As more teams wire AI review bots into CI, the failures shift from model quality to pipeline plumbing. A small team's review bot, running on MonkeyCode's free model access and a free server, started flooding every pull request with duplicate comments and drained a 10 million token budget in two days. The model was innocent; the fault lived in a single Git command that silently changed what the bot was allowed to see. This retrospective walks from symptom to root cause to fix, with a guard script that prevents the same failure.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The Symptom
The pipeline had run quietly for weeks. After a routine dependency bump, three things changed overnight:
- Every open PR received forty or more comments, many of them identical ("use optional chaining here" appeared on unrelated lines).
- The free token budget, expected to last a month, was nearly empty after two days.
- CI review times tripled because the model received far more context than any human reviewer would tolerate.
The team's first instinct was to blame the model. That instinct was wrong, and proving it wrong took less than an hour.
The First Suspect
The team ran the same review prompt manually against the same model with a clean, hand-written diff. The output was sensible, concise, and specific. The model had not regressed; the input had changed.
This is the core lesson of the retrospective: when an AI pipeline degrades, isolate the variable. The model is the last component to suspect, not the first, because the prompt and the context are the parts most likely to break silently.
Reproducing Locally
The team captured the exact input the bot had sent by adding a logging step to the pipeline. The log revealed the problem immediately: the diff file was 18,000 lines for a PR that changed 120 lines.
# capture what the bot actually sent
git log --oneline -5
git diff origin/main HEAD > /tmp/review.diff
wc -l /tmp/review.diff
# output: 18432 /tmp/review.diff
The diff contained changes from the entire upstream history, not just the branch's own commits. The bot was reviewing code the author never touched, and it commented on all of it.
Root Cause: Two-Dot Versus Three-Dot
The CI checkout used a shallow clone with depth: 1 to save time and disk space. The review script originally used the three-dot form, git diff origin/main...HEAD, which compares against the merge-base and shows only the branch's changes. That form requires the merge-base commit to exist locally, and a shallow clone does not have it.
Someone had "fixed" the resulting error by switching to the two-dot form, git diff origin/main HEAD, which compares the two branch tips directly. With a moving main branch, that command includes every commit main added since the branch diverged. The dependency bump had nothing to do with the model; it simply triggered a fresh clone that exposed the shallow-clone limitation.
# three-dot diff: needs the merge-base commit locally
git diff origin/main...HEAD # empty or fatal: bad object in a shallow clone
# two-dot diff: compares tips directly, includes unrelated upstream changes
git diff origin/main HEAD # thousands of lines the author never wrote
The Fix
The fix has two parts: compute the merge-base explicitly, and guard the context size before calling the model.
# fetch the base commit explicitly, then diff against it
git fetch origin main --depth 1
BASE=$(git merge-base origin/main HEAD || git rev-parse origin/main)
git diff "$BASE" HEAD > /tmp/review.diff
# guard against context explosion
LINES=$(wc -l < /tmp/review.diff)
if [ "$LINES" -gt 2000 ]; then
echo "diff too large ($LINES lines); skipping review" >&2
exit 0
fi
# rough token estimate before sending anything
BYTES=$(wc -c < /tmp/review.diff)
EST_TOKENS=$((BYTES / 4))
echo "estimated tokens: $EST_TOKENS"
The guard matters more than the exact command. A pipeline that silently sends 18,000 lines to a model will eventually burn any token budget, free or paid. The team also added a log line that records the diff size and the base commit SHA on every run, so the next regression is visible in the CI output instead of in the token balance.
A Reusable Debugging Checklist
The same investigation pattern applies to any AI-assisted pipeline failure:
- Reproduce outside the pipeline. Run the same prompt against the same model with a clean input to isolate the variable.
- Capture the exact input. Log the prompt, the diff, and the metadata before the model call.
- Inspect the data before the model. Check diff size, base commit, file count, and token estimate.
- Change one variable at a time. The dependency bump was a red herring; the real change was the clone depth.
- Add guards for context size. A hard limit on diff lines converts a silent budget drain into a loud, early failure.
Limitations
This approach assumes the failure lives in the pipeline, not in the model. Teams without any logging should add capture steps before debugging anything else. The free tier suits small and medium repositories, but monorepos with huge diffs will hit the guard and skip reviews by design. Shallow-clone behavior varies across Git versions and CI providers, so the merge-base fallback should be tested on the exact provider in use. Finally, an automated review, even with a correct diff, is not a substitute for human review on security-critical paths.
Closing
The model did not get worse; the diff did. A two-dot Git command turned a focused review bot into a noisy commentator, and the token budget paid for the mistake. Teams that want to reproduce this failure mode can run the guard script against their own repositories, and the free tier keeps the experiment cheap. The next time a review bot behaves strangely, check the input before blaming the model.
