Every stage of your delivery pipeline has tests except the stage that judges the code itself. Your AI reviewer is the only component whose output is trusted without assertions, which makes it the largest unverified surface in modern CI. Pinning a prompt and a model version does not pin behavior, because upstream updates change outputs silently and nobody notices. The fix is a planted-bug corpus that measures reviewer drift on a schedule you can actually afford.
Drift Is Invisible Until a Bug Ships
Application code fails loudly when a test breaks, but review prompts fail quietly when a model update shifts their judgment. A reviewer that caught every null dereference in January can miss half of them in June while still producing confident, well-formatted comments. Teams notice only when a defect reaches production, and by then the review log looks identical to the one from the good months. Recent DEV discussions have asked who tests the reviewer, and the honest answer is almost nobody.
The root cause is that review output has no oracle. A linter has a rule set, a unit test has an assertion, and a human reviewer has a second human in code review. An AI reviewer has only the prompt, and the prompt is evaluated by a model that changes underneath it. If you cannot state what the reviewer must catch, you cannot detect when it stops catching it.
Pinning the Config Does Not Pin the Model
Pinning your review configuration to a specific model version is necessary but insufficient, because the identifier you pin is a moving target. Providers ship updates, adjust quantization, and alter default parameters without changing the version string you recorded. The configuration stays stable while the behavior underneath it drifts, which means your regression suite must assert on outputs rather than versions. Treat the reviewer like a dependency with behavioral contracts, not like a settings file.
This is the gap between the common advice to pin your AI review configuration and the reality of model deployment. Pinning gives you a reproducible starting point, but it gives you nothing to compare against when the model changes. What you actually need is a baseline of expected behavior and a cheap way to re-measure it on a regular cadence.
The Artifact: A Planted-Bug Review Corpus
The practical fix is a small corpus of synthetic pull requests, each containing one planted defect with a known bug class and an expected verdict. You build it once from your own incident history, then run the reviewer against it on a schedule. Here is a minimal corpus entry:
{
"corpus_version": "2026-08",
"min_acceptable_rate": 0.8,
"cases": [
{
"id": "null-deref-01",
"bug_class": "null_dereference",
"should_catch": true,
"file": "src/cache.js",
"before": "const entry = cache.get(key);\nif (entry.expiresAt < Date.now()) {",
"after": "const entry = cache.get(key) || null;\nif (entry.expiresAt < Date.now()) {"
}
]
}
The before and after fields render as a synthetic diff, and should_catch records the verdict a competent reviewer must reach. Include clean cases too, because a reviewer that flags everything is as useless as one that flags nothing. The scoring script below is a minimal template; adapt the review() function to your own tool.
import json
import subprocess
import sys
def review(prompt: str) -> str:
"""Adapt this to your reviewer's CLI or API."""
return subprocess.run(
["your-reviewer", "review"],
input=prompt, capture_output=True, text=True
).stdout
def build_prompt(case: dict) -> str:
return (
f"Review this change to {case['file']}.\n"
f"Before:\n{case['before']}\n"
f"After:\n{case['after']}\n"
"Report only concrete defects with file and line."
)
def main(corpus_path: str) -> None:
corpus = json.load(open(corpus_path))
caught = 0
for case in corpus["cases"]:
output = review(build_prompt(case))
flagged = case["id"] in output or case["bug_class"] in output
caught += int(flagged)
print(f"{case['id']}: {'caught' if flagged else 'MISSED'}")
rate = caught / len(corpus["cases"])
print(f"catch_rate={rate:.0%}")
if rate < corpus["min_acceptable_rate"]:
sys.exit(1)
if __name__ == "__main__":
main(sys.argv[1])
The script exits nonzero when the catch rate drops below your threshold, so the regression run plugs directly into CI. The verdict check is intentionally naive, and a real implementation should parse the reviewer's structured output instead of searching for an ID string. The template contains no vendor-specific calls, because the corpus should outlive any single reviewer tool.
Why Free Infrastructure Changes the Habit
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
A regression suite only becomes a habit when its marginal cost is zero, because teams skip paid checks in the weeks when budgets are tight. MonkeyCode's free model access and free server option remove that excuse by making the weekly corpus run a background process rather than a purchasing decision. The economics matter because reviewer drift is slow, and a quarterly paid audit will miss the week when the catch rate quietly halves. A free weekly run turns drift detection into something you notice before the bug ships, not after.
The Workflow in Five Steps
- Build a corpus of ten to twenty planted bugs from your recent incident history, one defect per case, plus a few clean diffs.
- Run a baseline and record the catch rate for each bug class so you know what the reviewer actually detects today.
- Schedule the corpus as a weekly cron job that fails when the catch rate falls below your threshold.
- When it fails, diff the current review output against the baseline to decide whether the prompt or the model changed.
- Prune the corpus quarterly, because bug classes your team no longer writes will inflate your confidence.
The fifth step matters more than it looks. A corpus that never changes measures history rather than current risk, and stale cases produce a catch rate that feels reassuring while the reviewer drifts on the bugs you actually ship. Maintenance is part of the artifact, not an afterthought.
Who Should Not Use This
Planted bugs are not real bugs, and a corpus of fifteen cases produces percentages with wide error bars, so treat the number as a trend indicator rather than a benchmark. Teams without an existing AI review step should not build this harness first, because measuring a reviewer you do not use is pure overhead. Small repositories with few weekly PRs will get more signal from a second human reviewer than from a drift detector. The corpus also catches failures of detection, not failures of judgment, which is a real limit of any output-based check.
If you already run an AI reviewer, the cheapest correctness test you can add this week is ten planted bugs and one cron job. The reviewer judges your code, and it is time something judged the reviewer.
