Two weeks ago I watched a pull request get merged with an AI-authored review comment that said "LGTM, nice defensive coding here" — attached to a diff that introduced an unguarded array index into a payment reconciliation job. The array could be empty. It has been empty, twice, in production, since. Nobody had tested the reviewer.
That's not a hypothetical. It's the exact failure mode buried in the discourse this week when developers started asking why AI "promoted every developer to reviewer" without anyone stopping to ask what happens when the reviewer itself is wrong. Here's my hot take: AI code review should never be a merge gate. It should be a colleague you consult, not a bouncer you trust with a badge — and if your CI pipeline doesn't already know that, your pipeline is broken, not your reviewer.
The steelman: AI review is genuinely useful
Let's not pretend the tools are bad. Point an LLM at a diff and it will catch things a tired human on their fourth PR of the day will miss: a missing null check, a docstring that no longer matches the function signature, an SQL query concatenated instead of parameterized, an off-by-one in a loop bound. It does this in seconds, for free, on every single PR, without getting bored or defensive. Teams report real velocity gains from using it as a first pass — fewer round-trips, fewer "please add a test for the empty case" comments a human reviewer would've had to type out by hand.
None of that is in dispute. The problem isn't that AI review is bad at reviewing. The problem is that nobody has decided who reviews it.
Why "who reviews the reviewer" isn't a philosophical question
Run the same diff through the same model twice and you can get two different verdicts. That's not a bug you can file — it's the nature of the tool. A human reviewer who flip-flops on the same code gets a performance conversation. An LLM that flip-flops gets nothing, because nobody's tracking it. There's no regression suite for the reviewer, no versioned behavior, no way to say "this is worse than it was last month," because you never wrote down what "good" looked like in the first place.
Compare that to everything else that gets to block a merge in a mature pipeline: a test suite has a known pass/fail history. A linter has a changelog. A coverage threshold is a number you can graph over time. When any of those regress, you can point at the exact commit that broke them. When your AI reviewer starts approving diffs it should reject, you find out from the incident, not from the pipeline.
That asymmetry is the actual hot take: it's not that AI review is untrustworthy, it's that teams have quietly given a non-deterministic, non-versioned, non-tested system the same authority as a deterministic one — and then act surprised when it behaves like the former.
The fix: keep the gate deterministic, keep the AI advisory
If you're running CircleCI, this is a fifteen-minute change to your config.yml, not a philosophical debate. Split "AI says this looks fine" from "this diff is allowed to merge" into two separate workflow jobs, and only put the deterministic one in branch protection's required checks.
version: 2.1
workflows:
pr-checks:
jobs:
- test-and-lint:
filters:
branches:
ignore: main
- ai-review:
filters:
branches:
ignore: main
requires: [] # runs in parallel, blocks nothing
jobs:
test-and-lint:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run: npm ci
- run: npm run lint
- run: npm test -- --coverage
- run:
name: Enforce coverage floor
command: npx nyc check-coverage --lines 80
ai-review:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run:
name: Post advisory AI review comment
command: node scripts/ai-review.js --post-comment --no-fail-on-issues
The --no-fail-on-issues flag on the AI review job isn't a suggestion, it's load-bearing: that job's exit code should never be able to fail the build. In GitHub or GitLab branch protection, only test-and-lint goes in the required-checks list. The AI reviewer gets to talk. It doesn't get to vote.
Test the reviewer the way you test everything else
Here's the part most teams skip entirely: if you're going to keep an AI reviewer around — and you should, it's useful — give it the same regression discipline you give your code. Build a small golden set of past PRs where you already know the right call: five that should have been rejected (the null-check miss, the SQL injection, the empty-array bug that got through), five that were legitimately fine. Store the diffs and the expected verdicts in the repo.
Then add a scheduled CircleCI pipeline — nightly or weekly — that replays that golden set through whatever model and prompt you're currently using, and fails loudly if the verdicts drift:
workflows:
nightly-reviewer-regression:
triggers:
- schedule:
cron: "0 6 * * *"
filters:
branches:
only: main
jobs:
- reviewer-regression-test
jobs:
reviewer-regression-test:
docker:
- image: cimg/node:20.11
steps:
- checkout
- run:
name: Replay golden PR set against current reviewer
command: node scripts/reviewer-regression.js --golden-set ./fixtures/reviewer-golden --fail-on-mismatch
This costs you maybe an hour of setup and a handful of pipeline credits a month. What it buys you is the one thing "AI promoted every developer to reviewer" discourse keeps skipping past: proof, over time, that the reviewer you're trusting today still catches the bug it caught six months ago, on the model version you're actually running now — not the one you tested when you first turned it on.
Landing the argument
The hot take isn't "don't use AI to review code." It's that review authority and review assistance are different jobs, and collapsing them into one CI step is how a genuinely useful tool quietly becomes an ungoverned one. Keep your merge gate boring, deterministic, and versioned — tests, lint, coverage, security scan. Let the AI reviewer talk as loudly as it wants in the PR comments. And once a quarter, put the reviewer itself through the same regression suite you'd demand of any other piece of code that gets to have an opinion about what ships. If you wouldn't merge a linter you'd never run against a test case, don't merge-gate on a reviewer you've never tested either.
