Most AI Second Opinions Are Fake. I Built a Two-LLM Review Engine to Prove It.

Most AI "second opinions" are fake.

Not because there is no second model. Because the second model usually sees the first model's framing, assumptions, and conclusion before it does any real reasoning of its own. At that point, the system looks like review, but it is structurally biased toward agreement.

That bothered me enough to build AdversarialDebate, an open-source review engine where two LLMs analyze the same artifact in isolation, commit their reviews independently, and only then debate each other point by point.

I wanted to answer one practical developer question:

If you actually force independence, do you get better review quality, or do you just get a more expensive version of the same answer twice?

So I field-tested it on 70 real pull requests from projects like Kubernetes, Prometheus, Go, Rails, and Django. I ran 411 debates across 6 model pairs for a total cost of $0.53.

The result was more interesting than the original thesis:

• independence really does matter
• model pair choice matters more than I expected
• disagreement is often more useful than forced consensus
• and the hardest part of shipping a system like this is not getting two models to speak, but getting them to disagree honestly

This is the first post in a short series about what worked, what almost failed, and what I would change in v0.2.0.

If you build agent critics, verifier loops, judge systems, or multi-model review flows, this is the part I think matters most: independence is not a prompt trick. It is a system property.

Before I get into the build, here is the shortest honest summary of the field test:

That mix matters. This was not a system that always converged. It was a system that usually either argued productively or told me the disagreement was still real.

Why Most Second Opinions Collapse Before They Start

The standard pattern in AI tooling is easy to recognize.

One model writes a review, classification, summary, or recommendation. A second model is then asked to validate it. But the validation prompt usually includes the first model's answer. Sometimes that's framed as context sharing. Sometimes as critique. Sometimes as self-reflection.

In all three cases, the same thing happens: the second model is no longer reasoning independently.

That matters because the failure mode we're trying to catch is often not missing information. It is prematurely accepted reasoning.

The PR that looks safe because the first explanation was clean.

The migration plan that sounds fine until someone independent asks about rollback.

The incident hypothesis that gets accepted because it is coherent, not because it is correct.

Human teams already know how to handle this. We use independent review, delayed judgment, and structured challenge. If two senior engineers review a risky change, we do not force one to read the other's opinion before writing their own first. We know that contaminates the review.

AI systems mostly ignore that lesson.

I built AdversarialDebate because I wanted an engine that preserved the useful part of multiple reviewers: separate judgment before interaction.

The One Rule I Refused to Compromise On

The non-negotiable rule in the project is simple:

Reviewer B cannot see reviewer A's output until reviewer B has fully committed its own review.

Not "should not." Not "please avoid bias." Cannot.

That meant I had to build independence into the architecture rather than rely on prompt wording.

The v0.1.0 loop looks like this:

1. A PR diff is normalized into a reviewable artifact.
2. Reviewer A analyzes it in isolation.
3. Reviewer B analyzes the same artifact in isolation.
4. Both reviews are persisted separately.
5. The revelation gate opens only after both are committed.
6. Each side responds to the other's claims in bounded debate rounds.
7. The system produces either a joint verdict or a disagreement report.

That last output matters more than it first sounds.

I did not want a system that forced agreement because agreement is often the least trustworthy output in multi-agent systems. If both sides still disagree after structured exchange, the useful result is not a synthetic summary pretending everything is resolved. The useful result is a report that says:

• what is still disputed
• why it is disputed
• what evidence mattered most
• what would resolve it

That is much closer to how real engineering decisions get made.

What v0.1.0 Actually Ships

The product in v0.1.0 is intentionally narrow.

It ships as:

• Python library + CLI
• PR-review focused
• local-first SQLite transcript storage
• model-agnostic via adapters
• evidence- and schema-driven rather than freeform chat

The core components are straightforward:

• Input normalizer for PR diffs and metadata
• Independent reviewer engine with separate sessions
• Debate controller with bounded rounds
• Evidence tracker for claims, objections, concessions, unresolved points
• Synthesis layer for verdicts or disagreement reports
• Audit log for full transcript lineage

I kept the MVP lean on purpose. No hosted service. No UI. No GitHub Action. No attempt to solve every review domain at once.

The only question v0.1.0 needed to answer was this:

Can two isolated LLM reviewers surface something a single reviewer would miss, or at least produce a disagreement artifact that improves a human call?

If that answer was no, the project should die early.

How I Tested It On Real PRs

I did not want a benchmark made of toy examples or curated one-liners.

So I used real public PRs and documented outcomes. The final corpus covered 70 PRs across repositories like:

• kubernetes/kubernetes
• prometheus/prometheus
• golang/go
• django/django
• rails/rails

I tested 4 models:

• GPT-4o-mini
• Gemini 2.5 Flash
• DeepSeek-V3
• Mistral Small 3.2

And 6 model pairs:

• GPT + Gemini
• Gemini + DeepSeek
• GPT + Mistral
• Gemini + Mistral
• DeepSeek + Mistral
• GPT + GPT as a homogeneous control

That produced:

• 411 debates
• 70 real PRs
• 6 model pairs
• $0.53 total cost

The low cost matters. It let me run enough tests to discover uncomfortable truths instead of stopping at the first clean demo.

First Proof: The Debates Were About Real Problems

For 49 of the 70 PRs, I had a documented ground-truth outcome: revert reason, fix description, or advisory context.

I compared debate claims against those outcomes.

The top-line result:

• 49 of 49 PRs with documented outcomes had at least one debate claim that matched the actual cause
• 6,145 of 7,612 claims matched
• 1,467 were partial matches
• 0 were no-match

That does not mean the system is magically correct. It does mean the debates were not wandering off into irrelevant nonsense. Even weak pairs were usually arguing about the right things.

This was the release bar that mattered most to me.

I did not need proof that debate solved review. I needed proof that independence plus debate could produce inspectably useful review artifacts on real code, not just neat demos.

It cleared that bar decisively.

Just as important, the engine was not forcing verdicts to get there:

• 152/411 debates reached verdict
• 259/411 debates stayed disputed
• 1/411 was theater

That is the shape I wanted. If everything converges, I stop trusting the system. If everything deadlocks, the system is too expensive to justify. This landed in the middle: enough verdicts to be useful, enough disagreement to be believable.

The Surprise: The Pair Mattered More Than the PR

The strongest field-test signal was not the PR itself. It was the pair.

Here was the ranking by average convergence score and verdict productivity:

The most diverse pair and the least diverse pair were not just a little different. They behaved like different systems.

On the same PRs, with the same engine, with the same prompts, one pair reached verdict after verdict while another stalled almost every time.

Here is what that looked like on real PRs:

That is the part I keep coming back to. Same artifact. Same engine. Same debate rules. Completely different behavior depending on the pair.

That was the first big learning I did not fully expect: for adversarial review, model pairing is a first-order product decision, not a tuning detail.

If a user picks the wrong pair, they may conclude the system is mediocre even if the architecture is sound.

Why This Should Matter To Anyone Building Agent Systems

If you're building with multi-agent patterns, this should feel uncomfortably familiar.

We often think in terms of orchestration first:

• planner vs critic
• reviewer vs judge
• proposer vs verifier
• generator vs evaluator

Those roles matter. But this project pushed me to think harder about something lower in the stack:

What kinds of models can productively disagree?

That is a more specific question than "which model is smarter?"

An adversarial pair needs at least three things:

1. enough difference to spot different risks
2. enough overlap to evaluate each other's evidence
3. enough willingness to concede when outmatched

Miss any one of those and the debate quality degrades.

Too similar, and both sides stubbornly rebut forever.

Too different, and one side capitulates instead of engaging.

Too unstructured, and both sides perform disagreement without moving state.

I think a lot of agent builders are going to rediscover this the hard way as they move from demos to systems that people actually depend on.

What Worked Better Than I Expected

Three things worked better than I expected.

1. The isolation invariant held

The biggest architectural risk was fake independence. If reviewer sessions leaked context or if the revelation step happened too early, the whole product thesis would collapse.

The transcript structure and the control behavior held up under tests and field runs. The homogeneous GPT+GPT pair helped here too. If leakage had been happening, identical-model behavior would have looked far more suspiciously aligned.

2. The disagreement report turned out to be a feature, not a fallback

I went in thinking verdicts would be the hero output.

By the end of the field test, I trusted structured disagreement more than fast agreement. The would_resolve_if field is one of the best parts of the system. It gives the human an actionable next question instead of fake certainty.

The shape of the artifact is simple:

verdict: DISPUTED
unresolved:
  - claim: "Expand/contract migration is safe for pr-482"
    agent_a: sufficient as written
    agent_b: needs lock-timeout analysis on orders table (~40GB)
    would_resolve_if: load test at production row count

That is much closer to what a developer or reviewer actually needs. Not just "the agents disagree," but what they disagree about and what evidence would settle it.

3. The economics were better than expected

411 debates for $0.53 changed how I think about evaluation scope. Cheap enough to test real behavior. Cheap enough to discover methodology mistakes. Cheap enough to iterate on design instead of protecting a fragile benchmark from bad news.

What Went Worse Than Expected

This release earned its keep mostly because the field test was willing to embarrass the original story.

1. I underestimated how much pairing would dominate outcomes

I expected some pair effect. I did not expect pairing to shape the system more than the PR itself. That means user guidance for pair selection is a bigger product requirement than I originally thought.

2. I discovered too late that some "good" outcomes were low-quality outcomes

The strongest pair by convergence later turned out to have a major quality caveat: capitulation cascades. One side would concede everything in round 1 without real rebuttal. That deserves its own article because it changes how aggregate success metrics should be interpreted.

3. Release hardening surfaced a real packaging bug

The clean-venv install smoke test caught a runtime dependency problem: pyyaml was only in dev dependencies even though the scripted reviewer path imported it at runtime. That is exactly the kind of issue release checks are supposed to catch, and exactly the kind of thing I should have detected earlier.

That was a good reminder: passing CI in the repo is not the same thing as being ready for pip install in the wild.

It is also why I trust release checklists more than "it worked on my machine" confidence. The clean-venv smoke test caught something the happy-path build never would have exposed.

The field test had the same effect on the product itself. It found 14 issues during pipeline development. Twelve were real bugs I fixed. Two were not bugs at all; they were valid findings that contradicted my expectations. That is exactly what a useful release process should do.

What I Would Change In v0.2.0

If I were restarting v0.1.0, I would change three things.

1. I would define pair strategy earlier

The current result makes pair selection look obvious in hindsight, but I only learned the real diversity story by running the field test. In the next cycle, pair strategy needs to be a documented product surface, not just a research observation.

2. I would treat verdict quality and verdict frequency as separate metrics from day one

A system can produce lots of verdicts for bad reasons. That distinction should have been first-class in the initial scorecard.

3. I would tighten the release docs earlier

The product idea, WBS, and field-test thinking were strong early. Some of the release-facing docs and artifact paths lagged behind the implementation and needed cleanup close to ship. That is survivable, but it increases release friction for no good reason.

Where The Project Goes Next

The next version should not just add more adapters. It needs to deepen the truthfulness of the debate itself.

The highest-value directions right now look like:

• separating genuine verdicts from capitulation verdicts in scorecards
• adding better pair guidance based on observed behavior
• running more flakiness sweeps on additional pairs, especially homogeneous ones
• testing beyond PR review into adjacent high-stakes reasoning domains
• improving report quality so unresolved disagreement is even more actionable

I do not think the lesson of v0.1.0 is "multi-agent debate solved review."

I think the lesson is narrower and more useful:

if you want a second opinion from AI, independence must be enforced, not implied.

Without that, you're mostly building agreement with extra steps.

With it, you at least have a chance to get real adversarial pressure on a conclusion before a human trusts it.

That is a much smaller claim than the average AI launch post makes. It is also a much more defensible one.

Questions I Want Developers To Argue With Me About

I expect comments on this one, because the whole premise invites argument. Good.

These are the questions I think are worth debating:

1. If you use a second model today, how do you know it is not contaminated by the first model's framing?
2. Should disagreement artifacts be treated as first-class outputs in developer tooling, instead of failure cases?
3. If model pairing changes behavior this much, should agent products expose pair strategy directly instead of hiding it behind a generic "review mode"?
4. What is the right standard for proving independence in multi-agent systems: transcript structure, architecture guarantees, or something stronger?

If you've built critique loops, judge systems, planner/critic agents, or verifier flows, I want to hear where you've seen fake independence show up.

Because I think this problem is much more common than most agent demos admit.

AdversarialDebate is live here:

• Repo: https://github.com/deghosal-2026/adversarial-debate
• PyPI: https://pypi.org/project/adversarial-debate/
• Full field-test report: https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.1.0/FIELD_TEST_REPORT_full_corpus.md
• Learnings: https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.1.0/learnings.md

Next in the series: the most important engineering lesson from the whole build, where three prompt lines dropped debate theater from 89% to 0.2%.