I Added a Fourth Model Mid-Run. It Changed What My Field Test Could Prove.

Latest release: v0.2.2 — Aug 29, 2026

I did something I usually try hard not to do in a field test. I changed the design after it had already started.

Halfway through validating AdversarialDebate, I realized the model set was too narrow to answer the most important question in the project. So I added a fourth model in the middle of the run.

That was messy. It wasted work. It made the corpus inconsistent for a while. It also turned out to be one of the best decisions in the whole release.

This post is about a lesson I trust far more now than I did before building this project: a field test is not just there to produce numbers. It is there to reveal whether your experiment can actually answer the question you think it is answering.

The Setup I Started With

I began with three models: GPT-4o-mini, Gemini 2.5 Flash, and DeepSeek-V3. That gave me three useful pairings — GPT + Gemini, Gemini + DeepSeek, and GPT + GPT as a homogeneous control. Three labs, two regions, one same-model control. Reasonable spread.

I ran the small corpus first, just 3 PRs, to validate the pipeline.

The diverse pair was ahead. The weak pair was struggling. The homogeneous control was doing something interesting. If I had stopped there, I would have told a clean story — and it would have been the wrong one.

The Problem Was Not The Data. It Was The Coverage.

The issue was not that the first three models were bad. The issue was that the experiment could only see part of the diversity spectrum. With those three models, the farthest useful pairing I had was US + China. I did not have a genuinely cross-continent pair that could show what happened at the far end of diversity.

The test could suggest whether diversity helped. It could not show whether maximum diversity behaved differently from moderate diversity. That is a major blind spot when the whole thesis is about pairing behavior.

I needed a fourth model — not because more models always make a benchmark better, but because this specific thesis needed one more corner of the matrix.

Why I Added Mistral

I added Mistral Small 3.2 because it gave me three new pairings instantly: GPT + Mistral, Gemini + Mistral, and DeepSeek + Mistral. That last one was the important one. DeepSeek + Mistral created the strongest diversity pairing in the run: China + EU.

Once that pairing existed, the experiment could finally observe a fuller spectrum: homogeneous, weak diversity, moderate diversity, strong diversity. That made the field test qualitatively better, not just quantitatively bigger.

The Thesis Changed Because The Matrix Changed

Without Mistral, the story would have been: diversity helps, US + China looks strongest, homogeneous is interesting but secondary.

With Mistral, the actual story became: diversity helps but not monotonically, weak diversity can be worse than no diversity, maximum diversity can create capitulation cascades, and the highest-scoring pair is not automatically the safest pair.

That is a much more useful conclusion.

What The Fourth Model Revealed

1. The strongest pair was stronger than expected. DeepSeek + Mistral dominated the full run: 0.982 average score, 97% verdict rate, 2,352 concessions. Without the fourth model, that pairing never exists, and the thesis looks weaker than it really is.

2. Maximum diversity had a failure mode. The same pair that looked best on paper also had the most dangerous behavior: 44 capitulation cascades, 65% capitulation rate within the pair. The fourth model did not just strengthen the thesis — it exposed the limit of the thesis. Without it, the release would have shipped a much cleaner and much less honest story.

3. The homogeneous control became more meaningful. Once the spectrum widened, GPT + GPT stopped looking like a curiosity and started looking like evidence that weak diversity was a specific failure mode, not just a weak result.

4. The field test stopped being a pipeline check and started being a product test. Three models were enough to validate that the system ran. Four models were what made it possible to learn something product-level about pairing strategy. That is the difference that matters.

The Mid-Run Change Was Messy

I do not want to romanticize this. Adding a model after the run started created real waste. By the time Mistral was added, GPT, Gemini, and DeepSeek had each already reviewed about 148 PRs. Mistral started late and only reviewed about 70 to 73 PRs. The Mistral-containing pairs could only use the 70 PRs that all four models had in common.

Roughly 228 single-pass reviews were effectively stranded. In money terms, that waste was about $0.15 — tiny. In experimental-design terms, it was a real smell. It meant the model set had not been planned tightly enough before the field test widened.

Why I Still Think It Was The Right Call

Normally I would view mid-run design changes as contamination risk. Here, the opposite was true. Not making the change would have contaminated the conclusion. The fourth model was not changing the thesis after the fact to make the result look better. It was correcting an experiment that was too narrow to test the thesis properly.

And because the run was cheap — the whole thing cost $0.53 — I could afford to make that correction. That is one of the most practical lessons from the whole project: cheap field tests let you notice a gap, fix it, and still finish the release. If each debate had cost even a few dollars, I would have been locked into the earlier design and probably would have shipped the wrong explanation with much more confidence.

What I Would Have Shipped Without The Fourth Model

Without Mistral, I probably would have shipped some version of: diversity helps, the US + China pair is strongest, no serious capitulation problem observed, homogeneous control is weaker than the best heterogeneous pair as expected.

That story would have sounded smart. It also would have hidden three real truths: GPT + GPT beat weak diversity, DeepSeek + Mistral revealed capitulation as a first-class failure mode, and the diversity curve was not linear at all.

The experiment gives you enough evidence to tell a story, but not enough evidence to know whether it is the right story.

What I Learned

Three models were not enough for this thesis. They were enough to validate the pipeline. They were not enough to map the behavior space.

A better field test is one that can embarrass your favorite story. The fourth model did exactly that. It made the product thesis stronger and less clean at the same time. That is a good sign.

Coverage beats premature scale. If the matrix is incomplete, more rows will not save you.

Cheap experiments create room for methodological honesty. The low cost of the run was not just a nice operational detail. It directly enabled mid-course correction.

The right question is not "did I have enough data?" The right question is "could this experiment see the failure modes and interaction shapes I actually care about?" That is a much better standard for agent evaluation.

What I'd Change Next

1. Lock the model set before scaling the run
2. Treat diversity strata as first-class design dimensions
3. Add another model only if it expands the behavior space meaningfully
4. Separate pipeline validation runs from product-thesis validation runs more explicitly

I still think adding the fourth model was the right move. I just do not want the next release to need that move in the middle of the run.

Questions I Want Developers To Argue With Me About

1. When do you decide an experiment is too narrow to answer its own thesis?
2. Would you rather ship a clean, incomplete benchmark or a messy, more truthful one?
3. How do you distinguish "more data" from "better coverage" in agent evaluations?
4. If a mid-run change reveals a real failure mode you would otherwise miss, is that bad methodology or responsible methodology?

I think most of us are still too willing to trust clean benchmark stories and not willing enough to ask what the matrix failed to observe.

AdversarialDebate is open source at github.com/deghosal-2026/adversarial-debate. Latest release: v0.2.2.

v0.2.1 Update: The Fourth Model Was the Key Variable

This article told the story of adding Mistral as a fourth model mid-run and how it changed the entire thesis. The v0.2.1 separating experiment confirmed what that story suggested but could not prove: Mistral was the key variable, not diversity.

I ran DeepSeek+GPT-4o-mini on the same 150-artifact corpus — two different labs, no Mistral. If the diversity hypothesis were correct, this pair should produce productive debate.

DeepSeek+GPT converged at 0.246 — statistically indistinguishable from two identical GPT models (0.273). The fourth model I added was not just another diverse model. It was a model with a specific training property that no other model in the test shared.

v0.2.2 Update: The Causal Gap in the Mistral Effect

The v0.2.1 separating experiment confirmed that Mistral drives productive debate — but not why. Two equally consistent causal mechanisms remain:

• Positive — Mistral's European RLHF approach creates a model that holds positions under pressure while conceding when evidence demands.
• Negative — non-Mistral models share instruction-tuning data and safety alignment targets; they rubber-stamp each other, and Mistral happens to break the symmetry.

The recommendation ("always include Mistral") does not change. But the next experiment shifts from "test more lab-diverse pairs" to "test RLHF-distance directly." An uncensored model paired with a strongly safety-aligned model would distinguish the two mechanisms.

Also in v0.2.2: the noise-floor baseline adds confidence intervals to every pair comparison. The D