Eleven hours after the model-comparison post went up, a reader named Vinh Nguyen left a comment that unraveled its headline:
FATAL 0 against FATAL 0 is a tie in counts, but it is 29 items run once each. [...] Zero fatal out of 29 single trials is consistent with a true fatal rate up to about 10% at 95% confidence. [...] Running the whole exam five times per model and comparing fatal rates per item would tell you whether the tie is real or just the resolution of one pass, and it is the same order of cost as the run you already did.
He's pointing at a self-contradiction I published without noticing. My own post says, in the temperature section: "final scores should come from multiple runs, reported as a pass rate." And then the comparison itself — the whole "won by exactly one question" headline — sat on one pass per model.
No defense available. So I ran it.
The rerun
29 questions × 5 fresh runs × 2 models = 290 trials. Same harness, same unpinned temperature as the original — that variance is the thing being measured. Every raw answer sheet saved. All 290 completed this time (the original expensive-model run had dropped one question to a rate limit).
Haiku (cheap) Sonnet (3x price)
trials 145 145
FATAL 0 0
RISKY 1 1
MISSED 0 0
HARMLESS 4 5
clean 140/145 139/145
per run 29·28·27·28·28 28·28·28·28·27
Two things in that table killed my original framing.
The question that picked the winner was a coin flip
The original comparison came down to exactly one question — the lids. The customer writes: "10 boxes of the clear PET 300, and 5 bags of the white lids." The catalog has four lids to pick from:
CAP-300-W white lid, 300-neck (fits the PET-300 bottle)
CAP-300-BK black lid, 300-neck
CAP-500-W white lid, 500-neck (fits the PET-500 bottle)
CAP-500-BK black lid, 500-neck
"White" narrows it to two — but which neck? The clue sits in the same sentence: the customer is ordering PET 300 bottles, and nobody pairs 300 bottles with 500 lids. The expensive model read that context and confirmed the 300 white lid — and I called that the win. The cheap model asked "which neck?" instead.
Across five fresh runs, the expensive model never did that again. Zero confirms in five passes — it asked "which lid?" every single time. The only model that produced the winning confirm even once was the cheap one, in one run of five.
So the one-question gap my title stood on disappears the moment the exam is rerun. It was never a real ability gap between the two models. It was a coin flip that happened to land the expensive model's way on the one day I ran the exam once — and I published that single flip as the result.
The two RISKY trials landed exactly where the flag said
Each model produced one RISKY trial in 145 — the grade for "confirmed something ambiguous; right this time, fatal next time."
- Haiku, run 3, on "the usual, 3 boxes": two items in the order history both fit, and it presented only one of them as a candidate instead of both
- Sonnet, run 5, on the ultra-abbreviated "250 5": five products start with 250, and it confirmed one anyway
Both questions carry rule_decidable: false — the flag a reader talked me into adding: "can the reference data alone pin the answer down to exactly one?" On decidable questions, the run-to-run wobble drifted safe — extra clarifying questions, nothing worse. The dangerous direction — confirming what cannot be decided — showed up only on the flagged questions, at about 0.7% of trials per model.
The flag didn't just audit my answer key. It predicted where run-to-run variance turns dangerous.
So does the original conclusion survive?
The conclusion of that post — it's a tie on fatal errors, so use the cheap model — survives. What gets replaced is the evidence it stands on.
The old evidence was one pass: I ran the exam once, fatal errors came out 0 to 0, and the whole visible difference was that one lid question. The lid question is out — it was luck. What takes its place: fatal errors stayed 0 to 0 through 145 trials per model, and RISKY came out tied too, 1 to 1. By the rule of three, 145 trials tighten the fatal-rate bound on this exam from about 10% (at 0/29) to about 2%. On questions the model hasn't seen, I still have only 29 kinds — so that ceiling stays near 10%. The tie is no longer one lucky afternoon — it is a measured result. And on a tie, take the cheap one: the conclusion stands.
What doesn't stand is the margin I published. Clean counts wobbled between 27 and 29 per run for the cheap model and 27–28 for the expensive one. On any single pass, either model can rank first by a question or two. A one-pass margin of one question is below this exam's resolution — which is what the comment said before I spent a cent measuring it.
Takeaways
- A single pass ranks models with ±1–2 questions of noise. Don't publish a one-pass margin — mine didn't survive its own rerun
- "Zero fatal" means little until you say over how many trials — and of what. 0/29 caps this exam's fatal rate near 10%; five reruns (0/145) cap it near 2%, on this exam. New questions are a separate count. The reruns cost about as much as the first run
- Variance has grades too. On decidable questions it drifted safe (over-asking). The dangerous drift — confirming the undecidable — appeared only on questions pre-flagged as rule-undecidable
- The exam caught its author again. Not in the answer key this time, and not in the grader — in how confidently I read a single run
P.S. All 290 raw answer sheets, the 5× driver, and the aggregator are in the repo → github.com/ramses203/llm-test-harness
P.P.S. If you'd rather get new experiments by email: ramses203.substack.com/subscribe
