The rule of three for evals says zero failures in N runs is a count, not a rate. With 0 failures in N independent runs, the exact 95% upper bound on the true failure rate is 1 - 0.05^(1/N), which 3/N approximates. After 100 clean runs you still cannot rule out a 2.95% rate, about 1 in 34.
Here is the reading that bites you. Your eval harness runs the agent 100 times, prints "0 failures," and the tile goes green. Someone screenshots it into the launch thread. The unspoken translation is "the failure rate is zero." It is not what the data says.
I wrote a small script to make the gap concrete, so I ran a real gate over 200 deterministic agent runs first, counted honestly, and got the dashboard everyone trusts:
gate: spend<=budget over 200 deterministic agent runs
observed failures: 0 (distinct scenarios: 200)
naive point rate : 0.00% binomial SE: 0.00 pp
naive 95% CI : [0.00%, 0.00%] <- zero width: false certainty
Look at the standard error. For a zero count the binomial SE is sqrt(0*1/200), which is exactly 0, so the naive 95% interval collapses to [0.00%, 0.00%]. A zero-width confidence interval. The math is telling you it is completely certain, from 200 samples, that the true rate is precisely zero. That is obviously wrong, and it is the exact shape of every "all green" board I have ever trusted too much.
TL;DR
- "0 failures in N runs" is an observed count, not a rate. The naive binomial SE of a zero count is 0, which is why a green board looks like proof and isn't.
- The honest number is the one-sided upper bound. With 0 failures in N runs, the 95% upper confidence limit on the true failure rate is
1 - 0.05^(1/N). The rule of three,3/N, approximates it and rounds the risk slightly up. - At N=30 the bound is 9.50% (about 1 in 11). At N=100 it is 2.95% (1 in 34). At N=1000 it is 0.30% (1 in 334). Zero failures in 30 runs is compatible with a 1-in-11 true failure rate.
- To rule out a 0.1% rate at 95% you need about 2995 clean runs, not 50. "We ran it fifty times" and "we are 99.9% sure" are different sentences.
- Correlated runs make it worse. 200 runs that are really 20 scenarios replayed ten times give an effective N of 20, and the bound jumps from 1.49% to 13.91%.
What this is and is not. The gate in the script is benign by construction: I built it so it observes zero failures, because a clean eval is exactly the case I want to talk about. The bound over that count is a theorem, not a fitted effect. There is no tuned constant anywhere in this that, if I nudged it, would flip the conclusion. The only inputs are the count (0), the number of runs (N), and the confidence level. I show that explicitly in section 6 of the run, because "the number rides on a fixture I chose" is the failure mode I most distrust in this kind of post.
Why zero observed failures is not a zero failure rate
Counting is not estimating. If you flip a coin twice and it lands heads twice, you have observed a 100% heads rate, and nobody sane concludes the tails face does not exist. Two runs is not enough to rule much out. The same logic holds when the observed count is zero and the sample is 30, or 100, or 1000: a rare failure can hide behind any finite streak of clean runs, and the question is not "did it fail" but "what rate is still consistent with what I saw."
The naive dashboard answers a different, easier question. It reports the point estimate, 0/N = 0%, and its standard error, which for a zero count is zero. So the interval has no width and the board is green with no caveat. This is the statistical version of a monitor that has been silently dead for three weeks: a check that always returns "fine" and a check that has stopped running produce the identical screenshot. Silence is not the same as health, and a zero-width interval is not the same as certainty. This is the whole of the tracking-is-not-control idea in one number: the board is counting what already happened and calling the absence of an event a guarantee about the next one.
So throw out the point estimate and ask the honest question instead. Given that I saw zero failures in N runs, how high could the true failure rate be and still make a run this clean plausible?
The rule of three, and why it is exact at zero
The upper bound falls straight out of the binomial, no approximation required. The only way to observe zero failures in N independent runs is for every run to pass, and the probability of that is (1-p)^N. The Clopper-Pearson upper confidence limit is the failure rate p that makes an outcome this clean just barely believable at your confidence level: set (1-p)^N = 0.05, solve, and you get p = 1 - 0.05^(1/N). Because the binomial tail at x=0 is a single term, this is exact, not a normal approximation that wobbles at small N. One word of caution on exact: it means the coverage is guaranteed to be at least 95%, not that this is the tightest interval available. Clopper-Pearson sits at the conservative end of the family (Wilson and Jeffreys intervals run a hair narrower), and I take the conservative end on purpose when the thing I am bounding is a rare and expensive failure.
The rule of three is the pocket version. The true constant is -ln(0.05) = 2.9957, and 3/N is that rounded to a number you can do in your head. Here is the exact bound beside the rule of three across N, all assuming zero observed failures:
N | exact 95% upper | 3/N | ~1 in K | exact-3/N (pp)
------------------------------------------------------------------
10 | 25.89% | 30.00% | 4 | -4.113
30 | 9.50% | 10.00% | 11 | -0.497
50 | 5.82% | 6.00% | 17 | -0.184
100 | 2.95% | 3.00% | 34 | -0.049
300 | 0.99% | 1.00% | 101 | -0.006
1000 | 0.30% | 0.30% | 334 | -0.001
Read the N=30 row. Thirty green runs, zero failures, and the true failure rate could still be 9.50%, roughly one bad run in eleven, without your eval being unlucky at all. The 3/N column sits a hair above the exact bound in every row (the gap is negative because exact minus 3/N is small and negative), so the mnemonic slightly over-states the risk. That is the direction you want a shortcut to err. By a thousand runs the two columns read the same to two decimals (0.30% and 0.30%), and the rule of three stops being an approximation worth distinguishing.
None of these numbers are typed in. They come out of 1 - 0.05^(1/N) in the script, and I cross-checked the row values against an independent recompute before I quoted them, because quoting a bound I hadn't actually run would be its own kind of lie.
How many clean runs actually buy a target?
Flip the question around. If you want to be able to rule out a given failure rate, how many clean runs does that take? Invert the bound, 1 - 0.05^(1/N) <= target, and you get N >= ln(0.05)/ln(1-target):
target = the failure rate you want to be able to rule out at 95%.
bound <= 5.00% needs N = 59 clean runs (exact 58.4)
bound <= 1.00% needs N = 299 clean runs (exact 298.1)
bound <= 0.30% needs N = 998 clean runs (exact 997.1)
bound <= 0.10% needs N = 2995 clean runs (exact 2994.2)
This is the section that changes how I plan an eval budget. Ruling out a 1% failure rate takes 299 clean runs. Ruling out 0.1%, the neighborhood where you'd actually feel safe shipping an agent that can move money or delete data, takes about 2995. For anyone paying a judge model per eval item, those Ns are a line straight to a bill, and the honest version of "prove it's safe" is expensive in a way the green tile hides. If your rare-but-catastrophic action got 50 clean runs, the most you can claim at 95% is that its failure rate is probably under 6%. That is not a number I would put next to "delete the production database."
The confidence level is a knob too, and it is worth seeing that "three" is not sacred:
conf | -ln(alpha) | name | upper@N=100
----------------------------------------------------
90% | 2.3026 | rule of 2.3 | 2.28%
95% | 2.9957 | rule of 3 | 2.95%
99% | 4.6052 | rule of 4.6 | 4.50%
The rule of three is specifically the 95% case, because -ln(0.05) rounds to 3. Want 99% confidence in your upper bound? That is the rule of 4.6, and at a fixed N it gives a looser bound (4.50% versus 2.95% at N=100), because more confidence over the same data costs you sharpness. Pick the confidence deliberately instead of inheriting 95% because a stats class used it.
The trap in the number: correlated runs shrink your N
Everything above assumes the runs are independent. In agent evals they very often are not, and the failure is quiet. You run the same 20 scenarios with 10 seeds each and call it 200 runs. The board says 200; the informative count is 20. Common-mode inputs, the same prompt replayed, the same fixture, the same upstream data, inflate N on the dashboard and leave the real bound where the distinct cases put it.
nominal runs: 200 distinct scenarios: 20 observed failures: 0
bound if you count nominal N=200 : 1.49% (~1 in 67)
bound at effective N=20 : 13.91% (~1 in 7)
Same zero failures. Same 200 lines in the log. But the bound you're entitled to is 13.91%, about 1 in 7, not 1.49%. This one cuts against the reassuring direction, which is exactly why I put it in: replays and shared inputs shrink your effective N, and a smaller N only ever loosens the bound, never tightens it, since 1 - 0.05^(1/N) falls as N grows. A harness that counts replays as independent runs is quietly reporting a safety you don't have. If you can't argue your runs are independent draws from the distribution you care about, use the count of distinct cases, not the count of executions.
What the bound does not tell you
Three honest limits, because the number is narrow on purpose.
It is about frequency, not identity. 1 - 0.05^(1/N) bounds how often a failure could happen. It says nothing about which failure, or how bad. A 1% rate of "off-by-a-cent" and a 1% rate of "wired the whole treasury" are the same number here and nothing alike in practice. The bound does not replace worst-case analysis or reading results by severity class; it sits next to them.
It bounds the rate on your eval distribution, not on production. If your 100 clean runs don't cover the inputs production will throw, the bound is honest about the runs you did and silent about the ones you didn't. A clean eval on the wrong distribution is a different problem, closer to contamination in the eval itself.
It is an upper bound, not an estimate. Zero failures does not mean the rate is 3/N. It means the rate is probably below 3/N. The true rate could be much lower, or, with bad luck, right at the bound. The point is not to name the rate. It is to stop pretending a clean run named it zero.
Why there is nothing here to fake
I have dropped enough of my own drafts to be suspicious of any post where the punchline sits inside a constant the author picked. So I built this one to have no such constant. The conclusion depends on three inputs and nothing else: the observed count, N, and the confidence level. Section 6 of the run makes the point by brute force:
re-run the gate: failures still 0 of 200 -> bound 1.49%
Shuffle which scenario the gate sees first, run it twice, reorder the whole fixture: the count stays 0, so the bound is byte-for-byte the same. There is no effect size to permute, no ratio whose denominator is secretly doing the work. That is the opposite of the class of eval post that gets dropped, where the "finding" is really a fixture setting in disguise. Here the only thing I observe is a count, and the only thing I do to it is bound it. If you distrust the bound, you are distrusting the binomial, which is a fine thing to argue about and a very different argument from "he tuned the numbers."
The whole thing runs in a second: Python 3.13.5, stdlib only, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. Picking the right statistical test is a sibling problem, and I wrote up the paired-data version of it in why your A/B eval probably needs McNemar, not a two-proportion SE. Both belong to the same family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.
What does your board show
The next time a tile goes green on "0 failures in N runs," do the one-line translation out loud. Zero in 30 is a rate that could be 1 in 11. Zero in 100 is a rate that could be 1 in 34. Neither is zero, and the dashboard will not tell you that unless you make it.
I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I don't have a clean answer to: when your eval last went green, did anyone write down the N, or did "0 failures" quietly get filed as "the failure rate is zero"? I want to know how many runs your team treats as enough, and why that number and not ten times it.
AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the gate, and the reading are mine. Every output block is pasted from one real local run on 2026-07-22. zero_failures.py sha256 d3b3f205909640fc…, run output sha256 cc7aca328fdd7236…. Recompute: python3 zero_failures.py prints the same bytes, exit 0, empty stderr, stdlib only.
