A red gate and a silent one turned out to mean the same thing

I wrote up an incident here a couple of days ago: CI on a Rust project I maintain showed red for weeks, and not one of those jobs had ever started. The cause was billing. The scheduler was refusing each job in a couple of seconds, and the tooling kept telling me a log was missing when the truth was that no log had ever existed. That piece is here, and I'm not going to re-tell it.

What I left out of it is the part that turned out to matter more: the same mistake showed up twice more inside the same fortnight, in two places that had nothing to do with CI, and I only recognised the second and third because the first had just cost me thirteen days.

The shape is that red was doing double duty for two unrelated facts. "We checked and it failed" and "we never checked" were rendering identically. Those need telling apart, because the fix for one is to fix the code and the fix for the other is to fix the account, and conflating them wastes the outage twice: once by not addressing the real problem, and again by never learning whether the code was fine the whole time.

The same shape showed up a few days earlier in a completely different context, checking whether a set of recorded conversation turns had actually all been captured somewhere durable. The check used substring matching, a short six-character probe pulled from each turn, searched against the archive. Short probes threw false positives, six characters matches almost anything, so the probe length went up to twelve to cut the noise. That fixed the false positives and created a different problem nobody had anticipated: any turn whose distinguishing text was under twelve characters couldn't generate a probe long enough to search with at all, and those got silently counted as MISS, not found, indistinguishable in the output from turns that had genuinely gone unrecorded. Eighteen items out of sixty-five got reported as missing purely because they were too short to test, not because anything was actually wrong with them.

Both of these are the same mistake wearing different clothes: collapsing "I don't know" into one of the two answers you do have a column for. The fix, once you see it stated plainly, is almost embarrassingly simple and still easy to skip under time pressure: keep three values, not two. Hit, miss, and untestable, and never let untestable get counted as a miss just because your tooling only has two buckets to put things in. Tightening a threshold to kill false positives doesn't make the untestable cases go away, it just changes which direction they get miscounted in.

The last one is at my own expense, and it happened while writing this. I went to add a third example: a monitoring script meant to flag suspicious session activity, prompt repetition, oversized pastes, which I had on record as a fake that counted lines and returned PASS unconditionally. Before publishing I opened the file instead of trusting my own note. Seventy-nine lines. It parses each record, tests prompt content against two thresholds, and prints WARN with an itemised list when either trips. It also has a NO DATA branch for when there's nothing to read, which is the exact three-value discipline this whole piece is arguing for. My note was stale, and I was one paste away from publishing a confident claim about code I hadn't re-read.

Which is the actual ending, better than the one I'd planned. A stored verdict is a signal like any other. "I already checked this" is a green, and it goes stale the same way, and the only real defense is the same in every case: refuse to trust a signal, red, green, or silent, until you've confirmed it came from something that actually ran.

This is why the limitation lines in tracefold are copied character for character out of the constant the binary prints, with a test that goes red when the two drift apart. A verdict nobody re-derives is already going stale, mine included.