The Negative Control That Tested The Wrong Field

Originally published on hexisteme notes.

A disk loader got built for an artifact that had a writer and no reader — the build delegated, the job of proving its most important guarantee left to me: break something, watch the check turn red, and don't take anyone's word for whether it actually did. I described a related failure mode in an earlier piece, The check that cannot fire — checks whose thresholds were miscalibrated for the scale of the input, so they never tripped no matter what you fed them. That is not what happened this time. This control fired on schedule, every time. Green went to red exactly when I expected it to.

The catch took two more experiments to surface: firing is not the same as firing at the right thing. A control can pass every observable test of "does it react" while staying structurally blind to the one property it was built to protect.

A negative control flipping from green to red does not by itself prove it targeted the property it exists to guard. A bypass collapsing next to that property produces the identical signal.

A writer with no reader

The artifact was a plan object serialized to JSON by a writer that had shipped without a matching reader — a write-only artifact nobody could read back. The spec for the loader was a general restorer rather than a hand-written field list: walk the dataclass's declared type hints and rebuild each field recursively, so that a field added later can't be silently dropped by a loader that never learned about it.

The restorer had one structural problem to solve. Python's standard dataclass-to-dict conversion flattens every tuple into a list and leaves nested dataclasses as plain dicts. One of the object's fields was a tuple of tuples of floats — a range list, doubly nested — and its field tree held seven distinct nested dataclass types, one of them reachable only two levels down, nested inside another nested field rather than sitting directly on the plan. A naive Cls(**raw) reconstruction would silently accept every one of those as the wrong type: lists standing in for tuples, dicts standing in for objects, and the mismatch invisible until something downstream did an equality comparison and got False for no comprehensible reason.

I checked that this mattered by running a naive loader — no tuple restoration — through the same recursive type-checker the real loader had to pass. Eight declared-type violations, fifty-six stray lists where tuples belonged. The proper loader, on the same input: zero of both. Same checker, opposite verdict on two inputs that differed only in the one thing under test — which is what makes that checker informative rather than tautological. That distinction, tautological versus informative, turned out to be exactly the axis the rest of the story runs on, just applied to a different check.

Three spots in the restorer needed to fail loudly instead of quietly: a mismatch between the raw JSON's key set and the dataclass's declared fields (a field renamed or dropped between versions), a union type with more than one non-None branch (the restorer refuses to guess which one a raw value belongs to), and a fixed-length tuple whose element count didn't match its declared type. That third one is worth a beat, because the natural implementation is a bare zip(value, declared_types), and zip stops at the shorter sequence without complaint — feed it three elements where two were declared and one disappears in silence. The fix compares the two counts before any zip runs and raises, naming both, if they disagree.

The control that fired — at the wrong thing

With the loader built, the strongest available proof of correctness was end-to-end: feed the loaded object back through the downstream pipeline and diff the resulting JSON, byte for byte, against a live, already-approved reference output. Call that check B. On the unmodified round trip it passed — 8,576 bytes against 8,576 bytes, identical.

A passing check proves nothing by itself; it needs a negative control that shows it can fail. The first attempt (call it C1) nudged one boundary of the range field by 1e-9 and re-ran a plain equality check between the loaded object and the original. That one worked cleanly — the objects stopped being equal, as expected.

The obvious next move was to apply that same tiny nudge to check B. Before running it, the problem surfaced: downstream, that field passes through a frame-quantization step (one video frame, 1/30 second), and 1e-9 is nowhere near that resolution. Run as planned, the perturbation would have been silently absorbed into a "no difference" reading that meant nothing. Catching that in advance was the right call. So I moved the mutation to a field the quantization step couldn't touch — a plain strategy-label string that flows straight through to the downstream object unchanged.

B reacted. Green to red. I nearly wrote down "sensitivity demonstrated" and moved on.

Except that string doesn't participate in any geometric computation downstream. It gets copied. What that mutation actually demonstrated was "B reacts if you change any field that ends up in the JSON" — which is a real fact, and also a fact about json.dumps working correctly, not about the property check B exists to protect: that the geometry the loader reconstructs produces the same timeline downstream. The control had fired. It had not fired at the thing it was standing guard over.

Going back to the real question

Back to the actual question — is B sensitive to geometry? — I moved the mutation onto the range field directly.

C4: shift one internal boundary of the range by two frames (0.066667 s — comfortably clear of the one-frame quantization step, so this wasn't going to be swallowed the way C1's nudge was). The result wasn't a byte mismatch. It was an exception, raised before the comparison JSON could even be built: a downstream junction-matching step enforces that every internal range boundary sits within 1e-6 of a corresponding cut point's snapped timestamp, and moving the range alone broke that cross-field invariant on contact.

That's not a failed experiment. It's information: in this pipeline, the range and its matching cut point are not two fields you can perturb independently. There's a fork here worth naming. One path is to go hunting for yet another field that dodges the invariant — mutate, fail, try a different field, repeat until something passes. That path quietly inverts the logic of testing: you're no longer designing a mutation to answer a question, you're searching for whichever mutation produces the answer you already wanted. The other path is to design a mutation that satisfies the invariant instead of evading it. Those are not the same move, even though both end with a passing check.

A mutation that respects the invariant

C6 took the second path: shift the range boundary and its matching cut point's snapped timestamp by the same two frames, together. The invariant holds, the downstream JSON gets built, and it differs from the reference in exactly one place — a clip's end-frame value, shifted by exactly the delta applied: 1252 → 1254.

That's the first mutation in the whole exercise that moved geometry and produced a geometric difference in the output. Not a copied string. A number that only exists because a coordinate moved.

Evidence the second control isn't relabeled success

The obvious objection to C6 mirrors the one from the previous section: you kept adjusting the experiment until it passed, so what makes this different from chasing the result? The answer is that C4 stayed in the suite. It wasn't deleted once C6 worked.

C4 and C6 prove two different things, and neither one subsumes the other. C4 proves the cross-field invariant is actually enforced, not just documented. C6 proves that once that invariant is respected, the check does react to geometry. Keep only C6 and you lose the record that the invariant exists at all — a future reader could easily re-break C6's careful pairing and get a confusing exception instead of understanding why. Keep only C4 and the original question — does this check see geometry? — is still unanswered. Both stayed, tagged with what each one actually showed, including an explicit field in the report recording that geometric sensitivity specifically had — or had not yet — been demonstrated, so that a partial result couldn't quietly read as a complete one.

The reference file that C1 through C7 were all compared against never changed across the entire run — same modification time, same checksum, start to finish. Nothing about proving the loader was accomplished by moving the goalposts.

A quieter asymmetry three lines away

One more thing turned up in the same function, unrelated to the mutation-targeting problem but sitting right next to it. The restorer had two branches for two structurally different cases — a union type with multiple possible branches, and a fixed-length tuple. The union branch had just been fixed to raise loudly on ambiguity. The tuple branch, a few lines below it, still rebuilt via zip(value, declared_types) — which, as noted above, truncates in silence rather than complaining. Feed it three elements where two were declared, and one vanishes without a trace.

This one wasn't in the self-reported summary from the pass that implemented the fix. It surfaced from reading the function directly: one branch had been made to fail loudly, its neighbor a few lines down had not, and the asymmetry was visible on the page once someone looked at the whole function rather than the one field that had prompted the fix. "This field is patched, therefore safe" and "does this entire function share the same class of risk" are different questions, and only the second one catches this.

What transfers

1. A negative control's mutation has to target the property the check exists to protect. "Something changed and the check reacted" is not sufficient — that something might be a pass-through field with no connection to why the check exists. Ask whether the mutation touches the target property or a bypass sitting next to it.
2. When a mutation fails somewhere unexpected — an exception, a different check firing — that's information, not a dead end. Record it, then design a second, co-existing mutation that satisfies whatever it revealed. That's a different move from chasing mutations until one happens to pass, which is choosing the answer first and searching for an input that produces it.
3. "It failed as expected" is necessary, not sufficient, as a self-report. What was actually mutated has to be audited — a green-to-red flip alone can't distinguish a control that hit its target from one that hit a bypass, because both render as the identical binary signal.
4. When one branch gets a loud failure, check its siblings. A union branch and a fixed-length-tuple branch sitting a few lines apart in the same function don't get fixed together just because they're close — closeness in the source isn't closeness in the fix. It takes a deliberate second pass over the whole function.
5. Delegating implementation is not the same as verifying it. A collaborator's or a sub-agent's "all three mutations failed as expected" doesn't get copied into a record without being re-run and re-read. In this case, re-reading the code directly is what surfaced both the missing invariant test and the sibling asymmetry — neither one was in the handoff.

Where this could be wrong

• This is close kin to "weak mutant" problems in the mutation-testing literature — a rediscovery of an established idea in a concrete case, more than a new principle. What I think keeps it distinct from my earlier piece on checks that cannot fire is that this failure has a different shape: that one covered checks that never trip at all; this one covers a check that trips reliably, just not always because of what you think. Whether that distinction reads as meaningfully different or as the same lesson twice is for the reader to judge.
• The sample size here is one function, one round-trip verification run. Generalizing "this kind of misdirected negative control is common" from a single instance is thin evidence.
• The sibling-branch asymmetry (the union/tuple pair) has a different root cause than the mutation-targeting problem above it — one is about test design, the other about code review discipline. Bundling them risks a vague moral ("check things twice") that neither specific lesson actually supports on its own.
• I did not extend the co-designed-invariant technique (like C6) to the other output fields the reference comparison covers — only one of them was individually exercised this way. The byte comparison logically covers the rest, but I have not watched each of them move individually, so "geometry sensitivity" here is demonstrated for one field and inferred, not separately shown, for its neighbors.

More notes at hexisteme.github.io/notes.