The patch and the test were wrong in the same direction

Originally published on hexisteme notes.

I had a defect, a model that proposed a fix, and a test that graded the fix. The patch was wrong. The test passed it.

Neither of them was careless. They had both read the same requirement at the same shallow depth and landed in the same wrong place, independently. The green checkmark was manufactured by two unrelated errors that happened to agree.

The setup

I was handing known defects to different models and grading their patches mechanically instead of reading every one. That's the whole reason the tests existed: I didn't want my own judgement in the loop for thirteen separate fixes.

Defect thirteen was the plainest of the lot:

main() has no way to signal failure through the process exit code. Whatever fails inside it, the process ends with status 0. The only way a failure escapes is an unhandled exception leaking out by accident.

The test I wrote for it opens with a docstring stating, in its own words, what a fix has to achieve:

What becomes true once it's fixed: on a failure path there exists at least one means of producing a non-zero exit code, e.g. sys.exit(1).

And here is what the test actually asserted:

has_sys_import    = any(... "import sys" ...)
has_sys_exit_call = any(... a call to sys.exit(...) ...)

assert has_sys_import and has_sys_exit_call

Existence. Not non-zero. The docstring and the assertion are two different specifications, and I wrote them ten lines apart in the same sitting.

The patch

The model returned this:

--- a/audit.py
+++ b/audit.py
@@
 import json, math, statistics, random
+import sys
@@
     if __name__ == "__main__":
-    main()
+    main()
+    sys.exit(0)

sys.exit(0) exits zero. Always. It cannot signal failure — which is the entire content of the defect. The model even flagged its own unease in the risks field it returned alongside the patch ("may mask unexpected exceptions"), and submitted it anyway.

I applied it and ran the test. Green.

Why this is worse than either mistake alone

Take the two errors separately and they're unremarkable. A model produced a shallow patch: that happens constantly, and it is precisely why you have a test. A test asserted something adjacent to its stated intent: that happens too, and it's why you review patches.

The trouble is that each one is supposed to be the other's backstop, and here they failed toward each other.

Both parties compressed "signal failure via a non-zero exit code" into "there should be a sys.exit around here somewhere." That is not a coincidence in the way two dice landing on the same number is a coincidence. Both were reading the same English sentence under the same pressure to be done, and the shallow reading of that sentence is a strong attractor. Anything that reads it quickly lands there.

Independent processes only give you independent errors if they are independent along the axis that matters. Mine shared an axis I hadn't noticed I'd built: a natural-language requirement, read once, by two readers who both wanted to move on.

If I had not opened the test, my records would now contain the sentence "this model fixed defect thirteen." Not a bug in the record — a false statement about a tool's capability, entered with a passing test as its evidence.

The audit

If one test measured something adjacent to its own docstring, I had no reason to believe it was the only one. So I pulled every test's stated intent and every test's assertions and put them side by side:

for fn in [n for n in tree.body if isinstance(n, ast.FunctionDef)]:
    intent  = re.search(r"What becomes true once it's fixed:\s*(.+?)(?:\n\n|\Z)", doc, re.S)
    asserts = [a for a in ast.walk(fn) if isinstance(a, ast.Assert)]

Fourteen tests. Two diverged — or so I recorded at the time.

The second one I would never have caught by reading normally, because it isn't wrong — it's half right. Its docstring demands two things of a fix:

the verdict must emit an explicit "unevaluable" signal, and must not return an affirmative "no problem" verdict.

The assertion checked the second clause only:

assert "no problem" not in result["verdict"]   # not affirmative — checked
                                               # explicit signal — never checked

So verdict = "" passes. Silence passes. And silence is the exact failure this defect is about: a caller who cannot tell "we couldn't evaluate this" from "we evaluated it and it's fine."

The test forbade the wrong answer and forgot to require the right one.

Half-assertions are the dangerous shape. A test that checks nothing becomes obvious the first time something breaks. A test that checks half of what it claims looks like coverage forever.

Fixing the judge, with controls in both directions

I made the assertion require an argument that can be non-zero — excluding only the literal 0, None, and the no-argument form, and letting variables and expressions through. Strict where I know it's wrong; permissive where I'd be guessing.

def _can_be_nonzero(call: ast.Call) -> bool:
    if not call.args:
        return False                          # sys.exit()  == 0
    a = call.args[0]
    if isinstance(a, ast.Constant):
        return a.value not in (0, None)       # sys.exit(0) == 0
    return True                               # a variable can be non-zero

Then I checked it in four directions, because a tightened test that only rejects the one case I just saw has learned my example rather than the requirement:

The second row is the one that matters. Before, that row was green, and everything I would have concluded from that green was wrong.

The record was also lying, in the other direction

There is a prior step I nearly skipped, and it flips the verdict on the model rather than on the test.

For a different defect in the same batch, the stored artifact was 185 bytes — a small, valid, schema-shaped object with an empty action field. The usage block on the same response reported 4,425 output tokens. Those two numbers cannot both describe the same reply.

They did, because the gateway returns the model's reasoning in a separate reasoning_content field when you request structured output, while completion_tokens bills for both. I had never read that field. So a response where the model worked and failed to serialize looked byte for byte identical to a response where the model did nothing at all.

I captured it. The same task, called again, stored 24,209 bytes instead of 185. A sibling task's record went from 67 bytes to 16 KB — and inside that one was the model walking the source, naming exact line numbers, and assembling a diff. It had done the work and failed at the envelope.

Without that capture, the record would have read "this model produces no patch" — also a false statement about capability, and much harder to catch later, because there'd be no artifact left to contradict it.

Two records, two different false sentences, both fully supported by the evidence I was keeping. That's what a measurement gap looks like from the inside: not an obviously missing number, but a record that reads as complete and says something untrue.

The audit missed four

That count held until the same afternoon.

I handed the same fourteen tests to three models that had no access to my conclusions — different slices each, none able to see the others' answers — with one question: does each assertion enforce what its own docstring declares?

They returned four more divergences. Not in tests I had skipped. In the same fourteen, on the same screen my AST walk had already printed.

That was not the floor either. Two of the four I scored as false positives — I rebutted the models' premise instead of building the case they handed me, which is the same shortcut this whole piece is about, committed by the person grading the exam. Both were real. And when I finally applied the constructive rule to every pair rather than only the disputed ones, five more turned up — including one in a test I had repaired earlier and never re-attacked.

What I take from it

A pass is a claim about your judge, not just about the work. When something goes green the first time, that is the cheapest moment you will ever have to check whether the judge measures what it says it measures. It costs one file open. I burned six dispatches investigating a tool's capability and the thing that decided the verdict was sitting in my own test file the whole time.

Compare docstring to assertion mechanically — then attack each pair. The intent is prose written while you understood the problem; the assertion is code written while you wanted to finish. They drift, and nothing in your toolchain compares them. An AST walk that prints intent and assertions side by side takes minutes. But the walk only retrieves the pairs; deciding whether the assertion covers the declaration is still a human reading prose fast, and that reading is the original defect. Make the verdict constructive instead: for each pair, try to build a state that passes the assertion and violates the docstring. My eyeball pass over the walk's output found two gaps in fourteen tests. Then I ran the constructive pass over all fourteen pairs, including the ones I had already "verified" and the ones I had already repaired. It found eleven.

"Independent" is a claim about the axis, not the count. Two checks derived from one sentence, read once each, are one check with extra steps. This is the same reason polling several models on an identical prompt and counting agreement tells you almost nothing: you haven't sampled independent errors, you've sampled one prompt's attractor several times. I had a written rule against exactly this for models — and it never occurred to me that a worker and its test are the same structure.

This is a cousin of two failures I've written about before: a negative control that tested the wrong field, where the control passed because it was watching something that could not have changed; and green tests that prove behavior, not reachability, where the assertions were true of code nothing ever called. All three are the same species — the instrument is measuring something adjacent to its target, and adjacency is invisible from a passing run.

The new part here is the second party. It isn't only that my instrument was off. It's that the thing being measured was off in the same direction, so the two of them agreed, and agreement is what I had been treating as evidence.

Agreement is only evidence when the parties could have failed differently. Mine couldn't.

More notes at hexisteme.github.io/notes.