A new arXiv paper watched coding agents get denied the facts they needed. They did not stop. They invented.
On August 17th, a group of researchers posted a paper to arXiv with an unglamorous title and a genuinely unsettling core finding. The paper is "The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks" (arXiv:2608.16630), by Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, and Laurent Bindschaedler. Before going further, one honesty note that will hold for this whole piece: I have read the paper's abstract, not its full text, and every quotation below comes from that abstract. It is enough for what this essay is about, because what this essay is about is one sentence.
The setup first. The authors model repository-scale coding as reconstructing a web of coupled facts. Every edit an agent makes needs certain facts, and each fact arrives through one of two channels: it is either in the recent context, or it is in the model's memorized knowledge. Facts covered by neither channel are what the authors call coherence debt. Their experiment supplies and withholds each channel deliberately, injecting faults across "seven models and five harnesses" (abstract), and then watches what the agents do when a needed fact simply is not there.
The comfortable prediction is that a competent agent, denied a fact, stops and says so. Here is what the authors report instead: "A missing fact produces wrong work rather than absent work" (abstract). The agent asked to act, acts. In the paper's words, "an agent asked to act acts, fabricating the file or guessing the value" (abstract).
That much is alarming in a familiar way. Everyone who works with these systems has a story about an invented function or a guessed constant. The abstract has sharper findings than the fabrication itself, though. When the researchers renamed a real library to defeat memorized knowledge, the failure was collective: "all seven fail in the same place, passing and missing the same tests" (abstract). Seven different models, one shared cliff. And the paper's economics finding deserves a frame on any wall where infrastructure budgets get set: harness configurations that all pass every test "differ more than tenfold in tokens consumed" (abstract), and the extra spending recovers nothing when the facts are withheld. Paying ten times more did not buy truth. It bought a more expensive invention.
But the sentence this essay exists for is about none of that. It is about the watchers. Because agents fill the holes they are denied, the paper notes, "instruments built on reads look for a hole already filled" (abstract).
Sit with that one. The obvious way to check whether an agent had the information it needed is to watch what it read. Trace the file opens. Log the retrievals. If the agent never read the config file, flag the edit that depended on it. This is reasonable, cheap, and widely done. And the paper's finding is that it is blind precisely in the case it exists for, because an agent that lacked a fact does not leave a hole for your instrument to find. It fills the hole on the way through, with something plausible, and the trace shows an agent that read some things and then wrote some things, exactly like a healthy one. An absence that has been filled in looks exactly like a success.
The general shape
The reason that sentence deserves an essay rather than a citation is that nothing about it is specific to coding agents. It is the general shape of any measurement whose null result and whose broken result are the same value.
Think about what a zero means when a probe returns it. It might mean the thing you searched for is absent. It might also mean your probe never connected to the data at all: wrong path, wrong layer, wrong format, wrong vocabulary. Both conditions produce the same output, and the output is the one that reads as reassurance. Every other number a probe can return is at least weak evidence that the machinery ran. Zero is the only value an instrument produces both when it is working perfectly and when it is not plugged in.
The paper found this shape at the agent layer. I want to show you it sitting one layer up, in the small tools engineers write to check their own work, because that is where most of us actually live.
Four zeros, none of them true
What follows is original data in the smallest possible sense, and I want to be exact about its scope in the same breath as I introduce it. This is not a reproduction of the paper's experiment. It is four toy probes at a different layer, built on synthetic data, reading no real system. Each probe is written the way a working engineer writes one: quickly, reasonably, to answer a question and move on. Each returns zero. In all four cases the zero is a property of the probe, not of the data. The correct answers are 1, 1, 2, and 1.
PROBE NAIVE CORRECT
------------------------------------------------------------------------
1 phrase wrapped across a newline 0 1
2 marker one level up from where the probe looked 0 1
3 MM/DD/YYYY compared as text 0 2
4 searched the wording, not the claim 0 1
------------------------------------------------------------------------
probes run: 4
probes returning a FALSE ZERO (naive==0 while correct>0): 4
Every number in that table comes from a script called repro_absent_vs_zero.py, published alongside this essay. It asserts its own result, so if you run it and get a different table, you have found a bug in the script rather than a disagreement with the piece.
The four cases are worth a sentence each, because the point is what they do not have in common.
Probe one searches a document for a phrase that happens to wrap across a line break. The search is exact; the phrase is present; the newline defeats it. Zero occurrences reported. This is an encoding assumption.
Probe two looks for a status marker in a nested data structure, one level deeper than the marker actually lives. Every record it examines genuinely lacks the marker, so it reports a perfectly healthy corpus. This is a schema assumption.
Probe three compares dates. The records store them as MM/DD/YYYY text, and the probe compares them against an ISO-format string with an ordinary greater-than. Text comparison happily runs, means nothing, and finds no records past the cutoff. Two actually are. This is a type assumption.
Probe four is the subtle one. It searches a document to confirm a claim was corrected, but it searches for the wording of the old mistake rather than the substance of the fix. The fix is present, stated in words that share no tokens with the search terms. Zero matches, wrongly read as "the correction never landed." This is a vocabulary assumption.
An encoding assumption, a schema assumption, a type assumption, and a vocabulary assumption. The causes have nothing in common. There is no shared bug to learn, no single habit to break. What they share is the failure's shape: each produced the exact output that means "all clear," and produced it because the instrument never truly touched the thing it claimed to measure. Four different roads into the same hole, and the hole is already filled.
The discipline cannot be "notice when a zero looks suspicious," because the entire failure mode is that it never does. A suspicious zero is the one you got when you expected fifty. The dangerous zero is the one you got when you expected zero, checked the box, and shipped.
The Monday fix
There is a fix, it is old, and the lab sciences have been using it since before software existed: the control.
A positive control means running your probe against something you already know it should find, before you trust it finding nothing. Searching a log for a rare error? First search it for a common string you can see with your own eyes. If the probe comes back empty on that, the instrument is broken, and the zero you were about to report was never data. This costs one extra invocation and converts "the search found nothing" into "the search demonstrably ran, and found nothing," which are different sentences with different values.
Then add the negative control, because a positive control proves an instrument fires, not that it discriminates. A probe that matches everything passes every positive control ever written. Run it against something you know it should not find, and make sure it stays silent. An instrument that passes both controls has earned a modest thing: the right to have its zero believed.
None of this is novel, which is rather the point. Assay validation is a solved discipline in fields where a false negative kills someone. Software mostly skipped it, because software's probes are so cheap to write that we forgot cheap instruments still need calibrating. The paper's agents are an expensive, autonomous version of the same skip: the harnesses watched reads because reads were what there was to watch, and nobody ran the control that would have shown the watcher was blind.
The honest limit
The researchers could measure all of this because they built the world their agents ran in. They held both channels, knew every fact an edit required, and could therefore see fabrication for what it was. That is what makes the paper valuable: it is a place where the ground truth was, for once, actually available.
Most of us never get that. We audit our own systems with instruments we also built, which means our blind spots come pre-installed in the tools we would use to find them. I cannot inspect my way out of that circle and neither can you, which is why the discipline has to be procedural rather than perceptive. Not "get better at sensing bad zeros," but "no zero gets believed until the instrument that produced it has passed a control, this time, on this data." The rule is mechanical precisely because the failure is invisible.
The paper's agents, denied a fact, invented one rather than stopping. Our probes, denied a connection to the thing they measure, do something quieter and worse: they return the number that means everything is fine. The agent at least fabricates something you might later catch being wrong. The instrument fabricates an absence, and an absence never contradicts anything.
So before you believe the next quiet zero, check the thing that produced it, against a truth you can see without it. The hole you should worry about is not the one your instrument reports. It is the one it fills on the way through.
The four-probe table in this piece is produced by repro_absent_vs_zero.py, which uses synthetic data only, asserts its own result, and is published alongside the essay. Every quotation from arXiv:2608.16630v1 is from the paper's abstract, which is the locator throughout; the full paper was not read, and this piece does not claim to reproduce its experiment.
Sources
Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, Laurent Bindschaedler, "The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks," arXiv:2608.16630v1, submitted August 17, 2026. https://arxiv.org/abs/2608.16630v1 (locator for all quotations: abstract).
arXiv API record for the same identifier, used to verify every quoted string and the submission date. https://export.arxiv.org/api/query?id_list=2608.16630v1
repro_absent_vs_zero.py, published with this essay: the source of every computed number in the piece; synthetic data only; self-asserting.
An absence never contradicts anything
The failure in this piece is not that a tool was wrong. It is that a tool reported nothing, and nothing is what a working tool and a disconnected one both return. The defence is a record of what actually ran: which inputs were touched, in what order, and whether the step that produced a number reached the data at all. Chain of Consciousness is that record for AI agents, tamper-evident and written as the work happens rather than reconstructed once somebody asks. It does not make an agent right. It makes the basis of its answer checkable, which is the only thing that turns "found nothing" into evidence.
Hosted Chain of Consciousness · Verify a record
pip install chain-of-consciousness · npm install chain-of-consciousness
