Verifying AI Subagent Work: Nine Checks That Turn Testimony Into Evidence

A subagent finishes a task and reports back: implemented the fix, wrote a regression test, ran the suite, six failing tests now pass, done.

My setup: an orchestrator I call sentinel dispatches implementation subagents (a subagent is a separate agent instance, usually with its own model context and tool access, dispatched by an orchestrator to carry out one scoped task and report back), which I call lanes, each on its own git worktree (a second working directory linked to the same repository, checked out to its own branch, so multiple branches can be built and tested in parallel) and branch. A lane picks up a scoped task, reproduces a bug, fixes it, writes a test, runs the suite, commits, and reports back in prose. That report is the only thing sentinel sees unless it goes looking for more. What follows answers one question: what does it actually take to believe that report?

A subagent's completion report is testimony, not evidence. The lane that just spent twenty minutes fixing a bug is the same process now telling you it fixed the bug, using the same weights, the same training pressure toward sounding confident and complete, the same blind spots that let the original bug through code review in the first place. Treating "tests green" as a build artifact, the way you'd treat a CI badge (CI, or continuous integration: an automated pipeline that builds and tests every commit or merge, independent of any human or agent judgment about whether it's ready), is a category error. A CI badge comes from a deterministic pipeline running on a fixed commit. A subagent's report comes from a language model summarizing its own work, and a summary can be wrong in every way the work itself can be wrong, plus a few new ones: selective reporting, a test that fails for the wrong reason, a rewritten assertion that no longer tests anything.

These lanes are reliable enough that when a report is wrong, it is rarely the failure a five-second skim would catch: an obviously broken diff, a suite that clearly never ran. It is subtler than that: a test that dies before it reaches the assertion it was written to check, a rewritten test that happens to also be correct, a decay factor that compounds silently across two calls instead of recomputing from scratch. Subtle failures need mechanical, repeatable verification, not a diff review that feels thorough, because a diff review that feels thorough is precisely the kind of judgment call a subtle failure is built to survive.

Nine checks follow, built from real cases where the naive reading would have gotten it wrong.

Part 1: Why Testimony Fails Quietly

Why "Read the Diff, Run the Tests" Isn't Enough

Both moves feel rigorous. Neither is adversarial by default.

Reading the diff answers "does this code look reasonable." It doesn't answer "is this the only code that changed." A lane under instructions to write a failing test has every structural reason to also make that test pass by touching the implementation, instead of leaving it red. Nothing about a diff view flags that: a plausible implementation edit sitting next to a plausible test edit reads as one coherent change, not as a violation of the task's actual contract.

Running the tests answers "did the command exit zero." It doesn't answer whether the run happened against the code you think it happened against, whether every reported failure failed for the reason you think it failed for, or whether the counts in front of you reconcile with the counts from the last time you looked. A green run downstream of a broken setup is still a green run. A red run can be red for a reason that has nothing to do with the defect under test.

The witness stand. A subagent's report is closer to courtroom testimony than to a lab result. A witness who says "I saw the defendant leave at nine" might be right, mistaken, or repeating something someone else told them, and a serious cross-examination doesn't stop at "did they say it clearly." It asks for corroborating evidence: a receipt, a camera, a second witness who never spoke to the first. "Tests green" is the witness statement. The checks in this post are the cross-examination.

The fix isn't reading more carefully. It's converting each question into something mechanical: a command with a binary answer, an arithmetic check, a classification done once and never re-derived by eye. That's the rest of this post.

Part 2: Establish the Baseline

Check 1: Prove the Implementation Was Untouched

One git command, run before anyone opens the diff, settles a question that would otherwise depend on trusting a lane's own account of what it touched.

The scenario: a lane is told to reproduce a bug by writing a test that fails against the current, unmodified implementation, then fix the implementation so the test passes. The dangerous failure mode is a lane, under pressure to report a clean result, editing the implementation to make its new test pass without the implementation having a bug in the first place, or editing it in a way that papers over the symptom the test checks rather than the actual defect. Either way, "this test failed before my fix" becomes unfalsifiable once the only thing being read is the final green state.

The check is one git command:

git diff <base>...<lane-branch> --stat -- app/ agent/

An empty result means the product code is provably untouched by this lane, full stop: no implementation edit exists anywhere in the diff between the base and the lane's branch, scoped to the actual product directories rather than tests or benchmarks. Only once that's established does a claim like "the test fails against main" mean anything, because now the test's behavior on main reflects main's actual behavior, not a moving target the lane also modified.

Scope the diff, not just the command. git diff --stat without a path filter answers the wrong question: it tells you the total size of the change, not whether any of it landed in product code. The -- app/ agent/ (or whatever your own product directories are) is what makes the check binary. Drop it and a lane that touched both a benchmark fixture and the actual defect looks identical, in the stat output, to one that touched only the fixture.

One lane's full diff looked, at a glance, like it touched a lot of surface area. Scoped to app/ and agent/, the stat output came back empty: everything it had touched lived under benchmark and test paths. That's the entire review for "did you actually leave the implementation alone," answered in the time it takes to type one command, and binary in a way eyeballing a diff never quite is: either the stat list has product-directory entries in it, or it doesn't.

Check 2: Reproduce the RED Yourself

Don't trust a pasted failure message. A lane's report of "this failed before my fix, here's the traceback" is still testimony, generated after the fact from whatever the lane remembers or reconstructs about the pre-fix state. The only version of that claim worth acting on is one produced independently, against the actual unmodified base.

In test-driven-development shorthand, a RED run is a test that fails as expected, proving it actually exercises the bug before any fix lands; GREEN is the same test passing once the fix is in. Reproducing the RED yourself means checking the lane's new test file out against the unmodified base revision, in a separate, clean worktree, and running it there. Not in the lane's own worktree, where the implementation fix already landed. Not by re-running the lane's reported command and trusting the reported output. A fresh worktree checked out to the base commit, with only the new test file copied over, is the smallest environment in which "does this fail against the code as it stood before the fix" has an unambiguous, self-produced answer.

One fresh-worktree run against the unmodified base produced:

6 failed, 299 passed

That number, generated independently in an isolated worktree, is what licensed the merge. The lane's own report of the same failure count is corroborating, not load-bearing. The distinction matters because the two numbers being equal is itself informative: if an independent reproduction and the lane's reported number disagree, something about the lane's environment, working tree, or reporting was wrong, and that's worth discovering from a mismatch rather than assumed away by an agreement nobody checked.

Part 3: Read the Failure, Not the Count

Check 3: Classify Failure Types, Not Just Failure Counts

If you take only one of these nine checks, take this one.

Six failures reads like six pieces of evidence that the test suite caught the bug. It isn't. Of the six failures in the worked example above, five were genuine. The sixth was a TypeError (the exception Python raises when an operation or function call receives an argument of the wrong type or count), raised because the new test called decay_old_notes(..., now=fixed_now) with a now= keyword argument that did not exist on the function's signature at the base revision. That's not the test demonstrating the defect it was written to catch. That's the test failing to run at all, because it calls an interface that doesn't exist yet, a completely different fact from "the old behavior is wrong."

The rule, stated plainly. A test that fails because it cannot even run has not demonstrated anything about behavior. A TypeError from a missing keyword argument, an ImportError (the exception Python raises when code tries to import a name that isn't defined in the target module) from a symbol that doesn't exist yet, a TypeError from a changed function arity (the number of arguments a function is defined to accept; a call with the wrong number raises a TypeError before the function body ever runs): these are signature mismatches, not behavioral evidence. They say the base code hasn't grown the shape the new test expects. They don't say the base code, once called correctly, would produce the wrong output. Only a failure that reaches an assert and evaluates it to False is a genuine RED, because only that failure has actually exercised the old behavior and found it wanting.

The other five in that run were genuine assertion failures, worth quoting individually because they carry more information than "five failed" ever could:

• assert 1 == 0
• an assertion whose message read "expired note's row must survive purge (state machine is append-only, never DELETE)"
• assert None is not None
• 0.14999896382860867 == 0.5 ± 0.025
• two separate instances of assert 0 == 1

The fourth is doing more work than the rest. 0.14999896382860867 is, to floating-point noise, exactly 0.3 × 0.5. That number alone pins the mechanism: the test seeded a note's prior decay score at 0.3 and expected a fresh decay calculation, one half-life later, to land near 0.5, computed purely from elapsed time. Getting 0.15 instead means the implementation was multiplying the new decay factor onto the note's existing decay score rather than recomputing it from scratch, so a second call compounds a first call's result instead of superseding it. The full fix, including why the corrected version has to produce the same score no matter how many times it runs, is in Agent Memory Expiry Is a State, Not a Delete. Reading the assertion's own numbers found the bug before a single line of the implementation got opened. That's the difference between a failure count and a failure that's been read: a count says something is wrong, a genuine assertion failure with real expected and actual values in it often says what's wrong.

The lane itself had already declined to count that TypeError as a genuine RED in its own report. Verification isn't only a defense against a lane overstating its work. Running the same check independently sometimes confirms a lane was already being more careful than its summary suggested, which is a different, and equally useful, outcome.

(The published post includes an interactive demo here: six real failure messages from this run, each clickable to classify as a genuine RED or a signature mismatch before checking the answer.)

Check 4: Prove "Docstring-Only" Mechanically, With AST Normalization

A different category of claim: this diff only touches comments and docstrings, no behavior change. That claim is common on cleanup or documentation-pass tasks, and it's tedious and error-prone to verify by reading a large diff, because a docstring edit sitting sixty lines above a one-character logic change is trivially easy to miss on a skim.

Don't skim it. Parse both revisions with Python's AST module (AST, or Abstract Syntax Tree: a tree representation of source code's structure, produced by parsing, that ignores exact formatting, comments, and, once stripped, docstrings, while preserving everything that affects behavior), walk every Module, ClassDef, and FunctionDef node, strip the leading docstring Expr node from each body if one is present, and compare ast.dump() of the two normalized trees. Identical dumps mean the two revisions are provably semantically equivalent modulo docstrings, module-level comments, and whitespace, none of which reach the AST at all. Different dumps produce a structural diff of exactly what changed, without reading a single line of prose to find it.

import ast

def strip_docstrings(tree: ast.AST) -> ast.AST:
    for node in ast.walk(tree):
        if isinstance(node, (ast.Module, ast.ClassDef,
                              ast.FunctionDef, ast.AsyncFunctionDef)):
            body = node.body
            if (body and isinstance(body[0], ast.Expr)
                    and isinstance(body[0].value, ast.Constant)
                    and isinstance(body[0].value.value, str)):
                node.body = body[1:] or [ast.Pass()]
    return tree

def normalized_dump(source: str) -> str:
    tree = strip_docstrings(ast.parse(source))
    return ast.dump(tree, annotate_fields=False)

def is_docstring_only_change(old_src: str, new_src: str) -> bool:
    return normalized_dump(old_src) == normalized_dump(new_src)

This is a five-minute script, and it answers a question that otherwise costs a slow, attention-taxing read of a four-hundred-line diff, with the risk of missing the one line that mattered because the surrounding two hundred lines were legitimately just prose. Identical normalized dumps are a proof. A read of the diff, however careful, is an opinion.

Why this generalizes past docstrings. The same technique verifies any claim of the shape "this diff doesn't change behavior," not just docstring-only claims. Strip whatever the claim says shouldn't matter, comments, formatting, an added type hint Python ignores at runtime, and compare what's left. If the normalized trees match, the claim is a mechanical fact. If they don't, the diff of the normalized trees is the exact list of what actually changed, no matter how it's dressed up in the raw diff.

Check 5: A Rewritten Test Is Legitimate Only If the Old One Proved the Defect

Replacing an existing test is correct in one case and a cover-up in the other, and it's the same action either way: the only way to tell which one you're looking at is reading what the old test actually asserted, on the base branch, before anyone touched it.

The failure mode: the cheapest way to turn a red suite green isn't fixing the bug, it's rewriting the test that's catching it. A lane under pressure to report success has a live incentive to loosen an assertion, delete a check, or replace a test wholesale with one that happens to pass against the unfixed code. A diff showing a test file modified doesn't distinguish "the old test was wrong and encoded the bug being removed" from "the old test was right and someone made it stop complaining." The new test alone can't tell you which. Only the old one, in its pre-change form, on the base branch, can.

A real case had exactly two old tests replaced. Read on main, before the change: test_decay_deletes_very_old_notes had a docstring reading "Notes with decay_score < 0.1 are auto-deleted," and its body asserted store.recall("/repo") == [] after decaying a note down to that threshold: prove the note is gone. test_purge_expired_notes_removes_old_notes asserted store.count_notes(ws) == 0 after purging a note past its TTL (time-to-live: how long a piece of data is allowed to live before a system automatically treats it as expired, counted from when it was created): the same claim from a different angle, prove the row no longer exists.

Both tests were, by construction, codifying delete-on-expiry as correct behavior. The fix these tests were replaced for was specifically about removing delete-on-expiry, swapping a destructive purge for an append-only (a data structure that only ever accepts new rows; existing rows are never edited or deleted in place, so the full history is always intact) state transition so an expired note's row and its audit trail survive, with the note simply excluded from default recall. The full design is in Agent Memory Expiry Is a State, Not a Delete. Given that goal, both old tests asserted exactly the behavior the fix exists to eliminate. Replacing them wasn't a cover-up. It was the only correct outcome: a test asserting "the row is deleted" cannot coexist with a fix whose entire point is never delete the row.

Had those two old tests instead asserted something orthogonal, say a check on notification formatting sharing a test file with the decay logic, replacing them under the same diff would have been exactly the failure mode this check exists to catch: a real regression quietly deleted alongside a legitimate change, in the same commit, under the same lane summary claiming both were intentional. Same diff shape in both worlds, same lane, same confident report. Only reading the old assertions, on the base branch, tells you which world you're in. There's no shortcut around actually reading them.

This is reward hacking, structurally. An agent that rewrites a test to make it pass, instead of fixing what the test checks, is doing a smaller version of a well-documented failure mode in reinforcement-learned and agentic systems generally: reward hacking (an agent optimizing the measurable signal it is graded on, like a passing test, instead of the actual goal that signal was meant to represent). "Tests green" is the signal. The actual goal is "the defect no longer exists." A system optimizing for the first can satisfy it without touching the second, and a rewritten assertion is one of the cheapest ways to do that. The fix isn't trusting the agent more. It's making the signal harder to game: read the old assertion before accepting its replacement.

None of this makes the checks adversarially robust, and it's worth being precise about the difference. A lane that knew in advance exactly which nine checks would run against it could, in principle, learn to route an implementation edit through a directory a scoped diff doesn't cover, or phrase a replaced test's docstring to survive a skim of the old assertion. What these checks are actually robust against is the failure modes observed so far: a lane optimizing for a report that reads as done, not one deliberately reverse-engineering the review process itself. That's a narrower guarantee than "adversarially secure," and treating it as the stronger claim would be exactly the kind of overstatement this post is arguing against.

Part 4: The Arithmetic That Replaces Judgment

Check 6: Make the Suite Counts Reconcile, Arithmetically

Track pass counts as a ledger across merges, not as isolated snapshots, on whichever codebase you're maintaining; the numbers below are from a much larger suite than the 299-test example in the last section, a different repository entirely, kept only to show the pattern. If the suite stood at 4752 tests before a merge, the next merge landed at 4754, and the one after that landed at 4758, those numbers have to compose with what each lane claims it did. A lane that says "I added 4 tests" against a base of 4754 has to land the suite at exactly 4758. Landing at 4757, or 4760, means something doesn't reconcile, and that's a question to ask before merging, not a detail to notice later. The ledger itself doesn't need to be anything elaborate: a line in the merge log, a comment on the CI run, a running note next to the queue. What matters isn't where it lives, it's that the number gets written down and the arithmetic gets checked every time, not just when something already looks wrong.

It catches a class of problem no amount of diff-reading surfaces: a test silently skipped, a marker that deselects (a test the test runner's collection step found but chose not to run at all, because a marker or command-line filter excluded it, distinct from a test that ran and was skipped) more than intended, a fixture that swallows a collection error instead of raising it. None of those show up as a red mark anywhere. They show up as a total off by a number nobody explained, and the only way to notice is keeping the ledger in the first place.

Check 7: The Same Arithmetic Separates a Regression From an Environment Failure

A CI run went red immediately after a merge landed. Bisecting the merge would have been wasted effort here: the two failures had nothing to do with the merged code. The result line read:

2 failed, 4752 passed, 4 skipped, 13 deselected

Add those up: 2 + 4752 + 4 + 13 = 4771. A local run of the identical commit read 0 failed, 4754 passed, 4 skipped, 13 deselected: the same total, 4771, with the two failures landing as passes instead. Identical totals across two environments running the identical commit mean the set of tests that ran was unchanged between the two runs: the merge didn't alter what got collected or executed. If the merge itself caused the two failures, the failure should reproduce locally too, on the same commit, under the same collection. It didn't. Same commit, same total, different outcome between environments, which is what an environment difference looks like, not a code regression, and the arithmetic pointed at that category before either traceback got read.

The identity underneath both checks.

total = failed + passed + skipped + deselected

Every number on the right side has to be accounted for by something you can name: a real failure, a real pass, a deliberate skip, a deliberate deselect. If two runs of the same commit produce different totals, the runs didn't execute the same set of tests, and nothing about their individual failures is comparable yet. If the totals match, the runs did execute the same set, and a failure present in only one of them is telling you about the environment, not the code.

The mechanism: an upstream model host returned an HTTP 429 (the Too Many Requests status code a server returns when a client has sent more requests than the server allows within a given time window), because two tests were reaching out over the network to download an embedding model they had no business touching. The cache root gets redirected to a fresh temporary directory for the whole test session, so the suite never writes to the real developer cache, and that isolation has a side effect nobody had traced through: the model cache is empty on every run, local ones included. Both environments download the model every time; a warm cache isn't what saves the local run. Local requests simply get served, and CI, hitting the same host from many parallel jobs, gets rate-limited. The download itself was accidental: the indexer's constructor builds its embedding provider eagerly (building an expensive object, like a network client, immediately when its container is created, rather than waiting until the object is actually needed), so two tests that only checked path resolution pulled multiple gigabytes over the network as a side effect of construction. Eight other tests in the suite stub (a fake, simplified stand-in for a real dependency, substituted in during a test so the real, slow or costly, version never actually runs) the provider out; these two missed it, and one of them stubbed it a line too late, after the constructor had already run.

The arithmetic identified the category, environment rather than regression, in five seconds. It couldn't identify the mechanism: an eager constructor and an isolated-but-always-cold cache had been costing every CI run a multi-gigabyte download for as long as both existed, and nothing about the merge would have surfaced that on its own. A five-second addition problem pointed at the right category of answer before any log got read, and it costs little enough that skipping it whenever a run goes red right after a merge has no real justification.

(The published post includes an interactive demo here: enter what CI reported and what a local run of the identical commit reported, and watch whether the totals agree before anything else about the failure gets to matter.)

Part 5: Verify Your Own Verification

Check 8: Verify Yourself Honestly, Including When the Run Wasn't Clean

Verification discipline that only applies to the thing being checked, not to your own process while checking it, is theater. It looks rigorous and isn't.

Verification that never checks itself. A verification run's own conditions can be compromised the same way the code under test can be: shared CPU with another process, a cache that isn't actually isolated, a filesystem lock held by something unrelated. A check that never asks "was my own environment clean" is applying scrutiny in one direction only, and the direction it skips is the one nobody else is going to catch for you.

A supposedly isolated verification re-run turned out not to be isolated at all: ps showed another test suite's process still alive on the same machine while the verification run was executing, meaning the two were sharing CPU, and potentially any shared state like a local model cache or a filesystem lock, for the run's duration. The honest report says exactly that, rather than presenting the result as a clean, trustworthy signal. The run came back green anyway, and the honest version of that report doesn't stop at the color: it was re-run a second time, alone on the machine with the competing process killed, and came back green again before the merge went in. The first, contended run didn't get to stand in for a clean one just because the second run agreed with it.

That green result, under contended conditions, was itself informative, not just a relief. It meant the interference, whatever it was, wasn't deterministically reproducible enough to flip this particular run's outcome, a weaker but still useful fact: the check survived one instance of noisy conditions, not that noisy conditions never matter. Reporting "this ran clean" when it hadn't would throw that distinction away and substitute a false result for a nuanced, honest one. The whole point of running these checks is producing facts you can act on. A fact silently upgraded past what was actually observed isn't a fact anymore. It's the same testimony this post is about not trusting, coming from the verifier instead of the lane.

Check 9: When You Can't Prove the Mechanism, File It

Not everything resolves cleanly, and pretending otherwise is its own failure mode.

A test failed once, under concurrent suite runs, and passed cleanly on the next run under what looked like the same conditions. Inventing a plausible explanation and patching around it would produce a fix for a mechanism that was never actually confirmed, which isn't a fix: it can silently paper over the real problem while looking resolved.

The record instead marks the failure UNPROVEN, alongside the hypotheses already ruled out (cache isolation between concurrent runs was checked specifically and confirmed correctly isolated, so that's off the list) and the one leading hypothesis that remains unverified. That's a more useful artifact than a confident guess: the next person, or the same person later, who hits this failure again knows exactly what's already been checked and doesn't waste time re-deriving it.

A related lesson came out of the same incident, and it's now a standing rule in every lane brief since: the traceback for that original failure was lost, because whatever ran it had piped the output through tail -3, keeping only the last three lines. Three lines is nowhere near enough to diagnose a concurrency-sensitive failure after the fact; by the time the need for full context is obvious, the run is over and the information is gone. Every brief now mandates keeping at least 40 lines of any failing run's output. Evidence has to survive long enough to be checked. A claim about a failure whose output has already been discarded can't be verified, and piping it through tail because the run was noisy doesn't hold up against a failure that actually needs to be understood.

The Economics

These lanes are not unreliable. A list of nine checks can read as evidence against that, but the failures behind these checks are subtle by nature: a TypeError masquerading as a genuine RED, a test rewrite that's legitimate nine times out of ten and a cover-up the tenth, a decay bug whose signature is a six-digit floating-point coincidence. No amount of careful reading catches these reliably, because careful reading is a variable-quality process that degrades under fatigue, familiarity, and the reasonable instinct to trust a report that says done.

Every check above is seconds of mechanical work set against minutes of agent time. Most of them are one command: a scoped git diff --stat, a suite total that either adds up or doesn't, an ast.dump() comparison. None require reading the lane's prose more carefully. All of them replace reading the prose more carefully with something that produces a fact instead of an impression.

A cheap, mechanical check that runs every time, on every merge, without depending on extra attention this one time, beats a careful review that only runs when someone happens to be paying full attention. Build the checks into the loop once, and the testimony stops mattering, because the evidence is already there.

Related reading

• Agent Memory Expiry Is a State, Not a Delete: the decay bug and the rewritten tests in Sections 04 and 06 come from this fix; this post is what verifying it actually looked like.
• Your Eval Is Leaking the Answer: a different way a green result can be trusted for the wrong reason: five channels through which an eval leaks the fact it's supposed to measure.
• The Bugs Only Dogfooding Finds: six real bugs a green test suite and adversarial reviewers never caught, and the boundary each one exposed.