The tests, in principle. But the agent wrote those too, out of the same understanding of the task, so they agree with the code even when the code is wrong.
Why you care: this is the difference between an agent that checks its own work and one that confirms it, and the setup is a file, a runner and one CI job.
The test is the same idea written twice: once as behaviour, once as an assertion about that behaviour. Get the idea wrong and both are wrong together, agreeing. Coverage does not tell you which kind you have: every line runs either way.
Which leaves the question: how do you get an agent to write a test capable of contradicting its own code?
Have the agent break what its own test guards, on purpose, and report what the test did.
Here is the smallest possible case. A function averages two numbers, and one test guards it. Is that test good?
avg(a, b) = (a + b) / 2 the implementation
avg(4, 4) must be 4 the whole test
First, mutate it. Change the + to a -. Now avg(4, 4) returns 0, and the test goes red. Red is the good outcome here: a mutation that gets caught means the test would notice if this line broke, and that it is a good test.
Now put a placebo in its place. Throw the body away and return a. On this test it returns 4, so the test stays green. Green is the bad outcome here: a surviving placebo means it is a bad test, one that cannot tell a correct implementation from a wrong one.
Same test, two opposite verdicts, because the two ask different questions. The mutation asks whether the test is wired to that line at all; the placebo asks whether it knows what it is for. Believe the placebo: a test that lets a wrong-but-plausible implementation through is not testing that implementation.
Notice the bottom row. Everything in the placebo's column is invisible to the mutation, which only asks whether the line was covered.
What to set up
All of it fits in one request. Paste this into the agent, in the repository you care about:
Set up placebo proofs for this repo.
1. Add to CLAUDE.md: "A check is not finished until a deliberately
wrong implementation has died against it. Write the placebo
BEFORE the fix, never after."
2. Create a committed proofs file. One entry per check:
id, the test name, the package
inject - edits planting the placebo in PRODUCTION code
weaken - edits switching off that check's own verdict
note - which defect this imitates; refuse an empty note
Every edit is an exact anchor plus replacement, or an append.
3. Write a runner. ONE entry per invocation, never a batch:
concurrent builds produce false "survived" verdicts. It must:
- take the commit SHA as an argument, one SHA per whole run
- refuse to start if anything is uncommitted
- work in a throwaway checkout, never my working tree
- strip GIT_* from the environment before every git call
- check each anchor occurs EXACTLY once, counting
occurrences, not matching lines
- compile as a separate step first: a placebo the compiler
rejects is not a defect and must not read as success
- require evidence the target test RAN, by exact name; a
typo gives exit 0 and an empty failure list
- take the verdict from the NAME of the red test, never the
exit code, deciding in this order:
compile error, infra error, did not run, killed, survived
- restore the tree afterwards and verify it by hash
- keep the raw log of every phase
Three phases, each from a clean checkout:
phase 1 clean code nothing is red
phase 2 + placebo THIS test is red
phase 3 + placebo + weakened check green again
4. Require every check to carry //guard:proof <entry-id> in its
doc comment, and add a test enforcing it:
- read the marker only from the doc comment, not the body
- the entry named must exist, must name THIS test, and must
carry both an inject and a weaken
- an entry no check cites is a violation
- "none" is legal, and counted
- fail if the scan finds no checks at all
5. Add a CI job running every proof. No allow_failure.
Then pick the check whose silent failure would cost most, write
its placebo, and tell me whether it survived.
Run this and you find out where you are unprotected: the places where a check exists, passes, and would not notice if the thing it guards broke tomorrow.
Fix the worst one first: the check where nothing else would notice.
This is how you get an agent to check its own work instead of confirming it. It has to break what it built and show you what the test did, and from then on the tests it writes are tests that can fail.
From a set of pieces on the rules coding agents actually follow. Earlier: Your CLAUDE.md is full of wishes, on why most rules in that file are not enforced by anything.
