How to Check Your Test Can Catch the Bug

When an agent's PR says "added a regression test, all tests pass", I want to know one thing: would that test have caught the bug? Plenty of them can't. In my own experiments, grading tests that failed on the unfixed code and passed on the upstream fix still accepted every agent patch on two tasks, and two small probes written later rejected 8 of them. I wrote up that evidence, and when a green suite deserves trust, in When to Trust AI Agent Tests and Reviews.

This post is the hands-on part. I've told readers before to check that a test fails when you break the code, most recently in the post on agents declaring victory. I never showed how. So here's the procedure on a bug small enough to hold in your head: five lines of Python, four checks, real output from every command, and a table of which check catches which false pass.

Your new test is an oracle for one bug, and the only question is whether it can tell the buggy code from the fixed code. Each of the four checks catches a false pass the others miss:

1. Fail on the base, for the right reason. Catches the vacuous test, and filters out tests that go red for an unrelated reason.
2. Break the repair. Catches a test that still passes when the fix is subtly wrong.
3. A preservation probe. Catches the collateral regression the fix's own test never looks at.
4. An oracle the test's author didn't write. Catches a test and code that share one wrong assumption.

The Fixture: Five Lines and an Off-by-One

The demo is chunk(items, size), which splits a list into lists of size. The base version drops the trailing partial chunk: chunk([1, 2, 3, 4, 5], 2) returns [[1, 2], [3, 4]].

def chunk(items, size):
    """Split items into lists of `size`; the last list may be shorter."""
    if size <= 0:
        raise ValueError("size must be positive")
    return [items[i:i + size] for i in range(0, len(items) - size + 1, size)]

chunk([1, 2, 3, 4], 2) returns [[1, 2], [3, 4]] on both versions, so a test that only uses exact multiples passes the bug and the fix alike. That's what bothers me about this bug: nothing in such a test is wrong except the numbers it picked. The repair changes one range() bound:

-    return [items[i:i + size] for i in range(0, len(items) - size + 1, size)]
+    return [items[i:i + size] for i in range(0, len(items), size)]

The whole fixture is a tiny git repo you can rebuild in a few minutes. The code above is src/chunks.py. Commit it and tag it base, apply the diff on main and tag that repair. pyproject.toml needs two sections:

[tool.pytest.ini_options]
pythonpath = ["src"]

[tool.mutmut]
source_paths = ["src/"]
pytest_add_cli_args_test_selection = ["tests/test_bug.py"]

Two test files go in with the repair. tests/test_bug.py is the test under suspicion, with one test, test_keeps_the_partial_last_chunk, asserting chunk([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4], [5]]. tests/test_vacuous.py holds two tests an agent might write: test_chunk_runs calls chunk() and asserts nothing, and test_chunk_returns_something asserts the result is truthy.

A wrong-reason tag, one commit on top of repair, adds a chunk_count(items, size) helper that returns len(chunk(items, size)), plus two tests that both assert chunk_count([1, 2, 3, 4, 5], 2) == 3. tests/test_count.py::test_counts_the_partial_chunk imports chunk_count at the top of the file. tests/test_count_runtime.py::test_counts_at_runtime imports it inside the test function. That difference matters, as you'll see. The probe files for Check 3 and the two scripts get added on main as you reach them.

You need Python 3.12 or later (Check 4 uses itertools.batched), pytest and git. mutmut is optional and POSIX-only, so on Windows run it in WSL. The sed -i commands below are GNU sed. On macOS write sed -i ''. Every output below came from Python 3.12.15, pytest 9.1.1 and mutmut 3.8.0, run in this order on main. The output blocks keep the pytest summary lines and drop tracebacks.

python3.12 -m venv .venv && . .venv/bin/activate
pip install pytest==9.1.1 mutmut==3.8.0
export PYTHONDONTWRITEBYTECODE=1    # before the first run, in this shell

Set that variable before anything runs. It stops Python writing new .pyc files, but it doesn't stop Python loading old ones, and a same-length edit made within the same second as the last run can execute the previous code. In an existing checkout, delete the caches first: find src tests -name __pycache__ -prune -exec rm -rf {} +. A .gitignore keeps caches out of commits. It doesn't disable them.

Check 1: Fail on the Base, for the Right Reason

The key move is tests from the repair, source from the base. Check out the whole base commit and you run the base's old tests, which say nothing about the new one. (Benchmarks call the code before the fix the base and the reference fix the gold patch. SWE-bench sorts tests into FAIL_TO_PASS and PASS_TO_PASS.)

check_test.sh does the swap in a throwaway git worktree, for one test node. It matches the node id literally, so quote a parametrized one: ./check_test.sh base repair 'tests/test_x.py::test_partial[5]'.

#!/usr/bin/env bash
# Usage: ./check_test.sh BASE_REF REPAIR_REF TEST_NODE_ID
# Runs one test from REPAIR_REF against BASE_REF's source, then REPAIR_REF's.
set -euo pipefail
base=$1 repair=$2 node=$3
export PYTHONDONTWRITEBYTECODE=1
wt=$(mktemp -d); trap 'git worktree remove --force "$wt"' EXIT
git worktree add -q --detach "$wt" "$repair"
run() { (cd "$wt" && pytest -q -p no:cacheprovider -rA --tb=line "$node" 2>&1) | sed "s|$wt|<wt>|g"; }
git -C "$wt" checkout -q --no-overlay "$base" -- src/     # base source, exact tree
b=$(run) && bcode=0 || bcode=$?
git -C "$wt" checkout -q --no-overlay "$repair" -- src/
r=$(run) && rcode=0 || rcode=$?
echo "--- base (exit $bcode)";   grep -E '^(FAILED|ERROR|PASSED|SKIPPED|E )' <<<"$b" || true
echo "--- repair (exit $rcode)"; grep -E '^(FAILED|ERROR|PASSED|SKIPPED)' <<<"$r" || true
has() { awk -v l="$1" 'index($0, l " - ") == 1 || $0 == l { f = 1 } END { exit !f }'; }  # literal id
if [ $bcode -eq 1 ] && has "FAILED $node" <<<"$b" && ! grep -q '^ERROR' <<<"$b" \
   && [ $rcode -eq 0 ] && has "PASSED $node" <<<"$r"; then
  echo "CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?"
else
  echo "REJECT: this run does not show the test catching the bug"; exit 1
fi

--no-overlay is there because a plain git checkout <ref> -- src/ only adds and overwrites files. On a test repo where the repair added one source file and deleted another, the "base" run without the flag still had the repair's new file, and the swap back left the base's deleted one behind. Any git failure, like a mistyped ref, aborts the script before a verdict.

From the repo root, on main:

$ ./check_test.sh base repair tests/test_bug.py::test_keeps_the_partial_last_chunk
--- base (exit 1)
E   assert [[1, 2], [3, 4]] == [[1, 2], [3, 4], [5]]
FAILED tests/test_bug.py::test_keeps_the_partial_last_chunk - assert [[1, 2],...
--- repair (exit 0)
PASSED tests/test_bug.py::test_keeps_the_partial_last_chunk
CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?
$ ./check_test.sh base repair tests/test_vacuous.py::test_chunk_runs
--- base (exit 0)
PASSED tests/test_vacuous.py::test_chunk_runs
--- repair (exit 0)
PASSED tests/test_vacuous.py::test_chunk_runs
REJECT: this run does not show the test catching the bug
$ ./check_test.sh base wrong-reason tests/test_count.py::test_counts_the_partial_chunk
--- base (exit 4)
ERROR: found no collectors for <wt>/tests/test_count.py::test_counts_the_partial_chunk
E   ImportError: cannot import name 'chunk_count' from 'chunks' (<wt>/src/chunks.py)
ERROR tests/test_count.py
--- repair (exit 0)
PASSED tests/test_count.py::test_counts_the_partial_chunk
REJECT: this run does not show the test catching the bug
$ ./check_test.sh base wrong-reason tests/test_count_runtime.py::test_counts_at_runtime
--- base (exit 1)
E   ImportError: cannot import name 'chunk_count' from 'chunks' (<wt>/src/chunks.py)
FAILED tests/test_count_runtime.py::test_counts_at_runtime - ImportError: can...
--- repair (exit 0)
PASSED tests/test_count_runtime.py::test_counts_at_runtime
CANDIDATE: fails on base, passes on repair. Read the base failure: is it the bug?

The last two are the trap. Both tests need chunk_count, which the base doesn't have. The first imports it at the top of the file, so the base run dies during collection and the script rejects it. The second imports it inside the test function, so it fails at run time with exit 1 and a FAILED line, and the script calls it a candidate. It looks red. It proves nothing.

So the script narrows, it doesn't prove. It filters out what an exit code can show (a pass on the base, a collection error, a skip, a repair that doesn't pass), then hands you the base failure to read. The first run's E line is the missing [5], which is the bug. The fourth run's is an ImportError, which isn't.

Drop pythonpath from pyproject.toml and pytest -q tests/ stops on 6 collection errors with exit 2, on the base too. A check that only asks "did it fail on base?" would wave every test through.

One more hole: a test that returns its comparison instead of asserting it. With return chunk([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4]] in tests/test_return.py, pytest -q tests/test_return.py prints 1 passed, 1 warning, though the comparison is False. Add -W error::pytest.PytestReturnNotNoneWarning and it prints 1 failed.

The other half of the check matters as much. In my experiments, the first diagnostic probe for a Click regression bundled two properties, and it failed the upstream fix because of a separate, known bug. That made it invalid, and it was thrown out. A test that can't pass the real fix is measuring something else.

A team I work with hit the base-side failure in July. A regression test's setup rebuilt a database view with CREATE ... IF NOT EXISTS, silently reused a stale view from earlier work, and passed whatever the new SQL said. Running it once against the unfixed code would have shown it green. In September that team adopted OpenClaw's public test-audit skill, and its authoring gate puts it best: "A regression test that never demonstrably failed proves the mock, not the fix."

Check 2: Break the Repair and Watch the Test Go Red

A mutant is a copy of the code with one small deliberate change. If a test fails, the mutant is killed. If every test passes, it survived, and you've found something your tests can't see. Check 1 shows the test sees the original bug. Check 2 asks whether it would see a slightly wrong fix.

Start by hand. Read the repair diff and write the mistake an agent would most plausibly make in that exact line. Here the fix is a loop bound, so the obvious wrong fix is a different off-by-one in the same bound. That's mutant A, applied to the repaired source on main and then undone:

$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py
$ pytest -q tests/test_bug.py tests/test_vacuous.py
FAILED tests/test_bug.py::test_keeps_the_partial_last_chunk - assert [[1, 2],...
1 failed, 2 passed in 0.01s
$ git checkout main -- src/

The bug test kills it. Both vacuous tests pass it. The zero-tool version is a source rollback: revert one changed source file, and the test must fail. In my experiments, three of eight hand-made mutants that reverted urllib3 and Click code to obsolete API contracts passed the narrow feature test, and only the broader regression module caught them.

For a PR, that hand-written mutant of the changed line is the default. A tool sweep is broader, optional work. It costs one test run per mutant plus a manual read of every survivor, so skip it when that's more than the PR is worth.

I hadn't used a mutation tool before this experiment. Here's what mutmut found on the fixture. Its config is the two keys in pyproject.toml above, and older blog posts still show the deprecated paths_to_mutate and tests_dir. A small sweep.sh writes the test list into the second key, deletes mutants/, runs mutmut run and lists survivors:

#!/usr/bin/env bash
# Usage: ./sweep.sh TEST_PATH...
# Sets mutmut's test selection in pyproject.toml, runs a fresh sweep, lists survivors.
set -euo pipefail
sel=$(printf '"%s", ' "$@"); sel="[${sel%, }]"
sed -i.bak "s|^pytest_add_cli_args_test_selection = .*|pytest_add_cli_args_test_selection = $sel|" pyproject.toml
rm -f pyproject.toml.bak && rm -rf mutants
mutmut run > mutmut.log 2>&1 || { cat mutmut.log; exit 1; }
tr '\r' '\n' < mutmut.log | grep '🎉' | tail -1
mutmut results

Still on main, with the repair restored (🎉 killed, 🙁 survived):

$ ./sweep.sh tests/test_vacuous.py
13/13  🎉 4 🫥 0  ⏰ 0  🤔 0  🙁 9  🔇 0  🧙 0
$ ./sweep.sh tests/test_bug.py
13/13  🎉 8 🫥 0  ⏰ 0  🤔 0  🙁 5  🔇 0  🧙 0

The bug test alone left five survivors: three mangled error messages and two changes to the guard. One of them, from mutmut show chunks.x_chunk__mutmut_2:

-    if size <= 0:
+    if size <= 1:

Nothing calls chunk() with size=1, so nothing notices. The bug test was never meant to guard that line, and that's Check 3's job.

What the sweep missed matters as much. None of its 13 mutants shifted the bound the way mutant A does, and none deleted the guard. Its operators swap + for -, 0 for 1, values for None. The two mutants I'd most expect from an agent came from reading the diff, not from the tool.

After you add a test for a survivor, re-run the sweep. I've watched an agent publish a mutation table it never re-ran.

Know the limits. mutmut run exited 0 with 9 survivors, and cosmic-ray exec exited 0 with 16 of 21. Stryker fails a build only when thresholds.break is set, and its default is null. mutmut 3 doesn't mutate module-level code, so a wrong constant is invisible to it. Some survivors are equivalent mutants that no test can kill, so I'd read survivors one by one and skip the score. Google's 2018 report describes the same idea at scale: mutants only on the lines a change touches, shown during code review.

Check 3: A Preservation Probe Has to Pass Twice

A preservation probe tests behavior the change must not touch. It passes on the base, passes on the repair, and fails a mutant that breaks the preserved behavior. That's the mirror image of the bug test. A probe that fails the upstream fix is invalid, like the bundled probe in Check 1. A probe that fails the base isn't a preservation probe at all: it's a second bug test.

Why bother, when the project already has tests? Because a fix's own test only looks at the line that changed, and an agent's patch rarely stops there. It renames, simplifies a guard, reorders arguments. The two probes behind those 8 rejections in my experiments were exactly this kind: one checked that a Click multiple-value callback still received its collected arguments, the other that a zero-byte urllib3 read didn't stop a later full read from being cached.

On the fixture, the probe guards the size check, and it can be vacuous too. Mutant B deletes the guard. The weakest probe, tests/test_preserve_typeonly.py, only checks pytest.raises(ValueError) at size=0:

$ sed -i '/if size <= 0:/,+1d' src/chunks.py
$ pytest -q tests/test_preserve_typeonly.py
1 passed in 0.00s
$ git checkout main -- src/

It passes because range() raises ValueError for a zero step on its own. At size=-1 there's no error at all: range(0, 3, -1) is empty, and the guard-less chunk() returns []. OpenClaw's skill lists this exact junk pattern: "negative controls that pass for an unrelated reason".

tests/test_preserve_first.py adds match="size must be positive" and an exact-multiple case, chunk([1, 2, 3, 4], 2) == [[1, 2], [3, 4]]. That kills mutant B at size=0, and the sweep finds the next gap:

$ ./sweep.sh tests/test_bug.py tests/test_preserve_first.py
13/13  🎉 11 🫥 0  ⏰ 0  🤔 0  🙁 2  🔇 0  🧙 0
    chunks.x_chunk__mutmut_2: survived
    chunks.x_chunk__mutmut_4: survived

The first survivor is the size <= 1 guard from Check 2. The second raises "XXsize must be positiveXX". The pytest docs say match "is matched with the re.search() function", so the mangled message still matched. Neither trap is exotic. Both are a probe leaning on something other than the behavior it names.

The final probe, tests/test_preserve.py, probes -1, anchors the pattern and covers size=1:

import pytest
from chunks import chunk

@pytest.mark.parametrize("size", [0, -1])
def test_rejects_non_positive_size(size):
    with pytest.raises(ValueError, match=r"^size must be positive$"):
        chunk([1, 2, 3], size)

def test_size_one_is_allowed():
    assert chunk([1, 2, 3], 1) == [[1], [2], [3]]

def test_exact_multiple_unchanged():
    assert chunk([1, 2, 3, 4], 2) == [[1, 2], [3, 4]]

Pass on the base, pass on the repair, fail on mutant B, then mutant A and the sweep (summary lines only):

$ git checkout --no-overlay base -- src/ && pytest -q tests/test_preserve.py; git checkout main -- src/
4 passed in 0.00s
$ pytest -q tests/test_preserve.py
4 passed in 0.00s
$ sed -i '/if size <= 0:/,+1d' src/chunks.py && pytest -q tests/test_preserve.py; git checkout main -- src/
FAILED tests/test_preserve.py::test_rejects_non_positive_size[0] - AssertionE...
FAILED tests/test_preserve.py::test_rejects_non_positive_size[-1] - Failed: D...
2 failed, 2 passed in 0.01s
$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py && pytest -q tests/test_preserve.py; git checkout main -- src/
FAILED tests/test_preserve.py::test_size_one_is_allowed - assert [[1], [2]] =...
1 failed, 3 passed in 0.01s
$ ./sweep.sh tests/test_bug.py tests/test_preserve.py
13/13  🎉 13 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0  🧙 0

The probe also catches mutant A, through its size=1 case: chunk([1, 2, 3], 1) loses the [3]. The probe wasn't aimed at mutant A. That's the bonus of pinning plain boundary behavior: it catches more than the regression you had in mind.

On an agent PR, I'd pick probes from what the changed function promised before the change: its error cases, its boundary inputs, the flags or state it leaves behind for the next caller. One assertion per promise, run on the base first. If it fails there, you've misread the old behavior, and that's worth knowing before you judge the patch.

Check 4: An Oracle the Test's Author Didn't Write

A test and its implementation can agree because both misread the issue. Then Checks 1 through 3 all pass, and both are wrong. That's the risk when one agent writes the code and the expected values from the same reading: as Kent Beck puts it, copying computed values into expectations "defeats double checking".

On the fixture, the standard library supplies an independent oracle: itertools.batched already splits an iterable into batches where "the last batch may be shorter than n". oracle.py sits in the repo root:

"""Independent oracle: compare chunk() to itertools.batched on random inputs.
Exit 0 = agrees everywhere, 1 = first disagreement."""
import itertools, random, sys
sys.path.insert(0, "src")
from chunks import chunk

rng = random.Random(0)
for _ in range(2000):
    items = list(range(rng.randrange(0, 12)))
    size = rng.randrange(1, 6)
    want = [list(b) for b in itertools.batched(items, size)]
    if (got := chunk(items, size)) != want:
        sys.exit(f"DISAGREE chunk({items}, {size}) = {got}, expected {want}")
print("oracle: 2000 random cases agree")

On the repair, the base and mutant A:

$ python oracle.py
oracle: 2000 random cases agree
$ git checkout --no-overlay base -- src/ && python oracle.py; git checkout main -- src/
DISAGREE chunk([0, 1, 2, 3, 4, 5], 4) = [[0, 1, 2, 3]], expected [[0, 1, 2, 3], [4, 5]]
$ sed -i 's/range(0, len(items), size)/range(0, len(items) - 1, size)/' src/chunks.py && python oracle.py; git checkout main -- src/
DISAGREE chunk([0, 1, 2, 3, 4, 5, 6], 3) = [[0, 1, 2], [3, 4, 5]], expected [[0, 1, 2], [3, 4, 5], [6]]

The oracle passes Check 1 itself: it disagrees on the base and agrees on the repair. It also found its own counterexamples, [0, 1, 2, 3, 4, 5] with size 4, not the five-item list the bug test uses. The seed is fixed, so a disagreement reproduces every time.

Most PRs have no batched to lean on. Then I'd start smaller: before opening the patch, write two input and output pairs from the issue text, so the agent's code can't shape what you expect. When the issue has no examples, a reference implementation (a slow brute-force version, or the old code on inputs the change shouldn't affect) or one recorded real input from the system that produces it does the same job.

In June a webhook relay my agents built passed its tests, its review rounds and CI while silently dropping real comments: the code and its tests were written against the same invented webhook shape, and the real producer sent a flat one. Logging one real payload exposed it.

What I Ask of a Test Before I Trust It

Here's the whole map:

Two rows lean on Check 1, which is why it comes first. The last row is the only one that running the agent's own test more carefully can never fix, because the mistake sits in what the test expects.

The bar I set for my own graders is the bar I'd set for any test offered as evidence: it fails on the base for an assertion, passes on the fix, and goes red when a changed source file is reverted. My grading harness has since adopted one more rule: each task's hidden tests must also reject at least two targeted wrong patches. The fixture already has two good kinds. Mutant A is a partial repair and mutant B a regression, and Checks 2 and 3 are that rule in its cheapest form.

If you only run one check on an agent's test, run Check 1. It's one script and two pytest runs, and it filters the two failures in the table that look most like success. Build the five-line fixture, run ./check_test.sh base repair tests/test_bug.py::test_keeps_the_partial_last_chunk, and then point it at the next agent PR that says the tests pass.

When an agent's PR says "tests pass", which of these checks do you actually run before merging, and on which PRs do you skip them?

More on this on potapov.dev:

• When to Trust AI Agent Tests and Reviews
• Why AI Agents Declare Victory Too Early