An agent patch that passes its own tests is a baseline, not a verdict. The same model wrote the code and the tests, so both share the same blind spots. Mutation testing scores the tests themselves: inject a fault, run the suite, and see whether it notices. In practice, the first mutant often survives.
Previous rounds on this account established three gates before merge: property checks, fixtures, and a freeze on flaky tests. This round adds a fourth gate that runs after the suite is green. It answers a different question — not "does the patch work?" but "would the tests catch it if it didn't?"
Why green tests from an agent are weak evidence
Code coverage measures execution, not detection. A test can execute a line and still miss the bug on it. A suite that only checks is_even(2) and is_even(4) runs both lines, passes both assertions, and stays blind to a mutation that flips == to !=.
Agents produce this shape of test by default. They follow the happy path, mirror the implementation, and rarely probe boundaries. The result is a suite that is green, fast, and weak for regression.
Mutation testing converts that intuition into a number. For each small fault, rebuild and rerun. If the tests fail, the mutant is killed. If they pass, it survived — and you found a hole in the suite, not in the code.
A minimal harness
The harness below applies one mutation at a time to the implementation file, compiles it together with an unchanged test file, runs the resulting binary, and records the outcome. It is deliberately small: regex-based, two files, no dependencies beyond a compiler.
#!/usr/bin/env python3
# mutate.py — score a test binary against source mutations.
import re
import subprocess
import sys
import tempfile
from pathlib import Path
MUTATIONS = [
("eq_to_neq", r"==", "!="),
("lt_to_le", r"<", "<="),
("add_to_sub", r"\+", "-"),
("zero_to_one", r"return 0;", "return 1;"),
]
def mutate_once(src: str, pattern: str, replacement: str, nth: int) -> str:
matches = list(re.finditer(pattern, src))
if nth >= len(matches):
return src
m = matches[nth]
return src[:m.start()] + replacement + src[m.end():]
def main() -> int:
src = Path(sys.argv[1]).read_text()
test_file = sys.argv[2]
results = []
for name, pattern, replacement in MUTATIONS:
for nth in range(len(re.findall(pattern, src))):
mutated = mutate_once(src, pattern, replacement, nth)
with tempfile.NamedTemporaryFile("w", suffix=".cpp", delete=False) as f:
f.write(mutated)
tmp = f.name
build = subprocess.run(
["g++", "-std=c++20", tmp, test_file, "-o", "/tmp/mutant_test"],
capture_output=True,
)
if build.returncode != 0:
results.append((name, nth, "COMPILE_FAIL"))
else:
run = subprocess.run(["/tmp/mutant_test"], capture_output=True)
results.append((name, nth, "KILLED" if run.returncode != 0 else "SURVIVED"))
Path(tmp).unlink()
killed = sum(1 for _, _, r in results if r == "KILLED")
print(f"mutations={len(results)} killed={killed} survived={len(results) - killed}")
for name, nth, r in results:
print(f" {name}[{nth}]: {r}")
return 0
if __name__ == "__main__":
sys.exit(main())
The script takes two arguments: the implementation file and the test file. Only the implementation is mutated; the tests stay untouched. That separation is the whole point — you are scoring the tests, not rewriting them.
Worked example: a survivor in the first run
Here is the patch under review: a scoring function with tests.
// score.cpp — implementation only
int score(int points, int bonus) {
if (points < 0) return 0;
return points + bonus;
}
// test_score.cpp — tests only
#include <cassert>
int score(int, int);
int main() {
assert(score(-5, 10) == 0);
assert(score(5, 10) == 15);
}
The suite passes. The harness does not agree:
$ g++ -std=c++20 score.cpp test_score.cpp -o test_score
$ ./test_score
$ python3 mutate.py score.cpp test_score.cpp
mutations=3 killed=2 survived=1
lt_to_le[0]: SURVIVED
add_to_sub[0]: KILLED
zero_to_one[0]: KILLED
Two mutants are killed: flipping + to - breaks score(5, 10) == 15, and changing return 0; to return 1; breaks the negative case. The survivor is the interesting one. points < 0 became points <= 0, and the suite did not notice, because no test exercises points == 0.
That is a boundary hole, not a theoretical one. The next caller who passes score(0, 10) gets 0 instead of 10 once the mutation lands. One assertion fixes it:
assert(score(0, 10) == 10);
Re-run the harness:
mutations=3 killed=3 survived=0
The gate now passes. Note what changed: not the implementation, not the test count — the test's ability to detect a fault.
A merge policy for the gate
The harness produces a number; the policy decides what to do with it.
- Build the test binary from the agent's patch and run the suite once. Any failure here rejects the patch before mutation testing starts.
- Run the harness over the files the patch touched, not the whole tree. Mutating unrelated code wastes rebuild time.
- Compute the kill ratio as
killed / (killed + survived), ignoringCOMPILE_FAIL. Start with a threshold of 0.8 and adjust per project; the threshold matters less than the review of survivors. - Review every survivor. Classify it as an equivalent mutant (behavior unchanged) or a missing test case. For each missing case, add a test and re-run.
- Treat a flaky run as a failed run. If the suite is nondeterministic, the harness output is noise; freeze the merge until the flake is fixed.
Step 4 is where the value lives. The kill ratio is a signal, but the survivor list is the actual review artifact. It tells you exactly which behaviors the agent's tests cannot distinguish.
Limitations and who should not use this
Mutation testing is expensive. Each mutant is a full rebuild plus a test run, so a patch touching many files can take hours on a large project. Mitigations: mutate only the patch's files, keep the mutation list short, and run the gate on a machine you do not need for anything else.
The regex-based harness is fragile. It can mangle code and produce COMPILE_FAIL noise, and it cannot express type-level or structural mutations. Production projects should use a real mutation tool instead of this script; the script is for small patches and for learning the shape of the problem.
The kill ratio is a proxy, not proof. A suite can kill every mutant and still miss races, I/O errors, or integration failures. This gate complements sanitizer runs; it does not replace them.
Skip this gate when a rebuild takes more than a few minutes, when the patch is a throwaway script, or when the test suite is already flaky. Fixing flakiness first is a precondition, not an optimization.
Running the loop without burning the laptop
Mutation testing is compute-heavy: dozens of rebuilds per small patch. The loop has no interactive step, so it maps cleanly onto a remote runner. MonkeyCode's free server option is a fit for this shape of workload, and the free model access can generate the initial patch and tests.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The harness is small enough to read in one sitting. Point it at your next agent patch and read the survivor list before you merge. The first survivor will tell you more about the suite than the green checkmark did.
