What Mutation Testing Frameworks Do When a Timeout Kills Them

Code: Megapixel99/assay-checks

assay audits mutation harnesses: the scripts that deliberately break your code and check that a test notices. It looks for seven properties, each one a way a harness can report success without having run anything, and its README admits in writing that the seven have a hole underneath them. SIGKILL cannot be caught, blocked or handled, so no handler runs and no finally runs. The ordinary way to be SIGKILLed is a timeout rather than an impatient person; subprocess.run(..., timeout=...) kills the child outright, and so does the kill step of a CI runner that has waited long enough. A harness satisfying all seven, invoked under a timeout it then exceeds, leaves your tree mutated exactly as though it carried none of them.

That argument had never been pointed at anybody else's tool. So I built a conformance suite and pointed it at four: mutmut 3.7.0, cosmic-ray 8.4.6, Stryker 8.7.1 and PIT 1.16.1, all four of them other people's work, all four pinned in their own container. The method is one sentence: hash the tree, start the framework, wait until a source file is observably mutated on disk, kill it at that instant, and hash the tree again.

I expected four dirty trees. I got one.

Dirty means a file under test was left mutated. Scratch means the code under test came back and the run left other paths behind. Clean means every file is byte-for-byte what it was.

Three of the four never mutate your tree at all

This is the part I had not modelled, and it makes the SIGKILL question inapplicable rather than answered. mutmut copies the project into mutants/ and mutates the copy; Stryker copies it into .stryker-tmp/ and mutates that. PIT does not write a mutated .java file anywhere at all, because it mutates bytecode in memory. Their source survives a kill because their source was never dirty, and no signal handler had anything to do with it.

The hole is closed by architecture, and that is a stronger guarantee than handling a signal; handling a signal is a promise a process makes about code it will get to run. There is no code that runs after SIGKILL. There is nothing to clean up only if you never put anything there.

The one that does mutate in place fails earlier than I predicted

cosmic-ray edits the file under test and restores it afterwards, which is the design the whole argument was written about. Its own log says so during a clean run:

INFO:cosmic_ray.mutating:Applying mutation: path=calc/__init__.py, op=<...NumberReplacer object...>, occurrence=2
INFO:cosmic_ray.testing:Running test (timeout=60.0): python -m pytest -q tests

An uninterrupted run takes 71.58 seconds on this fixture and restores everything it touched. A mutation is on disk within half a second of the start, and if you kill it there, calc/__init__.py stays mutated.

What surprised me is which signal does it. I built the suite for SIGKILL, and cosmic-ray does not get that far: a plain SIGTERM delivered to the cosmic-ray process, which is what timeout and subprocess.run(..., timeout=...) send before they escalate, leaves the file mutated too. SIGTERM is inside assay's seven properties rather than beyond them, so this is a failure of the check I already had, found by the suite I built for the one I did not.

I want to be precise about the scope, because cosmic-ray is a careful project and this is a narrow finding. The uninterrupted run is correct: 51 jobs, every mutation restored, tree clean apart from a .pytest_cache/ that the default test command writes. The failure needs a signal to arrive inside a window that is a fraction of a second wide on a fixture chosen to make that window as wide as possible. On a real codebase with a slow suite that window is proportionally narrower per mutant and there are far more of them, which is the same total exposure arriving in smaller pieces.

Two of my own numbers were artifacts

The first table I generated said Stryker survived everything cleanly. Stryker had crashed.

The image was node:22-bookworm-slim, which does not ship procps, and Stryker shells out to ps to reap its test runners. The run died with spawn ps ENOENT immediately after the dry run, having mutated nothing, and left a spotless tree. Four passing verdicts about a run that never happened, which is exactly the evidence property that assay audits harnesses for, arriving one level up inside the thing built to check for it. Every framework now has to print a declared proof-of-work string in its baseline or the whole row reports NO-RUN instead of a pass.

The second one was worse, because it produced plausible numbers rather than obviously broken ones. I was invoking cosmic-ray through sh -c and Stryker through npx, so when the suite sent a signal to "the process it started", it was signalling a wrapper. The wrapper died and the framework was orphaned; the framework then ran to completion unwatched, and tidied up after itself. Both reported clean, and both were measuring a signal that never reached the thing under test.

The probe now records whether anything outlived the process it signalled, and both invocations were changed so that the framework itself is the process. Correcting that flipped Stryker's leader column from clean to scratch and cosmic-ray's from scratch to dirty. Two of eight cells in that column were artifact, and the cosmic-ray one was the headline.

Why the clean rows mean anything

A column of clean verdicts proves nothing unless the probe can be shown to report dirty when the tree really is dirty. Otherwise "nothing found" and "nothing looked" are the same output, which is the failure assay's seven properties exist to name. So the suite carries a calibration row that is not a framework at all: a harness I wrote that mutates in place and satisfies all seven properties. assay runners says so, rather than a comment claiming it:

$ PYTHONPATH=python python3 -m assay --root conformance/frameworks/control-inplace runners
  ok       mutations_calc.py

That harness is clean under both SIGTERMs, because it turns SIGTERM into an exception so its finally runs, and dirty under SIGKILL, because nothing turns SIGKILL into anything. It is the README's claim compiled and executed, and it is the reason the three clean rows above are evidence rather than an absence of evidence.

The kill is also timed on an observation rather than a stopwatch. The probe hashes the tree every 50 ms and fires the moment a watched file changes, because a blind timeout can land between two mutants where every framework looks clean. Where no in-place mutation ever appears, the report says so and kills mid-run anyway at half the framework's own measured baseline, so that "we never managed to interrupt it" and "it survived being interrupted" stay different findings. Every fixture's tests sleep 400 ms apiece for the same reason: a suite that finishes instantly cannot be interrupted in the middle of one.

What I got wrong, and what it changes

My model of a mutation harness was the one I had written myself: something that edits the file under test and puts it back. Three of the four most widely used tools do not work that way. So the remedy the README assigns to the invoker, check that the tree came back rather than trust that the harness was given the chance to put it back, is load-bearing for a narrower class than I had written. It is correct, and it is the right check to run in CI; for mutmut, Stryker and PIT it will simply never fire.

The advice that generalises is not the one I set out to confirm. If you are writing a mutation harness, the fix for the signal you cannot catch is not to catch more signals; it is to mutate a copy. Every property in that table of seven is a promise about code that runs, and the whole point of SIGKILL is that no code runs.

Three properties are still unmeasured here, and I would rather say so than let the table imply otherwise. dead-vs-real, parses-mutant and restore-verified are claims about how a framework scores what it sees, not about what it leaves on disk. Probing them means handing each framework a mutant it ought to refuse to count and then reading its report. That is a different instrument, and unlike this one it has to be written once per framework.

The suite is in conformance/, the committed results are in conformance/results/, and it needs Docker and Python and nothing else. Adding a framework is a Dockerfile and a JSON file. If one of these numbers is wrong, python3 conformance/run.py cosmic-ray takes about ninety seconds to say so.