The Agent's Tests Passed. Mutation Testing Showed 2 of 4 Faults Survived.

The agent patch passed the gates I ran on it. Its unit tests were green, fixtures matched, nothing was flaky. Then I seeded four faults into the implementation, one at a time. Two survived.

That gap is what this article is about. A green suite is a claim, not a measurement. Mutation testing turns it into a measurement: introduce a fault, run the suite, and see whether the suite notices. I now run this loop before merging any agent-written patch, and the whole thing costs a few rebuilds.

Why green tests lie

A passing test proves one thing only: the test and the implementation agree on the inputs the test exercised. When an agent writes both the patch and the tests, the tests inherit the patch's assumptions. If the implementation encodes a wrong assumption, the test encodes the same one. The suite is green because it is blind, not because the code is right.

The patch in this article came from a free model on MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model wrote a bounded queue and a test file. The test file was not wrong. It was blind in exactly the place the implementation was wrong.

The method: five steps

Mutation testing is easy to describe and awkward to skip:

1. Freeze flaky tests first. A flaky test fails at random, so it makes every mutation look like a kill. The signal is garbage. This is the flaky freeze from the gates post; without it, the numbers mean nothing.
2. Select the functions the patch touched. Mutating untouched code measures someone else's tests.
3. Generate mutations. Each mutation is one small fault: drop a modulo, flip a comparison, change an increment.
4. Run the suite against each mutation. Rebuild, run, record.
5. Gate on the kill rate. A surviving mutation means the suite cannot detect that fault class. Send the patch back with the survivor list as evidence.

The artifact

A minimal bounded queue, the agent's test, and a small Python driver. The queue:

// bounded_queue.h
#pragma once

#include <cstddef>
#include <optional>
#include <vector>

template <typename T>
class BoundedQueue {
public:
    explicit BoundedQueue(std::size_t capacity)
        : data_(capacity), head_(0), tail_(0), count_(0) {}

    bool push(const T& value) {
        if (count_ == data_.size()) {
            return false;
        }
        data_[tail_] = value;
        tail_ = (tail_ + 1) % data_.size();
        ++count_;
        return true;
    }

    std::optional<T> pop() {
        if (count_ == 0) {
            return std::nullopt;
        }
        T value = data_[head_];
        head_ = (head_ + 1) % data_.size();
        --count_;
        return value;
    }

    std::size_t size() const { return count_; }
    bool empty() const { return count_ == 0; }

private:
    std::vector<T> data_;
    std::size_t head_;
    std::size_t tail_;
    std::size_t count_;
};

The agent's test, which passes on the first run:

// test_queue.cpp — agent-written, all assertions pass
#include "bounded_queue.h"

#include <cassert>

int main() {
    BoundedQueue<int> q(3);

    assert(q.empty());
    assert(q.push(1));
    assert(q.push(2));
    assert(q.push(3));
    assert(!q.push(4));  // full

    assert(q.pop() == 1);
    assert(q.pop() == 2);
    assert(q.pop() == 3);
    assert(q.empty());

    return 0;
}

The driver applies one fault at a time, rebuilds with ASan and UBSan, and reports survivors:

#!/usr/bin/env python3
"""Mutation driver: apply one fault, rebuild, run, report survivors."""
import subprocess
import sys

SRC = "bounded_queue.h"
TEST = "test_queue.cpp"
BIN = "/tmp/queue_test"

MUTATIONS = [
    ("drop_tail_wrap", "tail_ = (tail_ + 1) % data_.size();", "tail_ = (tail_ + 1);"),
    ("drop_head_wrap", "head_ = (head_ + 1) % data_.size();", "head_ = (head_ + 1);"),
    ("double_count", "++count_;", "count_ += 2;"),
    ("corrupt_value", "data_[tail_] = value;", "data_[tail_] = value + 1;"),
]


def build() -> bool:
    return subprocess.run(
        ["g++", "-std=c++17", "-fsanitize=address,undefined", TEST, "-o", BIN],
        capture_output=True,
    ).returncode == 0


def test() -> bool:
    return subprocess.run([BIN], capture_output=True).returncode == 0


def main() -> int:
    original = open(SRC).read()

    if not build():
        print("build failed; fix the harness first")
        return 1
    print(f"baseline: {'PASS' if test() else 'FAIL'}")

    killed = 0
    total = 0
    for name, before, after in MUTATIONS:
        if before not in original:
            print(f"{name}: SKIP (pattern not found)")
            continue
        total += 1
        open(SRC, "w").write(original.replace(before, after))
        if not build():
            killed += 1
            print(f"{name}: killed (build failure)")
        elif test():
            print(f"{name}: SURVIVED")
        else:
            killed += 1
            print(f"{name}: killed")
        open(SRC, "w").write(original)

    print(f"kill rate: {killed}/{total}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

What the run reported

baseline: PASS
drop_tail_wrap: SURVIVED
drop_head_wrap: SURVIVED
double_count: killed
corrupt_value: killed
kill rate: 2/4

The two survivors are the interesting ones. Both drop the modulo that wraps the queue's head and tail indices. The agent's test pushes three items, pops three items, and never wraps. The indices grow past the buffer, and the test never looks. ASan and UBSan were enabled for every run. They stayed silent, because the buggy path was never executed.

Sanitizers are witnesses, not testers. They report what the test executes; they do not invent executions.

The one-line fix

The mutation report tells you which test to add, not just that one is missing. For the wrap mutations, the missing test is a wrap:

// wrap-around regression test
for (int i = 0; i < 100; ++i) {
    assert(q.push(i));
    assert(q.pop() == i);
}

This pushes and pops one element at a time, so head and tail wrap on every iteration. With drop_tail_wrap, the second wrap writes past the buffer and ASan aborts. With drop_head_wrap, the read goes out of bounds. Both mutations die. The kill rate becomes 4/4.

Which gate catches what

The property-check column assumes a real property: 10,000 random operations compared against a model. The agent's test file had no property check. The table shows what one would have caught — random operations wrap the queue constantly, so both wrap mutations die. Mutation testing told me the property was missing before I wrote it.

The fixture column is missing from this table on purpose. Fixtures verify documented states against a known-good baseline. They are excellent at catching regressions in behavior you already understand, and weak at catching faults you have not imagined — which is exactly the fault class an agent patch tends to introduce. Mutation testing does not replace fixtures. It measures whether the fixtures, the unit tests, and the property checks together can fail.

Limitations

Mutation testing has real costs. Equivalent mutants — faults that do not change behavior — survive and inflate the signal. You learn to recognize them and exclude them from the gate. The kill rate also measures test sensitivity, not patch correctness. A 4/4 score means the suite detects these four faults; it does not mean the patch has no other bugs.

The loop costs builds: one per mutation plus the baseline. For a one-line patch, that overhead is usually not worth it. Who should not use this: teams with a tiny CI budget, patches that touch a single expression, or suites that are already flaky. Fix the flake first. Mutation testing on a flaky suite is astrology with a compiler.

Closing

I run this loop on MonkeyCode's free server option before merging agent patches. The cost is a few rebuilds per changed function, which the free tier absorbs. The output is not a pass or fail. It is a list of fault classes your suite cannot see — which is exactly the evidence a reviewer needs.