Dead Probes Pass: Mutation-Check Your Characterization Tests Before You Refactor

A passing characterization suite is not a safety net. It is a net only if it fails on wrong behavior. Here is how to verify that in twenty minutes.

You wrote probes against a messy legacy module. They all pass. You feel ready to refactor. Do not trust that feeling. Passing on current code proves nothing. A probe earns trust by failing on mutated code.

This post walks through a three-mutation audit. It finds dead probes before they fail you. Then it makes the smallest safe change with evidence.

The core idea

A characterization probe locks in current behavior. Some probes lock in nothing. They pass on every plausible bug. Those probes are dead weight. They create false confidence.

Mutation testing fixes this. Inject a small fault into the legacy code. Run the probes. A probe that still passes is dead. A probe that fails is alive. Keep only the alive ones.

The audit workflow

Four steps. No paid tools. No local GPU.

1. Draft probes with a free model.
2. Draft three mutations per behavior.
3. Run the probe-versus-mutation matrix.
4. Keep only probes that catch at least one mutation.

Step 1: Draft probes with a free model

Here is the legacy function for this walkthrough.

# legacy_pricing.py — messy on purpose
def price(order, customer):
    total = 0
    for item in order["items"]:
        p = item["price"]
        if item.get("discount"):
            p = p * 0.9
        if customer["tier"] == "gold" and p > 100:
            p = p - 10
        total = total + p
    if len(order["items"]) > 5:
        total = total * 0.95
    return round(total, 2)

Ask MonkeyCode's free model to draft characterization probes. Give it the function and one rule. Probe observable behavior, not implementation. Review every probe. The model drafts. You decide.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model returns three probes.

def test_item_discount_applies():
    order = {"items": [{"price": 100, "discount": True}]}
    assert price(order, {"tier": "basic"}) == 90.0

def test_gold_tier_boundary():
    order = {"items": [{"price": 100}]}
    assert price(order, {"tier": "gold"}) == 100.0

def test_bulk_discount_applies():
    order = {"items": [{"price": 10}] * 6}
    assert price(order, {"tier": "basic"}) == 57.0

Check the math. Item discount: 100 times 0.9 equals 90. Gold boundary: 100 is not greater than 100, so no cut. Bulk: 60 times 0.95 equals 57. All correct.

Note the second probe. It tests the boundary, not the middle. Boundary probes catch off-by-one mutations. That matters in Step 2.

Step 2: Draft three mutations per behavior

Now ask the model for three small mutations. One per behavior. Each mutation changes a decision or a value.

MUTATIONS = [
    ("bulk_threshold_gt_6", "if len(order['items']) > 5:", "if len(order['items']) > 6:"),
    ("gold_gt_to_ge", "p > 100", "p >= 100"),
    ("discount_removed", "p = p * 0.9", "p = p"),
]

These are the faults your probes must catch. A probe that passes on all three protects nothing.

Why three? One mutation per behavior is the minimum. Three gives you a signal. If a probe misses two of three, it is weak. If it misses all three, it is dead. Add more later. Start with three.

Step 3: Run the matrix on a free server

You need a runtime for the matrix. MonkeyCode's free server option works here. No local setup. Just a Python process.

Concatenate the legacy function and your probes into one file. The runner mutates that file in memory.

# audit_probes.py
def run_probes(source):
    ns = {}
    exec(compile(source, "mutated_legacy.py", "exec"), ns)
    failures = []
    for name, fn in sorted(ns.items()):
        if name.startswith("test_") and callable(fn):
            try:
                fn()
            except AssertionError:
                failures.append(name)
    return failures

def collect_probe_names(source):
    ns = {}
    exec(compile(source, "legacy_pricing.py", "exec"), ns)
    return {n for n in ns if n.startswith("test_")}

def audit(original_with_probes, mutations):
    alive = set()
    for label, old, new in mutations:
        mutated = original_with_probes.replace(old, new, 1)
        failed = run_probes(mutated)
        alive.update(failed)
        print(f"{label:24} -> {failed}")
    dead = collect_probe_names(original_with_probes) - alive
    print(f"dead probes: {sorted(dead) or 'none'}")

Run it. Read the matrix.

bulk_threshold_gt_6     -> ['test_bulk_discount_applies']
gold_gt_to_ge           -> ['test_gold_tier_boundary']
discount_removed        -> ['test_item_discount_applies']
dead probes: none

Every probe failed on its own mutation. The suite is alive. That is the result you want.

Now add a lazy probe. It checks a type, not a behavior.

def test_bulk_discount_returns_float():
    result = price({"items": [{"price": 10}] * 6}, {"tier": "basic"})
    assert isinstance(result, float)

Run the audit again.

bulk_threshold_gt_6     -> ['test_bulk_discount_applies']
gold_gt_to_ge           -> ['test_gold_tier_boundary']
discount_removed        -> ['test_item_discount_applies']
dead probes: ['test_bulk_discount_returns_float']

The type probe never failed. It is dead. It adds confidence without adding protection. Delete it.

Step 4: Keep alive probes, delete dead ones

Use this decision table.

The third row is a warning. A probe that fails on every mutation is brittle. It may assert implementation details. Treat that as a signal to rewrite.

Run the audit after every probe change. Dead probes reappear as you edit. Keep the matrix output in your commit message.

Then make the smallest safe change

The audit is the gate, not the refactor. Pick one small change. Extract the item pricing into a pure function.

def item_price(item, tier):
    p = item["price"]
    if item.get("discount"):
        p = p * 0.9
    if tier == "gold" and p > 100:
        p = p - 10
    return p

def price(order, customer):
    total = sum(item_price(item, customer["tier"]) for item in order["items"])
    if len(order["items"]) > 5:
        total = total * 0.95
    return round(total, 2)

Run the surviving probes. They must pass. Then measure the diff. Small diff. Same behavior. Verified probes.

Limitations

This audit has real limits. Mutations are guesses, not real bugs. A probe can catch a mutation and still miss a regression. The audit only covers behaviors you probed. Side effects, database calls, and network I/O stay invisible.

The audit also depends on probe quality. Weak probes produce weak verdicts. Boundary values catch more mutations than middle values. Write probes like you are hunting off-by-one errors.

Who should not use this

Skip this if your module already has strong tests. Skip it if the legacy code has heavy external side effects. Skip it if you cannot review the probes yourself. The audit amplifies your judgment. It does not replace it.

The takeaway

A suite that never fails on a mutation is decoration. Mutation-check it before you refactor. Then make the smallest change. Let the alive probes guard it.

Try this on your messiest module. If you need a free runtime for the matrix, MonkeyCode's free server is one option. Bring your own review. The net is only as good as the probes you keep.