A Characterization Suite That Never Caught a Bug Won't Protect Your Refactor

A characterization suite that never caught a bug will not protect your refactor. It locks behavior only if it fails when behavior changes. You need proof before you trust it.

The fastest proof is mutation scoring. Seed a bug. Run the suite. Count the misses. Plan for about 30 minutes on a messy repo.

Why characterization tests fail silently

Characterization tests record current behavior. They do not assert intended behavior. That makes them the right tool for legacy code.

The catch: a test can pass forever without testing anything. It might exercise a path nobody changes. It might assert the wrong output. You will not know until a refactor breaks it.

Mutation scoring removes that blind spot. It changes behavior on purpose. Then it checks whether your suite notices. This is the same idea behind tools like Stryker and PIT. You do not need those tools here. A few patches and a shell script are enough.

Step 1: Capture behavior as golden masters

Pick the module you plan to refactor. Build a small set of representative inputs. Record the outputs before you change anything.

// capture.js — record current behavior as golden masters
const fs = require('fs');
const { execSync } = require('child_process');

const cases = [
  { name: 'valid', file: 'fixtures/valid.json', args: [] },
  { name: 'empty', file: 'fixtures/empty.json', args: [] },
  { name: 'strict', file: 'fixtures/broken.json', args: ['--strict'] },
];

for (const c of cases) {
  const out = execSync(
    `node src/parser.js ${c.args.join(' ')} < ${c.file}`,
    { encoding: 'utf8' }
  );
  fs.writeFileSync(`golden/${c.name}.snap`, out);
}

Run it once. Commit the snapshots. Now you have a baseline.

Choose inputs that cover the edges. Empty input. Malformed input. Maximum input. Each edge is one more thing the refactor cannot break.

Step 2: Seed mutations and score the suite

Create small patches that change behavior. Flip a comparison. Swap an index. Remove an early return. Change a default value. Each patch is one mutation.

#!/usr/bin/env bash
# score-suite.sh — seed a bug, see if the suite catches it
set -euo pipefail

caught=0
missed=0

for mutation in mutations/*.patch; do
  git apply "$mutation"
  if npm test > /dev/null 2>&1; then
    echo "MISSED: $mutation"
    missed=$((missed + 1))
  else
    echo "CAUGHT: $mutation"
    caught=$((caught + 1))
  fi
  git checkout -- .
done

echo "caught=$caught missed=$missed"

Record the results in a table. Here is an example from one run.

Aim for a catch rate above 80 percent. If you miss, add more golden inputs. Do not refactor until the score is acceptable.

Step 3: Ask the model for the smallest safe change

Now the suite is a gate. A free model endpoint can propose the refactor step. The suite decides whether the proposal is safe.

Use a constrained prompt. The model should not redesign anything. It should produce one atomic change.

You are refactoring a legacy module. Characterization tests lock current behavior.
Propose ONE change that:
- touches the fewest lines
- keeps every characterization test green
- does not rename public functions or alter return types
Return a diff and a one-line rationale.

Small changes are easier to review. Small changes are easier to revert. Small changes isolate failures. A model that rewrites the whole file defeats the purpose of the suite.

Step 4: Apply, run, inspect

Apply the proposed diff. Run the golden suite. Inspect the diff size.

git apply refactor.patch
npm test
git diff --stat

Green suite plus a small diff means the change is safe. Red suite means the model misunderstood the behavior. Revert and re-prompt with the failing case.

Repeat the loop. One atomic change at a time. The diff budget stays visible in git diff --stat.

Limitations

Golden masters only cover the inputs you recorded. Untested paths stay invisible. A high mutation score is a lower bound, not a guarantee.

Free model endpoints can be non-deterministic. The same prompt may return different diffs. Always review the output before applying it.

The suite protects behavior, not quality. It will not tell you if the code is better. That judgment stays with you.

Who should not use this

This workflow is for messy legacy code. Greenfield projects have real unit tests. Use those instead.

Teams that cannot review diffs should not auto-apply model output. A gate is only as good as the human behind it.

Where to run the loop

You need compute for the golden-master runs. You need a model for the candidate diffs. MonkeyCode's free server option covers the first. Its free model access covers the second. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Measure the suite first. Then let the model refactor. That order prevents silent regressions.