Witness Maps: Never Refactor a Line That No Test Executes

A 41-file cleanup diff lands on a Tuesday morning, the CI suite is green, and the reviewer approves it in nine minutes. Two days later an invoicing edge case breaks in production, inside a function that no test has ever entered. The suite was not lying — it simply never executed the lines that changed.

That gap is the entire problem. A green run tells you the paths your tests cover still behave, and it stays silent about the paths they never touch. Aggregate coverage numbers hide this further, because a repository reported at 78% line coverage still leaves roughly a fifth of its lines anonymous to every assertion in the build.

This post describes a small gate you can build in an afternoon: a witness map. It answers one narrow question before you accept a refactor — for each line inside the spans you plan to touch, which existing test executes it?

Why suite-level green is the wrong unit of evidence

Coverage tools report per file, per function, or per branch, and reviewers read those aggregates as a proxy for safety. The unit that actually matters during a refactor is the individual line you are about to rename, extract, or reorder. Three properties of real code make the aggregate misleading:

1. A module can sit at 90% coverage while the exact ten lines you plan to extract sit in the remaining 10%.
2. Coverage counts execution, not assertion — a line can be executed by a test that never checks its result.
3. AI-generated cleanup diffs tend to touch many small spans at once, so the probability that at least one span is unwitnessed grows quickly with diff size.

If you have been feeling the review-queue pressure that shows up in so many recent posts about AI-assisted refactoring, this is the mechanical cause. The bottleneck is not generating the diff; it is localizing which lines of the diff are safe.

Step 1: Freeze the candidate diff against the current tree

A witness map only works if old-side line numbers match the code you measured. Produce the candidate diff against the branch you actually intend to modify, and verify it applies cleanly before doing anything else.

# candidate.diff must be relative to the current HEAD, not to a rewritten branch
git apply --check .evidence/candidate.diff

If a model or an agent produced the diff on a scratch branch, rebase that branch onto your target commit first, then regenerate with git diff --unified=3. Three lines of context are enough for the mapping and keep hunk spans small.

Step 2: Capture executed lines from the suite you already have

You do not need new tests yet. Run the existing suite under coverage and export machine-readable data.

mkdir -p .evidence
python -m coverage run --source=app -m pytest -q
python -m coverage json -o .evidence/cov-before.json
git diff --unified=3 > .evidence/candidate.diff

If your suite needs a database or a seeded fixture, wrap the whole command in whatever your project already uses for a clean environment. The only requirement is that cov-before.json describes the same commit that .evidence/candidate.diff was generated against.

Step 3: Intersect hunk spans with executed lines

The script below parses old-side hunk ranges out of the unified diff and intersects them with the executed_lines arrays in the coverage JSON. Lines that fall inside a hunk but never appear in executed_lines are reported as orphans, and any orphan fails the gate.

# tools/witness_map.py
import json, re, sys
from pathlib import Path

HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")

def old_side_spans(diff_text):
    spans, path = {}, None
    for line in diff_text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            spans.setdefault(path, [])
        elif line.startswith(("--- ", "diff ", "index ")):
            continue
        m = HUNK.match(line)
        if m and path:
            start, count = int(m.group(1)), int(m.group(2) or 1)
            spans[path].append((start, start + count - 1))
    return spans

def executed_lines(cov_path):
    data = json.loads(Path(cov_path).read_text())
    return {f: set(v["executed_lines"]) for f, v in data["files"].items()}

def main(diff_path, cov_path):
    spans = old_side_spans(Path(diff_path).read_text())
    covered = executed_lines(cov_path)
    orphans_total = 0
    for path in sorted(spans):
        hit = covered.get(path, set())
        for start, end in spans[path]:
            touched = set(range(start, end + 1))
            witness = touched & hit
            orphans = sorted(touched - hit)
            pct = 100 * len(witness) / len(touched)
            print(f"{path}:{start}-{end} {len(witness)}/{len(touched)} ({pct:.0f}%)")
            if orphans:
                orphans_total += len(orphans)
                print("  orphans:", ",".join(map(str, orphans)))
    print(f"\norphan lines in planned spans: {orphans_total}")
    return 1 if orphans_total else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Run it and treat exit code 1 as a stop sign, not a warning:

python tools/witness_map.py .evidence/candidate.diff .evidence/cov-before.json

Sample output from a small demonstration repository, included only to show the format (numbers are illustrative, not a benchmark):

app/billing/invoice.py:118-146 4/29 (14%)
  orphans: 121,122,123,127,131,133,134,135,136,138,141,142,143,144,145,146
app/billing/tax.py:12-40 28/28 (100%)

orphan lines in planned spans: 16

That output changes the plan immediately. The tax module is cleared for extraction because every line in the span already has a witness. The invoice span is not, and no amount of careful renaming fixes the missing evidence.

Step 4: Turn orphans into characterization tests, one rung at a time

Each orphan is a small assignment, and the smallest safe change means resolving them before editing rather than after. Work in this order:

1. Read the orphan line and write a test that executes it through the public entry point, not by importing a private helper.
2. Assert the current output verbatim, including awkward formatting, so a later behavior change fails loudly.
3. Re-run the harness; when a span reaches 100%, you may edit only the lines inside that span.
4. Commit the test and the refactor separately, so a revert of the change never removes the evidence.

Some orphan lines are genuinely unreachable: dead branches, defensive guards for states the parser cannot produce, or code behind a feature flag that no longer ships. Delete those in their own commit and record the reason in the commit message. A witness map that stays red because of dead code stops being trusted within a week.

Step 5: Detect witness drift after the change

The gate does not end at the diff. After applying the refactor, re-run coverage and compare executed lines against the baseline for the touched files. A line that used to run and now does not is witness drift, and drift usually means a condition or a call path silently disappeared.

# tools/witness_drift.py  ->  python tools/witness_drift.py before.json after.json app/billing/invoice.py
import json, sys
from pathlib import Path

def executed(path):
    data = json.loads(Path(path).read_text())
    return {f: set(v["executed_lines"]) for f, v in data["files"].items()}

before, after = executed(sys.argv[1]), executed(sys.argv[2])
failed = False
for p in sys.argv[3:]:
    lost = sorted(before.get(p, set()) - after.get(p, set()))
    if lost:
        failed = True
        print(f"witness drift {p}: {lost[:12]}{' ...' if len(lost) > 12 else ''}")
sys.exit(1 if failed else 0)

Renames and extractions shift line numbers, so expect a small amount of noise on the first run. Filter it by comparing the set of executed line contents rather than line numbers if your diff is mostly moves. The signal you care about is a behavior that stopped being reached, not a line that slid upward by four positions.

Decision table for a span under review

Where a free model and a free server fit

The witness map is deliberately boring local tooling, but two costs around it are real: producing candidate diffs, and running full-suite coverage captures. MonkeyCode, an open-source coding assistant, offers free model access and a free server option that push both costs down — I generate the first batch of candidate diffs with it, then run the coverage capture on the server option when a full suite takes longer than a coffee break. Terms, quotas, and limits change, so confirm them in the project's own documentation before you plan around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important boundary is that the gate stays local and offline. A model can propose the diff, but only your repository's coverage data can witness it, and no external service gets to overrule a red span.

Limitations and who should skip this

• Line coverage is not behavior coverage. Async continuations, threads, subprocess calls, and template rendering can execute without attributing lines to your source files, producing false orphans.
• Moves break the mapping. A wholesale file move or a large extract-method produces spans too wide to reason about; split the change into smaller rungs first.
• No harness means no gate. If the project has no runnable test entry point, build a smoke test before anything else; the witness map cannot substitute for the first test.
• Execution is not correctness. Pair the map with an occasional deliberate fault injection so you know your witnesses can actually fail.
• Skip this approach if your refactor is really a rewrite, if type checking already covers the surface you are changing, or if the team will not maintain the harness when it turns red.

A short checklist to run before the next cleanup diff

1. Regenerate the candidate diff against the exact commit you will modify.
2. Export executed lines from the unmodified suite.
3. Refuse any span with an orphan unless the line is proven dead.
4. Change only witnessed lines, in one rung per commit.
5. Re-measure after the change and revert on drift.

The suite going green is the start of the conversation, not the end of it. A witness map turns that vague green into per-line evidence, which is the only thing that makes the smallest safe change actually small and actually safe. If you try it on a messy module, generating a few candidate diffs through the free model access is a cheap way to see how fast orphan lines show up.