The Diff Budget: Four Checks Before Merging an AI-Generated Refactor

A messy repo does not reward courage. It rewards measurement.

The smallest safe change is a number, not a feeling. If you cannot define it, you cannot delegate it. A free coding model can produce a 900-line diff for a 20-line refactor. That diff is not a refactor. It is a hostage situation.

This article defines a diff budget: four checks that turn "small and safe" into a script. Write characterization tests first. Slice the refactor into one transformation. Delegate that slice. Verify with the budget. Reject anything that fails.

Why characterization tests come first

Most AI refactor failures are not logic errors. They are contract errors. The model changes behavior because nobody told it which behavior is sacred.

Characterization tests are that contract. They capture current behavior, including the parts you secretly want to fix. In a messy repo, current behavior is the spec. The suite runs before the change and after the change. Both runs must produce a byte-identical transcript.

Do not describe the function in the prompt. That is opinion. Give the model the suite instead. The suite is the only part of this loop you fully control.

The diff budget: four checks

Every delegated step must pass all four checks.

1. Lock. The characterization transcript matches golden.txt byte-for-byte.
2. Scope. The diff touches at most one file.
3. Size. The diff changes at most 50 lines.
4. Tokens. The diff contains no behavior-signal tokens.

A ledger records what changed. A budget rejects what is too big. That difference matters when the diff author is a model.

The gate script

Save this as diff_budget.sh:

#!/usr/bin/env bash
# diff_budget.sh - reject a delegated refactor that is not the smallest safe change.
set -euo pipefail

GOLDEN="${1:-golden.txt}"
MAX_FILES="${MAX_FILES:-1}"
MAX_LINES="${MAX_LINES:-50}"
FORBIDDEN='^[+-](import|from|except|global|class|def)'

echo "== check 1: locked behavior =="
pytest -q >/dev/null 2>&1 || { echo "FAIL  suite failed"; exit 1; }
if diff -q "$GOLDEN" transcript.txt >/dev/null; then
  echo "PASS  transcript matches golden"
else
  echo "FAIL  transcript drifted"
  diff -u "$GOLDEN" transcript.txt | head -30
  exit 1
fi

echo "== check 2: scope =="
touched=$(git diff --name-only | wc -l | tr -d ' ')
if [ "$touched" -le "$MAX_FILES" ]; then
  echo "PASS  $touched file(s) <= $MAX_FILES"
else
  echo "FAIL  $touched file(s) > $MAX_FILES"
  exit 1
fi

echo "== check 3: size =="
churn=$(git diff --numstat | awk '{a+=$1; d+=$2} END {print a+d}')
if [ "$churn" -le "$MAX_LINES" ]; then
  echo "PASS  $churn changed lines <= $MAX_LINES"
else
  echo "FAIL  $churn changed lines > $MAX_LINES"
  exit 1
fi

echo "== check 4: tokens =="
if git diff | grep -nE "$FORBIDDEN"; then
  echo "FAIL  behavior-signal tokens found"
  exit 1
fi
echo "PASS  no behavior-signal tokens"

echo "BUDGET OK"

The workflow:

1. Write the characterization suite. Make it emit transcript.txt with sorted rows and a pinned locale.
2. Run it. Copy the output: cp transcript.txt golden.txt.
3. Slice the refactor. One transformation per prompt.
4. Ask the model for a diff. Give it the suite, not a repo tour.
5. Apply the diff. Run diff_budget.sh.
6. Reject on any FAIL. Fix the prompt. Repeat.

A rejection is information

Consider an illustrative run on a legacy order-total function. The model was asked to extract one discount calculation. It extracted the calculation, renamed three variables, and re-wrapped two joins.

$ MAX_FILES=1 MAX_LINES=50 ./diff_budget.sh
== check 1: locked behavior ==
PASS  transcript matches golden
== check 2: scope ==
FAIL  3 file(s) > 1

The behavior was intact. The scope was not. The patch was correct and unreviewable. The budget rejected it. The next prompt was narrower: "only touch total()."

The rename that leaks into a string is check 1's catch, when coverage exists. The sneakier failure is structural drift. The model adds an import to dodge a typing error. It wraps a call in except to hide a crash. The suite still passes. Check 4 rejects both moves.

Delegate these, not those

The gate proves nothing observable changed. It does not prove the new structure is good. Naming and layout remain a human call.

Where the free tiers fit

The loop is deterministic except for one part: the model. That makes it cheap to run. MonkeyCode's free model access generates the candidate diff. The free server option runs the verification pass in a clean environment, away from a half-configured legacy workspace. Each rejection costs a prompt correction, not a manual audit.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep the prompt small. Give the model the suite and one sentence: "Extract this calculation. Behavior must not change." Do not give it a guided repo tour. The suite is the spec.

Who should not use this

Skip this workflow in four situations.

• The repo cannot run. No executable test, no golden master. Grep-based characterization is theater.
• The behavior must change. If the refactor is secretly a bug fix, the golden master fights you. Write a new expectation test first.
• The change is tiny. A 100-line script does not need a transcript and a gate. A careful diff is faster.
• The reviewer is the only gate. The budget automates the boring checks. It does not replace a human reading the final diff.

The point

A diff budget is not a review tool. It is a delegation tool. It tells a free model, in advance, that scope is not negotiable.

Define your budget before your next refactor. Mine is four checks. Boring is the goal.

If you already gate AI patches with a script, the comments are open. What is in your budget?