Review the Contract, Not the Diff: Auditing Agent-Generated PRs Hunk by Hunk

Most agent-authored PRs are not hard to read. They are hard to bound. The code is usually fine line by line; what you cannot see is the scope the agent silently negotiated with itself while writing it. So you spend forty minutes reconstructing intent that was never written down, then approve because the tests are green.

There is a cheaper order of operations. Write the scope down before the agent runs, commit it next to the task, and let a deterministic script tell you which hunks deserve your attention. The script does not judge correctness. It tells you where correctness is even worth arguing about.

The failure mode: silent scope expansion

A typical agent PR says "handle 429 responses without infinite retry." The diff touches:

• src/http/client.py — expected, that is the task.
• src/config/defaults.py — not expected. A default changed from 3 attempts to 1.
• tests/http/test_client.py — six assertions replaced by one assert response is not None.

No single line is obviously wrong. That is the problem. You are being asked to review a decision that was never argued for, mixed into a diff where most hunks are mechanical, and the review UI renders all of it with the same visual weight.

Reviewers then pick one of two bad strategies: read every hunk with equal attention (slow, burns out by the third PR of the day), or skim and trust the tests (fast, and exactly how assertion weakening gets merged). Both are workarounds for missing metadata.

Step 1: commit the scope contract

scope.json lives next to the task description, in the same PR, before any code exists.

{
  "task": "handle 429 from provider without infinite retry",
  "allowed_paths": ["src/http/*", "tests/http/*"],
  "forbidden_paths": ["src/config/*", ".github/*", "pyproject.toml"],
  "allow_delete": false,
  "max_files": 8,
  "max_changed_lines": 250,
  "human_required_if": ["out_of_scope", "deps", "ci", "skip_added", "assertion_weakened"]
}

Three rules make this file useful rather than decorative:

1. Deny wins over allow. src/http/* allowed and src/config/* forbidden means a config edit is a finding, not a judgment call.
2. Budget the size, not just the paths. An agent that needs 40 files for a 5-file task is telling you something about the task, not about its speed.
3. allow_delete: false by default. Deletions are the cheapest way for any diff — human or generated — to make failing tests pass.

Step 2: generate a review packet

The packet script is intentionally boring: stdlib only, reads git, no network, no model calls. Determinism is the point. A classifier that changes its answer between runs cannot gate a merge.

#!/usr/bin/env python3
"""review_packet.py - classify an agent PR diff against scope.json.

Exit 0 = no human-required signal. Exit 1 = human review required.
Globbing is fnmatch-simple; see limitations below.
"""
import argparse, fnmatch, json, re, subprocess, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@ ?(.*)$")
MECHANICAL = re.compile(r"^\s*(?:import\s|from\s+\S+\s+import\s|#|//|/\*|\*|$)")
SKIP_ADDED = re.compile(r"(?:pytest\.mark\.skip|\.skip\(|xit\(|xdescribe\(|t\.Skip\()")
WEAKENED = re.compile(r"(?:assert\s+True|assert\s+[\w.]+\s+is\s+not\s+None|except\s+Exception\s*:\s*pass)")

RISK = {
    "deps":  ["requirements*.txt", "pyproject.toml", "package.json", "package-lock.json", "go.mod"],
    "ci":    [".github/workflows/*", ".gitlab-ci.yml", "Dockerfile*", "Makefile"],
    "db":    ["*/migrations/*", "*.sql"],
    "infra": ["terraform/*", "*.tf", "helm/*"],
}
TESTS = ["tests/*", "*/tests/*", "test_*.py", "*_test.go", "*.test.ts", "*.spec.ts"]


def git(*args):
    return subprocess.run(("git",) + args, capture_output=True, text=True, check=True).stdout


def matches(path, globs):
    return any(fnmatch.fnmatch(path, g) for g in globs)


def files_changed(base):
    out = git("diff", "--numstat", "--no-renames", f"{base}...HEAD")
    rows = []
    for line in out.splitlines():
        add, dele, path = line.split("\t")
        rows.append({"path": path, "add": int(add or 0), "del": int(dele or 0)})
    return rows


def hunks(base, path):
    out = git("diff", "--no-color", "-U0", "--no-renames", f"{base}...HEAD", "--", path)
    start, header, body = None, "", []
    for line in out.splitlines():
        m = HUNK.match(line)
        if m:
            if start is not None:
                yield {"line": start, "header": header, "body": body}
            start, header, body = int(m.group(1)), m.group(3), []
        elif start is not None and line[:1] in "+-" and not line.startswith(("+++", "---")):
            body.append(line)
    if start is not None:
        yield {"line": start, "header": header, "body": body}


def label(path, hunk, contract):
    added = [l[1:] for l in hunk["body"] if l.startswith("+")]
    text = "\n".join(l[1:] for l in hunk["body"])
    if matches(path, contract["forbidden_paths"]):
        return "out_of_scope"
    for kind, globs in RISK.items():
        if matches(path, globs):
            return kind
    if any(SKIP_ADDED.search(l) for l in added):
        return "skip_added"
    if WEAKENED.search(text):
        return "assertion_weakened"
    if not matches(path, contract["allowed_paths"]):
        return "out_of_scope"
    if all(MECHANICAL.match(l) for l in added):
        return "mechanical"
    return "semantic"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--contract", default="scope.json")
    ap.add_argument("--format", choices=["json", "markdown"], default="markdown")
    args = ap.parse_args()

    contract = json.load(open(args.contract))
    rows = files_changed(args.base)
    gate = set(contract["human_required_if"])

    findings, changed = [], sum(r["add"] + r["del"] for r in rows)
    for f in rows:
        for h in hunks(args.base, f["path"]):
            findings.append({"kind": label(f["path"], h, contract), "path": f["path"],
                             "line": h["line"], "header": h["header"]})

    human = [f for f in findings if f["kind"] in gate]
    if len(rows) > contract["max_files"] or changed > contract["max_changed_lines"]:
        human.append({"kind": "over_budget", "path": "-", "line": 0,
                      "header": f"{len(rows)} files, {changed} lines"})

    if args.format == "json":
        print(json.dumps({"task": contract["task"], "human_required": human,
                          "all": findings}, indent=2))
    else:
        print(f"# Review packet - {contract['task']}")
        print(f"changed: {len(rows)} files, {changed} lines\n")
        print(f"## human required ({len(human)})")
        for f in human:
            print(f"- [{f['kind']}] {f['path']}:{f['line']}  {f['header']}")
        print("\n## mechanical, in scope")
        for f in findings:
            if f["kind"] == "mechanical":
                print(f"- {f['path']}:{f['line']}")

    return 1 if human else 0


if __name__ == "__main__":
    sys.exit(main())

Run it against the PR branch, not your working tree:

python scripts/review_packet.py --base origin/main --contract scope.json > packet.md
echo "$?"   # 1 means: a human has to open this, and the packet says where

Output for the 429 example, trimmed:

# Review packet - handle 429 from provider without infinite retry
changed: 4 files, 75 lines

## human required (3)
- [out_of_scope] src/config/defaults.py:12  -retries = 3 +retries = 1
- [assertion_weakened] tests/http/test_client.py:88  def test_retry_budget()
- [skip_added] tests/http/test_client.py:140  def test_gives_up_after_limits()

## mechanical, in scope
- src/http/client.py:3

What the packet lets you skip

mechanical is the only label that earns a fast pass. Import churn, comment-only edits, formatting inside allowed paths. Skim it, run the suite, move on. Everything else gets read.

semantic inside scope is where the real work sits: control flow, error handling, new defaults, changed numeric literals. These hunks need the new test to fail on the base commit before you believe it. There is no packet signal that substitutes for that check.

What the packet forces you to test

The right-hand column is deliberately mechanical. It exists so you do not have to re-derive a policy at 6pm with eleven tabs open.

Where a model actually helps — and where it must not

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The classifier above labels shape: which file, which hunk, which risk class. It cannot say why src/config/defaults.py was touched. That is a job for a language model, and it is a narrow job: draft one sentence per out-of-scope hunk explaining the likely reason, so a human can accept or reject the explanation in seconds instead of re-reading the diff.

That note goes in a notes field. It never touches kind, never flips the exit code, and never suppresses a human required row. Practically: run the deterministic packet first, then call a model with the packet plus the out-of-scope hunks only. You get the cheap tokens spent on the ambiguous 5% instead of on a whole-diff summary nobody reads.

This is where free model access matters for the workflow. A gate that costs money per invocation gets disabled the week someone notices the bill; a gate that runs on a free tier keeps running. The operator states the current offer includes 10M free tokens and a free server option — treat that as a snapshot of a promotional offer, not a permanent SLA, and check the live terms before you make a pipeline depend on it. If you want to host the packet endpoint as an internal webhook instead of a laptop script, the payload is tiny (paths, line counts, hunk headers), which is exactly the shape a small free server can serve.

One hard constraint: do not paste diffs from private repositories into any hosted endpoint without reading its retention terms. Running the model call locally against free access keeps the diff on your machine. That is the configuration I would pick first, and the one I would keep as the default.

Limitations, including cases where this is the wrong tool

• The hunk classifier is syntactic. Whole-file renames, vendored code, generated code, and reformat-only commits will be mislabeled. If your PRs routinely include git mv plus edits, --no-renames will report deletions you did not make.
• Globbing is fnmatch, not gitignore. **/migrations/* does not match nested paths. Replace matches() with a pathspec-based check if your tree is deep.
• It only works if the contract comes first. Retrofitting scope.json onto a diff that already exists turns the gate into rationalization: everything gets declared in scope, and the packet politely agrees.
• In-scope but wrong is invisible. A retry loop inside src/http/client.py is correctly labeled semantic and still loops forever. The packet routes attention; it does not supply judgment.
• Skip this if you have no CI runner that can execute git against the PR base, or your agent already writes directly to a protected branch. In that case the missing control is branch protection, not a review script. And if no one on the team will read a packet, adding one is theater with extra steps.

Monday, in order

1. Write scope.json for the next agent task before the agent starts. Keep it under ten lines.
2. Run the packet against your last five merged agent PRs. Look only at the human required section. When something there was merged without discussion, that is your baseline risk.
3. Add one CI job that fails closed on exit code 1. Never make it advisory on day one; advisory gates decay into comments.
4. Only then add model notes for ambiguous hunks — after the deterministic part has been trusted for a week.

The review time you save is real, but the durable win is different: every merged agent PR now carries a written statement of what it was allowed to touch. The next reviewer does not have to guess, and neither does the audit.

If you want to try this end to end, copy review_packet.py and scope.json first — they work with no account and no network. Add the model notes when the packet has already taught you which hunks are worth a second pair of eyes.