Your CI pipeline just turned green, and the AI-generated patch looks clean. Why does that combination still make me nervous?
Because green CI proves the code compiles and the tests you remembered to write still pass, and it says nothing about the behavior you fixed three months ago and forgot to protect. I have watched a model-generated patch sail through review, pass every unit test, and quietly reintroduce a bug the team had already fixed twice. The fix is not more tests. The fix is a receipt gate: a small, repeatable check that compiles the candidate, runs a fixed set of behavior receipts, and reports per-case results.
Receipts are not test cases. They are promises the code must keep.
Step 1: Write behavior receipts, not test cases
A test case documents what you think the code should do. A receipt documents what the code once did and must keep doing. The difference is the intent, and the intent changes what you store.
Each receipt needs an id, an input, an expected output, and a tolerance. The id is the part that saves you later, because a score without per-case ids is just a number. When the total stays the same but one receipt flips from PASS to FAIL, the id tells you which promise broke.
[
{
"id": "normalize-whitespace",
"input": " hello world ",
"expected": "hello world",
"tolerance": 0
},
{
"id": "normalize-utf8",
"input": "caf\u00e9",
"expected": "caf\u00e9",
"tolerance": 0
},
{
"id": "normalize-empty",
"input": "",
"expected": "",
"tolerance": 0
}
]
Three receipts beat thirty that all exercise the same path. The UTF-8 case is the one that catches the regression. The empty case is the one nobody writes until production finds it.
Step 2: Compile the candidate, run the receipts
The gate is deliberately boring. It compiles candidate.cpp, runs it against each receipt, and compares stdout with the expected value. The intelligence lives in the receipts and the grader, not in the runner.
import json
import subprocess
import sys
def build():
return subprocess.run(
["g++", "-std=c++17", "-O2", "candidate.cpp", "-o", "candidate"],
capture_output=True,
text=True,
)
def run_receipt(receipt):
result = subprocess.run(
["./candidate"],
input=receipt["input"],
capture_output=True,
text=True,
timeout=5,
)
return result.stdout.strip()
def grade(receipt, actual):
expected = receipt["expected"]
tolerance = receipt.get("tolerance", 0)
if tolerance == 0:
return actual == expected
try:
return abs(float(actual) - float(expected)) <= tolerance
except ValueError:
return False
def main():
build_result = build()
if build_result.returncode != 0:
print("BUILD FAILED")
print(build_result.stderr)
sys.exit(1)
with open("receipts.json") as fh:
receipts = json.load(fh)
passed = 0
for receipt in receipts:
actual = run_receipt(receipt)
ok = grade(receipt, actual)
passed += ok
print(f"[{'PASS' if ok else 'FAIL'}] {receipt['id']}")
if not ok:
print(f" expected: {receipt['expected']!r}")
print(f" actual: {actual!r}")
print(f"{passed}/{len(receipts)} receipts kept")
sys.exit(0 if passed == len(receipts) else 1)
if __name__ == "__main__":
main()
The runner is short because it should be run, not maintained. A gate that needs infrastructure will be skipped. A gate that fits in one file will be run on every patch.
Step 3: Route the whole loop through a free tier
Now the workflow gets practical. The model generates a candidate patch, and the gate verifies it. Both halves cost something: model calls consume tokens, and a build machine costs time to provision.
MonkeyCode's free models can generate the candidate patch, and its free server can run the gate without you spinning up a VM. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
That combination removes the two excuses that kill eval loops: "I don't want to pay for model calls" and "I don't have a box to run this on." The gate stays the same whether it runs on a laptop, a CI runner, or a free server. The artifact is the receipts, not the hosting.
Treat the free tier as a claim to verify, not a permanent fact. Free models and free servers change quotas and availability without warning, so pin the versions you rely on and re-check the policy before you build a workflow around it.
Grader selection: a decision table
The grader is a contract, and each contract has a failure mode. I use this table when I set up a gate for a new function.
| Grader | Use when | Failure mode |
|---|---|---|
| exact | deterministic output like normalized strings | brittle to harmless formatting changes |
| numeric tolerance | measurements, timings, scores | hides slow drift toward the boundary |
| regex | logs and error messages | narrow; misses semantic changes |
| LLM rubric | subjective summaries, release notes | noisy; needs a versioned prompt |
Exact match is the default because it is cheap and impossible to argue with. The other graders exist for cases where exact match is wrong, not for cases where it is inconvenient. An LLM rubric can grade things with no single correct form, but a noisy grader turns a regression detector into a coin flip.
Where this gate breaks
The gate is only as good as its receipts. Stale receipts rot: a promise that no longer matches the intended behavior fails forever, and teams start ignoring red rows. Review receipts the same way you review code, because they are code.
The gate does not prove correctness. It proves that a fixed set of behaviors survived. Fuzzing, sanitizers, and formal verification cover different ground, and I have written about sanitizer harnesses and race detection separately. A receipt gate complements those tools; it does not replace them.
Who should not use this: teams with no existing test culture, because the gate becomes another ignored report. Teams with a spec that changes weekly, because the receipts chase a moving target. And teams that need proof rather than signals, because receipts are evidence, not proof.
The next patch you review
Your next AI-generated PR deserves a receipt, not just a green checkmark. Pick a function that has burned your team before, write five receipts, and run the gate before you approve. The first regression it catches will pay for the whole setup.
