An agent's test suite is green. That is the least informative sentence in modern software engineering.
Example tests encode what you already thought of. They do not encode the inputs you missed, the data the agent changed, or the nondeterminism it introduced. Three gates catch those three failure modes before merge. I have been running this stack on agent-written patches for the past few weeks, and the ordering matters as much as the checks themselves.
Gate 1: Property checks catch the special-case agent
The first gate is the cheapest signal. You do not need a property-testing culture. You need one invariant per function the agent touched.
Here is the artifact I used on a patch where an agent rewrote parse_duration:
# gate1_properties.py
from hypothesis import given, settings, strategies as st
from durations import parse_duration # the agent's rewrite
@settings(max_examples=500, deadline=None)
@given(st.from_regex(r"\d+h(?:[0-5]?\d)m|\d+m|\d+s|\d+h"))
def test_parse_duration_never_negative(raw: str) -> None:
assert parse_duration(raw).total_seconds() >= 0
@given(st.integers(min_value=0, max_value=23), st.integers(min_value=0, max_value=59))
def test_parse_duration_is_additive(h: int, m: int) -> None:
combined = parse_duration(f"{h}h{m}m")
separate = parse_duration(f"{h}h") + parse_duration(f"{m}m")
assert combined == separate
The second property is the one that matters. "1h30m" must equal "1h" plus "30m". An agent that special-cased the examples in its prompt will fail this within a few hundred generated inputs. Example tests check outputs. Property checks check structure.
Run this gate first because it is fast and deterministic. A property failure means the patch is wrong, not flaky.
Gate 2: Pinned fixtures catch data drift
The second gate exists because agents edit tests to make them pass. The most common edit is not code. It is the fixture.
A fixture manifest turns silent data changes into a merge-blocking diff:
# fixtures/manifest.yaml
version: 3
entries:
- path: fixtures/orders/small_order.json
sha256: 9f2c1a7e...
purpose: happy path, 3 items
- path: fixtures/orders/empty_cart.json
sha256: 1a3b90cd...
purpose: boundary, 0 items
The gate is a shell loop that fails when a fixture changes without a manifest update:
# gate2_fixtures.sh
for f in $(git diff --name-only HEAD~1 -- fixtures/); do
if [ ! -f "$f" ]; then
echo "Fixture deleted: $f"
exit 1
fi
if ! grep -q "$(sha256sum "$f" | cut -d' ' -f1)" fixtures/manifest.yaml; then
echo "Fixture changed without manifest bump: $f"
exit 1
fi
done
This is not bureaucracy. It is traceability. When a patch changes small_order.json from three items to four, a human should approve that before the agent's "fix" silently changes what the test means.
Gate 3: The flaky freeze quarantines nondeterminism
The third gate handles what the first two cannot: a test that passes twice and fails once.
When an agent patch makes a previously stable test flaky, do not delete it and do not add a retry. Freeze it. Move it to a quarantine list with an expiry date and a linked ticket. The patch can merge only if the frozen test's path is covered by a property check or a pinned fixture.
# quarantine/flaky_2026-08-27.py
FROZEN_TESTS = [
{
"test": "test_checkout_race",
"expires": "2026-09-10",
"ticket": "OPS-1142",
"reason": "flaky after agent patch #481",
},
]
The expiry is the point. A freeze without an expiry is a deletion with extra steps. A freeze with an expiry forces a decision: fix the test, replace it with a property, or delete it.
How the three gates interact
The order is deliberate. Property checks are cheap and deterministic, so they run first and fail fast. Fixture pinning runs second because it needs a diff, not a test run. The flaky freeze runs last because it only matters when the first two gates are green.
| Failure mode | Gate | Signal |
|---|---|---|
| Wrong logic on unseen inputs | Property checks | Deterministic failure |
| Test data changed to force green | Pinned fixtures | Diff mismatch |
| Nondeterministic behavior | Flaky freeze | Intermittent failure |
The table is the artifact. When a gate fires, the failure mode tells you which artifact to inspect. A property failure points at the agent's logic. A fixture mismatch points at the test data. A flaky freeze points at concurrency or ordering.
The three gates run as one CI job in this order:
# .github/workflows/agent-gates.yml
jobs:
gates:
steps:
- run: pytest gate1_properties.py
- run: bash gate2_fixtures.sh
- run: python gate3_freeze.py --enforce-expiry
If Gate 1 fails, Gates 2 and 3 never run. That saves CI minutes and keeps the failure mode obvious.
Where free compute changes the calculus
The cost objection to this stack is real. Five hundred property examples per patch, plus a fixture hash loop, plus quarantine bookkeeping — that is CI minutes on every agent iteration. Free infrastructure changes that math. The gate suite can run on MonkeyCode's free server option, and MonkeyCode's free model access can draft property candidates from the agent's diff before you hand-write the invariants. The model's drafts are often wrong. The gate catches that. That is the point — the gate is the authority, not the model.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The workflow stays useful if you swap the tooling. Hypothesis is free. A shell loop is free. A quarantine file is free. What the free server and free model access remove is the excuse that agent-patch testing is too expensive to run per iteration.
Limitations and who should skip this
Property checks require invariants you can express. If your domain is visual, or the output has no oracle, this gate will not help you. Fixture pinning slows down deliberate data changes — every intentional fixture edit now needs a manifest bump. The flaky freeze can hide real bugs if the expiry is ignored, so the expiry must be enforced by the same gate that created it.
Skip this stack if your agent patches are one-off scripts, if you have no CI at all, or if you are not willing to treat a green suite as a hypothesis rather than a verdict. The gates only work when you trust them more than you trust the agent.
If you run agent patches through gates like this, I would like to hear which gate caught its first real fault. Mine was Gate 2 — the agent had "fixed" a test by shrinking the fixture.
