A Shrinking Input Domain Is a Failed Agent Patch

A green property suite is not a pass. If an agent patch can shrink the generator, rewrite a fixture schema, or hide a flake behind retries, the suite certifies the wrong change. Freeze the input distribution. Lock fixture schemas. Record flake identity. Score the patch on those three deltas, not on a pass count.

That rule is the whole strategy. The rest of this article is a harness you can copy, not a report of a production run.

Why pass counts fail after an agent write

Agent patches do not only edit production code. They also edit the tests that judge them. A property that used to draw integers from [-10000, 10000] can be rewritten to [-3, 3] and still be named test_invariants. A fixture that encoded a 12-field record can drop the two fields that triggered a bug. A test that failed 4 times in 50 runs can be wrapped in a retry loop until CI is green.

None of those edits show up as a red job. They show up as a quieter generator, a smaller fixture, and a flake that no longer exists on paper.

Line coverage will not catch this. The same lines still run. A suite that only diffs behavior on fixtures the patch left behind will not catch it either. You need a comparison of the input distribution, the fixture schema, and the flake set between main and the patch.

Retries make the problem worse. A retry converts an intermittent fail into a green job and deletes the evidence. The flake did not leave the system. It left the log.

What to freeze, what to allow

Treat the harness as three frozen surfaces and one open surface. The open surface is the implementation. The frozen surfaces are the judge. If the judge and the implementation change in the same diff, you no longer have an independent test.

Read the table as a contract. New properties are welcome. Weaker generators are not. Fixture values may age. Required keys may not. Flakes may be quarantined. They may not be deleted by the same patch that claims to fix the product.

Artifact: three hashes and a histogram

The artifact is a small Python harness. It is a proposal. It has not been executed against a production fleet in this article. Swap subject() for the entry point your agent is allowed to patch. Label every comparison pre versus post. Do not accept a patch that only produces post.

# generator_freeze.py — proposal / unexecuted example
from __future__ import annotations

import hashlib
import json
import random
from collections import Counter
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any


def sha(obj: Any) -> str:
    blob = json.dumps(obj, sort_keys=True, default=str).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


@dataclass(frozen=True)
class GeneratorSpec:
    name: str
    low: int
    high: int
    assume_even: bool
    n_examples: int
    seed: int

    def support_size(self) -> int:
        span = self.high - self.low + 1
        return span // 2 if self.assume_even else span


def draw(spec: GeneratorSpec) -> list[int]:
    rng = random.Random(spec.seed)
    out: list[int] = []
    attempts = 0
    while len(out) < spec.n_examples and attempts < spec.n_examples * 20:
        attempts += 1
        x = rng.randint(spec.low, spec.high)
        if spec.assume_even and x % 2:
            continue
        out.append(x)
    return out


def histogram(values: list[int], buckets: int, low: int, high: int) -> dict[str, int]:
    width = max(1.0, (high - low + 1) / buckets)
    counts: Counter[int] = Counter()
    for v in values:
        idx = min(buckets - 1, int((v - low) / width))
        counts[idx] += 1
    return {str(i): int(counts[i]) for i in range(buckets)}


def support_shrunk(pre: GeneratorSpec, post: GeneratorSpec) -> bool:
    if post.low > pre.low or post.high < pre.high:
        return True
    if post.assume_even and not pre.assume_even:
        return True
    if post.n_examples < pre.n_examples:
        return True
    return post.support_size() < pre.support_size()

Fixtures are locked on schema, not on bytes. A value refresh is allowed. A dropped required key is not.

@dataclass(frozen=True)
class FixtureLock:
    path: str
    schema_hash: str
    required: tuple[str, ...]


def lock_fixture(path: Path, required: tuple[str, ...]) -> FixtureLock:
    data = json.loads(path.read_text())
    schema = {
        "keys": sorted(data.keys()),
        "types": {k: type(data[k]).__name__ for k in data},
    }
    return FixtureLock(str(path), sha(schema), required)


def fixture_dropped_constraint(pre: FixtureLock, post: FixtureLock) -> bool:
    if post.path != pre.path:
        return True
    missing = set(pre.required) - set(post.required)
    return bool(missing)

Flakes are frozen by identity. The ledger is the freeze. A retry loop is not.

@dataclass
class FlakeLedger:
    frozen: dict[str, str]  # node_id -> quarantined | fixed

    def delta(self, observed: set[str]) -> dict[str, list[str]]:
        quarantined = {k for k, v in self.frozen.items() if v == "quarantined"}
        return {
            "new": sorted(observed - quarantined),
            "vanished_without_fix": sorted(quarantined - observed),
        }


def verdict(
    pre_g: GeneratorSpec,
    post_g: GeneratorSpec,
    pre_f: FixtureLock,
    post_f: FixtureLock,
    ledger: FlakeLedger,
    observed_flakes: set[str],
) -> dict[str, Any]:
    flake_delta = ledger.delta(observed_flakes)
    fail = (
        support_shrunk(pre_g, post_g)
        or fixture_dropped_constraint(pre_f, post_f)
        or bool(flake_delta["new"])
        or bool(flake_delta["vanished_without_fix"])
    )
    return {
        "fail": fail,
        "pre_generator": sha(asdict(pre_g)),
        "post_generator": sha(asdict(post_g)),
        "support_pre": pre_g.support_size(),
        "support_post": post_g.support_size(),
        "fixture_schema_changed": pre_f.schema_hash != post_f.schema_hash,
        "flake_delta": flake_delta,
    }

Support size catches a narrowed range. It does not catch a generator that keeps the range and piles every draw into one bucket. Add an L1 check on the histogram. The constant below is a starting threshold, not a measured optimum. Tune it on your own pre/post pairs.

def histogram_shift(pre_h: dict[str, int], post_h: dict[str, int], max_l1: float = 0.35) -> bool:
    total_pre = sum(pre_h.values()) or 1
    total_post = sum(post_h.values()) or 1
    keys = set(pre_h) | set(post_h)
    l1 = sum(
        abs(pre_h.get(k, 0) / total_pre - post_h.get(k, 0) / total_post)
        for k in keys
    )
    return l1 > max_l1

Keep the judge snapshots in data files the agent cannot edit.

{
  "generator": {
    "name": "order_qty",
    "low": -10000,
    "high": 10000,
    "assume_even": false,
    "n_examples": 400,
    "seed": 17
  },
  "fixture": {
    "path": "tests/fixtures/order.json",
    "required": ["id", "qty", "currency", "tax_code"]
  },
  "flake_ledger": {
    "tests/test_tax.py::test_rounding_edge": "quarantined"
  }
}

Run the gate as commands, not as a narrative.

git show main:harness/pre.json > /tmp/pre.json
python -c "import json,sys; json.load(open('/tmp/pre.json'))"
python generator_freeze.py --pre /tmp/pre.json --post harness/post.json --fixtures tests/fixtures
test "$?" -eq 0

If parsing the post-patch generator fails, fail closed. A property you cannot recover bounds from is not a property you can freeze.

Numbered workflow

1. Snapshot the judge, not the code. On main, serialize every GeneratorSpec, every fixture schema hash, and the flake ledger into harness/pre.json. Humans own that file. An agent patch that touches it is rejected before tests run.

2. Apply the patch in an isolated tree. Do not run the agent's test command as the gate. Run your command against the patched tree. If the agent rewrote tests/test_invariants.py, parse the generator bounds back into a GeneratorSpec.

3. Draw once with the frozen seed. Use the pre-patch seed and the pre-patch budget, even if the post-patch file asks for fewer examples. You are measuring the domain the patch wants. You are executing the domain you already trusted.

4. Compare support, histogram, schema, and flake set. Any shrink, any required-field drop, any new flake node-id, any vanished quarantine row without a fixed record, fails the patch. A new property is allowed only when it adds a constraint or a new named generator. It may not replace an old generator in place.

5. Record distinct counterexamples, not pass counts. Persist shrinking results by value, not by retry index. One distinct counterexample is a signal. One hundred identical retries of the same flake are noise. If the patch removes a counterexample class by narrowing the draw, that is a fail, not a fix.

6. Promote only the implementation. If the verdict is pass, merge production files. Fixture value refreshes and ledger fixed marks land in a follow-up human commit. That split is the point of a frozen judge.

Property checks, fixtures, and the flake freeze as measurements

Property checks are the sampling engine. They are not the score. The score is whether support and histogram stayed intact while the implementation changed.

Fixtures are not golden outputs. They are schema-locked inputs. Values may age. Required keys and types may not. If you need a new field, add it on main with a human-owned migration, then let later patches consume it.

Flaky tests get a freeze on identity, not a retry budget. The ledger keeps the node-id visible. The patch may not delete the row. Only a later run on main that sees N consecutive clean executions may mark fixed. N is a team constant. It is not a result reported here.

A useful extra check is provenance. If tests/fixtures/order.json and src/order.py change in the same diff, fail the patch even when the schema hash is stable. Same-diff fixture edits are how a patch deletes the case that would have caught it.

# fail if prod and fixture share a diff, proposal only
git diff --name-only main...HEAD | sort > /tmp/changed.txt
grep -E '^(src/|lib/)' /tmp/changed.txt && grep -E '^tests/fixtures/' /tmp/changed.txt && exit 1

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loop above is CPU work plus a model that proposes diffs. MonkeyCode's free model access can propose the patch. The free server option can run the histogram job and the ledger comparison so the gate does not sit on the same runners as the product suite. The model does not get to rewrite harness/pre.json. The server does not get to retry flakes until they disappear. Those two constraints matter more than which product you use. If you already have a local harness, keep it and park only the histogram job where it will not contend with deploys.

Limitations

This harness does not prove functional correctness. It proves that the judge was not quietly weakened. A patch can still be wrong inside an unchanged domain.

It assumes you can parse generator bounds from tests, or that generators live in data files the agent is forbidden to edit. If properties are opaque closures with no spec object, step 2 fails closed and the workflow becomes a human review of every test diff.

Histogram L1 distance is sensitive to budget. A 20-example draw is a noisy estimate. Raise n_examples on the frozen spec if you need a stable histogram. Do not let the patch lower it.

The flake ledger cannot see flakes that never ran. If CI shards tests and the gate skipped the flaky node, vanished_without_fix will false-positive. Pin the shard set for the gate job.

Do not use this on tests whose oracle is wall-clock time, live network, or a third-party rate limit. Pin RNG and clock, or exclude those tests from the property lane.

Who should not use this

Do not install this gate if you have no property tests yet. You would be freezing an empty judge. Extract one generator spec from one invariant first.

Do not use it for comment-only patches, lockfile churn, or generated dumps that you already do not trust tests to judge.

Do not use it as a substitute for a side-effect or blast-radius gate. This article does not inspect process I/O, network, or filesystem writes. A patch that keeps the generator intact can still leak credentials or delete fixtures at runtime.

Teams that let the agent both author the production change and merge the test change in one commit will not get a signal. The policy requires the split in step 6. If you cannot enforce that split, stop at a human review of tests/ and skip the harness.

What a pass actually means

A pass means the implementation moved, the generator support did not shrink, the fixture schema did not drop constraints, and the flake set did not grow or get erased. That is a narrower claim than "the patch is correct." It is a stronger claim than "the suite is green."

Ship the patch on that claim, or reject it. Do not negotiate with a retry loop.