An agent was handed a crash on a checkout request. The real payload had a nested coupon block: a tiered discount, a bound expiry, a currency override, and an empty surcharge array. It stripped the request down to find the cause, and after a few rounds of trimming it had a three-field sample that ran clean. It called the bug fixed and closed the loop. The original payload still crashed.
What happened between the two requests was not a fix. It was minimization that went too far: somewhere in trimming noise, the agent removed the exact combination of fields that put the code on the failing branch. The toy case passes because it no longer touches that branch at all, and a passing case that never reaches the bug is not evidence the bug is gone.
Minimization removes shape, and the shape can be the bug
Debugging by reduction is a good instinct: drop a field, re-run, keep the failure, repeat. You want the smallest input that still reproduces, so you can reason about it without fifty unrelated fields fighting your attention. The failure mode appears when the reduction stops removing noise and starts removing cause.
The trigger lives in a combination, not one field. A crash that needs a non-empty surcharge array and an expired-tier coupon and a currency mismatch looks like three optional details. Remove any one and the code takes the happy path. The agent keeps removing fields until the response is clean and never notices it deleted the conjunction.
Optional fields read like safe to drop. An empty array, a null override, an unused-looking block all look like clutter to a reducer that has no model of which branch each field feeds. It treats them as removable rather than diagnostic.
A passing toy case is easy to mistake for a resolved bug. The loop's reward is green output; a clean run on the minimized sample reads as success, even though the original failure was never replayed.
The agent keeps the sample it built, not the artifact it received. Once it substitutes its reconstructed input for the original, the loop has no way to compare them, and the drifted input becomes the new definition of the bug.
What it usually looks like in the diff
A test built from the minimized sample. The added regression case uses a tidy two or three-field payload that passes, while the failing production payload never appears in the test at all.
The fix guards a single field. The change patches the one detail the sample still contains, leaving the conjunction that triggered the real crash untouched.
Logs point at a field that was present only in the original. You re-run the production request, the traceback names a block the agent never mentioned, and the added test doesn't cover it.
The report says it could not reproduce. When the minimized sample happens to pass on the first try, the agent often concludes the bug is environmental or already fixed instead of questioning its own reconstruction.
The gate that stops it: lock the original, minimize additively
Keep the original failing artifact untouched in the loop. The request body, the recorded stack, the exact input from the report stay pinned as the baseline. A minimized sample is allowed to be written for clarity, but it is never swapped in as the thing under test.
Replay the original, not the sample, before calling done. The last check must be the exact input that crashed. If that still fails, the bug is open regardless of how clean the tiny sample runs.
Treat reduction as narrowing, not replacing. The point of minimizing is to understand the smallest path to the branch, not to discard the branch. When a removed field changes the outcome, that field was part of the trigger and belongs in the regression case.
Write the regression around the conjunction. If the crash needs two or three fields together, the test should carry that combination. A test with one isolated field proves a guard on that field, not the interaction that failed.
Ask what was removed, not just what passes. Before accepting the fix, list the fields or conditions dropped during minimization and check each one against the original stack. The trigger is usually among them.
The same drift on a recorded API request
This is not unique to unit tests. An agent debugging an endpoint reconstructs a sample request from memory, replays that, declares the response correct, and never sends the recorded payload that actually failed. The reconstructed sample matches the happy path, so the check looks green while the real request shape drifts. The reliable habit is to keep the captured request as the source of truth and compare it against the contract rather than against a hand-built approximation. That source-of-truth replay is the loop we built into Powerduck: the recorded request and the spec outrank any reconstructed sample, and a clean approximation never counts as proof.
Before you accept 'fixed'
- Did the agent replay the exact original failing input, or only a sample it reconstructed?
- Which fields or conditions were dropped during minimization, and does the stack name one of them?
- Does the regression test carry the full combination that triggered the crash, or just one isolated field?
- Does the fix guard the original branch, or only the branch the minimized sample happens to hit?
Minimization finds the path to the bug. It does not remove the bug, and the moment a passing sample replaces the original failing artifact, the loop is validating a problem that no longer exists.
