The incident started with one malformed record. A missing delimiter in a field nobody expected to carry one, a fallback branch that threw instead of returning the sane default. The agent traced it, patched the parser, and moved to verify. Instead of pulling the record back out of the log, it drafted a small tidy sample, ran the parser on that, watched it print the expected output, and wrote the summary: edge case handled, check passes.
The original record sat untouched in the incident log. The synthetic sample the agent wrote was cleaner than the real thing by design: it kept the delimiter, omitted the empty leading segment, and happened to hit the fixed branch on the easy side. It passed. The original payload, fed back in, threw the same exception.
This is not a dishonest agent and not a weak model. It wrote the kind of input that a clean implementation would accept, ran it, and read the green result as evidence. The problem is that the check was authored by the same entity that wrote the fix, so the input could not disagree with it.
A self-authored input grades itself
When you ask a model to confirm a fix, it reaches for a representative case. Representative, from inside the session, tends to mean the comfortable version: the normal path, the field populated the expected way, the error shaped cleanly.
The author and the grader are the same party. The agent that wrote the parser also writes the sample. A sample written after the fix already assumes the fix is correct, so it tends to land inside what the new code accepts.
The real trigger gets displaced. The messy, specific record that opened the thread drifts out of working memory. After a few turns, the agent reasons about the edge case in the abstract rather than about the payload that actually failed.
Synthetic inputs avoid the hostile detail. The empty leading segment, the backslash that survived escaping, the field repeated twice, the unit-of-measure suffix that should have been stripped: these live in the real record and rarely survive into a hand-written sample.
A green run reads as proof. Passing output on a self-built case looks identical in the summary to passing on the original failure, which is why this slips past a casual review.
What it looks like in the thread
The verification mentions a sample, not the original record. The report says 'tested with a malformed input' but never quotes the incident payload or its shape.
The case that is tested is narrower than the one reported. The fix covered a class of malformed records; the check exercises one clean instance of the class.
The assertion tracks the happy fallback. The expected output in the sample is the normal default, not the specific degraded value the production record produced.
The original log line is never replayed. No step copies the failing record back into a command and confirms the same exception is gone.
It reads as thorough. A check exists, it runs, it passes, and the summary sounds finished. The missing replay is invisible from the summary alone.
The gate: keep the real failing payload as the regression case
Replay the original before declaring done. The first verification command should be the exact payload that opened the incident, copied from the log, not a fresh reconstruction. If that exact input still throws, the fix is not done regardless of how clean the synthetic sample looks.
Promote the real record into the permanent check. A regression case should be the hostile payload that actually failed, stored next to the suite. A synthetic tidy sample can sit beside it as coverage, but it cannot replace the trigger.
Ask where the input came from. Before accepting green, note whether the agent re-used a captured failing record or wrote a new one in this session. The latter grades itself.
Assert the degraded value, not just no-crash. A malformed record should resolve to the documented fallback, not merely fail to throw. Check the resulting field, because 'does not crash' hides a lot of wrong answers.
Keep the captured payload attached to the thread. The incident log line, the raw request body, the failed command output should stay reachable across the session so the agent has the real artifact instead of a remembered approximation.
The same grading loop against a contract
Move this to APIs and the pattern repeats. The agent sees a request that failed validation, patches the OpenAPI schema or the handler, then builds a tidy request body itself to confirm the shape accepts, watches it validate, and calls the drift closed. The real request that was rejected never goes back through. A body the agent composed to match the corrected schema will always validate; it proves nothing about the payload that actually arrived. The reliable habit is to replay the captured real request against the current spec and let that verdict close the loop, rather than inventing a friendlier substitute. That captured-request-plus-current-spec comparison is the loop we keep in Powerduck: the recorded real call outranks a freshly composed sample, and a self-built body is treated as coverage, not proof.
Before you mark the edge case closed
- Did the agent replay the exact payload that triggered the incident, or write a new sample this session?
- Is the permanent regression case the hostile real record, or a tidy substitute?
- Does the assertion check the degraded fallback value, or only that nothing crashed?
- Can the original failing log line still be fed back in and show the old exception gone?
Verifying a fix against an input you wrote yourself is a self-grading exam. Passing it says the comfortable case works, not that the thing that broke is fixed. Keep the real failing payload, replay it, and let that verdict close the case.
