Last week I watched an agent finish a task with a clean exit 0, a green test suite, and a confident Done. in the chat. Nothing in the system had changed.
The migration script it claimed to have run had actually errored out on the first row, and the agent hadn't checked stderr. The tests it ran were pointed at the old fixture because the working directory was different from what it assumed. The database still held the pre-migration rows. Nobody noticed until the first user hit the endpoint.
This is the failure mode I see most often with agentic coding, and it isn't hallucination. It's worse than that. The agent ran real commands. It just didn't verify that the commands had the effect it described.
Exit 0 is not a success receipt
A process exiting zero only tells you the process itself didn't segfault or throw an uncaught exception. It says nothing about whether the intended side effect landed. A migration script can exit 0 after processing zero rows. A test runner can exit 0 against an empty fixture list. A linter can exit 0 because it found no files to lint, not because it linted cleanly.
Humans used to catch this because we'd scroll the output. Agents don't scroll. They sample the last few lines, see no error string, and move on.
The cheap gate I now require
For any agent task that claims it changed something in the system, I ask for two things in the PR description before I trust it:
-
The actual diff. Not
git diff --stat— the real diff, or a checksum / row count that moved. -
The output that proves the state moved. The migration log line that says
processed 4,218 rows, thebefore/aftercount from aSELECT COUNT(*), the curl response showing the new field present.
If the agent can't paste that back, the task isn't done. It's running on vibes.
This is a remarkably low bar. It doesn't require a formal replay system or an observability stack. It just closes the loop between "I ran the command" and "the thing I was asked to do is now true."
Where this bites hardest
The trap shows up most on tasks that look like they should be easy:
-
Data migrations — script runs, exits 0, zero rows actually updated because of a
WHEREclause that matched nothing. - Config changes — file saved, but the service never restarted, so the live config is still the old one.
- Schema edits — OpenAPI file updated in the repo, but the live endpoint still returns the old shape because no deployment happened.
- Test runs — suite passes, but against a fixture from a previous task that didn't get cleaned up.
Each of these is a story about the agent trusting its own exit code instead of checking the world.
Applied to API work
I kept running into this exact shape on the API contract side: an agent would update an OpenAPI spec, report Done, and the spec would drift from the live endpoint by three or four endpoints. The exit code was fine. The diff was real. But the contract no longer described the server.
So I built the verification loop into Powerduck: after an agent edits the spec, the first thing the tool does is re-run the spec against the live endpoint and show you the mismatch table. No trusting the chat log. No trusting the exit code. The diff has to match reality, or the agent didn't actually finish.
That's the principle, not the product: require the output that proves state moved, and don't accept Done as a substitute.
What this replaces
You don't need a heavier agent harness. You don't need a new framework. You need one sentence in your prompt or your PR template:
"Paste the command output that proves the state changed."
Agents that can't produce that line haven't finished. Agents that can are suddenly much easier to trust, because you're reading evidence instead of taking their word for it.
The exit code was never the receipt. The world changing was.
