Last week I watched an agent close a ticket labeled "fix login timeout." It had rewritten the retry loop, added backoff, updated the unit tests, and green-lit the branch. The PR description was confident. The CI was green.
I clicked "run" on the local dev server, opened the login page, typed a fake email, and waited. The spinner went forever. Same bug. The unit tests were green because they only exercised the retry function in isolation — they never actually booted the HTTP server, never hit the real route, never rendered the page.
This is a structural failure mode, not a one-off.
Why the agent stops at the unit test
An agent works inside a feedback loop you set up. If the loop is "run npm test," it will optimize for npm test going green. It will not, unprompted, also:
- Boot the dev server
- Wait for the port to come up
- Open a headless browser
- Click the login button
- Wait for a response
- Tell you the actual status code
None of that is in the test command. So it isn't in the agent's world.
The unit tests are green because the agent wrote them against the unit it just refactored. The integration boundary — where the retry loop meets the real HTTP client meets the real auth endpoint meets the real frontend — is a no-man's-land the agent never has to cross.
The cheap gate
You don't need a full end-to-end suite to close this gap. You need one observable thing that proves the app actually started and the route actually responded.
The cheapest version I've landed on:
- After the agent says "done," ask it to boot the app in the background.
- Curl the route it just touched. Print the status code and the first line of the response body.
- If the boot fails or the curl times out, the task is not done — no matter what the unit tests say.
That's it. Three lines of shell. It catches the class of bug where the unit test passes and the route 500s because of a missing env var, a bad import, or a port-binding typo the unit tests never see.
When the work touches API behavior specifically, the gate gets narrower: re-run the spec against the live endpoint and look for a mismatch between what the OpenAPI says and what the server returns. We built that loop directly into Powerduck because I kept doing it by hand — boot the server, hit each documented path, diff the response against the spec, and only call the PR done when the two agree.
The trap of trusting the green tick
The deeper mistake is reading "tests passed" as "the thing works." They are two different claims:
- Tests passed: the code the agent wrote behaves the way the tests expect.
- The thing works: a user can actually do the thing the ticket asked for.
The second claim requires the app to be running. The first doesn't.
Agents will keep delivering the first claim until you require the second. Until then, green CI is a report about the unit test file, not about the product.
What to change in the prompt
The single prompt edit that moved the needle for me: replace "make the tests pass" with "boot the app, exercise the route, paste the output." The first sentence optimizes for the test file. The second optimizes for the running system.
It's a small shift. It moves the agent's definition of "done" from the test runner to the thing you actually care about.
