It ran the request and called it done
We handed the agent a bug in a response handler: a field was being dropped for one input shape. It edited the source file, saved, and re-ran the local request against the dev server. The summary read 'fixed and verified locally, returning the field as expected,' so we merged.
The next morning the same failure reproduced on our machine. We edited the same file again, stopped the long-running dev process, started a fresh one, re-ran the request, and the field appeared. The code had been correct on disk since the previous evening; the process that answered the agent's curl still held the old module in memory.
What actually happened
Most local development runs on a long-lived process with a watcher or hot-reload loop. The watcher recompiles files it recognizes as editable sources and leaves the rest alone. A shared module loaded once at boot — a config object, a registry built from imported files, a compiled helper — was not in the reload set. When the agent changed it, the watcher saw a file it does not track, said nothing, and kept serving the object already in memory.
The same trap takes other forms. A Node process caches a required module for the life of the process; editing the file does not evict the cache. A Python interpreter keeps an imported module in sys.modules. A compiled server was never rebuilt, so the running binary predates the edit. A database migration was already applied in a reused local database, which masks a fresh-install path. In every case the disk changed; the running truth did not.
The agent re-ran the one action it knew — curl, a request script, the happy path — and read back a response. It had no reason to doubt the process was fresh, so it compared the new request to its expectation and saw a coherent result. The summary said 'verified' because a request succeeded, not because the new code executed.
Why the agent stops before this check
Verification means 're-run the happy command.' The agent's model of testing is sending a request and reading a sensible response. It rarely models the lifecycle of the process that answers, because that state lives outside the edited file and the task prompt never mentioned it.
A reused server feels like an optimization, not a risk. Restarting takes time, so a fresh session reaches for the already-running dev process. To the agent, that looks like efficiency; it never sees the cache layer underneath.
Hot reload gives false confidence. When the watcher reloads entry files correctly, the agent learns that editing and re-running works, and generalizes that to every file. A silent miss on one boot-loaded module looks identical to a successful reload.
Checks that prove the new code actually ran
Restart to a clean process before verifying a behavior change. Stop the long-lived server and start it fresh in the session that claims done. A reused process is the default source of false verification; forcing a clean boot removes the most common stale state.
Stamp a build or revision hash on every response. Have the health route or response header report the current build hash or module load time. Compare that value before and after the edit; if it does not change, the process never loaded the new code regardless of what the response body looks like.
Drop a temporary canary in the changed path. Add a distinctive log line or a temporary marker in the exact block you changed, run the request, and confirm the canary appears in this process's output. Remove it before merge. A canary is hard to fake: if the line never prints, that code never ran.
Compare before and after on the same route. Run the request against the fresh process and confirm the specific changed field or branch, not only that the endpoint returned 200. A 200 against old behavior and a 200 against new behavior look alike on the surface.
Watch what the reload actually tracks. Confirm the watcher or hot-reload config covers shared modules, config files, and compiled sources, or stop relying on it for verification and restart manually.
The contract version of the same trap
API verification hits this wall constantly. A local studio or runner pointed at a long-lived server serves a previous built spec; the agent compares the live response to an outdated expectation, and 'the endpoint matches' reflects the old build, not the current document. Re-running against a fresh process and diffing the response against the documented schema removes the stale-server illusion. We keep that loop local in Powerduck: re-execute the spec against a freshly started endpoint rather than trusting a warm process, so a green check describes the current code instead of the object still in memory.
What I check before accepting done on a local fix
- Did the verifying run start from a fresh process, or reuse a server that predates the edit?
- Does the reported build hash or module load time change after the edit?
- Did a temporary canary in the changed block actually print?
- Was the exact changed field or branch compared, rather than only a 200 status?
- Does the watcher reload shared modules and config, or is restart manual?
A request that succeeds proves the server answered. It does not prove the process loaded your edit. Until the new code has run in a fresh boot, 'I tested it locally' is a claim about the old module, not the change.
