The incident started with a support ticket: customer paid, order number never arrived, but the payment provider webhook showed a successful charge.
We dug in. The checkout flow had been updated two weeks earlier. The agent's commit message was correct and boring: "add idempotency-key support to POST /checkout." The diff looked reasonable. It checked a Redis hash by Idempotency-Key, and if present, returned early. The test simulated a duplicate POST and asserted the second call returned 201. CI went green.
The bug was in what got stored.
What the agent actually wrote
The handler did roughly this:
key = req.Header.Get("Idempotency-Key")
if key != "":
cached, err := redis.Get(ctx, "idem:" + key)
if err == nil:
w.WriteHeader(int(cached))
return
// ... do the work, create order, charge card ...
w.WriteHeader(201)
redis.Set(ctx, "idem:" + key, "201", 24*time.Hour)
On the first request, the order was created, the card was charged, and the response body was rendered with the order number, the total, and the shipping estimate. The cache stored the string "201".
On the second request — same key, retried because the network dropped the first response — the handler read "201", called w.WriteHeader(201), and returned. No body. No order number. The HTTP status was correct. The payload was empty.
The mobile client's parser saw 201, treated it as "checkout accepted," and moved on. The user saw a success screen. The order record from the first request had, in fact, been committed. But under the retry path, when the first request timed out before the client received the response, the retry re-entered the handler, hit the cache, and got a bare status. The order had been created on the first pass — except when the first request itself had failed after the Redis write but before the order commit. In that narrow window, the second request returned a cached 201 for an order that did not exist.
That is the window that produced the support tickets.
Why the test missed it
The unit test looked like this:
func testIdempotentCheckout(t *testing.T) {
resp1 := doCheckout(key)
resp2 := doCheckout(key)
assert.Equal(t, 201, resp1.StatusCode)
assert.Equal(t, 201, resp2.StatusCode)
}
It asserted the status code. It never compared resp2.Body against resp1.Body. The agent wrote the test the way the feature was described in the ticket: "duplicate POST returns 201." It did exactly that. The gap between "status matches" and "response is byte-equivalent" was invisible.
The fix
Two changes. First, cache the whole serialized response, not the status:
body, _ := json.Marshal(orderResponse)
redis.Set(ctx, "idem:"+key, string(body), 24*time.Hour)
// on replay:
cached, _ := redis.Get(ctx, "idem:"+key)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(201)
w.Write([]byte(cached))
Second, write the cache entry only after the order commit has been confirmed. If the order creation fails, delete the key so a retry can start clean. Otherwise a partial-failure state permanently advertises a success that never happened.
The bigger lesson
Idempotency is not just "don't double-charge the card." It is "replaying the same request must produce the same observable outcome, including the response body the caller uses to drive its UI." A status code alone is not a replayable result.
This is also a contract problem. When the agent added the idempotency layer, it changed what a client sees on retry — but the OpenAPI document for /checkout still said 201 returns an order object. The document was right on first request and wrong on replay. We now replay recorded request/response pairs against the local spec on every change, so a handler that silently drops its body on a cached path shows up as a red diff in CI instead of a support ticket two weeks later. That is the same local-first loop we built into Powerduck: the spec stays the source of truth, and drift is caught before it ships.
What I check now
Any time an agent touches a POST endpoint that writes state, I ask three things before approving:
- What is stored for the idempotency key — status only, headers, or full body?
- At what point in the handler is the cache entry written — before or after the commit?
- Does the test replay assert the full response body, or just the status?
Number three is the one that catches this class of bug cheapest. Status-code-only idempotency tests are everywhere. Body-equality tests are rare.
