A customer taps "Pay ₹4,999." The request reaches your payment service, the card gets charged, money actually moves. And then, before the confirmation makes it back, the response gets lost somewhere on the way, a dropped connection, a flaky mobile network, a load balancer timing out the wait. The customer's screen just sits there spinning.
They do the only reasonable thing: they tap Pay again.
If nothing is protecting that endpoint, your system charges them a second time. Not because anything crashed, not because your code has a bug in the usual sense, but because the payment already succeeded and nobody told the client. I put together an animated walkthrough of exactly this failure and how idempotency prevents it on SeeItFlow, if you'd rather watch the request flow play out than picture it from the sequence below.
The failure isn't the retry, it's not knowing what happened
Walk through what the client actually experiences: it sends the payment request, the bank charges the customer, and then the response is lost on the way back. From where the client is sitting, this looks identical to the request never having reached the server in the first place, or to the server crashing mid-operation. All three of those are indistinguishable to a client staring at a timeout.
And that's really the whole problem in one sentence: when a request times out, there are three things that could have happened, and the client has no way to tell which one occurred. Either the request genuinely never arrived, in which case retrying is completely safe. Or it arrived and the server crashed before finishing, in which case retrying is still safe, since nothing completed. Or, the dangerous case, it arrived, ran successfully, and only the response got lost on the way home. Here a retry means re-running an operation that already happened.
Since the client can't distinguish the safe cases from the dangerous one, refusing to retry isn't a real option either, that would mean every dropped response turns into a stuck transaction requiring the user to figure out what went wrong themselves. The only workable answer is to make retries always safe, so the client never has to guess.
What idempotency actually means
An idempotent operation produces the same result no matter how many times you run it. DELETE /orders/42 is a natural example: run it once and the order's gone, run it again and it's still gone, nothing changes on the second call. PUT /users/7 with {"status": "ACTIVE"} behaves the same way, the account ends up active whether you send that request once or five times.
POST /payments doesn't have that property on its own. Run it once and you've charged ₹4,999. Run it again and, unless something's stopping it, you've charged ₹4,999 a second time. The operation itself has a side effect that compounds with every repetition, which is exactly the shape of thing a retry can accidentally trigger.
The idempotency key, and what it actually buys you
The fix is for the client to attach a unique identifier to the operation, not to the individual HTTP request, but to the underlying thing it's trying to accomplish:
POST /payments
Idempotency-Key: pay_abc123
The first time the server sees that key, it processes the payment normally and stores the outcome against the key:
{ "key": "pay_abc123", "payment_id": 987, "status": "SUCCESS" }
Now say the response gets lost, same as in the opening scenario, and the client retries with the identical key:
POST /payments
Idempotency-Key: pay_abc123
The server checks its key store, finds pay_abc123 already resolved, and just hands back the stored result instead of touching the payment logic at all:
{ "payment_id": 987, "status": "SUCCESS" }
No second call to the bank. No second charge. The retry becomes a lookup instead of a transaction.
Internally that's the whole shape of it: on the first request, the key isn't found, so the server marks it in progress, does the actual work, and stores the result. On any retry, the key is found, so the stored response gets returned directly and the payment logic never runs again.
The race condition that breaks this if you're not careful
Here's the part that catches people who implement this correctly on paper but miss one detail. Suppose two identical requests, same idempotency key, arrive close enough together that both check the key store before either one has written anything back. Both see "key not found." Both proceed. Both charge the customer. The idempotency key didn't help at all, because the check and the write weren't a single atomic step.
This is the classic check-then-act race, and it's a real bug, not a hypothetical, it shows up whenever a client fires a request twice in quick succession, say a double-tap on a slow connection. The fix is to make claiming the key atomic: a Redis SETNX, or a database unique constraint on (scope_id, idempotency_key), so that whichever request arrives fractionally first wins the claim outright, and the second one gets rejected before it can do anything, typically with a 409 Conflict, rather than being allowed to race ahead.
This isn't a hypothetical problem, it's standard practice
Stripe's API uses an Idempotency-Key header on POST requests specifically so the same key always resolves to the same outcome. PayPal does the equivalent with PayPal-Request-Id to prevent duplicate payment creation. Amazon uses a ClientToken on operations like order creation, so the same token can never produce two orders. And on the product side, Uber's ride creation is built the same way, so a double tap on "Request Ride" doesn't put two drivers en route to you.
None of these are edge-case protections bolted on after an incident. They're a default assumption baked into how the API is designed, because at the scale these companies operate, "the response got lost" isn't rare, it's a Tuesday.
The pattern isn't always a header
An explicit idempotency key is the right tool for payments, order creation, and calls to external APIs, but it's not the only shape this problem takes. Sometimes a natural unique constraint does the same job for free: UNIQUE(email) on a users table means a duplicated registration request simply fails on the second attempt rather than creating a second account, no key required. Sometimes an upsert is the cleanest fix, INSERT ... ON CONFLICT DO UPDATE means running a configuration sync job three times in a row leaves the system in the same state as running it once. And sometimes the fix is just choosing the right verb: PUT { "quantity": 5 }, which states the target value, is naturally idempotent, while PATCH { "delta": +1 }, which states a change relative to the current value, isn't, since applying it twice doesn't give you the same result as applying it once. If you find yourself needing delta semantics badly enough to justify the risk, that's exactly the situation an idempotency key exists for.
Where this goes wrong in practice
A handful of mistakes show up repeatedly once idempotency actually ships, and they're worth knowing about before they surface as an incident.
Generating a new key on every retry defeats the entire mechanism. If retry #1 uses key A, retry #2 uses key B, and retry #3 uses key C, every single retry looks like a brand-new operation to the server, because as far as the idempotency system can tell, it is one. The key needs to be generated once per business operation, at the moment the user clicks Pay, not once per network attempt, and then reused for every retry of that same attempt.
A TTL shorter than your retry window quietly reopens the same hole. If the stored key expires before a slow client finishes retrying, that late retry looks like a fresh key to the server, and you're back to processing it as new. The expiry needs to comfortably outlast the longest realistic retry sequence a client might run.
Storing only the key, not the response, sounds like it should be fine and isn't. If all the server remembers is "pay_abc123 = processed," a retry has nothing to actually return, so implementations end up improvising something, and what comes back on retry #1 doesn't quite match what comes back on retry #2. Store the full response alongside the key, so a replay is a genuine replay.
An unscoped key namespace is a quieter problem, but a nastier one. If keys aren't scoped per user, there's nothing stopping user A's pay_abc123 from colliding with user B's pay_abc123, and depending on how your lookup works, that's either a broken payment or, worse, one user's stored response leaking to another user's retry. Scope every key by user or API key, something like user_id:idempotency_key, so collisions across different customers simply can't happen.
Where it actually applies
Payments, order creation, and anything that moves money should treat idempotency as non-negotiable, not optional hardening. Inventory reservations and resource creation usually want it too, since double-booking or double-provisioning tends to be expensive to unwind after the fact. Things like email or SMS sending sit in a grayer zone, a duplicate notification is annoying but rarely catastrophic, so whether it's worth the engineering effort depends on how much your users would actually mind. Analytics and metrics ingestion can usually tolerate occasional duplicates and get deduplicated downstream instead. And GET or HEAD requests don't need any of this in the first place, they're safe to retry by definition, since they were never supposed to change anything to begin with.
The mental model worth keeping
Idempotency isn't really about preventing duplicate requests, requests get duplicated constantly, by flaky networks, by users double-clicking, by mobile clients reconnecting after losing signal, and there's no way to stop that from happening. What it actually prevents is duplicate business outcomes: the same money moving twice, the same order getting created twice, the same ride getting requested twice. Once you accept that retries are unavoidable in any system talking over a real network, the only real choice is whether a retry is safe or not. Idempotency is what moves that answer from "it depends" to "always."
References
Want to see the request flow, the race condition, and the failure scenarios above played out step by step instead of just reading through them? The animated version on SeeItFlow walks through a payment retry from the client's first tap through to the replayed response.
Has a missing idempotency key ever caused a real incident on something you've worked on? I'd like to hear how it surfaced, usually these show up in the least convenient way possible.
