Idempotency Keys - The Double-Charge Trap

The Problem

You click Pay. The server charges your card, but the response is lost on the way back (network drop, timeout, tab freeze). The UI thinks it failed, so you click Pay again and get charged twice.

A timeout means "I don't know what happened", not "payment failed".

The Idea

Idempotent = doing the same thing twice has the same effect as doing it once.

An idempotency key is a unique ID for one payment attempt. The UI sends it with the request. If the same key arrives again, the backend says "I've already handled this one" and returns the saved result instead of charging again.

• UI's job: create the key once, keep it, and resend the same key on every retry.
• Backend's job: remember each key and its result, so repeats are not executed again.

Full Flow Diagram

Example: user pays USD 50 for order O100.

sequenceDiagram
    actor User
    participant UI as Browser UI
    participant SS as sessionStorage
    participant API as Backend
    participant DB as Database
    participant PSP as Payment Provider

    User->>UI: Click "Pay"
    UI->>SS: Key for O100?
    SS-->>UI: None
    UI->>UI: key = crypto.randomUUID() → K1
    UI->>SS: Save payment-key:O100 = K1
    UI->>UI: Disable Pay button, show "Processing..."

    UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1
    API->>DB: Insert (user, K1, O100, processing)
    DB-->>API: Inserted (K1 is new)
    API->>PSP: Charge USD 50 (provider key derived from K1)
    PSP-->>API: Success, payment P900
    API->>DB: Update K1 → succeeded, result P900
    API--xUI: 200 OK (response LOST)

    UI->>UI: Timeout → show "Checking payment status..."
    UI->>SS: Read key for O100
    SS-->>UI: K1 (same key)
    UI->>API: POST /orders/O100/pay<br/>Idempotency-Key: K1 (retry)
    API->>DB: Find K1
    DB-->>API: K1 = succeeded, P900
    Note over API,PSP: Provider NOT called again
    API-->>UI: 200 OK, payment P900 (saved result)
    UI->>SS: Remove payment-key:O100
    UI-->>User: Show receipt ✅ (charged once)

Step-by-Step

Step 1: User clicks "Pay" — UI gets a key

The UI first checks whether this order already has a key (from an earlier attempt or before a refresh).

• Key found → reuse it. This is a retry of the same payment.
• No key → generate a new one with crypto.randomUUID().

payment-key:O100  →  K1   (e.g. "6f13d95a-e6c1-4bf2-9459-99b502098dc1")

Use crypto.randomUUID() — it is random and practically never collides. Do not use Date.now(), Math.random(), or the amount; two different payments could produce the same value.

Step 2: Save the key BEFORE sending the request

sessionStorage.setItem("payment-key:O100", key);

Save first, then send. If the page crashes or reloads mid-request, the key is still there and the next attempt reuses it.

Where to store on the UI:

Also disable the Pay button while the request is in flight. This stops accidental double clicks, but it is only a UX helper; the key is the real protection.

Step 3: Send the payment request with the key

POST /orders/O100/pay
Idempotency-Key: K1
Cookie: session=...   (user is logged in)

The UI sends only the order ID and key. The backend loads the amount from the order itself; never trust a price sent by the browser.

Step 4: Backend checks the key

Why does the backend store the key? Because the UI can't know whether the first request was charged — only the backend can remember.

The backend saves one row per key:

When a request arrives:

The insert must be atomic (a unique constraint on user + key). If two identical requests arrive at the same moment, only one insert succeeds, so only one charge happens.

Step 5: Backend charges and saves the result

The backend calls the payment provider and passes its own provider idempotency key derived from K1, saved before the call. If the backend crashes after the provider charged but before saving the result, a retry reuses that provider key, so the provider returns the original charge instead of creating a new one.

Then it updates the row to succeeded with result P900 and responds.

Step 6: UI handles the response

The key rule: only remove the key when you know the final outcome.

Step 7: Retry after timeout

The UI retries with the same K1 (a few times, with increasing delays like 1s, 2s, 4s).

Two possible cases, both safe:

• First request never reached the backend → K1 is new → backend charges once now.
• First request was already charged → K1 = succeeded → backend returns P900, no second charge.

That's the whole point: the UI doesn't need to know which case happened.

Step 8: User refreshes the page

flowchart TD
    A[Page loads / user clicks Pay] --> B{Key for O100 in sessionStorage?}
    B -- Yes --> C[Reuse the same key]
    B -- No --> D[Generate new key with crypto.randomUUID]
    D --> E[Save key in sessionStorage]
    C --> F[POST /orders/O100/pay with key]
    E --> F
    F --> G{Response}
    G -- Success --> H[Show receipt, remove key]
    G -- Declined --> I[Show error, remove key]
    G -- In progress / timeout / 5xx --> J[Show 'Checking...', keep key, retry later]
    J --> F

After refresh, the UI finds K1 in sessionStorage and sends the request with K1 again. The backend returns the saved result or "in progress" — never a second charge.

UI Code Example

Illustrative only; status codes depend on your API.

function getOrCreateKey(orderId) {
  const name = `payment-key:${orderId}`;
  let key = sessionStorage.getItem(name);
  if (!key) {
    key = crypto.randomUUID();
    sessionStorage.setItem(name, key); // save BEFORE sending
  }
  return key;
}

async function pay(orderId) {
  const key = getOrCreateKey(orderId);

  try {
    const res = await fetch(`/orders/${orderId}/pay`, {
      method: "POST",
      headers: { "Idempotency-Key": key },
      credentials: "include",
      signal: AbortSignal.timeout(15000),
    });

    if (res.ok) {
      sessionStorage.removeItem(`payment-key:${orderId}`);
      return { status: "paid", payment: await res.json() };
    }
    if (res.status === 402) {
      // Clear decline: not charged, a new attempt may use a new key
      sessionStorage.removeItem(`payment-key:${orderId}`);
      return { status: "declined" };
    }
    // 409 in progress, 5xx, etc.: outcome unknown, keep the key
    return { status: "unknown" };
  } catch {
    // Timeout or network error: outcome unknown, keep the key
    return { status: "unknown" };
  }
}

On "unknown", show "Checking payment status..." and call pay(orderId) again after a delay — it automatically reuses the same key.

sessionStorage can throw (e.g. storage full or blocked). In that case, keep the key in memory for this page and avoid letting the user start a fresh payment while one is unresolved.

When Is a New Key Needed?

Simple rule: same intention → same key. New intention → new key.

Security Checklist

• Store only the key in sessionStorage. Never store card number, CVV, or tokens.
• The key is not a password. The backend must still check that the logged-in user owns order O100.
• The backend scopes keys per user, so one user's key can't read another user's payment.
• Never trust amount or status from the browser; the backend decides both.
• Clear payment keys on logout.

What If sessionStorage Is Lost?

New tab, cleared storage, or another device → the UI generates a new key for the same order. The idempotency key can't catch this, so the backend needs one more rule:

An order that is already paid, or has a payment in progress, cannot start another charge.

Keys stop duplicate requests. Order rules stop duplicate payments for the same order. You need both.

System Design: Key Creation, Change Detection, and Cancellation

So far, one key = one payment. Real checkouts are messier: the user changes the cart, applies a coupon, or switches cards while a payment is unclear. This section designs a setup that answers three questions:

1. When should the UI create a key?
2. How do UI and backend detect that the payment details changed?
3. How is an old attempt safely cancelled so a new key can be used?

Core idea: a key belongs to a checkout snapshot

A key represents "pay for exactly this checkout". The checkout is identified by a small snapshot:

Same snapshot      → same intention → reuse the key
Different snapshot → new intention  → old attempt must be finished/cancelled, then a new key

orderVersion is the important part: the UI doesn't have to compare cart items itself. The backend increases the version whenever anything affecting the price changes.

Architecture

flowchart LR
    subgraph Browser
        UI[Checkout UI]
        KM[Key Manager]
        SS[(sessionStorage)]
        UI --> KM
        KM <--> SS
    end

    KM -->|Pay / Cancel + Idempotency-Key| GW[API Gateway<br/>auth, rate limit]
    GW --> PS[Payment Service<br/>idempotency check]
    GW --> OS[Order Service<br/>cart, orderVersion, lock]
    PS <--> IDB[(Idempotency Store<br/>key, fingerprint, status, result)]
    PS <--> OS
    PS -->|charge with provider key| PSP[Payment Provider]
    PSP -->|webhook: final result| WH[Webhook Handler]
    WH --> PS
    RJ[Reconciliation Job] -->|resolve stuck attempts| PS
    CJ[Cleanup Job] -->|expire old finished keys| IDB

Data model

Idempotency store (one row per key):

Order (relevant columns):

The backend calculates the fingerprint from its own data (loaded order + amount), not from values the browser claims.

Attempt lifecycle

stateDiagram-v2
    [*] --> processing: Pay request accepted (new key)
    [*] --> cancelled: Cancel arrives before Pay
    processing --> succeeded: Provider approved
    processing --> declined: Provider declined
    processing --> processing: Provider timeout, waiting for webhook / reconciliation
    processing --> failed: Reconciliation found no charge
    succeeded --> [*]
    declined --> [*]
    failed --> [*]
    cancelled --> [*]

• Cancel is only possible before charging starts. Once processing, the UI must wait for the final result.
• A succeeded payment cannot be "cancelled". Undoing it is a refund, a separate operation with its own key.

When should the UI create a key?

Rule: create the key at the moment of commitment (the Pay click), never earlier.

Backend rules on POST /orders/O100/pay

The request now carries the snapshot:

POST /orders/O100/pay
Idempotency-Key: K1
Content-Type: application/json

{ "orderVersion": 3, "paymentMethodId": "pm_123" }

The backend checks in this order:

1. Authenticate and confirm the user owns O100.
2. Look up the key. If it already exists:
   • Fingerprint differs → 422 KEY_REUSED_WITH_DIFFERENT_DETAILS. Nothing is charged.
   • succeeded / declined / failed → replay the saved result.
   • processing → 409 IN_PROGRESS.
   • cancelled → 409 ATTEMPT_CANCELLED. This blocks a late, delayed request from charging after the user cancelled.
3. New key. In one database transaction:
   • Request orderVersion ≠ current version → 409 ORDER_CHANGED (UI must refresh the checkout).
   • Order is paid or already has an active_attempt → 409 PAYMENT_ALREADY_ACTIVE.
   • Otherwise insert the key as processing and set order.active_attempt = K1.
4. Charge using the saved provider key, then save the result:
   • Success → key succeeded, order paid.
   • Decline → key declined, clear active_attempt.
   • Timeout → stays processing; webhook or reconciliation finishes it later.

Key lookup comes before the version check, so a retry of an already-paid attempt still gets its saved receipt.

Cart edits while a payment is active

The Order Service rejects cart changes while active_attempt is set:

PATCH /orders/O100/cart  →  409 PAYMENT_IN_PROGRESS

The UI also disables editing during payment. The backend check is what actually guarantees it (another tab or device could try to edit).

Cancel endpoint

POST /orders/O100/payment-attempts/cancel
Content-Type: application/json

{ "key": "K1" }

The key goes in the body, not the URL, so it doesn't end up in access logs or browser history.

Why insert a tombstone for an unknown key? Because the original Pay request might be stuck in the network and arrive later. The tombstone uses the same unique (user_id, key) slot, so whichever arrives first wins, and the late Pay is rejected instead of charging.

Scenario: user changes the cart after a timeout

sequenceDiagram
    actor User
    participant UI as Checkout UI
    participant SS as sessionStorage
    participant API as Backend

    User->>UI: Click Pay (cart v3, card pm_123)
    UI->>SS: Save {key: K1, snapshot: "v3:pm_123"}
    UI->>API: POST /pay, key K1, v3
    API--xUI: Timeout (outcome unknown)
    UI-->>User: "Checking payment status..."

    User->>UI: "I want to change my cart"
    UI->>API: Cancel K1
    alt K1 never reached backend
        API->>API: Insert K1 as cancelled (tombstone)
        API-->>UI: cancelled
        UI->>SS: Remove K1
        UI-->>User: Cart unlocked
        User->>UI: Edit cart → backend bumps to v4
        User->>UI: Click Pay
        UI->>SS: Save {key: K2, snapshot: "v4:pm_123"}
        UI->>API: POST /pay, key K2, v4
        API-->>UI: Success ✅
        Note over API: If delayed K1 arrives now → cancelled, no charge
    else K1 is processing
        API-->>UI: processing
        UI-->>User: "Payment in progress, please wait"
    else K1 already succeeded
        API-->>UI: succeeded, P900
        UI->>SS: Remove K1
        UI-->>User: Show receipt (refund if they still want changes)
    end

Scenario: backend crashes mid-payment (K1 stuck in processing)

The key reached the backend and was saved as processing, but the status was never updated. Now the user retries with K1. What happens?

Why K1 gets stuck:

1. UI sends Pay with K1
2. Backend saves K1 = processing (+ provider key PK1)   ✅ saved
3. Backend calls the provider
4. One of these happens:
   a) Backend crashes before or during the provider call  → maybe NOT charged
   b) Provider charged, backend crashed before saving      → charged
   c) Provider is slow or timed out                       → unknown
5. K1 is still "processing" in the DB

The backend itself doesn't know whether money moved.

What the retry does:

UI      -> Backend: Pay O100 with K1 (retry)
Backend -> DB:      Find K1
DB      -> Backend: K1 = processing
Backend -> UI:      409 "Payment in progress"   (provider NOT called)
UI:     shows "Checking payment status...", asks again in a few seconds

Rule: a retry request never starts a charge for a key that is already processing. It could be case (b) or (c), where money already moved or is moving.

Who fixes the stuck status?

1. Provider webhook (usually first). When the provider finishes the charge, it notifies the backend's webhook handler, which updates K1 to succeeded or declined.
2. Reconciliation job (safety net). Every minute or so, it finds attempts stuck in processing for longer than a threshold (e.g. 2 minutes) and asks the provider about them using PK1, which was saved with K1 before calling the provider.

Retrying with the same PK1 is safe: if the charge already exists, the provider returns it instead of creating a second one.

If the provider can't be reached, or offers no idempotency or status lookup, K1 stays processing and the order stays locked. It is flagged for manual review, never charged blindly.

sequenceDiagram
    participant UI as Checkout UI
    participant API as Backend
    participant DB as Database
    participant PSP as Payment Provider
    participant RJ as Webhook / Reconciliation

    UI->>API: Pay with K1
    API->>DB: K1 = processing, save PK1
    API->>PSP: Charge with PK1
    PSP->>PSP: Charge succeeds
    Note over API: 💥 Backend crashes. K1 still processing
    API--xUI: No response

    UI->>API: Retry with K1
    API->>DB: Find K1 → processing
    API-->>UI: 409 In progress (no charge)
    UI->>UI: "Checking payment status..."

    RJ->>PSP: Status of PK1?
    PSP-->>RJ: Succeeded, P900
    RJ->>DB: K1 = succeeded, P900, order paid

    UI->>API: Retry with K1
    API->>DB: Find K1 → succeeded
    API-->>UI: 200 Paid, P900 (saved result)
    UI->>UI: Show receipt, remove K1

Timeline:

12:00:00  UI: Pay with K1
12:00:00  Backend: K1 = processing, PK1 saved
12:00:01  Backend calls provider with PK1 → provider charges
12:00:01  Backend CRASHES (K1 still processing)
12:00:15  UI timeout → "Checking payment status..."
12:00:20  UI retries K1 → 409 in progress (no charge)
12:00:30  UI retries K1 → 409 in progress (no charge)
12:00:40  Webhook arrives: PK1 succeeded → K1 = succeeded, P900
12:00:45  UI retries K1 → 200 "Paid, P900" (saved result)
          UI shows receipt, removes K1 from sessionStorage
          Customer charged ONCE ✅

If the webhook never arrives, the reconciliation job performs the same update around 12:02.

What the user sees:

• While processing: "Payment in progress, please don't close this page." Pay button and cart stay locked.
• After a long wait (e.g. 1-2 minutes): "We're confirming your payment. We'll email you when it's done." The UI can stop polling; the backend still finishes the work.
• On refresh: the UI reads K1 from sessionStorage and asks again, getting the latest status.
• If the attempt ends as failed (not charged), the backend replays it like a decline (e.g. 402 in the UI example), so the UI removes K1 and the next Pay creates a new key.

Why it's safe:

Key Manager code (UI)

Illustrative; extends the earlier example with a snapshot and cancel.

const nameFor = (orderId) => `payment-attempt:${orderId}`;
const snapshotOf = (c) => `${c.orderVersion}:${c.paymentMethodId}`;

// Called only when the user clicks Pay
function getOrCreateAttempt(checkout) {
  const name = nameFor(checkout.orderId);
  const saved = JSON.parse(sessionStorage.getItem(name) ?? "null");
  const snapshot = snapshotOf(checkout);

  if (saved && saved.snapshot === snapshot) {
    return saved; // same intention: reuse key
  }
  if (saved) {
    // Details changed but the old attempt is unresolved
    throw new Error("Resolve the previous payment attempt first");
  }

  const attempt = { key: crypto.randomUUID(), snapshot };
  sessionStorage.setItem(name, JSON.stringify(attempt)); // save BEFORE sending
  return attempt;
}

// Called when the user wants to change details during an unclear payment
async function cancelAttempt(orderId) {
  const saved = JSON.parse(sessionStorage.getItem(nameFor(orderId)) ?? "null");
  if (!saved) return { status: "none" };

  try {
    const res = await fetch(`/orders/${orderId}/payment-attempts/cancel`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      credentials: "include",
      body: JSON.stringify({ key: saved.key }),
    });
    if (!res.ok) return { status: "unknown" }; // keep key, try later

    const { status } = await res.json();
    if (["cancelled", "declined", "failed", "succeeded"].includes(status)) {
      sessionStorage.removeItem(nameFor(orderId)); // final: key no longer needed
    }
    return { status }; // "processing" → keep key and wait
  } catch {
    return { status: "unknown" }; // network error: keep key
  }
}

UI reactions:

• cancelled / declined → unlock the cart; the next Pay creates a new key.
• processing / unknown → keep the cart locked, show "Payment in progress".
• succeeded → show the receipt; offer a refund if the user still wants changes.
• Pay returns 409 ORDER_CHANGED → the order changed elsewhere (another tab). Reload the checkout, show the new total, and let the user confirm. The old key was never accepted, so remove it; the confirmation Pay creates a new key.

Design summary

Summary

Click Pay
  → UI: reuse key, or create one with crypto.randomUUID()
  → UI: save key in sessionStorage
  → UI: POST with Idempotency-Key
  → Backend: new key? charge and save result. Known key? return saved result
  → UI: success/decline → remove key | unknown → keep key and retry

More Details:

Get all articles related to system design
Hashtag: SystemDesignWithZeeshanAli

systemdesignwithzeeshanali

Git: https://github.com/ZeeshanAli-0704/front-end-system-design

⬆ Back to Top