The webhook dedupe everyone copies has a hole in it

If you have ever searched for how to stop processing the same webhook twice, you have seen this snippet:

INSERT INTO processed_events (event_key)
VALUES ($1)
ON CONFLICT (event_key) DO NOTHING
RETURNING event_key;

Got a row back, you are the first one here, do the work. Got nothing, somebody already handled it, return 200 and move on. It is correct, it is atomic, and it beats the check-then-insert version that races itself under load.

It also has a hole. Claim the key, start the work, and have the process die before the work commits: the key stays in the table, marked as handled. Every retry after that hits the conflict branch and returns 200. The sender is happy. The event never happened.

I want to walk through that specific failure, because most dedupe tutorials stop one paragraph before it.

Why you are seeing the same event twice at all

Duplicates are not a provider bug you can report. They fall out of how at-least-once delivery works.

Your handler gets an event, commits the write, and then the response back to the sender gets lost, or your reply lands two seconds after the sender's read timeout. From where the sender sits, that delivery failed, so it sends the event again. Stripe retries failed events for up to three days in live mode, and you can resend anything from the dashboard on top of that. GitHub lets you redeliver anything from the last three days out of the UI.

There is no timeout value that closes the window. The duplicate is a normal outcome of a system that would rather send twice than lose one, so the handler has to make repeats harmless.

The snippet, and where it breaks

Here is the whole thing in Python, roughly as it gets copied into production:

claimed = db.execute(
    "INSERT INTO processed_events (event_key, source) "
    "VALUES (%s, %s) ON CONFLICT (event_key) DO NOTHING "
    "RETURNING event_key",
    [event_key, source],
).fetchone()

if not claimed:
    return 200  # already handled

fulfil_order(event)   # the actual work
return 200

Two things commit here, and they commit separately. The claim goes in first. The work goes in second. Anything that kills the process in between leaves you with a key that says "done" and a side effect that never ran: a pod restart mid-deploy, an OOM kill, a connection drop to the downstream API, a SIGTERM your worker does not handle gracefully.

The window is small. It is also hit constantly, because deploys happen during traffic and webhook volume is bursty. And the failure is quiet: no error, no retry, no alert. The event is just gone, and you find out when a customer emails about the thing they paid for.

Worse, the ordering is what makes it quiet. Do the work first and crash before the claim and you get a duplicate, which is loud and recoverable. Claim first and crash before the work and you get a loss, which is neither.

Fix one: make it a single commit

If the side effect is a write to the same database, this is easy. Put the claim and the work in one transaction and stop thinking about it:

with db.transaction():
    claimed = db.execute(
        "INSERT INTO processed_events (event_key, source) "
        "VALUES (%s, %s) ON CONFLICT (event_key) DO NOTHING "
        "RETURNING event_key",
        [event_key, source],
    ).fetchone()

    if not claimed:
        return 200

    fulfil_order(event)   # same transaction, same commit

Now a crash rolls back both. The key is not there, the next retry claims it cleanly, and the work runs exactly once in the sense that matters: one visible effect. ON CONFLICT still does the concurrency half, and the transaction does the crash half.

One thing to watch: the transaction stays open for as long as fulfil_order runs. If that function calls a third party API with a thirty second timeout, you are holding a database connection and a row lock for thirty seconds per event. At low volume nobody notices. At a few hundred events a minute you run out of connections.

Which brings us to the case this pattern cannot cover.

Fix two: claim with a state, when the work leaves the database

Sending an email, charging a card, calling somebody's API. None of that rolls back with your transaction, so a single commit does not exist to hide behind. What you can do is stop pretending the claim is binary. It has three states, not two.

CREATE TABLE processed_events (
    event_key    TEXT PRIMARY KEY,
    source       TEXT        NOT NULL,
    status       TEXT        NOT NULL,   -- 'in_progress' or 'done'
    claimed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at TIMESTAMPTZ
);

The claim becomes an upsert that will take the key back off a worker that has clearly died:

INSERT INTO processed_events (event_key, source, status)
VALUES ($1, $2, 'in_progress')
ON CONFLICT (event_key) DO UPDATE
    SET status = 'in_progress', claimed_at = now()
    WHERE processed_events.status = 'in_progress'
      AND processed_events.claimed_at < now() - interval '5 minutes'
RETURNING event_key;

Three outcomes, and you have to handle all three:

A row comes back and you own the event. Do the work, then set status = 'done' and completed_at = now(). That second update is the only thing that tells future retries to skip.

No row comes back and the stored status is done. Somebody finished this already. Return 200.

No row comes back and the stored status is in_progress with a fresh claimed_at. Another worker has it right now. Do not process, and do not return 200 either: return 409 or 503 so the sender retries in a minute. A 200 here is a lie that costs you the event if the other worker dies.

Since RETURNING cannot tell the last two apart, read the row when the claim fails:

if not claimed:
    status = db.execute(
        "SELECT status FROM processed_events WHERE event_key = %s",
        [event_key],
    ).scalar()
    return 200 if status == "done" else 503

Pick the lease window to be longer than your slowest realistic handler and shorter than your patience. Five minutes is a reasonable default when the work is an HTTP call with a thirty second timeout and a couple of retries. Too short and two workers process the same event; too long and a crashed worker parks the event until the lease expires.

Then make the write idempotent anyway

Dedupe tables fail. Somebody truncates one during a migration, a Redis instance without persistence restarts, a key expires earlier than the provider's retry window. So make the write itself survive being run twice, and the dedupe layer becomes an optimisation rather than the only thing standing between you and a double charge.

A unique constraint on the business key does most of it:

ALTER TABLE orders ADD CONSTRAINT uq_orders_event
    UNIQUE (provider_event_id);

A conditional update covers state transitions, so a replayed payment.succeeded on an invoice that is already paid changes nothing:

UPDATE invoices SET status = 'paid'
WHERE id = $1 AND status = 'pending';

Run either one five times and the database looks the same as after one. That is the actual goal. The dedupe table is just a way to avoid the wasted work.

Use a key that survives the retry

All of the above is worthless if the key changes between copies of the same event. Key on the event's identity, never on the moment it arrived.

Do not use a timestamp taken at receipt, a UUID your framework generates per request, or the retry count. Each of those makes every retry look brand new, which turns your dedupe table into an expensive log of things you processed twice.

Also: do not key on the object id inside the payload. A single charge produces several distinct events, and keying on ch_... means the second event gets swallowed as a duplicate of the first.

How to actually test this

Two tests, neither of which needs load:

Send the same delivery twice, in sequence, and assert the side effect happened once. That catches the changing-key mistake, which is more common than the race.

Then kill the process between the claim and the commit. Drop a sys.exit(1) inside fulfil_order, replay the event, restart, replay again, and check that the work eventually happens. If your second replay returns 200 without doing anything, you have the hole, and you now know exactly where it is.

Longer version of this, including the Idempotency-Key header that everyone confuses with webhook dedupe and why it points the other way, is on our blog: Idempotency keys for webhook consumers.