The Provider Retried My Webhook. My Model Ran Twice. Here's the Dedupe That Fixed It.

Webhooks look simple until the provider decides your response is too slow. I spent 48 hours running a small event classifier on a free server, and the pattern was painfully consistent: every slow response came back as a retry, and every retry re-ran the same model call. The retries were not the real problem, though. The real problem was that my code kept paying for work it had already done.

The setup was deliberately boring. A webhook endpoint received events, a background worker classified them with a free model, and a database stored the results. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I ran the receiver on MonkeyCode's free server option and used its free model access for the classification step, which kept the experiment free.

What I tried first: blocking the webhook

The first version was the most obvious one, and the most fragile. The endpoint parsed the event, called the model, saved the label, and returned the result. It read like a tutorial, and it failed like one too.

@app.post("/webhook")
async def webhook(request: Request):
    payload = await request.json()
    label = await classify(payload["text"])   # 6 to 15 seconds
    save_label(payload["event_id"], label)
    return {"label": label}

The model call took anywhere from six to fifteen seconds, and the provider's timeout was shorter than that. The response never arrived in time, so the provider retried, and the retry hit the same slow endpoint. Why did the retry re-run the model? Because my endpoint was still blocking, and retries are the only tool the provider has. Two events became six model calls in the first six hours, and the labels were still missing.

What broke: the retry storm and the memory trap

My first fix was an in-memory dedupe set, and it worked beautifully for about twelve hours. Then the free server recycled my idle process overnight, the set vanished with it, and the next retry ran the model again. An in-memory set is a cache, not a contract, and I had confused the two.

seen = set()  # gone after the next restart

@app.post("/webhook")
async def webhook(request: Request):
    payload = await request.json()
    event_id = payload["event_id"]
    if event_id in seen:
        return {"status": "duplicate"}
    seen.add(event_id)
    await queue.put(payload)

A table of the full 48 hours makes the pattern obvious, and it also shows why I stopped chasing the symptom.

The fix: 202 first, SQLite as the memory

The version I kept follows three rules: answer the provider immediately, dedupe against disk, and let the worker do the slow work. The endpoint returns 202 before the model is even awake, which means the provider has no reason to retry.

@app.post("/webhook")
async def webhook(request: Request):
    payload = await request.json()
    await queue.put(payload)
    return {"status": "accepted"}  # 202, fast, no model call

The worker then claims the event with an INSERT OR IGNORE, and this is where the magic lives. If the row already exists, the event was handled, and the model never sees it twice.

async def worker():
    con = sqlite3.connect("events.db")
    con.execute("CREATE TABLE IF NOT EXISTS processed (event_id TEXT PRIMARY KEY)")
    while True:
        payload = await queue.get()
        event_id = payload["event_id"]
        cur = con.execute(
            "INSERT OR IGNORE INTO processed (event_id) VALUES (?)",
            (event_id,),
        )
        con.commit()
        if cur.rowcount == 0:
            continue  # already classified
        label = await classify(payload["text"])
        print(f"{event_id}: {label}")

Three details matter here. First, the primary key is the provider's event ID, because that is the only value both sides agree on. Second, the INSERT happens before the model call, so a crash after the insert cannot cause a duplicate. Third, SQLite lives on disk, so the free server can recycle the process and the dedupe survives.

How I tested it: a retry-happy fake provider

I did not wait for the real provider to misbehave again, because I could simulate its worst habit in five lines. The fake provider sends the same event three times, which is exactly what a retry storm looks like from the outside.

# fake_provider.py — sends the same event three times
import requests, time

payload = {"event_id": "evt_42", "text": "order cancelled"}
for i in range(3):
    r = requests.post("http://localhost:8000/webhook", json=payload)
    print(i, r.status_code, r.json())
    time.sleep(1)

Run the fake provider and then count the model calls in the worker log. The first version printed three labels, while the fixed version prints one, and the other two requests return accepted without touching the model. That one-line difference is the entire point of the exercise.

What I'd repeat, and what I'd never do again

Given the same 48 hours, I would keep the 202-first pattern, the disk-backed dedupe, and the single worker without hesitation. I would never again block a webhook response on a model call, and I would never trust a set that dies with the process. The rules are short enough to tape to a monitor:

• Answer the provider in milliseconds, and do the slow work in the background.
• Dedupe by the provider's event ID, not by a hash of the payload.
• Persist the dedupe set, because free servers recycle processes without asking.
• Accept at-most-once processing when a lost event is cheaper than a duplicate.

Limitations and who should not use this

This design is at-most-once, not at-least-once, and the distinction is easy to miss. If the process dies between the INSERT and the model call, that event is gone, because the dedupe table now claims it was handled. For my classifier that was fine, since a missed label was cheaper than a double model call. If you are processing payments, sending emails, or doing anything where silence is expensive, do not copy this; you need a real queue with acknowledgements, a dead-letter topic, and idempotent side effects.

The webhook provider was never the enemy, because it was just doing the only thing it could: retrying a slow endpoint. The fix was to make my endpoint fast, my memory durable, and my model call a side effect that happens at most once.