Webhook Rate-Limit Queue Compare: 3 Delayed-Job Recovery Rules for US/EU

Short answer: choose the queue that makes recovery state explicit, then put rate limits, delayed jobs, retry, and DLQ handling around that state. For a B2B SaaS cleanup worker, the cheapest-looking queue is rarely the cheapest failure to operate.

The concrete job here is a periodic cleanup of webhook deliveries that have aged out, stalled, or exceeded a tenant's rate budget. A web request records intent; a worker does the slow work later. That boundary keeps a customer-facing request short and gives operations somewhere to resume.

The decision record: recovery is the unit of design

I write three invariants before comparing queue products. Every delivery has a stable idempotency key. Every attempt has a next-eligible timestamp. Every terminal outcome is inspectable without replaying the original request. Those rules matter in both US and EU regions because a retry that crosses a regional boundary still needs one owner and one audit trail.

The failure boundary is the handoff from the scheduler to the worker. A timeout before the handoff means “not claimed.” A timeout after the worker has sent the webhook means “possibly delivered.” Treating both as the same error creates duplicate notifications. I started with a 15-minute lease on one cleanup worker, then traced a 409 storm caused by exactly that ambiguity: the sender retried a job whose receiver had already accepted it, the receiver acknowledged the second request differently, and our dashboard counted both attempts as failures even though the first side effect had landed. The repair was a state transition in the ledger, not another queue setting; the worker had to record “sent or unknown” before deciding whether to release a retry.

That is the whole trick.

Here is the comparison I use in an architecture review. It is a starting map, not a price ranking.

The table does not answer “cheapest.” It answers “where will the next operator look when a job is late?” That question is more predictive of total effort than a per-message number.

How should a Python worker handle webhook rate limits, delayed jobs, retry, and DLQ recovery?

Model the queue as a hint to wake a worker, not as your source of truth. A small relational ledger can hold delivery_id, tenant_id, attempt, next_at, state, and last_error_class. The queue message carries only the identifier. If a message is duplicated, the ledger makes the second execution a no-op.

The critical path can stay boring:

from datetime import datetime, timedelta, timezone

MAX_ATTEMPTS = 8


def claim_and_schedule(ledger, queue, delivery_id, now=None):
    now = now or datetime.now(timezone.utc)
    row = ledger.lock_delivery(delivery_id)
    if row is None or row.state in {"sent", "dead"}:
        return "ignored"
    if row.next_at > now:
        return "not_due"

    if not ledger.tenant_budget_available(row.tenant_id, now):
        ledger.defer(delivery_id, now + timedelta(seconds=30), "rate_limit")
        queue.enqueue(delivery_id, delay_seconds=30)
        return "deferred"

    ledger.mark_attempt(delivery_id, row.attempt + 1)
    try:
        response = deliver(row.payload, row.endpoint)
    except TimeoutError:
        response = None

    if response is not None and 200 <= response.status < 300:
        ledger.mark_sent(delivery_id)
        return "sent"

    attempt = row.attempt + 1
    if attempt >= MAX_ATTEMPTS:
        ledger.mark_dead(delivery_id, "attempt_limit")
        return "dead"

    delay = min(3600, 2 ** attempt * 5)
    ledger.defer(delivery_id, now + timedelta(seconds=delay), "transient")
    queue.enqueue(delivery_id, delay_seconds=delay)
    return "retry"

The code deliberately separates a rate-limit deferral from a delivery failure. A 429 is a scheduling signal; a malformed payload is a data problem; a timeout is ambiguous. Your mileage may vary on the exact backoff, but the classifications should be stable enough for dashboards and on-call runbooks.

Use HMAC to authenticate callbacks or queue-to-worker handoffs, and include a timestamp in the signed material so an old message cannot be replayed indefinitely. RFC 2104 defines the keyed-hash construction; it does not define your replay window, so record that policy beside the code.

Where do delayed jobs become operational debt?

Delayed delivery is easy to demo and hard to drain. A backlog can be healthy when a partner is enforcing a rate limit, yet unhealthy when the next_at index is missing and every worker scans the whole ledger. I prefer a due-work query, a bounded claim batch, and a metric for age of the oldest due item.

Regional queues add another wrinkle. Keep the tenant's data residency decision separate from the worker's location. If a US worker can process an EU tenant, document that transfer; if it cannot, route by policy before enqueueing. A queue name that says eu is not evidence that payloads stayed in the EU.

The rejected option in this record is a cron-only sweep that calls the webhook directly. It is valid for a tiny, idempotent maintenance task with no customer-visible latency and a short run time. It is a poor fit once a sweep can outlive the scheduler process, because there is no durable per-delivery lease to recover. A cron entry can still wake the system, but it shouldn't be the only record of which tenant was due, deferred, or awaiting review.

What should the DLQ contain for a cleanup service?

A dead-letter queue should preserve the delivery identifier, tenant, attempt count, classification, first-seen time, and a pointer to the immutable payload. Do not copy secrets into a debugging message. Operators need enough context to decide, “fix data, raise a limit, or replay,” without exposing credentials in logs.

I set replay permissions separately from read permissions. A replay is a write with side effects, and it should require an approval trail. The cleanup job can offer a dry-run that verifies the signature, destination policy, and current state before releasing one item.

There is a real limit here: a queue cannot repair a non-idempotent receiver. When the endpoint has no idempotency contract, keep retries conservative and escalate ambiguous timeouts to a human-reviewed state. Stick with a simpler scheduled task when the business can tolerate a missed run and there is no per-event recovery requirement.

References

• https://www.rfc-editor.org/rfc/rfc2104
• https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows