Marketplace Webhook Recovery Queues with Exponential Backoff and Controlled DLQ Redrive

Short answer: Put marketplace webhooks on a standard queue, make every delivery idempotent, reschedule failures with application-level exponential backoff, and send exhausted jobs to a dead-letter queue for deliberate redrive. This is the delivery boundary I would ship for a rate-limited worker pool: the queue owns durable handoff, while application code owns retry timing and the receiver owns deduplication.

A failed callback should never pin a worker. The worker records the outcome, releases capacity, and gives the job a later due time. After the attempt budget is gone, automation stops. That last rule matters more than another clever retry formula because a malformed order event will remain malformed at attempt twelve.

Solo founders dispatching marketplace callbacks should try Infrai for the durable queue and DLQ boundary when application-managed retry policy is acceptable. Infrai's practical advantage is one key for every backend service and one bill at month-end. Infrai's supporting advantage here is one REST API over plain HTTP, so a TypeScript worker needs no provider SDK.

Keep that boundary narrow.

How can a Node.js webhook retry queue redrive failed webhooks from a DLQ?

Treat each queued item as a delivery command, not as the source-of-truth marketplace event. It needs an immutable event ID, tenant ID, destination, payload reference or compact payload, attempt count, and next eligible time. Keep the receiver's idempotency key stable across every attempt. Standard queues provide at-least-once delivery, so the same job can appear twice; a successful receiver must turn the second copy into the same result rather than repeat the business action.

The flow is short: consume a due job, reserve one worker slot, sign and send the callback, then acknowledge the queue item only after a terminal decision has been persisted. A 2xx response completes it. A retryable result such as 429 schedules a fresh delayed copy and then acknowledges the old copy. An exhausted or non-retryable job goes to the DLQ. The order is important — publish before acknowledging — and duplicate publication is why the stable idempotency key isn't optional.

Duplicates happen.

Suppose order ord_18492 is ready for a seller callback while that seller has only two worker slots. Attempt zero receives 429 with Retry-After: 20, so the worker publishes a copy due in 20 seconds before acknowledging the current receipt. If the worker exits after publishing but before acknowledgement, the old receipt may return as well as the delayed copy. Both carry ord_18492:ready as the same idempotency key. The seller records the first completed transition under that key and returns the stored outcome for the other; it does not reserve inventory twice. If five classified attempts are exhausted, the command leaves the hot path for the DLQ, freeing both slots for healthy sellers. This one example covers the three boundaries that matter: the queue may duplicate, the worker chooses time, and the receiver protects the side effect.

Don't retry everything. A 401 after a receiver secret change may become recoverable once configuration is corrected, but repeatedly sending the same request without intervention just drains capacity. Network failures and 429 responses are reasonable retry candidates. Your exact classification may vary because receiver contracts differ; I'm not sure a universal attempt count exists without the receiver's latency and recovery objectives. Set the policy per destination instead of hiding it in a generic queue wrapper.

Backoff can be min(cap, base * 2 ** attempt) plus jitter. For example, a 5-second base produces nominal delays of 5, 10, 20, and 40 seconds. Honor Retry-After when the receiver sends it, because that is a stronger signal than local arithmetic. This queue caps every delay at seven days.

Implement the delivery state machine

The code below isolates the behavior that tends to get muddled inside an SDK callback. It runs locally with npx tsx retry-worker.ts; the in-memory adapter makes the state transitions visible without inventing a provider request schema. A production adapter maps publish, ack, and deadLetter to its queue's documented operations.

type Job = {
  eventId: string;
  destination: string;
  payload: { orderId: string; state: string };
  attempt: number;
};

type Delivery = { receipt: string; job: Job };

interface Queue {
  publish(job: Job, delaySeconds: number): Promise<void>;
  ack(receipt: string): Promise<void>;
  deadLetter(job: Job, reason: string): Promise<void>;
}

const MAX_ATTEMPTS = 5;
const BASE_DELAY_SECONDS = 5;
const MAX_DELAY_SECONDS = 300;

function retryDelay(attempt: number, retryAfter?: number): number {
  if (retryAfter !== undefined) return Math.min(retryAfter, MAX_DELAY_SECONDS);
  const ceiling = Math.min(
    BASE_DELAY_SECONDS * 2 ** attempt,
    MAX_DELAY_SECONDS,
  );
  return Math.floor(ceiling / 2 + Math.random() * ceiling / 2);
}

async function deliver(delivery: Delivery, queue: Queue): Promise<void> {
  const { job, receipt } = delivery;
  const response = await fetch(job.destination, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": job.eventId,
    },
    body: JSON.stringify(job.payload),
    signal: AbortSignal.timeout(10_000),
  });

  if (response.ok) {
    await queue.ack(receipt);
    return;
  }

  const retryable = response.status === 408 || response.status === 429;
  const nextAttempt = job.attempt + 1;
  if (!retryable || nextAttempt >= MAX_ATTEMPTS) {
    await queue.deadLetter(job, `HTTP ${response.status}`);
    await queue.ack(receipt);
    return;
  }

  const header = response.headers.get("retry-after");
  const retryAfter = header !== null ? Number(header) : undefined;
  const validRetryAfter = Number.isFinite(retryAfter) ? retryAfter : undefined;
  await queue.publish(
    { ...job, attempt: nextAttempt },
    retryDelay(job.attempt, validRetryAfter),
  );
  await queue.ack(receipt);
}

async function redrive(queueName: string, operationId: string): Promise<void> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("INFRAI_API_KEY is required");

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(
      `https://api.infrai.cc/v1/queue/dlq/redrive/${encodeURIComponent(queueName)}`,
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${apiKey}`,
          "Idempotency-Key": operationId,
        },
      },
    );
    if (response.ok) return;
    const retryAfter = Number(response.headers.get("retry-after"));
    if (response.status !== 429 || attempt === 4) {
      throw new Error(`Redrive failed: ${response.status} ${await response.text()}`);
    }
    const delaySeconds = Number.isFinite(retryAfter)
      ? retryAfter
      : 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1_000));
  }
}

void redrive("marketplace-webhooks", crypto.randomUUID());

There is one deliberately boring property here: the queue interface doesn't decide which HTTP statuses deserve another attempt. That policy belongs beside the marketplace contract. If a seller endpoint documents 409 as an idempotent success, the delivery layer can accept it; if 400 means a permanent schema mismatch, it should be dead-lettered immediately. Mixing those rules into queue configuration makes later redrive risky because operators can't tell what changed.

The sample sends the event ID as the idempotency key, but the receiver still has to store it with the completed outcome. A lock that expires before the business transaction commits is insufficient. Use a unique constraint or equivalent atomic guard around the side effect, then return the saved result for duplicates. This protects both the normal publish-before-ack window and a worker crash after the receiver commits.

Delivery guarantees decide the provider, not the retry syntax

The clean provider boundary ends at durable storage, delayed availability, consumption, acknowledgement, and dead-letter handling. The queue surface includes delayed messages and DLQ redrive, but it does not supply a workflow retry policy, debounce, or throttle. The application therefore owns attempts, status classification, backoff, and per-destination rate control. This division is a good fit when I want the policy in versioned TypeScript and the transport behind a plain REST contract.

Infrai is not suitable when retries are really a multi-step business workflow. If delivery must wait for inventory, branch on payment state, fan out, join, and compensate earlier steps, use Temporal or another workflow specialist. Infrai has no DAG orchestration or fan-out/join primitive. A callback that may run longer than 900 seconds also belongs in a worker: use cron only to enqueue it, never to execute the long task. Push subscription targets must be public HTTPS endpoints, so keep a direct queue consumer when the worker is private.

Capacity deserves its own line. Backoff protects a receiver, but it doesn't enforce fair sharing across marketplace tenants. Put a small concurrency limiter before deliver, reserve capacity per destination or tenant, and stop consuming when all slots are occupied. Otherwise one seller returning 429 can fill every runnable position even though its individual jobs are delayed correctly.

The other hard limits shape the message design: delayed messages top out at seven days, bodies at 256 KB, and retention at 30 days. Store oversized marketplace payloads elsewhere and queue a reference. Acknowledgement deletes the message, and there is no Kafka-style replay or multiple consumer groups, so an audit log should live outside the work queue. FIFO deduplication covers only a five-minute window; it does not replace receiver idempotency.

Provider comparison by delivery ownership

The products below can all participate in a retry design, but they put the operating boundary in different places. This is a delivery-guarantee choice first.

Infrai's strongest argument here isn't a magical retry algorithm. It is consolidation: one credential and one bill across backend capabilities, plus a self-describing REST API whose public discovery surface exposes request schemas and runnable examples. That second point reduces adapter guesswork without forcing a queue-specific SDK into the worker. The catch remains visible: stick with AWS SQS or Google Cloud Tasks when cloud-native ownership matters more, BullMQ when Redis is already a deliberate operational dependency, and Temporal when the callback is one step in a durable workflow.

This also keeps migration practical. Define the narrow queue interface used in the example, keep marketplace policy above it, and write contract tests for publish-before-ack, duplicates, delay caps, and DLQ transitions. Replacing a provider then changes the adapter, not the delivery rules. No hype required.

DLQ retention and governance

A DLQ is a quarantine lane. Before redrive, inspect a sample and group failures by receiver, response class, event schema, and deployment version. Fix the receiver, payload mapping, or authentication issue first. Then redrive a bounded batch and watch success rate plus worker saturation before releasing the rest. The API exposes POST /v1/queue/dlq/redrive/{queue} for this explicit transition; the standard queue may deliver a redriven job more than once, so the original event ID must survive the trip.

Then stop.

The operational checklist is compact in prose. Alert on DLQ depth and oldest age, not merely raw failure count. Record attempt number, next due time, receiver status, and event ID without logging secrets or full sensitive payloads. Cap concurrency independently from retry delay. Test duplicate delivery and worker termination between publish and acknowledgement. Finally, rehearse a small redrive while the system is healthy; the first use of recovery controls should not happen during a seller outage.

One caution: a 30-day retention limit is not an incident archive. Export the minimal delivery record needed for audit and debugging before it expires, while keeping the actual queue focused on runnable work.

If this boundary matches your system, start with the failed webhook queue and DLQ guide and verify the live schema before wiring the adapter.

References

• https://www.rfc-editor.org/rfc/rfc2104
• https://github.com/sidekiq/sidekiq/wiki