6 Node.js Background Queue Diagnostics — Poison Retries, Attempt Caps, DLQ Redrive

Short answer: cap delivery attempts in durable state, move exhausted webhook jobs to a dead-letter queue, and make redrive a separate, bounded operation.

For an e-commerce system, that is the practical way to stop one malformed order event from consuming worker time or delivering the same outbound webhook indefinitely. The deciding constraint is latency versus cost: short retries recover transient failures quickly, while every extra attempt spends compute and increases duplicate-delivery risk.

The simple design is a worker that catches every error and puts the same job back on the queue. It looks ship-ready. It isn't. If the attempt count lives only in the Node.js process, a restart can erase it; if redrive returns a dead letter to the original policy, the retry budget can start over. A poison message then has two loops instead of one.

Treat the retry budget as data, not worker memory.

1. What should you pause before inspecting the attempt counter?

Pause redrive first. Do not erase or rewrite the suspect job; preserve its identifiers and timestamps, then reduce the active paths that can create another copy. This is incident containment, not a permanent scheduling policy. If valid order webhooks share the same queue, isolate the affected destination or delivery ID rather than stopping unrelated merchants.

One loop is enough.

The first instinct is to inspect the worker's maxAttempts value. I start one level earlier — with the producers — because an accurate counter cannot stop a second scheduler, an operator replay, or a recovery task from creating a replacement job. The simple design is a worker that catches every error and puts a fresh job back on the queue; after a restart, its local counter may also disappear. Both behaviors make a finite-looking policy run without a system-wide bound.

2. How does a Node.js background job queue replace poison message attempts?

Draw every path that can enqueue the webhook. The obvious path is the initial order event, but delayed retry, worker-crash recovery, manual replay, and DLQ redrive can each create another delivery. A maxAttempts check inside one path does not govern the others. The debugging question is therefore not “Did this worker reach six?” but “Which durable delivery identity owns the six-attempt budget, and can any path replace that identity?”

For outbound e-commerce webhooks, use one immutable delivery ID for a destination and event pair. Keep the business event ID too, because one order event may legitimately target several merchant endpoints. The receiver should be able to deduplicate the delivery ID, while the sender uses it to correlate attempts. A retry keeps the same delivery ID. An operator-approved replay gets a new replay ID linked to the original, plus its own small budget.

This distinction matters. If a worker creates a fresh job ID after each failure, a queue-level attempt counter may truthfully report 1 forever. The poison payload is not defeating the counter; the enqueue path is continually replacing the counted object. Look for changing IDs, attempt values that fall after a deploy, and timestamps that reveal two schedulers acting on the same event.

Consider a concrete trace. Order event order.paid produces delivery del_8F2 for merchant endpoint A. The first request fails, so delayed retry must retain del_8F2 and advance its durable attempt. If a recovery process instead creates job_B, then a manual action creates job_C, three queue records can all refer to the same destination and event while each presents a low local count. Grouping logs only by queue job ID hides the cycle. Group by delivery ID, business event ID, destination, replay lineage, and policy version; then sort all enqueue decisions and reservations on one timeline. No invented “global attempt” field will repair old data, but this view identifies the path that minted each new budget and tells you where to enforce the boundary.

3. Make the attempt cap atomic

The cap must survive process exits and competing workers. Reserve an attempt in the same durable operation that decides whether the job is still eligible. A read followed by a separate increment leaves a race: two workers can both observe attempt 4, both send, and both claim the fifth slot. In-memory counters are suitable only for local tests because they disappear precisely when crash recovery becomes relevant.

Here is the focused interface I would put between a TypeScript worker and its storage layer. The transaction belongs in the adapter; the worker never calculates eligibility from a stale object.

type DeliveryLease = {
  deliveryId: string;
  attempt: number;
  maxAttempts: number;
  endpoint: URL;
  payload: unknown;
};

type Completion =
  | { kind: "delivered" }
  | { kind: "retry"; runAt: Date; reason: string }
  | { kind: "dead-letter"; reason: string };

interface DeliveryStore {
  reserveAttempt(jobId: string, workerId: string): Promise<DeliveryLease | null>;
  complete(lease: DeliveryLease, result: Completion): Promise<void>;
}

reserveAttempt returns null when another worker owns the lease or the durable cap is already exhausted. complete records one terminal outcome for that reservation. The storage implementation also needs an expiration rule for abandoned leases so a terminated process does not hide a valid job forever, but expiration must not refund the attempt: the outbound request may have reached the merchant even when the worker died before recording success.

That last ambiguity is unavoidable in an ordinary HTTP delivery flow. A sender cannot infer “not delivered” from “no success recorded,” so retries may produce duplicates. Design for at-least-once delivery: send the stable delivery ID, document deduplication for receivers, and make handlers idempotent where the business action permits it. Exactly-once language would hide the actual failure mode.

4. Classify failures by the value of another attempt

Retrying every failure is how malformed payloads become expensive. Split outcomes into retryable, terminal, and ambiguous classes. A connection interruption can justify another attempt. A locally detected schema violation cannot become valid through waiting, so it should go directly to the dead-letter queue with a concise reason. An ambiguous post-send failure consumes an attempt because delivery may already have happened.

Then apply exponential backoff to retryable outcomes. Exponential backoff increases the wait between attempts, reducing repeated contention compared with a constant tight loop. Add bounded jitter so many failed deliveries do not wake at the same instant. The policy below is an example, not a universal constant; its six-attempt cap and 15-minute ceiling should be replaced with values derived from the merchant's acceptable delivery latency and the sender's retry budget.

const MAX_ATTEMPTS = 6;
const BASE_DELAY_MS = 2_000;
const MAX_DELAY_MS = 15 * 60_000;

function nextDelayMs(attempt: number, random: () => number): number {
  const exponential = BASE_DELAY_MS * 2 ** (attempt - 1);
  const capped = Math.min(exponential, MAX_DELAY_MS);
  return Math.floor(capped * (0.5 + random() * 0.5));
}

function decideNextStep(
  lease: DeliveryLease,
  failure: { retryable: boolean; reason: string },
  now: Date,
  random: () => number,
): Completion {
  if (!failure.retryable || lease.attempt >= lease.maxAttempts) {
    return { kind: "dead-letter", reason: failure.reason };
  }

  return {
    kind: "retry",
    runAt: new Date(now.getTime() + nextDelayMs(lease.attempt, random)),
    reason: failure.reason,
  };
}

Don't let the callback that schedules runAt also reset attempt. Test the boundary explicitly: attempt five may schedule attempt six, while attempt six must dead-letter. Also test a restart between reservation and completion, two workers reserving one job, a permanently invalid payload, and a receiver that accepts the request while the sender loses the response. Those tests are more useful than a happy-path throughput demo.

5. Turn DLQ redrive into change control

A dead-letter queue is quarantine, not a slower retry queue. Moving a job there should end automatic delivery. Store the payload, original delivery ID, final attempt number, failure class, first and last timestamps, and policy version needed to explain the decision. Avoid storing secrets that operators do not need.

Redrive needs an explicit gate: confirm the destination or payload problem has changed, select a bounded batch, assign a replay identity, and impose a new attempt cap that cannot recursively trigger another unlimited replay. Keep original and replay records linked. If redrive simply republishes the old payload to the same queue and the consumer initializes attempt = 0, the system has quietly converted a finite retry policy into an infinite cycle.

Do not schedule blind redrive with cron. Cron is a time-based job scheduler; time alone does not prove that a poison payload or destination has been corrected. A scheduled scan can alert an operator or prepare candidates, but release should depend on evidence such as a corrected payload version, a destination recovery check, or an approved replay batch. The catch is operational effort: manual approval adds latency and does not suit high-volume transient failures. Keep automatic bounded retry for those; reserve gated redrive for exhausted or terminal cases.

6. Price the latency budget with loop measurements

Queue depth can look stable while one delivery churns. Track attempts by delivery ID, exhausted jobs, age until success, time spent waiting versus executing, duplicate acknowledgements reported by receivers, and redrive outcomes linked to their originals. A useful alarm detects an attempt rate that rises without a matching rise in unique delivery IDs. That catches re-enqueue loops earlier than a depth threshold.

Watch cost and latency together. A longer backoff reduces active churn but delays recovery; a lower cap controls work but sends more cases to operator review. I'm not sure there is a portable “correct” cap because endpoint behavior and order urgency differ. Resolve it with a replay test set containing transient failures, invalid payloads, slow acknowledgements, and worker termination, then graph successful-delivery latency against total attempts and operator touches.

Small beats clever.

Before copying this design, measure how often failures recover on each attempt, how many deliveries are ambiguous after send, and whether receivers actually deduplicate the stable ID. Choose the smallest cap that captures the useful recovery tail, set a latency ceiling for backoff, and verify that no enqueue or redrive path can mint a fresh budget without leaving an audit link. That decision rule keeps the worker fast enough for orders without allowing a single poison message to own the bill.

References

• https://en.wikipedia.org/wiki/Exponential_backoff
• https://en.wikipedia.org/wiki/Cron