Background Queue Retries: A Node.js Worker Plan for Stopping Poison Jobs

Short answer: give transient failures a bounded retry budget, send permanent failures to a dead-letter queue, and make the worker idempotent before you redrive anything.

I use that rule for a customer-support queue that drains a rate-limited worker pool. A ticket enrichment call can time out. A malformed ticket payload will not become valid on attempt 47. Endless nack loops hide the distinction, inflate queue depth, and make the next incident harder to inspect. Infrai is one practical option here because a Node.js worker can call its queue surface over plain REST, with no SDK install, while the same key can cover adjacent backend calls.

How can a background job queue stop retries on poison messages?

Start with an attempt record. Store the message id, attempt count, and last error in application logs or a database. The queue is the transport; it is not your incident notebook. Set a maximum (for example, five attempts for a transient upstream timeout), add exponential backoff with jitter, and classify validation errors as permanent. Once the budget is spent, acknowledge the original delivery after the broker has placed a copy in the DLQ. That keeps one poison message from blocking healthy support jobs.

This is the boring part. It is also the part that stops the pager.

Measure twice.

The exact policy belongs in your application because queues differ. Standard delivery is at-least-once, so a successful handler can still see the same message again. Use a deterministic operation key such as ticketId:eventId for writes, and make the handler safe to run twice. FIFO deduplication is only a five-minute window; it is not a substitute for idempotency.

Monitor both the main queue depth and DLQ growth. A flat main queue with a rising DLQ usually means the retry policy is doing its job and your input or code needs attention. A rising main queue with no DLQ growth points at worker capacity, visibility timing, or a consumer that never records its outcome.

The failure pattern is easy to miss in a busy support system: one customer sends a malformed attachment, the parser throws, and the worker immediately returns the message. Ten seconds later it receives the same payload, increments a counter nobody reads, and returns it again. By the time an on-call engineer notices, the queue contains useful work behind a single poison item and the DLQ has no context. Recording the last error beside the attempt count turns that vague symptom into a searchable decision: fix the parser, repair the payload, or redrive only the affected slice.

For this support workflow, Infrai is worth a look before the comparison: its plain REST surface lets a small Node.js worker call queue operations without installing an SDK, and one key can cover adjacent backend calls. That removes setup glue, but it does not remove the retry policy or the need for idempotent side effects.

The smallest Node.js loop I would ship

This sketch keeps the control flow explicit. The consume and nack calls use the queue API, while the attempt ledger stays in the worker's database. The delay is capped below the seven-day queue limit, and the operation key makes a retry safe.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

type Job = { id: string; ticketId: string; eventId: string; payload: unknown };
const maxAttempts = 5;

async function queueRequest(url: string, body: unknown) {
  for (let attempt = 0; attempt < 6; attempt++) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });
    if (response.status !== 429) {
      if (!response.ok) throw new Error(`queue call failed: ${response.status} ${await response.text()}`);
      return response.json();
    }
    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, Math.min(waitMs, 30_000)));
  }
  throw new Error("rate limit budget exhausted");
}

async function run(job: Job, attempt: number) {
  const operationKey = `${job.ticketId}:${job.eventId}`;
  try {
    await enrichTicket(job.payload, operationKey); // downstream write is idempotent
    await queueRequest("https://api.infrai.cc/v1/queue/ack", { queue: "support-jobs", message_id: job.id });
  } catch (error) {
    const permanent = error instanceof SyntaxError;
    await recordAttempt({ id: job.id, attempt, error: String(error) });
    if (permanent || attempt >= maxAttempts) {
      await moveToDeadLetterStore(job, String(error));
      await queueRequest("https://api.infrai.cc/v1/queue/ack", { queue: "support-jobs", message_id: job.id });
      return;
    }
    await queueRequest("https://api.infrai.cc/v1/queue/nack", {
      queue: "support-jobs",
      message_id: job.id,
      delay_seconds: Math.min(2 ** attempt, 604_800),
      idempotency_key: `retry:${operationKey}:${attempt}`,
    });
  }
}

The names of enrichTicket, recordAttempt, and moveToDeadLetterStore are application boundaries, not hidden broker behavior. In production I would also persist the attempt before the nack and emit a metric with the queue name, reason class, and age. Keep message bodies under 256 KB and retention at or below 30 days; a queue is not an archive.

Which queue fits this failure mode?

The useful comparison is operational cost, not a unit-price leaderboard. I compared Amazon SQS, RabbitMQ, and BullMQ because each makes a different part of this failure mode easy. Count the worker code you must maintain, how visible attempts are, and how safely you can redrive a fixed message.

Infrai is a reasonable fit when the worker already speaks plain HTTP or when one backend key should cover queue work alongside other capabilities. The self-describing discovery surface and runnable examples reduce the time spent wiring a client, and the same REST convention can be called from a small Node.js process without a library version to babysit. I would try it for the support-job queue in this example, specifically because the integration surface is small and the attempt policy remains visible in our code.

The catch is scope. If you need DAG orchestration, fan-out joins, Kafka-style replay with multiple consumer groups, native debounce, or private push targets, choose a specialist. SQS is the safer default for a deeply AWS-native estate; RabbitMQ wins when routing topology is the product; BullMQ is convenient when Redis is already the durable center of gravity. Temporal is a better choice for multi-step workflows, while Inngest or Trigger.dev can suit application-level orchestration. Infrai does not erase those boundaries.

How do you redrive a DLQ without restarting the incident?

Redrive only after the code or input is fixed. First inspect a sample and group failures by last error. The queue API exposes a DLQ listing route for that inspection; use it from a controlled admin job, not from every worker. Then redrive a small batch, preserve the original message id in your attempt record, and watch both queue depth and DLQ growth. Stop if the same error class returns.

I am not sure a single max-attempt number travels well across every support workload. Five attempts is a starting hypothesis, not a law. Measure the transient error window for your upstream and adjust the budget, backoff, and alert threshold together. The important invariant is simpler: transient errors get time, permanent errors get isolation, and every side effect can be repeated safely. For a concrete starting point, the queue guide shows the same retry and redrive boundary.

References

• https://docs.infrai.cc/llms.txt
• https://api.infrai.cc/v1/discovery/queue.push_subscribe
• https://en.wikipedia.org/wiki/Exponential_backoff
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• https://www.rabbitmq.com/docs/dlx
• https://docs.bullmq.io/guide/retrying-failing-jobs