Background Job Queue Triage: Stopping Poison Retries Before Dead Letter Redrive

Short answer: Stop retrying a background job after a fixed attempt budget, quarantine the poison message in a dead letter queue, and redrive it only through the same idempotent execution path used for first delivery. For a rate-limited developer-tools worker pool, that rule prevents one malformed indexing request from consuming capacity forever while transient throttling still gets time to clear.

The deciding constraint is not the queue's retry feature. It is whether the system can distinguish a request that may succeed later from one that cannot succeed without changing its input. A retry counter without that classification merely delays the incident.

I've been paged for both sides of this failure: work that never ran and work delivered twice. The runbook lesson is blunt. Treat delivery as at-least-once, make the effect idempotent, and give every replay a bounded budget.

How can a Node.js worker contain background queue poison retries?

Start with three outcomes, even if the queue library exposes a dozen states. A worker completes the job, schedules a transient failure for another attempt, or quarantines a permanent failure. A missing repository identifier is permanent. A rate-limit response is transient. An ambiguous timeout is transient only if repeating the operation cannot duplicate its effect.

That last case is where tidy retry diagrams meet production. Suppose a developer-tools platform drains indexing jobs against an upstream API with a strict request limit. The worker sends an index request, loses the response, and cannot tell whether the upstream accepted it. Retrying with a fresh operation ID may create two index builds. Dropping the job may create none. The stable idempotency key, derived from the workspace and requested revision, makes the second delivery refer to the same logical operation rather than a new one.

Keep the retry budget in durable job metadata. Do not infer it from process memory, worker restarts, or log lines. Increment it before scheduling the next delivery, and copy the final value into the quarantine record. A redrive gets its own bounded redrive_count; it must not silently erase the history that exhausted the original budget.

Here is a concrete policy for one queue. The numbers are examples to tune with load tests and upstream limits, not universal defaults.

This boundary matters more than the exact number six. If every error is called transient, a poison message is immortal. If every ambiguous result is called permanent, brief dependency trouble becomes lost work.

Govern the retry ledger before writing code

The queue should transport intent; the worker should enforce the business invariant. For an indexing job, define a stable key such as workspace_id:revision, claim that key in a durable store, and commit the index result under the same logical identity. A duplicate delivery can then observe done and acknowledge without repeating the effect.

The Go sketch below shows the control flow behind a Node.js queue consumer. It is deliberately independent of a queue client: the Node.js process can pass the same fields over a generic worker boundary, and the safety decisions remain testable without library-specific retry behavior.

package worker

import (
    "context"
    "errors"
    "fmt"
    "math/rand"
    "time"
)

const maxAttempts = 6

type Job struct {
    ID             string
    WorkspaceID    string
    Revision       string
    Attempt        int
    RedriveCount   int
    IdempotencyKey string
}

type Store interface {
    IsDone(context.Context, string) (bool, error)
    MarkDone(context.Context, string) error
}

type Queue interface {
    Retry(context.Context, Job, time.Duration) error
    Quarantine(context.Context, Job, string) error
    Ack(context.Context, string) error
}

type Indexer interface {
    Build(context.Context, string, string, string) error
}

var (
    ErrRateLimited = errors.New("rate limited")
    ErrBadInput    = errors.New("invalid input")
)

func Handle(ctx context.Context, job Job, store Store, queue Queue, indexer Indexer) error {
    if job.WorkspaceID == "" || job.Revision == "" {
        return queue.Quarantine(ctx, job, "permanent: missing job identity")
    }

    done, err := store.IsDone(ctx, job.IdempotencyKey)
    if err != nil {
        return err
    }
    if done {
        return queue.Ack(ctx, job.ID)
    }

    err = indexer.Build(ctx, job.WorkspaceID, job.Revision, job.IdempotencyKey)
    switch {
    case err == nil:
        if err := store.MarkDone(ctx, job.IdempotencyKey); err != nil {
            return err
        }
        return queue.Ack(ctx, job.ID)
    case errors.Is(err, ErrBadInput):
        return queue.Quarantine(ctx, job, "permanent: rejected input")
    case errors.Is(err, ErrRateLimited) && job.Attempt+1 < maxAttempts:
        job.Attempt++
        return queue.Retry(ctx, job, retryDelay(job.Attempt))
    default:
        return queue.Quarantine(ctx, job, fmt.Sprintf("attempt budget exhausted: %v", err))
    }
}

func retryDelay(attempt int) time.Duration {
    base := time.Second * time.Duration(1<<min(attempt, 7))
    jitter := time.Duration(rand.Int63n(int64(base / 4)))
    return base + jitter
}

There is a deliberate gap in this sketch: MarkDone and the external index operation are not one transaction. If the upstream accepts the operation and the worker exits before recording done, the delivery returns. The idempotency key must therefore be honored at the effect boundary, not merely checked in a local table before calling a non-idempotent dependency. If that dependency cannot deduplicate, use an outbox or another transactional handoff that owns the effect. Don't label a read-before-write check as exactly-once delivery; it isn't.

Backoff reduces synchronized pressure by spacing retries, while jitter keeps a group of failed jobs from returning at the same instant. Cap the delay so recovery does not drift beyond the service objective. I'm not sure there is a defensible universal cap: the right value depends on the upstream reset window, queue depth, and how stale an index may become. Those three measurements resolve the choice.

Implement an idempotent effect boundary

First, pause automatic redrive rather than the whole queue. Preserve healthy throughput while the on-call engineer samples quarantined records and answers four questions: Is the attempt field increasing? Does the failure classification remain the same? Is the idempotency key stable across deliveries? Is a scheduler inserting a new job that only looks like a retry? Follow one logical indexing request across enqueue, lease, failure, delay, and the next lease; line up timestamps and confirm that its persisted attempt changes exactly once. Then group the surrounding records by workspace, revision, and failure class. If delivery IDs keep changing while the logical key stays fixed and each record starts at attempt zero, the retry mechanism is not looping at all: the producer is creating new jobs. If one delivery lineage advances past the configured maximum, inspect which component owns the counter and whether a rejection is returning the job to the queue without recording the attempt. This reconstruction takes longer than toggling a retry setting, but it tells the operator which state transition is broken and preserves the evidence needed after recovery.

The scheduler question is easy to miss. Cron starts work on a time schedule; it does not know that yesterday's logical job is still retrying. If a periodic producer emits the same indexing request every minute with a random job ID, the dashboard can resemble an endless retry even when each individual message reaches its maximum. Correlate by logical key and revision, not only by delivery ID.

Watch rates, not anecdotes — enqueue, start, success, retry, quarantine, redrive, and duplicate-suppressed counts should reconcile over a selected window. Also record attempt age, queue age, failure class, and upstream throttle responses. A growing oldest-job age with flat worker utilization suggests a dispatch or lease problem; high utilization dominated by one failure class points back to retry policy. These are diagnostic interpretations, so confirm them against traces and queue state before changing production.

One alert deserves its own sentence.

Page when retries consume enough worker capacity to threaten healthy jobs, not for every isolated failure. A poison message belongs in a bounded review queue; it should not become an on-call metronome.

Test redrive as a production change

Redrive is a controlled production change. Select records by an explicit failure class and time range, inspect the stored payload, retain the original idempotency key, and release a small batch below the rate-limited pool's spare capacity. Do not bulk-copy an entire dead-letter queue into the live queue and hope the backoff policy absorbs it.

Before release, test four cases: a valid first delivery, an immediate permanent rejection, a transient failure that later succeeds, and a duplicate arriving after success. Then inject a worker exit between the external call and local completion recording. The expected result is one logical index build, a later delivery acknowledged as duplicate, and no reset of the original attempt history.

During production verification, compare the redriven cohort's completion, quarantine, and duplicate-suppression counts with its selected total. Stop the batch if queue age rises beyond the rollback threshold, upstream throttling accelerates, or the failure class has not changed. Rollback means disabling further redrive and leaving remaining records quarantined; it does not mean deleting evidence or resetting counters.

The catch is operational complexity. A durable idempotency ledger and controlled replay path are not suitable when every job is a pure, cheap, deterministic calculation with no external effect; a bounded retry followed by discard may be enough there. At the other extreme, stick with a transactional outbox and an effect consumer when the database write and job publication must move together. A queue-level deduplication flag cannot replace that transaction boundary.

Ship the policy with a runbook owner, a maximum delivery count, a maximum redrive count, and a tested pause control. Then a stuck queue is an observable state transition, not a mystery loop.

References

• https://en.wikipedia.org/wiki/Cron
• https://en.wikipedia.org/wiki/Exponential_backoff