Node.js Job Workers: Recovering a Background Queue After At-Least-Once Delivery

Short answer: A Node.js background job queue should treat at-least-once delivery as normal: give each job a stable idempotency key, commit its local effect with a durable receipt, and acknowledge only after that commit. The deciding constraint is the gap between a side effect finishing and the consumer telling the queue it finished; a process can disappear in that gap, so a retry is the correct outcome even though the work may already be visible.

That sounds fussy until a job sends a notification, grants a credit, or starts an expensive model call twice. Then it is just accounting.

The tempting fix is to keep a Set of processed job IDs in the worker. It looks clean in a local test and disappears on deploy, restart, or a second consumer. Lowering retries has the same flaw: it trades duplicate effects for lost work. I would rather make a repeated delivery cheap and truthful than pretend it cannot occur.

What causes duplicate jobs in an at-least-once Node.js queue consumer?

The most useful mental model is that the queue owns delivery and the application owns the business effect. A consumer can update its database, lose its process before sending an acknowledgement, and later receive the same message again. The broker has no way to inspect the database commit, so redelivery follows directly from at-least-once semantics.

There are less obvious versions of the same timing problem. A worker can run beyond its visibility lease while another consumer receives the message. A network call can reach a remote service while its response is lost. A retry policy inside an HTTP client can overlap with the queue's retry policy. None of those cases proves that the original business action did not happen.

Short leases make this easier to see.

No magic setting fixes it.

Dead-letter queues are useful for isolating messages that have failed repeatedly, not for making earlier side effects disappear. AWS documents a design trade-off for FIFO queues: attaching a dead-letter queue can break the exact ordering of messages, so workflows that depend on order need an explicit redrive plan rather than an automatic escape hatch.

How should a Node.js queue consumer process duplicate jobs with retries?

Start with an identifier for the business action, not the delivery. invoice:9821:issue-credit is stable across retries and replays; a UUID created inside the consumer is not. Put that identifier in the message when the work is created, preserve it during redrive, and use it at the boundary that owns the durable change.

For an effect contained in one relational database, a unique receipt plus the state update in one transaction is a compact pattern. The receipt table needs a unique constraint on idempotency_key. In this example, the Database interface stands in for the transaction API of the database client already used by the application.

type CreditJob = {
  accountId: string;
  credits: number;
  idempotencyKey: string;
};

type Database = {
  transaction<T>(work: (tx: {
    query(sql: string, values: unknown[]): Promise<{ rowCount: number }>;
  }) => Promise<T>): Promise<T>;
};

export async function applyCredit(
  db: Database,
  job: CreditJob,
): Promise<"applied" | "duplicate"> {
  return db.transaction(async (tx) => {
    const receipt = await tx.query(
      `INSERT INTO job_receipts (idempotency_key)
       VALUES ($1)
       ON CONFLICT (idempotency_key) DO NOTHING`,
      [job.idempotencyKey],
    );

    if (receipt.rowCount === 0) return "duplicate";

    await tx.query(
      `UPDATE accounts
       SET credits = credits + $1
       WHERE id = $2`,
      [job.credits, job.accountId],
    );

    return "applied";
  });
}

The important detail is transactional placement. Imagine two consumers receiving the same business action after a lease expiry. Both call the handler. The first INSERT wins because the unique constraint is the arbiter, not a preliminary SELECT that both consumers can observe as empty. The other transaction waits for the conflict to resolve and then returns duplicate after the first transaction commits. If the first transaction rolls back before the account update becomes durable, its receipt rolls back too, so a later delivery is allowed to do the work. If the receipt commits before the account update, a crash can leave an unfinished operation marked as done. If the update commits before the receipt, a redelivery can repeat it. A single database transaction makes the receipt and the local state transition commit together or roll back together; the caller can acknowledge the queue message after applyCredit resolves, including the duplicate result. That is why an in-memory cache isn't enough: it cannot arbitrate between processes or survive a restart.

Do not acknowledge on receipt. Acknowledgement is the queue's permission to stop trying, so it belongs after the handler has reached a durable outcome. For retryable errors, leave the message eligible for the queue's retry policy. For invalid payloads or exhausted attempts, move the message to a failure lane and retain the original idempotency key for inspection and redrive.

Put the guarantee next to the effect

Queue-level de-duplication can reduce traffic, but it cannot prove that an email, payment, database mutation, or external request happened once. The system that records the business state is the place that can enforce a unique invariant. This distinction matters when a worker fleet changes size, moves hosts, or handles manual replays.

External effects need a different shape. Holding a database transaction open during a network request creates contention and still does not create a shared commit. Write the local change and an outbox record atomically, then deliver the outbox record separately with the same key when the destination supports one. If the destination has no idempotency contract, describe the recovery path honestly: reconciliation can discover mismatches, and compensation can reverse a business action when reversal exists.

The catch is retention. Deleting receipts too early lets a delayed redelivery repeat an old action; keeping them forever adds index and storage cost. There is no universal number. Choose a window from the maximum delivery delay, replay policy, audit needs, and the harm caused by a duplicate, then measure receipt-table growth before turning retention into a background cleanup job.

This approach is not suitable when several systems require one indivisible commit and none exposes a shared transaction or an idempotency key. Use a business-level reconciliation or compensation design there. Calling that arrangement “exactly once” would hide the part operators need to understand.

Which retry rules keep a queue from multiplying work?

Assign retry timing to one layer. If the queue schedules exponential backoff, the HTTP client should return a structured failure rather than independently scheduling the same operation. HTTP 429 Too Many Requests signals rate limiting, and MDN notes that a response may include Retry-After; a consumer can use that value as input to its selected retry policy. A malformed message is a different class of failure and should not spend attempts that are meant for transient conditions.

Keep the event record useful under pressure. Log the business key, delivery identifier, attempt number, start and finish time, outcome, and any requested retry delay as separate fields. Then compare deliveries with unique keys and committed effects. A rise in duplicate outcomes with flat committed effects means the guard is absorbing redelivery; a rise in both points to an unprotected effect.

Test the timing, not just the happy path. Run two consumers concurrently with the same key. Simulate completion of the database transaction followed by loss of the acknowledgement, then redeliver the message. Run a handler long enough to overlap its lease. Finally, send a rate-limited response through the retry boundary and verify that exactly one scheduler creates the next attempt. These tests cost less than debugging an inflated bill or a duplicated customer action after the fact.

Before adopting this pattern, measure the longest realistic replay interval, contention on the receipt index, queue age, failure-lane volume, and how the destination explains an ambiguous response. Those measurements tell you whether the issue is duplicate processing, capacity, or a retry policy that has become two policies.

References

• AWS SQS dead-letter queues documentation
• MDN: HTTP 429 Too Many Requests