Reservation Queue Recovery for Failed User Notifications with Idempotency and DLQ Redrive

Short answer: retry failed reservation reminder notifications with an at-least-once queue, but make the consumer idempotent before adding exponential backoff; nack transient failures, ack confirmed sends, and redrive the DLQ only after checking the send ledger.

For a healthtech reservation hold, I would optimize for bounded notification delay, not the smallest possible worker bill. A reminder that arrives after the hold expires has little value. The practical design is a database-backed send ledger keyed by reminder, channel, and provider send record, with queue delivery treated as a trigger rather than proof that a notification has or has not been sent.

This is the constraint: retries recover transient failures, while the ledger prevents duplicate user notifications. You need both.

How should a Node.js queue consumer retry failed reminder notifications?

Start the transaction by claiming a stable reminder ID such as hold_8f2:expiry-warning:sms. If the ledger already says sent, acknowledge the message without contacting the notification provider. If another worker owns a live sending claim, release the delivery for a later attempt. Only the worker that acquires the claim may send. After a confirmed send, it records the provider send reference and final status before acknowledging the queue message.

There is an awkward boundary here. A process can send successfully and stop before it records that result. No queue setting can remove that gap. The cleanest option is a provider-side idempotency key when the notification provider supports one, backed by the application ledger for retries that outlive any provider or FIFO deduplication window. Without provider idempotency, an ambiguous timeout needs reconciliation against the provider's send record; blindly retrying it can duplicate a reminder.

Classify failures narrowly. Authentication errors, malformed destinations, and rejected payloads are terminal and should not burn through retries. Timeouts, connection failures, and explicit rate limits are retryable. On a retryable failure, persist the attempt and nack with a delay based on exponential backoff plus jitter. On a terminal failure, persist failed and acknowledge so the same poison message does not cycle forever.

The order matters.

A focused state-machine example

The following TypeScript is a runnable model of the consumer decision. Transport adapters for AWS SQS, Google Cloud Pub/Sub, BullMQ, or a REST queue can map ack, nack, and dead-letter to their delivery controls without changing the idempotency rule. Run it with npx tsx reminder-consumer.ts.

type Outcome =
  | { kind: "sent"; providerSendId: string }
  | { kind: "retryable"; code: string }
  | { kind: "terminal"; code: string };

type LedgerRow = {
  status: "sending" | "sent" | "failed";
  attempts: number;
  providerSendId?: string;
};

type Disposition =
  | { action: "ack"; reason: string }
  | { action: "nack"; delaySeconds: number; reason: string }
  | { action: "dead-letter"; reason: string };

const ledger = new Map<string, LedgerRow>();
const maxAttempts = 5;

function retryDelaySeconds(attempt: number, random = Math.random): number {
  const cap = 300;
  const base = Math.min(cap, 2 ** attempt);
  return Math.max(1, Math.floor(base / 2 + random() * base / 2));
}

function consume(
  reminderId: string,
  outcome: Outcome,
  random = Math.random
): Disposition {
  const current = ledger.get(reminderId);
  if (current?.status === "sent") {
    return { action: "ack", reason: "already sent" };
  }

  const attempts = (current?.attempts ?? 0) + 1;
  ledger.set(reminderId, { status: "sending", attempts });

  if (outcome.kind === "sent") {
    ledger.set(reminderId, {
      status: "sent",
      attempts,
      providerSendId: outcome.providerSendId
    });
    return { action: "ack", reason: "send confirmed" };
  }

  if (outcome.kind === "terminal") {
    ledger.set(reminderId, { status: "failed", attempts });
    return { action: "ack", reason: `terminal ${outcome.code}` };
  }

  if (attempts >= maxAttempts) {
    return { action: "dead-letter", reason: outcome.code };
  }

  return {
    action: "nack",
    delaySeconds: retryDelaySeconds(attempts, random),
    reason: outcome.code
  };
}

const id = "hold_8f2:expiry-warning:sms";
console.log(consume(id, { kind: "retryable", code: "RATE_LIMITED" }, () => 0));
console.log(consume(id, { kind: "sent", providerSendId: "send_1042" }));
console.log(consume(id, { kind: "sent", providerSendId: "send_1043" }));

The first delivery is nacked with a deterministic one-second delay in this example, the second is acknowledged after recording send_1042, and the third is acknowledged without replacing that send record. In production, acquire the ledger claim with a unique constraint or compare-and-set transaction. A plain SELECT followed by INSERT is racy when two consumers receive the same reminder together.

Five attempts and a 300-second cap are example policy values, not measured optima. I'm not sure there is one universal retry schedule: provider rate-limit guidance, the reservation hold window, and p95 send latency should decide it. Your mileage may vary.

For an Infrai queue, the following runnable TypeScript sends a caller-supplied, schema-validated nack payload to the verified route. Keeping the payload in an environment variable avoids inventing fields; obtain its current JSON Schema from public discovery. The wrapper uses an explicit method, checks every response, retries HTTP 429 with Retry-After or exponential backoff, and supplies an idempotency key.

const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.QUEUE_API_BASE_URL;
const rawPayload = process.env.QUEUE_NACK_PAYLOAD;

if (!apiKey || !baseUrl || !rawPayload) {
  throw new Error(
    "Set INFRAI_API_KEY, QUEUE_API_BASE_URL, and QUEUE_NACK_PAYLOAD"
  );
}

const payload: unknown = JSON.parse(rawPayload);
const idempotencyKey = crypto.randomUUID();

for (let attempt = 0; attempt < 5; attempt += 1) {
  const response = await fetch(`${baseUrl}/queue/nack`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json",
      "Idempotency-Key": idempotencyKey
    },
    body: JSON.stringify(payload)
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("Retry-After"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    continue;
  }

  if (!response.ok) {
    throw new Error(`Queue nack rejected: ${response.status} ${await response.text()}`);
  }

  console.log(await response.json());
  break;
}

DLQ redrive is an operation, not another retry loop

A dead-letter queue needs ownership. Record the reminder ID, attempt count, last failure class, first and last attempt times, and final state in the application database so support can explain a missed reminder without reconstructing it from short-lived queue data. Before redrive, separate a resolved transient incident from permanently invalid destinations. Redrive the first group in a bounded batch; close or repair the second group before it can send again.

Don't redrive the whole DLQ on a timer. That turns a useful quarantine into an expensive retry loop and can push stale reservation notices after the hold has expired. The worker should compare the current time with holdExpiresAt before every send. If the hold is already expired, mark the reminder expired and acknowledge it rather than notifying the user. This is where latency beats theoretical delivery completeness: a late message can be technically delivered and still be wrong for the product.

A useful alert combines DLQ depth with the age of the oldest message. Depth catches volume; age catches a single reminder that nobody owns. I would also watch attempts per successful send, duplicate suppressions, terminal-failure rate, and end-to-end time from reminder eligibility to provider confirmation. Those measurements tell you whether backoff is protecting the provider or merely hiding slow recovery.

Choosing a queue by latency, cost, and control

The queue is replaceable if the consumer contract remains claim, send, record, ack or nack. The operational differences still matter.

Infrai is a strong fit when a solo team values a plain HTTP contract over provider-specific SDK work. One key and one bill cover a broad capability surface, while public discovery exposes request schemas and runnable examples without requiring a key; that shortens the work of validating the nack adapter above. The contract stays stable if the underlying queue vendor changes.

The catch is that Infrai has no topic fan-out or Kafka-style replay and multiple consumer groups, so stick with Pub/Sub for native one-to-many distribution and choose a log platform such as Kafka when replay is the central requirement. Use Temporal rather than a queue when the reservation process needs durable multi-step orchestration, joins, or compensation. Its push subscriptions require a public HTTPS target, so it is not suitable for a private-only worker endpoint. A cron task calls a public HTTP URL and has a 900-second execution limit; long work should use cron to enqueue and a worker to consume.

What to measure before copying this design

Set the retry policy from the reservation clock. Measure notification eligibility-to-confirmation latency at p50, p95, and p99; the fraction recovered on each attempt; provider rate-limit responses; DLQ age; duplicate suppressions; and worker time per successful send. Then compare the value of another attempt with the chance that the hold expires first.

Cost belongs in that same decision, but it is broader than queue requests. Count database claim transactions, idle polling, notification-provider calls, support investigation, and engineering time spent maintaining provider-specific adapters. A low request price can lose if aggressive polling dominates a quiet workload. A managed push path can reduce idle work, while a controlled pull consumer can make concurrency and backpressure easier to reason about.

Ship the ledger and expiry guard first. Add sophisticated backoff only after the basic metrics show where failures cluster — and keep the DLQ review process boring, explicit, and owned.

References

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
• https://cloud.google.com/pubsub/docs/overview