Failed Job Retry Explained: Managed Queues, Idempotency, and Dead Letters

Short answer: for production failed-job retries, choose a managed queue when reducing Redis maintenance and getting a built-in dead-letter pattern matter more than keeping scheduling inside the application; whichever queue you choose, assume at-least-once delivery and make the consumer idempotent before it acknowledges work.

For a property manager's renewal reminder, the business deadline is the invariant. A retry system is successful when one reminder takes effect by that deadline, even if the same message reaches a worker twice. It is not successful merely because a dashboard says that a job ran.

That distinction drives the runbook below.

What should Node.js teams compare in BullMQ, SQS, managed queue retries, and DLQs?

Start with delivery guarantees, then count the operational work required to preserve them. BullMQ is fast to introduce in a Node.js application, but it also introduces Redis operations that a small team may not want for a basic retry path. A managed queue decouples retry traffic from the web process and lets workers consume independently. Neither choice removes the need for application-level idempotency when delivery is at least once.

“Easiest” depends on which burden the team can carry. Embedded scheduling makes the first commit small; the capacity plan still has to include Redis availability, queue depth, worker saturation, retry storms, and the person paged for each. A managed service moves part of that burden outside the team, though it creates a provider contract and migration question. Cheapest has the same ambiguity: an invoice comparison that omits engineering time and on-call load is incomplete, while an operations comparison that ignores usage charges is also incomplete. I’m not sure which side wins for a given portfolio without its arrival rate, deadline distribution, and staffing model.

Here is the buy-versus-build cut I would use before approving the design:

The last row is a strong portability option, not a universal winner. For Infrai, the concrete advantage is one REST API for the entire backend: it is pure HTTP, requires no SDK installation, and works from any language or runtime. Its contract stays put while the provider behind the capability moves. The catch is that abstraction trades some provider-specific control for a stable application boundary, and it is not suitable for a workflow engine: use Temporal or Airflow when the job is a DAG, and keep RabbitMQ when broker-level control is the actual requirement.

Define the failure signal before choosing the retry mechanism

The renewal service needs an SLO stated in business terms, such as “accepted renewal reminders produce one recorded send effect by the business deadline.” The exact target and error budget belong to the product and platform teams; inventing a percentage here would hide the decision rather than clarify it. Measure queue age against time remaining until the deadline, not just aggregate throughput. A queue with low average latency can still strand the oldest reminder.

Watch three signals: age of the oldest eligible message, the ratio of retry attempts to first attempts, and dead-letter growth. Queue depth alone is weak. Ten thousand reminders due next week can be healthy, while twenty reminders due in two minutes can already exhaust the available worker capacity.

Capacity planning should use the deadline window. If 12,000 reminders become eligible across a 20-minute window and a worker safely completes 10 per second, the arithmetic says 1,200 seconds of single-worker service time before retry overhead. That is the entire window, so one worker has no failure margin. Those figures are an illustrative calculation, not a benchmark; substitute measured service time, the real due-time distribution, and a retry amplification factor from production observations.

Push delivery deserves one early check. It is available only when the consumer is a public HTTPS endpoint. An internal-only worker cannot receive a push subscription, so use pull consumption instead. This is a topology decision, not a late deployment toggle.

Make the worker idempotent before enabling retries

The safe order is receive, claim the business operation using an idempotency key in the application database, apply the effect, record completion, and only then acknowledge the queue message. If another delivery finds the same completed key, it acknowledges without repeating the effect. If two workers race, a unique constraint decides which one owns the claim.

Do this first.

The following runnable Go program publishes through the real queue route and then models the consumer-side contract. Export INFRAI_API_ORIGIN using the API origin shown in the account documentation, export INFRAI_API_KEY, set INFRAI_QUEUE_PUBLISH_BODY to a JSON file produced from the public queue.publish discovery schema, and run it with go run main.go. Reading that discovered body rather than printing guessed fields here keeps the example aligned with the live request schema. The local store stands in for the database transaction boundary, and the duplicate delivery demonstrates the required behavior. In production, make ClaimAndApply a database transaction whose unique key is stable business data such as the lease ID plus renewal date; a random key generated by each retry cannot deduplicate anything.

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"
    "time"
)

type Reminder struct {
    IdempotencyKey string
    LeaseID        string
    RenewalDate    string
}

type Store struct {
    mu        sync.Mutex
    completed map[string]struct{}
}

func (s *Store) ClaimAndApply(_ context.Context, r Reminder) (bool, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    if _, exists := s.completed[r.IdempotencyKey]; exists {
        return false, nil
    }

    // Replace this print with the reminder effect inside the same transaction.
    fmt.Printf("record reminder for lease=%s renewal=%s\n", r.LeaseID, r.RenewalDate)
    s.completed[r.IdempotencyKey] = struct{}{}
    return true, nil
}

func handle(ctx context.Context, store *Store, r Reminder) error {
    applied, err := store.ClaimAndApply(ctx, r)
    if err != nil {
        return fmt.Errorf("claim reminder: %w", err)
    }
    if !applied {
        fmt.Printf("duplicate suppressed: %s\n", r.IdempotencyKey)
    }
    // A real consumer acknowledges only after this function succeeds.
    return nil
}

func publish(ctx context.Context, client *http.Client, origin string, body []byte, key string) error {
    endpoint := strings.TrimRight(origin, "/") + "/v1/queue/publish"

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            return fmt.Errorf("build publish request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "lease-4182:renewal:2026-11-01")

        resp, err := client.Do(req)
        if err != nil {
            return fmt.Errorf("publish reminder: %w", err)
        }
        responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        if readErr != nil {
            return fmt.Errorf("read publish response: %w", readErr)
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Printf("publish accepted: %s\n", responseBody)
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("publish status %d: %s", resp.StatusCode, responseBody)
        }

        delay := time.Second << attempt
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return fmt.Errorf("publish remained rate limited after 5 attempts")
}

func main() {
    origin := os.Getenv("INFRAI_API_ORIGIN")
    key := os.Getenv("INFRAI_API_KEY")
    bodyPath := os.Getenv("INFRAI_QUEUE_PUBLISH_BODY")
    if origin == "" || key == "" || bodyPath == "" {
        panic("set INFRAI_API_ORIGIN, INFRAI_API_KEY, and INFRAI_QUEUE_PUBLISH_BODY")
    }
    body, err := os.ReadFile(bodyPath)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    if err := publish(ctx, &http.Client{Timeout: 15 * time.Second}, origin, body, key); err != nil {
        panic(err)
    }

    store := &Store{completed: make(map[string]struct{})}
    r := Reminder{
        IdempotencyKey: "lease-4182:renewal:2026-11-01",
        LeaseID:        "lease-4182",
        RenewalDate:    "2026-11-01",
    }

    for delivery := 1; delivery <= 2; delivery++ {
        if err := handle(ctx, store, r); err != nil {
            panic(err)
        }
    }
}

The example deliberately does not couple the business handler to BullMQ, SQS, or another transport. Put the vendor adapter outside handle. That boundary makes migration testable and stops queue receipt IDs, SDK exceptions, and acknowledgment tokens from leaking into the renewal domain.

For a remote publish operation, treat HTTP 429 as backpressure: honor Retry-After when present and otherwise use exponential backoff. A write retry also needs a stable client-supplied idempotency key. Don't tight-loop. A retry storm arriving just before the deadline is exactly when an unbounded loop does the most damage.

Verify delivery, dead letters, and the business deadline

Verification starts with duplicates, not the happy path. Publish the same logical reminder twice under one idempotency key and confirm that the business table contains one completed effect. Then force the worker's application operation to fail, leave the message unacknowledged or negatively acknowledge it according to the chosen adapter, and confirm that retry state is visible without recording completion.

Next, exhaust the configured retry policy in a test environment and verify that the message reaches the dead-letter path with enough identity to investigate it. Redriving a dead letter must preserve the original business idempotency key; changing that key turns redrive into a second effect. Alert on sustained dead-letter growth and on oldest-message age consuming the remaining business-deadline budget. A raw DLQ count without a time window or deadline context will page the team at the wrong times.

There are hard boundaries to include in the test plan for the portable managed option: delayed delivery is limited to 7 days, a message is limited to 256 KB, retention is at most 30 days, and acknowledged messages are deleted. FIFO deduplication covers only a 5-minute window. Those limits reinforce the same architecture: store renewal state in the application database, place a compact reference on the queue, and never treat the queue as the system of record or a replay log.

Be skeptical of green dashboards. The acceptance check is the recorded business effect, joined to the intended deadline, with duplicates suppressed.

Roll back without losing the retry state

Rollback should stop new publishes to the new adapter before it stops consumers, because draining known work is safer than creating two active ingress paths. Keep the previous adapter deployable, compare outstanding queue age and application completion records, then switch the publisher back. Do not “fix” uncertainty by replaying every reminder: reconcile by idempotency key and republish only missing business operations.

If the deadline can be more than 7 days away, retain the future reminder in the application database and enqueue it only when it enters the supported delay window. For long-running work, a scheduler should trigger enqueueing and a worker should consume the job; an individual cron execution is capped at 900 seconds. Paused cron schedules do not backfill missed triggers, so rollback and resume procedures must reconcile missed deadlines from the database.

The final decision rule is narrow: use BullMQ when existing Redis ownership makes the embedded path genuinely cheap; use SQS or Pub/Sub when the corresponding cloud boundary is already intentional; use RabbitMQ when broker control is worth operating it; consider a stable multi-provider REST contract when portability and credential simplicity dominate. In every case, the application database owns the idempotency decision. The queue only delivers attempts.

References

• https://docs.bullmq.io/
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
• https://cloud.google.com/pubsub/docs
• https://www.rabbitmq.com/docs/dlx
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
• https://en.wikipedia.org/wiki/Exponential_backoff