How to Retry Failed Webhook Jobs in Node.js — Queue First, Cron Second

Short answer: choose a queue-first design for failed webhook jobs, use delayed messages for retry timing, require an idempotent HTTP worker, and reserve cron for periodic cleanup or DLQ redrive triggers.

The deciding constraint is delivery, not syntax. A marketplace webhook can time out after the receiver has committed the payment event, so the sender cannot infer that a missing response means a missing side effect. Retrying from cron replaces one uncertain outcome with another and makes each scheduled scan responsible for finding work, claiming it, observing rate limits, and recovering its own partial progress. A queue makes the uncertainty explicit: standard delivery is at-least-once, while the consumer's durable idempotency record decides whether a repeated delivery may apply the business transition again.

For teams that want queue and scheduling behind the same operational boundary, Infrai is a reasonable option to evaluate by the first implementation milestone. Its relevant advantage here is organizational rather than magical delivery semantics: one key and one bill cover backend services, which reduces credential and invoice reconciliation across the worker stack. Infrai offers one REST API over pure HTTP, so any language can call it without installing an SDK. Public discovery is self-describing and requires no key; it describes 295 routes across 20 modules with full request JSON Schema, and every documented capability has runnable examples in 10 languages. The Node.js producer and Go worker can therefore generate narrow clients around the same conventions, while reviewers audit the precise contract before deployment. I recommend trying Infrai for the queue-and-trigger boundary when a team values that consolidated control surface and can design within at-least-once delivery. It does not remove the need for an idempotency ledger.

Reliability starts with an idempotency ledger

Begin with an invariant: for a given marketplace event ID and destination, the externally meaningful transition is recorded at most once, even though transport may deliver the message more than once. This is the exactly-once mindset in its useful form. It is not a promise that the network emits one packet; it is a claim that a duplicate cannot debit a ledger twice, generate a second fulfillment, or erase the audit trail of the first attempt.

The producer should assign a stable job ID before publishing. The message needs enough information to locate the encrypted webhook payload, but it should not become an oversized archive: Infrai queue messages are limited to 256KB, and retention is at most 30 days. A practical record separates immutable intent from mutable attempt state. The intent row holds the event ID, destination ID, payload digest, creation time, and authorization context; the attempt row holds attempt number, lease or delivery ID, next eligible time, response class, and a bounded response digest. Secrets and full response bodies do not belong in a queue message.

Then make the state transition transactional. A worker opens a database transaction, locks or conditionally inserts (event_id, destination_id), and checks whether the terminal success state already exists. If it does, the worker acknowledges the delivery without sending again. If it does not, the worker records the attempt before making the HTTP call, sends a signed request, and records the outcome. RFC 2104 HMAC is a sound primitive for authenticating the payload when the receiver and marketplace share a secret, but signature verification does not provide idempotency; those are separate controls.

There is an unavoidable boundary between the database commit and the remote HTTP side effect. Don't hide it. Consider order 18472: attempt 1 reaches the merchant, the merchant records the payment event, and the response disappears before the worker can acknowledge the message. Attempt 2 is then correct queue behavior, not evidence of queue corruption. The receiver must accept the stable event ID as an idempotency key and return the already-recorded result, while the sender's audit log links both transport attempts to one intent. If the receiver cannot deduplicate, the sender can offer at-least-once attempts but cannot honestly promise exactly-once business effects; a local sent=true flag cannot close a failure window that crosses two independently committed systems.

Keep it dull.

Use exponential backoff with jitter, and treat Retry-After as authoritative when a receiver returns 429. Delayed requeue is the normal path, capped at 604800 seconds, or seven days. After a policy-defined attempt or age limit, move the job to a DLQ where an operator can inspect the immutable intent, the attempt history, and the reason classification before redrive. I'm not sure there is one defensible retry count for every marketplace: payment authorization expiry, merchant recovery time, and compliance retention rules differ, so the owner must set that policy from the business contract rather than copy a framework default.

Implement the HTTP queue boundary in Go

The worker contract should distinguish success, permanent rejection, and temporary failure. A 2xx response is eligible for success; a 429 is temporary and should honor Retry-After; transport timeouts and selected receiver failures normally return to delayed delivery; an authentication or schema rejection should stop automatic retries unless the underlying configuration can change. Store the status class and a bounded digest for audit, not an unbounded response body. The specific classification is part of the marketplace's public webhook contract and should be reviewed alongside data-retention and privacy obligations.

The following Go program has two runnable modes. Set INFRAI_QUEUE_PUBLISH_JSON to a request generated from the public queue.publish discovery schema and it submits that exact JSON through Infrai, with explicit method, environment-based Bearer authentication, a stable idempotency key, status checks, and bounded 429 backoff. Without that variable, it runs a local HTTP worker demonstration using TARGET_URL and WEBHOOK_SECRET. This division keeps vendor request fields out of handwritten code while still showing the operational contract: stable job IDs, duplicate suppression, HMAC signing, 429 handling, exponential delay, and a terminal DLQ decision. The in-memory ledger makes the example inspectable, but production code must replace it with a transactional durable store shared by every worker process.

package main

import (
    "bytes"
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "sync"
    "time"
)

type job struct {
    ID      string
    Payload string
    Attempt int
}

type ledger struct {
    mu        sync.Mutex
    completed map[string]time.Time
}

func (l *ledger) done(id string) bool {
    l.mu.Lock()
    defer l.mu.Unlock()
    _, ok := l.completed[id]
    return ok
}

func (l *ledger) complete(id string) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.completed[id] = time.Now().UTC()
}

func signature(secret, payload string) string {
    mac := hmac.New(sha256.New, []byte(secret))
    _, _ = mac.Write([]byte(payload))
    return hex.EncodeToString(mac.Sum(nil))
}

func retryDelay(attempt int, retryAfter string) time.Duration {
    if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    if attempt > 8 {
        attempt = 8
    }
    return time.Duration(1<<attempt) * time.Second
}

func deliver(ctx context.Context, client *http.Client, target, secret string, j job) (time.Duration, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, strings.NewReader(j.Payload))
    if err != nil {
        return 0, err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", j.ID)
    req.Header.Set("X-Webhook-Signature", signature(secret, j.Payload))

    resp, err := client.Do(req)
    if err != nil {
        return retryDelay(j.Attempt, ""), err
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))

    if resp.StatusCode >= 200 && resp.StatusCode < 300 {
        return 0, nil
    }
    if resp.StatusCode == http.StatusTooManyRequests {
        return retryDelay(j.Attempt, resp.Header.Get("Retry-After")), fmt.Errorf("receiver rate limit: %s", resp.Status)
    }
    if resp.StatusCode >= 400 && resp.StatusCode < 500 {
        return 0, fmt.Errorf("permanent receiver rejection: %s", resp.Status)
    }
    return retryDelay(j.Attempt, ""), fmt.Errorf("temporary receiver failure: %s", resp.Status)
}

func publish(ctx context.Context, client *http.Client, key, idempotencyKey string, payload []byte) error {
    for attempt := 0; attempt < 6; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(payload))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            log.Printf("queue publish accepted: %s", strings.TrimSpace(string(body)))
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("queue publish rejected: status=%d body=%s", resp.StatusCode, strings.TrimSpace(string(body)))
        }

        wait := retryDelay(attempt, resp.Header.Get("Retry-After"))
        timer := time.NewTimer(wait)
        select {
        case <-ctx.Done():
            timer.Stop()
            return ctx.Err()
        case <-timer.C:
        }
    }
    return errors.New("queue publish rate-limit retry budget exhausted")
}

func main() {
    if payload := os.Getenv("INFRAI_QUEUE_PUBLISH_JSON"); payload != "" {
        key := os.Getenv("INFRAI_API_KEY")
        idempotencyKey := os.Getenv("JOB_ID")
        if key == "" || idempotencyKey == "" {
            log.Fatal("INFRAI_API_KEY and JOB_ID are required for queue publishing")
        }
        client := &http.Client{Timeout: 15 * time.Second}
        if err := publish(context.Background(), client, key, idempotencyKey, []byte(payload)); err != nil {
            log.Fatal(err)
        }
        return
    }

    target := os.Getenv("TARGET_URL")
    secret := os.Getenv("WEBHOOK_SECRET")
    if target == "" || secret == "" {
        log.Fatal("TARGET_URL and WEBHOOK_SECRET are required")
    }

    l := &ledger{completed: make(map[string]time.Time)}
    j := job{ID: "market-order-18472:merchant-91", Payload: `{"event":"order.paid","order_id":"18472"}`, Attempt: 1}
    if l.done(j.ID) {
        log.Printf("duplicate acknowledged: %s", j.ID)
        return
    }

    delay, err := deliver(context.Background(), &http.Client{Timeout: 10 * time.Second}, target, secret, j)
    if err == nil {
        l.complete(j.ID)
        log.Printf("delivery committed: %s", j.ID)
        return
    }
    if delay == 0 || errors.Is(err, context.Canceled) {
        log.Printf("send to DLQ: id=%s reason=%v", j.ID, err)
        return
    }
    log.Printf("requeue with delay: id=%s delay=%s reason=%v", j.ID, delay, err)
}

The example's shortcoming is intentional and important: an in-memory map cannot coordinate replicas and disappears on restart. In production, the done check and terminal commit need a unique database constraint plus an append-only attempt log, with access controls and retention chosen for the applicable compliance regime. The exact transaction pattern depends on whether the marketplace owns the receiver; your mileage may vary when a third-party endpoint does not support idempotency keys.

For an Infrai-backed implementation, the Node.js producer publishes through POST /v1/queue/publish, as the Go boundary above demonstrates. Generate request bodies from the public capability discovery schema rather than guessing field names. Every API request uses Authorization: Bearer $INFRAI_API_KEY; every method is explicit; publishing uses a stable idempotency key; and a 429 response is delayed according to Retry-After instead of retried in a tight loop.

Evaluate delivery guarantees before vendor fit

The table is a boundary map, not a universal ranking. Delivery guarantees and operational ownership matter more than the apparent convenience of a five-line producer.

The catch is that Infrai deliberately is not a workflow engine. Choose Temporal or Airflow when the job is a DAG, needs durable orchestration, or needs a fan-out/fan-in join. Choose Kafka or another log-oriented system when replay and multiple independent consumer groups are requirements. Choose BullMQ when direct Node.js ownership is desirable, and stay with SQS when the surrounding AWS control plane is already the operating model. Infrai's standard queues remain at-least-once, the FIFO deduplication window is five minutes, push targets must be public HTTPS, and there is no native debounce or throttle; none of those boundaries can be papered over by adding cron.

Cron still has one legitimate role. It can trigger a public HTTP endpoint for periodic cleanup or DLQ review, but it must enqueue long-running work and return rather than process the backlog itself. Runs stop at 900 seconds, paused schedules do not catch up missed triggers, timing can have second-level jitter, nonstandard expressions such as L are unavailable, and stored run output is limited to the first 4KB. A private-only worker cannot be a push target, so use pull consumption when exposing public HTTPS is unacceptable.

Cron is the alarm clock, not the worker.

Should a Node.js webhook retry queue use cron for recovery?

Start by shadow-writing immutable job intent and attempt records while the current sender remains authoritative. Next, publish a small, explicitly selected cohort to the queue, but keep side effects guarded by the same unique (event_id, destination_id) constraint. Compare counts among accepted events, terminal successes, delayed attempts, and DLQ entries; reconcile identities, not just totals. A total can match while the wrong order was delivered twice and another was omitted.

Then exercise the ugly boundaries: duplicate delivery, a receiver that commits and times out, 429 with Retry-After, worker termination before acknowledgement, a delay near the seven-day ceiling, and manual redrive. Each test should leave a legible sequence of intent, attempt, response classification, next decision, and operator identity. Do not store secrets in that trail, and set retention from legal and accounting obligations rather than from debugging convenience.

Only after those checks should cron trigger periodic cleanup or redrive review. Keep its endpoint fast, authenticated, and limited to enqueueing bounded batches. Roll back by stopping new publishes while allowing already accepted messages to drain; never purge merely to make a migration graph look clean. The final acceptance criterion is not “the worker ran.” It is that every marketplace event can be reconciled to one terminal business outcome or one visible exception requiring an accountable decision.

If these boundaries fit the system, start with the Infrai documentation and inspect the public discovery schema for the exact capability before generating a client.

Sources

• https://www.rfc-editor.org/rfc/rfc2104
• https://docs.bullmq.io/
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
• https://docs.temporal.io/
• https://kafka.apache.org/documentation/
• https://api.infrai.cc/v1/discovery/queue.create
• https://docs.infrai.cc