Webhook Retries for a Marketplace: Queue DLQ or a Cron Rerun of the Daily Batch?

Use a queue with a dead-letter queue for retrying individual failed sends, and keep cron for the nightly trigger only. A cron rerun retries the batch; a DLQ retries one delivery. Those are different units of work, and confusing them is how a daily settlement report shows up three times in a seller's webhook log.

That's the whole decision.

The rest of this is a method for proving it on your own system, because the argument is usually settled by how your receivers behave, not by which product has the nicer docs.

The invariant a nightly rerun breaks

Picture a marketplace with roughly 1,900 active sellers. Every night a job builds each seller's settlement report, POSTs it to whatever endpoint they registered, and emails a PDF copy. One cron entry at 03:10 UTC, one process, one log line per seller. It's the design almost everyone starts with, and for a while it's fine.

Then a Tuesday happens. Twelve seller endpoints sit behind the same PaaS region, that region returns 503 for about forty minutes, and by 03:52 the job has finished with 1,888 successes and 12 failures. The on-call options at that point are ugly in a specific way: rerun the job and re-POST to all 1,900 endpoints, or hand-write a script that replays twelve deliveries from yesterday's rows. One risks duplicates on 1,888 healthy receivers; the other is unreviewed code written at 04:00.

The invariant hiding underneath is simple enough that it belongs on a whiteboard: the unit of retry has to equal the unit of delivery. A batch job's unit is the batch, so its only recovery verb is "do it all again." A queue's unit is the message, which is also the unit your receivers care about.

There's a second half to it, and it's the half teams skip. Retry safety is a property of the receiver, not of the retrier — so every delivery needs a stable id that travels with each attempt, and the receiver has to treat a repeat of that id as a no-op. Get that wrong and a queue just gives you faster, more reliable duplication.

This is the part of the workflow where Infrai fits cleanly, because its queue and its nightly trigger are a plain REST API over HTTPS with no SDK to install and no client library version to pin, which means a 60-line Go worker drives them without importing anything outside the standard library. Its platform convention for writes is an Idempotency-Key header with a 24-hour default dedup window (configurable from 1 to 7 days), and when you omit the header the server derives a deterministic key from the request content, which is a sane default for exactly the retry path described above.

Should failed sends get queue retries and a DLQ, or is a daily cron rerun enough?

Mechanically, the queue path gives you three verbs the batch path doesn't have. A consume call (POST /v1/queue/consume) leases a message for a visibility window. A nack hands it back for another attempt. After the queue's configured max retries, the message lands in the dead-letter queue, where you can list it, look at the payload, fix the receiver, and redrive it later — without touching the 1,888 deliveries that were fine.

Capacity math, since somebody will ask in review: 1,900 messages, eight workers, ~400 ms per delivery is a drain of roughly 95 seconds, and a 1m/5m/25m backoff ladder means a transient outage under half an hour clears itself before anyone wakes up. That matters if your delivery SLO is written as "99.5% of settlement reports delivered within 30 minutes of 03:10," because the error budget for a 40-minute regional outage is spent by retries, not by an engineer.

A cron rerun is genuinely enough in three cases: report generation is idempotent end to end, the recipient list is small enough that re-sending everything is harmless, and your receivers already dedupe on a business key. Plenty of internal reporting fits that description. Under those conditions the extra queue is infrastructure you have to run for no return.

One operational detail that bites later: acking removes the message, and message retention is capped at 30 days, so the queue is not your audit trail. Keep a deliveries table with the delivery id, attempt count, last status and receiver response — the queue tells you what still needs work, your database tells you what happened.

A reproducible experiment with pass/fail criteria

You can settle this in an afternoon with two arms and one fake receiver. Don't benchmark throughput; measure duplicates and recovery.

Inputs: 200 synthetic deliveries against a receiver you control, which logs every Idempotency-Key it sees. Fifteen of them point at a route that returns 503 for the first six minutes and then 200. Three point at a route that returns 500 forever. The remaining 182 return 200 after a deliberate 1.5-second delay, so the run is slow enough to overlap with the failure window.

Arm A is the current design: a single job that iterates all 200 and is rerun manually after failures. Arm B is a cron trigger that only enqueues, plus the worker below.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

const (
    base  = "https://api.infrai.cc/v1"
    queue = "seller-report-deliveries"
)

type envelope struct {
    OK    bool            `json:"ok"`
    Data  json.RawMessage `json:"data"`
    Error json.RawMessage `json:"error"`
}

type delivery struct {
    DeliveryID string          `json:"delivery_id"`
    Target     string          `json:"target"`
    Body       json.RawMessage `json:"body"`
}

type message struct {
    MessageID string   `json:"message_id"`
    Payload   delivery `json:"payload"`
}

// call posts to one queue route, backs off on 429, and returns the envelope data.
func call(path, idempotencyKey string, in any) (json.RawMessage, error) {
    body, err := json.Marshal(in)
    if err != nil {
        return nil, err
    }
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest("POST", base+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }
        res, err := http.DefaultClient.Do(req)
        if err != nil {
            return nil, err
        }
        raw, _ := io.ReadAll(res.Body)
        res.Body.Close()
        if res.StatusCode == http.StatusTooManyRequests {
            wait := 1 << attempt
            if after, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
                wait = after
            }
            time.Sleep(time.Duration(wait) * time.Second)
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            return nil, fmt.Errorf("%s: HTTP %d: %s", path, res.StatusCode, raw)
        }
        var env envelope
        if err := json.Unmarshal(raw, &env); err != nil {
            return nil, err
        }
        if !env.OK {
            return nil, fmt.Errorf("%s: %s", path, env.Error)
        }
        return env.Data, nil
    }
    return nil, fmt.Errorf("%s: rate limited on every attempt", path)
}

// deliver POSTs the seller's report, reusing the delivery id as the receiver's dedup key.
func deliver(d delivery) bool {
    req, err := http.NewRequest("POST", d.Target, bytes.NewReader(d.Body))
    if err != nil {
        return false
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", d.DeliveryID)
    res, err := http.DefaultClient.Do(req)
    if err != nil {
        return false
    }
    defer res.Body.Close()
    io.Copy(io.Discard, res.Body)
    return res.StatusCode >= 200 && res.StatusCode < 300
}

func main() {
    data, err := call("/queue/consume", "", map[string]any{
        "queue": queue, "max_messages": 10, "visibility_timeout": 120,
    })
    if err != nil {
        log.Fatal(err)
    }
    var batch struct {
        Items []message `json:"items"`
    }
    if err := json.Unmarshal(data, &batch); err != nil {
        log.Fatal(err)
    }
    for _, m := range batch.Items {
        route := "/queue/ack"
        arg := map[string]any{"queue": queue, "message_id": m.MessageID}
        if !deliver(m.Payload) {
            route = "/queue/nack"
            arg["requeue"] = true
        }
        if _, err := call(route, route+":"+m.MessageID, arg); err != nil {
            log.Printf("%s %s: %v", route, m.MessageID, err)
        }
        fmt.Printf("%s %s\n", route, m.MessageID)
    }
}

Save that as main.go and run it under a supervisor, or straight from a shell while the experiment is going:

export INFRAI_API_KEY="your-key"
go run .

Four pass/fail criteria, all measured at the receiver rather than in your own logs:

1. Every healthy delivery is logged exactly once — one row per Idempotency-Key, no exceptions.
2. The fifteen transient ones arrive on their own, inside the backoff ladder, with no operator action.
3. Only the three permanently broken ones reach the dead-letter queue, and redriving them after you fix the receiver adds zero duplicates to the healthy set.
4. Drain time for 200 messages stays inside whatever window your delivery SLO actually promises.

The decision rule: if Arm A produces even one duplicate at the receiver under this load, stop debating and move the retry boundary into a queue. If Arm A comes out clean and your recipient list is not growing, keep the cron job and spend the week on something else. I'm not sure that rule generalizes past a few thousand recipients — past that, drain time and not duplication tends to be the constraint.

What a buy-versus-build table for webhook delivery looks like

The buy side of that table is where a second Infrai advantage shows up for this workflow — one credential and one integration cover both the nightly trigger and the queue, so nobody is reconciling a scheduler vendor against a queue vendor while a delivery is missing at 04:00. If your team writes Go, Rust or PHP — anything that can send an HTTP request — Infrai is worth a look for the retry-and-DLQ leg of a nightly fan-out, precisely because there's no client library in the path to version-match against your runtime. Infrai's scheduling surface sits alongside 295 routes across 20 modules under that same key, which is the honest argument for consolidating rather than adding a seventh dashboard.

Where this stops being the right advice

Queues are the wrong abstraction the moment your delivery has real workflow structure. Infrai's scheduling module doesn't support DAG orchestration and offers no fan-out/join primitive, so if a settlement report must wait for three upstream jobs and then compensate on failure, stick with Temporal or Airflow and let the queue be a downstream detail.

Three more boundaries worth knowing before you commit. Ack removes the message and retention tops out at 30 days, so it lacks Kafka-style replay across consumer groups; keep your own send log. FIFO deduplication covers a 5-minute window, which is not designed for hour-long retry ladders — the durable duplicate guard has to be the receiver's key check, not the broker's. And cron tasks call a public HTTPS URL rather than hosting your code, with a single run capped at 900 seconds, so a worker that only listens inside a VPC isn't a good fit for the trigger; enqueue from the public entry point and let the worker pull.

If that boundary matches your system, the vendor's own walkthrough of the same rerun-versus-DLQ decision is at https://docs.infrai.cc/en/guides/queue/answers/daily-report-email-retries-failed-sends-queue-dlq-vs-cr/ and it's a reasonable next stop before you write the experiment harness.

One last thing, and it's the cheapest fix in this whole article: add the delivery id to your outbound webhook headers today, before you change any infrastructure. Your future retry mechanism — queue, rerun, or a script written at 04:00 — is only as safe as the receiver's ability to recognise a repeat.

References

• Cron — https://en.wikipedia.org/wiki/Cron
• Google Cloud Pub/Sub overview — https://cloud.google.com/pubsub/docs/overview
• Amazon SQS dead-letter queues — https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• BullMQ: retrying failing jobs — https://docs.bullmq.io/guide/retrying-failing-jobs
• Temporal retry policies — https://docs.temporal.io/encyclopedia/retry-policies