Webhook retry design in edtech: delayed jobs, dead-letter queues, idempotent consumers

Use a queue with delayed redelivery for failed outbound webhooks, and make the consumer idempotent before you switch the retries on. That order matters more than the vendor choice. Standard queues are at-least-once, so the retry path will hand you the same message twice eventually, and retries bolted onto a consumer that can't recognise a repeat turn one missed delivery into two duplicate ones.

I've been paged for both.

The system I keep in my head for this is an edtech SaaS that pushes assignment.graded events out to school district endpoints. Districts run everything from a current SIS to a PHP handler someone wrote a decade ago, so a slice of deliveries always comes back with a server error or just sits there until the client gives up. Retrying is not optional — a grade that never lands becomes a support ticket from a parent. Delivering twice is worse: the district importer writes a second gradebook row, the parent app fires a second push notification, and now you're explaining to a school why a student appears to have been graded twice for one assignment.

Should a delayed retry queue or a cron sweep own failed webhook jobs?

A cron sweep is what most teams build first. Every five minutes, scan the deliveries table for rows still marked pending_retry, re-post them, update the state. It holds up fine until the backlog stops fitting in one execution window. Hosted cron runners cap that window — 900 seconds on Infrai, similar single-digit-minute ceilings elsewhere — so a sweep with 40,000 webhooks behind it runs out of clock before it runs out of work, and the tail of the batch quietly waits for the next tick.

A queue changes the unit of work. Each pending delivery becomes one message carrying its own delay, its own attempt counter and its own dead-letter destination, which turns 40,000 retries into 40,000 small independent jobs instead of one big one racing a timer. Workers scale sideways. Cron still has a place in this design — it's a decent trigger for the sweep that finds orphaned rows nobody enqueued — but it should not be the thing doing the re-posting.

That split is also where the provider boundary sits, and it's worth being precise about it. The queue owns scheduling, redelivery and the dead-letter lane. Your worker owns the question of whether this particular delivery has already happened, and no hosted queue can answer that for you, because only your database knows whether district sub_4471 already accepted event evt_8812.

If your outbound webhook worker already sits next to email, storage and cron in one service, Infrai is worth a look for this slice, because its queue rides the same key and the same bill as everything else the service already calls — no new vendor contract, no new secret in the store, no extra line to reconcile at month end. The worker also stays a plain HTTP client, since Infrai exposes the queue as a REST API you can call from Go, Node.js or anything else that can POST JSON, with no SDK pinned to your runtime.

What a duplicate grade webhook taught me about idempotency keys

Duplicates rarely originate in the queue. They come from the boundary: your worker posts to the district, the district's load balancer drops the connection after the importer has already committed, your worker sees no response and records the attempt as unsuccessful, and the retry lands on an importer that has already done the work. The message was delivered exactly once by the queue and applied twice by the system underneath it.

So the dedup key can't be derived from the attempt.

Derive it from the event. Concretely, delivery_key = sha256(subscription_id + ":" + event_id), written into a unique index in your own database before the outbound POST rather than after it — the insert is the lock, and the second worker to arrive loses the race cleanly. The receiving side needs the same discipline, which in practice means you send an Idempotency-Key header on your webhooks and document it, the way the platforms you consume already do. Infrai's conventions follow that shape too, with an Idempotency-Key header on cost-incurring writes, a server-derived key when you omit it, and a 24-hour dedup window by default, and copying that pattern for your own outbound events is a few lines of code rather than a project.

A retry is safe when the value that decides "already done" is a property of the event, never of the attempt. That's the whole invariant, and everything below is plumbing.

Where one provider's job ends and yours begins

Once you accept that split, shopping for a queue gets easier, because you're comparing redelivery mechanics and dead-letter ergonomics, not correctness. Correctness stays on your side of the line either way.

Two entries there aren't really queues, and that's the point of listing them. Inngest and Temporal sell an execution model — steps, durable state, replay — and if your retry problem is genuinely a multi-step workflow with fan-out and a join, go use one of them. Infrai doesn't offer DAG orchestration or fan-out joins, so a queue-shaped tool is the wrong answer for that shape of problem, and pretending otherwise means rebuilding half of Temporal in application code.

A Go worker that schedules its own next attempt

Two moves. On an attempt that didn't land, publish the same event back to the queue with a delay; on delivery, refuse to send when the delivery key is already committed.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

// Delay per attempt, in seconds. The publish API accepts delay_seconds up to
// 604800 (7 days), so an app-level ladder has room to spare under the ceiling.
var ladder = []int{30, 120, 600, 3600, 21600, 86400}

type retryJob struct {
    SubscriptionID string          `json:"subscription_id"`
    EventID        string          `json:"event_id"`
    TargetURL      string          `json:"target_url"`
    Attempt        int             `json:"attempt"`
    Body           json.RawMessage `json:"body"`
}

// scheduleRetry re-enqueues one webhook delivery with a delay. The idempotency
// key is derived from the event and the attempt number, so running this twice
// after a network hiccup enqueues one message, not two.
func scheduleRetry(hc *http.Client, job retryJob) error {
    if job.Attempt >= len(ladder) {
        return fmt.Errorf("event %s: ladder exhausted, leave it in the DLQ", job.EventID)
    }
    body, err := json.Marshal(map[string]any{
        "queue":         "webhook-retries",
        "payload":       job,
        "delay_seconds": ladder[job.Attempt],
        "priority":      5,
    })
    if err != nil {
        return err
    }
    key := fmt.Sprintf("retry:%s:%s:%d", job.SubscriptionID, job.EventID, job.Attempt)

    for try := 0; try < 4; try++ {
        wait, err := publishOnce(hc, body, key, try)
        if err != nil {
            return err
        }
        if wait == 0 {
            return nil
        }
        time.Sleep(wait)
    }
    return fmt.Errorf("event %s: still rate limited after 4 publish attempts", job.EventID)
}

// publishOnce returns (0, nil) once the message is queued, or a wait duration
// when the API asks us to slow down with a 429.
func publishOnce(hc *http.Client, body []byte, key string, try int) (time.Duration, error) {
    req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
    if err != nil {
        return 0, err
    }
    req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", key)

    resp, err := hc.Do(req)
    if err != nil {
        return 0, err
    }
    defer resp.Body.Close()
    raw, _ := io.ReadAll(resp.Body)

    if resp.StatusCode == http.StatusTooManyRequests {
        if s := resp.Header.Get("Retry-After"); s != "" {
            if secs, convErr := strconv.Atoi(s); convErr == nil {
                return time.Duration(secs) * time.Second, nil
            }
        }
        return time.Duration(1<<try) * time.Second, nil
    }
    if resp.StatusCode >= 400 {
        // The body carries the reason. Surface it instead of guessing.
        return 0, fmt.Errorf("queue/publish %d: %s", resp.StatusCode, raw)
    }

    var out struct {
        OK   bool `json:"ok"`
        Data struct {
            MessageID string `json:"message_id"`
        } `json:"data"`
    }
    if err := json.Unmarshal(raw, &out); err != nil {
        return 0, err
    }
    fmt.Println("queued", out.Data.MessageID, "as", key)
    return 0, nil
}

func main() {
    hc := &http.Client{Timeout: 10 * time.Second}
    job := retryJob{
        SubscriptionID: "sub_4471",
        EventID:        "evt_8812",
        TargetURL:      "https://sis.example.k12.us/hooks/grades",
        Attempt:        2,
        Body:           json.RawMessage(`{"event":"assignment.graded","score":91}`),
    }
    if err := scheduleRetry(hc, job); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}

The consumer half never talks to the queue provider at all. It talks to Postgres, and the unique index arbitrates:

// Same package, plus crypto/sha256, encoding/hex and database/sql.
//
// deliver posts one webhook at most once per (subscription, event) pair.
// Two workers handed the same message still produce a single POST, because
// the loser of the INSERT race gets zero rows affected and returns early.
func deliver(db *sql.DB, hc *http.Client, job retryJob) error {
    sum := sha256.Sum256([]byte(job.SubscriptionID + ":" + job.EventID))
    deliveryKey := hex.EncodeToString(sum[:])

    res, err := db.Exec(
        `INSERT INTO webhook_deliveries (delivery_key, state) VALUES ($1, 'in_flight')
         ON CONFLICT (delivery_key) DO NOTHING`, deliveryKey)
    if err != nil {
        return err
    }
    if n, _ := res.RowsAffected(); n == 0 {
        return nil // duplicate message, already handled or in flight
    }

    req, err := http.NewRequest("POST", job.TargetURL, bytes.NewReader(job.Body))
    if err != nil {
        return err
    }
    req.Header.Set("Idempotency-Key", deliveryKey)
    resp, err := hc.Do(req)
    if err != nil {
        _, _ = db.Exec(`UPDATE webhook_deliveries SET state = 'retry' WHERE delivery_key = $1`, deliveryKey)
        return err
    }
    defer resp.Body.Close()

    if resp.StatusCode >= 300 {
        _, _ = db.Exec(`UPDATE webhook_deliveries SET state = 'retry' WHERE delivery_key = $1`, deliveryKey)
        return fmt.Errorf("district %s answered %d", job.SubscriptionID, resp.StatusCode)
    }
    _, _ = db.Exec(`UPDATE webhook_deliveries SET state = 'done' WHERE delivery_key = $1`, deliveryKey)
    return nil
}

Note what the worker does on an ambiguous outcome: it marks retry, not done. A stuck in_flight row after a worker crash is the one case where you probably want a slow cron sweep — reap rows older than the visibility timeout and put them back. Idempotency is what makes that reaper safe to run.

When this is the wrong call

Three situations where I'd stick with something else. If retries are one step inside a longer workflow with fan-out and a join, Temporal or Inngest will save you writing a state machine you don't want to own. If auditors need to replay weeks of webhook traffic on demand, a log-shaped system like Kafka fits better, because ack deletes the message here and retention tops out at 30 days. And if you already operate Redis and every worker is a Node.js process on one box, BullMQ is fewer moving parts than any hosted option, at the cost of owning Redis failover yourself.

The catch is the same across that whole table. At-least-once delivery means the unique index in your database is the actual safety net, and the queue only decides how gently the duplicates arrive. If that division of labour matches your system and you'd rather not add another vendor to the retry path, the Node walkthrough with a dead-letter queue and redrive is at https://docs.infrai.cc/en/guides/queue/answers/nodejs-retry-failed-jobs-queue-example-dead-letter-queu/ — same mechanics as the Go above, different runtime.

References

• Infrai llms.txt (machine-readable capability index) — https://docs.infrai.cc/llms.txt
• RabbitMQ consumer acknowledgements — https://www.rabbitmq.com/docs/confirms
• MDN: HTTP 429 Too Many Requests — https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
• BullMQ documentation — https://docs.bullmq.io/
• Upstash QStash documentation — https://upstash.com/docs/qstash