Renewal Reminders on a Deadline: Delayed Queue, Retry, and Public HTTPS Endpoints

Pick the delivery guarantee before picking the scheduler. A renewal reminder that has to reach a customer before a contractual deadline needs at-least-once delivery plus an idempotency key at the receiving endpoint; the delayed queue message is only a trigger, and the delivery row in your own database is the fact you will later be asked to prove. A five-minute retry is a policy written on top of that mechanism, never a substitute for it.

Timers are cheap. Evidence is the expensive part.

The constraint is a business deadline, not a delay timer

In a payments or insurance backend, a renewal notice has a legitimate window rather than a moment. The contract fixes how far in advance the customer must be told, the operations team wants the notice recorded against the same reference the invoice carries, and an auditor asks a very boring question months later: for this contract, on this date, what did we send, when, and how do you know it went out exactly once? The scheduling input is a calendar computation — renewal date minus the notice period, rounded forward to the next business hour in the customer's jurisdiction — rather than now + delay.

That distinction decides the whole design. A delay is a duration, and durations drift: a broker restart, a paused consumer, a daylight-saving transition, or a queue that was drained during an incident all move a duration-based reminder without moving the deadline it was supposed to protect. A due timestamp doesn't drift, because it's a stored column that a worker can compare against the clock on every pass.

So the durable object is a reminder record with due_at, and the queue message is a hint that due_at has probably arrived.

How should a delayed queue schedule a webhook retry to a public HTTPS endpoint after five minutes?

Write the reminder and its first delivery row in one transaction, then publish a small message whose only job is to wake a worker at roughly the right time. The message carries identifiers, not the payload — a delivery ID, the contract reference, and the attempt number — because brokers cap message size, and because a payload that lives in two places will eventually disagree with itself.

The worker then does four things in a fixed order: claim the attempt, call the endpoint, record the outcome in the same transaction that ends the claim, and acknowledge the message last. Acknowledgement order is the part teams get wrong. Broker acknowledgements confirm that a consumer took responsibility for a message, and a consumer that dies before acknowledging causes redelivery; that is documented behaviour, not an edge case, and it means at-least-once is the guarantee you actually operate under even when the marketing word on the box is "exactly once."

Exactly-once is a property of the effect, not of the transport. You build it at the receiver.

For an outbound reminder, the receiver is the notification service sitting behind a public HTTPS endpoint, and the contract with it is an Idempotency-Key header that stays constant across every retry of the same logical delivery. The IETF draft for that header describes exactly this: the client generates the key, the server stores the result of the first successful processing, and a replay with the same key returns the recorded result rather than performing the work again. Attempt two of delivery dlv_8817 carries the same key as attempt one. The attempt counter goes in a separate header, where it belongs, so the receiver can log retries without changing its dedup behaviour.

Five minutes is a reasonable first backoff for a deadline that is days away, and a terrible one for a deadline that is nine minutes away. Compute the retry from the remaining margin instead: min(300s, remaining_margin / 4), capped by the number of attempts you're willing to defend in a reconciliation meeting. When the response carries Retry-After, honour it — the receiver knows more about its own capacity than your backoff constant does.

Claiming the attempt before you call out

The claim has to be durable and single-winner, otherwise two workers wake on the same redelivered message and both call the endpoint. A row lock plus a due-time predicate handles it without a distributed lock service:

-- Single-winner claim. The unique index on (delivery_id, attempt) is the guard;
-- the queue is only the wake-up.
update webhook_delivery
   set attempt    = attempt + 1,
       state      = 'in_flight',
       claimed_at = now()
 where delivery_id = (
         select delivery_id
           from webhook_delivery
          where state = 'due'
            and due_at <= now()
          order by due_at
            for update skip locked
          limit 1)
returning delivery_id, endpoint, attempt, idempotency_key;

The Go worker below performs one attempt against that claim. It signs the body, sends the stable idempotency key, classifies the response into delivered, retryable, or terminal, and returns the delay for the next attempt. Nothing here is Go-specific: a Node.js task queue worker keeps the same four-step order, and the library you pick only changes who owns the timer.

const baseRetry = 5 * time.Minute

// deliver runs one attempt inside the transaction that already claimed it.
// Every branch writes an outcome row; a silent return would leave the delivery
// stuck in 'in_flight' with no audit trail.
func deliver(ctx context.Context, tx *sql.Tx, hc *http.Client, a Attempt, secret []byte) (time.Duration, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, a.Endpoint, bytes.NewReader(a.Body))
    if err != nil {
        return 0, err
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write(a.Body)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", a.IdempotencyKey) // constant across retries
    req.Header.Set("X-Delivery-Attempt", strconv.Itoa(a.Attempt))
    req.Header.Set("X-Signature-256", hex.EncodeToString(mac.Sum(nil)))

    resp, err := hc.Do(req)
    if err != nil {
        // No response means no evidence either way: retry with the same key.
        return baseRetry, record(ctx, tx, a, 0, "retryable", err.Error())
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, io.LimitReader(resp.Body, 8<<10))

    switch {
    case resp.StatusCode < 300:
        return 0, record(ctx, tx, a, resp.StatusCode, "delivered", "")
    case resp.StatusCode == http.StatusTooManyRequests, resp.StatusCode >= 500:
        wait := baseRetry
        if s, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && s > 0 {
            wait = time.Duration(s) * time.Second
        }
        return wait, record(ctx, tx, a, resp.StatusCode, "retryable", "")
    default:
        return 0, record(ctx, tx, a, resp.StatusCode, "terminal", "")
    }
}

Two details are load-bearing. The transport error path returns a retry rather than a failure, because a timeout tells you nothing about whether the receiver processed the request — only the idempotency key can settle that. And the response body is drained under a limit before the connection goes back to the pool, which is the kind of thing that costs you an afternoon when a receiver starts answering with a 40 MB HTML page.

Which scheduling mechanism carries the guarantee?

Compare mechanisms by where the guarantee physically lives, since that is what you will be defending during reconciliation:

The pragmatic combination for deadline-bound reminders is the second row plus the first: a due-time table as the source of truth, a delayed message as the low-latency wake-up. If the queue loses a message, the poller finds the row. If the poller lags, the message wakes the worker early. Neither path can create a second business effect, because the unique constraint and the idempotency key both sit below them.

There are cases where this shape doesn't fit. A reminder that is one step of a compensating, multi-step process is not suitable for a single delayed message; stick with a durable workflow engine when steps must converge or roll back together. If several independent consumers need the same renewal event, a delivery table plus fan-out queues is the wrong tool, and a retained log with consumer groups fits better. The catch with the log is that it deletes by retention policy rather than by acknowledgement, which is a different audit story than the one your compliance team has already approved.

Rolling it out without a reconciliation gap

Run the new path in shadow first: create reminder and delivery rows, let the worker compute due_at and the attempt schedule, and post to an internal sink instead of the customer-facing endpoint. Compare computed due timestamps against the business calendar for a full week, including at least one weekend and one holiday, before a single real notice goes out. Aggregate counts balancing is not evidence — reconcile by delivery ID.

The failure drills that matter here are narrow and repeatable: kill the worker between commit and acknowledgement, replay a duplicate message, return 429 with Retry-After: 900, return 503 twice then 200, and move a customer's renewal date backwards while a delivery is in flight. The expected result of each is one business effect, one audit row, and a delivery state a human can read.

Instrument the queue and the table separately. Attempts by state, age of the oldest due row, and time-to-first-attempt tell you whether the schedule is holding; broker depth alone tells you almost nothing, since a healthy broker with a stalled poller looks identical to an idle one. Keep the audit record on the retention schedule your regulator expects, not on the queue's — acknowledgement deletes messages, and that's a delivery mechanism, not an evidence policy.

Cost lands mostly on people, honestly. A polled table is boring and cheap to run but adds a schema, indexes and a migration path; a broker adds a component someone has to patch, size and page for at 03:00. Your mileage may vary on which of those your team already knows how to operate, and that familiarity is usually a better tiebreaker than any benchmark you'll find on a vendor page.

References

• https://www.rabbitmq.com/docs/confirms
• https://www.rabbitmq.com/docs/priority
• https://datatracker.ietf.org/doc/html/draft-ietf-httpapi-idempotency-key-header
• https://www.rfc-editor.org/rfc/rfc9110
• https://www.postgresql.org/docs/current/sql-select.html
• https://opentelemetry.io/docs/concepts/signals/metrics/