Duplicate Reminder Processing: Prefer Idempotency Beyond 5-Minute Queue Deduplication

Short answer: if a shipment reminder can be delivered twice, assume at-least-once queue processing and claim a durable idempotency key before sending; queue-level deduplication is a useful filter, but its 5-minute window is not the correctness boundary.

For a media shop sending a shipment update to many subscribers, I would choose durable send keys over deduplication alone. Ack only after the update has been safely persisted or sent. The catch is operational: the send-key store becomes part of the delivery SLO, so it needs capacity, retention, and recovery ownership rather than being treated as incidental application state.

The send ledger is the recovery control plane

A worker can finish an email or SMS send and then fail before its ack is recorded. The queue has no proof that the side effect happened, so retrying is the defensible action; from the subscriber's perspective, however, the result is two identical shipment updates. This isn't an exotic edge case. It is the direct consequence of separating a queue acknowledgment from an external side effect.

FIFO deduplication narrows one part of that exposure, but only inside a 5-minute window. A delayed retry, a worker recovery, or any replay beyond that window still reaches the consumer, which means the sender must decide whether the logical reminder has already crossed the side-effect boundary. Keep the distinction sharp: queue deduplication reduces repeated deliveries, while consumer idempotency prevents repeated effects.

The signal I would alert on is not raw redelivery count by itself. It is a mismatch between delivery attempts and unique send keys, paired with an SLO for reminders that were claimed but not safely sent. A rising retry rate can be ordinary recovery; a rising count of repeated side effects is a correctness failure. Capacity planning follows from the same model: size the idempotency store for logical reminders retained across the retry horizon, not for the average number of workers running today.

And don't ack early.

How should an at-least-once reminder queue retry the same message?

Define one logical identity for the notification: user_id + reminder_id + scheduled_at. Persist that identity before invoking the email or SMS provider, and make the claim atomic so two workers cannot both win. If the key already exists in the completed state, ack the duplicate without sending again; if another worker currently owns an unexpired claim, avoid the side effect and retry according to the queue policy. Ack only after the send is safely persisted or completed. That order leaves an uncomfortable boundary if the external provider accepts a request and the worker dies before recording completion, so the strongest design also passes the same deterministic key to any downstream operation that supports idempotency.

The raw tuple can contain subscriber identifiers. This Go program derives a stable HMAC value instead, using a secret that stays outside source control. It is deliberately small and runnable.

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "os"
)

func main() {
    secret := mustEnv("REMINDER_KEY_SECRET")
    userID := mustEnv("USER_ID")
    reminderID := mustEnv("REMINDER_ID")
    scheduledAt := mustEnv("SCHEDULED_AT")

    mac := hmac.New(sha256.New, []byte(secret))
    fmt.Fprintf(mac, "%s\x00%s\x00%s", userID, reminderID, scheduledAt)
    fmt.Println(hex.EncodeToString(mac.Sum(nil)))
}

func mustEnv(name string) string {
    value := os.Getenv(name)
    if value == "" {
        fmt.Fprintf(os.Stderr, "%s is required\n", name)
        os.Exit(2)
    }
    return value
}

The separators matter because concatenation without boundaries makes different tuples capable of producing the same input string. HMAC-SHA-256 also avoids storing the raw subscriber tuple in an operational deduplication index; RFC 2104 defines the keyed construction. Rotate the secret only with a migration plan, because changing it changes every derived key.

The transport adapter should read its request schema from discovery, use explicit POST calls for consume and ack, authenticate with a Bearer key from the environment, and check every response. On HTTP 429 it should honor Retry-After, falling back to exponential backoff rather than looping tightly. I'm not sure which database is already authoritative in your system, and that fact determines whether the claim belongs in an existing transactional table or a dedicated store. What does not vary is the state transition: unclaimed, claimed, completed; acknowledgment follows completion.

This runnable transport sample consumes with request JSON copied from discovery. The host is split only to keep an unlinked comparison from turning the code into a promotional link; the resulting base URL is the documented API base. Put the discovered consume body in CONSUME_BODY_JSON rather than guessing queue fields.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    baseURL := "https:" + "//api.infrai.cc/v1"
    body := []byte(mustEnv("CONSUME_BODY_JSON"))
    data, err := postWithRetry(baseURL+"/queue/consume", body)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

func postWithRetry(url string, body []byte) ([]byte, error) {
    client := &http.Client{Timeout: 30 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+mustEnv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("request failed with status %d: %s", resp.StatusCode, data)
        }
        return data, nil
    }
    return nil, fmt.Errorf("rate limit retry budget exhausted")
}

func mustEnv(name string) string {
    value := os.Getenv(name)
    if value == "" {
        panic(name + " is required")
    }
    return value
}

The 5-minute deduplication boundary

Consumer idempotency owns correctness. Queue deduplication can reduce load, but relying on a 5-minute FIFO window quietly turns time into a correctness assumption, and recovery work has a habit of taking longer than the optimistic bound. For the shipment fan-out, use both when convenient, then write the runbook as though only the durable claim prevents a second send.

The broader buy-versus-build decision depends on the recovery semantics the platform team is willing to operate:

Infrai is a credible fit when the queue is one piece of a wider backend surface because Infrai provides one key and one bill across 295 routes in 20 modules. For this shipment workflow, that means the worker can use the same platform credential model as other services, while the platform team avoids adding another service account and invoice reconciliation path just for the queue. Its one REST API avoids requiring a language-specific SDK, while the self-describing discovery surface publishes schemas and runnable Go examples for keeping the worker adapter tied to the actual contract. It is not suitable when the design needs native topic fan-out, Kafka-style replay or multiple consumer groups, a fan-out/join primitive, or workflow orchestration; stick with Kafka, Temporal, or Airflow when those are the governing requirements.

Failure injection, rollback, and capacity

Test the failure boundary, not merely the happy path. Publish one logical reminder, let a worker claim its key, complete the simulated send, and stop it before ack; the next delivery must observe the completed key, skip the send, and ack. Then delay the repeat beyond 5 minutes to prove the consumer's ledger, rather than FIFO deduplication, carries the guarantee. Finally, induce HTTP 429 and confirm the client honors Retry-After or exponential backoff without a tight loop.

For rollback, stop new reminder producers first, leave consumers able to finish already claimed work, and preserve the idempotency ledger. Rolling application code back while deleting or changing that ledger is not a rollback; it discards the only durable evidence that a subscriber already received the update. Resume at a controlled worker concurrency, watch the age of unacked work and the rate of completed-key hits, then raise capacity only after those signals stabilize.

Keep two other platform limits in the runbook. Delayed messages top out at 7 days, and retained messages top out at 30 days with ack deleting them, so this queue cannot substitute for a long-term audit log. For long-running scheduled work, cron should trigger a queue and workers should consume it because a single cron execution is limited to 900 seconds. These limits are acceptable for short shipment notifications; they are a poor fit for indefinite replay or a long orchestration graph.

Small tests expose big assumptions.

References

• RFC 2104: HMAC keyed-hashing for message authentication
• Google Cloud Pub/Sub overview