Small-App Job Recovery: Retry Idempotency, Deduplication, and FIFO Trade-offs

Short answer: make the effect idempotent in the database first, use a standard queue for most retry traffic, and choose FIFO only when a single business key has order-dependent state transitions. A queue can reduce duplicate delivery; it cannot prove that a payment, invoice, or ledger mutation happened exactly once after a worker disappears.

That distinction is the design constraint. Retry policy, deduplication, and ordering then become separate controls that can be tested and audited instead of three names for the same hope.

The queue is not the ledger.

The invariant sits below the queue

An at-least-once consumer is the honest default for a distributed worker. A process can finish an external call, lose its network connection before acknowledging the message, and receive the same message again. Celery's introduction describes this possibility and advises that task bodies be idempotent. The duplicate is therefore a normal delivery outcome, not an exceptional broker failure.

For a finance-shaped backend, the durable invariant belongs in a transactional store: one business operation maps to one ledger intent, and every later attempt can find that intent by a stable key. The key should come from the business event, such as payout:{invoice_id}, rather than a broker-generated message identifier. Redelivery changes the latter; it must not change the former.

The database constraint is the useful boundary because reconciliation has a long memory. A transport's duplicate-suppression cache might last minutes, while settlement files, chargebacks, and audit reviews can arrive months later. PCI DSS v4.0 also treats audit history and availability as operational controls. A short broker window can protect worker capacity, but it is not an accounting record.

How should a small business app retry failed jobs when ordering, deduplication, and idempotency collide?

Start by classifying the effect, not the queue. If two deliveries describe the same business event, the handler should return the existing result. If they describe two different events on the same account, ask whether applying them out of sequence creates an invalid state. Only the second question is an ordering requirement.

This is the smallest reliable write path I use for a retryable ledger intent. It relies on a unique index over idempotency_key; the conflict branch returns the previously created row, so acknowledgement of a redelivery does not create a second effect.

type Job struct {
    Key       string
    AccountID int64
    AmountMinor int64
}

func (w *Worker) RecordIntent(ctx context.Context, j Job) (int64, error) {
    tx, err := w.db.BeginTx(ctx, nil)
    if err != nil {
        return 0, err
    }
    defer tx.Rollback()

    var id int64
    err = tx.QueryRowContext(ctx, `
        INSERT INTO ledger_intents (idempotency_key, account_id, amount_minor, state)
        VALUES ($1, $2, $3, 'pending')
        ON CONFLICT (idempotency_key) DO UPDATE
          SET idempotency_key = EXCLUDED.idempotency_key
        RETURNING id`, j.Key, j.AccountID, j.AmountMinor).Scan(&id)
    if err != nil {
        return 0, err
    }
    if err := tx.Commit(); err != nil {
        return 0, err
    }
    return id, nil
}

The external side effect needs the same key when the provider supports idempotent requests; otherwise, record an explicit pending state and reconcile the provider reference before retrying. Do not infer success from a timeout, and do not mark a job permanently failed merely because the acknowledgement was lost. Those states need a durable transition and an operator-visible trail.

Deduplication is narrower. It suppresses repeated messages inside a defined identity and time window, which lowers load during a retry burst. It does not cover a late replay, a manually re-created job, or two distinct messages that carry the same business meaning under different message IDs. Idempotency is the property that makes all of those cases safe.

When is FIFO ordering worth its operational cost?

Ordering is valuable when the state machine is order-sensitive. Consider authorize, capture, and refund for one authorization: each operation can be individually idempotent, yet a refund processed before capture is still invalid. Grouping by authorization or account can prevent that inversion. Global ordering is usually wasteful because unrelated customers do not share a causal sequence.

The cost is head-of-line blocking. One poison message can hold every later message in its group until its retry schedule, visibility timeout, and dead-letter policy release it. Throughput is also shaped by the number of active groups and the broker's per-group limits. A FIFO transport is a poor fit when jobs are independent, when a worker can safely compare versions in the database, or when latency matters more than sequence fidelity.

The catch is that FIFO does not remove the need for the unique constraint. A consumer can crash after applying an effect but before acknowledging the message; the broker will still redeliver it. Conversely, a standard queue can be correct for an order-sensitive workflow if the handler stores a sequence number and rejects or parks a future version until its predecessor is present. Pick the simplest mechanism that enforces the invariant you actually have.

A retry loop is an audit workflow, not a timer

Retries should carry an attempt count, an exponential delay with jitter, and a hard ceiling. Persist run_after and the last error with the job so that a restart does not reset the schedule and create a retry storm. A useful state history distinguishes a connection timeout from a rejected business rule: the first can be retried after the provider's idempotency window is checked, while the second should usually be parked immediately. Record the transition that made that decision, the response classification, and the next eligible time; otherwise an operator sees only a growing integer called attempts and has no way to tell a transient outage from a permanent validation error. After the ceiling, move the record to a parking table or dead-letter stream that retains the payload hash, business key, error class, timestamps, and every attempt. Deleting it loses the explanation a reconciliation analyst will eventually request.

For a database-backed worker, FOR UPDATE SKIP LOCKED allows concurrent consumers to claim different ready rows without waiting on a locked one. The ordering clause is yours to define: per-account sequence, due time, or an explicit priority. Keep the claim and state transition in one transaction, and make the lease visible so a crashed worker's job becomes eligible again.

Observability should count more than failures. Track duplicate-key conflicts, age of the oldest ready job, attempts by error class, parked-job volume, and the gap between an external request and its confirmed reference. Those measures tell you whether the queue is protecting the system or hiding an unresolved side effect.

Three tests are disproportionately valuable: deliver one message twice and assert one ledger row; deliver it again after the row is settled and assert no downgrade; and kill the worker between the external call and acknowledgement, then verify reconciliation converges on one provider reference. Run these against a real transactional database in CI. An in-memory fake cannot reproduce uniqueness, locking, or crash timing.

Rollout: make correctness boring before changing transport

Add the unique index and stable key derivation first. Then instrument conflict counts and park records without changing queue selection. This gives a baseline for the duplicate behavior already present in production.

Next, make retry state explicit and replay parked jobs through the same idempotent handler. If a sequence invariant remains after that work, introduce a per-business-key group or a database version check; migrate one key space at a time and watch head-of-line delay.

Stick with a standard queue when work is independent and the database can reject stale versions. Choose FIFO when preserving order for a narrow key is cheaper than buffering and repairing out-of-order events. Neither choice is suitable when the real problem is an unknown external outcome; that requires reconciliation and an audit trail, not a different queue label.

References

• Celery introduction and task execution semantics — https://docs.celeryq.dev/en/stable/getting-started/introduction.html
• PostgreSQL SELECT, FOR UPDATE, and SKIP LOCKED — https://www.postgresql.org/docs/current/sql-select.html
• PCI Security Standards Council document library — https://www.pcisecuritystandards.org/document_library/
• Stripe API idempotent requests overview — https://docs.stripe.com/api/idempotent_requests