Weekly Media Digest Recovery: DLQ Redrive Service Boundaries for Failed Background Jobs

Short answer: use a durable queue with a dead-letter queue (DLQ), a bounded retry policy, and an operator-approved redrive path; make the weekly digest worker idempotent so a second delivery cannot send a second digest. For a small US/EU media SaaS, operational recovery matters more than finding the fewest API calls. The schedule should create work, while the queue and the application ledger decide whether that work is safe to repeat.

This is an architecture decision record for retrying failed background jobs, not a vendor ranking. The useful question is where each failure stops, what evidence survives it, and how an operator can recover a known set of customers without turning a transient delivery problem into duplicate email or duplicate billing-side effects.

The duplicate-send boundary for a weekly digest

Start with invariants. The weekly scheduler must enqueue one logical digest job per customer and include a stable operation ID, the digest period, the customer region, a schema version, and a payload hash. The worker may receive that message more than once because ordinary queue delivery is at-least-once. It must therefore consult durable idempotency state before sending, record the outcome, and acknowledge the message only after the business result and audit record are committed.

The order is the important part. An acknowledgement says that the transport has accepted completion; it does not prove that a media digest was sent, that the recipient list was correct, or that the provider accepted the request. The application record should retain the operation ID, customer ID, period, payload hash, attempt number, transition, actor, reason, request ID, and deployment identifier. Those fields let an on-call engineer distinguish a duplicate delivery from a newly approved retry and reconcile a customer-facing result with an administrative action.

Keep the queue payload small and durable references explicit. A stack trace, rendered report, or reconciliation export belongs in controlled storage, with an immutable reference and integrity hash in the job. The DLQ is a recovery boundary for poison messages; it is not an accounting archive, a permanent event log, or proof of exactly-once execution.

Three words: send once.

For a weekly job, a cron-like trigger should enqueue bounded work and finish. A worker then owns retries, backoff, timeout handling, and the final transition to the DLQ. This separation is easier to operate than having a scheduler perform network calls for every active customer, because a missed or slow downstream request cannot hold the schedule hostage. Cron is a time trigger, not a redrive controller.

Why the scheduler must stop at the queue

The selected shape is schedule -> queue -> worker -> idempotency store and audit trail, with the DLQ beside the queue and a reviewed redrive command feeding selected messages back into the normal worker path. The decision preserves a clear recovery boundary: transient failures remain retryable, poison payloads become visible, and business mutations remain governed by application state.

The table is deliberately about failure semantics rather than price. A queue that is already inside the team's identity, logging, and incident boundary may be simpler to recover than a technically similar service that introduces a second control plane. The choice should be recorded with its delivery guarantee, retention window, visibility timeout, maximum payload, redrive permissions, and regional data-handling rules.

Exactly-once is a business invariant here, not a transport promise. The durable idempotency record should be unique on the logical operation ID, and the send path needs a clear state machine such as pending, committed, and reconciled. If the email provider supports an idempotency key, pass the same operation ID through; if it does not, record the provider request and reconcile its result before any redrive. A retry that skips this check is not recovery. It is another attempt at an irreversible side effect.

How should operators redrive failed background jobs through a DLQ service?

Redrive should be an administrative workflow with selection, approval, bounded execution, and reconciliation. Do not copy every DLQ message back into the live queue because the calendar changed. Select by failure class, digest period, customer status, and payload schema; exclude messages whose business action is already committed; record who approved the batch and why; then send the selected messages through the same validation and idempotency path as ordinary work.

A minimal command can express the critical path without binding the article to a commercial SDK. The endpoint below is intentionally pseudonymous: the transport contract is the design point, while the real queue client or HTTP API must supply its documented redrive operation.

package main

import (
    "context"
    "errors"
    "fmt"
    "time"
)

type DigestJob struct {
    OperationID string
    CustomerID  string
    Period      string
    PayloadHash string
}

func redrive(ctx context.Context, job DigestJob, store Store, sender Sender) error {
    if job.OperationID == "" || job.CustomerID == "" || job.Period == "" {
        return errors.New("invalid digest job")
    }

    state, err := store.Lookup(ctx, job.OperationID)
    if err != nil {
        return err
    }
    if state == "committed" || state == "reconciled" {
        return nil
    }
    if state == "" {
        if err := store.MarkPending(ctx, job.OperationID, job.PayloadHash); err != nil {
            return err
        }
    }

    requestID := fmt.Sprintf("digest:%s", job.OperationID)
    if err := sender.Send(ctx, job.CustomerID, job.Period, requestID); err != nil {
        return fmt.Errorf("send digest: %w", err)
    }
    return store.MarkCommitted(ctx, job.OperationID, time.Now().UTC())
}

// Store and Sender are implemented by the application boundary.
type Store interface {
    Lookup(context.Context, string) (string, error)
    MarkPending(context.Context, string, string) error
    MarkCommitted(context.Context, string, time.Time) error
}

type Sender interface {
    Send(context.Context, string, string, string) error
}

The example leaves one hard case visible: a timeout after the provider accepted the send but before the worker recorded committed. Consider a Tuesday redrive for the previous Friday's digest. The worker validates the customer and payload, calls the email provider with digest:customer-42:2026-W31, and waits; the provider commits the request, but the network connection disappears before the response reaches the worker. The job remains pending. A naive operator sees no success record and presses redrive, which can send the same digest twice. A careful operator first asks the provider whether that request ID was accepted, compares the answer with the audit record, and records a reconciled result before selecting the message. If the provider has no request lookup and no idempotency behavior, the system cannot prove that another send is safe; its honest choices are to accept a possible duplicate under an explicit policy or route the customer case to manual review. That is why the audit trail needs a provider request ID or a reconciliation query, and why a redrive approval should wait for that check. A second attempt can be safe only when the application can establish that the first side effect did not commit, or when the downstream system deduplicates the same operation ID.

Keep this boundary visible.

During operations, watch DLQ depth, age of the oldest message, retry count, redrive batch size, send-provider response classes, and the time between approval and reconciliation. Alerting only on queue depth misses a slowly growing poison-message class; alerting only on worker errors misses a successful transport acknowledgement followed by a failed ledger write. The metrics should carry region and schema version so US and EU recovery can be investigated separately without placing customer data in log messages.

The governance limits of this recovery design

A queue with a DLQ is not suitable when the workload requires a durable history for several independent consumer groups, arbitrary historical replay, a DAG with joins, or long-lived human approval state. Use a replayable log for the first requirement and a workflow-oriented design for the latter two. A scheduler is also the wrong place for per-customer network work when the digest can exceed its execution window or needs backfill semantics.

The rejected option is a cron task that directly loops over active customers, sends each digest, and retries the loop after any error. It is acceptable for a small maintenance script whose side effect is idempotent and whose missed-run behavior is explicitly harmless. It is a poor default for customer communication: one slow recipient can delay the rest, a process restart obscures the completed prefix, and a retry of the whole loop can duplicate messages unless every customer operation has its own durable identity.

Compliance does not change those mechanics. Retention, access control, deletion, regional processing, and evidence requirements must be mapped to the applicable policy and assessor; a queue's message retention is not a substitute for an audit schedule. Your mileage may vary because the decisive constraints are the email provider's idempotency behavior, the team's regional boundary, and whether customers require a legally significant delivery record. Verify those before selecting the transport.

I would reject any design that calls a message acknowledged merely because the worker returned HTTP 2xx, or that allows an unreviewed operator to redrive the entire DLQ. The durable record, the approval boundary, and the reconciliation step are the recovery system. The queue is only one component of it.

References

• Wikipedia: Cron
• Google Cloud Pub/Sub overview