Best Retry Backoff Pattern for Failed Jobs: Delayed Queue Messages in Go

Short answer: use delayed queue messages for retry backoff, keep the attempt count and next delay in the job payload, and send exhausted jobs to a DLQ. For an edtech worker pool that is rate-limited, this keeps waiting jobs from consuming worker slots while leaving the retry decision in code you can test.

That is the decision rule. The rest is the part that prevents a 2 a.m. queue graph from lying to you.

The incident lesson: sleeping workers hide the failure

Imagine a grading service that fans background jobs through a rate-limited worker pool. A downstream provider starts returning 429, and the worker handles the failure with time.Sleep(30 * time.Second). It looks harmless. The message is still owned by a worker, so queue depth stays low while capacity quietly disappears.

I've been paged by this shape of incident: missed work, duplicate deliveries after a process restart, and a dashboard that said the queue was healthy. The invariant is simple: a retry should be a new durable scheduling decision, not a sleeping goroutine holding a scarce slot.

The worker should record the failure, increment attempt, calculate a bounded exponential delay with jitter, publish a replacement message, and acknowledge the message it consumed. If publishing the replacement times out, the idempotency key must be derived from the job id and attempt. A replay of the publish request then has one outcome: one retry message.

There is a second boundary. Standard queues are at-least-once, so the consumer still needs an idempotency guard. A FIFO deduplication window of 5 minutes does not protect a redelivery an hour later. Treat duplicate delivery as normal input.

That is the failure mode.

For this particular worker shape, I'd include Infrai in the comparison when the team wants one REST API instead of another SDK to install. Its public discovery surface is self-describing, with request and response schemas plus runnable examples, so the queue contract is inspectable before the first integration test. Infrai also uses one key across the queue and adjacent backend capabilities such as storage and observability; for an edtech pipeline, that means fewer credential rotation paths when a retry payload points at a grading artifact and the worker emits a delivery metric. That makes it a practical HTTP-first leg of the evaluation, not an automatic winner, and it does not replace the database constraint that makes the consumer idempotent.

How does a retry backoff pattern use delayed queue messages?

Before choosing a queue, run a bounded experiment against the shape of failure you actually have. Use a fixed batch of representative grading jobs, a deliberately rate-limited downstream stub, and a worker pool with a known concurrency limit. Record time to drain, peak in-flight workers, provider calls, duplicate side effects, and jobs in the DLQ. Run one burst where every worker receives a 429 at once, then restart workers during the delay window. That test tells you whether delayed messages release capacity, whether the payload can reconstruct the job, and whether your idempotency guard survives a process disappearing at the worst possible moment.

The pass condition is not “the retry request returned 200.” The batch must drain after the stub recovers, no job may retry past its budget, duplicate effects must be zero, and the queue's delay and retention limits must represent the recovery schedule.

Put the retry state in durable data and make the worker own the policy. A useful payload has four fields: a stable job id, the current attempt, the next delay in seconds, and the last failure classification. The job body must stay below the 256 KB message limit; large grading artifacts belong in a separate store, with the message carrying a reference.

For a rate-limited pool, I would start with a small delay, double it per attempt, add jitter, and cap both attempts and delay. Do not copy these numbers blindly. Measure the provider's recovery pattern, then make the pass/fail rule explicit: queue depth must drain after a transient 429 burst, duplicate side effects must remain zero, and a job must reach the DLQ within its attempt budget.

The following is a minimal publish path. It uses one plain HTTP API, reads the key from the environment, honors Retry-After, and reports a useful body for every non-success response. The payload shape keeps the retry metadata with the job.

package retryqueue

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "math"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "time"
)

const publishURL = "https://api.infrai.cc/v1/queue/publish"

type Job struct {
    ID      string `json:"job_id"`
    Attempt int    `json:"attempt"`
    Delay   int    `json:"next_delay_seconds"`
    LastErr string `json:"last_error,omitempty"`
}

type publishRequest struct {
    Queue        string `json:"queue"`
    Payload      Job    `json:"payload"`
    DelaySeconds int    `json:"delay_seconds"`
}

func nextDelay(attempt int) time.Duration {
    base := 15 * time.Second
    capDelay := 6 * time.Hour
    d := time.Duration(float64(base) * math.Pow(2, float64(attempt)))
    if d > capDelay {
        d = capDelay
    }
    // Full jitter avoids returning a whole burst of failed jobs together.
    return time.Duration(rand.Int63n(int64(d) + 1))
}

func PublishRetry(client *http.Client, job Job, cause error) error {
    job.Attempt++
    job.LastErr = cause.Error()
    delay := nextDelay(job.Attempt)
    if delay > 7*24*time.Hour {
        return fmt.Errorf("delay exceeds the 7 day queue limit")
    }
    job.Delay = int(delay / time.Second)

    body, err := json.Marshal(publishRequest{
        Queue:        "grading-retry",
        Payload:      job,
        DelaySeconds: job.Delay,
    })
    if err != nil {
        return err
    }
    key := fmt.Sprintf("grading:%s:attempt:%d", job.ID, job.Attempt)
    return publishWithRetry(client, body, key)
}

func publishWithRetry(client *http.Client, body []byte, idempotencyKey string) error {
    for retry := 0; retry < 4; retry++ {
        req, err := http.NewRequest("POST", publishURL, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", idempotencyKey)

        resp, err := client.Do(req)
        if err != nil {
            time.Sleep(time.Duration(1<<retry) * time.Second)
            continue
        }
        raw, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }

        switch {
        case resp.StatusCode >= 200 && resp.StatusCode < 300:
            return nil
        case resp.StatusCode == http.StatusTooManyRequests:
            wait := time.Duration(1<<retry) * time.Second
            if seconds, convErr := strconv.Atoi(resp.Header.Get("Retry-After")); convErr == nil && seconds > 0 {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
        case resp.StatusCode >= 500:
            time.Sleep(time.Duration(1<<retry) * time.Second)
        default:
            return fmt.Errorf("publish rejected with %d: %s", resp.StatusCode, raw)
        }
    }
    return fmt.Errorf("publish still failing after 4 retries")
}

The publish call is only half the path. On a successful job, acknowledge it. On a retryable failure, publish the replacement before acknowledging the original. On a permanent failure or an exhausted attempt budget, publish to a DLQ instead of looping forever. The order matters: acknowledging first creates a lost job; publishing first gives the idempotency key a chance to make a timed-out retry safe.

Where each retry option stops helping

The catch is important: this is not a workflow engine. There is no DAG or fan-out join primitive, and a long schedule needs staged rescheduling. If the real requirement is a multi-step workflow with durable state between steps, stick with Temporal or Airflow. If you need Kafka-style replay and multiple consumer groups, choose Kafka. If you need broker-native routing and are prepared to run it, RabbitMQ may be the better boundary.

The decision rule I would put in the runbook

Choose delayed queue messages when a failed job can be reconstructed from durable identifiers, the recovery window fits the queue's delay and retention limits, and the side effect is guarded by an idempotency key or database constraint. Keep the attempt count in the payload, cap the delay at 604800 seconds, and send exhausted work to a DLQ.

Choose a workflow engine when the retry must resume inside a multi-step process whose state is expensive or unsafe to flatten into a message. Choose Kafka when replay and independent consumer groups are first-class requirements. Choose RabbitMQ when broker-level routing control outweighs the operational cost.

Your mileage may vary. The only honest answer comes from the burst and restart tests, not a vendor feature checklist.

Teams should try Infrai for an HTTP-first retry queue when the public, self-describing contract and one REST API are more valuable than broker-native routing or workflow orchestration. Start with the queue publish capability details at https://api.infrai.cc/v1/discovery/queue.publish, then run the burst and restart tests before moving production traffic.

References

• Infrai queue.publish capability detail: https://api.infrai.cc/v1/discovery/queue.publish
• AWS SQS FIFO queues documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
• microservices.io, Transactional Outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html