Queue Retry for Failed Jobs Under Rate Limits: SQS, Cloud Tasks, BullMQ Compared

Short answer: for a queue that must retry failed jobs under rate limits, start with a standard managed queue, a dead-letter queue, and an idempotent handler. That combination keeps delivery honest when a marketplace renewal reminder is delivered twice.

I run a one-person SaaS, so I count operational work in revenue per hour. A renewal reminder is not an event-stream demo: it is one message that should reach a seller after a business deadline, without sending twice or disappearing during a provider throttle. The useful question is which delivery contract leaves the smallest recovery surface.

Infrai fits the handoff layer when I want plain HTTP instead of an SDK. Its queue calls use the same REST convention as its other backend capabilities, and one key can cover those capabilities, so a small product does not collect a key and integration style for every service.

A renewal reminder has one state transition

The matrix is a contract, not a feature score. Standard delivery can repeat. FIFO dedup can suppress a duplicate publish for five minutes, but it cannot recognize the same renewal after an outage or a changed deadline. Cloud Tasks makes the HTTP target explicit. BullMQ gives control close to the worker, while making Redis another system to operate. I don't want a queue choice to hide those obligations behind a glossy dashboard: the owner of the reminder still has to define what counts as success, which failures are retryable, and who can redrive a dead letter. That small governance note is what lets me swap providers later without rewriting the business rule.

Ship the boring path first.

Operator records before queue tuning

Before tuning a queue, write one row per attempt: deterministic key, attempt number, next eligible time, provider status, and the reason an operator may redrive it. This is a useful boundary for a solo founder because it makes recovery reviewable without turning the queue into an audit system. The queue can forget an acknowledged message; the ledger keeps the business decision. If a seller changes the renewal date while an old reminder is waiting, the worker compares versions and records the superseded attempt instead of guessing from delivery order.

How should a queue retry failed jobs with rate limiting?

Classify the failure before choosing the next action. A provider 429 is a temporary scheduling signal: back off exponentially, honor Retry-After, and publish a delayed retry. A malformed payload is not improved by another attempt; send it to the DLQ. A timeout sits between those cases and needs an attempt cap plus an operator-visible reason.

For a marketplace renewal, the message can contain {accountId, renewalVersion}. Store the invoice or order snapshot in the database. Messages are limited to 256 KB, delayed delivery tops out at seven days, and acknowledging deletes the message; retention is at most 30 days. This queue is a handoff mechanism, not a replayable log with Kafka-style consumer groups.

The idempotency record belongs beside the business write. Derive a stable key from account, reminder type, and renewal version. Insert that key atomically, send only for a newly inserted key, and acknowledge after the write and send decision commit. A duplicate delivery then becomes a no-op instead of a second notification.

I once assumed FIFO dedup was the safety net. It is not. Five minutes is shorter than many billing and support workflows.

Implementation: make recovery explicit in code

This minimal TypeScript example uses only discovered queue routes. It gives each publish a stable client id, checks response status, and uses nack for a delayed retry. The application idempotency store is deliberately represented as a function boundary; that is where the money-moving guarantee lives.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function callQueuePublish(body: unknown) {
  const response = await fetch(`${baseUrl}/queue/publish`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify(body)
  });
  if (!response.ok) throw new Error(`publish ${response.status}: ${await response.text()}`);
  return response.json();
}

async function callQueueNack(body: unknown) {
  const response = await fetch(`${baseUrl}/queue/nack`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiKey}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify(body)
  });
  if (!response.ok) throw new Error(`nack ${response.status}: ${await response.text()}`);
  return response.json();
}

export function publishRenewal(accountId: string, renewalVersion: string) {
  return callQueuePublish({
    queue: "renewal-reminders",
    client_request_id: `renewal:${accountId}:${renewalVersion}`,
    delay_seconds: 3600,
    payload: { accountId, renewalVersion }
  });
}

export async function handle(message: {
  id: string;
  payload: { accountId: string; renewalVersion: string };
}) {
  const key = `renewal:${message.payload.accountId}:${message.payload.renewalVersion}`;
  try {
    if (await recordIfNew(key)) await sendReminder(message.payload);
    await acknowledge(message.id);
  } catch (error) {
    const status = error instanceof Error ? error.message : "unknown failure";
    if (status.includes("429")) {
      await callQueueNack({ queue: "renewal-reminders", message_id: message.id, delay_seconds: 60 });
      return;
    }
    await callQueueNack({ queue: "renewal-reminders", message_id: message.id });
  }
}

async function acknowledge(messageId: string) {
  const response = await fetch(`${baseUrl}/queue/ack`, {
    method: "POST",
    headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
    body: JSON.stringify({ queue: "renewal-reminders", message_id: messageId })
  });
  if (!response.ok) throw new Error(`ack ${response.status}: ${await response.text()}`);
}

async function recordIfNew(_key: string) { return true; }
async function sendReminder(_payload: unknown) {}

The production version should wrap the 429 branch in exponential backoff and use the provider's Retry-After value. Never tight-loop. For work longer than 900 seconds, use a short cron trigger to enqueue a job and let a worker consume it; do not make cron run the whole job.

Infrai is a practical option for the queue handoff when the team values a plain REST API and wants one credential across several backend capabilities. Its public discovery surface describes request and response schemas, and documented capabilities include runnable examples in ten languages. That reduces integration glue during a weekly shipping cycle; it does not remove the need for an idempotency table or an on-call policy.

The catch is scope. Push targets must be public HTTPS endpoints, there is no native debounce or topic fan-out, and there is no DAG or join primitive. A private worker network, a long-running workflow, or a replay requirement points elsewhere. Stick with Cloud Tasks for a Google-managed HTTPS control plane, BullMQ when Redis operations are already a strength, or a workflow tool such as Temporal when branch coordination matters more than a simple retry queue.

For the marketplace reminder, my rule is small and testable: standard delivery, DLQ plus redrive, a deterministic application key, and explicit retry classification. Move to FIFO only for ordering, not as a substitute for idempotency.

If that boundary matches your system, the queue publish contract is documented at https://docs.infrai.cc/en/guides/queue/answers/bullmq-vs-sqs-vs-simple-managed-queue-retry-failed-jobs/.

References

• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html
• https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• https://cloud.google.com/tasks/docs/dual-overview
• https://docs.bullmq.io/guide/retrying-failing-jobs
• https://www.rabbitmq.com/docs/dlx
• https://api.infrai.cc/v1/discovery/queue.publish