A customer asked why their invoice email never arrived. I opened the worker logs, grepped for the invoice id, and found nothing. No error. No retry. No trace. The message had been published, consumed, and then the consumer died before acking. RabbitMQ redelivered it once, the visibility window passed, and the broker moved on. From the queue's point of view, everything was fine. From ours, the work simply never happened, and no query on earth could tell us that.
That was the week I stopped calling the queue our state.
The only record of a job was the message
The job existed as a JSON blob in a routing key. Once a consumer acked it, it was gone. There was no jobs table, no status column, no attempts counter you could join against. So the three questions that actually matter in production all had the same answer: ask the broker.
- What is pending? You can get a message count per queue. You cannot get the payloads, the ages, or which customer is waiting.
- What failed? Only what is still sitting in the dead-letter queue. Anything that failed and expired is erased.
- What did we already do? Nothing. Acked means forgotten. Double-sending an invoice was as easy as replaying a batch.
We had a #alerts-dlq channel. It had forty-one unread messages and a pinned note from an engineer who had left the company.
Visibility timeouts are a guess, not a guarantee
Most brokers offer at-least-once delivery with a visibility or ack timeout. If your job runs longer than that window, the broker assumes the consumer died and hands the message to someone else. Now two workers are doing the same thing, or the first one finishes and the second one clobbers it.
The timeout is set where the message is published, but the runtime is decided by the job. A nightly export that took ninety seconds in staging takes eleven minutes when a customer has a year of data. Nobody measured it, because the publish site looked unrelated to the slowness.
If you cannot move the work off the queue, measure the real distribution before setting the number: log start and end timestamps per job type, take the p99, and set the timeout above it โ not at the median. And make every handler idempotent, because the broker will hand out that message again.
The same window governs retention. Queues are built to hold messages briefly and drop them. Message TTL, queue expiry, and DLQ retention all mean the history of what you tried is deleted on a schedule you probably did not choose deliberately. Storage for a queue is a buffer, not an archive.
"Just replay the queue" is not recovery
Replaying works exactly once: when the messages still exist. After an outage, the interesting messages are the ones that expired, were purged by the DLQ's own retention, or were never published because the producer crashed between committing and publishing. Replay cannot resurrect a message that no longer exists.
Worse, replay re-executes everything that succeeded too. Without a durable record of side effects, you cannot tell a completed charge from a pending one, so recovery becomes a manual reconciliation against the payment provider's dashboard.
Make the database the source of truth
The fix is an outbox, or a job table. Write the job to a table in the same transaction as the business change. The row is the state. The queue carries an id and nothing else.
CREATE TABLE jobs (
id bigserial PRIMARY KEY,
kind text NOT NULL,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending',
attempts int NOT NULL DEFAULT 0,
run_after timestamptz NOT NULL DEFAULT now(),
locked_until timestamptz,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
finished_at timestamptz,
CONSTRAINT jobs_status_check
CHECK (status IN ('pending','running','done','failed'))
);
CREATE INDEX jobs_pending_idx ON jobs (run_after)
WHERE status = 'pending';
CREATE UNIQUE INDEX jobs_dedupe_idx
ON jobs (kind, (payload->>'idempotency_key'));
The producer writes the row and commits. A publisher reads unpublished rows and sends {"job_id": 12345} to the broker. The consumer loads the row, and every state transition is an ordinary update: status='running' with locked_until = now() + interval '15 minutes', then done on success or failed with last_error after the attempt counter hits the limit. A reaper resets rows whose locked_until has passed back to pending, so a crashed worker costs you a retry, not a lost job.
The payload->>'idempotency_key' unique index is the part that makes replay safe. Insert the key in the same transaction as the side effect, and a duplicate delivery hits a conflict instead of sending a second invoice.
INSERT INTO job_effects (job_id, effect_key)
VALUES (%s, %s)
ON CONFLICT (effect_key) DO NOTHING
RETURNING job_id;
If that returns no row, the effect already happened. Skip it.
With the table in place, the questions answer themselves:
SELECT status, count(*)
FROM jobs
WHERE created_at > now() - interval '1 day'
GROUP BY status;
SELECT id, kind, attempts, last_error, created_at
FROM jobs
WHERE status = 'failed'
ORDER BY created_at DESC
LIMIT 50;
Now the dead-letter queue is a signal, not a filing cabinet. It tells you something is wrong; the table tells you what.
The queue is good at exactly one thing: waking up a worker promptly. Let it do that. Keep the record somewhere you can query, index, and back up.
I write about production failures in Postgres, queues, and distributed systems.
Subscribe by email ยท RSS ยท Bluesky
