Design a Notification System: The Four-Point Answer, and the Duplicate-Send Trap

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

"Design a notification system" is one of the most common system-design interview questions, and one of the easiest to answer badly — because it sounds like a feature question.

It is not. The interviewer is asking whether you treat notification as shared infrastructure that every other service calls, or as code each service bolts on for itself. Everything else in the answer follows from that one choice.

Candidates who miss it start describing SendGrid. Candidates who get it start describing a boundary.

Here is the whole answer, in the order one event travels.

1. One central notification service

Your order service publishes an event:

order.shipped   { "user_id": 4192, "order_id": 8871 }

That is all it does. It does not know — and must not care — whether that event becomes a push, an email, or an SMS.

Why this is the point being marked: the alternative is real, and it is what most systems actually grow into. payments adds a mailer to send receipts. auth adds a second one for password resets. orders adds a third for shipping. Now you have three half-built notification systems, three template stores, three sets of provider credentials, three places that do or do not check whether the user opted out — and no single place to add a channel. Adding WhatsApp means editing three services and testing three deploys.

A central service inverts that:

Producers emit domain events. The notification service owns templating, channel selection, provider integration, preferences, and delivery.

The last row is the one people underrate. A per-user cap is not even expressible when five services send independently — none of them knows what the others sent this morning.

2. A queue absorbs the burst

A promo goes out. Ten million users need a message. Those ten million sends must not arrive at your SMS provider in the same second — you will be rate-limited, throttled, or simply cut off.

So the producer writes to a queue and returns immediately. Workers drain the queue at whatever rate the downstream provider actually tolerates. The queue is the shock absorber between a spiky producer and a provider with a fixed budget.

# producer: fast, synchronous, and done
await queue.publish("notify.email", {
    "event_id": evt.id,          # the dedupe seed — see the trap
    "user_id": evt.user_id,
    "template": "order_shipped",
})
# returns in ~1ms. It does NOT wait for SendGrid.

It is also where retries live, and that is the part people forget. A provider returning a 503 is normal. The right response is a bounded retry with exponential backoff, and a dead-letter queue for the messages that still fail — so a permanently bad address does not block the partition behind it.

Retry logic buried in a request handler is retry logic you cannot observe, cannot cap, and cannot replay. The DLQ is what turns "notifications are broken" into a number you can look at.

3. Push, email and SMS are separate channels

They look like three names on a list. They are three different systems:

Different providers, different failure modes, different latency budgets — so give each channel its own queue and its own worker pool.

The reason is isolation. When your SMS vendor has a bad night, a shared worker pool means every push and every email is stuck behind a retrying SMS. Your workers are all blocked on the one provider that is slow, and a password-reset email that costs nothing and never fails is now four minutes late. With split pools, one channel degrades and the other two do not notice.

This is also where fan-out belongs. One event resolves to a per-user channel list, and each channel gets its own message. One event in, N channel-messages out — which, note, is also N chances to send a duplicate.

4. Per-channel rate limits, and the user's preferences

Two things must be true before a worker calls a provider.

A cap per channel. Say 50 pushes per user per day. Without it, one buggy loop in one producer empties your SMS budget overnight and trains every user to disable notifications permanently. The cap is cheap insurance against a bug you have not written yet.

The user's preferences. Opt-out is not a filter you apply at the end — it is a check on the hot path, per channel, because "email me but never text me" is the normal case, not an edge case. Quiet hours matter for the same reason: a 03:00 push is worse than no push. Hold it and send in the morning.

if not prefs.allows(user_id, channel):        return DROP
if quiet_hours(user_id, channel, now):        return DEFER_TO_MORNING
if rate_limiter.exceeded(user_id, channel):   return DROP

Three checks, in the worker, before the provider call. Not in the producer — the producer emitted a domain fact, and whether that fact reaches a human is not its decision.

And then dedupe.

The trap: not sending it five times

The hard part was never sending one notification. It is not sending the same one five times.

Count the at-least-once guarantees in the pipeline above:

• The producer retries, because it did not see the broker's ack — but the message was already enqueued.
• The queue redelivers, because the worker crashed after calling the provider but before acknowledging.
• The worker retries, because the provider timed out — after it had already accepted the message.

Every one of those is individually correct. Every one of those is a duplicate waiting to happen. And they compose: three independent at-least-once hops is not "three times more careful", it is a multiplier.

The fix is an idempotency key the whole pipeline agrees on:

key = f"{event_id}:{user_id}:{channel}"

# claim it BEFORE the provider call, atomically
if not redis.set(key, "sent", nx=True, ex=86400):
    return  # someone already sent this. drop it.

provider.send(...)

Write it to a store with a TTL before you call the provider, conditionally — SETNX, or a unique constraint on a table. If the key is already there, drop the send.

Before, not after. If you claim the key after a successful send, the crash-between-send-and-claim window is exactly the window the queue's redelivery will hit. Claiming first means the worst case is a lost notification, not a duplicate one — and for notifications that is the right way round: a missed shipping email is a support ticket, five identical buzzes at 2am is an uninstall.

Now the producer can retry, the queue can redeliver and the worker can crash mid-flight, and the user still gets exactly one message.

Users do not file bug reports about duplicate notifications. They mute the app, and you never find out.

The same shape, one level up: LLM and agent pipelines

If you build with LLMs, you have already built most of this system without calling it a notification system — and the interview answer transfers almost line for line.

Two things get worse in the LLM version, which is worth saying out loud if the interview drifts that way.

A duplicate costs money. A redelivered notification wastes a push token. A redelivered generation burns real tokens at real prices, and a queue that silently redelivers on a slow worker can double your inference bill without a single error in the logs.

A retry is not the same answer. Notifications are deterministic: resending produces the identical message, so a duplicate is merely annoying. Generation is not. Retry a timed-out completion and the user may get two different answers to the same question — which is why the idempotency key has to guard the call, and why the response gets cached against that key rather than regenerated.

And the last hop usually is a notification: the job finishes, and something has to tell the user. Webhook, push, email — the same fan-out, the same dedupe problem, one layer up.

The verdict

Answer the question that was asked. "Design a notification system" is asking where the boundary goes, not which vendor you like.

The whole answer in four lines:

1. One central service → every producer publishes an event and stops there. The alternative is N half-built mailers and no place to add a channel.
2. A queue → absorbs the burst, and is where retries, backoff and the DLQ live.
3. Separate channels → separate providers, separate queues, separate pools, so one bad provider night stays local.
4. Limits + preferences + dedupe → capped, opted-in, quiet-hours-aware, and exactly once.

Then close on the trap yourself, before the interviewer asks. Saying "the hard part here isn't sending one notification — it's not sending the same one five times, because every hop in this pipeline is at-least-once" is the sentence that separates someone who has drawn this diagram from someone who has operated it.

Watch the reel: the 90-second version walks the same four points end to end, and closes on the duplicate-send trap.