Every Rails app grows a notification system eventually.
It usually starts like this:
class Comment < ApplicationRecord
after_create :notify_watchers
private
def notify_watchers
post.watchers.each do |user|
NotificationMailer.new_comment(user, self).deliver_later
user.notifications.create!(message: "New comment on #{post.title}")
end
end
end
This works beautifully — until a post has 40,000 watchers and someone comments. Now you have 40,000 inserts and 40,000 mailer jobs inside a request cycle, a timeout, a half-written notification table, and an on-call ping.
The fix isn't a gem. It's four architectural decisions. Let's walk through them.
1. Separate the event from the delivery
The single most important split: what happened is not who gets told and is not how they get told.
Event (comment.created)
└─> Fan-out (who should know?)
└─> Notification records (in-app inbox)
└─> Deliveries (email, push, Slack, SMS)
Three separate concerns, three separate failure domains. If Firebase is down, your in-app inbox still works. If fan-out is slow, the comment still saved.
Model it explicitly:
# events: what happened, once
create_table :events do |t|
t.string :kind, null: false # "comment.created"
t.jsonb :payload, null: false, default: {}
t.references :actor, polymorphic: true
t.timestamps
end
# notifications: one row per recipient
create_table :notifications do |t|
t.references :recipient, null: false
t.references :event, null: false
t.string :kind, null: false
t.jsonb :payload, null: false, default: {}
t.string :idempotency_key, null: false
t.datetime :read_at
t.timestamps
end
add_index :notifications, :idempotency_key, unique: true
That unique index is not decoration — it's what makes the whole pipeline safely retryable. More on that in a second.
2. Never fan out in the request
The web request creates the event and enqueues exactly one job. That's it.
class Comment < ApplicationRecord
after_commit :publish_event, on: :create
private
def publish_event
event = Event.create!(kind: "comment.created", actor: author,
payload: { comment_id: id, post_id: post_id })
Notifications::FanOutJob.perform_later(event.id)
end
end
Use after_commit, not after_create. Enqueueing inside a transaction is the classic race: the job starts, queries for a record the DB hasn't committed yet, and dies with RecordNotFound.
The fan-out job then resolves the audience and chunks it:
class Notifications::FanOutJob < ApplicationJob
queue_as :notifications
BATCH = 1_000
def perform(event_id)
event = Event.find(event_id)
Notifications::Audience.for(event).find_in_batches(batch_size: BATCH) do |users|
Notifications::WriteBatchJob.perform_later(event_id, users.map(&:id))
end
end
end
One long job becomes N short jobs. Short jobs retry cheaply, spread across workers, and don't hold a database connection for four minutes.
3. Bulk insert, don't loop
create! in a loop is 40,000 round trips. insert_all is 40 round trips at batch size 1,000.
class Notifications::WriteBatchJob < ApplicationJob
def perform(event_id, recipient_ids)
event = Event.find(event_id)
now = Time.current
rows = recipient_ids.map do |id|
{
recipient_id: id,
event_id: event.id,
kind: event.kind,
payload: event.payload,
idempotency_key: "#{event.id}:#{id}",
created_at: now,
updated_at: now
}
end
Notification.insert_all(rows, unique_by: :index_notifications_on_idempotency_key)
Notifications::DeliverBatchJob.perform_later(event_id, recipient_ids)
end
end
unique_by turns this into an upsert-style no-op on conflict. Now if Sidekiq retries this job after a network blip, you don't double-notify anyone. Idempotency is what lets you retry aggressively, and retrying aggressively is what makes the system reliable.
You skip callbacks and validations with insert_all — that's the trade. Notifications are derived data with a fixed shape, so it's a trade worth making. Just keep the shape enforced in one place (the row builder above).
4. Make channels pluggable and preference-aware
Delivery is where the fan-out multiplies again: one notification, three channels. Keep each channel a small object with the same interface.
module Notifications
module Channels
class Email
def self.deliver(notification)
NotificationMailer.with(notification: notification).notify.deliver_now
end
end
class Push
def self.deliver(notification)
PushClient.send(token: notification.recipient.device_token,
title: notification.title, body: notification.body)
end
end
end
end
REGISTRY = { email: Channels::Email, push: Channels::Push, slack: Channels::Slack }
Then resolve preferences once, in bulk, instead of per-user:
class Notifications::DeliverBatchJob < ApplicationJob
def perform(event_id, recipient_ids)
notifications = Notification.where(event_id: event_id, recipient_id: recipient_ids)
.includes(:recipient)
args = notifications.flat_map do |n|
Notifications::Preferences.channels_for(n).map { |ch| [n.id, ch.to_s] }
end
Notifications::DeliverJob.perform_all_later(args.map { |a| Notifications::DeliverJob.new(*a) })
end
end
perform_all_later (Rails 7.1+) pushes the whole batch to Redis in one call instead of one round trip per job. On Sidekiq you can also reach for Sidekiq::Client.push_bulk.
Each leaf job handles exactly one delivery, so failures are isolated:
class Notifications::DeliverJob < ApplicationJob
retry_on Net::OpenTimeout, Errno::ECONNREFUSED,
wait: :polynomially_longer, attempts: 5
discard_on ActiveJob::DeserializationError
def perform(notification_id, channel)
notification = Notification.find(notification_id)
Notifications::REGISTRY.fetch(channel.to_sym).deliver(notification)
end
end
Give email, push, and in-app their own queues with their own concurrency. A backed-up SMS provider should never starve your push notifications.
The unread badge problem
Here's the query that quietly kills you at scale — it runs on every single page load:
current_user.notifications.where(read_at: nil).count
Two fixes, use both:
A partial index, so Postgres only indexes rows you actually query:
add_index :notifications, [:recipient_id, :created_at],
where: "read_at IS NULL",
name: "index_unread_notifications"
A read watermark, so "mark all as read" is one write instead of 10,000:
# users.notifications_read_at
def mark_all_read!
update_column(:notifications_read_at, Time.current)
end
def unread_count
notifications.where(read_at: nil)
.where("created_at > ?", notifications_read_at || Time.at(0))
.limit(100).count
end
The limit(100) is deliberate. Nobody needs to know they have 8,412 unread notifications — "99+" is the same information at a fraction of the cost.
What to watch in production
Four numbers tell you whether this is healthy:
- Fan-out lag — event created → last notification row written
- Delivery lag — notification created → provider accepted
- Queue depth per channel — the first thing to spike when a provider degrades
- Delivery failure rate by channel — a broken push cert looks exactly like a quiet week unless you're graphing this
Emit them as counters and histograms and alert on the p95, not the average. Notification systems fail at the tail.
The short version
- Write one event, enqueue one job in the request. Never loop.
- Fan out in chunks, insert in bulk.
- Put a unique idempotency key on every notification so retries are free.
- Split delivery into per-channel jobs on per-channel queues.
- Solve the unread badge with a partial index and a watermark.
None of this requires a new gem or a new service. It's the same tables you'd have written anyway — just arranged so that the number of recipients stops being your app's problem.
How are you handling fan-out in your app — dedicated tables, an outbox, or a third-party service? I'd be curious what's held up at scale.
