Design a real-time analytics system: the stream-processing walkthrough

Design a real-time analytics system: the stream-processing walkthrough

The request to design a real-time analytics system sounds like a reporting feature — show some live numbers on a dashboard — and that framing is the trap. Picture it as a table you query on demand and you reach for a database and a GROUP BY; picture it as a firehose of events arriving a billion a minute, each one needing to move a counter that a dashboard reads a few seconds later, and it becomes one of the defining streaming problems in distributed systems. This is one of the most revealing real-time analytics system design prompts because the hard part is not storing the data — it is that you can never afford to scan the raw events when the dashboard loads. The numbers have to be computed as the data arrives and served pre-aggregated.

The round turns on one structural insight. "Show live metrics" hides the fact that the input is unbounded and bursty — it never ends, and it spikes without warning — so the only workable shape is a pipeline that aggregates continuously: ingest → process → store → serve. Raw events pour into a durable log; a stream processor rolls them up into windowed aggregates the instant they land; those rollups go to a store tuned to answer a dashboard query in milliseconds. Say that spine out loud and the system design interview becomes three decisions inside it: how to absorb the firehose without dropping data, how to aggregate a never-ending stream correctly when events arrive late and out of order, and how to serve queries sub-second over an enormous history while the numbers stay trustworthy. This walkthrough follows the path a strong candidate draws on the whiteboard — requirements, estimate, API and data model, high-level design, then the deep dives — naming the follow-up the interviewer uses at each step to tell a memorised answer from real understanding.

How the real-time analytics round is actually run

A real-time analytics round runs about 60 minutes on a shared canvas — you draw the boxes and arrows while you talk, and the interviewer interrupts. The extra time over a 45-minute round is not slack; it is because this problem has three substantial deep dives that each deserve real minutes, and a weak candidate spends the hour drawing one giant pipeline while a strong one budgets it across ingestion, aggregation and serving. The one-line problem opens onto deep water fast: the moment you draw "events go into a thing that counts them," the question becomes what counts as the right count, and every answer after that compounds from how you handle time, lateness and scale.

The interviewer rarely lets "we aggregate the events" stand. "Count the unique users in the last five minutes across a billion events a minute — exactly, or is approximate OK? Walk me through where that number actually gets computed and stored." That question — correctness and cost under a real firehose — is the one a copied answer can never rehearse, and the one this round is really testing.

The rounds here are modelled on how the loop runs, not scraped from any candidate's recall. What matters for prep is the shape: a blank canvas, a 60-minute clock, and a follow-up after every decision you make.

The five-step framework for any HLD round

Every high-level design answer fits the same spine. Say it out loud at the start so the interviewer knows you have a plan, then walk it:

1. Clarify requirements — functional and non-functional, and what you are explicitly not building.
2. Estimate — events per second and the burst factor versus query volume, to justify every later choice.
3. High-level design — the boxes and arrows: producers, the ingestion log, the stream processor, the serving store, and the query layer.
4. Deep dive — the two or three decisions that actually decide this system.
5. Follow-ups — bottlenecks, failure modes, and what you would change at ten times the scale.

What the debrief measures

The score is not "right or wrong design." A good HLD debrief reads on four dimensions, and every move below maps to one: problem solving (did you reframe it as pre-aggregate a stream, don't scan raw events, and reach for windows and approximation), communication (did you say in one sentence why approximate counts are fine for a live dashboard but a batch layer earns the exact numbers), technical execution (a partitioned log, a stream processor doing event-time windows with watermarks, rollups in an OLAP store, at-least-once plus idempotent writes), and time management (did you spend your 60 minutes across ingestion, aggregation and serving, not sink the hour into watermark edge cases). Reading a walkthrough teaches the design; it does not teach you to deliver it on the clock while someone pushes back.

Step 1 — Requirements: what a real-time analytics system must do

Spend the first few minutes here. Jumping to boxes before you have scoped the problem is the fastest way to design the wrong system.

The sentence that earns points: "This is a streaming aggregation problem, not reporting-over-a-table — I can't scan raw events per dashboard load at this volume, so I pre-aggregate the stream as it arrives and serve the rollups, and I'll decide per metric whether approximate-and-fast or exact-and-slightly-delayed is the requirement." Say that and you have framed the entire design before drawing a box.

Step 2 — Capacity estimation (back of the envelope)

Estimation here is about proving one thing: the load is a continuous write/aggregate stream, not a read workload — and that one asymmetry drives every later choice.

Two conclusions fall out, and you should say both out loud. First, because writes dwarf reads, you pre-compute aggressively: do the expensive aggregation once as data streams in, so every dashboard read is a cheap lookup of a rollup rather than a scan. Second, there are two regimes you must name: freshness (a live dashboard wants seconds-old numbers and can tolerate a small, bounded approximation) versus correctness (billing or official reporting wants exact numbers and can tolerate minutes of delay). Deciding which a given metric needs is what later justifies HyperLogLog over exact counting and lambda over a single pipeline. You have now pointed the interview at the real deep dives — ingestion, aggregation, and serving — before anyone asked.

Step 3 — API and data model

The API is deliberately small: producers push events in, consumers read pre-aggregated metrics out.

The data model separates the raw truth you can replay from the rollups you actually serve:

The point to state: raw events are kept only long enough to replay and reprocess; the rollups are the served truth, and they are what the storage layer is built to answer fast.

Step 4 — High-level design: ingest, process, store, serve

Now the boxes. Events flow left to right through four stages, with a batch path on the side for exact numbers:

• Producers and ingestion collector: clients, services and edge emit events to a thin collector that validates and forwards — it does no aggregation, just gets events onto the log fast.
• Ingestion log (Kafka): a durable, partitioned append-only log is the buffer between producers and processing. It decouples the two, absorbs bursts, and is replayable.
• Stream processor (Flink / Spark Structured Streaming): consumes the log partitions and computes windowed aggregates per key — this is where the real work happens.
• Serving store (OLAP / time-series — Druid / Pinot / ClickHouse): holds the pre-aggregated rollups in a columnar, pre-rolled layout built for fast range and group-by queries.
• Query service and dashboards: read rollups from the serving store; a live subscription pushes each window to dashboard tiles as it closes.
• Batch layer (optional, for exactness): reprocesses the raw log on a schedule to produce exact numbers that reconcile the fast approximate layer.
• Sharding: everything partitions by key — all events for a key land on one partition, so aggregation for that key is ordered and local.

Say explicitly that the design's whole job is to move the aggregation off the read path: pre-aggregate as data arrives so a dashboard query is a lookup of a rollup, never a scan of raw events. Naming that — a partitioned log feeding a stream processor that rolls up into an OLAP store — is the signal the interviewer is listening for.

Deep dive: ingesting the event firehose

Start at the front door, because if ingestion drops data the rest of the system is computing wrong numbers confidently. The follow-up is the spike: traffic jumps ten times in a flash sale — what happens to the events you can't process fast enough?

The answer is to put a durable, partitioned append log (Kafka) between producers and the stream processor, never to let producers write straight into the aggregator:

• Decouple producers from consumers: producers only have to append to the log, fast and fire-and-forget; the processor consumes at its own pace. Neither blocks the other.
• Absorb bursts as lag, not loss: when the firehose spikes past the processor's throughput, events pile up in the log and the processor falls behind — back-pressure becomes consumer lag, not dropped data. It catches up when the spike passes, because the data is safely durable.
• Replayable: because the log retains events with offsets, you can rewind and reprocess — essential for fixing a bad aggregation, backfilling a new metric, or recovering a crashed processor from its last checkpoint.
• Partition by key: the log is split into partitions keyed by the aggregation key, so all events for a key land in one partition in order, and the processor scales by assigning partitions across workers. Ordering is guaranteed per partition, not globally — which is all windowed aggregation per key needs.

The point to state plainly: the log turns a scary, unbounded, bursty input into something the rest of the system can consume calmly and re-consume if it has to. A spike is not an outage; it is lag that drains.

"Traffic spikes ten times in a flash sale — what happens to events you can't process fast enough?" Producers keep appending to the partitioned log, which is durable, so nothing is dropped — the stream processor simply falls behind and its consumer lag grows. When the spike passes, it catches up by reading the backlog; the dashboard is a few seconds staler for a while, but no event is lost and no count is wrong. The log absorbing the burst is exactly why I don't let producers write to the aggregator directly.

System Design
HLD · Media & analytics · 60 min
Design a real-time analytics system out loud, with follow-ups

Practise on Zynter →

Deep dive: stream processing, windows and approximation

Ingestion gets events in safely; this decides what the numbers actually are. It is the heart of the round and the richest source of follow-ups, because it is where correctness lives. The follow-up is the one that separates memorised answers from real ones: an event arrives two minutes late, after its window already closed — what happens?

The processor computes windowed aggregates — a metric per key over a bounded slice of time — and three decisions make them correct:

• Tumbling vs sliding windows: tumbling windows are fixed and non-overlapping (every minute, a fresh bucket) — the default for dashboards; sliding windows overlap (a trailing five-minute count updated every minute) when you need a smooth moving metric. Say which and why.
• Event time, not processing time: bucket each event by when it actually happened (its event timestamp), not when it reached the processor. A mobile client offline for a minute sends events whose event time is a minute old; counting them by arrival time would put them in the wrong window and make every historical number drift.
• Watermarks close windows: since you aggregate by event time but events arrive out of order, you need a signal to decide a window is done. A watermark is a heuristic — "I believe I've now seen all events with timestamps up to time T" — that lets the processor close a window and emit its aggregate. It is a deliberate trade between waiting longer (more correct) and emitting sooner (fresher).
• Late events past the watermark: an event that lands after its window closed is handled by an explicit policy you must name — drop it (fine for a live dashboard), route it to a side output for later reconciliation, or re-open and correct the window (needed for exact numbers). Never pretend late events don't happen.

Then the scale problem. At a billion events a minute, some aggregates won't fit in memory if computed exactly:

• Approximate distinct counts (HyperLogLog): counting unique users exactly means holding every id you've seen — gigabytes per window. A HyperLogLog sketch estimates cardinality in a couple of kilobytes with a small bounded error (~2%), and sketches merge, so per-partition sketches combine into a global count. For a live "unique users" tile, that trade is almost always right.
• Heavy hitters (count-min sketch): to find the top-K keys (hottest pages, biggest spenders) without a per-key counter for millions of keys, a count-min sketch estimates frequencies in fixed memory.
• Emit rollups: the processor writes pre-aggregated rollups — (metric, key, window, measures) — downstream, so the serving store never sees raw events.

The sentence that shows range: for a live dashboard I'll accept a HyperLogLog estimate within a couple of percent because freshness matters more than the last digit; if the same number feeds billing, I run a batch job over the raw log to get it exact and reconcile. That is the freshness-versus-correctness trade the round is testing.

"An event arrives two minutes late, after its window already closed — what happens?" It depends on the policy I chose, and I'd state it: aggregate by event time, use a watermark to decide a window is complete, and for anything past the watermark either drop it (acceptable for a live dashboard), send it to a side output to reconcile later, or re-open the window and emit a correction if the metric must be exact. The watermark is a tunable trade between freshness and completeness — I don't pretend events arrive in order.

System Design
HLD · Media & analytics · 60 min
Defend event-time windows and watermarks out loud

Practise on Zynter →

Deep dive: storage, serving and correctness (lambda vs kappa)

Aggregation produces the numbers; this decides how a dashboard reads them in under a second over a huge history, and how you keep them correct. The follow-up is the one about scale on the read side: the dashboard must render in under a second over a trillion rows — how?

The serving layer is an OLAP / time-series store (Druid, Pinot or ClickHouse), not a general-purpose transactional database, and the reasons are specific:

• Pre-aggregated and columnar: it stores the rollups, already grouped by key and window, in a columnar layout, so a range + group-by query touches only the relevant columns and segments — not a trillion raw rows. The expensive work was done once on the write path.
• Time-partitioned, hot and cold: data is partitioned by time, so a query for "last hour" scans a tiny recent segment; older data is rolled up to a coarser granularity (minutes → hours → days) and eventually expired, keeping the hot set small and fast.
• Built for this access pattern: these stores are tuned exactly for "aggregate a metric across a time range, grouped by a dimension," which is every analytics dashboard query, so they answer sub-second where a row store would table-scan.

Then correctness at the architecture level — name the two shapes and pick one:

• Lambda architecture: run a speed layer (the stream pipeline — fast, approximate) beside a batch layer (reprocesses the raw log periodically for exact numbers), and merge them at query time (recent from the speed layer, settled history from batch). It gives you both freshness and exactness, at the cost of two codebases computing the same logic twice.
• Kappa architecture: keep one streaming pipeline and, when you need to reprocess (a bug, a new metric, a correction), replay the log through it rather than maintaining a separate batch system. Simpler and less duplication; it leans on the log being replayable, which it is.

Finally, delivery semantics, because duplicates corrupt counts:

• Exactly-once or effectively-once: the processor's checkpointing can give exactly-once within the pipeline; where it can't, use at-least-once delivery plus idempotent writes — an upsert keyed by (metric, key, window) — so re-processing the same event or replaying after a crash overwrites rather than adds, and a duplicate never double-counts.

The point to state plainly: the read path is fast because all the aggregation already happened on the write path, and the system stays correct because the raw log is replayable and the writes are idempotent — a crash or a correction is just a reprocess, not a lost or doubled number.

The follow-ups that decide the real-time analytics round

Scan the probes from each deep dive and they rhyme. Master these five and you are ready for a version of the question that is not drawn exactly like this one.

• "Why not just run a GROUP BY over the raw events when the dashboard loads?" → Because at a billion events a minute that scan is impossibly slow and expensive per load. You pre-aggregate the stream as it arrives and serve cheap rollups; the read becomes a lookup, not a scan. (problem solving)
• "Exact or approximate counts?" → Approximate for live dashboards — HyperLogLog gives a distinct count within a couple of percent in kilobytes, which beats storing every id; exact only where it must be (billing), computed by a batch job over the raw log and reconciled. State the freshness-versus-correctness trade. (communication)
• "An event arrives late — what happens?" → Aggregate by event time, close windows on a watermark, and for anything past it apply a stated policy: drop, side-output to reconcile, or re-open and correct. The watermark trades freshness against completeness. (technical execution)
• "The dashboard must render sub-second over a trillion rows — how?" → Serve pre-aggregated, time-partitioned, columnar rollups from an OLAP store; the aggregation already happened on the write path, so the query touches tiny recent segments, not raw rows. (technical execution)
• "How do you keep numbers correct across crashes and reprocessing?" → At-least-once plus idempotent upserts keyed by (metric, key, window) so duplicates never double-count, a replayable log so reprocessing is a rewind, and lambda or kappa to reconcile the exact numbers. (problem solving)

Each maps to a debrief dimension. Reading this list teaches you the answers; it does not teach you to deliver them while someone interrupts — that only comes from saying them out loud.

How to prepare: a one-week plan

You do not need fifty systems. You need this one cold and the framework automatic, so the next "design X" is just new nouns in the same five steps.

• Days 1–2: Learn the five-step framework until it is reflex. Do the real-time analytics system end to end on paper — requirements, estimate, API, data model, and the ingest/process/store/serve boxes.
• Days 3–4: Drill the aggregation core: tumbling vs sliding windows, event time vs processing time, watermarks and late-event policy, and HyperLogLog vs exact counting. Say each tradeoff in one sentence.
• Days 5–6: Own serving and correctness — why an OLAP store answers sub-second, lambda vs kappa, and at-least-once plus idempotent writes. Answer each of the five follow-ups out loud.
• Day 7: A full timed 60-minute round, narrating the whole way, with someone (or something) interrupting. This is the rep that counts.

This post is the deep walkthrough behind one row of the broader Top 25 system design interview questions — once the real-time analytics pipeline is automatic, the same spine carries the rest of the list.

Frequently asked questions

How do I design a real-time analytics system step by step?

Follow the five-step HLD framework. Clarify requirements (ingest a high-volume event stream, compute windowed aggregate metrics per key, serve live dashboards and ad-hoc queries with seconds-level freshness, and decide per metric whether approximate-and-fast or exact-and-delayed is needed); estimate to show the load is write/aggregate-heavy and read-light, that raw retention is huge so you roll up and expire raw, and that the unit of work is the (key, window) tuple; define the small API (ingest_event, query_metric, stream_metric) over a raw-event log plus a rollup model; draw the design (producers → partitioned Kafka log → stream processor → OLAP serving store → query service, with a batch layer for exact numbers); then deep-dive ingestion, stream processing with windows and approximation, and storage/serving with lambda vs kappa. Finish on the follow-ups about late events and sub-second reads.

What is the difference between stream processing and batch processing here?

Stream processing aggregates events continuously the moment they arrive, so metrics are fresh within seconds but are computed on incomplete, out-of-order data and often use approximations. Batch processing reprocesses a complete, bounded set of raw events on a schedule, so it is slower but can produce exact numbers and backfill new metrics over all of history. A real-time analytics system usually needs both: the stream pipeline for the live dashboard, and a batch pass (or a replay through the same pipeline) to settle exact figures. The lambda architecture runs them side by side and merges at query time; the kappa architecture keeps one streaming pipeline and reprocesses by replaying the log.

What is a watermark, and how are late events handled?

A watermark is the stream processor's heuristic estimate that it has now seen all events with event timestamps up to some time T, which lets it close a time window and emit that window's aggregate. Because you bucket events by when they happened (event time) rather than when they arrived, and because events arrive out of order, you need this signal to decide a window is complete. An event that arrives after its window's watermark has passed is a late event, and you handle it with an explicit policy: drop it (acceptable for a live dashboard), route it to a side output to reconcile later, or re-open the window and emit a correction (for metrics that must be exact). The watermark is a tunable trade-off between freshness (emit sooner) and completeness (wait longer).

How do you count unique users across billions of events without running out of memory?

Use a probabilistic data structure instead of storing every id. A HyperLogLog sketch estimates the number of distinct values (cardinality) using only a couple of kilobytes per window, with a small bounded error of roughly two percent, where an exact set would need memory proportional to the number of unique users — gigabytes at this scale. HyperLogLog sketches also merge, so each partition computes its own sketch and they combine into a global distinct count. For finding the heaviest keys (top-K hot pages or users), a count-min sketch estimates frequencies in fixed memory. You accept a small, bounded approximation for live metrics because freshness matters more than the last digit; where an exact number is required, a batch job over the raw log produces it.

What is the difference between the lambda and kappa architectures?

Both aim to serve fresh metrics while still being able to produce exact, corrected numbers. The lambda architecture runs two layers: a speed layer (the real-time stream pipeline, fast but approximate) and a batch layer (reprocesses the raw log for exact results), merged at query time — powerful but it means maintaining two codebases that implement the same aggregation logic twice. The kappa architecture keeps a single streaming pipeline and, whenever it needs to reprocess for a correction or a new metric, replays the retained log through that one pipeline — simpler, with no duplicated logic, relying on the log being durable and replayable. Pick kappa for simplicity when one pipeline can meet the latency and correctness needs; pick lambda when a separate heavy batch computation genuinely earns its keep.

How do I practise a system design interview out loud?

Reading a walkthrough is not the same as defending a design while someone interrupts. On Zynter, the HLD track runs live voice rounds modelled on how real system design loops run — you talk through a shared design on the canvas, and a voice interviewer pushes with the follow-ups above one at a time, then gives you a debrief on problem solving, communication, technical execution, and time management. The score is a practice signal, not a hiring decision.

Practise it on Zynter

You can read what a watermark is. The round is won on defending the trade-offs out loud.

Pick the Real-Time Analytics round, draw the ingest-process-store-serve pipeline on the canvas, and defend it out loud to a voice interviewer that keeps asking "exact or approximate — and why?" and "an event arrives late — what happens?" — then read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies including the HLD system design track (Sep 2026).

Practise the Real-Time Analytics round →
Browse all interview tracks