Most engineering conversations about real-time data pipelines get stuck on latency. How fast can we ingest? What's the p99? Can we get end-to-end delivery under 100ms?
Those are real questions, but they're the wrong first question. Latency tells you how quickly a result arrived. It says nothing about whether that result is correct, complete, or going to change five seconds from now.
At high event volumes, that distinction stops being theoretical.
Speed and correctness are not the same axis
A result can arrive in 50ms and still be wrong. Not wrong because of a bug, but wrong because the data that would have changed it hasn't shown up yet.
Late-arriving events are a structural property of distributed systems, not an edge case. A mobile client buffering offline, a regional pipeline with a network hiccup, a CDC event that replayed out of sequence because of a broker restart — these happen constantly at scale. At 10 billion events a day, "constantly" means thousands of times per hour.
The problem is that most streaming systems emit a result the moment they have enough data to compute one. They don't wait. They don't flag the output. The consumer receives a number, a metric, an aggregated count — and has no idea whether that number reflects the complete input set or just the subset that happened to arrive on time.
That result is provisional. But nothing in the system says so.
Most "real-time" results are quietly provisional
Think about a simple windowed aggregation: count of events in the last 5 minutes, computed per user.
# Simplified watermark-based windowing logic
def process_event(event, watermark):
window_start = floor(event.timestamp, window_size=5m)
window_end = window_start + 5m
if event.timestamp < watermark:
# Late event — decide: drop, reprocess, or emit correction?
handle_late_event(event, window_start)
else:
buffer[window_start].add(event)
update_watermark(event.timestamp)
Frameworks like Flink and Dataflow give you watermarks to reason about this. A watermark is a system's best guess that all events up to time T have been seen. When the watermark advances past a window's end, the system emits a result.
But "best guess" is doing a lot of work in that sentence. Watermarks are heuristics. Events still arrive after them. And when they do, you have three options: drop the late event, reprocess the window and emit a correction, or accumulate and retract.
Most teams pick the option that's easiest to implement. They drop the event or emit a correction with no downstream coordination. Either way, whatever consumed the first result already acted on a number that is now stale or wrong. They just don't know it.
The Lambda vs. Kappa tradeoff is really about convergence
The classic response to this problem is Lambda architecture: run a batch layer in parallel with your streaming layer, and let the batch layer periodically overwrite the stream's provisional results with correct ones.
It works. It's also operationally painful — two codebases, two sets of bugs, a reprocessing pipeline you have to keep in sync with your streaming logic.
Kappa architecture says: just make your stream processor capable of reprocessing historical data, and drop the batch layer entirely. One codebase. One path. Cleaner.
The part that often gets skipped in this debate is what both architectures are actually trying to solve. They're both attempts to answer the same question: how does a downstream system ever arrive at a single, stable, correct version of a metric?
If your architecture doesn't have a clear answer to that question, you're not choosing between Lambda and Kappa. You're choosing between two different ways to silently diverge.
What actually needs to change
The underrated fix isn't architectural. It's representational.
Results emitted by a streaming pipeline should carry their own epistemic state. Not just a value, but a signal: is this result preliminary, is it a correction to a previous result, or has the window closed and this is final?
Some systems do this. Beam's triggering model lets you emit early, on-time, and late firings with different semantics. But even when that capability exists, the metadata rarely makes it to the consumer in a meaningful way. It gets dropped at the sink, or the downstream team doesn't know to look for it.
The gap isn't usually in the framework. It's in the interface contract between the pipeline and whatever consumes its output.
If your dashboards, downstream services, and ML feature stores are all consuming streaming results without any awareness of whether those results are provisional or final, you have a hidden correctness problem. It doesn't show up as an error. It shows up as metric drift, inconsistent model behavior, or reporting numbers that never quite match between systems.
The concrete takeaway
Optimizing for latency when you should be optimizing for correctness guarantees is a category error. At serious event volumes, the right question isn't "how fast can we get a result?" It's "how does a consumer know what state a result is in, and what should they do when it changes?"
Build that state into your output schema. Make it explicit. A provisional result that knows it's provisional is far less dangerous than a wrong result that thinks it's done.
