Why You Can't Just Call an LLM for Every Kafka Event

The idea sounds clean: a Flink job consumes from a Kafka topic, and for every event it fires off an LLM call to detect something interesting, an anomaly, a sentiment signal, a risky pattern. Streaming agent, done.

Except at any meaningful throughput, that's a billing disaster waiting to happen.

The math hits fast

Say your Kafka topic is ingesting 50,000 events per minute from a financial feed. Even with a cheap model and tiny prompts, you're looking at tens of millions of tokens per hour. Token costs that feel trivial in a chatbot prototype become load-bearing infrastructure costs at stream scale. The model itself becomes the bottleneck, and the invoice.

This is the core tension nobody talks about enough: LLMs are designed to reason over context, but most individual events in a high-frequency stream carry almost no context on their own. Calling a model per event is like hiring a senior engineer to read every line of a server log in real time. It's overkill for 99% of the lines, and you can't afford it for the 1% that matter.

Complex Event Processing as a pre-filter

Steffen Hoellinger's approach at Berlin Buzzwords points at something practical: use Complex Event Processing (CEP) inside Flink to pattern-match across sequences of events before any LLM ever sees the data.

CEP is a mature technique. Flink has a CEP library that lets you define patterns, things like "three consecutive price drops within 10 seconds" or "order placed, then cancelled, then re-placed by the same user within a minute", and only emit a match when the full pattern fires. Most events get filtered. Only the stateful sequences that actually mean something bubble up.

Then you call the LLM. On the output of CEP, not the raw stream.

What this changes architecturally

Instead of: Kafka topic → Flink → LLM (per event) you get something closer to: Kafka topic → Flink CEP → filtered pattern matches → LLM → action

The LLM now operates on pre-compressed, semantically meaningful windows. You're giving it signal, not noise. Token usage drops dramatically. Latency actually improves because the model isn't being hammered by irrelevant calls. And you can be more generous with prompt context on the patterns that do fire, because you're not spending that budget on junk events.

The statefulness problem underneath

There's a subtler issue here too. LLMs are stateless by default, each call is independent. But the reason you're using CEP is precisely because individual events only carry meaning in sequence. CEP is doing the stateful heavy lifting that the model can't do on its own.

This is worth internalizing. CEP and LLMs aren't alternatives; they're complementary. CEP handles temporal pattern detection across the stream. LLMs handle reasoning about what a matched pattern actually means in context. Neither replaces the other.

Practical tradeoffs to consider

A few things worth thinking through if you're designing something like this:

Pattern latency vs. LLM latency. CEP patterns that span long time windows introduce their own delay. If you're waiting for a 5-minute pattern to complete before calling the model, your "real-time" response is already 5 minutes stale. Design patterns to be as tight as possible.

False negative cost. CEP filters aggressively, which means you might miss edge cases that don't fit predefined patterns. If your LLM is supposed to catch novel anomalies, a rigid CEP front-end can suppress exactly the events you care about. Hybrid approaches, CEP for known patterns, lightweight statistical filters for novelty detection, tend to work better in practice.

Pattern maintenance overhead. CEP rules need to be maintained as your domain evolves. In financial data especially, what counts as a suspicious sequence changes as market structure changes. That operational burden is real and often underestimated.

The broader principle

High-volume streams and LLMs have fundamentally different cost curves. Streams are cheap to move, expensive to reason over at full resolution. LLMs are cheap to call once, expensive to call millions of times. The engineering work is building the layer in between that compresses a stream into a set of moments actually worth reasoning about.

CEP is one tool for that. Windowed aggregations are another. Statistical anomaly detection is another. The pattern is the same: reduce before you reason.

If you're building streaming agents on top of Kafka and Flink, this framing might save you a significant infrastructure bill before you hit production.