Why Benchmarking Stream Processing Is Harder Than It Looks

A workshop paper quietly dropped at TPCTC this August and it's worth paying attention to: SECBench, a benchmarking framework specifically designed for stream processing across the sensor-edge-cloud continuum.

Most people in the data engineering world would say, "we already have benchmarks for streaming." And they'd be partially right. But SECBench is pointing at something different, and the gap it's filling is real.

The benchmark gap nobody talks about

Traditional stream processing benchmarks, think Yahoo Streaming Benchmark or NEXMark, were designed with a relatively tidy assumption: your data lives in one place (usually a broker like Kafka), and you measure throughput and latency from there.

The sensor-edge-cloud model breaks that assumption immediately. You have data originating at the edge, sometimes on constrained hardware, moving through intermediate aggregation layers, and finally landing in cloud infrastructure for deeper processing. The latency profile at each hop is different. Resource constraints at the edge are different. And the failure modes are genuinely different from anything a pure-cloud benchmark captures.

Why this matters beyond IoT

It's easy to mentally file "sensor-edge-cloud" under IoT and move on. But the same architectural pattern is showing up in places that don't call themselves IoT at all.

Crypto infrastructure is a good example. On-chain event listeners, block indexers, and price oracles are all operating in a continuum: data originates at the protocol layer (think of the node as your "sensor"), gets preprocessed by indexers and RPC endpoints (your "edge"), and then lands in cloud-based analytics or ML pipelines. Benchmarking only the cloud layer tells you almost nothing about the actual end-to-end latency your users experience when a liquidation event triggers.

Financial market data follows the same shape. The "sensor" is an exchange feed. The "edge" is a co-located normalizer. The cloud is where the models run. The bottleneck is almost never where you think it is until you measure the whole path.

What good benchmarks actually need to capture

SECBench's framing is useful because it forces you to think about a few things that single-layer benchmarks ignore:

Resource heterogeneity. A benchmark that runs the same workload on uniform cloud VMs tells you nothing about what happens when your edge node is an ARM device with 2GB of RAM.

Network variability. Sensor-to-edge links are not your datacenter's 10Gbps backbone. Benchmarks that ignore variable, higher-latency links are benchmarking a fiction.

State management across tiers. Where does your windowing state actually live? If you're doing a 5-minute rolling aggregation and your edge node buffers data during a connectivity gap, what happens to your stream when it reconnects? This is an operational reality that most benchmarks don't touch.

End-to-end vs. component latency. Optimizing your Flink job's internal throughput while ignoring the 800ms edge-to-cloud hop is a very easy way to fool yourself with good numbers.

The broader point

The fact that a TPCTC workshop paper is specifically addressing multi-tier stream processing benchmarking is a signal that the community is catching up to where real deployments already are. Most production streaming systems are not single-tier. They never were. Benchmarks that assume they are have been measuring the wrong thing for a while.

If you're designing or evaluating a streaming pipeline today, it's worth asking which tiers your benchmark actually covers, and being honest when the answer is "just the last one."