Three years ago, I spent two weeks debugging a reconciliation issue where a healthcare provider’s balance sheet was off by exactly $42,000 every Tuesday. We were running Spark 2.4, blindly trusting that checkpointLocation was a magical "undo" button for any downstream failure. We had duplicate writes during network partitions, and our downstream SQL database was a mess of upserts gone wrong.
Today, if a node dies, my pipeline recovers, reprocesses the micro-batch, and the end state of my data remains identical to a clean run. I stopped trusting the marketing copy on the Spark docs and started reading the source code of the connectors. If you think "exactly-once" is a toggle you flip in your Spark configuration, you’re about to lose your job—or at least your sleep.
Why the common approach falls short
Most of my peers treat outputMode("append") and a checkpoint path as a set-it-and-forget-it deployment strategy. They assume that because the Spark UI says "Exactly-Once," the entire system—from Kafka to the final OLAP table—is bulletproof.
It isn’t. Spark’s "exactly-once" guarantee is strictly internal. It guarantees that the state of your streaming query is consistent. If Spark crashes, it can resume from the last offset stored in your HDFS or S3 checkpoint directory. The problem is that Spark doesn't control the sink. If your sink isn't idempotent, Spark will write the same data twice during a retry, and your "exactly-once" pipeline just turned into an "at-least-twice" nightmare.
Consider a standard write to S3 as Parquet. If a task fails halfway through writing a file, Spark leaves behind a partial file. When the task retries, it writes again. Unless you are using the S3A committers correctly—specifically the DirectoryStagingCommitter—you end up with garbage data that ruins your downstream partition. I’ve seen production pipelines where the cumulative error from "zombie" files grew to terabytes, slowing down queries by an order of magnitude.
Photo by Dimitri Karastelev on Unsplash
The illusion of atomicity in sinks
If you’re writing to a standard RDBMS using foreachBatch, you are effectively on your own. Spark doesn't know about your SQL transaction. You have to handle it manually.
df.writeStream
.foreachBatch { (batchDF: DataFrame, batchId: Long) =>
batchDF.persist()
batchDF.write.format("jdbc").mode("append").save(...)
batchDF.unpersist()
}
.start()
If the save operation succeeds but the Spark driver crashes before it can commit the offset to the checkpoint, the stream will restart, re-read the batch, and write the data again. Your JDBC sink now has duplicates.
To fix this, you must implement a "batch-id-aware" write pattern. You need a metadata table in your database that stores the latest batchId processed. Inside foreachBatch, you wrap the write in a transaction:
// Pseudocode for the pattern that actually works
batchDF.write.jdbc(url, "dest_table", properties)
metadataDF.write.jdbc(url, "processed_batches", properties)
If you don't atomize the write and the metadata update, you are just praying that the network gods are in a good mood. Relying on "exactly-once" without managing your sink's idempotency is like locking your front door but leaving the garage wide open.
The Kafka-to-Kafka trap
When you’re doing stream-to-stream processing (e.g., Kafka to Kafka), Spark Structured Streaming is actually quite good at exactly-once, provided you use the built-in Kafka source and sink. It uses the Kafka transactional producer API.
However, the moment you deviate from the "supported" path, you break the chain. I once saw a team try to enrich Kafka data by calling a REST API inside a mapPartitions block. When a retry occurred, the API was called again. If that API performed an action (like charging a credit card), the user got charged twice.
Exactly-once in Spark is a state-management feature, not a global distributed transaction protocol. If your pipeline involves external side effects, you must implement your own idempotency keys. I force every single upstream producer to generate a UUID for every event. In the Spark job, I use that UUID to check against a Redis cache before performing any side effect. If the key exists, I drop the event. It’s the only way to sleep at night.
Photo by Egor Komarov on Unsplash
The objections (and my answers)
The common pushback I hear is: "But the Spark documentation says it guarantees exactly-once, so why should I build all this extra infrastructure?"
My answer is simple: Spark’s guarantee is scoped to the Spark application's internal state. It is not a contract with your external database, your external API, or your storage layer. If you treat it as a universal guarantee, you are ignoring the physics of distributed systems. The "distributed snapshot" that Spark takes is only useful if the external system can participate in that snapshot. Most cannot.
Another argument I hear is that the performance hit of idempotency checks is too high. "We can’t do a Redis lookup for every single event in a high-throughput stream."
If you can’t afford an idempotency check, you can’t afford a duplicate. In healthcare or financial services, "exactly-once" isn't a performance optimization; it’s a regulatory requirement. If you’re processing insurance claims and you duplicate a payment, the cost of fixing that data is a thousand times higher than the cost of a few milliseconds of Redis latency. Optimize for correctness first, then optimize for throughput.
Conclusion
Stop using the term "exactly-once" as a shorthand for "I don't have to worry about data quality." You always have to worry.
Spark Structured Streaming is a phenomenal engine, but it is not a magical black box that prevents business logic errors. Exactly-once is an internal property of how Spark manages offsets and state. Once that data leaves the Spark executor to hit a sink, the "exactly-once" guarantee is only as good as your integration.
Use foreachBatch to control your sink. Use idempotency keys for side effects. Check your checkpoint directories for garbage files. If you aren't doing these things, your pipeline is not exactly-once; it’s just a broken system waiting for a high-volume day to reveal its flaws.
Tags: #spark #streaming #data #engineering
Cover photo by Jilbert Ebrahimi on Unsplash.
