In Cassandra, a delete is a write.
That sentence sounds like trivia until it takes down a read path. DELETE doesn't remove anything. It writes a tombstone, a marker saying "this data is dead as of timestamp T." The actual removal happens later, during compaction, and only after a grace period. Between those two moments, your deleted data has negative value. It's gone from your application's point of view but still physically present, and every read that touches its range must process it.
Why the marker has to exist
Cassandra is a distributed, eventually consistent system. Suppose a delete simply removed data, and one replica was down when the delete happened. When it comes back, it still has the row, and to the rest of the cluster, that looks like data the others are missing. Repair would helpfully copy the deleted row back to everyone.
Deleted data returning from the dead is called zombie data, and tombstones exist to prevent it. The tombstone outranks the older value everywhere it's seen.
Which is why tombstones must survive for gc_grace_seconds, default 864000, ten days, before compaction may purge them. Ten days is the window you have to repair a down replica. Shrink gc_grace_seconds without shrinking your repair interval and you've quietly signed up for zombies. Repair at scale is its own post, and it's coming.
How reads pay
A read merges data across memtable and SSTables. Every tombstone in the requested range must be read, held, and reconciled against live data before Cassandra can answer. A partition that has accumulated a million tombstones makes you process a million dead cells to return whatever's alive.
Cassandra tells you when this is happening, in two escalating tones:
WARN ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted
The warning fires past tombstone_warn_threshold, 1,000 by default. Past tombstone_failure_threshold, 100,000, the read is killed mid flight with TombstoneOverwhelmingException. That's the database choosing to fail your query rather than let it OOM the node.
The thresholds are guardrails, not tuning knobs. Raising them treats the symptom and keeps the disease.
Where tombstone farms come from
Every one of these I've met in production.
- Queue like workloads. Insert, process, delete, repeat, in the same partition. The partition becomes a graveyard the consumer must scan past on every poll. Cassandra as a queue is the canonical anti pattern for exactly this reason.
-
Collection overwrites.
UPDATE t SET mymap = {...}replaces a whole collection, which writes a range tombstone over the old one first. Prefer additive updates,mymap = mymap + {...}. - Inserting NULLs. Binding null in a prepared statement writes a tombstone for that cell. ORMs and "just bind every column" code generate these invisibly. Use unset values, not nulls.
- TTL everywhere. Expired TTL cells become tombstones too. Fine if your compaction strategy is built for it.
Finding them, then fixing them
# per-SSTable estimate
sstablemetadata /var/lib/cassandra/data/ks/table-*/ *-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83
0.83 means an estimated 83% of that SSTable is purgeable tombstones.
For query level visibility, TRACING ON in cqlsh shows tombstone cells scanned per query. For a live table, nodetool tablestats and watch average tombstones per slice.
The durable fixes are modeling fixes.
- Time series with TTL, use TimeWindowCompactionStrategy. Whole SSTables age out and get dropped as files, so tombstone processing largely disappears. Compaction strategy trade offs are the next post.
- Stop deleting in place in hot partitions. Partition by time bucket and let whole partitions expire instead.
- Audit for null binding and collection overwrites. These are the tombstones nobody meant to write.
Deletes in Cassandra are cheap to issue and expensive to have issued. Model as if every delete is a small loan against your read latency, because it is, and gc_grace_seconds is the repayment schedule.
