The request was simple. "There's about 2TB of expired data in this collection. Can you delete it tonight?"
The honest answer was no, and being able to explain why, not just refuse, is most of the job. A missing TTL index had let short lived data pile up for months in one of our busiest collections. The team wanted one big deleteMany overnight. Here's what that would have actually done.
Why a mass delete hurts, and it isn't locking
People reach for "locks" as the explanation. Wrong database era. WiredTiger uses document level concurrency, so a huge delete doesn't freeze the collection. The damage arrives through three quieter channels.
1. The oplog becomes a firehose. Every deleted document is an individual entry in the oplog. Delete 500 million documents and you've written 500 million oplog entries, which every secondary must pull and apply. Replication lag climbs. If lag exceeds what the oplog window can hold, a secondary falls off the oplog entirely and needs a full resync. Now your delete has cost you redundancy.
2. Cache churn. To delete a document, WiredTiger loads its pages into cache. A 2TB sweep drags cold data through a cache that was carefully full of hot data. Your working set gets evicted, p99 latency on unrelated queries climbs, and someone opens an incident that will never mention the word "delete."
3. Acknowledgment pressure. If your writes use w:"majority" (they should, see the write concern post), the delete's progress is gated on those same struggling secondaries. Everything compounds.
I demonstrated exactly this in a lower environment. Kicked off the naive delete, watched rs.printSecondaryReplicationInfo() lag grow, and the "tonight" deadline converted itself into a plan.
The pattern: chunk, sleep, watch the lag
const CHUNK = 10000;
const SLEEP_MS = 500;
const cutoff = ISODate("2026-01-01");
let total = 0;
while (true) {
const ids = db.events.find({ created: { $lt: cutoff } },
{ _id: 1 }).limit(CHUNK).toArray().map(d => d._id);
if (ids.length === 0) break;
db.events.deleteMany({ _id: { $in: ids } });
total += ids.length;
sleep(SLEEP_MS); // let the oplog breathe
if (total % 1000000 === 0) print(`${total} deleted, lag check...`);
}
Two operational rules around it.
Run in off peak windows only. We ran nightly windows and paused at business hours. Duration was about a week and a half for the full 2TB. Nobody noticed, which was the definition of success.
Watch replication lag as your throttle. If lag climbs past your comfort line, raise SLEEP_MS. The loop's speed limit is the cluster's health, not your patience.
sleep() between chunks looks unbearably conservative to developers. That is the point. The delete has no deadline once the growth is stopped, and the growth is stopped by the real fix.
The real fix: TTL, so this never becomes a project again
db.events.createIndex({ created: 1 }, { expireAfterSeconds: 2592000 }) // 30 days
Details worth knowing about the TTL monitor, because they surprise people.
- It wakes every 60 seconds, so expiry is approximate, not instant. Fine for cleanup, wrong tool for business logic that needs precise expiry.
- It deletes in background batches, effectively running my chunked loop forever, gently. On huge backlogs it can lag, which is why we cleared the 2TB manually first and let TTL own steady state.
- Index the field the data actually ages by. TTL on the wrong date field is a slow motion data loss incident.
We closed it with a runbook, the chunk script, the lag thresholds and the window schedule, so the next 2TB request is a lookup, not a debate.
The fastest way to delete 2TB is slowly.
