What 3 Database Race Conditions Taught Me About Distributed Locks
When you start building backend systems with asynchronous workflows, everyone tells you about race conditions. You think, "Sure, I'll just throw a Redis lock on it and call it a day."
That's exactly what I thought, until production proved me wrong—three different times, in three different ways.
I was working on a system where synchronous APIs, async webhooks, and background workers were all trying to read and write the same MongoDB documents at the exact same time. The root problem was always the same: two writers, one document. But I quickly learned that there is no silver-bullet fix. The right way to handle a race condition depends entirely on what's actually happening inside that split-second execution window.
Here are the war stories, the trade-offs I had to make, and what this experience actually taught me about concurrency.
Lesson 1: The Stale Snapshot (Or, why you shouldn't lock across network calls)
What happened:
I had an API endpoint that opened a MongoDB transaction, read a document, and then made a synchronous call to an external vendor. That vendor call was slow—sometimes taking up to 15 seconds. I had put a Redis lock on this operation, but I gave the lock a Time-To-Live (TTL) of 10 seconds.
You can guess what happened. At second 11, the lock expired. An asynchronous webhook arrived, saw the lock was free, updated the document, and committed. By the time my slow API got its response at second 15 and tried to commit, MongoDB threw a WriteConflict. The document had changed underneath the transaction's initial snapshot.
How I fixed it:
My first instinct was to just make the lock longer, but out-guessing network timing is a losing game. Instead, I decided to catch that specific WriteConflict. When it happens, I abort the stale transaction, take a fresh, un-cached read of the document, and only write what is actually still missing.
The Trade-off:
I had to accept a microscopic residual race window. In the split-second my reconciliation logic reads and writes, the webhook's transaction could theoretically conflict instead.
Lesson 2: The Fast Local Update (The "Silent Timeout" Trap)
What happened:
A specific record could be updated by two different api's hitting at the exact same time. But unlike the first case, there was no slow external network call in the middle. It was just a local update.
How I fixed it:
Because this was completely internal, wrapping it in a short-lived Redis lock (with a 1-second TTL) keyed to the resource ID worked for time being.
The Trade-off:
The danger here isn't concurrency; it's future maintenance. Hardcoding a 1-second lock TTL without a massive warning sign in the comments is a trap. Ultimately, this was just a rapid production hotfix that we later migrated to async processing.
Lesson 3: The Blast Radius Mismatch (Locking the wrong thing)
What happened:
Incoming webhooks always carried a specific ID (let's call it the CHILD ID). Naturally, I set up my workers to lock on that CHILD ID before processing.
The problem was my code didn't just update the Order. It updated shared state across several collections tied to a higher-level PARENT ID. Since one Plan could have multiple Orders, two webhooks for different Orders processed at the same time, grabbed two different locks, totally bypassed each other, and collided while writing to the shared Plan state.
How I fixed it:
The ID on the payload is rarely the ID you actually need to lock on. I had to add a step to resolve the CHILD ID up to its owning PARENT ID first, and then lock on that.
The Trade-off:
This introduced a slight delay in processing of webhooks due to async nature, but we were fine with eventual consistency.
My Final Takeaways
If I had to boil down everything I learned into a few rules of thumb for my future self:
- Lock only what actually needs serialization When we take Locks it should only be for specific purpose just adding it as cover for multiple processes is a potential issue waiting to happen.
- Async processing for the win Not everything has to be realtime in distributed systems we should encourage off-loading processing to async queues leveraging its strength is key.
- Lock scope our lock should always be scoped from an all-inclusive system POV. If you only lock the specific child record you are working on, but your transaction updates shared parent resources, you are just locking the front door and leaving the back door wide open.
- Name your trade-offs out loud. Every concurrency fix leaves some edge case on the table. Whether it's a residual window or a hardcoded timeout, that's fine—just make sure you write down why you accepted it.
