One worker at a time across many machines
Day 152 of 155
👉 Full deep-dive with code examples
The Coffee Shop Key
One restroom, one key, tied to a wooden spoon so nobody pockets it. Whoever holds the spoon has the room; everyone else waits.
Now the awkward part: someone walks out still holding it, and the restroom stays shut all afternoon.
Distributed locks solve that with an expiry date.
Three Copies, One Job
Your nightly billing job runs on three instances for availability. At midnight all three wake up and start billing the same customers.
A language-level mutex will not help. Each process has its own memory. The lock has to live somewhere all three can see.
Taking the Key
One atomic operation that checks and claims together:
SET lock:billing worker-7 NX PX 45000
-
NXmeans "only if the key is absent" - the race is settled in one step. - The value
worker-7is your owner identity. -
PXsets a lease, so a crashed holder does not block the job forever.
Release by comparing first, usually in a small Lua script so check and delete are one operation:
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
end
A plain DEL would wipe out someone else's lock once your lease had expired.
The Honest Limit
Your process can freeze - garbage collection, a network hiccup - after the lease expires and before your write lands.
The lock service took no part in that pause, so it cannot rescue you.
That is why careful systems add a fencing token: a number that increases on each acquisition, checked by the storage layer so a late writer is turned away.
Idempotence is a separate defence, not a stronger one. It stops a repeated operation from doing damage.
It does nothing about a stale one: if your delayed write sets qty = 6 while newer work already set qty = 4, replaying it safely still leaves the row wrong.
In One Sentence
A distributed lock is a shared, expiring claim that lets one worker act at a time, and it is only as safe as the resource behind it.
🔗 Enjoying these? Follow for daily ELI5 explanations!
Making complex tech concepts simple, one day at a time.
