Support pasted a screenshot at 9:40am: a customer's invoice page showed a balance of $0.00, and a second tab showed $412.60. Same account, same minute. No deploy that morning. The database was right. Our cache was wrong, and nothing in the system was responsible for telling it so.
We had a TTL. That is the part that stings. Every key expired in five minutes, which felt careful at the time, and for two years it was fine because nobody reconciled two tabs at once.
A TTL is a bet, not a correctness mechanism
A TTL says: I believe this answer stays true for N seconds. That is a guess about your write patterns, disguised as a config value. It has no idea that invoices.balance.8842 was just invalidated by a payment webhook. When a TTL-driven cache is wrong, it is not broken — it is behaving exactly as designed and answering a question about the past.
The honest framing: a cache is a stored copy of a decision you already made. The database decided the balance was $412.60. Our cache held a copy of the decision that said $0.00, made before the payment landed. The only interesting question is which writes make that copy wrong, and which code paths know about those writes.
If you want to know how much staleness you actually tolerate, measure it. Write the timestamp of the cached write into the value, and log now - write_time on every read of a key that later turns out stale. That distribution tells you what your TTL is really costing. Do not guess the number; sample it.
The write path is the only thing that knows
Reads cannot invalidate. A read has no idea whether another writer is mid-transaction. The write path — the one place that changes the underlying decision — is the only code that knows the old answer is now garbage.
We ran cache-aside: the reader did GET, missed, queried Postgres, then SETEX. It looked clean. The bug was ownership. Nobody owned invoices.balance.{id}. The reader created it. The payment service, which actually changed the balance, had never heard of the key. So the payment service wrote to Postgres and the cache kept serving its stale copy until the TTL swept it away.
Write-through inverts that. The writer updates the store and the cache in one place, so the key has an owner:
def record_payment(account_id, amount, conn):
with conn.transaction():
conn.execute(
"UPDATE invoices SET balance = balance - %s WHERE account_id = %s",
(amount, account_id),
)
row = conn.execute(
"SELECT balance FROM invoices WHERE account_id = %s", (account_id,)
).fetchone()
# The writer owns the key. Reads never create it.
redis.set(f"invoices.balance.{account_id}", row["balance"], ex=300)
redis.publish("invoice.changed", str(account_id))
The ex=300 is now a safety net for a missed write, not the correctness story. That is the difference. When invalidation is owned by the writer, the TTL becomes a backstop you hope never fires.
Be careful with delete-on-write too. DEL then re-populate races: a reader can miss, read the pre-commit row, and SETEX the old value back on top of your fresh write. Delete outside the transaction and re-populate from the committed row, or version the key.
Expiry is a coordinated attack on your database
The other failure nobody plans for: everyone's key expires at the same moment. A popular key with a 300-second TTL, written at 10:00:00 during a batch job, expires at 10:05:00 for every reader at once. All of them miss, all of them hit Postgres, and the database that was comfortably handling 200 reads per second suddenly takes the full load.
Three things helped. Jitter the TTL so keys written together do not die together. Collapse concurrent misses onto a single in-flight fetch so one request rebuilds and the rest wait on it. And for the hottest keys, refresh in the background before expiry so the miss never happens in the request path. Test this deliberately: expire your top key by hand during peak traffic and watch the database.
Make invalidation an event, not a cron
The fix that finally held was deleting the sweeper job. We had a cron that walked recent invoices and refreshed their cache entries every five minutes. It was a second source of truth about when data changed, and it was always slightly behind the writes it was chasing.
Instead, the payment service publishes invoice.changed with the account id. A subscriber deletes the balance key. The domain already had a name for this moment — a payment was recorded — so the event existed before the cache did. The cache became a consumer of a fact the business cared about, not a timer with an opinion.
That reframing is the whole job. Stop asking what TTL to pick. Ask which writes make this answer wrong, and who in your code knows they happened. If the answer is "a cron job, eventually," you have a staleness budget, not a cache.
What the comments added
@compoundlabs found the hole in the record_payment snippet above, and it is worth fixing rather than hand-waving: redis.set runs after with conn.transaction() closes, so the commit and the invalidation are two separate failure domains. A crash in between leaves the old balance in the cache, and the only thing that saves you is ex=300 — which means that window was still a staleness budget, just a short one. I presented the TTL as a backstop while quietly depending on it.
The fix is a transactional outbox. Write the intent to invalidate in the same transaction that changes the balance, and let something else publish it:
def record_payment(account_id, amount, conn):
with conn.transaction():
conn.execute(
"UPDATE invoices SET balance = balance - %s WHERE account_id = %s",
(amount, account_id),
)
conn.execute(
"INSERT INTO outbox (topic, payload) VALUES (%s, %s)",
("invoice.changed", json.dumps({"account_id": account_id})),
)
A relay tails outbox, publishes invoice.changed, then marks the row sent. The intent to invalidate now commits atomically with the fact that made it necessary, so there is no window where the database moved and nothing was told.
Delete, do not set. Outbox delivery is at-least-once, so the consumer has to be idempotent. Deleting a key is; SETting a recomputed value is not, because a delayed duplicate can put an older balance back on top of a newer one. That is the strongest argument for invalidating instead of writing — and as a bonus, a consumer that only deletes does not care about duplicates or out-of-order delivery at all. The moment you cache a value derived from the event, you owe each key a version and an "ignore anything older than what I already applied" rule.
The window does not close, it moves. Reads during relay lag still see the old value, so the thing to alert on is relay lag and rows piling up in outbox — not cache hit rate. Keep the TTL as the backstop for a relay that is simply down, but stop describing it as the mechanism.
If you cannot add a table to that database, make the read safe instead: version the cached value from a column the write already bumps, and have the reader refuse a value older than the row it can see. That is an outbox in disguise, and it needs no relay.
I write about production failures in Postgres, queues, and distributed systems.
