A storage migration needed forty minutes with the primary database accepting no writes. We had a read only mode for exactly this, built two years earlier, exercised once in a drill, and we had told customers a week in advance that browsing and reporting would work and that checkout would not.
We turned it on at ten in the evening. Ninety seconds later we turned it off and took a full outage instead, because nobody could sign in and the site was returning 503s to everybody.
Sign in writes. It updates a last login timestamp and inserts a session row, and no amount of read only banner on the checkout page changes that. Worse, our health check endpoint wrote a heartbeat row, so within one interval every instance reported itself unhealthy and the load balancer removed all of them. We had built a mode that served a friendly page and then removed every server capable of serving it.
Underneath that were more. The audit log wrote a row on every read, by design, because reading a customer record is an auditable event. The rate limiter kept its counters in the same database. The feature flag library persisted evaluation counts, which means the flag that enables read only mode required a write to be read.
None of this is exotic and none of it was hidden. It is simply that read only mode had been designed at the level of features, by people thinking about checkout and refunds, while the writes that actually mattered live in the platform layer that every request passes through and nobody owns.
We rebuilt it against evidence rather than intention. Staging was pointed at a genuine read only replica and the whole test suite was run against it, which produced a list of thirty one distinct write paths, twenty three of which were in shared libraries. Sessions became signed cookies. Audit events go to a queue that buffers. Rate limit counters moved out of the primary. The health check stopped writing anything.
Then the part that matters most. Read only mode runs in production for ten minutes on the first Tuesday of every month, during the quiet window, whether we need it or not. The first two of those found something.
We thought we had three states: working, degraded and down. We had two, and the middle one was a document.
– Sergey Shinder
