We lost the primary database of our order system at twenty past eight on a Thursday morning, during a storage migration that had been through three reviews. The restore from snapshot took thirty eight minutes, which is the figure we had been quoting in continuity documents for two years, and we were briefly quite pleased with ourselves.
Customers could not search for a product until twenty to six that evening.
Our search index is built from the database by a pipeline, and it was not in the backup set, because it is derived data and everyone knows derived data can be rebuilt. It can. It took nine hours and twenty minutes, limited by how fast the indexer may read without hurting a database that was also trying to serve the site. Nobody had ever run that job from empty against production volumes. The last time anyone measured it, the catalogue was a third of the size.
There was more underneath. The cache was cold, so the database took about six times its normal read rate for the first hour and we had to shed traffic to keep it standing. A materialised view behind the account pages needed a refresh, which is forty minutes by itself. Our stated recovery objective was two hours, and we had been rehearsing the first thirty eight minutes of it.
The drill is what changed. It runs to the point where the system is serving normal traffic at normal latency, not to the point where the database accepts connections, and the stopwatch does not stop in between. The index is snapshotted alongside the database and restored in parallel. The rebuild path still exists and is now resumable and able to serve partial results instead of nothing. A warm up step replays a day of the most common queries before we take traffic back.
The first drill under the new definition took four hours and ten minutes. That was an uncomfortable number to write down, and it was the first honest one we had ever had.
A restore is a step. Recovery is the moment the people who use the thing can use it again, and we had never measured those two with the same clock.
– Sergey Shinder
