I have a question I ask any team that tells me their data is safe: when did you last restore from a backup, end to end, into a working system. The confident answers are rare. The usual answer is some version of "the backups are running, we get the success emails." That is not a backup strategy. That is a hope strategy with good email deliverability.
The uncomfortable truth is that a backup is not a thing you have. It is a thing you can do. Until you have actually restored it, you do not know if the file is complete, if the format still imports, if the restore fits in your maintenance window, or if the runbook for doing it even exists. Plenty of teams discover on their worst day that the backup job silently started failing months ago, or that it backed up the schema but not the data, or that restoring takes eleven hours when the business expected one.
I once inherited a system with a beautiful automated backup pipeline. Green checkmarks everywhere. The first time we needed it, we learned the backups were encrypted with a key that lived only on the server that had just died. Technically we had backups. Practically we had encrypted noise. Nobody had ever tried a restore, so nobody knew.
Since then I treat restore drills as non-negotiable, the same way a serious team treats fire drills. On a schedule, we take a real backup and restore it into an isolated environment, and we time it. The drill validates three things at once: that the backup is usable, that the procedure is correct, and that the recovery time actually meets what we promised the business. All three drift silently if you never check.
The drill also surfaces the human gaps. The first time we ran one, it took two people most of a day and a lot of guessing, because the knowledge was scattered. The fifth time, one person did it in an hour from a runbook. That improvement is the whole point. You do not want to be learning the recovery procedure while the business is down and everyone is watching.
Test your restores. Everything else is faith.
– Serguey Shinder
