The scariest deploys I have ever run were not code deploys. They were schema migrations. Code you can roll back by shipping the previous version. A migration that has already rewritten or dropped data has, in the worst case, nowhere to roll back to. Once the column is gone, it is gone, and "just revert it" stops being a sentence that means anything. I learned to treat database changes as a fundamentally different, more dangerous category than shipping application code.
The mistake I see most is treating a migration like any other change in the pull request: write it, review the diff, let the pipeline run it on deploy. That works fine right up until the migration locks a large table for the duration of the change, and every query that touches that table queues behind the lock, and a routine "add an index" turns into an outage while the whole application waits. On a small table nobody notices. On the table that has been growing for five years, the same statement is a landmine.
What I do now is treat migrations the way a surgeon treats an operation: plan it, stage it, and never do the irreversible part in one blind step. Additive changes first. Add the new column, backfill it gradually in the background, deploy code that can read from both the old and the new shape, and only once everything is confirmed working do you remove the old one, as a separate change, later. It is slower and it is several steps instead of one. It is also the difference between a migration you can pause when it misbehaves and one that has already committed you before you notice anything is wrong.
Expand and contract, done in separate deploys, is the pattern that has saved me the most grief. The expand phase adds the new structure without breaking the old. The application runs happily on both. The contract phase, days later, removes the old structure once nothing depends on it. At no single moment is the system in a state you cannot back out of.
Application code is reversible and forgiving. Data has memory and holds grudges. Respect that difference, move in small reversible steps, and you stop having the deploys that turn into all-night recovery jobs.
– Serguey Shinder
