A storage controller failed on a Tuesday afternoon and took our order system down for five hours. The outage itself was handled properly. What turned it into a fortnight of clearing up was everything that carried on running while we worked.
The integration that sends consignments to our carrier retried patiently throughout, and when the database came back part way through its restore, it dispatched around two thousand seven hundred duplicate requests, several hundred of which became real vans at real addresses. The overnight reconciliation ran at eleven as usual, found thousands of mismatches in a database that was still being rebuilt, and helpfully corrected them. Customer despatch notifications went out a second time. A self healing rule restarted the application every four minutes, which is why the engineer trying to capture a diagnostic could not get one.
None of that was a fault. Every job did exactly what it was written to do, with good logging and a green result, because each of them had been given a start condition and none of them had ever been asked whether it should be acting on a system that was broken.
We run somewhere over sixty automated jobs, built up over three years by people including me, and until that Tuesday there was no answer to the question of how you stop them. Not a documented one, not a tested one, not even a list. You went looking for schedules in four different consoles while the incident was live, which is precisely when nobody has the attention to do it.
There is a procedure now. Each job declares the service it belongs to and whether it is safe to run while that service is degraded, which took a dull fortnight of asking authors questions they had never considered. Disabling a group is one command, and bringing it back is an ordered list that is verified rather than remembered. Quiescing is the second item on the major incident checklist, after declaring one. We rehearse it twice a year, and the first rehearsal found two jobs that can only be stopped from a vendor's console, one of which is the vendor's own.
We spent three years asking what our automation could do for us and never once asked what it would keep doing while we were not looking.
– Serguey Shinder
