Our expenses and purchasing platform was degraded for most of a Tuesday in May. We raised a ticket at fourteen minutes past nine. A human being replied at twenty to two. It was working again by seven in the evening, and to this day the supplier's explanation is one paragraph long.
What I remember is not the outage. It is the six hours in the middle, during which the total technical capability of my department consisted of reloading a status page and telling our own people that we had no news. Three directors rang me. I had nothing to offer any of them that they could not read for themselves, and I have never in twenty years felt less useful.
Our recovery plan says four hours for that service. That number was written in 2016, when the system ran on two machines in our own building and a colleague could go and stand next to them. Nothing about the promise was revised when the application moved, because the migration was assessed on cost, resilience and features, and recovery was assumed to improve rather than to change hands.
We went through all seventeen significant services afterwards and read the support terms properly, which took a fortnight and was the most useful fortnight of the year. Response times are committed for the first reply and not for a fix. Severity definitions are the supplier's and two of ours require a named contact to raise a high priority case, both of whom had left the company. Four services have no telephone route at all. One contract promises a target rather than an obligation, which is a sentence I had read before and never understood.
Two services moved to a support tier that costs real money. For three, the board has accepted in writing that a day without them is possible. Every recovery figure now states the supplier's committed response and our own time on top, so the honest number for that platform is fourteen hours rather than four, and the people who depend on it have seen it.
We are still accountable for the outage. We are no longer in a position to end one.
– Serguey Shinder
