We run a continuity exercise every January. The first two were sensible tabletop discussions about the order system, the finance system and what each department would do without them, and both were judged a success. This year the facilitator began differently. She said that our sign in service was unavailable, because that is how a genuinely bad morning tends to start, and asked me to begin.
We stopped after about twenty five minutes, because my department could not do anything at all.
The on call rota lives in our service management platform, behind that sign in. So do the supplier escalation contacts, with the account numbers you have to quote before anyone will talk to you. The runbooks are in a wiki behind the same authentication. The asset list that tells me which depot has which switch is in the same tool, as is the incident log we would have been writing. The conference bridge we use for a major incident requires a login. And the mobile numbers of my own engineers existed, for me, only in the company directory.
The business's side of the plan was in decent shape. It had simply been written on the assumption that my department would be present and working, which is an assumption nobody had noticed making, because every previous exercise had tested their half of the day.
What we keep now is unglamorous and costs nothing. Once a quarter we export the rota, the escalation list with contract references, twelve critical runbooks, a site and equipment summary and the department's own telephone numbers. Those go to four people, printed and on encrypted sticks, at home and at two sites. We declared a group on a consumer messaging application as the fallback channel, which I resisted for about three years and which took ten minutes to set up. And our own tooling is now classified like any business system, with a stated recovery time that somebody has tested.
I had spent a decade planning continuity on behalf of other departments. It had genuinely never occurred to me to ask what mine would be able to do.
– Serguey Shinder
