We run twenty three applications this company genuinely depends on. In 2014 they sat on about a hundred and forty servers in two rooms. Nineteen servers are left, and the list that matters now is a list of forty seven suppliers.
In May our payslips did not arrive. Two thousand six hundred of them, because the bureau that runs our payroll uses a document delivery partner we have never contracted with, never assessed and could not have named that morning. My department had no account with them, no telephone number and nothing to offer the people asking. We were a customer of a customer.
Afterwards we wrote to seventeen material suppliers and asked who they depend on. Twelve answered in any useful detail. Of those, nine run on the same infrastructure provider and six in the same region of it, which means a single regional event takes our expenses platform, our human resources system, our carrier's booking portal and, pleasingly, our monitoring. Four use the same identity provider. Two use the same transactional mail service, and one of those two is how our customers are told an order has been dispatched. We had bought a diversity of vendors and accidentally bought a concentration of infrastructure.
This is the part of the next decade I find hardest to plan for. Everything I was taught about resilience assumes components I can see: a second circuit, capacity somewhere else, a failover that can be rehearsed on a Sunday. None of it applies to a dependency I cannot see, cannot test and am not permitted to switch. The work moves from designing systems to interrogating counterparties, and that is a different craft, practised by procurement people and lawyers as much as by engineers.
We have made a start. There is a register of our suppliers' suppliers, collected when a contract is signed and asked again each year. Concentration is a risk in its own right now rather than a footnote against individual systems. Three agreements require notice of a material change of subprocessor, and several suppliers declined to offer it, which is itself information. And our continuity exercises include a day on which the only available tools are a telephone and a manual process.
For most of my career the question was what our systems are made of. It is becoming who they are made of, and that is a list I still cannot finish.
– Serguey Shinder
