The pickers at our largest depot told us the handheld scanners had become slow. They were right. Each scan was taking about one and a half seconds to come back, against a third of a second three years ago, and in a shift of nine hundred scans that is twenty minutes of somebody standing still.
Nothing was broken. The application had moved to a provider whose data centre is twenty miles from that depot. The scanner's traffic went from the warehouse over a private circuit to head office, through a firewall and a web filter, out of the single internet connection in that building, across to the provider, and all the way back again. We had built a network in 2016 in which every site reached the world through one room, and in 2016 that was the correct design, because everything the company used was in that room.
It is not any more. Of the systems this business genuinely runs on, most are now somewhere else entirely, and we were paying for private circuits to carry internet traffic to a building in order to inspect it in a place where almost nothing lives.
The bill for that was the smaller half of the problem. In February a fault on the head office connection lasted a little over three hours, and every site in the company lost access to applications that were healthy the entire time, hosted by suppliers with excellent availability figures, unreachable because of a cable in a town none of those suppliers had heard of.
Putting it right took most of a year and was not purely technical. Each site now breaks out to the internet locally, with a second connection over mobile at the smaller ones. Filtering and inspection are delivered as a service instead of a box, which meant rebuilding rules that had been tuned for nine years, and we deliberately left the depots on the old path for six months until we could prove the new one behaved. Scan times are back to a third of a second and our circuit spend is down by about a third.
A network is a set of assumptions about where things are. Ours was written down once, was true when written, and nobody's job description included asking whether it still was.
– Serguey Shinder
