A supplier sends us a price file every Tuesday. In February they changed the system that produces it, and the new one gave us prices in pounds where the old one had always given us pence. Fourteen hundred lines came in at a hundredth of their real cost. We sold against those prices for eight days, to about ninety trade customers, and the loss we could not recover was a little over thirty one thousand pounds.
Nothing in the chain objected, and that is the part worth writing down. The transfer completed. The file parsed. The row count was within the normal range. The load reported success and wrote a log entry saying so. Every check we had was a check on whether the mechanism had worked, and the mechanism had worked perfectly.
Compare that with a price one of our own staff types in. It sits outside a tolerance, it goes to a category manager, and if the change is large it needs a second approval. We built those controls carefully over about a decade, and every one of them faces inwards. A change fourteen hundred lines wide, arriving from outside, went through none of them, because data from a supplier arrives through a pipe rather than through a screen and a pipe is treated as infrastructure.
It was a depot manager who found it. He rang to ask why a ladder was showing at four pounds.
Inbound files now land in a staging area and are compared with the previous version before anything becomes live. A line that moves by more than a fifth, or a file that changes more than a twentieth of its rows, holds the whole load until the category manager approves the difference from a summary that shows it. That approval is a task in his list, not a message in a mailbox, because I did not want this to depend on somebody reading their email.
We also found four other feeds with no checking whatsoever, including credit limits and a carrier's surcharges.
Most of what our systems act on is now written by somebody else's system, and we had spent ten years governing the shrinking part that we type ourselves.
– Serguey Shinder
