One of the quietest, most expensive problems I keep running into is not a system that is down. It is two systems that are both up, both confident, and both reporting a different version of the truth. The CRM says the customer is on the premium plan. The billing system says they are on the free tier. Support trusts one, finance trusts the other, and the customer is caught in the middle of a disagreement that nobody realizes is happening, because each system looks perfectly healthy on its own.
This happens because data gets copied, and every copy is a promise you have to keep in sync forever. A field gets duplicated into a second service to save a lookup. A nightly job syncs records between two databases. An integration writes the same fact in two places. Each of these seemed reasonable when it was built, and each one quietly created a second source of a truth that used to live in one place. Now, when they drift apart, and they always eventually drift apart, there is no principled way to say which one is right, because both were designed to be authoritative.
The discipline that prevents this is deciding, explicitly, where each important fact actually lives, and treating every other copy as exactly that, a copy, subordinate to the source and never allowed to be edited on its own. When the billing system and the CRM disagree about the plan, there should be no debate about which one wins, because one of them was declared the owner of that fact long ago and the other merely reflects it. Ambiguity about ownership is the root of the disagreement, and you fix it by removing the ambiguity, not by building yet another reconciliation job to paper over it.
Reconciliation jobs, in fact, are usually a sign you have already lost this battle. If you need a nightly process to detect and repair the divergence between two systems, you have accepted that they will diverge, and you are just cleaning up after it on a schedule, hoping the window between drift and repair never lands on the wrong customer at the wrong moment.
Decide where the truth lives. Make everything else defer to it. Two confident systems that disagree do not average out to something reasonable. They just give everyone permission to believe whichever one they already trusted.
– Serguey Shinder
