Yesterday I argued that a zero is only information if something independent proves the counter could have been non-zero, and that for anything a user initiates the answer is a synthetic probe running from outside your app.
mickyarun replied with the part I had got slightly wrong, and it is worth more than the original:
the value is not that it is outside, it is that it produces a second number capable of disagreeing. A prober running in the same cloud region, resolving through the same DNS, trusting the same cert store, produces a number that agrees for exactly the reasons your first one does.
That is the correction. "Outside" is a proxy for the property you actually want, and a bad one.
The property you want is disagreement
Their example is payments, and it is worth stealing precisely because it is not clever.
A payment system has a counterparty. A second organisation independently counts the same events, and at the end of the day the two counts are compared. An attempt that never reached you still exists in the bank's number, so the discrepancy surfaces without anybody on your side having instrumented the failure.
Read that last part again. The instrument that finds the problem is a disagreement between two organisations that share no codebase, no clock and no deploy. Nobody wrote a check for it. The check is the existence of a second party with its own reasons to count.
Most web applications have no counterparty. That is not a gap in their monitoring, it is a structural fact about them, and it is why this whole class of failure stays invisible there.
The test
So here is the question to ask of any pair of instruments you are relying on:
Can these two fail independently? If not, you have one instrument printed twice.
It is uncomfortable how often the answer is no.
Two error trackers, both loaded as a <script> on your page. Belt and braces, except that a tightened Content-Security-Policy blocks both, an ad blocker blocks both, and a bundle that stops including your init calls drops both. You bought a second vendor and got the same measurement twice.
An uptime check running in the same cloud region as your app. It crosses the same network segments and egresses through the same path. A regional event takes out the thing and the thing that watches it, together, and the graph stays flat because the graph stopped being written.
A prober resolving through the same DNS resolver as your CI. A bad or stale record is invisible to both. They agree, confidently, for identical reasons.
Your application's own "request handled" log, next to your APM. Same process, same deploy, same crash. One instrument.
Each of those passes the naive test (there are two of them) and fails the real one (they die to the same cause).
How to actually check
Do not ask whether the second instrument is external. List what the two of them share, and be literal about it:
- The same region or datacenter
- The same DNS resolver
- The same TLS trust store or client library
- The same deploy or config push
- The same credentials or API key
- The same code path
Every shared item is a way they fail together, and a way your two green ticks turn out to be one green tick. The value of a probe is not its distance from your servers, it is the length of that list.
You do not need zero overlap, and you will not get it. You need to know the list, so that when both instruments agree you know what that agreement is worth.
The counterparty most web apps do have
Here is the part that changed how I think about our own product.
Most applications have no bank on the other side counting. But nearly all of them have something close: the people using it.
A person who tells you the site did not work is a genuinely independent measurement. Their machine, their network, their resolver, their browser, their cached bundle, none of it shares a failure mode with your monitoring. They are the one instrument in the entire system that can disagree with all of yours at once, which makes them the closest thing to a counterparty most teams will ever have.
And in almost every organisation, that instrument reports in the lowest-fidelity channel that exists. "It's broken." "It doesn't work on my end." No URL, no browser, no timestamp, no error text, arriving days late through three people.
That is a reading from the only sensor pointed at the failures your own sensors cannot see, and it arrives unreadable.
The instinct when monitoring fails you is to add another monitor. It is usually cheaper, and usually more informative, to improve the resolution of the one independent instrument you already have.
Thanks to mickyarun, whose comment is the entire argument here.
