A while back I wrote about a scheduler of ours that did not run for six days while every check we had said it was fine. In the comments, to21as produced the sharper version of the problem, from their own system:
Ours ran every morning, logged that it ran, and collected zero rows for three days, because the machine's wifi is off overnight. systemd, the heartbeat and the dashboards were all green, because all three answer "did it run", and it did.
Three independent instruments. All green. All correct. And all answering a question nobody cared about.
That is worth separating from ordinary monitoring failure, because nothing was broken in the monitoring. The checks were accurate. They measured the machinery and reported honestly on it, while the outcome the machinery existed to produce was zero for three days.
The shape of it
A check tells you something true about the process. You read it as something about the result. Those come apart the moment the process can complete without doing anything, which is most of the time.
Once you look for it, the web is full of the same shape.
Your error tracker reported no errors this week. Is the application clean, or did the bundle stop including the init call, or the DSN get rotated, or a tightened CSP start blocking the ingest request? All four produce an empty dashboard. Only one of them is good news.
Your feedback form received nothing. Is nobody hitting problems, or is the submit endpoint returning 500, or the mail rule filing it somewhere nobody reads? Silence arrives identically down every one of those paths.
A funnel step shows zero conversions. Nobody is completing it, or the event stopped firing when the button was rewritten. The chart is the same chart.
In each case the instrument is installed, the dashboard loads, and the number is a real number. The number is zero, and zero is exactly what a working system with nothing to report looks like.
Zero is not a measurement
Here is the rule I would have wanted years earlier: a zero is information only if something independent proves the counter could have been non-zero.
Absent that proof, a zero is not a low reading. It is an absence of reading, wearing the costume of one.
to21as closed their own case by making each run record how many rows it wrote, and treating a completed run with zero rows as a failure rather than a success. The trace stopped asserting the execution and started asserting the collection. That works because the job was theirs: they could make it report on its own output.
The half you cannot fix that way
You cannot do it for anything a user initiates. Nothing in your system can log "a person hit a bug and did not tell us", because the event you want to count is the one that failed to arrive. There is no self-reporting silence.
For those paths the only honest instrument is a synthetic one. Send yourself the thing, on a schedule, and alert when it fails to land:
// Runs on a schedule, from outside your app. The point is not that it passes,
// it is that its absence is loud when it stops.
const probe = `canary-${Date.now()}`
const res = await fetch('https://example.com/api/feedback', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ message: probe, email: 'canary@example.com' }),
})
if (!res.ok) throw new Error(`feedback endpoint answered ${res.status}`)
// The half that actually matters: read it back out of the store your team
// reads from, not out of the response you just got.
const landed = await findInTracker(probe)
if (!landed) throw new Error('feedback accepted and never arrived')
The second half is the part people skip. A 200 from the endpoint proves the endpoint answered. It does not prove the report reached the place a human would look, which is the only thing you wanted to know and the exact distinction that made three green dashboards useless.
What to do with this today
Take the path you would most hate to be silently broken, the one where you would say "we would have heard about it". Ask what the last thing is that would have to happen for you to hear. Then ask what currently proves that thing happened, and be strict: a check on the step before it does not count.
If the answer is "nothing proves it", you do not have a quiet week. You have no information at all, and the two have been rendering identically on your screen for as long as you have had the dashboard.
Thanks to to21as, whose comment is the whole of the good idea here.
