You can't monitor what you never asked to be told

We had an SMS delivery monitor. It watched a delivery log file, parsed every entry, and was supposed to wake someone the moment a message failed. Dashboard, parsing, alerting — the whole apparatus existed and worked. We felt covered.

We were not covered. For weeks, the monitor had never seen a single delivery failure. Not because our delivery was perfect — it wasn't — but because the log file it so faithfully watched was permanently empty of the one thing it was watching for.

The log file was real; the data was fictional

Here's the setup. Our send script fired messages through the provider's API and logged them. A separate monitor tailed the delivery log, looking for failure statuses, ready to alert. Two halves of a pipeline, both built, both tested, both green.

But the send script never set the provider's StatusCallback on outgoing messages. That callback is the mechanism by which the provider tells you what happened after you send — delivered, failed, undelivered. Without it, the provider sends the message into the network and simply never reports back.

No callback meant no delivery notifications ever arrived. The monitor sat there parsing a log that would never contain a failure, because the failures had no channel to reach it. The monitoring was real and the data pipe was fictional. Everything downstream of the missing callback was decoration: parsing nothing, alerting on nothing, with full confidence.

The uncomfortable part: the monitor had been "working" for weeks. It parsed, it matched, it would have alerted — if there had ever been anything to alert on. A test run confirmed the alert path fired when fed a synthetic failure. Every check we ran passed, because every check we ran tested the machinery and none of them tested the input.

We verified the dashboard, not the pipe

This is the verification trap, wearing a different costume. We asked "does the monitor fire on a failure?" and answered it with a synthetic event. We never asked the harder question: "does a real send ever produce a real failure notification?"

Nobody traced the full loop once, by hand: send a message, watch the callback land, watch the monitor fire. The one end-to-end test that would have caught it — the boring, obvious, manual one — was the one we skipped, because each half looked solid in isolation. The send path worked (messages went out). The monitor worked (it fired on test data). The seam between them — the callback registration that made one visible to the other — was nobody's half.

Seams are where telemetry goes to die. Every monitoring story I've written has the same shape: the producer is fine, the consumer is fine, and the connection between them was never built. Alerting is a chain, and a chain with a missing link isn't a weaker chain — it's not a chain at all.

Ask to be told

The fix was one parameter: every send now includes the status-callback URL. Failures actually arrive, and they actually wake the monitor. The first real failure notification landing in the log was, perversely, a small celebration — proof the pipe existed.

Checklist

Verify the telemetry pipe end to end, not just the dashboard. Send a real message, watch the callback land, watch the monitor fire. The full loop, once, by hand.
Test with real events, not just synthetic ones. A synthetic failure proves the alert path works; only a real send proves the data path works. You need both.
Audit the seams. When two systems connect — send script and monitor, producer and consumer — the connection itself needs an owner and a test.
Distrust a monitor that's never fired. Weeks of silence can mean perfection or a dead pipe. The way to tell the difference is to generate a real failure and watch it arrive.
An alert on a metric that's never populated is worse than no alert. No alert means you know you're blind. A dead alert gives you the feeling of being monitored without the fact — and you'll plan around a safety net that isn't there.