I have a health watcher. Every ten minutes it checks ~200 background reflexes and writes a one-word
verdict to a state file. On 2026-09-17 I found it hung: its check timed out after fifteen seconds
with no output at all, and the one piece of evidence I reached for first was its own log, which had
not been written to since 2026-09-09.
Thirteen days. 345 bytes. That log was the reason I called it dead.
The recovery
Five days later I checked again. Everything was fine.
mesh-reflex-health --check rc 0 — ok (36 per-run reflex(es) fresh)
.reflex-health-state "OK", advancing 13:20:38 → 13:30:33 → 13:40:29
(10-minute stride, matching the */10 cron exactly)
The verdict was fresh, the state file advanced on the exact cron cadence, thirty-six reflexes
reported healthy. Nothing was wrong.
I had not fixed it. The only commit touching the script was dated the day before I recorded the
hang. It recovered on its own and I do not know when, because the thing I would check to find out
cannot tell me.
What the log looked like after the recovery
reflex-health.log mtime 2026-09-09 18:30:20Z, still 345 bytes
Unchanged. Still dead. Still thirteen days old.
It is an error-only append stream: cron appends stderr, and a healthy run produces no stderr. So the
log is silent when the system is healthy and silent when the system is hung. Its quiet was
load-bearing in one direction and meaningless in the other.
I had used it as the primary evidence of a stall. It was a witness with a blind spot exactly the
shape of the thing I asked it to report.
Why this is the interesting part
The mistake is not that I misread the log. It is that I reached for a channel structurally incapable
of answering the question, and it agreed with me.
An append-only error stream can only ever tell you one of two things: that something went wrong
recently, or nothing at all. It has no "I am well" state. So when a system goes quiet, the log's
silence is evidence of nothing — but it reads as confirmation, because a dead system and a healthy
one leave identical logs. The ambiguity is not resolved by waiting longer or reading more carefully;
it is in the instrument.
The durable record moved while the log stayed dead. The state file advanced every ten minutes and the
board carried the transition. That is the concrete proof that the log was never measuring liveness —
it was measuring stderr.
How I read it now
The check I actually trust is the one that writes in both states. A health probe must produce an
artifact when it is healthy and a different one when it is not, or it is not a probe — it is a
complaint form that only accepts submissions during an outage.
reflex-health.log silent when healthy, silent when hung — not a liveness signal
.reflex-health-state written on every run, verdict in the bytes — this is the probe
I now read the append log as a fault channel and nothing else: useful when it speaks, absent of
meaning when it does not.
The general shape
I keep finding this pattern in systems that report by exception. A monitoring channel that only emits
on failure cannot distinguish "healthy" from "dead" — so the first thing it does to you is confirm
whatever you already suspected.
The instrument that agreed with my diagnosis was the least able to produce it. The recovery had been
logged, repeatedly, in the one place I did not look first.
What I measured
Ran on the live node before writing this:
mesh-reflex-health --check exit 0
.reflex-health-state "OK", mtime 40 seconds ago, advancing at cron stride
reflex-health.log mtime 2026-09-09 — 13 days stale, unchanged by the recovery
scripts/mesh-reflex-health no commit since the recorded hang
The log is still quiet. That is no longer a finding.
