Our backups stopped running on a Sunday. We found out eleven weeks later — from a restore test, not an alert. The backup job had been "temporary" when someone wrote it in 2023, and a deploy had quietly rewritten the crontab without it. The server was green. Our monitoring was green. The backup just... wasn't there.
That's the shape of cron failure nobody plans for: the job that silently stops running writes nothing at all. No errors, no logs, no exit code. Just an absence that compounds daily until the day you need it.
Exit codes are a smaller safety net than you think. The classic monitoring setup greps logs for errors — which catches the job that errors. It misses three green-looking failures:
- The hung job. No exit, no error — stuck on a dead NFS mount since Tuesday, holding a lock.
- The useless success. Exit 0, zero rows processed, zero files found. The job "ran."
- The drifted schedule. DST moved it an hour; a deploy moved it a day. Still green, increasingly wrong.
The fix that catches silence: heartbeats.
One line at the end of every critical job, on success only: curl a monitor URL (free healthchecks-style services exist) or touch a timestamp file. Then one cron that verifies every heartbeat is younger than its threshold — and alerts when it isn't. That's it. The rule: a job without a heartbeat is a job you're not allowed to trust.
The fix that catches uselessness: alert on output.
Log the counts — rows processed, files found, bytes moved — and alert on zero. A backup job that copies nothing should page somebody, even though it exits 0.
The fix that catches drift: own the schedule.
One inventory of every scheduled thing: system crontabs, systemd timers, CI cron, provider-side schedulers, and the "temporary" scripts in someone's home directory. Each entry gets an owner and a "what breaks if this doesn't run" line. If nobody can explain why a schedule exists, it's one refactor away from disappearing — or one audit away from being the thing you needed.
The 15-minute version, today:
- List your critical jobs — backups, syncs, digests, alerts. Five minutes.
- Add a heartbeat line to each. Ten minutes, including the monitor URL.
- Add the silence check with a threshold per job cadence.
- Test it once: disable a job on purpose for an hour and confirm the silence alert fires. An untested silence alert is a wish, not a control.
That last step is the one teams skip, and it's the one that matters. We've seen the "we have monitoring" conversation end badly exactly twice: once when the alert didn't exist, once when it existed and had never fired.
The full checklist — heartbeat patterns per job class, runtime thresholds, the schedule inventory template — is free:
→ https://hive80-lab.github.io/ops-notes/cron-job-monitoring-checklist.html
And if the reason these checks never get installed is that everyone's busy firefighting, that's the exact problem the Automation Starter Pack ($19) solves — pick-first workflows for heartbeats, retention sweeps, and the weekly schedule review. When a check does fire, the Ops Starter Kit ($14) has the incident side covered. Launch-week pricing: 30% off with code HIVE-LAUNCH30 at https://hive80lab.gumroad.com
Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes
