We muted our own alert channel on a Friday. By Monday, nobody noticed we had.

We muted our own alert channel on a Friday. By Monday, nobody noticed we had.

That's the part nobody warns you about with monitoring: it fails silently, in the direction of more silence. The phone channel chimes all week with nonsense — single blips, CPU spikes that mean nothing, one ping from a free-tier monitor that lost its wifi — and nobody tunes the thresholds. Someone just mutes the channel. And then the real alert fires into a channel that has been muted for two weeks, and you find out from a customer.

We didn't buy better tooling to fix this. We bought a checklist, and it took one afternoon. Here's what stuck.

Five signals catch 90% of small-team outages

1. Is the service up — from outside? One HTTPS request per minute to your most important URL, from a box that isn't yours, checking the status code and page content, not just the TCP port. A port that answers with a 500 stack trace is not "up." Expect 10–20 false alarms a year. Still cheaper than one four-hour silent outage.

2. Is the disk filling up? The #1 silent killer of small-team servers. Alert at 80%. At 100%, everything fails at once: the database stops writing, your logs vanish, and the postmortem has no evidence.

3. Did the job actually run? Every cron job that matters pings a dead-man's-switch service when it finishes; alert when the ping is late. "The backup failed silently on Tuesday and we noticed on Thursday" is the standard small-team horror story. A free heartbeat service kills it.

4. Are certificates and domains expiring? An expired SSL cert or lapsed domain is a 30-second fix that produces a full outage. Check at 21 days out. Put renewals on a calendar that isn't owned by one person.

5. Does the login path actually work? A synthetic "can a user log in and do the one action that pays us" check. This catches what the others can't: service up, ports open, auth provider silently rate-limiting you.

The one alert rule that doesn't get you muted

Every phone-channel alert carries a runbook line — one sentence saying what to do. "Disk 82% — run du -x / | sort -h | tail, likely /var/log." If the person paged asks "so what do I do?", the alert was misconfigured, not the person.

And the corollary we learned the hard way: any alert that pages someone and requires no action gets fixed or downgraded within a week. Non-actionable alerts aren't facts about your servers. They're bugs in your configuration. One blip is weather; two consecutive failures is data. Yes, that doubles detection time — from one minute to two. Worth it. A phone channel that fires five times a week will be muted by Friday. One that fires once a month, with the fix attached, is the whole point.

Then we added a 10-minute weekly review: count the pages (more than ~5 a week means thresholds are wrong), check the disk curve rather than the threshold, audit that every important cron job has a heartbeat and an owner, sweep expiries, and answer one question: if the phone fired tonight, would the on-call person know what to do? If the answer is no, that's the week's fix.

What you can skip for now

You don't need Prometheus, ELK, or an observability engineer. Those are tools for teams with more servers than people. Five signals, a dead-man's switch, and one disciplined weekly review will beat an abandoned Grafana dashboard on every metric that matters: time-to-detect, time-to-restore, and whether the team trusts the alerts at all. Migrate to real dashboards when the five signals genuinely can't see a failure you've had — not before.

The free version

The full checklist — the five signals, the alert-rule table, the weekly review, the "what to skip" list — is on our ops notes site, next to the on-call rotation and disaster recovery guides it plugs into.

If you'd rather not assemble it yourself: the Ops Starter Kit Vol. 2 ($27) includes the full monitoring pack — the 5-signal checklist, the alert-rule table, the weekly review card — plus the advanced incident response and continuity templates. Launch week: 30% off any paid kit with code HIVE-LAUNCH30 at checkout.

And for the part that shouldn't need a human remembering it at all — health checks, heartbeats, log rotation, the weekly expiry sweep — the Automation Starter Pack ($19) covers the pick-first workflows that make monitoring boring.