A client pinged me on a Saturday morning: "your site is down."

A client pinged me on a Saturday morning: "your site is down."

It wasn't. Their site was fine. Their DNS was down — and that distinction is the whole lesson. Nothing in their own logs, nothing in their own monitoring, nothing on their dashboards. The thing that broke sits one layer above every request they serve, so every tool they own said everything was fine while every customer saw a "server not found" page.

The three-part trap:

1. The monitor that watches the site can't watch DNS.
Your uptime monitor asks for https://www.yourapp.com, gets an answer, marks green. But it only got an answer because DNS worked. The moment DNS breaks, your monitor is blind — the exact tool you trusted has nothing to check. The fix is embarrassingly cheap: resolve your apex, www, and one API host against 8.8.8.8 and 1.1.1.1 every five minutes, plus a daily check on the domain expiry date and that the nameserver set is still what the registrar says it is. Three checks, a five-line cron script, and you catch the outage before the tweet does.

2. The second nameserver is configured in peacetime or it doesn't exist.
When we dug in, their "DR plan" for DNS was a sentence: "we'd move to another provider." Moving DNS providers means re-entering or importing every record — at a second provider — during an outage. A, B, MX, TXT records for SPF/DKIM/DMARC, the CAA records nobody remembers. During the fire, every re-typed record is a fresh chance for a typo to become outage #2. The fix is boring: a zone export stored outside the provider, a free second DNS provider with the same zone already live, and a quarterly check that the copy is still current. Rehearsed, the restore is: flip NS at the registrar, dig +trace, done — ten minutes. Unrehearsed, it's hours and apologies.

3. TTLs are lowered before changes, not during fires.
The most human mistake in the whole story: the first thing they did when DNS broke was drop TTLs to 60 seconds. Doesn't help — resolvers obey the TTL that was in effect when they cached the record. Your 3am panic edit doesn't reach caches that already stored the old answer. Lower TTLs 24–48h before planned migrations; during an incident, republish correct records and let caches age out.

The 10-minute restore path, as a checklist:

1. Confirm scope (2 min): dig the failing names against two public resolvers and one ISP resolver. All fail + registrar portal slow → provider-side outage, not your config.
2. Decide the path (1 min): provider down → switch NS to the second provider. Own mistake → restore the exported zone to the primary.
3. Execute from the written runbook (5 min): exact NS strings, exact export filename, exact verification command. Typing nameservers from memory under pressure is how typos become second outages.
4. Verify and tell people (2 min): test from a phone on mobile data — not office Wi-Fi with its own cached answers — then post a status update. Even two lines. Silence turns a technical incident into a trust incident.

That last one stings every time. DNS was broken for 40 minutes; customers found out via a competitor's reply tweet two hours later. The DNS came back. The trust didn't.

The full runbook — the three upstream checks, the zone-export discipline, the TTL rules, and the 10-minute restore path — is free:

→ https://hive80-lab.github.io/ops-notes/dns-outage-runbook.html

If you want the incident layer this plugs into — the first-30-minutes card, severity matrix, and the comms templates we used to tell customers before they told us — we packaged it as the Ops Starter Kit ($14), and Vol. 2 ($27) adds the DR plan and evidence log for the review after. Launch week: 30% off with code HIVE-LAUNCH30 (ends tonight, Sep 11 23:30 ACST) at https://hive80lab.gumroad.com

Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes

And if the boring-but-critical jobs (the DNS resolvability check, the domain expiry sweep, the quarterly zone-export test) keep slipping because everyone's firefighting, that's the exact problem the Automation Starter Pack ($19) is built for — pick-first workflows so the checks run by themselves.