Could a New Teammate Follow Your Runbook Page Alone?

Your teammate starts their first shadow shift on Monday. On Sunday night you open the runbook page for the disk alert, to check it's ready. It has one line: "Check the usual stuff and clean up." You know what "the usual stuff" means. They won't.

A runbook is a short page that says what to do when one alert fires. Google's SRE book says that writing best practices down ahead of time, in what it calls a "playbook", gave roughly a 3x improvement in MTTR (mean time to repair, the average time to fix a problem) compared with improvising. That only works if someone new can follow the page alone.

Start with one alert, not the whole server

An alert is a message a monitoring tool sends when something crosses a line you set, like "disk more than 90% full". On call means you're the person who answers those alerts during your shift.

Pick one alert. One page covers one alert. If you try to write "everything about the photo server", the page gets long, and nobody reads it at 3 a.m.

Write the page in five parts

1. What the alert is telling you. One or two sentences, in plain words.
2. Who notices. Which users feel it, and how.
3. The first 3 checks. Exact commands, ready to copy.
4. A safe fix. Only a fix that can't make things worse.
5. When to escalate. To escalate is to call in someone with more context. Give clear numbers, not "if it looks bad".

Add a "last tested" line at the bottom. You'll see why below.

Try it on a toy service

Here is a page for a made-up service called thumbs. It makes small preview images of uploaded photos. It keeps them in a cache folder (a store of copies it can rebuild) and writes logs to /var/log/thumbs.

# Alert: thumbs-disk-high

## What the alert is telling you
The /var disk on a thumbs server has been over 90% full for 10 minutes.
At 100%, thumbs can't save new previews, and new uploads fail.

## Who notices
People uploading photos. Old photos still show.

## First 3 checks
1. How full is it now?
   df -h /var
2. Which folders are biggest?
   sudo du -xh --max-depth=1 /var | sort -h | tail -n 5
3. Is it the cache or the logs?
   sudo du -sh /var/lib/thumbs/cache /var/log/thumbs

## Safe fix
If the cache is the biggest folder:
   sudo /opt/thumbs/bin/clean-cache --older-than 7d
The cache only holds copies. thumbs rebuilds a preview when someone asks for it.

## Escalate when
- the disk is above 97%, or
- the biggest folder is neither the cache nor the logs, or
- the clean script fails, or frees less than 10%.
Escalate to: the thumbs owner on the team rota.

## Last tested
(date and name of the person who tried it cold)

Here's what each check does, so you can explain it in your own page:

• df -h /var shows how full the disk holding /var is. The -h flag prints sizes like 12G instead of long numbers, per the GNU coreutils manual.
• du adds up how much space folders use. -x skips folders on other disks, and --max-depth=1 stops after one level, so you get a short list.
• sort -h sorts sizes like 2G and 300M in the right order. tail -n 5 keeps the last five lines. The last one is the total for /var itself, and the four above it are the biggest folders.

The clean-cache script is part of the toy service. On a real system, only list a fix that your team has already agreed is safe.

Hand it to someone who hasn't seen it

This is the real test. Ask a teammate who has never worked on thumbs to follow the page on a test server.

1. Fill a test disk on purpose. Write a big dummy file into the cache folder, so the alert fires. Then run touch -d '8 days ago' on that file, so the 7-day clean in the safe fix can remove it.
2. Give them only the page. No hints, no chat.
3. Sit on your hands and take notes. Write down every place they stop, guess or ask a question.
4. Time it. Note how long the first 3 checks take.

Fix what they tripped on

Every stop in your notes is a gap in the page. Common ones:

• A command needs sudo (run as admin) and the page didn't say so.
• A path is wrong on the test server.
• "Escalate to the owner", but the page doesn't say how to find the owner.

Fix each one, then fill in the "last tested" line with the date and their name. When the service changes, the page needs another cold test.

Where this fits at DevGrid Staffing LLC

DevGrid Staffing LLC works in IT staffing for SRE, DevOps and CI/CD teams. People who join an on-call rota lean on runbook pages from their first shift, so we care about pages that work without the author in the room.

One thing to try today

Open one runbook page your team owns. Read it as if you'd never seen the service, and circle every word or step a new teammate would have to ask about.

Written at the DevGrid Staffing LLC workbench for people on SRE and pipeline teams · devgridstaffing.com