Your On-Call Runbook Should Be an Executable Playbook, Not a PDF

The first thing you reach for during an incident should not be a wiki page that is already out of date, but a runbook that behaves like code, with tests, version control, and an escape hatch for the moments your brain goes blank. I have spent enough 3 AM shifts decoding alerts with a flashlight in one hand and a mouse in the other, and the pattern that finally worked was turning every runbook entry into a small, executable rule set. In this post, I will show you a lightweight YAML-based runbook parser that maps alerts to first commands, enforces a freeze rule, and optionally asks a free-tier model to fill in the gaps when you are stuck.

Why a PDF runbook fails you at 3 AM

A static document is a snapshot of someone's knowledge, and by the time you open it, the cluster may have moved on. You need a runbook that can be executed, or at least filtered by the current alert, the time of day, and the deployment freeze window that your team promised the business. The usual alternative is to keep a mental model of every service, but that does not scale when you are sleep-deprived and the alert title is an acronym only the SRE team invented.

The executable playbook pattern

The core idea is simple: describe each response plan as a YAML entry with four fields — patterns, first_commands, escalation, and freeze_rule. The patterns are regular expressions matched against the alert title, first_commands is a list of shell commands you can run immediately, escalation defines who to page after a certain timeout, and freeze_rule tells the script whether this entry is allowed during a change freeze. Here is a complete example for a database connection pool exhaustion alert:

# runbook.yaml
freeze_windows:
  - name: 'holiday_freeze'
    starts: '2026-12-24T00:00:00'
    ends: '2026-12-26T23:59:59'

entries:
  - name: 'DB connection pool exhausted'
    patterns:
      - '.*db.*pool.*'
      - '.*connection.*timeout.*'
    first_commands:
      - 'kubectl get pods -l app=api | grep -i pending'
      - 'kubectl logs -l app=api --tail=200 | grep ''conn pool'''
    escalation:
      contact: 'db-oncall@example.com'
      after_minutes: 15
    freeze_rule: deny

Notice that the freeze rule is not a moral suggestion, but a hard gate that your script can check before it prints a restart command. If you are on call during a freeze and this entry says deny, the tool will show you only read-only diagnostics and remind you to escalate instead.

A lightweight parser that turns alerts into actions

Now let's write a small Python script that consumes this YAML. The script accepts an alert title as an argument, finds the matching entry, checks the freeze context, and prints the commands you should run. It is deliberately short, because the last thing you want at 3 AM is a 400-line monster that needs its own debugging session.

#!/usr/bin/env python3
import os, re, sys, yaml
from datetime import datetime

def load_runbook(path='runbook.yaml'):
    with open(path) as f:
        return yaml.safe_load(f)

def is_in_freeze(runbook, now):
    for w in runbook.get('freeze_windows', []):
        start = datetime.fromisoformat(w['starts'])
        end = datetime.fromisoformat(w['ends'])
        if start <= now <= end:
            return w['name']
    return None

def match_entry(runbook, alert):
    for entry in runbook['entries']:
        for pattern in entry['patterns']:
            if re.search(pattern, alert, re.IGNORECASE):
                return entry
    return None

def main():
    if len(sys.argv) < 2:
        print('Usage: oncall.py [alert title]')
        sys.exit(1)
    alert = sys.argv[1]
    rb = load_runbook()
    entry = match_entry(rb, alert)
    if not entry:
        print('No local runbook entry found. Trying AI support...')
        return
    freeze = is_in_freeze(rb, datetime.now())
    print('Alert: ' + alert)
    print('Runbook: ' + entry['name'])
    if freeze and entry.get('freeze_rule') == 'deny':
        print('Freeze window ' + freeze + ' active - restart commands are blocked.')
        contact = entry['escalation']['contact']
        minutes = entry['escalation']['after_minutes']
        print('Escalate to ' + contact + ' within ' + str(minutes) + ' minutes.')
    else:
        print('First commands:')
        for c in entry['first_commands']:
            print('  $ ' + c)

if __name__ == '__main__':
    main()

This script does not require any third-party request library, just pyyaml for parsing. You can install it with pip install pyyaml, then run python oncall.py 'DB pool exhausted on api' and you will get a short, action-oriented checklist instead of a wall of documentation.

Where the freeze rule fits in

The freeze_rule field is what separates this toy from something your change management team will respect. Instead of relying on everyone remembering the freeze calendar, the script checks the current time against the windows defined at the top of the YAML. That way, a restart or a deploy command is not even suggested when it would violate a freeze, and your escalation path takes over immediately. You can also add an allow rule for entries that are safe during freezes, like increasing logging verbosity or collecting metrics.

Testing the playbook before the pager fires

A runbook without tests is just a hope with YAML formatting. The beauty of the executable pattern is that you can assert the behavior of your matching and freeze logic with a few lines of pytest:

from datetime import datetime
from oncall import load_runbook, is_in_freeze, match_entry

def test_freeze_blocks_restart():
    rb = load_runbook('runbook.yaml')
    assert is_in_freeze(rb, datetime.fromisoformat('2026-12-25T12:00:00')) == 'holiday_freeze'

def test_alert_matches_entry():
    rb = load_runbook('runbook.yaml')
    assert match_entry(rb, 'DB pool exhausted on api')['name'] == 'DB connection pool exhausted'

Run pytest in your repo and your team will know in seconds whether a new alert title accidentally matches the wrong runbook. You can even add a check that every deny entry has an escalation contact, which catches a surprisingly common gap.

Making it smarter with free model access

When the pattern match fails, or when the first commands list leaves you wondering what the output means, that is the moment you want an AI assistant that can reason about your alert without charging you for every token.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option, which means you can route a small support prompt from the same script to a model backend without standing up a dedicated GPU box. The idea is simple: when the local runbook has no entry, you call an OpenAI-compatible endpoint with the alert title and any recent log lines, and ask it to propose the next diagnostic command. Here is how to wire that into the script:

def ai_suggestion(alert, logs):
    import requests
    endpoint = os.getenv('MONKEYCODE_ENDPOINT')
    if not endpoint:
        return 'No MonkeyCode endpoint configured.'
    payload = {
        'model': 'free-model',
        'messages': [
            {'role': 'system', 'content': 'You are an on-call engineer. Suggest exactly one diagnostic command for the alert.'},
            {'role': 'user', 'content': f'Alert: {alert}\nLogs: {logs[:2000]}'}
        ]
    }
    r = requests.post(endpoint, json=payload, timeout=10)
    return r.json()['choices'][0]['message']['content']

I am not providing a concrete model name or quota, because those change over time; the pattern is what matters. The free server option lets you self-host the backend, which is attractive if you do not want alert payloads leaving your network.

Limitations and who should not use this

This approach will not save you if your runbook entries are poorly written, if you skip testing the YAML syntax, or if you expect AI to magically fix a broken schema. The script is a thin wrapper around your team's operational knowledge, so if no one writes down the recovery steps, the pattern matching will return nothing useful. Also, do not use this as the sole escalation mechanism for critical incidents without auditing the freeze windows, because a wrong date in the YAML is just as dangerous as a missing page.

The takeaway

An executable runbook is a small investment that pays off exactly when your cognitive load is highest. Start with one alert, add the YAML entry, wire the script into your alerting tool, and let the freeze rule keep you honest. If you are curious about the product side of this post, MonkeyCode is worth a look for its free model access and self-hosted server option, but the real win here is the runbook pattern itself.