Build an On-Call Handoff Agent: Shift Summaries That Don't Drop Context

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

What This Agent Does

An on-call handoff agent assembles the shift summary your outgoing engineer was too tired to write: what paged and why, what's still smoldering, which silences expire on the next person's watch, what shipped in the last 12 hours, and how much error budget the shift consumed. The facts come from Alertmanager, Prometheus, and the deploy log through read-only tools — the LLM never invents an incident, it only prioritizes and narrates what the tools returned. The output is a structured Markdown handoff posted to the on-call channel at shift boundary, and the outgoing engineer approves it with one click before the incoming one reads it.

The handoff is the most predictable context loss in on-call. Every rotation, twice a day, the person with 12 hours of accumulated situational awareness types two rushed Slack lines — "quiet shift, ignore the disk alert on node-7, it's known" — and logs off. The incoming engineer then re-derives everything else the hard way, usually at the exact moment something pages. Teams solve this with handoff templates, which fail for a human reason: filling in a template at 8 a.m. after a rough night is toil, so the fields decay to "n/a." An agent doesn't get tired, and almost everything a good handoff contains is already sitting in your monitoring stack with timestamps on it.

What a Handoff Actually Contains

Before any code, agree on the payload. A useful shift handoff answers five questions, and each maps to a queryable source:

The expiring-silences row is the one templates always miss and the one that produces the classic handoff failure: the outgoing engineer silenced a flapping alert for 12 hours at 21:00, and it comes back at 09:00 as a mystery page for someone who has never heard of it. Whether a silence expires mid-shift is pure date arithmetic — it should be computed in code, never left for the model to notice.

The Read Surface: Three Tools, All Boring

Same discipline as every ops agent on this site: fixed, parameterized, read-only tools, no free-form queries. If you already run the Alertmanager MCP server, the first two tools are a strict subset of it.

# handoff_tools.py — the agent's entire read surface
import os
from datetime import datetime, timedelta, timezone

import httpx

AM = os.environ["ALERTMANAGER_URL"]
SHIFT_HOURS = 12
NEXT_SHIFT_HOURS = 12

def _get(path: str, params: dict | None = None) -> list | dict:
    r = httpx.get(f"{AM}/api/v2/{path}", params=params or {}, timeout=15)
    r.raise_for_status()
    return r.json()

@mcp.tool()
def shift_alert_state() -> dict:
    """Currently firing alerts plus alerts that fired during the last
    shift window. Read-only snapshot from Alertmanager."""
    now = datetime.now(timezone.utc)
    shift_start = now - timedelta(hours=SHIFT_HOURS)
    active = [{
        "name": a["labels"].get("alertname", "?"),
        "severity": a["labels"].get("severity", "?"),
        "since": a["startsAt"],
        "summary": a["annotations"].get("summary", ""),
    } for a in _get("alerts", {"active": "true", "silenced": "false"})]
    fired_in_shift = [a for a in active
                      if datetime.fromisoformat(a["since"]) >= shift_start]
    return {"active": active, "started_this_shift": fired_in_shift,
            "shift_window_hours": SHIFT_HOURS}

@mcp.tool()
def silences_expiring_next_shift() -> list[dict]:
    """Silences that expire within the incoming shift. Each of these
    WILL resume paging on the next engineer's watch. Computed in code."""
    now = datetime.now(timezone.utc)
    horizon = now + timedelta(hours=NEXT_SHIFT_HOURS)
    out = []
    for s in _get("silences"):
        if s["status"]["state"] != "active":
            continue
        ends = datetime.fromisoformat(s["endsAt"].replace("Z", "+00:00"))
        if now < ends <= horizon:
            out.append({"matchers": s["matchers"],
                        "ends_at": s["endsAt"],
                        "created_by": s["createdBy"],
                        "comment": s["comment"] or "(no comment)"})
    return out

@mcp.tool()
def shift_deploys() -> list[dict]:
    """Deploys during the shift window from the CD system, newest first.
    Returns [{app, version, deployed_at, author}] — read-only."""
    ...

The third tool matters more than it looks. The most valuable sentence in a real handoff is usually a correlation — "payments latency alert at 14:40 fired twenty minutes after the 14:20 payments deploy; it resolved on its own but watch it" — and the model can only draw that line if deploys and alerts are both in context with timestamps. For budget state, reuse the get_burn_state tool from the error budget agent verbatim; the handoff only needs its one-line answer, not the full triage.

Generation: Forced Schema, Then a Deterministic Template

Don't let the model write free-form Markdown. Force a schema, then render the Markdown yourself — that separation is what makes the output testable and keeps the sections from silently vanishing when the model has a lazy day:

HANDOFF_TOOL = {
    "name": "compose_handoff",
    "description": "Compose the shift handoff from tool results ONLY. "
                   "Every item must cite which tool result it came from. "
                   "If a section is empty, say so explicitly.",
    "input_schema": {
        "type": "object",
        "properties": {
            "shift_character": {"enum": ["quiet", "noisy", "incident"]},
            "headline": {"type": "string",
                "description": "One sentence the incoming engineer must "
                               "read even if they read nothing else."},
            "watch_items": {"type": "array", "items": {"type": "object",
                "properties": {
                    "what": {"type": "string"},
                    "why": {"type": "string",
                        "description": "Evidence with timestamps, e.g. "
                                       "'fired 3x between 02:00-04:30'"},
                    "action_if_it_pages": {"type": "string"}},
                "required": ["what", "why", "action_if_it_pages"]}},
            "expiring_silences_ack": {"type": "boolean",
                "description": "True only if every expiring silence from "
                               "the tool result appears in watch_items."},
        },
        "required": ["shift_character", "headline", "watch_items",
                     "expiring_silences_ack"],
    },
}

Two deliberate choices here. watch_items forces the action_if_it_pages field — a handoff that says "keep an eye on Kafka lag" without saying what to do when it pages has only moved the anxiety, not the context. And expiring_silences_ack is a self-check the renderer verifies in code: if the silences tool returned two expiring silences and fewer than two appear in watch_items, the run fails loudly instead of shipping an incomplete handoff. Schema-level tripwires like this catch omissions far more reliably than prompt-level pleading.

The renderer is then thirty lines of string formatting: stable section order, timestamps normalized to the team's timezone, raw tool counts printed in a footer ("4 alerts fired, 2 silences expiring, 6 deploys") so a reader can spot at a glance if the narrative dropped something. Everything positional and structural is code; only judgment — what makes the headline, what's worth watching, what connects to what — is model output.

Wiring and the One Approval Gate

Run it from cron a few minutes before shift boundary, post the draft to the on-call channel, and have the outgoing engineer approve or edit it before it's pinned for the incoming one. This is the cheapest possible human-in-the-loop gate: the reviewer is the one person who lived the shift, review takes under a minute because they're checking a summary of their own last 12 hours, and their edits are a free quality signal — anything a human consistently adds is a candidate for a new tool; anything they consistently delete is noise to cut from the prompt.

Sequencing with the rest of an on-call agent stack: the handoff draws on the same evidence-first layout as the context engineering budget — live state first, history clearly labeled as history. And if an incident from the shift produced a reviewed postmortem via the postmortem agent, its record lands in incident memory — the handoff covers the next 12 hours; memory covers the next 12 months. Same sources, different half-lives; don't collapse them into one document.

Evaluating It: Hallucinated Incidents Are the Only Real Failure

Handoff generation is unusually easy to eval because the input is fully synthetic-able. Build fixture shifts as canned tool outputs — a quiet shift, a noisy-but-benign shift, an active-incident shift, a shift with two expiring silences — and assert three things on every run. Grounding: every alert name, version string, and timestamp in the output must appear in a tool result; a fuzzy substring check catches most fabrications, and a fabricated incident in a handoff is worse than no handoff because it gets trusted. Completeness: every expiring silence and every currently-firing alert must appear somewhere in the output — this is the tripwire above, promoted to a test. Calibration: the quiet-shift fixture must produce a short handoff; if your agent writes four paragraphs about a shift where nothing happened, the incoming engineer will stop reading handoffs within a week, and the whole system fails socially rather than technically.

Honest Limits

The agent summarizes what's instrumented, and shifts contain things that aren't: the vendor ticket you're waiting on, the customer escalation in a DM thread, the hunch that node-7's disk alert is actually a failing drive. That's why the approval step is an edit step — the human adds the uninstrumented 20%, and the agent's job is to make that the only part they have to write. It also inherits your monitoring's blind spots; an unmonitored failure mode is invisible in the handoff, which can lend a false calm — the footer counts help, but they can't count what was never measured. Start with the three tools, the forced schema, and the outgoing-engineer gate. The first time an incoming engineer gets paged at 09:05 by an alert whose silence expired at 09:00 — and the handoff already told them what it was and what to do — the rotation will defend this agent for you.

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.