The pager goes off at 3:11 a.m. It's the fifth time tonight, and it's the same alert: HighMemoryUsage on a node that's running a memory-mapped cache doing exactly what it was designed to do. You ack it half-asleep, knowing it'll fire again in twelve minutes. By the time the real incident shows up at 4:40 — a slow API degradation that's quietly eating your error budget — you're too fried to see it clearly. That's not a tooling failure. That's a design failure, and most of us have lived it.
I run production OpenStack, Kubernetes, Terraform, and the observability stack that watches all of it. I've spent more nights than I'd like fighting my own alerts. So when "AI for SRE" started showing up in every vendor deck, my first reaction was a tired no. The last thing on-call needs is an autonomous bot deciding to restart my database at 3 a.m. based on a hunch.
But there's a version of this that actually works, and it has nothing to do with autonomy. It's about using AI for the narrow set of things it's genuinely good at — clustering noise, summarizing storms, drafting hypotheses, surfacing the right runbook — while a human stays the final decision-maker. AI triages and proposes. You decide what pages a human and what gets fixed. That's what I mean by humanizing AI: not making the machine more human, but using the machine to keep the on-call human rested, focused, and in control.
The real problem is signal, not tooling
Alert fatigue isn't caused by too few dashboards. It's caused by alerts that don't map to a human decision. Every page should answer one question: does a person need to do something right now? If the answer is "no" or "not yet," it shouldn't page. We all know this. We violate it constantly because writing good alert rules is hard, and tuning them is a chore nobody prioritizes until they're drowning.
The honest baseline, before any AI enters the picture, is rule hygiene. If your alerts are symptom-based, tied to user-facing SLOs, and have sane thresholds, you've already eliminated most of the noise. I've written about this at length in designing alert rules that don't page you falsely, and the short version is: alert on what the user feels, not on what a single machine is doing. A node at 92% memory is not an incident. A checkout flow at 92% success rate is.
Here's a symptom-based, multi-window burn-rate alert — the kind that respects your error budget instead of paging on every blip:
groups:
- name: slo-burn-rate
rules:
- alert: CheckoutErrorBudgetFastBurn
expr: |
(
job:slo_errors:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
and
job:slo_errors:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
)
for: 2m
labels:
severity: page
slo: checkout-availability
annotations:
summary: "Checkout burning 30-day budget 14.4x too fast"
runbook: "https://runbooks.internal/checkout-availability"
The 14.4 multiplier on a 0.1% error budget is the classic fast-burn threshold: at that rate you'd exhaust a 30-day budget in roughly two days, so it deserves a human now. Pair it with a slower window for the grind-it-down failures. If multi-window burn rate is new to you, this walkthrough is the most practical reference I keep handing to my team.
Pro Tip: Before you let AI touch a single alert, audit how many of your current pages are actionable. Pull a month of pages and bucket them: "I took an action," "I acked and ignored," "false positive." If more than a third land in the last two buckets, fix your rules first. AI applied to a noisy rule set just gives you faster, more confident noise.
Where AI earns its place: the alert storm
Rule hygiene gets you far, but it doesn't solve the 3 a.m. storm. When a top-of-rack switch flaps or a Kubernetes node goes NotReady, you don't get one clean alert. You get forty: pod restarts, probe failures, downstream latency, queue backups, dependent service timeouts. Each one is technically "real." Collectively, they're a wall of text that you have to mentally parse while half-conscious.
This is the single best place to put AI to work, because it's a summarization and clustering problem — exactly what large language models are good at. You feed the model the firing alerts, recent deploys, and relevant topology, and ask it to do what a senior SRE does instinctively: collapse the storm into a root cause plus its downstream effects.
A good humanized output looks like this:
Alert storm summary — 41 alerts, last 6 min
Most likely root cause (confidence: high): Nodeworker-prod-07wentNotReadyat 03:08 UTC, 90s after theingress-nginxrollout (deploy #4821). 38 of 41 alerts are pods evicted from or routing through this node.
Downstream effects (not separate incidents):
checkoutp99 latency +400ms — pods reschedulingpayment-workerqueue depth rising — consumers restarting- 6x
KubePodCrashLooping— all onworker-prod-07Suggested next check:kubectl describe node worker-prod-07and the kubelet logs around 03:07. Recommended runbook: node-notready-triage. This summary is advisory. No action has been taken.
That last line matters. The AI clustered 41 alerts into one root cause and three downstream effects, attached a confidence level, and pointed at the next diagnostic step. It did not cordon the node, restart anything, or roll back the deploy. It handed me a hypothesis and a starting point, and I decide whether it's right. That cuts my time-to-understanding from "read forty alerts" to "verify one claim" — which is exactly where you want AI to save you minutes, because those minutes directly improve your MTTA and MTTR. (If you want to be rigorous about which incident metrics actually matter, MTTA is the one that alert fatigue quietly wrecks.)
The mechanical clustering can — and should — live in Alertmanager. AI is the layer that explains the cluster in plain language; Alertmanager is the layer that suppresses the redundant pages so they never wake you in the first place.
Let Alertmanager do the suppression, let AI do the explaining
Don't ask the LLM to be your routing engine. Deterministic grouping and inhibition rules are reliable, auditable, and free. Use them to fold the storm down before it ever reaches a human, then let AI summarize what's left.
route:
receiver: oncall-pager
group_by: ['alertname', 'cluster', 'node']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity = page
receiver: oncall-pager
- matchers:
- severity = ticket
receiver: ticket-queue
inhibit_rules:
# If a node is NotReady, don't page on the pods it took down with it.
- source_matchers:
- alertname = KubeNodeNotReady
target_matchers:
- alertname =~ "KubePodCrashLooping|KubePodNotReady"
equal: ['node']
That inhibition rule alone kills the most common storm pattern: a node failure dragging twenty pod alerts behind it. The equal: ['node'] clause scopes the suppression so a genuinely unrelated crashloop on a different node still pages. This is the deterministic floor. AI sits on top of it to explain the handful of alerts that remain, not to replace it. There's more nuance to getting this grouping right in cutting alert noise and designing alerts engineers actually trust, which pairs well with this approach.
Pro Tip: Treat every AI summary as a hypothesis with a confidence score, never a verdict. Require the model to emit a "next check" — a concrete command or query a human can run to confirm or refute it in under a minute. A confident-sounding root cause you can't quickly verify is worse than no suggestion at all, because it anchors your tired brain to the wrong path.
Prioritization: what actually deserves a human at 3 a.m.
Clustering tells you what is happening. Prioritization tells you whether it can wait until morning. This is the judgment call that burns out on-call engineers, and it's another place AI can genuinely help — as long as it ranks and recommends rather than decides.
The signal AI is good at synthesizing for prioritization:
- Blast radius: how many users, services, or SLOs are affected, pulled from the alert labels and topology.
- Budget velocity: is this burning error budget fast, slow, or not at all? A fast burn pages; a slow grind opens a ticket.
- Business context: is checkout down at peak, or is an internal batch job late at 4 a.m.?
- Recent change correlation: did a deploy, config push, or Terraform apply land in the last 30 minutes?
Feed those signals to the model and ask it to propose a severity and a routing decision with reasoning. The output should read like a recommendation you can override in one click:
Recommendation: open ticket, do not page (confidence: medium).
BatchExportLatencyHighis firing, but it's an internal nightly export with no SLO and no user impact. Error budget for the user-facing API is untouched. No deploy correlated. Suggest ticket for the morning; re-evaluate if it's still firing after 2 retries.
You, the human, glance at that and either agree or say "actually that export feeds the 6 a.m. finance report, page me." The AI never had the business context that the export feeds finance — and that's the point. It proposes; you supply the judgment it can't have. Over time you encode that judgment back into the rules so the AI gets it right next time. For a broader survey of what's actually useful here, I keep a running list of the best AI tools for SRE teams and what each is realistically good for.
Surfacing the runbook, not writing the fix
When you do get paged for something real, the next time-sink is finding the right runbook and remembering the current state of the system. AI is good at retrieval and assembly: given the firing alert and its labels, it can surface the most relevant runbook, pull the last three related incidents, and assemble a quick situation brief.
What it should produce:
- The runbook link that matches this alert's
runbookannotation, plus any siblings. - A one-paragraph history: "This alert fired twice in the last 30 days; both were resolved by draining the node, mean resolution 14 minutes."
- The current relevant state: recent deploys, open changes, who's already in the channel.
What it should not produce: the executed remediation. The model can draft the kubectl drain command and explain why. A human reads it, sanity-checks the node, and runs it. The gap between "here's the command" and "I ran the command for you" is the entire safety margin of this approach. Drain the wrong node and you've turned one incident into two.
Pro Tip: Wire your runbook links directly into alert annotations, like the runbook: field in the burn-rate example above. AI retrieval is dramatically more reliable when every alert already carries a canonical pointer — you're asking the model to fetch and summarize a known document, not to guess which runbook applies. Garbage retrieval in, confident-but-wrong brief out.
Closing the loop without a bot closing it for you
The part everyone skips is the follow-through. The storm ends, the incident resolves, and the action items — "tune that node alert," "add an inhibition rule," "fix the probe timeout" — evaporate into a Slack thread nobody reads again. Then the same storm wakes you next month.
AI helps here too, in its lane: it can draft the postmortem timeline from the alert and chat history, extract the action items, and propose the specific alert-rule changes that would have prevented the storm. But a human owns each item and a human merges the rule change. The discipline of actually closing the loop on incident action items is what turns last night's pain into next month's quiet. AI can draft the fix; it can't care that it ships. That's still on us.
This is the feedback loop that makes the whole system humane over time. Every overridden AI recommendation, every false page you tune out, every storm you cluster becomes encoded knowledge — better rules, better inhibition, better routing. The AI gets a sharper picture, the human gets quieter nights, and the error budget stays honest. If you want a deeper tour of the patterns, the incident response category on the site collects the workflows I actually run.
The line you don't cross
Let me be blunt about the boundary, because the vendors won't be. AI in your on-call loop should do four things: cluster noise, summarize storms, draft hypotheses, and surface runbooks. It should rank and recommend with explicit confidence levels and a verifiable next check. It should never silently take an action against production, never auto-close an incident, never decide on its own that something doesn't deserve a human.
The moment you hand the machine the keys to remediation, you've traded one kind of 3 a.m. terror — the noisy pager — for a worse one: the bot that did something you didn't sanction and now you're reverse-engineering its decision under pressure. Keep the human as the final approver. Use AI to make that human faster, calmer, and better-informed. That's the entire game.
The goal was never to remove the engineer from the loop. It's to make sure that when the pager does go off at 3 a.m. — and it will — it's for something that genuinely deserves you, with a clear summary, a ranked hypothesis, and the right runbook already in hand. That's a humane on-call. That's humanized AI.
James Joyner IV runs devopsaitoolkit.com, where he writes about keeping production systems and the humans who run them healthy. If you want to see this human-in-control approach in action, try the free AI Incident Response Assistant — it summarizes and triages, but you stay the one who decides.
