💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.
What This Agent Does
A Kubernetes capacity planning agent answers one question every week, per cluster, before anyone is paged: what runs out first, when, and what is the smallest change that buys thirty days? The forecasting is predict_linear in Prometheus recording rules — arithmetic, testable, no model involved. The LLM reads the forecasts through three read-only tools, decides which of the dozen "things filling up" actually matter, and drafts one pull request per finding: a bigger PVC, a higher Karpenter NodePool limit, a raised ResourceQuota. A human merges it. The agent never touches the cluster.
Capacity incidents are the dumbest outages an SRE team has, because the signal was on a dashboard for a week. A PVC hits 100% at 03:00 Sunday and Postgres stops accepting writes; a NodePool reaches its CPU limit and the next deploy sits in Pending for an hour; a namespace quota fills and CI can't schedule its runner. The Pod Pending / FailedScheduling guide covers the 2 a.m. version of this. This post is the version where it never gets to 2 a.m.
The Forecasts Stay in PromQL
Never ask a language model to extrapolate a time series. predict_linear and deriv exist for that, and recording rules make the results a stable, cheap read for the agent. Four rules cover most of what actually exhausts in a cluster:
# capacity-rules.yaml — the agent reads these, never writes them
groups:
- name: capacity-forecast
interval: 5m
rules:
# 1. Days until each PVC is full, from a 6h slope. NaN/Inf when not shrinking.
- record: capacity:pvc_days_to_full
expr: |
kubelet_volume_stats_available_bytes
/ clamp_min(-deriv(kubelet_volume_stats_available_bytes[6h]), 1)
/ 86400
# 2. Requested CPU per Karpenter NodePool vs its hard limit (0..1).
- record: capacity:nodepool_cpu_limit_ratio
expr: |
karpenter_nodepool_usage{resource_type="cpu"}
/ karpenter_nodepool_limit{resource_type="cpu"}
# 3. Where that ratio lands in 7 days at the last 24h's growth rate.
- record: capacity:nodepool_cpu_limit_ratio_7d
expr: |
predict_linear(capacity:nodepool_cpu_limit_ratio[24h], 7*24*3600)
# 4. ResourceQuota consumption per namespace and resource (0..1).
- record: capacity:quota_ratio
expr: |
kube_resourcequota{type="used"}
/ on (namespace, resourcequota, resource) kube_resourcequota{type="hard"}
Rule 1 is the one people get wrong. Dividing by deriv directly explodes when the slope is zero or positive (a volume that's shrinking or flat has infinite days-to-full, which is correct but unfriendly), so the clamp_min floors the shrink rate at one byte per second and the tool layer treats anything above 365 days as "not a concern". Rule 2 uses Karpenter's own karpenter_nodepool_usage and karpenter_nodepool_limit series (Karpenter v1; the nodepool and resource_type labels are the join keys) — if you run Cluster Autoscaler instead, substitute the ASG max-size math from the Karpenter vs Cluster Autoscaler comparison. Memory gets a mirror of rules 2 and 3 with resource_type="memory"; it's omitted here for space, not because memory fills up less often.
Alert on the recording rules too — capacity:pvc_days_to_full < 3 deserves a page with or without an agent. The agent's job is the slow burn the alert threshold doesn't catch: nine days out, on a Tuesday, when a two-line PR is still a boring change.
The Tools: Three Reads, Computed Numbers, No Raw PromQL
The agent gets no free-form query tool. Each tool wraps a fixed query and returns the decision-relevant number — days, ratios, counts — already computed, the same discipline as the read-only Prometheus MCP server with a narrower surface.
# capacity_tools.py — the agent's entire read surface
import os, httpx
from fastmcp import FastMCP
PROM = os.environ["PROM_URL"]
HORIZON_DAYS = 30 # anything further out is not a finding
mcp = FastMCP("capacity-agent")
def _series(query: str) -> list[dict]:
r = httpx.get(f"{PROM}/api/v1/query", params={"query": query}, timeout=20)
r.raise_for_status()
return [{**s["metric"], "value": float(s["value"][1])}
for s in r.json()["data"]["result"]]
@mcp.tool()
def pvc_forecast(max_days: int = HORIZON_DAYS) -> list[dict]:
"""PVCs projected to fill within max_days, worst first.
Includes the current size so the agent can propose a concrete new size."""
days = _series(f"capacity:pvc_days_to_full < {max_days}")
sizes = {(s["namespace"], s["persistentvolumeclaim"]): s["value"]
for s in _series("kubelet_volume_stats_capacity_bytes")}
out = []
for s in days:
key = (s["namespace"], s["persistentvolumeclaim"])
out.append({"namespace": key[0], "pvc": key[1],
"days_to_full": round(s["value"], 1),
"capacity_gib": round(sizes.get(key, 0) / 2**30, 1)})
return sorted(out, key=lambda x: x["days_to_full"])
@mcp.tool()
def nodepool_headroom() -> list[dict]:
"""Per NodePool: CPU limit ratio now and projected in 7 days,
plus pods currently unschedulable in the last hour."""
now = {s["nodepool"]: s["value"]
for s in _series('capacity:nodepool_cpu_limit_ratio')}
in7 = {s["nodepool"]: s["value"]
for s in _series('capacity:nodepool_cpu_limit_ratio_7d')}
pending = _series('sum(kube_pod_status_unschedulable) or vector(0)')[0]["value"]
return [{"nodepool": p, "cpu_ratio_now": round(v, 3),
"cpu_ratio_in_7d": round(in7.get(p, v), 3),
"unschedulable_pods_now": int(pending)} for p, v in now.items()]
@mcp.tool()
def quota_pressure(threshold: float = 0.8) -> list[dict]:
"""Namespaces where any ResourceQuota resource exceeds the threshold."""
return [{"namespace": s["namespace"], "quota": s["resourcequota"],
"resource": s["resource"], "ratio": round(s["value"], 3)}
for s in _series(f"capacity:quota_ratio > {threshold}")]
Notice what the tools don't return: raw byte counts, per-sample histories, or anything the model would have to do arithmetic on. pvc_forecast hands over days_to_full and capacity_gib so the only number the model produces is a proposed new size, and the wrapper validates that number before it goes anywhere.
The Verdict: One Finding Per Resource, Bounded Actions
The model is forced into a schema. Free text is allowed only in reasoning; every field a script acts on is an enum or a bounded number.
FINDINGS_TOOL = {
"name": "report_capacity_findings",
"description": "Report capacity risks inside the 30-day horizon.",
"input_schema": {
"type": "object",
"properties": {
"findings": {
"type": "array",
"items": {
"type": "object",
"properties": {
"kind": {"enum": ["pvc", "nodepool", "quota"]},
"target": {"type": "string",
"description": "namespace/name or nodepool name"},
"days_to_exhaustion": {"type": "number"},
"urgency": {"enum": ["this_week", "this_month", "watch"]},
"action": {"enum": ["expand_pvc", "raise_nodepool_limit",
"raise_quota", "investigate_growth",
"none"]},
"proposed_value": {"type": "string",
"description": "New size or limit, e.g. '200Gi' or '480'."},
"reasoning": {"type": "string"}
},
"required": ["kind", "target", "days_to_exhaustion",
"urgency", "action", "reasoning"]
}
}
},
"required": ["findings"]
}
}
SYSTEM = (
"You are the weekly capacity review for an SRE team. Use the tools, then "
"report findings within a 30-day horizon.\n"
"Urgency: 'this_week' under 7 days, 'this_month' under 30, else 'watch'.\n"
"Prefer 'investigate_growth' over raising a limit when a PVC's fill rate "
"implies more than 2x growth in 30 days — that is usually a leak or a "
"missing retention policy, not a capacity need.\n"
"Proposed sizes must be at most 1.5x the current value. You cannot change "
"anything; you propose, with numbers cited from the tools."
)
Then the wrapper enforces what the prompt asked for, because prompts are requests and code is policy:
def validate(finding: dict, current: dict) -> str | None:
"""Return a rejection reason, or None if the finding may become a PR."""
if finding["action"] == "expand_pvc":
new_gib = parse_gib(finding["proposed_value"])
cur_gib = current["capacity_gib"]
if new_gib <= cur_gib: return "PVCs cannot shrink"
if new_gib > cur_gib * 1.5: return "exceeds 1.5x cap"
if new_gib > 2048: return "above 2 TiB needs a human plan"
if finding["action"] == "raise_nodepool_limit":
if int(finding["proposed_value"]) > current["limit"] * 1.5:
return "exceeds 1.5x cap"
return None
The 1.5x cap is the guardrail that matters. A model that reasons "this volume fills in 4 days, let's make it 10 TiB" is being helpful in exactly the way that turns a capacity agent into a cost incident. Growth beyond the cap is a signal to a human that the workload needs a conversation, not a bigger disk.
Findings Become Pull Requests
Every validated finding becomes a small diff against the repo your GitOps controller already syncs — the PRs-not-kubectl pattern. A NodePool limit bump looks like this:
# infra/karpenter/nodepool-general.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: general
spec:
limits:
- cpu: "320"
+ cpu: "480" # capacity-agent: 0.84 of limit now, 1.02 projected in 7d
memory: 1280Gi
And the PR body is the evidence, written by a template from the schema, not by the model:
Capacity finding: nodepool/general — raise CPU limit 320 -> 480
- cpu_ratio_now: 0.84, cpu_ratio_in_7d: 1.02 (24h growth slope)
- unschedulable_pods_now: 0
- Urgency: this_week
- Est. cost delta at full use: +$1,420/mo (160 vCPU x m6i on-demand)
- Reasoning (agent): "General pool crosses its limit in ~6 days at the
current request growth; HPA on checkout-api added 22 replicas since Monday."
The cost line comes from a price lookup in the wrapper, not from the model. A capacity PR without a cost delta is half a PR — the reviewer's real question is whether the growth is worth paying for, and the FinOps counterpart in the FinOps agent is a natural second reviewer here: one agent proposes headroom, the other flags waste, and the human sees both.
PVC expansion has a prerequisite the agent must check before proposing: the StorageClass needs allowVolumeExpansion: true, and the PVC must live in Git (Helm values or a Kustomize patch), not only in the cluster. If the PVC was created by a StatefulSet's volumeClaimTemplates, editing the template does nothing to existing claims — the PR should say so and target the claim directly. Encode that as a hard rule in the wrapper: expand_pvc findings whose claim has no Git source get downgraded to investigate_growth with a note.
Run It Weekly, Shadow It First
Wire it as a CronJob on Monday morning, after the weekend's growth is in the 24h slope and before the week's deploys land:
apiVersion: batch/v1
kind: CronJob
metadata:
name: capacity-agent
namespace: sre-agents
spec:
schedule: "0 6 * * 1"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
serviceAccountName: capacity-agent # no cluster RBAC at all
restartPolicy: Never
containers:
- name: agent
image: ghcr.io/example/capacity-agent:1.4.0
env:
- name: PROM_URL
value: http://prometheus.monitoring.svc:9090
- name: MODE
value: "shadow" # shadow | propose
envFrom:
- secretRef: { name: capacity-agent-github }
The ServiceAccount has no Kubernetes RBAC because the agent needs none: it reads Prometheus over HTTP and writes to GitHub. That is the entire blast radius, and it's the reason a capacity agent is a good first agent for a team that hasn't run one — compare the RBAC surface to the least-privilege kubectl agent, which has to be far more careful.
MODE=shadow posts the findings to a channel and opens nothing. Run four weekly cycles that way and score them: every capacity incident in that month should have appeared as a this_week or this_month finding beforehand, and every this_week finding should have been a real risk — not a nightly batch job that fills a scratch volume and empties it at 04:00. Those cyclical volumes are the classic false positive: a 6h slope taken at 03:00 forecasts doom, and the 24h slope says nothing is wrong. If shadow mode surfaces them, widen the PVC rule's range to [24h] for volumes with a capacity-agent/cyclic: "true" annotation, which pvc_forecast can honor.
Honest Limits
Linear extrapolation is the right default and the wrong answer for anything with a step. A migration that doubles a table, a product launch, a Black Friday HPA ramp — none of these are visible in last week's slope, and this agent will confidently forecast forty days of headroom on Thursday and be wrong by Saturday. It complements the humans who know the calendar; it doesn't replace them. Pods without resource requests are invisible to the NodePool math (requests are what karpenter_nodepool_usage counts), so a cluster with sloppy requests will under-report its pressure — fix that with a Kyverno or LimitRange policy before trusting the numbers. Quota ratios say nothing about whether the quota is right; a namespace at 0.85 of a quota that was set arbitrarily two years ago is a conversation, and the agent's raise_quota action should be the one most often rejected in review. And the whole design assumes Prometheus retention covers at least a day of the series at 5-minute resolution; a monitoring stack that drops samples under load produces slopes that are noise.
Start with the recording rules alone — they're worth having with no agent at all. Add the tools and shadow mode in the second week. The first Monday the agent opens a PR that says "the Postgres volume fills in eleven days, here is 150Gi, here is what it costs, allowVolumeExpansion is on", and the on-call reviews it over coffee instead of at 03:00, capacity stops being an incident category.
📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.
