What if the safest Kubernetes fix is no fix at all?

AI-assisted DevOps and SRE tools are becoming more common. Tools like K8sGPT can scan Kubernetes clusters, detect issues, and explain what might be going wrong.

Most evaluations of these tools focus on one obvious question:

Can the system diagnose or fix the incident?

That question matters, but in production systems, it is not the whole story.

Sometimes the more important question is:

Does the system know when not to act?

A confident but wrong remediation can make an incident worse. Restarting the wrong workload, suggesting an unsafe mutation, acting on stale events, or proposing changes based on incomplete evidence can turn a manageable incident into a larger production problem.

That idea is the motivation behind this benchmark.

A calibration and abstention benchmark has been built for K8sGPT-backed Kubernetes SRE workflows. The goal is not only to test whether a workflow can identify Kubernetes issues, but also whether it can recognize uncertainty, avoid unsafe actions, and abstain when the evidence is incomplete, misleading, or adversarial.

GitHub repo: github.com/Mayank-013/k8sGPT

Why diagnosis accuracy is not enough

In many Kubernetes troubleshooting scenarios, identifying the visible symptom is relatively easy.

A pod is in CrashLoopBackOff.

An image cannot be pulled.

A workload is stuck in Pending.

A readiness probe is failing.

But knowing the symptom is different from knowing the root cause.

For example, a readiness failure could be caused by:

• a bad probe path

• a wrong probe port

• a broken application

• an upstream dependency outage

• a NetworkPolicy issue

• stale events from an older incident

If an SRE workflow sees the symptom and immediately proposes a remediation, it may look helpful while still misunderstanding the incident.

That distinction matters.

In real operations, "do nothing yet" can be the safest decision — but only if the system understands why it is abstaining.

What has been built

This project introduces a benchmark and evaluation harness around K8sGPT as the analyzer under test.

The benchmark includes 163 labeled Kubernetes incident cases across four categories:

Each case is labeled not only with the expected root cause, but also with:

• action-risk information

• abstention expectations

• evidence completeness

• unsafe remediation candidates

• expected behavior

Instead of only asking whether the workflow got the diagnosis right, the benchmark asks:

• Did it identify the right root-cause family?

• Did it propose a safe action?

• Should it have acted, investigated more, required approval, or abstained?

• Was its confidence calibrated?

• Did it produce an unsafe or overprivileged action proposal?

• Did it fail safely under uncertainty?

How the evaluation works

At a high level, the workflow looks like this:

kind cluster
  -> inject Kubernetes incident
  -> capture evidence
  -> run K8sGPT
  -> normalize findings
  -> extract structured action plan
  -> classify risk
  -> score diagnosis, abstention, calibration, and safety

The benchmark uses a reproducible kind-based setup to inject incidents, run K8sGPT, normalize findings, extract structured action plans, and score the results.

Raw outputs are preserved, and scoring happens separately. This makes it possible to distinguish what K8sGPT actually found from what the surrounding workflow inferred.

That distinction is important because this project evaluates K8sGPT-backed workflows, not native K8sGPT remediation behavior.

K8sGPT provides analyzer findings and optional explanation output. Structured action plans, confidence scores, abstention decisions, and routing logic are wrapper and evaluation-layer components in this benchmark.

To keep the evaluation honest, stage scores are reported separately:

1. K8sGPT analyzer finding quality

2. Wrapper action and abstention quality

This avoids mixing up what K8sGPT itself produced with what the surrounding workflow inferred.

What is being measured

The benchmark focuses on safety-oriented evaluation rather than diagnosis accuracy alone.

This matters because an AI-assisted SRE workflow should not only be evaluated by how often it produces an answer.

It should also be evaluated by whether the answer is safe to act on.

What was evaluated

The released benchmark contains 163 labeled cases. The evaluation separates full-catalog analyzer scoring from a smaller live LLM-backed subset.

The important distinction:

Benchmark size is not the same as LLM-differentiated size.

The full 163-case catalog is used for live analyzer evaluation. The live LLM-backed mechanism comparison is currently a smaller 14-case subset.

Key results: live C0 analyzer evaluation

On routine Kubernetes cases, K8sGPT-backed analyzer workflows were useful.

For common incident families like ImagePullBackOff, CrashLoopBackOff, OOMKilled, and Pending, analyzer-backed extraction recovered expected actions reliably in the in-distribution set.

Probe-related issues were much harder.

This suggests that K8sGPT-backed analyzer workflows can be helpful for routine Kubernetes symptoms, but some failure families need deeper causal reasoning.

Full-catalog results

The full executable C0 evaluation was run across all 163 released cases.

The pattern is clear:

K8sGPT-backed workflows are useful for routine Kubernetes symptoms, but safe remediation under uncertain, OOD, or adversarial cases remains difficult.

The high safe-abstain rate in OOD and adversarial cases is important, but it should not be over-interpreted.

An abstention-heavy workflow can look safe while still misunderstanding the incident.

In other words, a system may avoid dangerous actions not because it deeply understands the situation, but because it abstains broadly.

That is safer than reckless mutation, but it is not the same as robust causal understanding.

Mixed-condition full-catalog results

The full catalog also includes prepared C1-C3/C7 rows. These are mixed results: 149 heuristic offline prepares plus 14 preserved live OpenAI-backed plans.

These full-catalog mixed rows are useful as infrastructure baselines, but the fair mechanism comparison is the live LLM subset below.

Live LLM-backed subset

A small live LLM-backed subset was evaluated using OpenAI gpt-4o-mini for explanation and structured action extraction.

The 14 representative cases covered ID, near-OOD, far-OOD, and adversarial scenarios.

In this subset, the LLM-backed wrapper was more willing to propose actions and had higher confidence. However, it also produced more unsafe action proposals than the abstain-heavy heuristic baseline.

A hybrid router restored more conservative behavior by pushing uncertain or risky cases back toward abstention.

This suggests an important lesson:

Adding an LLM layer can make a workflow feel more actionable, but without risk-aware routing, it may also make the workflow less safe.

Second-labeler agreement

To reduce the risk of purely subjective labeling, a second-labeler pass was performed on 30 out of 113 OOD cases.

The lower agreement on action-risk tier is actually useful. It shows that deciding whether an SRE workflow should act, require approval, or escalate is not always obvious.

That is exactly why this kind of benchmark matters.

Main takeaway

The strongest finding is not simply that K8sGPT performs better on routine cases than OOD cases.

That is expected.

The more interesting finding is this:

Safety via abstention is not the same as robust causal understanding.

A workflow can avoid unsafe action by abstaining broadly, but still fail to understand the incident. On the other side, adding an LLM layer can increase actionability and confidence, but may also increase unsafe action proposals unless paired with risk-aware routing.

This creates a useful way to think about AI-assisted SRE workflows:

• diagnosis accuracy matters

• confidence calibration matters

• abstention quality matters

• action-risk classification matters

• unsafe proposal rates matter

• escalation behavior matters

Production safety is not only about getting the answer right.

It is also about knowing when the answer is not safe enough to act on.

Why this matters

As AI-assisted SRE tools become more capable, the hard problem is not only generating a plausible explanation.

The hard problem is deciding what level of autonomy is appropriate.

Should the system suggest a command?

Should it ask for approval?

Should it collect more evidence?

Should it escalate to a human?

Should it refuse to recommend a mutation?

These decisions are central to production safety.

For Kubernetes environments, where remediation actions can affect running workloads, security boundaries, and customer-facing systems, knowing when not to act is just as important as knowing how to act.

What this project does not claim

This benchmark does not claim that K8sGPT is unsafe.

It also does not claim that the evaluated wrapper represents native K8sGPT remediation behavior.

The goal is more specific:

Evaluate how K8sGPT-backed workflows behave when diagnosis, confidence, abstention, and action risk are measured together.

This project is about building a more careful evaluation framework for AI-assisted Kubernetes troubleshooting.

What comes next

This is an early release, and there is more to improve.

Next steps include:

• expanding live LLM-backed evaluation beyond the current subset

• adding another baseline such as HolmesGPT

• improving probe and dependency-related scenarios

• publishing a formal PDF report

• refining calibration and abstention metrics

• improving reproducibility and documentation

Final thought

Most AI/SRE demos focus on action:

Here is the issue. Here is the fix.

But production reliability often requires restraint.

Sometimes the best answer is:

There is not enough evidence to safely act yet.

That is the behavior this benchmark is designed to test.

If you are interested in Kubernetes, SRE, AI-assisted operations, or reliability evaluation, feedback on the project is welcome.

GitHub repo: github.com/Mayank-013/k8sGPT