Everyone is building AI agents. Most of them have far more permissions than they should.
One of the biggest questions facing platform engineering teams today is not whether AI can help operate production systems. The real question is: how do you prevent an AI agent from becoming the next privileged insider threat?
I wanted to explore that problem in a practical way. The result was NEXUS - Mesh Intelligence Hub - an AI-powered operations agent running on Amazon EKS inside an Istio service mesh, designed around a single principle:
The agent should be able to observe everything, understand what is happening, propose a solution, and still be physically unable to touch the workloads it monitors.
That principle shaped every architectural decision that followed.
The Problem
Traditional monitoring platforms are excellent at detecting symptoms.
Prometheus tells you error rates are increasing. Jaeger shows where requests are failing. Kiali visualises service-to-service communication. Dynatrace correlates application and database activity.
But none of these tools answer the question engineers immediately ask during an incident:
"What is most likely broken, and what should we do next?"
That gap is where NEXUS operates. Its purpose is not to replace engineers. Its purpose is to shorten the path from detection to diagnosis.
Zero Trust First, AI Second
The most important design decision was that NEXUS would never be trusted simply because it is AI. Instead, it is treated exactly like any other workload inside the cluster.
The agent runs in its own Kubernetes namespace with a dedicated ServiceAccount and a SPIFFE cryptographic identity:
spiffe://cluster.local/ns/ai-agent/sa/ai-agent
All communication is protected by Istio mTLS in STRICT mode. An AuthorizationPolicy grants NEXUS read-only access to Prometheus, Jaeger, and Kiali in the istio-system namespace. A separate DENY policy explicitly blocks it from reaching any application workload - lsd-frontend, lsd-backend, payment APIs - everything.
Even if the agent were compromised, the blast radius is intentionally constrained. The service mesh becomes the enforcement layer. The security model is not based on what the agent promises to do. It is based on what the platform physically allows it to do.
How Detection Works
Every 30 seconds NEXUS polls Prometheus and evaluates two metrics per service:
Error rate derived from istio_requests_total
p99 latency from the histogram bucket
When error rate exceeds the configured threshold, NEXUS gathers a full telemetry snapshot and submits it to Claude Sonnet 4.6 via the Anthropic API. The prompt mandates structured JSON output - severity, summary, root cause, numbered remediation proposal, a "cannot do" scope boundary, and the role required to approve. No freeform text. No markdown. Structured output consumed directly by the dashboard.
Chaos Engineering Test
To validate the design I injected a controlled failure using Chaos Mesh. A NetworkChaos fault was applied against the lsd-backend service inside the lsd-payments namespace.
Within a single 30-second polling cycle NEXUS detected a 56.5% error rate with no corresponding frontend degradation. Claude's diagnosis:
Correctly isolated the fault domain to lsd-backend specifically, ruling out mesh-wide or namespace-wide networking issues
Identified NaN p99 latency as a broken metrics pipeline signal rather than an application latency problem
Produced a 6-step remediation proposal covering pod health inspection, Envoy sidecar log analysis, DestinationRule outlier detection tuning, and Jaeger trace verification
Explicitly declared it cannot execute kubectl commands, cannot inspect running pods, cannot modify Istio configuration, cannot change production workloads
Human-in-the-Loop Operations
The dashboard keeps humans in control. When an incident is detected, three actions are available:
NEXUS AI - Auto-Remediate - Executes a tightly scoped remediation workflow using a ClusterRole limited to chaos-mesh resources. In this environment that means deleting the Chaos Mesh fault object. The agent cannot touch application workloads.
Escalate to Senior Support Engineer - Routes the approved diagnosis and proposal through Discord and Grafana IRM as a structured incident notification.
Dismiss - Acknowledges the event without action.
The principle does not change: AI proposes. Humans approve.
Incident Management and Notifications
NEXUS integrates with Grafana Cloud IRM and Discord for full incident lifecycle visibility.
Grafana Synthetic Monitoring runs uptime checks every 60 seconds from four global probe locations - Cape Town, London, North Virginia, and Sydney - with SSL expiry tracking and latency measurement per phase (connect, TLS, processing, transfer).
During the test run:
Synthetic monitoring detected service degradation from all four global locations
Grafana IRM automatically opened Incident #4 routed through the NEXUS-Alerts escalation chain
Discord received structured diagnosis notifications at alert, remediation, and recovery stages
Human approval triggered the auto-remediation
Recovery was confirmed through telemetry within 5 minutes of fault deletion
The entire lifecycle was end-to-end: detect, diagnose, approve, remediate, recover.
Why the Service Mesh Matters
Without a service mesh, enforcing these boundaries becomes significantly harder. Istio provides workload identity via SPIFFE/SPIRE, mutual TLS on every connection, AuthorizationPolicy enforcement at the sidecar level, and full request telemetry through Envoy - without touching application code.
The AI agent does not need to be trusted. The mesh verifies identity and enforces policy on every connection. That distinction is critical.
The Stack
Amazon EKS 1.34 - Istio 1.23 - Prometheus - Jaeger - Kiali - Dynatrace - Grafana Cloud IRM - Chaos Mesh - ArgoCD - External Secrets Operator - GitHub Actions OIDC - Amazon ECR - AWS ACM - Claude Sonnet 4.6 - Flask - Discord Webhooks
Final Thoughts
AI agents are becoming increasingly capable. That makes architectural guardrails more important, not less.
NEXUS was built to explore a different model: an AI agent that can see everything it needs to diagnose a problem, while remaining cryptographically prevented from becoming the solution itself.
The service mesh enforces the boundaries. The AI performs the analysis. The engineer remains accountable for the decision.
That may be the most practical path toward operating AI safely in production environments.
