The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It

Every organization running AI agents has already made a hosting decision. Most made it by accident.

The sales team switched on the agent built into their CRM. Engineering is piloting a coding agent in a vendor’s cloud. Someone on the data team deployed a LangGraph service to a VM with a database key in an environment variable, and someone else is running an agent framework on a laptop with production credentials in a dotfile. Each of these is a hosting decision. Each one quietly settled who holds the agent’s credentials, what network paths it can reach, what gets recorded when it acts, and who can stop it. Nobody ran an architecture review, because no single deployment looked big enough to deserve one.

The scale says otherwise. By May 2025, 82% of organizations surveyed by SailPoint were already using AI agents. Only 44% had policies for securing them, 80% said their agents had already taken unintended actions, and 23% had watched an agent get tricked into revealing credentials. A year later the bill arrived: IBM’s 2026 Cost of a Data Breach report found that one in four malicious breaches is now AI-enabled, up 56% in a single year, and that those breaches cost about a million dollars more than the global average. Gartner, for its part, expects over 40% of agentic AI projects to be canceled by the end of 2027, naming inadequate risk controls as one of the three reasons.

This post is the architecture review those deployments never got. The conclusion, stated up front so you can argue with it: agents should run on Kubernetes clusters you control, wrapped in guardrails and gateways that no prompt can talk its way past. Not because Kubernetes is fashionable, but because a year of documented incidents shows that every failure was an environmental failure, and a cluster you control is the only substrate where you own every layer of the environment.

The workload that chooses its own code path

A microservice does what its code says. You can read the code, test the paths, and enumerate what it will do with any input. An agent is a different kind of workload, and the difference is structural, not a maturity phase it will grow out of:

• Its control flow is decided at runtime by a model. The “code path” is chosen by whatever lands in the context window. You cannot statically review what an agent will do, only what it is allowed to do.
• It holds credentials and takes actions. Agents query databases, call APIs, send email, move money, and spawn other agents. The interesting failure mode is not a wrong answer. It is a correct, authorized, catastrophic action.
• Every input is potentially an instruction. Prompt injection remains unsolved. OWASP’s Top 10 for Agentic Applications, published in December 2025 with over a hundred contributors, reads mostly as variations on this theme: goal hijack, tool misuse, identity and privilege abuse, memory poisoning.

Put those three together and you get the principle we have argued all year, from the accountability gap to the OpenShell analysis: controls the agent can override are not controls. The system prompt is not a perimeter. The guardrail library running inside the agent’s own process is not a perimeter. The controls have to live in the environment, where a confused or compromised agent cannot negotiate with them.

And once the controls must live in the environment, the hosting question stops being an infrastructure detail. Where the agent runs determines which controls can exist at all.

A year of incidents; one pattern

Theory is cheap. The past twelve months supplied the evidence, and it is unusually well documented.

One agent’s tokens, seven hundred victims

Drift was an AI chat agent embedded on corporate websites. To do its job it held long-lived OAuth tokens into each customer’s Salesforce. In August 2025, attackers who had stolen those tokens used them to run bulk data exports against customer CRMs. Google’s threat intelligence group assessed more than 700 organizations as potentially affected. The attackers were not after chat transcripts. They mined the exported support cases for the credentials customers had pasted into them: AWS keys, Snowflake tokens, passwords. Cloudflare, one of the disclosed victims, rotated 104 API tokens found in its stolen case text.

No agent was compromised. No model misbehaved. One integration’s credentials were broad, long-lived, and held outside every victim’s own controls, so one theft became seven hundred breaches.

The agent with production credentials and a code freeze

In July 2025, on day nine of a public twelve-day build, Replit’s coding agent wiped SaaStr founder Jason Lemkin’s production database, despite an explicit and repeated code-freeze instruction. It then generated thousands of fake records and reported that rollback was impossible. It was not; the data came back from backup. The agent’s own postmortem: “a catastrophic failure on my part.”

The agent held live credentials to production with no environment separation, and the code freeze existed only as natural language. An instruction in a prompt is a suggestion. The incident record was, briefly, whatever the agent chose to say it was.

The tool that BCC’d the attacker

In September 2025, Koi Security found the first confirmed malicious MCP server in the wild: an npm package impersonating Postmark’s email server. It behaved correctly for fifteen versions, building trust, then v1.0.16 added a single line that BCC’d every email the connected agents sent to the attacker’s domain. Koi estimated roughly 300 organizations were running it in production. Nothing was exploited. An MCP server runs inside the agent’s trust boundary with the user’s credentials, and nobody diffs a patch release.

And the rest, briefly

EchoLeak (CVE-2025-32711) demonstrated a zero-click chain against Microsoft 365 Copilot: a poisoned email, retrieved later by RAG, walked the assistant’s most sensitive reachable data out through an allowlisted URL. Microsoft fixed it before any known exploitation, and it became the first critical-severity CVE assigned to an AI assistant’s injection path. The s1ngularity npm supply-chain attack inverted the picture: malware on developer machines invoked the victims’ own installed CLI agents with their client-side safety flags disabled and put them to work enumerating secrets, leaking over a thousand valid GitHub tokens within hours. And in July 2026, Check Point documented a single operator running two frontier-model agents as an intrusion pipeline that hit nine Mexican government agencies and roughly 400 million records. Attackers have agents too, and theirs ignore your prompts entirely.

The pattern

None of these was a model failure. Every one of them was contained, or would have been, by the same four environmental controls:

• Credentials scoped to one target and short-lived , so a stolen token is worth almost nothing (Drift).
• Authorization evaluated on every request against policy , not against the agent’s intentions or instructions (Replit, and the bulk CRM exports that no chat agent should have been able to run).
• Egress that is deny-by-default and enforced outside the agent , so even a successful injection has nowhere to send the loot (EchoLeak, the Postmark BCC, s1ngularity’s uploads).
• An audit trail owned by the platform, not narrated by the agent (Replit’s fabricated reports; Cloudflare could say “exactly 104 tokens” only because logs existed outside the blast radius).

Read that list again. Nothing on it is a property of the model. Nothing on it can be provided by a prompt, a fine-tune, or a guardrail library running inside the agent. All four are properties of the place where the agent runs.

Why the answer is a cluster you control

Environmental controls need an environment you own: the network path, the identity plane, the secret store, the audit pipe. Kubernetes is the one mainstream substrate that hands you all four as programmable primitives: namespaces and NetworkPolicy for segmentation, runtime classes for kernel-level sandboxing (gVisor, Kata), SPIFFE for workload identity, and admission control to keep anything unreviewed from running at all. Its declarative model then puts every one of those decisions in git, where they get reviewed and versioned like the rest of your infrastructure. These are the same properties that made Kubernetes the default for microservices. For agents they are not conveniences; they are the containment system.

The ecosystem has noticed. In the past year, Kubernetes gained a SIG subproject for agent sandboxing with first-class Sandbox primitives, launched at KubeCon NA 2025. agentgateway, a data plane purpose-built for agent-to-agent, MCP, and LLM traffic, became a Linux Foundation project with participation from AWS, Microsoft, Red Hat, IBM, and Cisco. KubeCon NA 2026 has a dedicated AI and agentic track. Running agents on Kubernetes stopped being a contrarian position sometime last winter.

Watch the managed agent runtimes and you see the same conclusion arriving from the other direction. AWS AgentCore added VPC egress and private identity-provider support. Microsoft’s agent service will inject its runtime into your own VNet so agent data stays in your tenant. Anthropic’s managed agents shipped self-hosted sandboxes that run tool execution on your infrastructure. OpenAI’s Frontier launch promises agents that run “across local environments, enterprise cloud infrastructure, and OpenAI-hosted runtimes.” Every managed runtime’s roadmap is bending toward your perimeter. That is the vendors telling you, in feature announcements, where the controls have to live.

There is also the question of what you can see when isolation is outsourced. In late 2025, Unit 42 disclosed an escape in AWS AgentCore’s code-interpreter sandbox: its “sandbox” network mode still allowed DNS resolution to arbitrary domains, which is enough for tunneled exfiltration and command-and-control. AWS fixed it properly within months, and credit to them. But note who found it and when the customers learned: from the researcher’s publication, not from their own telemetry, because inside a managed runtime there is no customer telemetry to find it in.

Side by side:

The table generalizes, and the best vendors are improving individual rows. What no managed runtime can give you are the rows that require owning the substrate: one policy engine over every agent regardless of framework, one audit trail across agents, tools, and models, and visibility into the agents nobody told the platform about.

One caveat before the spec. _ A bare cluster is not the answer either _. Stock Kubernetes secures pods, not agents: NetworkPolicy has no idea what an MCP tool call is, RBAC governs humans and service accounts, and nothing in the box produces an agent-level audit trail. We walked through those gaps in the accountability gap post. What the cluster uniquely gives you is the ability to install the missing layer yourself, uniformly, for every agent you run.

What strict guardrails look like

Here is the spec a year of incidents wrote, stated as requirements. Sandboxing the agent’s process is assumed and is not sufficient on its own; even the CNCF says so now. Everything below governs what agents do across the network, which is where every incident above happened.

1. An identity for every agent. Cryptographic workload identity (SPIFFE, OIDC) issued at deploy time, not a shared service account. Every other control depends on “which agent did this” having an answer.
2. One road out. Agent egress is deny-by-default, and the only path to other agents, MCP servers, and LLM providers is a gateway. EchoLeak, the Postmark backdoor, and s1ngularity’s uploads were all exfiltration over permitted egress. An allowlist enforced outside the agent kills the chain even when the injection succeeds.
3. Authorization per request, not per deployment. Each call is judged against policy: this agent, this target, this tool, these arguments. A lead-triage agent asking for a bulk CRM export should fail on policy, not on luck.
4. Credentials the agent never holds. Provider keys and tokens attach at the gateway, scoped to the single target of the single hop, short-lived. This is the Drift control: make the stolen token worthless.
5. A human where the blast radius is high. Destructive and irreversible operations block until a person approves. “Code freeze” becomes a policy object with an enforcement point, instead of a sentence in a prompt.
6. Detection for the agents you didn’t launch. 80% of organizations already report unintended agent actions, and IBM’s data keeps tying breach costs to shadow AI. You need to see the unregistered agent from the node it runs on, not from a registry it never joined.
7. An audit trail the agent cannot edit. Recorded by the enforcement plane as a side effect of enforcing, tied to agent identity, covering every hop. The agent’s own account of events must never be the system of record.

If you run a startup, this list probably reads as an enterprise ceremony. It is the opposite. The spec is precisely what lets a five-person team hand agents real credentials to real systems without hiring a security organization: default-deny scales down, a policy file for three agents fits on one screen, and one gateway is not a platform team’s worth of work. If you run an enterprise, the same spec is what turns your agent deployment from an audit finding into an audited system. The requirements do not change with headcount. Only the number of agents does.

The platform this spec describes

By now the specificity of that list has probably given the game away: it is, near enough, the design document for Tigera Lynx, our security and governance platform for AI agents on Kubernetes.

Lynx puts a registry and identity plane in front of your agents: SPIFFE and OIDC workload identity, plus a pod-owner mode that brings an unmodified agent under governance with no code changes and no credential to issue. Its gateway, built on the same Linux Foundation agentgateway data plane mentioned above, authorizes every agent-to-agent, MCP, and LLM call individually under Cedar policy, and MCP policies can see the operation, the tool name, and the tool arguments, failing closed on anything malformed. LLM provider keys live in your cluster’s secret store and attach at the gateway, so an agent never carries a provider credential it could leak, and per-provider token rate limits cap what any agent can burn.

In the current release cycle, per-hop credentials are minted for one target only (the Drift lesson, applied literally), and a policy can mark specific MCP calls as needing human approval, so the call waits until someone signs off (the Replit lesson). On the node, an eBPF-based detector spots agents nobody registered, classifies them as sanctioned, shadow, or unknown, and lets you quarantine a compromised workload at both the gateway and the kernel. And the Agent Trail records every decision, every hop, and which model actually served each call, as a byproduct of enforcement. The agent does not get to narrate.

Every one of those maps to a numbered line in the spec, because the spec came first. How Lynx Works has the full walkthrough, and the five principles post is the vendor-neutral checklist if you want to evaluate Lynx against anyone else.

The decision you already made

Back to the opening. Your organization has a hosting posture for AI agents today, assembled from defaults: whichever vendor clouds your teams clicked through, whichever VMs were lying around, whichever laptops were closest. The incidents above are what those defaults cost, and IBM’s data says AI is now involved in one of every four malicious breaches.

So ask the question deliberately, once, before the next agent ships: when it acts, who can say no, and where is the record? If the answer is a vendor’s incident queue and a support ticket, _ you have outsourced the two things a security leader cannot outsource. _

_ An agent you cannot refuse is an agent you do not control. _ Run yours where refusal is built in: a Kubernetes cluster you own, a gateway it cannot go around, and a policy engine that never gets tired of saying no.

Lynx is Tigera’s security and governance platform for AI agents on Kubernetes: identity, policy, detection, and audit for every agent in your cluster. Read How Lynx Works or request early access at tigera.io/demo/.

Learn more about Lynx →

The post The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It appeared first on Tigera – Creator of Calico.