The AI red teaming market grew up fast this year. OpenAI bought Promptfoo, Cisco and Microsoft shipped automated attack suites, and a seed-stage startup publicly compromised 50 of 55 live customer service bots. These platforms find real problems at a scale no human team can match. But when you read the findings closely, a pattern emerges: agents talked into refunds, transfers, and data leaks they had standing authority to perform. Patching the prompt fixes one phrasing until the next model update. Constraining the authority fixes the class. The first job belongs to a red team platform. The second belongs to your runtime, and no scanner will do it for you.
In April 2026, General Analysis raised a $10M seed round on the strength of an uncomfortable demonstration: its adversarial agent attacked 55 live customer service bots and compromised 50 of them. Not lab models, but live systems with real customers and real tool access. This post is about the market behind that demonstration: who now automates the attacker’s role, what the attacks keep finding, and why the fix that lasts is runtime policy rather than a better prompt.
What an AI red team agent actually does
A traditional red team is a group of people paid to break your system before someone else does. An AI red team agent automates that role for AI systems. It fires thousands of adversarial inputs at a model or agent, notices which ones partially work, mutates them, escalates across turns, and reports what got through, with severity ratings and reproduction steps. A human tester works through a checklist over a week. These platforms run 20,000 variations overnight and rank the results.
The canonical demo goes like this. A company deploys an agent that handles refunds and can reach the billing system. The red team platform embeds disguised instructions in customer messages until it finds the phrasing that makes the agent skip its verification step and process a refund it shouldn’t. The report names the exact prompt pattern so engineering can fix it.
Enterprises are buying for reasons that have little to do with fashion. Agents now do things, such as executing code, querying databases, sending emails, and calling APIs, often with minimal supervision, so every capability is an attack vector. The EU AI Act mandates adversarial testing for high-risk AI systems, and insurers and procurement teams have started asking for documented testing as a condition of coverage or vendor selection. And a quarterly pen-test calendar cannot keep up with an agent whose behavior shifts on every model update.
All of that is real, and the tooling is genuinely good. None of this is an argument against red teaming. The question is what you do on the morning the report arrives.
The AI red teaming market grew up in about a quarter
The speed of consolidation tells you how seriously the industry takes automated red teaming. In March 2026, OpenAI acquired Promptfoo for a reported $86M; the MIT-licensed scanner claims 350,000 developers and now ships inside OpenAI’s Frontier enterprise platform. Cisco AI Defense tests across 200+ risk subcategories mapped to OWASP, NIST, and MITRE ATLAS. Microsoft’s AI Red Teaming Agent wraps PyRIT’s multi-turn attack orchestration, including Crescendo and Tree of Attacks with Pruning, into Azure AI Foundry. Mindgard starts with reconnaissance: it maps your guardrails, tools, and shadow AI deployments before it fires a single probe. And General Analysis goes deepest on agentic systems: MCP server exploitation, memory poisoning, and multi-step permission bypass.
OWASP now publishes a quarterly landscape document just for this category. When a market gets its own recurring OWASP publication, it has stopped being a niche.
Read the findings like a defender
Look at what these platforms actually catch across every vendor’s case studies: an agent processing unauthorized refunds. An agent skipping its verification step. An agent revealing one customer’s data to another. A moderation model bypassed with encoding tricks. An agent walked, over several polite turns, across a permission boundary it was never supposed to cross. A poisoned memory that redirects behavior days later. A malicious MCP server that turns one compromised tool into several.
Sort those into two piles. In the first pile, the model said something it shouldn’t have (jailbreaks and moderation bypasses). Those are model-layer problems with model-layer fixes, and the labs keep getting better at them. In the second pile, the agent did something it shouldn’t have. Every headline finding in the agentic column lands here, and every one of them has the same anatomy: the agent was persuaded, and the environment obeyed.
The persuasion is novel every time. That is the entire point of automated attack generation; there is always another phrasing, another encoding, another escalation path. The obedience is identical every time. The agent asked for transfer_funds, and something executed it.
Prompt patches decay. Runtime policy doesn’t.
Here is the loop most teams fall into. The red team report names a prompt pattern. Engineering patches the system prompt or adds a guardrail rule. The finding closes. A model update ships three weeks later, and the finding reopens with different wording. The vendors know this; it’s why General Analysis sells regression testing for exactly the case where a passing agent configuration fails after a model update or prompt change, and why every platform now pitches continuous scanning rather than an annual engagement. Continuous testing is the industry’s honest admission that the layer under test never stays fixed.
We’ve made this argument before, in the accountability gap post and throughout the series: controls the agent can override are not controls. The same logic applies to instructions. An agent can be talked out of its system prompt; that is precisely what the red team platform just proved. It cannot be talked out of a deny that lives outside its process.
So a pile-two finding deserves two responses. Patch the prompt, by all means; make the attack more expensive. But the fix that lasts is changing what the persuasion can accomplish, and that change happens in the environment, at the point where the agent’s request becomes an action.
Rewriting a red team finding as runtime policy
Take the marquee finding, the one on every vendor’s landing page: an injected instruction convinces a finance agent to call transfer_funds without its verification step. As a reproduction step, that’s a ticket that will reopen. As policy, it looks like this:
@approvalRequired("Approve transfer_funds on treasury")
forbid ( principal, action == Action::"callMCPServer", resource )
unless { context.approval.granted }
when { context.mcp.tool.name == "transfer_funds" };
This is Cedar, evaluated at a gateway between the agent and its MCP servers. Any call to transfer_funds is denied unless a human has granted approval, and the annotation turns that deny into a hold: the gateway parks the request, a person approves or rejects it, and only then does anything move. Approval is an input to the policy decision, not an override of it. The injection can still convince the agent to try. Trying now produces a held request and an audit entry instead of a wire transfer.
The same translation works across the rest of pile two:
| What the red team keeps finding | Where the durable fix lives |
|---|---|
| Prompt injection triggers an unauthorized tool call | Per-request authorization at a gateway the agent cannot route around |
| Agent skips a verification or approval step | A policy hold that parks the call until a human decides |
| Agent leaks one customer’s data to another | Tool and egress policy scoped to the agent’s verified identity |
| Multi-step chains cross a permission boundary | Per-call decisions with the full chain correlated in one trace |
| The agent nobody tested, because nobody knew it existed | Runtime detection of unregistered agents |
That last row deserves a sentence. Mindgard leads with reconnaissance because most organizations cannot produce an accurate inventory of their own AI systems, and you cannot red team what you don’t know you’re running. Scoping the engagement is where a program fails first, before a single probe fires. Inventory is a runtime problem too.
This is the seam where Lynx sits. Its gateway authorizes every MCP call individually against Cedar policy under a workload identity (e.g. SPIFFE). The @approvalRequired hold shown above shipped in preview this cycle, with the pending queue and countdown surfaced in the dashboard. Agent Trail records the deny that started the hold and the human decision that resolved it. And runtime detection classifies the agents actually present in your cluster, so the inventory a red team engagement needs is a query rather than an archaeology project. None of that replaces adversarial testing. It’s what turns the testing’s output into something that stays fixed.
Starting with open source: Garak, PyRIT, and DeepTeam
If you want to start this quarter, the open source route is credible. Garak (by NVIDIA, under Apache 2.0 license) is the deepest free attack catalogue: 120+ probe modules, with multi-turn agentic attacks added in v0.15.0 this May. PyRIT (Microsoft, MIT) is the strongest orchestrator for custom multi-turn campaigns. DeepTeam (Apache 2.0 license) maps results onto OWASP’s Agentic Security Initiative taxonomy for the compliance evidence trail; disable its cloud scoring calls if your data can’t leave the building. One caveat worth knowing before you standardize: Promptfoo belongs to OpenAI and PyRIT to Microsoft, both model vendors. The attack logic is open source and auditable, which helps, but neither should be your only scanner when the system under test is that vendor’s own model.
Two practices make whichever stack you pick actually pay off. First, run it against a test double, not only production; we covered how to stub LLMs for exactly this, so you can exercise the agent’s authority paths deterministically. Second, file every confirmed agentic finding as two tickets: one for the prompt, one for the policy that makes the prompt’s failure survivable. If your enforcement point logs its decisions, the audit trail becomes your regression oracle. The next scan shouldn’t just show the attack failing; it should show the gateway denying it.
Key takeaways
- AI red team agents automate adversarial testing at a scale no human program matches, and the market consolidated fast: OpenAI bought Promptfoo, Cisco and Microsoft ship attack suites, and OWASP now tracks the category quarterly.
- The agentic findings share one anatomy: the agent was persuaded, and the environment obeyed. Persuasion varies infinitely. Obedience is the constant you can actually fix.
- Prompt patches decay with every model update, which is why the vendors themselves sell continuous regression testing. Policy enforced in the environment doesn’t decay with the model.
- Translate findings into runtime controls: per-request authorization, human approval holds on dangerous tools, identity-scoped access, and detection for the agents you didn’t know you had.
- Buy the red team. Build the enforcement. They are complements, and neither substitutes for the other.
Back to the fifty bots. What they had in common was not a worse model or a lazier system prompt than the five survivors. It was that persuasion and permission were the same thing: once the agent was convinced, nothing else got a vote. Your next red team report will show you where that’s true in your own stack. When it does, which findings will you patch, and which will you enforce?
The red team finds the words that move your agent. Your runtime decides what moving accomplishes.
Lynx is Tigera’s security and governance platform for AI agents on Kubernetes: identity, policy, detection, and audit for every agent in your cluster. Read How Lynx Works or request access at tigera.io/demo/.
The post AI Red Team Agents Automate Attacks on your AI Agents. Runtime Policies Automate their Defense. appeared first on Tigera – Creator of Calico.
