Why traditional security tooling breaks down for LLM and agent traffic, and what actually needs to sit in the request path
If you've put an AI agent into production, you've probably already noticed the uncomfortable gap. Your WAF is happily passing traffic because every request is a well formed JSON payload. Meanwhile the actual risk, a prompt telling the model to ignore its instructions or call a tool it has no business calling, sails straight through because nothing about it looks malicious at the protocol level.
That's the core problem. Security tooling built for the API era inspects syntax: headers, payload size, known attack signatures. LLM and agent traffic is an attack surface made of meaning, not syntax, and once you add agents into the mix, it's not even a single request anymore. It's a chain of autonomous decisions, any one of which can be hijacked.
Why agents make this worse
A plain LLM call that gets manipulated produces bad text. An agent that gets manipulated takes an action. It queries a database, hits an internal API, sends an email. The injected instruction doesn't have to come from the user typing something suspicious either. It can be buried in a PDF the agent retrieves, or hidden in the output of a tool it just called.
Rate limiting won't catch this because request volume looks completely normal. A compromised agent doing exactly what an attacker wants still fits well within your quota.
Where enforcement actually needs to live
The practical answer is a control point that sees every prompt, every tool call, and every completion before it goes anywhere, and applies policy inline rather than logging problems after the fact. That's the job of an AI gateway, and it's a genuinely different job than a guardrails library bolted onto one service, since guardrails have to be reimplemented per application while a gateway enforces consistently across all of them.
A gateway worth the name should be doing a few concrete things:
- Inspecting prompts, tool calls, and completions for injection attempts and policy violations in real time
- Scoping access per token, per agent, per tool, so a single compromised credential doesn't unlock everything
- Redacting PII in both directions, inbound before it reaches the model, outbound before it reaches the caller
- Rate limiting per user and per agent to contain both abuse and runaway agentic loops
- Logging everything in a tamper resistant audit trail
- Failing closed when a check can't complete, rather than letting the request through by default
That last point matters more than it sounds. If your injection classifier times out and the fallback is "allow," you've quietly turned every service degradation into a security hole.
Mapping this to a real threat model
The OWASP Top 10 for LLM Applications is the closest thing the field has to a standard reference here, and it maps cleanly onto gateway controls. Prompt Injection (LLM01) and Sensitive Information Disclosure (LLM02) are the two most obviously suited to inline inspection and redaction. Excessive Agency (LLM06), an agent given more tool access than the task actually requires, is where per-agent, per-tool permissioning earns its keep. Unbounded Consumption (LLM10) is what rate limiting and quotas are for.
Not everything on that list belongs at the gateway. Supply chain risk and training/fine-tuning data poisoning happen upstream of runtime traffic, so a gateway can restrict which model versions an app is allowed to call, but it's not a substitute for governance further back in the pipeline.
What a policy actually looks like
Concretely, this tends to be declarative config rather than code scattered across services:
policy:
fail_closed: true
rules:
- type: prompt_injection
action: block
sensitivity: high
- type: pii_detection
scope: [inbound, outbound]
action: redact
- type: agent_tool_access
allowed_tools: [search_docs, read_ticket]
denied_tools: [send_email, execute_code]
require_approval: [update_database]
- type: rate_limit
scope: per_agent
limit: 100
window: 1m
The specifics vary by platform, but the shape is always the same: block by default, scope tools tightly, redact before data leaves your boundary.
Compliance is starting to demand this anyway
This isn't purely a "nice to have" argument anymore. The EU AI Act requires high risk AI systems to support automatic event logging (Article 12) and requires providers and deployers to retain those logs, which pushes audit trails from best practice into a compliance requirement for a lot of teams shipping agentic systems into the EU.
Enforcement vs. assessment
It's worth separating two things that get conflated a lot: enforcing policy inline versus assessing agent risk and running adversarial tests against your workflows to figure out what that policy should even be. NeuralTrust splits these into TrustGate, the inline enforcement gateway, and TrustGuard, the governance and threat detection layer that informs it. If you want the fuller picture on agent-specific threats and how gateways fit into securing them, NeuralTrust's original writeup on this topic and their AI agent security overview are both worth a look, as is agentsecurity.com if you're mapping out the broader landscape of tooling in this space.
The bottom line: if your security stack can't read the meaning of a prompt or the intent behind a tool call, it isn't actually securing your LLM and agent traffic, it's just watching it go by.
