Your API Gateway Rate Limit Can't See the Agent Looping Your Search Tool

Here's the failure mode nobody's gateway catches.

An agent gets stuck. Not stuck like a crash — stuck like a loop: it calls web_search, gets results, doesn't like them, calls web_search again with a slightly different query. 10 times. 50 times. 400 times a minute. Each call is a perfectly valid HTTP request. Your API gateway sees normal traffic well under its per-key limit. Your bill disagrees.

The problem: gateway rate limits live at the HTTP layer. Agent abuse lives at the tool-call semantic layer. One HTTP request can fan out into many tool calls. One looping tool can spread its calls across many endpoints. A gateway counting bytes and paths never sees "the agent is hammering one expensive tool" — but that's exactly the unit an agent developer wants to budget.

I built the fuse that belongs at the tool-call layer.

What I built

tool-throttle — a zero-dependency Python package (stdlib only, Python ≥ 3.9) that puts a token-bucket rate limiter on MCP tool calls, per tool name. Wrap your tool dispatcher, set quotas, and over-quota calls get queued or rejected with a retry_after hint.

pip install tool-throttle

from tool_throttle import Throttler, ThrottlerConfig, Quota, REJECT

th = Throttler(ThrottlerConfig(
    default=Quota(calls=60, per_seconds=60),
    per_tool={
        "web_search":  Quota(calls=10, per_seconds=60),  # tight leash
        "expensive_*": Quota(calls=2, per_seconds=60),   # glob patterns
    },
    on_exceed=REJECT,  # or "queue": wait for a token, up to max_queue_wait
))

safe_search = th.wrap("web_search", mcp_client.call_tool)
safe_search("web_search", {"query": "..."})  # past quota -> ThrottledError

th.stats()  # per-tool allowed / rejected / queued / waited_seconds

Quotas live in the language of the agent — tool names and glob patterns, with per-tool bursts — not in the language of HTTP. Config round-trips through JSON so it can sit in a file next to your agent config:

cfg = ThrottlerConfig(
    default=Quota(calls=60, per_seconds=60),
    per_tool={"web_*": Quota(calls=10, per_seconds=60, burst=3)},
    on_exceed="queue",
    max_queue_wait=30.0,
)
cfg.save("throttle.json")

There's also a CLI for quick checks and a demo:

echo '{"tool": "web_search"}' | tool-throttle check --config throttle.json
echo '{"tool": "web_search"}' | tool-throttle acquire --config throttle.json
tool-throttle demo   # watch a looping agent get fused

How it works

Each distinct tool name gets its own TokenBucket, sized by the first matching quota in per_tool (glob patterns, insertion order) or default. Buckets refill steadily — Quota(calls=10, per_seconds=60) sustains 10 calls/minute with a burst of up to burst (default: calls).

On quota exhaustion:

reject — fail fast with ThrottledError, carrying a retry_after hint so the agent can back off intelligently instead of hammering blindly.
queue — block until a token frees up, bounded by max_queue_wait; if the bound would be exceeded, fail fast instead. Queuing keeps bursty-but-legitimate agents flowing while still capping the rate.

Everything is thread-safe, time is injectable (the test suite runs against a fake clock), and there are no dependencies — it's threading, time, json, fnmatch, and argparse.

Why this isn't just "rate limiting, again"

Because the layer matters:

Gateway limits are per key/IP. An agent looping on one expensive tool looks normal at the HTTP layer while burning money at the tool layer. tool-throttle counts tool-call intent.
Tool granularity. You don't want the same budget for get_time and web_search. Per-tool quotas with globs (expensive_*) express what you actually fear.
Fuse, not wall. Queuing with a bounded wait absorbs legitimate bursts instead of erroring on the first spike — it's a fuse that trips on sustained abuse, not a wall that punishes normal use.
Observability. stats() tells you which tool the agent was abusing, with waited-time accounting. That's the number you want when you're debugging a runaway agent at 2am.

It complements gateway rate limiting; it doesn't replace it. The gateway protects your infrastructure. tool-throttle protects your wallet from your own agent.