Hot take: smaller prompts can be more expensive
If you run agents that call tools, wait for external jobs, or stitch together multi-turn reasoning, you probably obsess about token counts. That’s natural — but token-minimization is the wrong first principle when provider caches are in play.
This article explains why prompt caching for agentic systems changes the economics and engineering tradeoffs, how aggressive compaction or retroactive pruning destroys cache hits, and what to do instead. It includes a short code example and a 5-step checklist you can try on one flow this week.
How prompt caching actually works (short primer)
Modern providers save work by storing the model’s internal key/value (KV) state after processing a prompt prefix. When a later request begins with the same prefix, the provider can reuse that KV state instead of reprocessing those tokens — reducing time-to-first-token (TTFT) and input billing.
Important provider realities you must design around:
- Exact prefix matching: caches typically match on verbatim prefixes or on explicit breakpoints. Any change in the cached prefix invalidates a hit.
- Minimum cacheable length: many models require a minimum prefix (commonly ~1,024 tokens on recent models) before a write is eligible.
- Billing asymmetry: cache writes may bill at a slightly higher rate (e.g., 1.25× on some newer model families) while cache reads are billed at a small fraction of full input price (often ~0.1×). That makes writes worthwhile when the prefix is reused.
- Routing and stickiness: requests are routed by a prefix hash (and optionally a prompt_cache_key). Each machine handles limited throughput per prefix (roughly ~15 requests/min). Excess traffic spills to other machines and produces cache misses.
These behaviors make caching a high-leverage optimization — when you structure prompts to get hits — and a fragile one when you break exactness unnecessarily.
Why aggressive compression or reordering backfires
When teams aggressively compress, reorder, or retroactively edit history to shave tokens, they often change the cached prefix. That causes:
- Cache misses and full re-prefill of the prefix.
- Extra diagnostic turns and retransmissions to recover lost anchors or explain missing context.
- Repeated cache writes for content that will never be reused.
A practical example: shrinking tool output by 38.4% in one study increased billed costs by 6.8% because the compaction invalidated cache hits and forced re-runs. In another coding-agent test, aggressive compression broke code anchors and reduced patch success from 27/40 to 15/40. The marginal token saved was dwarfed by extra agent turns, latency, and developer time.
The lesson: the monetary and latency value of a token depends on whether it’s served from cache or processed anew.
Principles: preserve cacheable anchors, isolate volatility
A simple rule that scales: keep cacheable anchors (system prompt, tool definitions, stable instructions) verbatim at the front of the prompt. Move volatile content (tool outputs, ephemeral runtime state, timestamps) outside the cached prefix.
Conservative tactics that work across providers:
- Pre-inject compressed summaries before a cache write instead of retroactively pruning cached history.
- Use explicit breakpoints (when available) to declare the end of the cached prefix and avoid writing volatile suffixes.
- Avoid inline timestamps, UUIDs, or session-specific strings in the cached area.
Small code example (conceptual)
This pseudocode shows the relocation pattern: compute and store a static prefix, then append a dynamic block at request-time.
cache_key = hash(static_prompt)
store_cache(cache_key, static_prompt)
prefix = get_cache(cache_key)
// dynamic_blob contains tool outputs, fresh observations, etc.
prompt = prefix + "
" + dynamic_blob
send_request(prompt)
And an OpenAI-style explicit breakpoint example (JSON snippet) that avoids caching the dynamic suffix:
{
"messages": [
{"role": "system", "content": "
"},
{"role": "user", "content": ""}
],
"prompt_cache_options": {"mode": "explicit", "ttl": "30m"},
"prompt_cache_key": "my-agent-shard-42"
}
(Provider fields differ; use the equivalent controls for your API.)
Operational knobs and recommended starting values
- TTL: Start conservatively. Many providers have defaults (OpenAI ~30m, Anthropic ~5m). For interactive sessions tune between 10–60 minutes depending on session churn.
- Keepalive / warm pings: When agents pause for external work (builds, human approvals), inexpensive keepalive reads can refresh a cached prefix and avoid eviction. Pick a ping interval safely under the provider’s eviction point (e.g., ~240–480s depending on provider) — but measure first to avoid pinging a dead cache and paying full write costs.
- Routing key: Use prompt_cache_key consistently for related prefixes to improve routing stickiness. Don’t overload a single key above ~15 RPM without sharding.
- Pre-injection threshold: Instead of compacting tool outputs aggressively at 1k of accumulated tokens, consider pre-injecting compressed summaries only when the block is large (for example, >10k). In one ops flow we switched from a 1k summarization threshold to a conservative 10k pre-injection policy; latency dropped and accuracy rose. The 1k setting had increased end-to-end latency by ~177% due to extra re-runs.
- Privacy guardrails: Redact or tokenise PII before caching and enforce invalidation on permission changes.
Checklist: 5 practical steps to try this week
1) Breakpoints — Identify static vs dynamic: cache system prompts and tool anchors; keep runtime state outside the prefix.
2) TTL — Start with a conservative TTL (10–60m) and tune by measuring cache hit rate vs staleness.
3) Keepalive interval — Use lightweight keepalives to sustain hot keys during pauses. Start ~60s and adjust to provider behavior.
4) Relocation trick — Inject frequently-changing data as a separate dynamic block or metadata header so the cached prefix remains byte-identical.
5) Privacy guardrails — Redact or tokenise sensitive data before caching and invalidate caches when permissions change.
Measure and iterate
Don’t guess: measure cached_tokens, cache_write_tokens, TTFT, hit-rate, end-to-end latency, and billed reads. Run an A/B on one flow (e.g., the most expensive agentic path) and log before/after. Important signals:
- Hit-rate: % of requests that read cached tokens.
- Cached read fraction: how much of the input was served from cache.
- Re-run count: diagnostic or retransmission turns per session.
- Patch/operation success rate (for coding agents).
If your token savings cause hit-rate to drop, the net bill or latency can increase. Optimize for net cost and latency, not raw token count.
Closing: think cache-first, not token-first
Prompt caching for agentic systems rewards stability and deliberate boundary control. Aggressive token-surgery that changes cached prefixes often destroys the very lever you were trying to exploit. Preserve anchors, move volatility to dynamic blocks, warm hot keys across pauses, and measure the real impact on hit-rate, latency, and billed reads.
Try the 5-step checklist on one high-cost flow this week. If you apply it to a coding agent, ops automation, or an interactive assistant, track hit-rate, TTFT, and total billed reads — you’ll usually find the marginal token saved wasn’t worth the cache hit lost.
Which flow in your stack would you try first?
