A 2026 survey of prompt caching among LLM API teams found that 72% of teams using LLM APIs are not using prompt caching at all. Prefix caching — the mechanism that actually determines whether you get the discount you think you're getting — is an exact-match game The headline 90% discount is real, but it's meaningless without prefix stability.
What is prefix caching and why does it matter?
Prefix caching stores the computed key-value state of a prompt's leading tokens so that subsequent requests sharing that exact prefix skip recomputation and are billed at a reduced cached-input rate. Every major provider — OpenAI, Anthropic, Google — offers some form of this. The discount is real: as of mid-2026, cached input tokens cost roughly 10% of the full price across the board. But the mechanism is fragile. Caching matches on an exact byte-for-byte prefix from the first token; a single changed character in the prefix invalidates the cache from the divergence point forward.
The economic stakes are high because inference now accounts for roughly 70% of total AI compute costs per DigitalOcean, and prefill computation scales quadratically with input length. That means attention computation quadruples when input length doubles. For agentic workloads — which consume 5 to 30 times more tokens per task than standard chatbot exchanges per Spheron — input tokens from repeated context resends dominate the bill. Output tokens never receive a caching discount, so the pricier a model's output, the less caching can save on total cost. Teams monitoring only output costs are flying blind on the actual cost driver.
How does prefix caching differ from prompt caching and semantic caching?
This is where most teams get confused. There are three unrelated things called "caching" in LLM serving, operating at three different layers, and conflating them costs money or correctness. Prompt caching is a provider billing feature: you mark a prefix, the API skips recomputing it and discounts those tokens. Prefix caching — the subject of this post — is an inference-engine feature: it reuses KV-cache tensors in GPU memory across requests. Semantic caching is an application-layer store that returns a previously generated response when a new query is embedding-similar to an old one.
The rule is simple: prompt and prefix caching skip recomputation and never change the output; semantic caching skips the model entirely and can. Treat the first two as free wins and the third as a correctness decision. If you're running a RAG pipeline or agent system, you need to understand which layer you're actually operating in. Self-hosted teams running vLLM or SGLang are working with prefix caching, which reduces latency and improves throughput without appearing on a bill. Hosted API users are working with prompt caching, which provides direct billing discounts. The SGLang inference engine uses RadixAttention for prefix caching, while vLLM's automatic prefix caching is enabled by default on the V1 engine, using a hash-based cache where each 16-token block is identified by a chain hash per packet.ai.
How much can prefix caching actually save you?
The headline discount is 90% off cached input tokens. DeepSeek bills cache hits at 2% of full input price per Foreverse. Anthropic charges cache reads at 0.1x base input price (90% discount) and applies a cache-write surcharge of 1.25x for a 5-minute TTL or 2.0x for a 1-hour TTL per packet.ai. OpenAI's caching is automatic for prefixes above a 1,024-to-2,048-token minimum depending on the model per packet.ai. Gemini offers implicit caching (90% discount on Gemini 2.5+, 75% on 2.0) with no guarantee of a hit, and explicit caching that guarantees the discount but requires manual setup and carries an hourly storage fee per packet.ai.
But here's the catch: the 90% headline discount is meaningless without prefix stability. In measured production traffic with identical byte-identical scripts, fixing prompt structure to preserve prefix stability raised cache hit rates from 46.5% to 89.9%, cutting per-session costs by roughly 3x on DeepSeek V4 Flash (from $0.20 to $0.054 per 100-round session). The difference between broken and correct prefix ordering can mean paying 2-4x more than identical traffic with proper layout. The tradeoff between write cost and cache persistence is explicit: shorter TTLs mean cheaper writes but more frequent cache misses as prefixes expire; longer TTLs preserve hits longer but cost more to write.
On September 1, 2026, Anthropic released Claude Fable 5.1, cutting cache-read pricing by 75% to $0.25 per million tokens (from $1.00) while keeping input at $10/M and output at $50/M per TechSpot. Anthropic estimates this yields approximately 25% savings on typical workloads and up to 45% on cache-heavy agentic tasks — but only if your hit rate is already high. If your prefix contract is broken, that 75% price cut applies to a much smaller base than you think.
What breaks prefix caching (and how to fix it)?
The single most common mistake that silences caching is placing variable, per-request content — user ID, timestamp, session value — ahead of static content in a prompt, which breaks the prefix match from the very first token per packet.ai. This is what I call the Stable Prefix Contract: cache economics are determined by client-side prompt architecture, not provider-side discounts. Teams with broken prefix ordering operate at 40-50% hit rates despite having caching enabled, paying 2-4x more than identical traffic with correct layout.
The fix is mechanical but requires discipline. Stable content — system prompt, tool definitions, few-shot examples, policy text — belongs at the top in a byte-stable order. Volatile content — user message, retrieved RAG chunks, session state, per-request values — belongs at the bottom. If you're using an agent framework, check where it injects dynamic content. Many popular frameworks inject timestamps, memory snippets, or tool results at the top of the prompt by default, silently destroying your cache hit rate. The prompt caching cost guide breaks down provider pricing and write premiums in more detail, but the architecture rule is universal: static first, dynamic last.
When should you use self-hosted vs. hosted prefix caching?
The tradeoff is straightforward. Self-hosted prefix caching in engines like vLLM and SGLang reuses KV-cache tensors in GPU memory to reduce latency and improve throughput without appearing on a bill per Dreaming Press. Hosted API prompt caching provides direct billing discounts — 90% off input tokens — but requires careful prompt engineering to hit. If you're already self-hosting for latency or data-control reasons, prefix caching is a free win. If you're on hosted APIs, the savings are real but conditional on hit rate.
The routing layer matters too. Naive round-robin load balancing is actively destructive to LLM inference economics, degrading cache hit rates linearly as replica fleets grow. The AI load balancing guide covers this in depth, but the principle is simple: sticky routing to cache-warm servers beats distribution for hit rate, until saturation.
| Approach | Layer | Cost Model | Hit Rate Dependency | Setup Complexity |
|---|---|---|---|---|
| Self-hosted prefix caching (vLLM, SGLang) | Inference engine (GPU memory) | No direct billing discount; saves compute/latency | Medium — depends on request routing and cache eviction | Low — often enabled by default |
| Hosted prompt caching (OpenAI, Anthropic, Gemini) | Provider API (billing) | 90% discount on cached input tokens | High — requires exact prefix stability and minimum token lengths | Low-Medium — automatic or manual per provider |
| Semantic caching (GPTCache, etc.) | Application layer (response store) | Skips model call entirely | Variable (5-90%) — depends on embedding similarity | Medium-High — requires threshold tuning and correctness monitoring |
The decision framework is simple: if your team has the expertise to manage GPU infrastructure and your workload has high prefix reuse, self-hosted prefix caching gives you latency wins for free. If you're on hosted APIs and your prompts have stable structure, prompt caching is the highest-leverage cost optimization available. If you need both, you're looking at a hybrid architecture — and that's where cache-aware routing becomes non-negotiable.
The bottom line
Prompt caching should be treated as a mandatory architecture review, not an optional optimization. The 72% of teams not using it, plus the majority with broken prefix contracts, will face existential cost scaling problems as agentic workloads go from 5-30x to 1000x token multipliers. The teams that audit and fix their hit rates first will capture a 2-4x cost advantage that has nothing to do with model choice. Start by logging your cache hit rate — most OpenAI-compatible endpoints report it in the usage object — and then audit your prompt structure. The savings are already there if you're willing to do the architecture work to claim them.
Originally published at SaaS with Alex
