A few months ago I was running a long-context RAG setup locally, feeding a model a big pile of documents and asking it questions across the whole thing. The model weights fit on my GPU with room to spare. Then I pushed the context past 60–70K tokens and the process died with an out-of-memory error, as if the model had suddenly gotten bigger.
It hadn’t. What grew was something most people never look at directly: the KV cache. Once I understood what it actually was, the OOM errors stopped being mysterious, and a lot of pricing decisions I’d seen from API providers, like charging less for “cached” input tokens, finally made sense too.
This is the explanation I wish I’d had before I started debugging blind.
The problem the KV cache solves
Transformers generate text one token at a time. To generate token number 5,000, the attention mechanism needs to look back at tokens 1 through 4,999. Specifically, it needs two numerical representations of each of those tokens: a Key vector and a Value vector.
Without caching, the model would recompute the Key and Value vectors for every previous token, every single time it generates a new one. Token 5,000 would require recomputing 1 through 4,999. Token 5,001 would recompute 1 through 5,000. That’s roughly quadratic work for no reason: the Keys and Values for token 1 don’t change just because you generated more text after it.
So instead, the model computes each token’s Key and Value once and stores them:
new token arrives -> compute its K and V -> store in cache -> reuse for every future token
That’s the whole idea. It’s a straightforward memoization trick, and it’s the reason autoregressive generation is fast at all.
But memoization has a cost: you have to keep the memo somewhere. And the “somewhere” is GPU memory that also has to hold the model weights, activations, and everything else. The more tokens you cache, the less room there is for everything else, and the more data attention has to read on every single step.
One detail that trips people up: the KV cache doesn’t store your text. It stores tensors, numbers produced by the attention layers. You can’t inspect it and see your prompt. That distinction matters when you’re debugging memory, because “the cache is huge” and “the model knows a lot” are unrelated facts.
Two very different phases: prefill and decode
I used to think of “running an LLM” as one homogeneous thing. It isn’t. Every request goes through two phases that stress the GPU in opposite ways.
Prefill happens when your prompt first arrives. The model processes all of your input tokens at once, in parallel, and builds the KV cache for the entire prompt in one pass. This is a big parallel matrix-multiplication job, so it’s compute-bound: the GPU is mostly limited by how fast it can crunch numbers.
Decode happens after that, one token at a time:
token 1 -> token 2 -> token 3 -> token 4 -> ...
Each step reads the entire KV cache built so far to compute attention for the new token, then adds one more entry to it. Very little new math happens per step relative to how much cached data has to be read from memory. This makes decode memory-bandwidth-bound: the bottleneck isn’t compute, it’s how fast the GPU can move data around.
That’s why a 100K-token prompt doesn’t just cost more memory. It costs more time per generated token, because every single step now has to read through a much bigger cache before it can produce the next word.
Doing the math on an actual model
The size of the KV cache isn’t a mystery, it’s a formula:
KV cache size = 2 x layers x KV_heads x head_dimension x bytes_per_value x tokens x batch_size
The 2 is because you store both a Key and a Value for every token, in every layer.
Let’s plug in Llama 3 70B: 80 layers, 8 KV heads (thanks to grouped-query attention, more on that below), head dimension 128, FP16 storage at 2 bytes per value, batch size 1.
I ran the numbers instead of eyeballing them:
Context length FP16 cache INT8 cache INT4 cache
-------------------------------------------------------
1,000 tok 0.31 GiB 0.15 GiB 0.08 GiB
8,000 tok 2.44 GiB 1.22 GiB 0.61 GiB
16,000 tok 4.88 GiB 2.44 GiB 1.22 GiB
32,000 tok 9.77 GiB 4.88 GiB 2.44 GiB
64,000 tok 19.53 GiB 9.77 GiB 4.88 GiB
128,000 tok 39.06 GiB 19.53 GiB 9.77 GiB
-------------------------------------------------------
That’s a single sequence at batch size 1. Serve four of those concurrently at 128K tokens in FP16 and you need roughly 156 GiB just for KV cache, before the model weights even enter the picture. This is why “the model fits on my GPU” and “I can serve this model with long contexts at any real concurrency” are two completely different claims.
Everything below is really just an attack on one term of that formula.
Shrinking the “KV heads” term: GQA and MQA
Standard multi-head attention gives every query head its own Key head and Value head. If a layer has 64 query heads, it also stores 64 separate KV heads, that’s a lot of cache per token.
Grouped-Query Attention (GQA) lets multiple query heads share one KV head:
standard MHA GQA
Q1 -> KV1 Q1 -,
Q2 -> KV2 Q2 -+-> KV1
Q3 -> KV3 Q3 -'
Q4 -> KV4 Q4 -,
Q5 -+-> KV2
Q6 -'
Llama 3 70B uses 8 KV heads instead of 64, an 8x reduction in that term of the formula, which is exactly why the numbers above are as small as they are. Multi-Query Attention (MQA) takes this to the extreme: every query head shares a single KV head. Maximum savings, but less representational flexibility, which can show up as a quality hit depending on the model and training setup.
This isn’t something you toggle at inference time, it’s baked into the architecture during training. If you’re picking a model to self-host, checking whether it uses GQA (most modern open-weight models do) tells you a lot about how expensive it’ll be to serve at long context.
Multi-Head Latent Attention (MLA), used in the DeepSeek model family, attacks a different term: instead of caching full-size Key/Value vectors, it caches a compressed latent representation and reconstructs what’s needed for attention on the fly. It can cut cache size dramatically, but taking advantage of it requires inference code built around that specific attention structure. You can’t bolt MLA onto a model that wasn’t trained with it.
Shrinking the “bytes” term: quantizing the cache
If you already have a trained model and can’t touch its architecture, this is the lever you actually get to pull. Keys and Values are typically stored in FP16 or BF16, 2 bytes per number. Store them at 8 bits instead and you halve the cache. Go to 4 bits and you quarter it.
The trade-off is precision. In my experience 8-bit KV cache quantization is close to a free lunch for most conversational and coding workloads. I couldn’t tell the difference in output quality. 4-bit is where I’d actually test carefully, especially for needle-in-a-haystack style retrieval over long documents, where small numerical errors in old cached tokens can matter more.
Here’s a copy-paste example using llama.cpp, which supports quantized KV cache natively and runs entirely on your own machine, no API, no cloud bill:
# Build llama.cpp (once)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON # drop -DGGML_CUDA=ON if you're CPU-only
cmake --build build --config Release -j
# Run a model with an 8-bit quantized KV cache instead of the FP16 default
./build/bin/llama-server \
-m ./models/llama-3-8b-instruct.Q4_K_M.gguf \
--ctx-size 32768 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--port 8080
Drop to q4_0 for --cache-type-k / --cache-type-v if you want to push further and are willing to test quality on your own workload.
If you’re running something closer to production serving, vLLM supports FP8 KV cache and is easy to spin up locally in Docker:
docker run --gpus all --rm -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model meta-llama/Meta-Llama-3-8B-Instruct \
--kv-cache-dtype fp8 \
--max-model-len 32768
Same idea in both cases: everything about the model stays the same, you’re just storing the cached numbers more compactly.
Shrinking the “tokens” term: eviction
The most aggressive lever is deciding you simply won’t keep every token’s cache around forever. A sliding window keeps only the most recent N tokens and discards the rest.
There’s a wrinkle here that surprised me the first time I read about it: some models exhibit “attention sinks,” where a handful of early tokens absorb a disproportionate amount of attention regardless of their actual content. Evict them naively and generation quality can degrade even though those tokens looked unimportant. That’s why streaming approaches like StreamingLLM keep a few sink tokens plus a recent window, and drop the middle:
[keep: sink tokens] -> [evicted: old middle] -> [keep: recent window]
The real risk with any eviction strategy is the one you can’t fully engineer around: you’re deciding what to forget before you know what the model will need later. Feed it a 200-page contract, mention a critical clause on page 40, evict that range because the model hasn’t referenced it in a while, then ask about that clause on page 190, it’s gone. For long-document work, legal or research use cases, I’d treat eviction as something to test thoroughly rather than adopt by default. For a chat assistant where only the last few turns matter, it’s close to free money.
The part that isn’t about shrinking the cache at all: PagedAttention and prefix caching
Everything above reduces the cache itself. This last piece is about using the memory you already have without wasting it.
Before PagedAttention (introduced by the vLLM project), serving engines often reserved one contiguous memory block per request, sized for a worst-case output length. If a request was allocated space for 2,000 tokens and stopped at 300, the rest sat reserved and unusable by anyone else: memory technically free, practically fragmented.
PagedAttention borrows the idea of OS virtual memory: it splits the KV cache into fixed-size blocks that don’t need to be physically contiguous, and tracks which blocks belong to which sequence:
Request A -> block 2 -> block 8 -> block 11
Request B -> block 1 -> block 5
Request C -> block 3 -> block 6 -> block 10
Same GPU, same model, but the effective usable memory goes up because nothing is over-reserved. This is one reason vLLM and similar engines can serve noticeably higher concurrency than a naive implementation on identical hardware.
The other big serving-side win is prefix caching. If many requests share an identical prefix (a system prompt, tool definitions, a repository’s worth of context that an agent resends on every call), the engine can compute the KV state for that prefix once and reuse it across requests, only doing fresh work on what’s actually new:
shared prefix -> computed once
request A = shared prefix + "fix this bug"
request B = shared prefix + "write tests"
request C = shared prefix + "explain this function"
This is what’s behind API providers pricing “cached” input tokens lower than fresh ones: you’re not paying full price to reprocess a prefix the server has already seen. If you’re building an agent that resends a large system prompt on every call, this alone can be the biggest cost lever available to you, and it requires no changes to the model at all.
One caution worth naming: reusing cached state across requests means a serving system has to isolate that state carefully so one user’s cached data never leaks into another user’s response. That’s a real engineering concern in multi-tenant deployments, not just a theoretical one.
What I’d actually check before optimizing anything
Technique Attacks Risk to output quality
--------------------------------------------------------------------
GQA / MQA KV heads Low (baked into training)
MLA cached repr. size Low, but needs matching kernels
INT8 KV quantization bytes per value Low for most workloads
INT4 KV quantization bytes per value Test before trusting
Token eviction tokens kept Workload-dependent, can be high
PagedAttention memory fragmentation None (pure allocation change)
Prefix caching repeated computation None (exact reuse)
--------------------------------------------------------------------
The ones at the bottom of that risk column, PagedAttention and prefix caching, are essentially free wins if your serving stack supports them, and they were the first thing I checked once I understood what was going on. Quantization to 8-bit was the next thing I turned on, and I didn’t notice a quality difference on my own workload. Eviction I still treat carefully, and only reach for it when I actually know old context won’t matter.
If you’re building on top of hosted APIs, you don’t get to choose GQA or MLA, that’s the model provider’s decision. What you do control is how much of your prompt repeats across calls (prefix caching territory) and how long your context genuinely needs to be. If you’re self-hosting, llama.cpp's --cache-type-k / --cache-type-v flags or vLLM's --kv-cache-dtype fp8 are the two lowest-effort changes worth trying before you reach for anything more invasive.
The lesson that actually changed how I think about long-context LLM apps: context length isn’t free just because it fits in the context window. Every extra token you send is memory you’re renting for the entire length of that generation, and there’s now a real toolbox (architectural, numerical, and purely operational) for deciding how much of that memory you actually need to pay for.
Tags: LLM, AI Engineering, GPU Optimization, Machine Learning, vLLM
