Originally published on mrsaynothing.dev
Originally published at mrsaynothing.dev.
Every local-model card now brags a 128k context window. On a desktop, that number is a lease agreement your RAM signs without reading. The thing that decides how far you can actually push a long document is not the weights — it is the KV cache, and it scales with context length like a tax you agreed to in someone else's currency.
The context window is not a feature you toggle. It is a memory lease you pay every second the server runs.
How much RAM does a 128k context actually use?
The arithmetic is public and short. As laid out in Transformer Inference Arithmetic, the KV cache holds two tensors — keys and values — for every layer, every KV head, every token:
cache_bytes = 2 × layers × context × kv_heads × head_dim × bytes_per_element
Take Llama 3.1 8B: 32 layers, 8 KV heads, head_dim 128, straight from the config.json on Hugging Face. At 131,072 tokens in fp16 (2 bytes):
2 × 32 × 131072 × 8 × 128 × 2 = 17,179,869,184 bytes ≈ 16 GiB
The weights of that same model in fp16 are about 16 GiB. At maximum context, the cache is as big as the model it belongs to. The "8B model that fits in 16 GiB" quietly becomes a 32 GiB problem the moment you set num_ctx to 128k.
Why does the cache grow with the window?
Because attention has to compare every token against every token that came before it, and inference is autoregressive — nothing gets recomputed, everything gets remembered. That is the entire trick that makes generation fast: each token's keys and values are computed once and stored. Storage per token is constant, so total storage is linear in context. 128k is not "a bigger number" — it is 128× the cache of a 1k window, allocated for the life of the session.
Grouped-query attention (GQA) is the model-side discount: fewer KV heads means a thinner cache. Qwen2.5 7B uses 28 layers with only 4 KV heads (config here), so the same 128k window costs about 7 GiB in fp16 — real relief, and one reason those models feel friendlier on modest machines.
| Model (fp16) | Layers × KV heads | KV cache @ 128k | Weights | Cache vs weights |
|---|---|---|---|---|
| Llama 3.1 8B | 32 × 8 | ~16 GiB | ~16 GiB | ~100% |
| Qwen2.5 7B | 28 × 4 | ~7 GiB | ~15 GiB | ~47% |
Run the numbers yourself — this is the whole check:
def kv_cache_gib(layers, kv_heads, head_dim, ctx, bytes_per=2):
return 2 * layers * ctx * kv_heads * head_dim * bytes_per / 2**30
print(kv_cache_gib(32, 8, 128, 131_072)) # Llama 3.1 8B -> 16.0
print(kv_cache_gib(28, 4, 128, 131_072)) # Qwen2.5 7B -> 7.0
What the marketing number leaves out
Here is the honest ledger of this argument — including where it bends:
- Prefill is the second bill. Before the first answer token, the whole prompt is processed at once. A 100k-token prompt means chewing 100k tokens before you see a character of output. On a 3060, that is minutes, not milliseconds.
- Sliding-window models break the math — in your favor. Mistral-style architectures cap attention at a fixed window per layer, so the cache stops growing. The simple formula overestimates them. This post's claim applies to full-attention transformers, which is most of what people run.
- KV quantization is real but partial. llama.cpp can cache KV in q8_0, roughly halving the cache at some quality risk. Half of 16 GiB is still 8 GiB.
- Specs rarely say KV heads. You will dig through config.json to find them, which tells you something about how the number is meant to be read.
None of this makes long context useless. RAG exists precisely because filling a window is the expensive way to say "read section 4". But a model card that prints "128k" without printing the memory bill is selling a car by its top speed and never mentioning the fuel tank.
Nobody reads 128,000 tokens. Your RAM pays for every one of them anyway.
The fix is boring: set context to what your workload actually uses. A 32k window costs a quarter of the 128k cache and covers nearly every prompt a single developer actually sends. This is the same ledger discipline as the RAM arithmetic for local LLMs — model size is only half the budget, and quantization levels shrink the weights but leave the cache math untouched.
If the cache is the real price of context, why do model cards sell the window and never the bill? And confessional round: what is the largest context you have actually filled to the last token — and did the output justify the gigabytes? The comments are open; the next piece in this series will take whichever side loses.
This ships daily at mrsaynothing.dev — the full archive, every piece in 21 languages, zero missed days. New posts land in the newsletter the moment they ship: join it here. Code at GitHub.
