vLLM: The Open-Weight AI Inference Server Built for Production LLM Serving — Day 11/30

TL;DR — vLLM is the inference server most teams reach for when they need to serve open-weight LLMs at scale, thanks to PagedAttention and continuous batching. This episode breaks down what those two mechanisms actually do to GPU utilization, walks through the VRAM and cost-per-million math for real hardware, and covers who should (and shouldn't) be running it.

Every model you've read about in this series so far has a weights file. What it doesn't have is an opinion about how to turn that file into something 50 people can hit at once without falling over. That's a different problem, and for most teams serving open weights in production, the answer to it has a name: vLLM.

Two ideas, one GPU that does more

vLLM's reputation rests on two mechanisms that sound like implementation details but are actually the whole product.

PagedAttention treats the KV cache — the running memory of everything a model has generated so far in a conversation — the way an operating system treats RAM: in fixed-size pages that can be allocated, freed, and shared, instead of one giant contiguous block reserved per request. That sounds abstract until you see the effect on hardware. As kubenatives.com puts it, PagedAttention is "the reason a single A100 80GB can serve a 7B model to 50+ concurrent users instead of 5" — a roughly 10x improvement in how many concurrent conversations one GPU can hold in memory at once, without changing the GPU.

Continuous batching is the scheduling half of the same story. Instead of waiting for a fixed batch of requests to all finish before starting the next batch, vLLM reconsiders the batch after every single decode step: a request that just emitted its end-of-sequence token is evicted immediately, its memory pages freed, and a waiting request slides into that freed slot on the very next step. The idea traces back to Orca (OSDI 2022), and as frontiercheckpoint.com describes it, this drops queueing delay "from 'wait for the batch' to 'wait for one token's worth of compute.'" The GPU stays busy instead of idling between batch boundaries.

Put together, these two mechanisms are why vLLM's own docs list "state-of-the-art serving throughput" first among its features, alongside chunked prefill, prefix caching, speculative decoding (n-gram, EAGLE, and more), and quantization support spanning FP8, INT4, GPTQ, AWQ, and GGUF — per docs.vllm.ai.

The math you do before you provision anything

Capacity planning for vLLM starts with one formula: weights ≈ params × bytes-per-parameter, where fp16/bf16 is 2 bytes, fp8/int8 is 1 byte, and int4 is 0.5 bytes. Then you multiply that by roughly 1.3–1.5x to cover the KV cache and framework overhead at moderate concurrency — more like 1.5–2.0x for long-context, high-traffic production, per the walkthrough at dreaming.press.

Run the numbers on Llama 3.1 70B in fp16: 70B × 2 bytes = 140GB of weights alone, which doesn't fit on a single A100 80GB, full stop — kubenatives.com uses exactly this example to make the point that memory math, not GPU count, is the real constraint. Qwen3-32B tells a friendlier story: 32B × 2 bytes = ~64GB in fp16, which forces two H100s via --tensor-parallel-size 2, but the same model in fp8 (Qwen ships an official FP8 checkpoint, not a lossy afterthought) drops to ~32GB and fits comfortably on one H100 with headroom for a real KV cache.

Memorize three numbers and you can size almost any serve in your head: H100 = 80GB, H200 = 141GB, B200 = 192GB of HBM. Then check what you're trying to fit under that ceiling — the open MoE giants making news right now are not single-GPU stories. DeepSeek V4 Flash is a 284B-parameter MoE (~13B active) that lands around 150–170GB in mixed FP4/FP8, requiring two H200s or a B200. Kimi K3, at roughly 2.8T parameters and ~594GB in native MXFP4, is a data-center-rack decision, not a workstation one — both per dreaming.press.

Once it fits, the question becomes cost. Cost-per-million-tokens = (GPU $/hr ÷ 3600) ÷ throughput-tokens/sec × 1e6. Qwen3-32B on a single H100 at roughly $2.50/hr and ~2,350 tokens/sec aggregate lands near $0.30 per million output tokens — which is roughly break-even with DeepSeek V4 Flash's hosted API price of $0.28 per million. The catch: that $0.30 assumes the GPU runs near 100% utilization. Drop to 40% and the effective cost nearly triples while the hosted price stays flat. Self-hosting with vLLM wins on control, privacy, and customization — not automatically on price, unless your GPU is actually busy.

Who should actually be running this

vLLM earns its default status in a specific set of jobs, and it's worth being honest about which ones.

Teams serving a fine-tuned open model behind an API. If you've fine-tuned Qwen3-32B or a similar dense model (see Unsloth from Day 4 of this series) and need an OpenAI-compatible endpoint without building your own server, vLLM's vllm serve command is genuinely one line, with tensor-parallel sharding, quantization flags, and an API key as the only real decisions left.
Companies with data-residency or compliance constraints. When a hosted API is off the table for a specific workload, vLLM lets you keep the same throughput techniques (continuous batching, paged KV cache) that hosted providers use, on infrastructure you control.
High-concurrency agent and chat platforms. Prefix caching and chunked prefill matter most when many requests share a system prompt or tool schema — exactly the shape of agentic traffic.
Frontier-scale open MoE deployments. vLLM isn't only for 30B-class models. The vLLM blog documents serving GLM-5.2 — a 744B-parameter MoE with 40B active parameters, using DSA sparse attention and MTP speculative decoding — across 24 NVIDIA B300 GPUs, combining disaggregated prefill/decode with speculative decoding to hit production SLAs. That's the same server, three orders of magnitude up the hardware ladder from a single-GPU Qwen deployment.

Who shouldn't reach for it: anyone serving one user at a time on a laptop or edge device. vLLM's whole value proposition — batching many concurrent requests to amortize GPU cost — evaporates at concurrency of one. That's llama.cpp territory, not vLLM's.

Where vLLM isn't the only answer

vLLM is the right default for high-throughput, datacenter-GPU serving where you want broad model coverage and an OpenAI-compatible endpoint without a build step. But per frontiercheckpoint.com, the field has real alternatives: Hugging Face's TGI has narrowed the gap by adopting continuous batching and paged attention itself; TensorRT-LLM can be faster on NVIDIA hardware via compiled engines, at the cost of a build step and flexibility; and SGLang, with RadixAttention's prefix-sharing radix tree, is often the pick for agent workloads dense with shared prefixes and structured generation. That last one is worth sitting with, because the tradeoffs aren't always obvious until you see them side by side.

Credits & sources

Thanks to the maintainers of the vLLM project and the vLLM engineering blog for documenting production deployments in detail. The VRAM and cost-per-million math in this episode draws on dreaming.press's walkthrough of serving open-weights LLMs. The PagedAttention concurrency framing comes from kubenatives.com's piece on running vLLM on Kubernetes. The continuous-batching and scheduler explanation is grounded in frontiercheckpoint.com's breakdown of the vLLM serving stack.

Tomorrow's episode stays in the serving layer and looks at SGLang, the server built around RadixAttention's prefix-sharing radix tree — worth understanding once you've seen where vLLM's batching stops and an agent's shared-prefix traffic begins.