Everyone who asks me "I'm going to run a model locally, vLLM or llama.cpp?" is really asking one thing: which one is faster. I think that's the wrong question. Both are fast; both were designed to solve a different problem, and what decides the choice isn't tokens per second — it's how you share GPU memory.
You see it most clearly here: vLLM reserves a large share of GPU memory for itself at startup, while llama.cpp moves as many model layers onto the card as it can fit. If you plan to run Stable Diffusion or a second model on the same card, that single sentence matters more to you than every benchmark table put together.
While preparing this piece I opened the official docs for both — and found that two things I "knew" are no longer true. I'll cover those as well, because in local inference, information that was correct six months ago will send you down the wrong path today.
The two engines aren't chasing the same goal
vLLM was born as a server. Its assumption: you have one or more GPUs, many requests arrive at once, and you want to maximize total throughput. Continuous batching, a paged KV cache, a multi-process scheduler — all of it comes from the "serve fifty users at the same time" problem.
llama.cpp was born from portability. Written in C/C++, requiring no external runtime, it starts on a laptop without CUDA, on Apple Silicon, and on an old AMD card through Vulkan. The repository today lists more than fifteen backends including CUDA, HIP, Metal, Vulkan, SYCL and WebGPU; on the CPU side there's AVX/AVX2/AVX512 and BLAS acceleration. I don't know another inference engine that covers such a wide hardware range.
So the comparison isn't "fast engine vs slow engine". It's "dedicated server vs embeddable library that fits anywhere". The moment you accept that distinction, half of the remaining decisions resolve themselves.
The real difference is in memory allocation
In vLLM, the space set aside for the KV cache is governed by --gpu-memory-utilization, and the documented default is 0.92. For years I had 0.9 in my head; the docs today say 0.92. It looks like a small difference, but the effect is large: at startup the engine claims 92% of the card's memory for its own pool, model weights come out of that pool, and what remains becomes KV cache.
Operationally this means vLLM rents the card at startup and doesn't hand it back unless you say so. If you want to run another job on the same GPU, you must lower that value by hand or the second process won't find memory. In exchange, because allocation happens up front, behaviour is predictable: memory pressure usually shows up not as a sudden crash but as requests being preempted and recomputed — that is, as latency.
If you're running vLLM on a shared card, do the arithmetic in advance. Subtract what the other workloads need from the card's total memory, set --gpu-memory-utilization deliberately according to what's left, and record it like any other deployment parameter. "Let's leave the default, it'll be fine" is the first thing to collapse the day a second job shows up. The reverse is true too: squeeze the value too far and the KV cache shrinks, long-context requests start queueing, and the engine hands the cost back to you as latency. There is no universally correct value for this setting — only a value that belongs to your workload.
llama.cpp's model is the opposite. On the server side, the number of layers moved to the GPU is set by -ngl, --gpu-layers, --n-gpu-layers, and the documented default is now auto — it used to be 0, meaning nothing went to the GPU at all. Part of the model can live in VRAM and the rest in system memory; when the card isn't big enough, the engine doesn't refuse to run, it just slows down.
That hybrid placement is what makes llama.cpp a homelab staple. On a 24 GB card you can run a model that doesn't fit, half on the card and half in RAM. Performance drops, of course — but it runs. In vLLM's world, the model either fits or it doesn't; there isn't much middle ground.
Concurrency: both have it, but not the same way
A common misconception is that continuous batching is unique to vLLM. It isn't. In the llama.cpp server, the -cb, --cont-batching flag is documented as "default: enabled", and -np, --parallel, which sets the number of server slots, defaults to -1, meaning automatic.
The difference is scale and scheduler maturity. vLLM's V1 architecture keeps the scheduler and the model executor together in one isolated execution loop (EngineCore) and moves the API server into its own process, so CPU-heavy work such as tokenization, multimodal input processing and detokenization overlaps with the core loop instead of blocking it. On top of that, V1 enables chunked prefill by default wherever possible, unlike V0 where it was switched on conditionally based on model characteristics.
In practice: at two or three concurrent requests either engine will do. At dozens of concurrent requests vLLM's architecture is the one designed for the job, while llama.cpp will remind you it's a single-machine embedded server. The question to ask your own setup is simple: how many people really query this model at the same time? If the answer is "just me", you're paying for scalability you never use.
The hardware reality: what happens on CPU
Here the picture is clear. vLLM has a CPU backend, but its boundaries are written plainly in the docs: on x86, AVX512F is recommended and AVX2 comes with limited features; on AMD, running vLLM on CPU requires at least a 4th-generation (Zen 4/Genoa) processor. Apple Silicon and IBM Z support are marked experimental with no prebuilt wheels, so you must build from source. Because float16 support is unstable on CPU, bfloat16 is the recommended dtype.
In other words, running vLLM on CPU is possible, but the documentation is politely telling you it isn't the main road. In llama.cpp the CPU is a first-class citizen; the project started there, and a Metal-backed setup on a MacBook is still one of its smoothest scenarios.
I should correct a misconception here, because until recently I also thought "vLLM means CUDA". vLLM's installation docs officially list AMD ROCm, Intel XPU and TPU alongside NVIDIA CUDA, and for Metal acceleration on Apple Silicon they point to vllm-metal, a community-maintained plugin using MLX as the compute backend. Paths beyond NVIDIA do exist.
None of them is as smooth as the CUDA route, though, and on laptop-class hardware they're hard-pressed to beat llama.cpp. That's why the top question in the decision tree isn't "which is better" — it's what you actually have.
Quantization: the GGUF world
llama.cpp's ecosystem revolves around GGUF, and the llama-quantize documentation lists everything from IQ1_S up to Q8_0. Note what it does not do: it doesn't tell you which one to pick. It gives measured tables — bits per weight, file size, prompt processing and generation throughput — plus one warning: quantization may introduce accuracy loss, usually measured in perplexity (ppl) and Kullback-Leibler divergence (kld), and that loss can be minimized with a suitable imatrix file.
So the answer to "which level is good enough" isn't in the table; you have to reach it by looking at your own outputs. My own practice is to start at Q4_K_M and move up if quality degrades, but that's my starting point rather than a recommendation.
There is one place where the docs do give explicit advice: for multimodal models, vision and audio encoders should be kept in a high-quality format such as bf16 or q8, because those components prepare the input for the model and their quality feeds straight into generation quality. Squeezing the model shouldn't mean squinting at the same time.
On the vLLM side the story differs: --quantization selects methods such as AWQ, GPTQ or FP8, and when unspecified the engine first checks the quantization_config attribute in the model config. You look for a ready-made quantized build in the model repository; you don't get GGUF's "produce your own file and carry a single artifact" convenience. If you want to build your quantization strategy around your card's VRAM, the arithmetic in my VRAM and quantization guide for local LLMs applies to both engines.
Client compatibility: both speak the same language
The good news: whichever you pick, your client code stays mostly the same. vLLM starts with vllm serve <model>, listens on port 8000 by default and exposes /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models and /health. The llama.cpp server defaults to port 8080 and serves the same OpenAI-compatible endpoints, plus an Anthropic-compatible /v1/messages endpoint listed in its docs.
That turns the engine choice into a reversible decision — and if there's one thing I love in architecture, it's reversibility. If your application talks to an OpenAI-compatible client, switching engines is barely more than changing the base URL. Set yourself up for that today: read the model name and base URL from configuration instead of hardcoding them.
So where does Ollama fit?
I can't skip this question, because most people running local models meet these engines through Ollama. Ollama behaves less like a third engine and more like the layer above: its own blog notes that version 0.30 shipped with GGUF model compatibility through llama.cpp, and it announced a separate fast path powered by MLX on Apple Silicon. So on the GGUF side llama.cpp runs underneath, and on Apple Silicon it's MLX.
The practical consequence: Ollama gives you model downloads, version tagging and an easy interface, and in return limits your access to the engine's finer controls. In a single-user setup that's an excellent trade. But the day you need to tune context window, slot count and GPU layer distribution with production concerns in mind, you'll want to drop below that abstraction and run the llama.cpp server or vLLM directly. Leaving Ollama behind one day simply means your requirements grew.
What you remember may be out of date
Now the part I promised at the start. The two things I had wrong: vLLM's memory fraction is 0.92, not 0.9, and llama.cpp's -ngl default is auto, not 0. Opening the docs added three more:
On the llama.cpp side, a single unified llama binary was introduced on 29 May 2026; you can now work through subcommands like llama serve and llama cli. The old binaries weren't removed — in the maintainer's words, llama serve and llama-server are the same thing. Your existing command line keeps working, but don't panic when reading the new docs and finding a command you don't recognize. Worth remembering too: the repository moved from ggerganov/llama.cpp to ggml-org/llama.cpp; old links still redirect, but that's the canonical address now.
In the same repo, some flags quietly retired. --draft and --draft-min are gone; the docs point you to --spec-draft-n-max / --spec-draft-n-min, or --spec-ngram-mod-n-max / --spec-ngram-mod-n-min on the ngram side. --spec-ngram-size-n and --spec-ngram-size-m were removed too, replaced by the --spec-ngram-*-size-n/m family and --spec-ngram-mod-n-match. If you have a script using speculative decoding, check the docs before upgrading — these flags aren't silently ignored, they error out.
On the vLLM side the break was more fundamental: the V1 engine has been the default since 0.8.0 and the documentation states V0 has been fully deprecated. V1 drops some V0 features such as best_of sampling, per-request logits processors and GPU↔CPU KV cache swapping. A two-year-old vLLM example you find online either won't run today, or will run on an entirely different architecture.
What's left exposed in production
Comparison articles usually stop here, yet this is where the real problems start. Three of them kept me busy:
Authentication doesn't cover everything. The vLLM docs carry a blunt warning about --api-key: it only authenticates endpoints under the /v1, /v2 and /inference path prefixes. The critical detail sits right there — other endpoints on the same server, including /invocations, which exposes the same inference capabilities as /v1, remain unauthenticated. Set the key and relax, and you may have left a second door to the model wide open.
The CORS defaults are wide open too: --allowed-origins, --allowed-methods and --allowed-headers all default to ['*']. Don't expose the engine directly to the internet; put a reverse proxy in front, terminate authorization there, and narrow those three values. I've yet to meet anyone running this on a shared network who had checked.
Memory tuning isn't set-and-forget. In vLLM, context length, max_num_batched_tokens and KV cache space are coupled. The docs note that with chunked prefill disabled, max_num_batched_tokens must exceed max_model_len; if you care about latency a smaller batch (2048, say) improves inter-token latency, while raw throughput favours going above 8192. You can't optimize both at once — deciding which you want is your job, not the engine's.
Don't mistake saturation for slowness. When vLLM's KV cache space runs out, the engine preempts requests and recomputes them, logging a "not enough KV cache space" warning; the total_cumulative_preemption_cnt counter sits among the Prometheus metrics. The docs suggest raising gpu_memory_utilization or lowering max_num_seqs / max_num_batched_tokens in that case. Without that counter on a dashboard you can't tell "the system got slower" from "the system saturated" — and on the llama.cpp side, requests quietly queue once the -np slots are full.
A model update is a deployment event. Swapping a GGUF file or pointing vLLM at a new model deserves the same seriousness as an application release: rollback plan, health check, a review of output quality. If you're planning a service on Kubernetes, my KServe vs vLLM comparison covers that layer separately.
Don't argue before you measure
Everything above is architecture and documentation. I can't tell you which one is faster on your hardware — nobody can, without measuring. Most tokens-per-second tables you find online were produced with a different model, a different quantization, a different context length and often a different version. Those numbers say almost nothing about your hardware.
When you set up your own measurement, fix three things: the same model family and parameter count, comparable quantization levels (GGUF Q4_K_M and FP8 are not the same thing — compare them knowing that), and the same prompt set with the same maximum generation length. Then collect two separate numbers: time to first token and inter-token latency for a single request, and total throughput at N concurrent requests. These two usually move in opposite directions, and a single average "speed" number hides both.
Don't forget warm-up either. The first request includes model loading, cache filling and kernel compilation; including that first call in your measurement is the easiest way to make an engine look slower than it is. Likewise, requests sent back-to-back with the same prefix benefit from prefix caching; if every prompt in your test set starts with the same long system message, what you're measuring may not be your real workload.
Finally, don't treat measurement as a one-off ceremony. Engine versions move fast and defaults change. Record the conditions under which you decided: which version, which model, which flags, which result. Six months later that note will be the only thing answering "why did we choose this?"
Decision checklist
Answer these five questions before you sit down to choose; the answers name the engine almost by themselves:
- Hardware: a datacenter NVIDIA/AMD GPU, or a laptop and Apple Silicon? If the latter, llama.cpp.
- Concurrency: how many simultaneous requests? Dozens and up means vLLM.
- Card sharing: is anything else using the GPU? If so, lower vLLM's up-front allocation deliberately.
- Model size: does the model fit in VRAM? If not, llama.cpp's hybrid placement is the only practical route.
- Operational load: who maintains it? llama.cpp means a single binary; vLLM means a Python service and its dependency tree.
My own preference is to use both: llama.cpp for development and single-user agent work, vLLM when a shared endpoint is required. Because both speak the same OpenAI-compatible interface, this isn't the cost of maintaining two systems — it's one interface with two implementations.
The real risk in local inference isn't picking the wrong engine. The real risk is going to production without knowing the engine's defaults: not knowing who took 92% of the card, which endpoint is sitting there unauthenticated, which flag retired last release. Benchmark tables go stale; the habit of opening the documentation doesn't.
