Choosing an On-Device Inference Engine in 2026: It's Not Just llama.cpp Anymore

Running models locally used to mean one thing: llama.cpp, some GGUF quantization, done. Fine for text generation. But the moment you run a vision language model on device, or need fast prefill for long contexts, the engine choice matters way more than the model itself.

We building Cider, an INT8 activation quantization SDK on top of MLX. The problem we kept hitting: MLX does weight quantization well, W4A16 and W8A16, but leaves activations in full precision. For VLMs processing screenshots at 4000+ tokens of visual context, that prefill cost adds up fast.

What actually changed in 2026

Three things shifted the landscape this year.

First, Apple silicon got real INT8 support. The M5 generation added hardware paths that make W8A8 and W4A8 viable without software emulation. Before M5, you could do INT8 weights but activations still ran in fp16 on the matrix units.

Second, vision models went mainstream on device. GUI agents, document parsers, visual QA. These models have massive prefill requirements because image tokens dominate the context. A single screenshot can be 2000+ tokens. The old decode speed benchmarks everyone focused on just dont tell the full story anymore.

Third, quantization got more nuanced. Its not just about making the model smaller. W8A8 per channel quantization on M5 Pro gives you roughly 1.8x prefill speedup over W8A16 baselines. Thats not a rounding error, its the difference between a usable agent and one that feels broken.

Some real numbers

On M5 Pro with 64GB RAM and 307 GB/s memory bandwidth, running a 4B VLM with 4516 tokens of context:

W8A16 baseline: 2.839s prefill, 80.1 tok/s decode.
W8A8 with Cider: 2.519s prefill, 79.5 tok/s decode.

Decode speed stays about the same because thats memory bandwidth bound either way. But prefill drops by roughly 12.7%, and on heavier contexts the gap widens further.

Picking an engine in practice

If you are doing text only inference on Apple silicon, MLX with standard weight quantization is totally fine. llama.cpp with Metal backend works great too. No reason to overcomplicate it.

Once you add vision, or need to handle contexts above 4K tokens regularly, you want to look at engines that support activation quantization. Cider is one option, it works with any MLX model and you can check it out at https://github.com/Mininglamp-AI/Mano-P under the Cider repo. There are others emerging too, and the right choice depends on your target hardware generation.

The bigger point: the engine layer is no longer a solved problem. Model selection gets all the attention, but the inference stack underneath determines whether your 4B model actually feels responsive or just technically runs.