When you type «nvfp4 vs» into Google it completes with «fp8», «gguf», «q4_k_m» and «bf16». The usual answer compares different models quantized by different people. On October 16 NVIDIA released Nemotron Labs Dory 1.0 4B in BF16, NVFP4, MXFP8 and GGUF at once, so I measured the same model in every format on an RTX 4070 Ti SUPER (Ada, 16 GB, no FP4 hardware), with the rules written down before the first run.
| Format | Engine | Generate (tok/s, batch 1) | Read 2048-token prompt (tok/s) | Weights |
|---|---|---|---|---|
| BF16 | vLLM | 51.0 | 7,003 | 7.69 GiB |
| FP8 (per tensor, W8A8) | vLLM | 78.5 | 10,118 | 4.67 GiB |
| MXFP8 | vLLM | 81.4 | 6,061 | 4.76 GiB |
| NVFP4 | vLLM | 116.8 | 6,978 | 3.28 GiB |
| GGUF Q5_K_M | llama.cpp | 58.8 | 1,710 | 3.42 GiB (file) |
- Among the vLLM formats, generation follows weight size. NVFP4 was 2.3 times faster than BF16. On Ada it runs through a weight-only Marlin kernel: the weights stay small, the math stays in 16 bits.
- Reading a long prompt, FP8 wins, because it is the only one of these formats whose multiplications Ada does in 8 bits.
- Quality, full GSM8K (1319 questions, 8192-token cap): BF16 1249, FP8 1241, MXFP8 1250, NVFP4 1238. All three tie with BF16 under the ±3-point rule; their 95 % intervals stay within ±2 points.
- Token by token it is another story. On fixed English and Spanish texts NVFP4 picks BF16's next token 89-90 % of the time, against 94-96 % for FP8 and MXFP8, while perplexity barely moves. MXFP8 is the most faithful to BF16.
The post also covers what broke before measuring (the llama.cpp converter rejects the public checkpoint, a prefix cache that inflated prefill, truncated reasoning being scored as answers) and a mistake the audit caught in my own first fidelity run.
Read the full measurements, the pre-registered sheet and the scripts → https://efraingaray.com/en/blog/nvfp4-vs-fp8-gguf-ada/
