MXFP8 vs Q8: 10x the weight error, 1% the perplexity

Key takeaways

• At the same 8.25 bits per weight, MXFP8 reconstructed real Qwen3-Coder-30B weights with 6.85% mean layer-output error against 0.67% for 8-bit affine — about 10x worse, on all nine tensor kinds.
• End to end that 10x collapses to +1.05% perplexity. Weight-reconstruction error is a real measurement and a bad predictor of model quality; I nearly published the first number on its own.
• The tiebreaker is speed, not accuracy: at identical footprint, affine decoded at 54.5 tok/s against MXFP8's 49.3 — about 10% faster for 1% better perplexity.
• The W8A8 path that justifies MXFP8 runs for exactly one shape on this hardware — M=1. Every other size raises 'NYI for the general case'.
• MXFP8 is a compute format for GPUs with native FP8 tensor cores. On Apple Silicon you pay its costs and collect none of its benefit — but the cost is far smaller than a weight-error table implies.

A page on this site has been ranking for mxfp8 vs q8 for months and it never once mentions Q8. That is my fault, so here is the comparison, run properly: the same weights, both formats, on one machine, with the numbers I actually got.

The short version is that MXFP8 lost, but the margin depends entirely on which number you look at, and I nearly published the wrong one. At identical storage cost it reconstructs real Qwen3-Coder weights about ten times less accurately than plain 8-bit affine. Run a whole model end to end, though, and that ten-times gap turns into one percent of extra perplexity. Both measurements are mine and both are correct; the distance between them is the most useful thing in this article, and it is a standing warning about proxy metrics.

The practical answer still comes out the same way — use affine — but for a much less dramatic reason than the weight table suggests, and it ends up being a speed argument rather than an accuracy one.

What the two things actually are

"Q8" means two different layouts depending on which world you came from, and the difference matters enough that I am going to pin both down before comparing anything.

In GGUF, Q8_0 is defined in llama.cpp's ggml-common.h:

#define QK8_0 32

typedef struct {
    ggml_half d;         // delta
    int8_t    qs[QK8_0]; // quants
} block_q8_0;

Thirty-two int8 weights, one fp16 scale, no zero point — symmetric. That is 34 bytes per 32 weights, or 8.5 bits per weight.

In MLX, the equivalent is mode="affine" at 8 bits, which is asymmetric: it stores a scale and a bias per group, so it can pin both ends of the group's range rather than assuming symmetry around zero. Do not take my word for what the library supports — the installed version will tell you, and it is the only source that cannot be out of date:

import mlx.core as mx
print(mx.quantize.__doc__)

On mlx 0.32.0 that prints:

MXFP8 is one of the OCP microscaling formats. Each element is an E4M3 float — one sign bit, four exponent bits, three mantissa bits — and each block of 32 shares one E8M0 scale, which is a raw power-of-two exponent with no mantissa at all. The element and scale types are pinned in MLX's own source rather than only in the docs — backend/cuda/quantized/qmm/qmm_utils.h maps mxfp8 to cutlass::float_e4m3_t elements with a cutlass::float_ue8m0_t scale, and the Metal path I am actually running selects the same scale type in backend/metal/kernels/fp_quantized.h (using ScaleType = metal::conditional_t<use_mx_scale, fp8_e8m0, fp8_e4m3>).

So the metadata budgets come out like this, and I measured them rather than deriving them, by quantising a 4096×4096 bfloat16 tensor and adding up the bytes of every array that comes back:

The first surprise is that MXFP8 is the smaller format. One byte of scale per 32 weights and no bias at all beats a 4-byte scale-and-bias pair per 64 weights. MXFP8 at 8.25 bits per weight is exactly the same size as affine at group 128, and smaller than GGUF's Q8_0.

That is the comparison worth running, so that is the one I ran: affine group 128 against mxfp8, both at 8.25 bits per weight. Anything else is comparing a format against a bigger one and calling the bigger one better.

The measurement

I pulled real weights. Not a synthetic tensor — a Gaussian random matrix has no outliers, and outlier structure is exactly the property that decides between a block-float format and an affine one, so measuring on one would have told me nothing.

The same comparison, measured two ways. The weight table says MXFP8 loses badly; a whole model running says it barely matters.

You do not need a whole model for this. Qwen/Qwen3-Coder-30B-A3B-Instruct is 61 GB across 16 shards, and one shard is 4 GB and contains the embeddings, the attention projections, the MoE expert tensors and the routers of the early layers:

hf download Qwen/Qwen3-Coder-30B-A3B-Instruct \
  model.safetensors.index.json --local-dir ./M
hf download Qwen/Qwen3-Coder-30B-A3B-Instruct \
  model-00001-of-00016.safetensors --local-dir ./M

For each tensor I quantised, dequantised, and measured two different errors, because they are not the same number and the second is the one that matters:

q  = mx.quantize(w, group_size=gs, bits=8, mode=mode)
dq = mx.dequantize(*q, group_size=gs, bits=8, mode=mode)

err_w = mx.linalg.norm(dq - w) / mx.linalg.norm(w)          # how far the weights moved
y, yq = mx.matmul(x, w.T), mx.matmul(x, dq.T)
err_y = mx.linalg.norm(yq - y) / mx.linalg.norm(y)          # how far the OUTPUT moved

Twenty-four tensors, mean relative L2 error on the layer output:

At 8.25 bits per weight in both cases, that is 10.2x more error for MXFP8. There is no tensor kind where it wins, or ties, or comes close. The gap is smallest on the embeddings and widest on the routers, which is the worst possible place to be wrong in a mixture-of-experts model — a router error does not degrade an answer, it sends the token to a different expert.

Hold that number loosely, though. It is a weight-reconstruction error, and I show further down that it overstates the damage by roughly an order of magnitude once a real model runs.

Why it loses

Three mantissa bits. That is the whole story.

E4M3 gives each element eight steps per binade, so between 1.0 and 2.0 the representable values are 1.0, 1.125, 1.25 and so on up to 1.875 — which is what a round-trip through the format confirms:

in : [1.0, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.875]
out: [1.0, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.75 ]

The shared scale cannot rescue this. E8M0 is a bare exponent, so it can only shift a block into a different binade — it can never subdivide one. Every weight in the block lands on roughly three significant bits no matter what the scale does. Affine int8, by contrast, spreads 256 evenly-spaced levels across whatever range the group actually occupies, and because it carries a bias as well as a scale it can pin both ends of that range.

You can watch that difference directly. I built blocks of 32 with a known maximum and checked what came back:

Affine reproduces the block maximum exactly, every time, by construction. MXFP8 frequently does not, and the pattern is hard to miss — whenever the maximum sits on a power of two it comes back 12.5% low. Across a whole 2048×2048 tensor the block maximum moves by 11.18% on average and up to 29.29% under mxfp8, against 0.000% under affine at any group size.

I want to be careful about what I am claiming there. I measured the behaviour; I did not manage to reverse-engineer MLX's scale-selection rule well enough to tell you whether it matches the OCP reference formula, and I am not going to guess at a bug I cannot demonstrate. The observable fact is that a power-of-two shared scale plus a three-bit mantissa does not pin the top of a block, and an affine scale-and-bias pair does.

The case for MXFP8, and why it did not show up

Block-float formats are supposed to earn their keep on outliers. A single huge weight in a block drags an affine scale so wide that every other weight in that block collapses into the same few levels, whereas a shared exponent rides it out. That is a real effect and I could reproduce it — here is a block of thirty-one 1.0s plus one outlier, showing what happens to the 1.0s:

At a 4096x outlier the affine block is destroyed and MXFP8 is untouched. So the mechanism is genuine. It just never fires on these weights: across the Qwen3-Coder tensors I measured, the median in-block dynamic range was 39–51 dB, or a ratio of roughly 90:1 to 350:1 between the largest and smallest non-zero weight in a block of 32. Int8's 256 levels swallow that comfortably. On my own numbers affine was still exact at a 256:1 in-block ratio and destroyed at 4096:1, so the breaking point sits somewhere in between — well clear of anything these tensors do. Trained transformer weights, at least these ones, are simply not that spiky at 32-element granularity.

There is a neat piece of corroboration for this sitting in public. The community MXFP8 build of Qwen3-Coder-Next, mlx-community/Qwen3-Coder-Next-mxfp8, has a base config of {"group_size": 32, "bits": 8, "mode": "mxfp8"} — and then 96 per-layer overrides, one for every mlp.gate and mlp.shared_expert_gate, each reading {"group_size": 64, "bits": 8} with no mode field. MXFP8 does not support group 64. So what mode do those layers actually load as? mlx-lm passes the override dict straight to to_quantized(), whose mode argument defaults to "affine", which I checked rather than assumed:

def pred(path, mod):
    return {"group_size": 64, "bits": 8} if path == "layers.0" else True

nn.quantize(m, group_size=32, bits=8, mode="mxfp8", class_predicate=pred)
# layers.0: mode='affine' group_size=64 bits=8
# layers.1: mode='mxfp8'  group_size=32 bits=8

The shipped MXFP8 model quietly keeps all 96 of its router layers on affine int8. That is the exact tensor kind where my measurements put MXFP8 at its worst.

It is not faster either

An accuracy loss buys something, normally. So I measured throughput on the same box — a Mac Studio, M3 Ultra, 60 GPU cores, 96 GB.

One trap first, because it cost me an hour. A single 4096² quantised matmul inside its own mx.eval() measures about 300 µs of Python and graph-building overhead and almost none of the kernel. Chain forty of them into one eval, mx.synchronize() before stopping the clock, and take the best of several runs. The same code went from 1559 µs to 41.5 µs. A bf16 matvec that reports 808 GB/s against this machine's 819 GB/s spec is a measurement you can trust; one that reports 108 GB/s is measuring your harness — I got that number the first time, because a model download was running in the background.

Decode shape, M=1, which is memory bound:

Prefill shape, M=512, which is compute bound:

Two things fall out. At decode, both formats beat dense bf16 because they move half the bytes, but affine converts that into more speed than MXFP8 does — it sustains 780 GB/s against MXFP8's 629, so MXFP8 gives up about a fifth of the bandwidth advantage its smaller footprint should have earned. And at prefill, every quantised mode is slower than dense bf16. Quantisation on this machine is a memory optimisation, not a compute one, and if your workload is prefill-heavy it is costing you.

Then there is the W8A8 row, which is the real point of MXFP8's existence. Quantising the activations as well as the weights is where a block-float format pays off, and MLX exposes it — nn.quantize takes a quantize_input=True flag, and it is supported for exactly two modes:

if mode not in ["nvfp4", "mxfp8"]:
    raise ValueError(
        f"Quantized activations are only supported for 'nvfp4' and 'mxfp8' modes, got {mode}."
    )

Affine cannot do this at all. This is the one genuine architectural advantage MXFP8 has. So I swept mx.qqmm across fourteen batch sizes from 1 to 256:

M=1     OK
M=2     [QQMatmul] NYI for the general case
M=4     [QQMatmul] NYI for the general case
M=8     [QQMatmul] NYI for the general case
...
M=256   [QQMatmul] NYI for the general case

One shape out of fourteen. And at that one shape it ran at 0.57x of dense bf16 — slower than doing no quantisation at all.

So why does this format exist

Because it was not designed for this machine. MXFP8 is a compute format, aimed at hardware with native FP8 tensor cores, where the win is that the multiply-accumulate itself runs in FP8 and you never materialise a wide intermediate. On an NVIDIA B200, PyTorch and TorchAO report end-to-end speedups of up to 1.26x for MXFP8 and 1.68x for NVFP4 — their numbers, not mine, on hardware I do not have.

Apple Silicon has no equivalent path today. Apple's own M5 write-up introduces GPU "Neural Accelerators" with dedicated matrix-multiplication units and reports up to 4x time-to-first-token against an M4. It does not say in so many words that M3 lacks them, so I will not claim that it does — I will only say what I measured on this M3 Ultra, which is that the W8A8 kernel is unimplemented for every shape but one, and that where the weight-only kernel does exist it is slower than the affine one.

That is the whole asymmetry. On Blackwell, MXFP8's accuracy cost buys a real throughput win. Here it buys nothing at all — which brings me to how large that cost actually is, because up to this point I have only shown you what happened to the weights.

None of this transfers to MXFP4 or NVFP4 by analogy. Those are 4-bit formats competing against 4-bit affine quantisation, where the trade-offs are completely different and block-float formats do much better. I measured 8-bit against 8-bit. Do not generalise it a bit-width sideways.

The number that nearly fooled me

Everything above is a proxy. Weight error and layer-output error are real measurements, but nobody runs a norm — they run a model. So I quantised an entire model three ways and measured what a user actually gets.

Qwen/Qwen3-4B, converted with mlx_lm.convert, evaluated on ~16k tokens of held-out TypeScript and TSX the model had never seen (this is a coding question, so the eval text is code, not wikitext), plus a decode-throughput run on each:

MXFP8 costs +1.05% perplexity against affine g64, and +1.03% against affine g128 at the identical footprint. Not ten times anything. A tenfold gap in weight-reconstruction error came out the far end as a one-percent difference in how well the model predicts code.

That is worth sitting with, because I had a finished draft that led with "ten times the error" and it would have been technically true and practically misleading. Transformers are enormously tolerant of weight perturbation — the error is spread quasi-randomly across a very wide network and most of it cancels rather than compounds. A relative-L2 table tells you what happened to the weights. It does not tell you what happens to the model, and the ratio between those two things is not fixed.

So the real case against MXFP8 here is not accuracy, it is that it loses on both axes at once. At an identical 4.15 GB and identical peak memory, affine g128 gives you slightly better perplexity and decodes about 10% faster. There is no trade to make. If MXFP8 had been meaningfully faster, a 1% perplexity cost would have been an easy price.

The two halves of this article use two different models, deliberately. Weight-level error is measured on real Qwen3-Coder-30B-A3B-Instruct tensors, because that is the model people actually ask about. The end-to-end run is Qwen3-4B, because quantising a model three ways needs all of it, and 61 GB was not going to finish over an unauthenticated Hugging Face connection. A 4B dense model is not a 30B MoE, and the MoE routers are exactly where MXFP8 measured worst — so if anything I would expect the end-to-end penalty on the coder to be a little larger, not smaller.

What to actually do

If you are running a coding model locally on Apple Silicon and you want 8-bit weights, use affine.

# -q is required — without it the other flags are ignored and nothing is quantised
mlx_lm.convert --hf-path Qwen/Qwen3-Coder-30B-A3B-Instruct \
  -q --q-mode affine --q-bits 8 --q-group-size 128 --mlx-path ./out

Group 128 is the one I would actually ship: it matches MXFP8's footprint exactly, edges it on perplexity, and gave me the fastest decode kernel of anything I measured. Group 64 buys a marginally better weight reconstruction for an extra 0.25 bits per weight and decoded slower, which on this evidence is not a trade worth making. In GGUF land Q8_0 is the same family of idea and lands at the same 8.5 bits per weight as affine g64.

Reach for MXFP8 when you are on hardware with native FP8 matmul and you are quantising activations too. That is the workload it was built for, and none of my criticism here touches it. On a Mac there is currently no configuration where it wins: same size, slightly worse perplexity, slower decode.

FP8 vs Q8: Qwen's -FP8 builds on Hugging Face are not MXFP8

If you searched for fp8 vs q8, you may be holding a different file from the one I measured. Qwen's builds whose names end in -FP8 are plain FP8, and they are popular: the Hugging Face API showed 8,961,550 downloads in total for Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 when I looked on 13 October 2026. Its config.json reads quant_method fp8, fmt e4m3, weight_block_size [128, 128] and activation_scheme dynamic, and the model card calls that fine-grained fp8 quantization with block size of 128.

So the element is the same E4M3 float that MXFP8 uses, and the blocks are different: 128 by 128 there, against 32 weights sharing one power-of-two scale in MXFP8. Qwen's build also quantises activations while the model runs. On this page the error tables, the perplexity run and the tok/s figures all have quantised weights only. The other plain FP8 you will meet is the llm-compressor recipe for vLLM, which lists static, per-channel quantization on the weights and dynamic, per-token quantization on the activations, for Nvidia GPUs with compute capability above 8.9 (Ada Lovelace, Hopper).

Qwen's card lists transformers, sglang and vllm as the frameworks for its -FP8 checkpoint, so I read Qwen's -FP8 builds as made for NVIDIA cards. I have not run one, and I have no number for plain FP8 against Q8_0 on a card that computes FP8 natively. On a Mac the choice is still the one above, MLX affine 8-bit or GGUF Q8_0.

What I did not test

I did not test MXFP8 on hardware that supports it natively, so all of this is a statement about Apple Silicon and not about the format in general. I did not test 4-bit formats at all. And the perplexity run is one model on one corpus of one language — it establishes the order of magnitude of the penalty, not a precise constant.

Every number here came off one Mac Studio this week, and the scripts are short enough that you can reproduce the lot in an afternoon. If you have run the same comparison on a Blackwell card, I would genuinely like to see where the crossover sits — at what point does the compute win pay back a 10x weight error?

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.