A fit calculator answers one question: will these weights load in your memory. A leaderboard answers a different one: is this model good at tests in general. The question that decides what you actually run is neither. It's "will Q4_K_M hurt my invoice extraction accuracy on my machine," and the only reliable way to answer it is to run your extraction on Q4_K_M and measure.

QuantProof automates that. Point it at a folder of real examples from your task. It sweeps every model you already have in Ollama (or whatever a Rapid-MLX server is serving), scores every output with deterministic scorers, measures time to first token, tokens per second, and peak memory, then recommends the smallest quant whose quality stays within 2% of the best measured result.

npm install -g quantproof
quantproof ingest my-tasks.md   # a local model drafts the task pack from your notes
# or: quantproof init my-task   # scaffold it yourself
quantproof run --pack my-tasks
quantproof report --markdown

Why measure instead of estimate

Three task packs ran on an M5 Max: ticket classification, invoice extraction, config generation. Three models, 540 scored generations. No model won all three.

An 8B at Q8_0 took classification at 0.950, beating a 30B at 45% of the memory. The 30B was the only model to score 1.000 on extraction. They tied at 1.000 on config generation, where the 8B needs about 17 GiB less.

The 8B's zero on extraction was a token-budget truncation issue, not a quality failure. On 57 of 60 generations it spent its entire 512-token budget on reasoning and never emitted content. Because every raw output is journaled, the report flags the zero trunc! and names the fix (raise max_tokens) instead of letting it read as "this model can't extract."

Which model is "best" changed with every task. That is the leaderboard problem in one paragraph: the ranking is a property of the model and the workload together, and a leaderboard only knows the model.

Determinism is a backend version property, not a model property

Every example runs 3 times at temperature 0 with a fixed seed, and repetitions are compared byte for byte. On Apple Metal, every candidate repeated exactly, across 240+ scored units. Then the same check ran on an RTX 5070. Same pack, same seed, same driver, two Ollama versions:

Both 8B quants became deterministic after the upgrade. The 1B flipped the other way.

Not even the direction of the effect survived a backend upgrade, which is why the check runs on every sweep instead of being trusted once. One example answered "As needed" on repetition 1 and "Weekly" on repetitions 2 and 3, at temperature 0 with a seed: identical conditions, different outputs.

Predictions print next to measurements

The fit prediction is deliberately simple: weights on disk, plus f16 KV cache for the context, plus a fixed 1 GiB compute allowance. On the 5070, three of four candidates measured within 15% of prediction; the 1B ran over 20% under its intentionally conservative estimate. Every report shows both. Use predictions for planning. Measured peak is your actual constraint.

Peak memory comes from the most honest source each machine has:

• nvidia-smi on NVIDIA GPUs
• Backend process RSS on Apple Silicon (validated against Ollama's own accounting)
• Metal accounting on Rapid-MLX
• /proc/meminfo-based RSS on everything else

A machine with no working source reports "not measured." Nothing is ever estimated.

Anyone can recheck the scores

quantproof report --bundle exports a zip with every raw output, all scores, model digests, sampler params, and the backend version. Re-scoring the 5070 sweep from bundle contents alone reproduced 264 of 264 scores exactly, on both Ollama versions. A number you can't recheck is marketing.

Run it on your task

Node 22+, Ollama or Rapid-MLX. Out-of-memory failures are recorded as results, not crashes, and the same packs can also sweep Claude models over the Anthropic API when no GPU is around.

Run your own 20 examples