MLX vs GGUF on Apple Silicon: Benchmarking the Same Local Model Two Ways

You've pulled a local coding model onto your Mac and you're staring at two downloads of the same weights: a GGUF build and an MLX build. Which one do you actually run? The folklore says MLX is faster on Apple Silicon but you pay for it in quality. We got tired of guessing — so we built a harness that makes local coding agents produce real software, grades that software by running it, and then pointed the whole thing at the exact same model in both formats. This is what it found, and how it works.

Why you can't just ask the model how it did

Local coding models are very good at declaring victory. The tests pass, the summary says done, the diff looks plausible — and the actual feature is quietly broken. If you grade a model on its own report, you are measuring its confidence, not its competence. So the first design decision was the most important one: everything is graded by running it. Build an inventory store, add two items, ask for the total, and check the number against a golden value. If the program doesn't run, or prints the wrong answer, it doesn't count — no matter what the model claimed.

The harness: swarm-gym

The harness is called swarm-gym, and it ships inside goose local-edition — our open-source fork of the goose agent, hardened for local models and extended with a multi-node build swarm. One run of swarm-gym does five things in sequence:

• Plan — a planner model decomposes the spec into a small DAG of subtasks (core module, CLI entry, tests, an integrate-verify step) with explicit file ownership.
• Execute — worker agents build those subtasks in parallel across the fleet, each confined to its own files, seeing the signatures of its siblings.
• Judge — a semantic judge watches each worker's live activity and intervenes only on real trouble (a worker looping on a failing test, or shipping a stub).
• Integrate-verify — a final sink task wires the modules together and runs a smoke gate: does the package import, does its entry point run.
• Grade — deterministic verification runs the built app against golden values and records everything to a ledger.

The whole run is captured — a structured event log, every worker's tool calls, per-device dispatch counts — so a result is never just a number; it's traceable back to what each agent actually did.

The swarm

"Swarm" isn't marketing. The build is genuinely distributed: the planner drafts a skeleton, the fleet details every subtask's spec in parallel, and workers run concurrently across three Apple Silicon nodes, each serving the model through LM Studio. A dynamic replanner fills idle workers with independent work so the machines stay busy, and a per-task idle watchdog re-routes a genuinely stalled worker to another node. The point of the swarm is to get a real, multi-module app out of small, weak local models — the kind that fall over if you ask a single context to build everything at once.

Two modes: benchmark and exploratory

swarm-gym runs in two explicit, ledger-tagged modes, and it matters which one produced a number.

• Benchmark — a frozen suite of fixed specs replayed identically every time, so results from different models or runtimes pair up. Reproducible by construction. This article is entirely benchmark mode.
• Exploratory — an operator-driven agent invents new prompts, vibes follow-ups, and tunes swarm knobs while an active monitor watches reasoning, output quality, and node utilisation (catching idle nodes and re-balancing). It's for finding failure modes and improvements, not for producing comparable numbers.

The archetypes

Benchmark mode replays three fixed app specs — the archetypes. They're chosen to stress the three axes where weak local models fail differently: data modelling, algorithms, and stateful logic. Each ships golden-value checks.

The transaction archetype is the mean one: it needs correct nested-rollback semantics and a multi-command exec path, and it's where models most often ship passing unit tests over a broken feature. Hold that thought.

The tiers

The suite scales through tiers — smoke (one app), light (one of each archetype), medium (five of each), high (ten of each), and extreme (ten of each, harder specs). More repetitions means the run measures the model's consistency, not just one lucky build. For this study we ran medium: 5 runs × 3 archetypes × 2 model builds = 30 full app builds.

The experiment

A paired A/B. Same model, qwopus3.6-27b, in two builds: the original GGUF (from jackrong) and the MLX translation (from mlx-community). Identical frozen prompts, identical harness binary, identical flags, identical 3-node fleet. Only the loaded build changed between the two arms, and the two builds differed in more than format: the GGUF was jackrong's Coder build with the MTP head, the MLX one mlx-community's conversion of the Coder model, which they publish at 4, 6 and 8 bits and in bf16. Our run notes call them qwopus3.6-27b-coder-mtp and qwopus3.6-27b-coder, and we didn't record the quantisation of either file.

# what each variant ran — byte-identical except the loaded model
python -m harness bench --tier medium --variant gguf
python -m harness bench --tier medium --variant mlx

Results: speed

Median build time per run — lower is faster:

Overall it's a wash. But the consistency diverges sharply: MLX had lower run-to-run variance (stdev 580s vs 751s), a lower p90 (3050s vs 3601s), and — the headline — it never once hit our 60-minute per-run cap. GGUF hit that cap twice, both on the crud archetype. More on those caps below, because they turned out to be our bug, not the model's.

I wrote up how far the same model's score moves between two builds on a longer agentic task separately: Are AI benchmarks reliable? We had 32 models build the same app twice.

Results: quality

Does the built app actually work:

The raw number favours MLX. Run every app and the gap closes to a tie — GGUF's lower raw score is dragged down by a grading artifact.

On raw checks MLX looks ahead, 87% to 80%. But when you actually run every app, the gap closes to a tie (~13 of 15 each). GGUF's lower raw number is dragged down by a grading artifact — one transient false-partial on a crud run whose app is fully correct when you run it. Judged on whether the software works, the two builds are even.

The interesting part: what running the apps revealed

A benchmark that only prints medians is boring. The value is in what the harness caught by watching itself and by running the software. Three findings mattered.

1. The caps were a swarm bug, not the model

GGUF's two capped runs weren't the model grinding — they were the swarm's own integrate-verify step still running after the app was already built and correct. The step is a heavy critical-path task with no wall-clock budget, and the judge's repeated "ok" verdict was a no-op with nothing to force it to finish. So a healthy-but-slow finisher could run until the external cap killed it. That's a scheduler bug. We shipped an env-gated fix (a wall-clock cap on that step). In a partial re-run of eight GGUF builds nothing hit the cap, but the slow finisher never came back either, so the fix is still unproven. This is the entire point of the harness: it doesn't just rank models, it turns each run into a concrete fix.

2. Passing tests, broken feature — on both builds

The transaction archetype produced the same failure three separate times, across both model formats: the model wrote a KV store whose own unit tests passed, while a required feature was broken or missing — an exec path that printed nothing, or a COUNT command it simply forgot to implement. The generated tests didn't cover it, so the model shipped it green. Because this showed up on GGUF and MLX alike, it isn't a runtime difference — it's a weak-model completeness limit. And it's the clearest possible argument for grading by running a spec-derived command instead of trusting the model's own tests. The harness's golden checks caught every one of these; the model's tests caught none.

If you're picking a model rather than a format, I wrote up which models came out best on our agentic coding benchmarks separately: Best model for agentic coding in October 2026.

3. Raw pass-rate lies; judged-by-running doesn't

If we'd stopped at the raw checks column, we'd have called MLX the quality winner by 7 points. Running the apps erased that lead: one of GGUF's three "failures" was a correct app the harness mis-scored (a transient), and the other two were the same tests-green, feature-broken transaction failure MLX hit too. The lesson generalises beyond this benchmark — report what the software does, not what the grader's first pass said.

The verdict

Remarkably close, and the folklore was wrong. MLX is not a downgrade: it ties GGUF on whether the app works, and it's the steadier of the two — lower variance, and it never blew the cap. GGUF is faster specifically on compute-heavy work, at the cost of being spikier. If you want a safe default on Apple Silicon, MLX is it. If your workload is compute-bound and you can tolerate more variance, GGUF has an edge. Either is a reasonable choice — which, given how much simpler the MLX story is to reason about, is itself a win for MLX.

We run a local text-to-speech model on MLX too, as the voice of our YouTube narrator. I wrote up how we picked it separately: How we made Elzi, our AI YouTube narrator, on a local TTS model.

If it's images you want to generate locally rather than code, I wrote that up separately: the best local image generation model on a Mac.

MLX vs GGUF speed: prompt processing vs generation (time to first token)

One thing this page didn't split: a request has two phases. Two numbers, not one. First the model reads your prompt (prompt processing, or prefill), and that sets the time to first token. Then it generates, and that's the tokens per second number everyone screenshots. I'd always look at both. The runtimes report them separately. llama.cpp's llama-bench calls them pp and tg, and mlx-lm prints a Prompt line and a Generation line. LM Studio's v1 chat API returns time_to_first_token_seconds next to the generation speed, and the older v0 API calls it time_to_first_token.

Our July numbers are wall-clock time for each whole app build, so they include both phases plus tool calls and grading. The harness recorded no token counts, so I can't tell you which phase produced the differences above. That's a gap in our run, not a finding. It's on me. It matters because a swarm re-sends long contexts all the time, and prompt caching decides how much of that gets recomputed. LM Studio added disk-backed KV-cache checkpointing for hybrid models like Qwen 3.5 and 3.6 in mlx-engine v1.8.5, announced on 5 June 2026. Which engine version each node ran on 2 July wasn't recorded.

If you want the split on your own Mac, measure both phases with the same prompt length and the same output length. llama-bench defaults to a 512-token prompt and 128 generated tokens (-p and -n). mlx_lm.benchmark defaults to 512 and 1024 (-p and -g, and there -n is the number of trials). So default runs of the two aren't like-for-like on generation. Set both explicitly. Then raise -p on both for a long-prompt run. A short prompt hides the phase a long-context agent pays for. llama-bench's -d is a different test: it fills the context first and measures at that depth. Our llama.cpp vs MLX MTP comparison measured the generation side only.

MLX vs GGUF in Ollama on a Mac: MLX is the default since 0.40

Our July swarm served the model through LM Studio on all three nodes, and the MTP comparison ran llama-server and mlx_lm directly. We didn't use Ollama in either. If you're on Ollama, the engine under you changed this year. Ollama's preview post of 30 March 2026 put one model, Qwen3.5-35B-A3B, on MLX and showed decode going from 58 to 112 tokens/s and prefill from 1,154 to 1,810, Ollama 0.18 against 0.19. Read the line under the chart, though: the 0.19 run used NVFP4 and the 0.18 run Q4_K_M, so the chart changes the engine and the quantisation at the same time. It isn't a clean MLX vs GGUF number. The post also asked for a Mac with more than 32GB of unified memory.

Ollama 0.40.0, whose binaries went up on 6 October 2026, says "model architectures supported by the MLX runtime will automatically run on MLX" on Apple Silicon. In practice the tag you pull decides. qwen3.6:27b now carries two builds, an MLX one in NVFP4 and a GGUF one in Q4_K_M, and on a Mac you get the MLX one. So by default you get the same quant mismatch as Ollama's chart. There is a switch, it's just hidden: ollama pull and ollama run take --runner mlx, llamacpp or ggml, and it isn't in --help. If I were comparing the two formats in Ollama, I'd pull a pair at the same bit width, qwen3.6:27b-bf16 (GGUF) against qwen3.6:27b-mlx-bf16, about 56 GB each. Then check what you actually got: ollama show prints the quantization and the runner, and ollama ps has a RUNNER column.

MLX 4-bit vs GGUF Q4_K_M: same label, different bits per weight

Both files can say 4-bit and still not be the same quantisation. mlx-lm's mlx_lm.convert -q defaults to affine mode, 4 bits, groups of 64 weights, and each group stores a bf16 scale and a bf16 bias. I quantised a 4096 by 4096 bf16 matrix with mx.quantize on mlx 0.32.4 and counted the bytes: 4.50 bits per weight, the same in every layer. Q4_K_M is a recipe. llama.cpp stores most tensors as Q4_K, also 4.5 bits per weight, but it puts the output layer in Q6_K, and in about half the layers it moves attention V and the FFN down projection up to Q6_K as well. Its own quantize README puts a Q4_K_M Llama-3.1-8B at 4.89 bits per weight, against 4.67 for Q4_K_S. So on that model a plain MLX 4-bit vs Q4_K_M test gives the GGUF file roughly 0.4 more bits per weight, spent where llama.cpp expects them to matter. When someone reports that Q4_K_M held up better than MLX 4-bit, I'd look at that before I'd blame the engine.

MLX can do the same trick. --quant-predicate mixed_4_6 keeps 4 bits but gives v_proj and down_proj 6 bits in the same layers llama.cpp picks, and lm_head 6 bits like llama.cpp's output layer. Its docstring says it makes similar choices to Q4_K_M. It isn't an exact copy. On a hybrid model like Qwen3.6, llama.cpp also bumps the linear-attention QKV projection in those layers, and mixed_4_6 leaves it at 4 bits. The newer --q-mode values carry different overhead again. My byte counts on the same matrix: mxfp4 4.25 bits per weight, nvfp4 4.50, mxfp8 8.25, affine 8-bit 8.50. Q8_0 is 8.50 in llama.cpp's table, so MLX 8-bit against Q8_0 is the one quantised pair where the bit budget matches out of the box. At 4 bits, compare mixed_4_6 with Q4_K_M, or print the bits per weight next to each result.

You can read all this off the files. An MLX model's config.json has a quantization block such as {"group_size": 64, "bits": 4, "mode": "affine"}, with per-layer entries added when a recipe was used. llama.cpp prints a file type line reading Q4_K - Medium, and the file size with its BPW, when it loads a GGUF. Our July run kept neither line, so we don't know whether our pair was matched on bits. So the tie holds for these two files. We haven't run this benchmark with the two formats matched on bits. Next time both BPW numbers go in the results table.

Run it yourself

swarm-gym is open. One command runs a tier and prints a stats report plus CSVs you can paste straight into a write-up — the same numbers behind this article. Load your local models, then:

python -m harness bench --tier medium --variant mlx

The full benchmark, with the per-spec breakdown, lives on our Agentic Benchmarks page — where you can also publish your own run with our system. If you've got a model and a Mac, we'd genuinely like to see your numbers.

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.