Colibri Review: Run a 744B Model on Hardware You Own

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Colibrì is a pure-C inference engine that runs frontier Mixture-of-Experts models — 744B to 2.8T parameters — on hardware you already own, by treating VRAM, RAM, and NVMe as one memory hierarchy instead of one memory requirement. It went from first commit on 2026-07-01 to 26,662 stars in two months.

• 26,662 stars, 2,920 forks, Apache-2.0, written in C with zero engine dependencies
• Eight model families run today: GLM-5.2 (744B), Inkling (975B), Kimi K3 (2.8T), GLM-5.3-Flash (321B, vision), DeepSeek V4 Flash (284B), Qwen3.8-Flash-Next (125B), Qwen3.6 (35B-A3B), OLMoE (7B)
• No GPU required for any of them — a GPU only ever makes it faster
• The trick: a 744B MoE activates only ~40B params per token, and only ~11 GB changes between tokens
• Real measured range: 0.05 tok/s on a 25 GB dev box, ~1.8 tok/s on a 128 GB CPU desktop, 5.8–6.8 tok/s on 6× RTX 5090
• The catch: you need 372 GB of disk for GLM-5.2 (1.6 TB for Kimi K3), and decode speed is set by your SSD

This is not a way to get ChatGPT speeds for free. It is a way to hold a frontier model instead of renting one — and to watch every expert fire while it answers you.

Quick Reference

The Core Idea: Placement, Not Residency

Every "you need 8× H100" claim assumes the whole model must sit in fast memory at once. For a Mixture-of-Experts model, that assumption is wrong in an exploitable way: GLM-5.2 has 744B total parameters but activates roughly 40B per token — about 5.4%. Colibrì splits the model along that line:

• The dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4, about 9.9 GB.
• The 19,456 routed experts (75 MoE layers × 256, plus the MTP head, ~19 MB each at int4) live on disk — ~370 GB — and get streamed on demand.

The README's framing is the clearest one I've seen: think of it as a JIT, but for weights. A compiler JIT never compiles the whole program; it watches what actually runs and compiles the hot paths just in time. Colibrì makes the same bet about a 744B parameter space — parameters are not resident state to be held, they are data to be staged across VRAM/RAM/NVMe exactly when the router proves they're needed.

The load-bearing commitment: placement only ever decides speed. Insufficient fast memory makes it slower; it must never quietly redefine the model.

What Makes It Fast Enough to Bother

A naive "read the expert when you need it" loop would be unusable. Most of the engine's cleverness goes into avoiding and overlapping disk misses:

• Batch-union reads. Each expert's three matrices are stored adjacent and read in one pread; batched positions read each unique expert once.
• Router lookahead (PILOT=1). A thread runs the router one layer ahead and prefetches. Routing is measurably 71.6% predictable one layer ahead — that predictability is the whole reason this works.
• A learning cache. The engine records which experts your workload routes to in .coli_usage, updated every turn, and pins the hottest automatically. It literally gets faster the more you use it.
• O(1) expert lookup (v1.8.0). A resident hit is one probe where the legacy worst case was 44 to 219.
• O_DIRECT (DIRECT=1). Bypasses the page cache: +34% decode measured on a Blackwell/Windows box, 4.25 → 9.69 GB/s in iobench on a GB10. Drive-dependent — QLC and DRAM-less disks can be neutral or negative.
• Dual-SSD striping. Put a second full copy of the model on a second drive and stream from both. A 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone.

That last one has a nice failure model. The mirror is validated at startup (byte-identical safetensors headers), never written to, and a read error falls back to the primary with one warning — so unplugging the second drive mid-run degrades instead of killing the server. A partial mirror is fine too: coli mirror plan / stage / verify ranks which shards to copy from the expert history the engine already learned.

MLA attention also stores 576 floats/token instead of 32,768 (57× smaller) and persists it to .coli_kv, so conversations reopen warm with zero re-prefill, byte-identical to an uninterrupted session.

Getting It Running

You need two things: the program (a few hundred KB) and the model (372 GB).

Get the engine — prebuilt archives exist for Linux, macOS, and Windows:

mkdir colibri && tar xzf colibri-v1.10.1-linux-x86_64.tar.gz -C colibri && cd colibri
python3 coli info                          # engine ready ✓

Or build from source, which needs gcc (or clang) with OpenMP:

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh                                 # checks gcc/OpenMP, builds, self-tests

Get the model. Use the group-scaled (gs64) container with the int8 MTP head:

https://huggingface.co/mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp

Or convert from the FP8 source with ./coli convert --model /nvme/glm52_i4 — one resumable command that never needs the full 756 GB on disk at once.

Run it:

COLI_MODEL=/nvme/glm52_i4 ./coli chat      # TUI; RAM budget, cache, MTP auto-detected
COLI_MODEL=/nvme/glm52_i4 ./coli plan      # inspect planned VRAM/RAM/disk placement
COLI_MODEL=/nvme/glm52_i4 ./coli doctor    # read-only readiness check
COLI_MODEL=/nvme/glm52_i4 ./coli tune      # measure this machine's fastest safe profile
./coli serve --model /nvme/glm52_i4        # OpenAI-compatible API + dashboard, headless

Run coli plan, coli doctor, and coli tune before your first real session. They're the difference between "why is this 0.05 tok/s" and knowing exactly which tier your experts landed in.

Switching models changes nothing about the command line — build the engine you want (make -C c inkling, make -C c kimi_k3), then point COLI_MODEL at the right directory. coli reads the model's config.json, picks the matching binary, and renders that family's chat template.

What Each Model Actually Costs You

This table is the single most useful thing in the repo, and misreading two rows together has confused people into thinking the requirements contradict each other. They don't — they're different models.

Note the shape of it: RAM barely moves from 8 GB to 32 GB across a 400× range in parameter count, while disk goes from 7 GB to 1.6 TB. That's the whole thesis in one table. The binding constraint moved from expensive scarce memory to cheap abundant storage.

The Benchmarks, Honestly

The project publishes a hardware ladder rather than a headline number, which I appreciate:

Quality is measured rather than assumed. The forward pass is validated against a transformers oracle with teacher-forcing typically 30–32/32, and the int4 container's quantization cost has its own ablations. The Qwen3.6 CUDA VRAM expert tier measured 1.44 → 10.05 tok/s (7.0×) on two 8 GB cards with output bit-identical to the CPU path — verified with cmp over a full 200-token generation. That's the right way to report a speedup.

Speculative decoding gets the same treatment. GLM-5.2's native MTP head drafts 2.2–2.8 tokens/forward when it pays, with two hard-won defaults: the MTP head must be int8 (int4 heads collapse to 0–4% acceptance), and draft and verify must compute the same function (SPEC_PIN=1). And the honest counterpoint is published too — MTP measured a 32% loss around 85% expert hit rate. Use DRAFT=0 when it doesn't pay.

Community Reaction

The Show HN — "Getting GLM 5.2 running on my slow computer" landed on 2026-07-10 and the reception split along a predictable line.

The enthusiasm is about access. r/LocalLLaMA and r/LocalLLM both ran threads within days, and commenters immediately started asking about RAID0 and multi-SSD setups. One: "it would be interesting if with a really fast drive, potentially raid0... you could get to 1 to 2 tokens a second — then for really critical work that you have extra time for and want right, I could see it being quite valuable on consumer hardware."

The skepticism is about arithmetic. From r/LocalLLM: "I don't know about people but for my SSD it will take 1s, so best case I'll do is 1 tok/s, given everything else is perfect. The only scenario where this works at any reasonable speed is for you to have enough RAM to fit the model anyway." That is a fair reading of the cold-cache case — and the learning cache plus lookahead prefetch is precisely the counter-argument, though it only pays on repeatable workloads.

The sharpest criticism wasn't technical. An r/LocalLLaMA commenter called out the AI-written README and noted 24 of the first 32 commits were authored by Claude. The maintainer's reply: colibrì is a one-person project written and tested entirely on a 12-core laptop with 25 GB of RAM, and the published numbers are the ceiling of what they could measure at home. Two months and 26k stars later, that ceiling has been raised mostly by contributors A/B-testing on hardware the author doesn't have.

An r/technology thread also flagged generations getting stuck in loops. Worth knowing: that was traced to the older per-row int4 containers, which measure ~9pp worse on quality. The gs64 container fixed it in controlled A/Bs — but the maintainers explicitly say it is not a general repetition guard. Use the gs64 build.

Honest Limitations

• Speed is your SSD. On a slow drive, expect a fraction of a token per second. No amount of engine cleverness fixes a cold cache on a QLC disk.
• Storage is the new price. 372 GB for GLM-5.2, 1.6 TB for Kimi K3. Cheaper than 8 GPUs, not free.
• No SLA on speed — the project says so explicitly. It's an engine you can run today and a research platform. Aggressive systems ideas ship behind flags.
• 97 open issues and a fast-moving release cadence (five releases in twelve days). v1.10.0 shipped broken prebuilt archives that v1.10.1 repaired the next day.
• Not every knob is a win. O_DIRECT is drive-dependent, learned pins can overfit a prompt, and CPU/GPU overlap gains vanish on fast CPUs at low residency. The docs say this out loud, which is unusual and good.
• TTFT is its own problem. 13 s to first token on a 6× 5090 rig is acceptable; on a cold streaming box it's a different conversation.

Who Should Actually Use This

Good fit: you have a big NVMe drive and normal RAM; you want a frontier model for batch work where 1–2 tok/s is fine (overnight analysis, offline drafting, air-gapped research); you want to study MoE routing — the Brain and Atlas pages visualize all 19,456 experts, 13,260 of them characterised, with 1,041 replicated specialists clustering by topic; or you want a C engine small enough to actually read.

Bad fit: interactive chat as a daily driver, anything user-facing with latency requirements, or a machine without hundreds of gigabytes to spare. A 30B dense model on the GPU you already have beats this every time.

FAQ

Does Colibrì really run a 744B model on 25 GB of RAM?
Yes, with the qualifier that matters: the 744B model's weights still occupy ~372 GB on disk. Only the dense portion (~9.9 GB at int4) plus an expert cache stays in RAM; the 19,456 routed experts stream from NVMe as the router calls them. On a 25 GB box that's 0.05–0.1 tok/s cold — correct, but slow.

Do I need a GPU?
No. None of the eight supported families requires one. A GPU only makes it faster — the DeepSeek V4 CUDA path makes prefill 5–10× and decode ~2.5× faster, and the Qwen3.6 VRAM expert tier measured 7.0× on two 8 GB cards. Backends exist for CUDA, Metal (experimental), and Vulkan, the last of which covers AMD cards via Mesa/RADV including ones ROCm dropped, like the RX 580.

How is this different from llama.cpp with mmap offloading?
mmap lets the OS page weights in reactively and leaves placement to the page cache. Colibrì treats placement as the engine's job: measured routing heat drives a per-layer LRU and a learned pinned hot-store, a lookahead thread prefetches next-layer experts using 71.6%-predictable routing, batch-union collapses duplicate reads, and O_DIRECT plus dual-SSD striping attack the I/O path directly. It's proactive scheduling versus reactive paging.

Will streaming from disk degrade output quality?
No — that's an explicit guarantee rather than a hope: placement decides speed, never semantics. The forward pass is validated against a transformers oracle (30–32/32 teacher-forcing), and the Qwen3.6 GPU tier produces bit-identical output to the CPU path. Quantization does cost quality, but that's the int4 container's doing, not the streaming. Use the gs64 container, not the older per-row mirrors.

Can I split it across multiple machines?
Yes, there's a local cluster mode. The coordinator keeps token generation, routing, and KV state local while disk-backed workers execute routed FFNs elsewhere; a layer's batched expert union goes as one persistent TCP request, so a token doesn't cost one round trip per expert. The transport stays disabled unless workers are configured, so the single-machine path is unchanged.

Is it production-ready?
No, and it doesn't claim to be. Treat it as a serious research engine with a usable front end — an OpenAI-compatible API, a web dashboard, persistent KV slots — but with 97 open issues, no speed SLA, and a release last week that shipped a broken archive. Pin a version, run coli doctor --deep, and measure your own hardware before depending on it.

Sources

• JustVugg/colibri on GitHub — README, benchmark tables, per-model docs
• Colibrì release notes — v1.7.0 through v1.10.1, 2026-08-20 to 2026-08-31
• Show HN: Getting GLM 5.2 running on my slow computer — 2026-07-10 launch thread
• r/LocalLLaMA: GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine
• r/LocalLLM: Is Colibri the right approach? — the skeptical arithmetic
• GLM-5.2 int4-gs64 container on Hugging Face