Colibri Review: Run a 744B Model on 25GB of RAM

Originally published on andrew.ooo — visit the original for any updates, code snippets that aged out, or follow-up posts.

TL;DR

Colibri is a lightweight, pure-C inference engine that runs GLM-5.2 — a 744-billion-parameter mixture-of-experts model — on a consumer machine with as little as 25 GB of RAM, by treating VRAM, RAM, and disk as one memory hierarchy and streaming routed experts off an SSD exactly when the router asks for them. It's a single C file, zero runtime dependencies, Apache-2.0, and it hit the Hacker News front page with ~922 points and 238 comments as a Show HN in July 2026.

The pitch is deliberately provocative: a frontier-scale model isn't something you rent behind an API — it's something you can open up, run on hardware you already own, and watch every expert fire in real time.

Key facts:

• ~14.7K GitHub stars, one of July 2026's fastest-climbing repos, from developer vforno / JustVugg
• Pure C engine (c/glm.c + small headers) — no BLAS, no Python at runtime, no GPU required
• Runs a 744B MoE by keeping the dense part (~9.9 GB int4) resident and streaming 19,456 routed experts (~370 GB) from disk
• Token-exact against a transformers oracle — placement changes speed, never precision or router semantics
• Web dashboard that visualizes all 19,456 experts as a living cortex, lighting up as they route

If you've ever wanted to hold a frontier model in your hands instead of poking it through a metered endpoint, this is the most interesting thing to happen to local inference this year.

What problem does Colibri actually solve?

The conventional wisdom about large language models is that the parameters have to fit. If a model is 744B parameters, you need enough fast memory (VRAM, or at least RAM) to hold the weights, or you can't run it. That's why frontier open models like GLM-5.2 have effectively lived in datacenters and on multi-GPU rigs.

Colibri's core insight is that a mixture-of-experts model doesn't need to fit — it needs to be placed. A 744B MoE activates only ~40B parameters per token, and of those, only ~11 GB actually change from token to token (the routed experts). So the trick is:

• The dense part — attention, shared experts, embeddings, ~17B params — stays resident in RAM at int4 (~9.9 GB).
• The 19,456 routed experts (75 MoE layers × 256, plus the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.

The mental model the author reaches for is a JIT compiler, but for weights. A JIT never compiles your whole program up front — it watches what actually runs and compiles the hot paths just in time. Colibri makes the same bet about a 744B parameter space: parameters aren't resident state to be held, they're data to be staged across a heterogeneous storage hierarchy (VRAM / RAM / NVMe), exactly when the router proves they're needed. The router runs a layer ahead so prefetch can hide the staging latency, and — like a JIT — the engine learns your workload: the more you run, the hotter the right experts get.

It works because routing has measurable structure. Colibri's "expert atlas" shows 13,260 characterized experts clustering by topic — poetry, law, Chinese, SQL — and structure is cacheable.

Installing and running it

Colibri needs two things: the program (a few hundred KB) and the model (~372 GB). There are prebuilt releases for Linux, macOS, and Windows — no compiler needed:

# Grab a prebuilt release and unpack it
mkdir colibri && tar xzf colibri-v1.1.0-linux-x86_64.tar.gz -C colibri && cd colibri

# Sanity check — engine ready?
python3 coli info

The coli launcher and its Python helpers are just glue — the engine itself is pure C with zero dependencies. Python 3 is only used by the launcher and the optional API gateway, never at inference time.

Or build from source (needs gcc/clang with OpenMP):

git clone https://github.com/JustVugg/colibri && cd colibri/c
./setup.sh   # checks gcc/OpenMP, builds, self-tests

The model is a pre-converted GLM-5.2 int4 container on Hugging Face — about 372 GB, so put it on a disk with room, ideally a fast NVMe:

COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect planned VRAM/RAM/disk placement
COLI_MODEL=/nvme/glm52_i4 ./coli doctor   # read-only readiness check
COLI_MODEL=/nvme/glm52_i4 ./coli chat     # RAM budget, cache, MTP auto-detected

Starting a chat looks like this:

$ ./coli chat
 🐦 colibri v1.1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
 ✓ ready in 32s · resident 9.9 GB
 › ciao!
 ◆ Ciao! 😊 Come posso aiutarti oggi?

Want an OpenAI-compatible endpoint? ./coli serve gives you the API only; ./coli web gives you the API plus a web dashboard on one port.

One critical gotcha: grab the model variant with int8 MTP heads. The original mirror ships int4 MTP heads, which collapse speculative-decoding acceptance to 0% (tracked in issue #8). Check with ls -l <model>/out-mtp-* — the int8 (correct) files are 3527131672 / 5366238584 / 1065950496 bytes.

The performance ladder — be honest about it

This is where Colibri earns trust: it publishes a full benchmark ladder, and it doesn't hide the slow end. Same engine, same int4 container — the hardware only changes where the experts live:

That 25 GB number is the headline, but read it correctly: at 0.05–0.1 tok/s cold, it is a proof of correctness, not a chat experience. The honest, usable configuration is a machine with a fast NVMe and enough RAM to keep the dense weights and a warm expert cache resident. The author is refreshingly clear about this — the 25 GB floor exists to prove the architecture is real, not to promise you a snappy assistant on a ThinkPad.

A few engineering details worth calling out, because they're the difference between "cute demo" and "actually correct":

• Token-exact validation. The forward pass is validated token-exact against a transformers oracle (teacher-forcing 32/32). Placement only ever decides speed.
• Compressed KV cache that persists. MLA attention stores 576 floats/token instead of 32,768 (57× smaller) and persists it across restarts (.coli_kv), so conversations reopen warm with zero re-prefill — byte-identical to an uninterrupted session.
• Speculative decoding done right. GLM-5.2's native MTP head drafts tokens the main model verifies in one batched forward (2.2–2.8 tokens/forward when it pays), with hard-won defaults like SPEC_PIN=1 so draft and verify compute the same function.
• Dual-drive streaming. Got a second SSD? Put a full copy of the model on it and the engine streams experts from both drives at once, routing each expert to a drive by deterministic hash weighted by measured bandwidth — a 9 GB/s + 3 GB/s pair reads ~33% faster than the fast drive alone.

What the community said

Colibri landed as a Show HN ("Getting GLM 5.2 running on my slow computer") and climbed to roughly 922 points and 238 comments, going from zero to over 9,600 stars in under two weeks. The reaction split along predictable but useful lines:

• "This is the good kind of hacking." The overwhelming top-line sentiment was admiration for the sheer audacity and cleanliness — a single C file, zero deps, a 744B model, and a real-time visualization of the experts firing. People compared its spirit to llama.cpp's early days: one person proving something was possible before the ecosystem caught up.
• "Tokens per second, though." The most common pushback was pragmatic: at sub-1 tok/s on realistic hardware, this is not replacing your API calls for interactive work. Commenters framed it as a research and learning tool — a way to study MoE routing and disk-streamed inference — more than a daily driver.
• "Disk endurance and read amplification." Streaming ~11 GB of experts per token off NVMe raised real questions about SSD wear and sustained read bandwidth. The dual-drive and O_DIRECT tuning knobs exist precisely because decode is disk-bound on most machines.
• "Show me the quality numbers." To the maintainer's credit, the quantization cost of the int4 container and the ablations are documented rather than waved away, which several skeptics acknowledged as unusually rigorous for a two-week-old project.

The through-line: people trust it because it refuses to oversell. The README leads with the slow floor, not the fast ceiling.

Honest limitations

Colibri is genuinely impressive, but it's a narrow tool, and you should go in clear-eyed:

• It is slow on realistic hardware. Unless you have a multi-GPU rig, expect 1–2 tok/s. This is for exploration, not production serving.
• You need ~372 GB of fast storage. The model container is large, and decode is disk-bound — a cheap QLC/DRAM-less SSD can be neutral-to-negative with O_DIRECT. NVMe with bandwidth headroom is effectively a requirement for anything usable.
• One model, for now. The engine is built around GLM-5.2's specific architecture (MLA attention, DSA sparse attention, the MTP head). It is not a general llama.cpp-style runtime that loads any GGUF.
• Setup has sharp edges. The int4-vs-int8 MTP-head trap, the model conversion step, and the tuning knobs (DIRECT, PIPE, SPEC_PIN, DRAFT) mean the "measure, keep what your hardware rewards" philosophy is real — this rewards tinkerers, not one-click users.
• Not accepting a broad contributor base yet. It's an Apache-2.0 project you can study, compile, and fork, but it reads as one person's deeply opinionated engine rather than a committee-built framework.

Who should actually use Colibri?

• Local-LLM enthusiasts who want to hold a frontier-scale model on hardware they already own, latency be damned.
• Systems and ML engineers curious about disk-streamed MoE inference, the memory-hierarchy-as-JIT idea, and how far the "place, don't fit" bet can go.
• Researchers probing MoE routing structure — the expert atlas and per-expert routing heat are a genuinely novel observability surface.

If you need an interactive assistant or a production endpoint, reach for a smaller model on a proper serving stack. Colibri is the opposite bet: maximum model, minimum hardware, honest about the tradeoff.

FAQ

Can I really run a 744B model on a 25 GB laptop?
Technically yes — the architecture is validated token-exact at that floor. But at 0.05–0.1 tok/s cold, it's a proof of correctness, not a chat experience. For usable speed you want a fast NVMe and enough RAM to keep the dense weights plus a warm expert cache resident; realistic decode is ~1–2 tok/s on CPU/single-GPU boxes and 5–7 tok/s on a 6×5090 rig.

How is this different from llama.cpp or Ollama?
Those keep the whole (usually much smaller) model resident in RAM/VRAM. Colibri deliberately does not: it streams GLM-5.2's routed experts from disk on demand, treating VRAM/RAM/NVMe as one tiered cache. It's specialized for one big MoE rather than being a general GGUF runtime.

Does streaming experts from disk hurt output quality?
No. Colibri's design goal is that placement only ever affects speed — the router's decisions and the weights' precision are identical whether an expert answers from VRAM or disk, and the forward pass is validated token-exact against a transformers oracle. The only quality cost is the int4 quantization of the container, which is measured and documented.

What hardware do I actually need?
A machine with ~372 GB of fast (ideally NVMe) storage for the model, enough RAM to hold ~9.9 GB of dense weights plus a warm expert cache, and gcc/OpenMP (or a prebuilt release). No GPU is required, but a GPU dramatically improves throughput by holding more experts resident. A second SSD roughly adds its bandwidth on top.

Is it production-ready?
Not for interactive or high-throughput serving on commodity hardware. It's best treated as a research, learning, and tinkering tool for disk-streamed MoE inference — and as an existence proof that frontier models don't have to be sealed inside datacenters.

The bottom line

Colibri isn't trying to be the fastest way to run a model — it's trying to prove that a 744B frontier model can run on hardware you already own, in pure C, with every expert visible as it fires. It succeeds at exactly that, and it's unusually honest about where the approach is slow. If you care about local inference, MoE internals, or the principle that intelligence should be something you can hold rather than rent, it's one of the most worthwhile repos of 2026 to clone and read.

Repo: github.com/JustVugg/colibri · License: Apache-2.0 · Model: GLM-5.2 int4 (with int8 MTP)