I Ran a 284B-Parameter LLM From 3.2GB of RAM — in Plain C

DeepSeek-V4-Flash has 284 billion parameters and takes up about 160GB on disk. My laptop does not have 160GB of RAM. It doesn't even have 32GB.

It ran the model anyway. Peak memory: 3.23GB. With a GPU and a bit more headroom, it generates at 1.6–1.7 seconds per token. No quantizing the model down to fit, no renting a multi-GPU box. Just C99, streaming weights off NVMe as they're needed.

This post is about how deepseek-v4-in-c actually works, and — more usefully — about three bugs I hit building it that taught me more than the parts that went smoothly.

The trick that makes this possible at all

DeepSeek-V4-Flash is a mixture-of-experts model: 256 experts per layer, 43 layers, but only the top 6 experts per layer actually fire on any given token. That's the whole game. You don't need 160GB resident in memory — you need whatever fraction of the checkpoint this specific token's routing decisions touch, which works out to about 3.2GB per forward pass.

So instead of loading the model, I stream it. Every token, the router decides which experts it needs, and the engine pulls just those off disk into an LRU cache. Give it more RAM and the cache gets bigger and hits more often; give it almost none and it still runs correctly, just slower, re-reading more from disk each time.

I didn't build the streaming layer from nothing — it's ported from Fareed Khan's kimi-k3-in-c, which does the same thing for Kimi K3. The safetensors reader, the disk streamer, the memory planner, roughly 40% of the codebase — that transfers directly. The other 60% doesn't: DeepSeek-V4 and Kimi K3 don't share any math, so every kernel had to be written fresh against DeepSeek's own reference implementation.

Why I didn't trust it until I could prove it

Here's the uncomfortable truth about hand-writing inference kernels: a model with a swapped nibble or a misindexed scale factor will still produce text that reads fine. Fluent output is not evidence of a correct implementation. It just means the bug isn't catastrophic enough to break language modeling entirely.

So before I believed any of my own benchmark numbers, I checked correctness at three levels against PyTorch:

• Per kernel — 14 kernels, checked individually, agreeing to 5e-7
• Per block — three layer types, 46 positions, same tolerance
• End-to-end — a real (tiny) deepseek_v4 checkpoint run through the actual C loader and actual inference path, matching PyTorch to 2.9e-6 with identical argmax at every single position The PyTorch side is a from-scratch reimplementation of DeepSeek's inference/model.py, not derived from my C code. If I'd built the reference from my own implementation, both could quietly agree on the same misunderstanding.

I went one step further for the C code itself: bit-exactness between the scalar path, the OpenMP path, and the AVX2 path. Not "close enough" — byte-identical, enforced with a fixed 16-accumulator reduction tree and -ffp-contract=off, checked at runtime by literally running both paths on the same input and diffing the bits. The GPU can't join that club — warp-level reduction order isn't deterministic the same way — so it gets held to relative error plus argmax match instead, and the CPU stays the ground truth.

The optimization that was actually silent corruption

This is the bug I'm most annoyed I shipped, and most glad I caught.

Reading experts off disk uses O_DIRECT, which demands the file offset, read length, and destination buffer all land on 4096-byte boundaries. The checkpoint's tensors don't naturally align that way, so my first version widened every read to the nearest aligned window, landed it in a staging buffer, then memcpy'd the actual payload into the cache slot. A 12.75MB copy, roughly 5,000 times per run. I'd written it off as cheap — "~1ms against the ~10ms the unbuffered read saves" — without actually measuring it.

I measured it. The read itself was 2.81ms. The copy was costing 3.60ms total per expert. The copy wasn't a rounding error, it was 22% of the cost.

The fix seemed simple: allocate cache slots 4096-aligned with a little slack, and place each read at the offset whose alignment residue matches the actual tensor, so the widened window lands exactly where the data belongs — no copy needed. My first attempt at that fix was wrong. It placed each read at the next offset with a matching residue, without accounting for the fact that an aligned window can begin up to 4095 bytes before the actual payload starts. So a second tensor's "aligned" read could reach backward and silently overwrite bytes belonging to the tensor before it.

Every existing test passed. Every one. Here's why: my correctness gates compared the cache against itself — serial mode against concurrent mode — and both were corrupting the data identically, so the comparison found nothing wrong. Worse, the corrupted version looked like a huge win: cache hit rate jumped from 52.6% to 95.5%, because the corruption had collapsed routing onto a tiny handful of experts. Disk reads dropped from 61GB to 5.76GB. It looked like I'd found a 5x speedup. The only thing that gave it away was that the generated token IDs were wrong.

I fixed it two ways: a new gate that compares the fast path against a plain buffered pread sharing no code with it (so it can't fail the same way), and a test fixture that actually mirrors how the real checkpoint splits an expert's tensors — three scale tensors in one disk region, three weight tensors ~341MB away at a different alignment residue. My synthetic fixture had been storing everything contiguously, which meant the bug's trigger condition literally never came up in testing. Real fix, properly verified: disk time down 16%, wall clock down 12%.

Lesson I'm keeping: if two things you're comparing can fail identically, your test proves nothing. I needed a reference path with zero shared code.

A 30x slowdown with the wrong explanation

Turning on --gpu moves the dense trunk's FP8 matrices into VRAM. Naively, that should be free for anything still running on CPU — the GPU does its thing, the CPU does its thing. It wasn't. I benchmarked a CPU-only FP4 matmul while the GPU was busy and watched throughput fall from 119 GF/s to 3.9 GF/s. A 30x collapse.

My first theory: CUDA's default sync mode spins the calling thread instead of sleeping it, so that thread competes for CPU cycles with my OpenMP workers. I switched to cudaDeviceScheduleBlockingSync, which sleeps instead of spinning. It should have fixed it.

It didn't, really — 5.0 GF/s instead of 3.9. Still a 25x collapse.

What actually fixed it: reserving one CPU core for the thread driving the GPU, and letting OpenMP use the rest. That removed the collapse completely, in either sync mode. My best guess now is DMA traffic from the GPU contending with a memory-bandwidth-bound kernel for DRAM bandwidth — but I haven't measured that, so I'm not claiming it in the README as fact. The fix works. The explanation I originally reached for was wrong, and I'd rather say that than pretend I nailed it on the first guess.

One caveat I want to be honest about: this collapse is much bigger in the microbenchmark than in the real model. The synthetic benchmark keeps the GPU saturated back-to-back; the actual model only touches it a few times per layer. On real generation, holding back a core is worth 10–15%, not 3x. An earlier draft of my README overstated this based on a heat-soaked benchmarking session, and I want to flag that explicitly rather than let an inflated number stick around.

Thermal state matters more than I expected

Speaking of that heat-soaked session — this is the correction I'm least proud of needing to make, and the most useful thing in this whole post if you benchmark anything disk-bound.

I re-ran every number in the README on a freshly rebooted, idle machine on AC power. The gap versus numbers taken after 24+ hours of sustained load was not the ±20% I'd assumed:

The pattern makes sense in hindsight: the more disk-bound a configuration is, the more it suffers when the machine is hot, because sequential O_DIRECT throughput itself drops under thermal load (5.3 GB/s cold vs 4.4 GB/s heat-soaked, measured directly). Low---budget runs are almost pure disk I/O, so they took the worst of it.

I now treat any single timing as ±20% at a fixed thermal state and up to 3x across states — and I resolve actual kernel changes with a dedicated microbenchmark (bench/matmul_bw.c) rather than by timing a full generation run, because a 70-second end-to-end run just can't resolve a 12% kernel improvement through that much noise.

The --budget flag is the one that actually matters

If you try this yourself, the flag to understand is --budget — it sets total RAM for the trunk plus the expert cache, and getting it wrong doesn't just make things slow, it can make the cache mathematically incapable of ever hitting.

One forward pass touches 258 experts across all layers — about 3.21GB. Below that, an LRU cache evicts every entry before its layer comes back around, so it can never hit. I measured this directly at an old default: 10,320 requests, 0 hits, 128GB pulled from disk for one run.

--budget    expert cache    hit rate    disk read
8 GB        1.6 GB          0%          128 GB
12 GB       5.6 GB          ~40%        ~80 GB
16 GB       9.6 GB          53%         61 GB

The engine now prints a warning naming the threshold if you're under it, because I hit this myself before I understood why my "optimized" cache was doing nothing.

Trying it yourself

make            # CPU-only build
make test       # 20 gates, no model weights required

# pack the checkpoint once
python3 tools/pack_trunk.py ~/models/dsv4-flash ~/dsv4-trunk
python3 tools/pack_tokenizer.py ~/models/dsv4-flash ~/dsv4_tok.bin

./bin/dsv4 ~/models/dsv4-flash \
    --trunk ~/dsv4-trunk \
    --tok   ~/dsv4_tok.bin \
    --prompt "The capital of France is" \
    --gen 25 --budget 16 --gpu

--chat applies DeepSeek-V4's real prompt format — I had to pull it from the checkpoint's own encoding_dsv4.py, because it isn't where most tokenizer tooling looks (tokenizer_config.json). --think opens a reasoning block before the model answers. CUDA is auto-detected; without nvcc, the build still succeeds and --gpu just reports no device found instead of failing.

What's still open

• DeepSeek-V4-Pro (61 layers, 384 experts, ~865GB checkpoint) is gated and planned in the test suite but never actually run — I don't have the disk for it. The planning gate already caught a real bug: a shape-validation rule that happened to compute the right answer for Flash by coincidence (both sides of the equation collapse to 4096) but would silently fail on Pro's dimensions. A Flash-only test suite would never have caught that.
• Batched prefill doesn't exist yet. Prompt tokens go through one at a time, same as generated ones, at roughly 1 second each — so a long prompt dominates latency far more than generation speed does. I've measured the expert-reuse ceiling that batching would unlock (up to ~8x weight reuse at 200+ tokens of prompt), but implementing the batched kernels is future work, not something I've shipped.

• Tool-calling loop. The model correctly emits its tool-call format, but actually driving an agent loop on top of it is a layer above this CLI that I haven't built.

  What I'd tell someone doing something similar

• Fluent output from a hand-written kernel is not proof it's correct. Get an independently-written reference and check bit-level agreement, not vibes.

• If your test compares two things that can fail the same way, it proves nothing. I needed a reference path with zero shared code to catch my worst bug.

• Measure the thing you think is expensive before you "optimize" it. My memcpy was 22% of cost, not the 1ms I'd assumed.

• Report the thermal state your benchmarks ran under, or don't trust them.
  Code's here, Apache-2.0: github.com/ronak-create/deepseek-v4-in-c