Running NVIDIA's new 120B Nemotron-3.5-Super on an 8 GB GPU — skipping the zeros, and finding the real bottleneck

NVIDIA released Nemotron-3.5-Super on October 15: a 120B mixture-of-experts model, 75 GB even in their 4-bit NVFP4 format. The usual answer for an 8 GB card is "not on your hardware". Two days later it runs on my home PC — one RTX 5060 with 8 GB — as an ordinary chat model in Open WebUI at ~8 tokens/s, it reads images, and its answers match the Hugging Face reference implementation.

Code: https://github.com/helgard-orlm/nemotron-super-8gb

Who did what, up front: I picked the model, set the goal and made the calls (including the idea to exploit the zeros after ReLU², below). The engine, kernels, server and measurements were written by Claude (Anthropic) in my sessions. "We" below means the two of us.

This is the third model in a series on the same box — after Qwen3.8-Flash-Next, 133 GB, 11 tok/s and DeepSeek-V4.1-Flash, 510 GB, 2.4 tok/s. Same idea, different model, and two new tricks.

The machine

RTX 5060 8 GB (Blackwell, PCIe 5.0 ×8), Core Ultra 5 225F, 32 GB DDR5, a Gen5 NVMe drive used only for model files. Nothing exotic.

The only number that matters: bytes per token

In a mixture-of-experts model most of the weights are experts, and each token uses only a few of them. So the question isn't "does 75 GB fit in 8 GB" — it's how many expert bytes does one token need.

Nemotron-3.5-Super: 88 layers (40 Mamba-2, 40 MoE, 8 attention), 512 experts per MoE layer, top-22. That sounds expensive, but NVIDIA made the experts tiny: they work in a 1024-wide "latent" space instead of the full 4096. One expert is up 2688×1024 + down 1024×2688, ReLU², no gate — 3.10 MB in NVFP4.

22 experts × 40 layers × 3.10 MB ≈ 2.7 GB per token

Between Qwen (1.3 GB/token) and DeepSeek (4.2 GB/token). Everything that isn't a routed expert — Mamba, attention, shared experts, router, latent projections, output head — is 6.6 GiB on the GPU, kept in the checkpoint's own FP8/NVFP4 formats and multiplied by our own Triton kernels. The 59 GiB of routed experts stay on the NVMe drive.

The engine, step by step

No existing runtime streams experts for this architecture, so it's a PyTorch + Triton engine with the transformers NemotronH code as the skeleton (Mamba mixers, attention, router). We replaced the experts, the dense layers and the per-token MoE step.

1. One block per expert on disk. Each expert's four tensors (up, its scales, down, its scales) are rewritten into one 3,096,576-byte block — exactly 756 pages of 4 KiB. One expert = one pread with O_DIRECT into pinned RAM. Repacking 59 GiB takes about 2.5 minutes.

2. A RAM cache in front of the disk. LRU over pinned memory, 32 reader threads. Routing is concentrated: with 24% of the experts in RAM, 81–84% of lookups hit.

3. Prefetch into the gap. Run the next MoE layer's router on the current input: it guesses the next layer's experts ~72% right. The missing guesses are read while the GPU is busy.

4. Skip the zeros (new). After ReLU², 92.5% of an expert's neurons are exactly zero for a given token. A zero neuron doesn't need its column of down — so why move it across PCIe? The catch: NVFP4 shares one scale per 16 consecutive values, so you can't just pick neurons out of the original layout. We re-laid down on disk once (a pure byte permutation, same file size): each neuron's 4-bit values are contiguous, scales grouped per 16 neurons. Now only up (1.55 MB) is copied to the GPU; the down kernel reads directly from pinned RAM over PCIe, and only the 16-neuron groups that contain a live neuron. +12%, 6.6 → 7.3–7.6 tok/s.

5. Chat server. OpenAI-compatible /v1/chat/completions with streaming, a -think model id for reasoning mode, and dialog snapshots (~170 MB: 8 attention KV layers + 40 Mamba states) so the next message only processes its new tail.

Where the time actually went — and where I was wrong

At 7.3 tok/s a token takes ~137 ms. The plan was CUDA graphs: thousands of small kernel launches per token, surely Python overhead is the problem. Before writing graphs, we profiled with Nsight Systems:

• GPU busy ~92 ms, idle ~54 ms per token;
• of the idle time, gaps between launches (what a CUDA graph would fix) were only ~13 ms.

Then we tried two "obvious" fixes for the remaining waits: reordering work so the shared expert runs before the router result comes back, and copying the predicted experts early. Both were bit-for-bit correct. Neither helped — the speculative copies made it 7% slower, because wrong guesses queued on the bus ahead of the right ones.

So we timed what the CPU was waiting for:

The disk was the biggest wait. An earlier measurement had said "disk ~3 ms" — but that run had GPU copies switched off, which hid the disk behind nothing. The first README said that; it's corrected now. (The bad number was pointed out by a second agent reviewing the README, not by me.)

The fix was boring: a bigger RAM cache. Before touching the live server we recorded the routing of 5 conversations and replayed an LRU cache offline. The simulation matched live miss counts within ±0.4 reads per token, so we could pick a size on paper:

20 GB live: 136.9 → 121.5 ms per token, 7.3 → 8.2 tokens/s. That leaves ~7 GB of RAM for the rest of a 32 GB system, which is why it stops there.

Results

A 32–40 token prompt: ~5 s. Load time: 7–27 s.

It reads images too

Nemotron-3.5-Super-VL ships a vision tower (C-RADIO, 1.3 GB). The engine's 6.6 GiB plus the tower don't fit on the card at once, so the tower's weights live in pinned RAM and are streamed to the GPU layer by layer while it runs (0.4–1.6 s per image, vs 10–31 s on the CPU). In Open WebUI you just attach a photo.

A test that's easy to fake and hard to pass: an invoice of 60 lines in a 15-px font, asked for the code, quantity and price of lines 21–30. At normal resolution (1047 image tokens) it got 7–9 of 10 per field — swapped letters, one made-up line. At NVIDIA's maximum resolution (2204 tokens, a separate -doc model id): 10/10 on all three fields, 116 s total. The vision code is in the repo too (vision/, tests with the invoice and its ground truth in tests/vision/).

How we know it's right

Speed is meaningless if the model says different things. The reference is transformers NemotronHForCausalLM in fp32 on the CPU, with weights read from the original checkpoint — not our repack — and unpacked with transformers' own FP4 code:

Mean KL ≤ 0.003. The one differing token is a real tie: the reference itself scores -layer and layer 0.11 apart, less than one BF16 step.

Plus checks that must fail: swap the two 4-bit halves of each byte and PPL jumps to 18,784 — so the unpacking order is tested, not assumed. Repack and re-layout were compared byte-for-byte with the original on 200 experts each.

One bug of ours was caught by running the same input three times: perplexity drifted between 5.19 and 5.29. Cause: index_add_ with repeated indices uses atomics in random order. Fixed with a per-(token, expert) buffer summed in a fixed order. A second source of run-to-run difference wasn't a bug: mamba-ssm Triton kernels pick configs by timing, so separate processes rounded differently; TRITON_CACHE_AUTOTUNING=1 makes them bit-identical.

What didn't work

• Putting the prompt's experts into the RAM cache — prompts got slower, decode didn't gain.
• Prefetching 6 guesses instead of 4 — more wasted reads.
• Skipping zeros per single neuron — re-reading scales per neuron was slower than copying the whole block. Per 16-neuron group worked.
• Speculative early copies (above) — correct, slower.
• Keeping recently used experts on the GPU — 40% of a layer's experts repeat on the next token, but holding them needs ~2.7 GB of VRAM that an 8 GB card doesn't have next to the 6.6 GB dense part.

Try it

The repo has the engine, the Triton kernels, the server, the repack/verify tools and every measurement script behind the numbers above. Honest caveat: it hasn't been run end-to-end from a fresh clone yet — the README lists the exact steps used on my machine. You need a Blackwell GPU (the kernels use the NVFP4 hardware path), ~70 GB free on a fast NVMe drive, and 32 GB of RAM.

The point is the same as in the earlier posts: for a mixture-of-experts model, "does it fit in VRAM" is the wrong question. Count bytes per token — and then measure where the time goes before you fix it, because the obvious culprit (here: Python launch overhead) can be the wrong one.

Engine, kernels, server, measurements and this write-up: Claude (Anthropic). Model choice, goal, the ReLU² idea and the decisions: me. All numbers are from logs on the machine described above.