Run GLM-5.3-Flash (321B) locally with Project Maya and EXL3 on one RTX 5090

GLM-5.3-Flash is a 321B mixture-of-experts model, and its 3.05 bpw EXL3 quant is 125 GB, too big to fit in my workstation's 128 GB of RAM next to the OS. It still runs locally here: on one RTX 5090 with 128 GB of RAM, with a 512K-token context, at about 25 tokens per second. This guide shows how, with Project Maya, and what we measured on the way.

Every number below was measured on our own machine on October 15, 2026. Where a number comes from someone else, I name the source.

This is the second of two articles. The first one covers Qwen3.8-Flash-Next with Strata at 79 tokens per second, and before that I wrote about Qwen3.8-Flash-Next with EXL3 and TabbyAPI on 40+ GB of RAM.

The model

GLM-5.3-Flash is an open-weight model from zai-org under the MIT license. Maya's README describes it as 321B parameters with about 18B active per token and a context of up to 1M tokens. It has a hybrid attention design and an MTP layer for drafting.

18B active parameters is three times the roughly 6B that the Qwen3.8-Flash-Next model card lists. That explains most of the speed difference below: more of the model has to move through the CPU and over PCIe for every token.

Why not TabbyAPI

I tried TabbyAPI first, with turboderp's GLM-5.3-Flash EXL3 quants. ExLlamaV3 keeps CPU experts in a RAM arena and has no SSD tier for experts. So the whole model has to fit in VRAM plus RAM, next to the OS. On 128 GB, the largest quant that worked was the self-calibrated 2.60 bpw revision:

The 3.05 bpw quant (125 GB) does not fit that way. You need a runtime that can leave part of the experts on the SSD.

Why Project Maya

Project Maya is a fork of Strata, built for this one model. Its engine keeps the experts your conversation uses most on the GPU, the next ones in RAM and the rest on the NVMe drive, and moves them while you chat. It measures your hardware and tunes itself, exposes OpenAI- and Anthropic-compatible APIs, and has a dashboard with a live monitor.

Maya ships its own GGUF quants (Maya-S 96.5 GB, Maya-M 116 GB, Maya-L 156 GB). It can also read turboderp's EXL3 quants in place from their safetensors, which its README marks as experimental and NVIDIA-only. I chose the EXL3 3.05 bpw route because I already trusted that quant from the TabbyAPI tests, and because Maya then decoded as fast as the smaller sc_2.60 quant did in TabbyAPI.

The machine

• GPU: RTX 5090, 32 GB of VRAM, also driving the desktop.
• CPU: Ryzen 9 9950X3D (16 cores, 32 threads).
• RAM: 128 GB DDR5 as 4x32 GB at 5,600 MT/s.
• Disk: one NVMe drive.
• OS: Arch Linux (Omarchy), NVIDIA driver 615.78, CUDA 13.4.
• Maya 1.1.11 (commit d187195, October 15, 2026).

Install

git clone https://github.com/mw00/project-maya.git
cd project-maya
./maya.sh --check

--check prints the GPUs, driver, CUDA toolkit, compiler, RAM and CPU. Then run setup non-interactively:

./maya.sh --setup --plain --yes --gpu 0 \
  --context 131072 --model EXL3-3.05bpw \
  --models-dir ~/Models/maya --download-model \
  --no-vision --draft mtp --no-start

What happens, according to Maya's setup docs and what we saw:

1. It checks the PC and installs its Python packages into .venv.
2. It fetches llama.cpp's source at a pinned commit and compiles the engine for your GPU. Maya's docs say 10-30 minutes, once.
3. It downloads the 125.3 GB quant (27 files) with resumable curl commands it prints first, and checks every file against its SHA-256.
4. It builds the pack, the engine's index of the model files, in the model folder. Here that was a 1.3 GB pack/ folder plus a 2 GB maya-exl3.gguf next to it.
5. It writes maya-exl3-3.05bpw.json (the engine config) and run-maya-exl3-3.05bpw.sh.

The flags that matter:

• --model EXL3-3.05bpw selects turboderp's EXL3 quant instead of a Maya GGUF.
• --draft mtp drafts with the model's own MTP block. The DFlash2 drafter is the other option, but its license is CC BY-NC-ND 4.0, non-commercial only, so I left it out.
• --no-vision skips the image encoder (a 10-20 minute build and 1.1 GB).
• --no-start installs without starting, so you can start it the way shown below.

Edit the config before the first start

Two settings go into the args of maya-exl3-3.05bpw.json:

"args": [
  "--glm-pack", "/home/you/Models/maya/EXL3-3.05bpw/pack",
  "--max-context", "524288",
  "--expert-cache", "auto",
  "--prefill", "auto",
  "--kv-streaming", "on",
  "--vram-reserve-mib", "3072"
]

• --vram-reserve-mib 3072: the VRAM Maya's expert pool leaves free. The default is 700 MiB. Our desktop runs on the same card, and with the VRAM full the screen flickered, so we keep 3 GB free everywhere.
• --max-context 524288: GLM-5.3-Flash supports up to 1M tokens, so this is not rope scaling, just a bigger limit. With --kv-streaming on, extra context costs RAM (about 13 KB per token, judging by the size of the sessions Maya saved to disk), not VRAM.

A new --setup run writes this file again. Maya's docs say it keeps settings like the reserve, but keep a backup anyway.

Start it outside your shell's memory limits

Maya pins most of the free RAM for its expert tier. On this machine it leaves about 4-5 GB free. That collides with two Linux defaults:

• A shell inside a systemd scope with a memory limit. Our agent's terminal scopes have memory.max of 4 GiB, and a big process started from there lands in swap.
• systemd-oomd, which Omarchy enables for the user's app.slice. It kills a cgroup after 20 seconds above 50% memory pressure, and a model pinning RAM during load triggers exactly that.

So we start it in a separate slice, without a memory limit:

systemd-run --user --scope --collect --slice=llm.slice \
  --unit="maya-$(date +%s)" -p MemoryMax=infinity \
  ./run-maya-exl3-3.05bpw.sh --plain

--plain keeps the output as plain text instead of Maya's full-screen dashboard, which is easier to log. Stop it with systemctl --user stop 'maya-*.scope'. The start takes a few minutes. Then:

curl -s http://127.0.0.1:8080/v1/models

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "glm-5.3-flash", "messages": [{"role": "user", "content": "Say hello in five words."}],
       "max_tokens": 2000}'

The dashboard is at http://127.0.0.1:8080/#monitor and shows decode and prefill speed, the VRAM hit rate and where the experts live. Keep the server on 127.0.0.1; if you open it to your network, use --host 0.0.0.0 together with --api-key.

Calibrate

Maya's calibration measures how to split the work between the CPU and the PCIe link, and how many CPU threads to use:

./maya.sh --calibrate --no-start

It loads the full model, so run it with Maya stopped and in the same way as the server (in llm.slice, without a memory limit). It refuses to run while more than 1 GiB of VRAM is in use on the GPU, to make sure no other engine is loaded. A desktop on the same card already takes about 2 GB, so on our machine it never started. I raised that threshold locally in busy_gpus() in maya.py from 1024 to 3072 MiB. That is a local patch: check it after every update, and only use it if you are sure nothing else heavy is on the card.

The calibration took about 10 minutes (Maya estimates 10-15). Its full output on our machine:

the engine's own split (PCIe share 0.60): 23.7 tok/s
PCIe share 0.00: 21.1 tok/s
PCIe share 0.10: 24.6 tok/s
PCIe share 0.25: 23.9 tok/s
PCIe share 0.50: 24.1 tok/s
PCIe share 0.75: 23.3 tok/s
16 CPU threads: 24.6 tok/s
12 CPU threads: 21.9 tok/s
11 CPU threads: 20.0 tok/s
8 CPU threads: 18.1 tok/s
4 CPU threads: 13.7 tok/s
confirming against the engine's own settings ...
[ok] tuned for this PC: STRATA_GLM_PCIE_SHARE=0.10 (25.7 tok/s)

It stores the result as an environment variable in the config. In the sweep, a PCIe share of 0.10 beat the engine's own split by about 4% (24.6 vs 23.7), and the confirmation run measured 25.7. Fewer than 16 CPU threads hurt a lot.

Benchmarks

How we measured: a client-side streaming bench against the OpenAI endpoint. Eight runs per setting, alternating a ~400-word Dutch prose prompt and a coding prompt (binary search with tests), greedy decoding, 512 output tokens, median reported. Prefill comes from a needle-in-a-haystack test at the given prompt length. The needle and filler text are fixed per length, and Maya caches prompt prefixes (also on disk, across restarts), so a later run can be flattered by an earlier one: read the prefill numbers as approximate. Nothing else heavy ran on the machine.

The first two rows ran with the 700 MiB default reserve, and you can see the VRAM peak at the edge of the 31.8 GiB card. With the 3 GB reserve, the peak stayed at or below 29.5 GiB. Reading 506K tokens took 262 seconds. Decoding at 512K was as fast as at 262K, because the KV cache streams from RAM.

Compared to TabbyAPI on the same machine: about the same decode speed as the smaller sc_2.60 quant (25.5, from TabbyAPI's server log rather than this client bench), with a bigger quant, about 1.7x the prefill speed, and twice the context.

For reference, Maya's README lists "30+ tokens/s" for an RTX 5090 with a Ryzen 9 7950X and 96 GB of DDR5, with Maya's own Maya-M quant. That is a different quant and a different bench, so do not compare the numbers one to one.

Quality check

We ran the same five coding tasks with hidden tests that we used for the Qwen setups: three trials each, sandboxed, reasoning effort medium, and each server's default sampling (Maya temperature 1.0 and top_p 0.95, Strata greedy, TabbyAPI temperature 0.7).

The one miss was invalid JSON (a trailing comma) in one trial of an incident-report task; all the code tasks passed. These tasks are too easy to say which model is smarter. What they do show is that a 3-bit GLM on a desktop produces working code, and that it takes about 1.5x the time per task because of the slower decode.

Pitfalls we ran into

Nothing else fits next to it. Maya pins RAM until about 4-5 GB is free and holds 29-30 GB of VRAM. Do not run a second model runtime, a game or a GPU miner next to it. Switching models means stopping one and starting the other.

Wait for the VRAM after a stop. VRAM takes several seconds to come back after a runtime stops. Start the next one too early and it fails a VRAM check or gets a smaller expert pool.

Unstable RAM corrupts downloads. Our four DIMMs at their 6,000 MT/s EXPO profile were not stable: kernel oopses, crashes, and most of a model download written in that time was corrupt. We run 5,600 MT/s now. Maya checks its own downloads against SHA-256; do the same for anything you fetch yourself.

Measure with fresh prompts. A repeated long prompt hits the prefix cache. One of our runs reported 70,624 tokens per second of "prefill" for exactly that reason.

Reasoning effort and sampling. Setup wrote "reasoning_effort": "high" with temperature 1.0 and top_p 0.95 as server defaults in the config. Long reasoning at 25 tokens per second adds up, so for interactive work consider sending a lower reasoning_effort from the client. On the Qwen setup, medium scored best on our own bench; for GLM we have not measured that yet.

Which setup we use when

On this one machine we now have three ways to run a large model, and we switch with a stop and a start script:

• Qwen3.8-Flash-Next UD-Q5_K_XL on Strata: about 79 tokens per second, 512K context, our daily default for coding and agent work.
• Qwen3.8-Flash-Next EXL3 4.05 bpw on TabbyAPI: about 68 tokens per second in its server log, at 262K context.
• GLM-5.3-Flash EXL3 3.05 bpw on Maya: about 25 tokens per second, a 321B model with 512K context, for when a second, bigger model should look at a problem.

The Strata setup, with its calibration A/B and our own nine-task bench, is in the companion article: Run Qwen3.8-Flash-Next at 79 tok/s with Strata and a 5-bit quant on 128 GB of RAM.

Update, October 21, 2026: the same machine now also runs DeepSeek-V4.1-Flash (748B) with Project Maya, and all four local models are compared on the same day in Four big local LLMs on one RTX 5090 and 128 GB of RAM, benchmarked side by side.

If you are deciding what hardware to buy for running AI locally, Voltti has a practical guide on the trade-offs between local and cloud AI for business use.