Tokens per second is a memory bandwidth number. Measure it in 2 seconds.

A 40B model that does 125 tok/s on a laptop, and a 7B model that does 1.5 tok/s on a 2011 CPU with the same amount of RAM. Same ballpark of memory, ~80× difference in speed. Almost every explanation you will read for that gap is wrong, and the wrong explanation costs people money.

Here is the number that actually decides it: how many bytes your machine can read per second. Not cores. Not GHz. Not TOPS.

The formula

Generating one token with a dense model requires reading the entire set of weights once. Every token. There is no caching trick around it — the weights do not fit in cache, so each token is one full sweep of DRAM.

tokens/second  ≈  memory bandwidth  ÷  bytes of weights read per token

That is the whole ceiling. Everything else — kernel quality, quantization, --threads — decides how close you get to it, never how far above it you can go.

I measured the bandwidth on my own box

The machine is an Intel i5-2520M from 2011 running as a NAS. 8 GB of DDR3-1600, one DIMM, nproc 4. Not a benchmark rig; it is the thing serving files in the other room.

import numpy as np, time

mb = 512
a = np.ones(mb * 1024 * 1024 // 4, dtype=np.float32)

a.sum()                                  # fault the pages in first
ts = []
for _ in range(5):
    t = time.perf_counter()
    a.sum()                              # read-only, no write traffic
    ts.append(time.perf_counter() - t)

print(f"{mb / 1024 / min(ts):.2f} GB/s read")

Results:

Three things worth noticing.

First, the copy number is not the bandwidth number. A copy reads each byte and writes it, so it moves twice the traffic for the same wall time. 4.7 GB/s of copying is ~9.5 GB/s of DRAM traffic — which is the same wall the read pass keeps hitting. If you benchmark with memcpy and compare it directly against a model's tok/s, you will be off by a factor of two and conclude your machine is magically twice as fast as physics allows. Use a read-only pass.

Second, 9.0 GB/s against a theoretical 12.8 GB/s for single-channel DDR3-1600 is 70% of peak. That is a healthy result. Old hardware is not secretly broken; it is just narrow.

Third, adding threads does not add bandwidth. Same 256 MB arrays, one thread per array, min of five runs:

It plateaus at ~10.5 GB/s and stops. On a dual-channel board four threads would have gone well past that — two channels of DDR3-1600 is 25.6 GB/s of theoretical peak, and you would expect to reach most of the way there. I get about 40% of it. So: one DIMM, and the DIMM is the ceiling, not the core count. Confirm it on your own box with sudo dmidecode -t memory | grep -E "Size|Locator|Speed|Rank" before you spend anything.

What that predicts, and what it does not

Plug the measured 9.0 GB/s in:

Real-world runs land at 60–80% of the ceiling. If you are seeing 1.5 tok/s from a 7B Q4 model on hardware like this, your setup is fine and there is nothing left to tune. If you are seeing 0.2 tok/s, something is wrong — usually it is swapping, not compute. On an 8 GB machine with 5.1 GB actually available, a 4.4 GB model plus the OS is right at the edge; the moment the kernel starts paging, you are no longer measuring DRAM bandwidth, you are measuring your SSD.

Now the 125 tok/s claim

I have not run Flint 40B, so I am not going to tell you it is fake. I am going to tell you what has to be true for the number to be real.

At 23 GB of weights per token, 125 tok/s requires 2,875 GB/s. Nothing portable has that. The fastest laptop memory shipping today is in the several-hundred-GB/s range. So a 40B model at 125 tok/s is not reading 23 GB per token — which means most of its parameters are not being touched for a given token. That is what mixture-of-experts buys you: 40B parameters resident, a couple of billion active. Divide the active set instead of the whole model and the arithmetic works out fine.

This is the actual shift in 2026's local models, and it is worth internalising: parameter count stopped being the thing that sets your speed. Active bytes per token is. A "40B" MoE can be several times faster than a "13B" dense model while using more RAM. Benchmarks that report only total parameters tell you nothing about tok/s.

The table you actually want

Bandwidth by machine, and the ceiling it implies. Only the first row is measured here; the rest are stated figures with a 75–85% efficiency factor applied, which is why they say "est."

Read this table before you buy anything. Then read it again.

What it changes about buying hardware

Three practical consequences, in order of how much money they save.

1. The cheapest upgrade on an old box is a second DIMM. My machine is single-channel. Adding a matching stick roughly doubles the read bandwidth, and therefore roughly doubles tok/s — for a part that costs less than a month of coffee. Check first:

sudo dmidecode -t memory | grep -E "Size|Locator|Speed|Rank"

One "Size: 8 GB" line means single channel. Two matched lines means dual. This is a bigger lever than any CPU you can put in the socket.

The caveat, and it is the reason this advice does not help me: if both slots are already populated you cannot add a channel, only replace the pair with larger sticks — same bandwidth, more capacity. Bandwidth is a property of the channel count, not the DIMM size. So the question to answer before shopping is how many slots are free, and dmidecode answers it in the same output.

2. A CPU upgrade buys you almost nothing for generation. My CPU is thermally degraded — it hits its 86 °C throttle line about three seconds into a sustained load and settles at 1.7–2.0 GHz, below its 2.5 GHz base clock. It has already given up 20–30% of its rated clock. Token generation does not care, because the cores are idle waiting on DRAM. Prefill (processing your prompt) is compute-bound and does care; the first ten seconds of a long prompt will get faster. Steady-state chatting will not.

3. If you want 40B-class dense models at conversational speed, you are buying bandwidth. That means a machine with wide memory — an Apple Silicon laptop, or a GPU with fast VRAM. Not a faster CPU, not more cores, not more RAM of the same narrow kind. More RAM lets you fit a bigger model; it does not make it faster.

That is the whole thing. Measure your read bandwidth, divide by the size of the weights, and you know what your hardware is going to do before you download 20 GB over your home connection.

Reproduce it

The snippet above is the entire method — numpy, five runs, minimum. It takes two seconds on any machine and it is more informative than any spec sheet.

If you want the shopping side of this — which used mini PCs actually have two DIMM slots and what they cost — I keep that list here: Best mini PC for running Ollama under $300. I also wrote up which cheap SSDs survive in a NAS, because the "it started swapping" failure above is exactly how people kill a consumer drive.

How this was made: drafted with AI assistance, which I am stating up front rather than leaving you to guess. The memory benchmark was run by me on the machine described above, and every number in the first two tables is a command you can re-run in two seconds. The Flint 40B figures are from the vendor's announcement — I have not run that model, and the arithmetic in that section is presented as arithmetic, not as a benchmark. The two links at the end point to my own site, which carries affiliate links that are labelled as such on the page.