Ollama Not Using GPU? Fix It on Linux, Windows and WSL

Related on the site: Nobody Talks About RAM. Every Local-LLM Regret Is a RAM Problem..

TL;DR

Run ollama ps while a model is loaded: the PROCESSOR column tells you the truth. 100% GPU means the GPU is fine and you can stop reading. A split like 40%/60% CPU/GPU means the model didn't fit in VRAM — use a smaller quant. 100% CPU means Ollama found no usable GPU: usually an outdated driver, a missing group membership (AMD on Linux), a pinned OLLAMA_LLM_LIBRARY, or a container started without GPU access. Fix the cause the logs name; there are only about five.

How do I check if Ollama is actually using the GPU?

Two commands, no guessing.

# In one terminal: load a model
ollama run llama3.2 "hello"

# In another: see where it runs
ollama ps

NAME            ID          SIZE     PROCESSOR        UNTIL
llama3.2:latest a80c4de17cd9 3.3 GB   100% GPU         4 minutes from now

The PROCESSOR column has three states:

• 100% GPU — every layer offloaded. Done.
• 48%/52% CPU/GPU — partial offload. The GPU is working, but the model plus context didn't fit in VRAM. See the VRAM section below.
• 100% CPU — inference is on the CPU. The GPU was either not detected or deliberately disabled.

Then read the server log, which names the hardware Ollama actually found at startup:

journalctl -u ollama --no-pager | grep -i "inference compute"

On a healthy NVIDIA box you want a line like:

inference compute id=GPU-xxxx library=CUDA compute=8.9 driver=12.4 name=NVIDIA GeForce RTX 4070

No line at all, or one that ends with a CPU-only fallback message, and you've found your problem. The rest of this post is the five causes, most likely first.

Why does Ollama say "no compatible GPU discovered"?

On NVIDIA, the usual culprit is the driver, not CUDA. Ollama ships its own CUDA runtime libraries, so you do not need the CUDA toolkit installed — but the bundled runtime needs a driver new enough to talk to it. nvidia-smi working is not proof; it only proves a driver exists, not that it's recent enough.

nvidia-smi --query-gpu=driver_version --format=csv,noheader

If the version is years old, update it and reboot:

# Debian/Ubuntu family
sudo apt install nvidia-driver-570
# Arch family
sudo pacman -S nvidia

After a driver update, restart the Ollama service so it re-detects devices — detection happens once at startup, not per request:

sudo systemctl restart ollama

If the log now prints your GPU with library=CUDA, you're done. If it still refuses, check that OLLAMA_LLM_LIBRARY isn't set anywhere — see the "after an update" section.

Why does Ollama use the GPU for only part of the model?

Partial offload is arithmetic, not a bug: the model weights plus the KV cache for your context window must fit in VRAM. A 7B model at Q4 is roughly 4–5 GB; give it an 8K context and the cache adds more. On an 8 GB card something has to stay on the CPU, and ollama ps shows the split.

Three ways to close the gap, cheapest first:

1. Smaller quant. Dropping from Q8 to Q4 halves the weight size with a modest quality cost. The trade-offs are laid out in GGUF quantization levels explained.
2. Shorter context. num_ctx dominates the cache size. 32K context on an 8 GB card means most layers stay on CPU.
3. Fewer GPU layers. The num_gpu option caps how many layers get offloaded. Setting it below the layer count guarantees a split — if someone set it in a Modelfile or API call, unset it.

Note the reverse trap too: a GPU that shows 100% GPU but runs slower than expected may be swapping over system RAM. Check ollama ps SIZE against your actual VRAM.

Why is Ollama not using my AMD GPU?

AMD on Linux needs three things, and all three are checkable:

1. ROCm support in the build. The official Linux install script bundles a ROCm build. Confirm what the server detected:

journalctl -u ollama --no-pager | grep -iE "rocm|inference compute"

2. Group membership. The ROCm runtime needs access to /dev/kfd and /dev/dri, which means the render and video groups:

sudo usermod -aG render,video $USER
# log out and back in, then:
sudo systemctl restart ollama

This single missing group is the most common "Ollama not using GPU on Ubuntu" post on every forum, and it survives driver reinstalls because the driver was never the problem.

3. A supported GPU — or an override. Unsupported RDNA2 consumer cards (gfx1031, gfx1032) fail detection even with a working ROCm stack. The standard workaround is claiming a compatible target:

sudo systemctl edit ollama

[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"

Then sudo systemctl restart ollama. This is an unsupported-but-widely-used override; if it misbehaves, remove it and you're back to official support territory. If you'd rather have full control over backends than fight autodetection, that's the core difference covered in llama.cpp vs Ollama.

On Windows, AMD support is narrower — check Ollama's supported-GPU list for your card before assuming the install is broken.

Why did Ollama stop using the GPU after an update?

Updates change one of three things, in this order of likelihood:

1. A pinned backend library. OLLAMA_LLM_LIBRARY forces a specific runner (cuda_v11, rocm, or even cpu). It's meant for debugging, it overrides autodetection silently, and it persists in shell profiles and service files long after the reason is forgotten. Find it and remove it:

systemctl show ollama --property=Environment | grep -i llm_library
env | grep OLLAMA

1. The driver fell behind the runtime. Ollama upgrades bundle a newer CUDA runtime; your driver doesn't move until you move it. Same fix as the driver section above.
2. The service is a container and the flags are gone. A recreated container without GPU flags is a CPU-only container. The NVIDIA invocation is:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

For AMD containers, the equivalent is device passthrough plus group adds:

docker run -d --device=/dev/kfd --device=/dev/dri \
  --group-add video --group-add render \
  -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

No --gpus=all, no GPU — Docker has no reason to be generous.

Does Ollama work in WSL2?

Yes, with the right driver in the right place: install the Windows NVIDIA driver, never a Linux driver inside the distro — an in-WSL driver breaks CUDA passthrough rather than fixing it. Then update WSL itself and confirm the passthrough device exists:

wsl --update   # from PowerShell
ls /dev/dxg    # inside WSL — must exist for GPU use

With /dev/dxg present and a current Windows driver, Ollama in WSL2 offloads to the GPU like a native install. If you'd rather skip the indirection entirely, the Windows build of Ollama runs natively and sees the GPU without WSL.

Does Ollama need a GPU at all?

No — a CPU-only run is functionally identical, just slower, and for small models on a fast CPU it can be perfectly usable. On Apple Silicon the question dissolves: Metal uses unified memory automatically, and the only limit is how much RAM you're willing to share with the model.

The five-minute checklist

Check in this order: ollama ps for the state, server logs for the detection list, then the table. Nine times out of ten the log line already told you which row you're in.

This ships daily at mrsaynothing.dev — the full archive, every piece in 19 languages, zero missed days. New posts land in the newsletter the moment they ship: join it here. Code at GitHub.