Qwen3 4B vs 0.6B on a 6 GB GPU: Speed, Memory, and Answer Quality

For my personal retrieval-augmented generation (RAG) study, I needed a local model to answer questions from documents. My laptop has an RTX 3060 Laptop GPU with 6 GiB of VRAM, so memory use mattered alongside speed.

I compared qwen3:4b and qwen3:0.6b using Ollama. The smaller model generated tokens roughly three times faster. The larger model passed all six document checks, while the smaller one passed four.

That gave me a practical starting point for the lab. Here is what I measured and what the failures showed.

The setup

The experiment ran on October 20, 2026, with a Ryzen 7 6800H, approximately 15 GiB of usable system RAM, and Ollama 0.40.6.

Both models used Q4_K_M quantization, a 4,096-token context, temperature 0, seed 42, and thinking enabled. I loaded one model at a time and warmed it up before measuring three short prompts and three document-heavy prompts.

The short workload requested a long list of document-organization tips. The document workload supplied 60 synthetic workspace records. Both workloads reached a 256-token output cap, so they measured generation throughput rather than completed answers.

Separately, six synthetic document questions tested lookup, exceptions, policy updates, multiple conditions, arithmetic, and missing information. Each question ran once per model with a 2,048-token output budget, including reasoning. Scoring compared parsed JSON against the expected answer; Markdown code fences were accepted. No schema-constrained decoding was used.

The measured tradeoff

Generation rates are medians of three warm requests per workload and include both reasoning and final-answer tokens. They do not describe how quickly a complete answer appears.

Both models reported their full model allocation in VRAM at the tested context. The memory figures above are sampled whole-GPU peaks, including other GPU activity; they are not model file sizes.

The full report contains every measured request and document answer.

What the smaller model got wrong

The first failure involved combining requirements. A pilot required an EU region, enabled backups, and at least 100 active users. Only Aster satisfied all three conditions.

Expected answer, also returned by 4B:

{"workspaces": ["Aster"]}

The 0.6B model returned:

{"workspaces": ["Aster", "Cedar", "Dune"]}

Cedar had backups disabled. Dune had only 60 active users. Both names were present in the supplied documents, but neither met every requirement. This was an answer-correctness failure.

The second failure involved missing information. The documents mentioned phone support but specified no guaranteed response time. The requested answer was:

{"hours": null, "sources": []}

The 4B model returned that structure. The 0.6B model returned:

{"hours": [null, null], "sources": [null, null]}

This was an output-format failure: arrays appeared where a scalar null and an empty source array were required. It did not invent a response-time guarantee. That distinction matters when diagnosing what an application needs to validate.

A thinking-mode mismatch changed the protocol

Preliminary attempts tried to establish a shared non-thinking baseline. The installed 4B tag continued emitting reasoning, exhausting the short output budget before some answers completed. Its metadata advertised only thinking enabled, while the installed 0.6B tag advertised both settings.

I excluded those attempts from the final comparison. The recorded run explicitly enabled thinking for both models and increased the document-check budget. All six document answers completed for each model without truncation.

This observation applies to the installed tags and runtime used here. It does not establish that every Qwen3 4B release behaves the same way. The repository preserves the excluded results and their explanations.

Run the comparison yourself

The repository includes a reusable runner using Python's standard library. It requires Linux, Python 3.10 or newer, local Ollama, and nvidia-smi for GPU telemetry.

Clone it and download the models:

git clone https://github.com/rafaself/lab-ollama-local-benchmarks.git
cd lab-ollama-local-benchmarks
ollama pull qwen3:4b
ollama pull qwen3:0.6b
ollama ps

With Ollama serving locally and no models loaded, choose an unused results directory:

python3 benchmarks/compare_models.py \
  --output runs/local-qwen3-comparison-001/results.json

The runner tests models sequentially and unloads them afterward. To check the published evidence without running inference:

python3 benchmarks/validate_run.py runs/2026-10-20-qwen3-4b-vs-0.6b

Model tags can change. Compare installed digests with the recorded configuration when attempting to reproduce this baseline.

What I would use next

This was an exploratory experiment on one laptop. Six questions cannot establish general accuracy. Model order was fixed, normal prompt caching remained enabled, and temperature, power policy, and desktop activity were uncontrolled. Embeddings and retrieval were not evaluated: every document question included its passages directly. The methodology documents these limits.

For this lab, 4B is my main answer-generation choice: it fit entirely in GPU memory and passed these checks. I would use 0.6B for quick iteration where outputs are easy to inspect. A useful next experiment would expand the document questions and alternate model order across independent sessions before making broader claims.