Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads

This article follows Comparing Qwen and Gemma on the Low End, which ran Qwen3.5-2B and Gemma 4 E2B with thinking off. Here both models think, on the same card, the same llama.cpp binary and the same 4-bit files.

https://github.com/xbill9/qwen-dev/tree/main/local-llamacpp-1650ti-qwen352b-q4_0

Thinking is worth 17.7 points to Qwen3.5-2B on BFCL tool calling and nothing measurable to Gemma 4 E2B. On GSM8K math, thinking alone adds 3.7 points to Qwen, a range that includes zero. With both models thinking, Gemma still leads by 10.3 points on math and 5.0 on tool calls. Qwen's thinking runs to a median of 7,607 tokens per math answer against Gemma's 808.

What is this project trying to Do?

The first article found Gemma 4 E2B ahead of Qwen3.5-2B on every task, with thinking off for both. Qwen3.5 is built to think, so the first question a Qwen user asks is what happens with thinking on.

This article answers it on the GTX 1650 Ti, a 4 GB laptop card, for the two generation tasks: GSM8K math word problems and BFCL simple tool calls. Both models run as 4-bit GGUF files: Qwen's standard Q4_0 and Gemma's QAT Q4_0.

At This Point You Should Have…

• An NVIDIA GPU with a CUDA driver — here a GTX 1650 Ti (4 GB, Turing)
• llama.cpp built with CUDA — commit 7182ef71e here
• The two GGUF files from the first article: Qwen3.5-2B-Q4_0.gguf and Gemma's gemma-4-E2B-it-q4_0-exact.gguf
• Python 3 for the task runner — the system python3, no virtualenv
• The repository cloned, and jev-tpu-v5e1/ holding gen_eval.py

Step 1 — Turn Thinking On

One server flag turns thinking on for both models:

llama-server -m Qwen3.5-2B-Q4_0.gguf --host 127.0.0.1 --port 8080 -ngl 99 \
  -c 131072 --parallel 4 -ctk f16 -ctv f16 -fa 1 -t 6 -tb 12 --reasoning on --metrics

llama-server puts the thinking in message.reasoning_content and the answer in message.content. Only content is scored; the task runner keeps the thinking in each record for reading later.

-c 131072 --parallel 4 gives each of 4 slots 32,768 tokens, so a request has room to think. The server log confirms the split:

0.01.412.127 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 32768, kv_unified = 'false'

Step 2 — Use Each Vendor's Sampling Settings

The thinking-off runs used greedy decoding. With thinking on, greedy decoding breaks Qwen. On 16 GSM8K problems with an 8,192-token limit:

trial-greedy-8k/gsm8k-qwen.jsonl: n=16 limit=8192 right=5 hit_limit=11 ends_in_loop=9 still_thinking=11

Qwen got 5 of 16 right. It hit the limit on 11, every one still thinking, and 9 of those ended in one sentence repeated dozens of times:

    Wait, I need to check if the prompt requires the thinking steps to be in the final output. Yes.
    Okay.
    Wait, I need to check if the prompt requires the thinking steps to be in the final output. Yes.
    Okay.

Qwen's model card gives settings for thinking mode, and Gemma's generation_config.json gives its own:

Qwen3.5-2B, thinking mode for text tasks: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
Gemma 4 E2B: "temperature": 1.0, "top_k": 64, "top_p": 0.95

Each model ran at its own vendor's settings. gen_eval.py merges them into every request from one variable:

JEV_SAMPLING='{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}' \
JEV_GEN_MAX_TOKENS=32000 python3 gen_eval.py run http://127.0.0.1:8080 Qwen/Qwen3.5-2B gsm8k gsm8k-qwen.jsonl \
  --ids gsm8k-300-ids.txt --concurrency 4

Step 3 — Give Thinking Room

Qwen's card recommends an output length of 32,768 tokens for most queries. Both models got a 32,000-token limit on both tasks.

At that length Qwen's math takes hours on this card, so GSM8K is a fixed random 300 of its 1,319 problems, drawn once with a fixed seed and used for both models. BFCL is all 400. With thinking off, the 300 score Qwen at 0.730 and Gemma at 0.927, against 0.734 and 0.901 on all 1,319: the sample matches the full set for Qwen and favours Gemma by 2.6 points. Gemma's math leads below are about 3 points larger than the full set would give: with thinking off its lead is 16.7 points on all 1,319 and 19.7 on the 300.

Step 4 — Score Both Models

Each model with thinking on is paired question by question against its own thinking-off records on the same 300 problems and 400 requests, with a 95% range from 10,000 resamples:

• Qwen rises 9.7 and 15.0 points, both ranges clear of zero.
• Gemma is level on math and loses 3.0 points on tool calls. Gemma with thinking off still beats Qwen with thinking on.
• On BFCL, Qwen now calls the tool. Replies with no tool call fall from 88 of 400 to 8. Its misses are now mostly wrong argument values.

Step 4 changed two things at once: thinking, and greedy decoding to each vendor's sampling. Step 5 separates them.

Step 5 — Separate Thinking From Sampling

One more run holds the sampling and turns thinking off: the same server, files, limit, problems and settings as Step 4, with --reasoning off. The greedy runs kept the first article's 2,048 and 512-token limits, and the sampled runs had 32,000, so the first step changes the sampling and the limit together. Three runs per model, compared in pairs:

• On tool calls, thinking is the whole of Qwen's gain: +17.7 points. With thinking off, Qwen still answers 83 of 400 in prose; with thinking on, 8.
• On math, thinking adds 3.7 points to Qwen, a range that includes zero. The other 6.0 points come with the sampling and the longer limit together. Greedy, 26 of 300 replies reached the 2,048-token limit, 24 of them with no loop at the end; with the card's settings and 32,000 tokens, 4 of 300 reached the limit. Scored as right, those 26 would put greedy at 0.797, level with the sampled 0.790.
• Gemma's 3-point BFCL loss is the sampling. Gemma's greedy replies stayed well inside both limits (longest 1,586 tokens on math, 92 on tool calls), so for Gemma that column is sampling alone. Thinking changes neither of Gemma's scores beyond its range.

At matched settings Gemma leads by 12.0 and 22.5 points with thinking off, and by 10.3 and 5.0 with thinking on.

Step 6 — Count What Thinking Costs

From the records and the task runner's own wall times, 4 requests at a time:

qwen gsm8k: n=300 right=248 wall_s=36269.8 wall_min=604.5 wall_h=10.07 s_per_right=146.2 tokens_total=3407907 tokens_per_right=13742 tokens_median=7607
qwen bfcl: n=400 right=335 wall_s=1456.4 wall_min=24.3 wall_h=0.40 s_per_right=4.3 tokens_total=109946 tokens_per_right=328 tokens_median=160
gemma gsm8k: n=300 right=279 wall_s=1872.6 wall_min=31.2 wall_h=0.52 s_per_right=6.7 tokens_total=270970 tokens_per_right=971 tokens_median=808
gemma bfcl: n=400 right=355 wall_s=949.5 wall_min=15.8 wall_h=0.26 s_per_right=2.7 tokens_total=117153 tokens_per_right=330 tokens_median=284

On math, Qwen spends about 14 times Gemma's tokens and 22 times its time for each right answer. On tool calls the two spend the same tokens.

31 of Qwen's 300 math replies used all 32,000 tokens; 20 of them were still thinking and 21 ended in repeated text. Scoring all 31 as right gives Qwen 0.913, still below Gemma's 0.930.

🔎 Tip: Do Not Run Qwen Greedy

Greedy decoding is the usual setting for repeatable scores. With thinking on it loops Qwen: 9 of 16 math problems ended in repeated text, and 5 of 16 were right against 12 of 16 at the card's settings. Use the card's settings, and keep presence_penalty in them; it is the one aimed at repetition.

🔎 Tip: The Token Limit Covers the Thinking

max_tokens counts thinking tokens as well as the answer. A limit sized for thinking-off answers (512 for tool calls, 2,048 for math here) cuts a thinking model off mid-thought, with an empty content. Size the limit, and each slot's share of -c, for the thinking.

Compare and Contrast

So, Which One?

Gemma 4 E2B, with thinking off and greedy decoding. It scores higher than Qwen with thinking on, on both tasks, and answers math in a fraction of the time.

If you use Qwen, use its card's sampling settings, and turn thinking on for tool calls: it is worth 17.7 points there for few tokens. On math, thinking adds 3.7 points that may not be real, at 14 times Gemma's tokens per right answer. Leave Gemma's thinking off for these tasks; it gains nothing.

Summary

The goal of this article was to measure Qwen3.5-2B and Gemma 4 E2B with thinking on, where the first article ran them with thinking off. The key to the solution was running each model at its vendor's thinking-mode sampling with room to think, and a third run with the same sampling and thinking off, so that thinking's effect is measured with everything else held, question by question. The results were:

• 🟢 Thinking lifts Qwen on tool calls: +17.7 points on BFCL, with replies that skip the tool down from 83 to 8.
• ⚠️ On math, thinking adds little to Qwen: +3.7, a range that includes zero; the other +6.0 came with the card's sampling and the longer limit together.
• ❌ Thinking does not lift Gemma: +2.0 on GSM8K and +0.3 on BFCL, neither clear of zero.
• 🟢 Gemma still leads: by 10.3 and 5.0 points with both thinking, and by 10.0 and 8.0 with only Qwen thinking.
• ⚠️ Qwen's math thinking is expensive: 13,742 tokens and 146 seconds per right answer, against Gemma's 971 and 6.7.
• ⚠️ Greedy decoding loops Qwen's thinking; its card's sampling settings fix it.

Scope: one GTX 1650 Ti (4 GB), llama.cpp 7182ef71e, the same Q4_0 files as the first article, measured 2026-10-18 and 2026-10-19. Two tasks: GSM8K on a fixed random 300 of 1,319 problems and BFCL simple on all 400; the classification suite was not run with thinking on. The two sampled runs used each vendor's thinking-mode sampling and a 32,000-token limit, 4 requests at a time, once per model; the greedy thinking-off records are from the first article's runs, with 2,048 and 512-token limits, 8 at a time. Thinking with greedy decoding was run on 16 problems only. Each sampled setting was run once, and a re-run gives different text, so the 95% ranges cover the choice of questions but not run-to-run variation. With --reasoning off, Gemma still wrote reasoning on 26 of 400 sampled BFCL replies, 25 of them right; the greedy runs have none. The task prompts and scoring code were written for this repository's Gemma work.

The strategy for measuring thinking on Qwen3.5-2B and Gemma 4 E2B on a local GPU was validated with an incremental step by step approach.

References

• Comparing Qwen and Gemma on the Low End | dev.to
• local-llamacpp-1650ti-qwen352b-q4_0 | GitHub
• Qwen/Qwen3.5-2B | Hugging Face
• google/gemma-4-E2B-it | Hugging Face
• xbill9/gemma-4-E2B-it-qat-q4_0-exact-gguf | Hugging Face
• llama.cpp | GitHub