A 7 GB 27B Model Lost to My 17 GB Default

A new 27B model landed on my RTX 5090 in a 6.66 GiB GGUF file. It generated 123 tokens per second. It answered all three factual checks correctly. I still did not make it my default.

The short answer: Ternary Bonsai 27B proved that a large model can fit in a small memory budget, but fit did not decide the job. My local test favored gemma4:26b because it generated faster and followed the requested output format. File size gets a model onto the machine. Task completion keeps it there. Canonical URL: https://bmdpat.com/blog/ternary-27b-vs-gemma4-26b-local-ai-2026

How did a 27B model fit in 6.66 GiB?

I tested the official Ternary-Bonsai-27B-Q2_0.gguf on July 16, 2026. llama-bench reported a 6.66 GiB file and 26.90 billion parameters. The model ran with 99 GPU layers on an RTX 5090 at a 4,096-token context.

The compact file comes from ternary weights. Instead of storing ordinary high-precision weights, the model uses values of negative one, zero, or positive one, plus shared scale factors. The official model card describes a roughly 7.2 GB deployed footprint for the language model. My downloaded file measured 7,165,121,600 bytes.

That is the first win. A 27B-class model that fits under 8 GB opens options for laptops, smaller GPUs, and multi-model servers. It also makes model size a poor proxy for task quality.

Was the smaller file fast on an RTX 5090?

Yes, with a caveat. Bonsai generated 123.37 tokens per second in three llama-bench repetitions. Prompt processing measured 3,885.47 tokens per second on the same run. Those are my local measurements, not vendor estimates.

My existing gemma4:26b default generated the same factual prompt at 179.95 tokens per second through Ollama. That comparison crosses two runtimes, so I treat it as a job-level spot check, not a clean model benchmark. It was still enough to answer the operator question: which model should keep the drafting slot on this machine?

The answer stayed Gemma. Bonsai was compact and fast enough to serve. It was not faster for the task I needed.

Why did instruction following decide the winner?

I asked both models three factual questions and required one exact FINAL line. Both models found Canberra, Ursula K. Le Guin, and Python 3.7. Bonsai put the correct answers in its reasoning, then used the 384-token cap before it emitted the required line. Gemma returned the exact line.

That difference matters in an agent loop. A correct answer in hidden reasoning is not the same as a parseable result. If the next step expects one line, the model that emits that line finishes the job. The model that explains past the cap creates a retry, a repair step, or a manual check.

This is why I separate model intelligence from model usefulness. Benchmarks can tell me whether a model has capability. My verifier tells me whether it completed my task.

What should a local model selection test measure?

Start with fit, but do not stop there. Record the actual loaded footprint at the context you plan to use. Context and cache settings can change the memory total after the weights fit.

Then measure the real output path. I care about generation speed, cold-load behavior, exact format compliance, and whether the result passes a deterministic check. My Q4_K_M versus Q5_K_M comparison uses the same idea: the quant label is an input, not the verdict.

Test the workload at its natural length too. My 8B long-form failure test showed that a quick answer can hide a length-control failure. A model that wins a short prompt may lose a full draft.

For each candidate, I now want one row with five fields: loaded memory, warm generation speed, cold-load time, verifier result, and failure reason. The default slot goes to the model with the best completed-task result, not the smallest download.

When would I still choose Ternary Bonsai 27B?

I would test it again on a memory-constrained machine or a server that needs several resident models. A 6.66 GiB file leaves much more room than my 17 GB Gemma default. That can matter more than raw speed.

I would also run it as a persistent server. My captured Bonsai load took 68.99 seconds, and other launches from the Windows E: drive took longer. Paying that cost for every request would erase the value of fast generation.

The model earned a place in the sizing data. It did not earn the default drafting job. That is a useful result. A good model test should be allowed to end with "fits, runs, does not replace."

Accompanying prompt

What the prompt does: It turns a local model comparison into a task-based keep, test, or reject decision.

Copy/paste this prompt:

Role:
You are a local LLM model-selection reviewer.

Context:
Paste the GPU, runtime, model files, context size, benchmark output,
task prompt, expected output format, and verifier result.

Task:
1. Compare loaded memory and cold-load time.
2. Compare warm generation speed on the same task.
3. Check exact output compliance and verifier results.

Output:
- A table with fit, speed, compliance, and failure reason.
- One decision: keep, test again, or reject for this job.

Constraints:
- Separate local measurements from vendor claims.
- Do not compare different runtimes as a clean model benchmark.
- Do not invent missing measurements.

Copy the block above.

I publish measured local model decisions in The 5090 Reports. Join the email list for the next result.

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=ternary-27b-vs-gemma4-26b-local-ai-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.