Why We Kept Ternary Bonsai 2 27B After Testing the Faster 9B

October 16, 2026

Our production classifier runs on a single RTX 2080 Ti with 11 GB of VRAM. It is often busy enough that requests spill over to a hosted model. When PrismML released Ternary Bonsai 2 9B on October 15, we had a practical reason to try it: its GGUF file is 2.25 GiB, against 6.70 GiB for the 27B we already use.

We replayed 1,015 production decisions. The 9B was 2.4–3.6× faster in our laptop benchmarks, but it downgraded 15 of 65 escalations that a second model had confirmed. On another task, 43% of its answers contradicted an application rule.

That was enough to rule out a straight swap. Here is what we tested and where it broke.

What we needed to improve

The 27B runs in llama.cpp with PQ2_0 weights, two parallel slots and a shared 32k context. Reasoning is disabled; responses use strict json_schema decoding at temperature 0.

Requests have an 8-second client deadline. When the gateway cannot offer a slot within that budget, it rejects the request and the application uses a hosted fallback. Over one 20-hour window, we logged 9,081 requests:

At the busiest hour, about 30% overflowed to the hosted model. Served requests took a median of 4.3 s and a p95 of 8.1 s of model time. A faster model could reduce both waiting and fallback costs.

The application makes three judgements about informal, non-English posts. Each request includes the current post and up to 11 earlier posts in the thread. Median prompt lengths range from about 0.7k to 2.3k tokens; responses are capped at 64–96 tokens of JSON.

Task A decides whether we can make the switch. We call an escalation “confirmed” when the hosted model agrees with the 27B. This gives us a useful reference, though it is still model agreement rather than expert ground truth.

How we tested

We sampled logged decisions from the current prompt versions, keeping rare outcomes well represented:

To estimate production rates, we weighted each case by its sampling and logging probabilities:

weight = (rows in stratum / rows sampled) × (1 / logging probability)

The Task A sample represents about 68k production checks.

We rebuilt requests with the application's own prompt builders, using stored posts and thread windows, the same schemas, token limits and temperature. Prompt token counts matched production in all 275 cases we checked.

The replay ran on an M1 Max under llama.cpp with the production server flags. The 27B used our production build, prism-b10709; the 9B required prism-b10777.

Since we already had the 27B's logged answers, we reran it on only 275 cases: a random control subset and every Task A and C disagreement. It reproduced 273 of 275 production decisions. The two borderline differences may reflect Metal/CUDA numerical differences. This check gave us confidence in the replay, without requiring a second full run.

Task A: fewer clear decisions, missed escalations

All outputs were valid. The problem was the decisions inside them.

The 15 downgraded confirmed escalations mattered most. The 27B reproduced all 15 as escalate in the local rerun. Twelve had its most severe, explicit subtype; three had the figurative-language subtype. The 9B assigned all 15 a low-severity subtype with self-reported confidence of 0.85–0.9. A confidence threshold would not have caught them.

We also read those cases ourselves. About 11–12 looked clearly positive, while 3–4 were borderline. Some stated the critical signal directly; others used metaphor or idiom. This was an informal review, not expert annotation.

The broader shift was toward clarify. Reweighted to production volume:

Weighted agreement was 61.2%. The 9B never moved an escalation straight to allow, but it moved many cases from both ends into the middle. Since clarification has a cost in our application, that would affect a large share of users.

It also doubled the weighted escalation rate despite missing confirmed cases. New escalations from the clarify stratum alone would add roughly 180 hosted-model reviews over the logged period.

Task B: valid JSON, contradictory answers

In 151 of 350 answers (43%), the 9B returned “intervention needed: true” alongside “continue as normal.” The application rejects that combination.

Our JSON schema permits it; the prompt tells the model not to use it. The 27B followed that instruction in our control runs. The 9B frequently did not. Constrained decoding kept the JSON parseable, but could not enforce a relationship we had left out of the schema.

Even among valid answers on positive cases, agreement was only 23 of 47. After reweighting, about 17% of all Task B checks would fail validation and go to the hosted fallback, adding to the load we wanted to reduce.

Task C: better agreement, limited evidence

The 9B agreed on 91 of 103 cases the 27B had flagged or marked urgent (88.3%). Disagreements included five flagged cases dropped to none and three urgent cases dropped to flagged.

We had only five none cases, too few to estimate false positives. This task gave us less reason for concern, but also much less evidence.

How much faster?

We ran llama-bench on the M1 Max with flash attention enabled. We also tested the 27B on the newer runtime to check whether the build itself explained the gain.

Against our pinned 27B build, the 9B was 3.6× faster on prefill and 2.4× on generation. Updating the runtime alone made little difference.

The production 2080 Ti measures 749 tok/s for prefill and 42.6 tok/s for generation with the 27B. Applying the laptop speed ratios suggests checks might take about 1.5–2 s instead of 4.3 s. We have not measured the 9B on that GPU, so this remains an estimate.

What we would try next

This test answers whether the 9B can replace our 27B with the existing prompts. It does not tell us the best the smaller model can do. We tuned those prompts for the 27B, disabled reasoning on both models and tested one language in one domain. We also did not run full-precision Qwen3.5 9B, so we cannot separate model-size effects from ternary quantisation.

The first change worth testing is Task B's output schema. Merging the boolean and action into one enum could make the contradictory combination impossible. We would then tune the prompts for the 9B and replay the same cases. A small reasoning budget might help too, at the cost of some speed.

Another possibility is to use the 9B as a first pass for Task A, sending its non-allow answers to the 27B. On this sample, that would skip the larger model for about half of production checks, letting through roughly 6% of the 27B's clarify cases and none of its escalations. We have not tested this as a pipeline. Keeping both models on an 11 GB card would be tight, and the second hop would add latency.

For now, the 27B stays. Replaying production traffic made that decision straightforward: the speed gain was real, but so were the missed escalations. Any replacement gets the same test, with no misses on confirmed positives and no increase in invalid answers.