Flash Onyx 2.1, one day later: my model spent 400 tokens thinking and returned an empty string

Yes, I posted about Flash Onyx 2
yesterday. I know. I'm back already.

Not because I found a typo. Because somebody said "it takes forever to answer"
and I went looking for why, and what I found was bad enough that it needed a
version bump rather than a patch note.

Short version: the system prompt I spent so long writing was the problem, and
the model was sometimes returning literally nothing.

The complaint

Onyx 2 runs on Gemma 4 through Ollama, 12B on consumer hardware. The report was
just that replies took minutes. Not that they were wrong. Just slow enough that
you would tab away and forget about it.

My first instinct was the output budget. num_predict was set to 8192, which
felt generous rather than reckless. Then I actually measured the generation rate
on the 12B and got about 9.5 tokens per second.

8192 divided by 9.5 is 862 seconds. Fourteen and a half minutes. The budget I
picked to be safe was a fourteen minute worst case, and I had never done that
division.

That was not the real problem, but it was a good sign I had not measured
anything.

The real number

I put one request through the API and read the timings instead of the text:

prompt_eval_count      27000ish
prompt_eval_duration   ~290s
eval_count             40

Two hundred and ninety seconds to read the prompt. Before generating a single
token of an answer.

My system prompt was 903 lines and about 27,000 tokens. The context window is
32K. So the model was starting every conversation with roughly 5K of room left
for the actual conversation, and paying nearly five minutes to load its own
instructions.

I had written a beautiful document explaining that the model should be fast.

The part that made me put my head on the desk

While measuring, I noticed something worse. Some responses came back as an empty
string. Not an error. done_reason: length, eval_count: 400, and
response: "".

Four hundred tokens generated. Zero characters out.

Gemma 4 has a separate thinking channel, and Ollama only surfaces it if you ask.
I asked. Here is what my carefully engineered agent was doing with its budget:

*   User pushed a commit to `main` containing an AWS secret key.
    *   Constraints: Flash Onyx 2 persona (fast, direct, few words,
        no fluff, no em-dashes).
    *   Primary action: Rotate/Revoke the key in the AWS Console.

It was spending its entire reply budget reasoning about its own personality.
Then hitting the cap. Then returning nothing.

My prompt contains the line "Never deliberate about tone, length, or word
choice." It was deliberating about tone, length, and word choice, and it never
got far enough to tell anyone.

Same question with the thinking channel off:

That is a request level flag, not a Modelfile setting, so if you are building on
a reasoning model through Ollama, go check what your client sends. Mine was
paying for reasoning it then threw in the bin.

Cutting 903 lines to 516

The compression itself was less clever than it sounds. Mostly it was admitting
that I had written the same rule four times in four sections, and that a lot of
sentences existed to sound good rather than to change behaviour.

Before:

Never assume anyone can answer a prompt for you. Take the non-interactive path
every time: pass -y, --yes, --noconfirm, --no-pager, and supply every
argument up front, because anything that waits on input can hang until it
times out. Pagers, confirmation prompts, REPLs, editors, -i flags, and a
missing required argument are all that same trap.

After:

Never assume anyone can answer a prompt. Take the non-interactive path: -y,
--yes, --noconfirm, --no-pager, every argument up front. Pagers, REPLs,
editors, -i flags, and a missing required argument all hang until they time
out.

Same rules, roughly half the words. 903 lines to 516, 27,000 tokens to 14,200,
prefill from about 290 seconds to about 152.

Then it got worse

Here is the bit I did not expect. I built the shorter version, ran it against
Onyx 2 on seven prompts covering the things it is supposed to be good at, and
lost four of them.

The new prompt had a section telling the model to be brief. It followed the
instruction and became shallow.

Asked to write a Slack message about a slipped deploy, Onyx 2 said:

Hey team, just a heads up that Thursday's deploy is pushed to Monday. The
staging DB migration failed, so we need the extra time to get it sorted.

2.1 said:

Thursday's deploy moved to Monday since the staging DB migration failed.

Technically shorter. Also not a message to a human being, it's a log line.

Same story in a negotiation: 2.1 gave a tight answer that immediately conceded
to the user's own maximum, while the older, wordier model worked out the
landlord's turnover costs and offered a trade. Short is not the same as good,
and I had written a prompt that could not tell the difference.

The fix went into the brevity section itself:

Compression is words, never substance. Dropping a step, a caveat that changes
the answer, a trade worth offering, or the manners a human message needs is
not brevity, it is a worse answer that happens to be short.

The lesson I actually didn't know

One test refused to move. The negotiation answer came back byte identical across
three rounds while I rewrote the negotiation rules twice.

The section sits around line 460 of 516. Deep in the middle. I moved a one line
version of the same rule to the very end of the prompt, changed nothing else,
and the answer changed immediately and included the trade it had been missing.

Position beat wording. Two careful rewrites in the middle did nothing, one blunt
sentence at the end did the job. If you are tuning a long system prompt and a
change appears to do nothing, try moving it before you try rewording it again.

Where it ended up

Four rounds of this, same seven prompts each time. Final tally: 2.1 won six,
tied one, lost none, and its answers came to 70% of Onyx 2's length. It was
shorter on all seven.

The honest caveats, because this is not a benchmark: seven short prompts, one
run each, fixed seed, hand graded by me against my own standards, 12B only. I
never tested the 31B. Treat it as a smoke test that caught real regressions, not
a leaderboard.

Two small things I enjoyed. Onyx 2 broke its own no em-dash rule in one of the
seven answers and 2.1 broke it in none. And 2.1 now knows the difference between
code and a one-off, which Onyx 2 did not:

# "quick one, move the March files"
mv ./dumps/report_2024-03-*.csv ./march/

The first version of my throwaway scripts section made that worse, by the way.
It produced a 663 character bash script with set -euo pipefail, a guard loop
and mkdir -p, because I had written a lot of words about skipping ceremony and
never actually said "if one command does it, that is the answer." Adding that
sentence took it to 202 characters.

If you want it

ollama run Natuworkguy/flash-onyx-2.1:12b

Flash is at github.com/Natuworkguy/Flash.
If you missed it, yesterday's Onyx 2 post
covers what the model actually does, and all of that still applies here.
Onyx 2 is still there and still builds, and honestly if you are on a machine
where prefill is free it is a fine model. On mine it was reading a novel to
itself before every conversation.

The thing I keep coming back to is that I could have found all of this on day
one by reading the timing fields in the API response instead of the text. Four
numbers. They were there the whole time.