Qwen3.8-Flash-Next vs Claude Opus 5.5: Six-way benchmark on a real Rust feature

Originally published at deepu.tech.

I recently posted on my blog and Reddit about my experience pitting Qwen 3.8 Flash-Next against Claude Opus in some tasks and some people on Reddit pointed out that it wasn't a fair comparison since I used different harnesses and thinking effort levels. So I did a more controlled comparison to see how they truly stack up against each other. It's still a non-deterministic comparison, but it provides a clearer picture of how they measure up against each other.

You can read the previous comparison in the blog below and on Reddit.

Forget Claude Code. All you need is Qwen3.8-Flash-Next running locally for agentic coding

Deepu K Sasidharan

Deepu K Sasidharan

Deepu K Sasidharan

Follow

Oct 6

Forget Claude Code. All you need is Qwen3.8-Flash-Next running locally for agentic coding

#ai #qwen38flashnext #localllm #qwen38

8 reactions

Add Comment

15 min read

TL;DR

• All 12 runs finished the task, and all 12 PRs passed the end-to-end test and here are the best averages for each model. The gap is quite thin IMO.
  • Opus 5.5 xhigh: 87%, $17.78, 43 min
  • Qwen3.8-Flash-Next medium, run locally: 69%, $0.32 ($0.02), 45 min
• Flash-Next took 1.1x to 2.1x as long as Opus 5.5 at the same effort (set 2).
• Average score, out of 8:
  • Claude Code + Opus 5.5 - 6.59 (82%)
  • Claude Code + Flash-Next - 5.44 (68%)
  • Pi + Flash-Next - 5.25 (66%)
• With Flash-Next, Claude Code and Pi scored the same on the review questions. CI passed on 4 of 4 Claude Code + Flash-Next PRs and 1 of 4 Pi PRs. Claude Code was also faster in the cleaner set, though Flash-Next had to answer its auto mode tool checks too.
• Opus xhigh scored 6.94 against 6.25 for medium, at 2.7x the cost. Flash-Next medium and xhigh scored almost the same.
• At API prices, a set 2 Opus run cost $6.52 (medium) or $17.78 (xhigh), and a Flash-Next run $0.32 to $0.67, but since I ran it locally, the actual cost was electricity which at my rates is less than 2 cents.

Combinations compared

I wanted to compare both harness differences and thinking effort differences so I decided to set up a six-way comparison with the help of Claude Code + Opus 5.5.

Note: This is Qwen Focused so I'm not gonna burn money testing Pi + Opus combinations.

Each combination ran twice: once where the agent measured save and restore at 100k tokens itself (set 1), and once with those numbers given in the prompt (set 2). Total 12 runs.

Opus 5.5 is a frontier cloud model with more than a trillion parameters, and Anthropic doesn't publish its actual size. Qwen3.8-Flash-Next is a mixture of experts model with 125B parameters, 6B active per token and a 51B n-gram embedding, and it runs on my laptop at ~Q4 quantization.

The Task

I picked a real feature from the LlamaStash roadmap. LlamaStash unloads a model when it needs room for another one or when it has been idle too long, and the next request then has to reprocess the whole prompt. llama-server can save a slot's KV cache to disk and load it back, so the task was to save the cache before an unload and restore it when the model comes back.

The agent had to measure save and restore at 100k tokens first (a spike), design the caps, cleanup and cache matching, prove it works end to end, add tests and docs, and open a PR.

You are in a fresh clone of llamastash at commit <SHA>, on branch <BRANCH>. Do all the work here.

Task (TODO.md, R14 batch 4):
"Keep a model's prompt cache across an unload. When make-room or the idle sweep stops a llama.cpp launch mid-conversation, the next request reprocesses the whole prompt. llama-server can write a slot's KV cache to disk and read it back: --slot-save-path PATH plus POST /slots/{id_slot}?action=save|restore (present on local build 11390 and in upstream tools/server/README.md, checked 2026-10-05). Nothing in src/ uses it. Spike first: measure save time, restore time and file size at a large context (100k tokens) against plain reprocessing. Then decide when to save (before an eviction only), where the files live, how they are capped and cleaned up, and how a restored slot is matched to the returning conversation."

Rules
- Work from scratch. Do not look at any other branch, pull request, clone or worktree.
- Do not ask me anything and do not stop to check in. If something is unclear, pick the reasonable option, note it for the PR, and continue until done.
- Do not use child agents. Do all the work in the main session.

Environment
- llama-server: <LLAMA_SERVER>
- Test model: <TEST_MODEL>
- Run llama-server on CPU only, with 8 threads, and one server at a time.
- Do not use /tmp for anything: it is a RAM disk. Put temporary files, slot save files and daemon directories under <SCRATCH>.
- Any llamastash daemon you start gets its own LLAMASTASH_STATE_DIR, LLAMASTASH_CONFIG_DIR and LLAMASTASH_CACHE_DIR and non-default ports. Do not send test requests to the model endpoint you are running on.
- While coding, run narrow tests only: cargo nextest run --features test-fixtures --lib <filter>, or --test <file>. The full suite runs once, at the end, in the pre-push hook.

Done means
1. Spike numbers at 100k tokens are in a doc under docs/spikes/: reprocess time, save time, restore time, file size.
2. End-to-end proof on an isolated daemon with the test model: send a prompt, let the idle sweep or make-room stop the launch, send the follow-up, and show that the prompt came from the restored cache (cached token count and time, before and after).
3. Unit tests cover the new code. Docs, config example, CHANGELOG and TODO.md are updated as AGENTS.md requires.
4. The pre-push hook passes. If it fails on tests your change does not touch, run it once more; if it then passes, note the flake. Do not use --no-verify.
5. A PR against main is open, titled <BRANCH>, with these sections: Spike numbers, Design decisions (when to save, where files live, cap and cleanup, slot matching), End-to-end proof, Tests, Known limits.

Then stop. Do not wait for CI or poll it; I check CI myself. End with a summary of at most 10 lines.

How I Ran It

An orchestrator session (Claude Code + Opus 5.5) started each run and collected the numbers. Every run got a fresh clone of LlamaStash at the same commit, the same prompt, a 3 hour limit and one "continue" if it stopped early. Claude Code ran in auto mode, which sends its tool checks to the same model, and Pi ran with its guard off. The PRs are in a public scratch repo, deepu105/llamastash-b4r2.

The agents, builds and tests ran on my Dell XPS 15. Flash-Next was served from my ASUS ROG Flow Z13 (Strix Halo, 128GB) by LlamaStash via Halogen, over the LAN.

In the first set, each agent ran the 100k-token spike itself on the XPS CPU, which took 48 to 53 minutes per run. This was a mistake, it should have used the GPU, and by the time I noticed all six runs finished and since quality won't depend on the spike I decided not to redo them. In the second set, the spike numbers were in the prompt and llama-server ran on the GPU.

Scoring

Each PR got seven review questions and a CI check, for a maximum score of 8:

• Q1: Is a restore into a different model or launch config refused?
• Q2: Are the save files capped and cleaned up?
• Q3: Is the eviction code still backend neutral? LlamaStash's AGENTS.md doesn't allow llama.cpp names in generic code.
• Q4: Do the tests run the new code?
• Q5: Are the docs, config example and CHANGELOG updated and correct?
• Q6: Does the PR avoid changes the task didn't ask for?
• Q7: Is the eviction safe? It should check for new requests after the save, and block a second eviction of the same model during the save.
• CI: Do the PR's tests pass on all platforms?

Each answer gets the grade of its worst problem:

For CI, a failing test that the PR added for working code gets 0.75. That includes two tests that hang on Windows, since the hang could be in the test rather than the code.

Four scorings went into the final score. Pi + Flash-Next and Claude Code + Opus 5.5 scored the PRs first. A second Claude Code + Opus 5.5 session checked every place they disagreed in the code and graded each problem by severity. Then Kimi K3 (in GitHub Copilot CLI) scored all 12 PRs on its own, without knowing which run made which PR. The final score is the average of these last two, except where Kimi missed something that was checked in the code. There the Opus grade is used.

Results

Did They Finish?

Yes. All 12 runs opened a PR, and all 12 PRs pass the end-to-end test. After LlamaStash unloads the model, the follow-up prompt gets 15,000 tokens from the restored cache and takes about 1.4 s, against 5.2 to 6.1 s without the feature. One Flash-Next run needed the "continue".

Speed

Set 2 times, where no run had to measure the spike:

In set 1 the gap was 1.9x to 3.5x, even after taking out the time spent waiting for the spike, because some Flash-Next runs built and tested on the same CPU the spike was using. Halogen served Flash-Next at 51 to 56 tokens/s in every run.

Code Quality

Opus wrote better code but not by a huge margin and every PR works.

• Scores are the average of both sets.
• Time, cost and tokens: from the set 2 runs since set 1 data except scores got skewed by the spike.
• Cost: at API prices. Opus 5.5 at Anthropic's list price. Flash-Next at the price of Qwen3.8-Flash, its hosted version on QwenCloud. I ran Flash-Next locally, so it cost me only electricity and Claude was using a 20€ subscription.
• Tokens: from the session logs, without counting tokens used for Claude Code's auto mode tool checks. "in" is input that was not read from the cache.
• CI: 4 PRs fail the Windows test jobs on a test they added: Opus medium #2, and Pi + Flash-Next #3, #9 and #12. The tests in #2 and #9 are wrong (one expects / as the path separator, one uses a helper that does nothing outside Unix). The tests in #3 and #12 hang until nextest kills them after 180 s. Each of these PRs gets 0.75 for CI.
• Acceptance: not scored, since all 12 PRs passed it.
• Diff size: not scored either. The Opus PRs added 1,373 to 2,729 lines, Claude Code + Flash-Next 867 to 1,223 and Pi + Flash-Next 1,439 to 1,962.

What Opus did and Flash-Next didn't

• A request during the save: The Opus PRs check for new requests after the save. The Flash-Next PRs don't, so a request that arrives during the save (under a second) gets cut off.
• Two evictions at once: Only the Opus xhigh PRs block a second eviction while the first one is still saving.
• Cache matching: The Opus PRs key the cache on the full launch command and the model file. The Flash-Next PRs key on less, 3 of them on the model path only, so a relaunch with different flags can restore an old cache.
• Tests: The Opus PRs and 2 Pi + Flash-Next PRs test save and restore through the eviction code. The other 6 Flash-Next PRs have unit tests only.
• Docs: 4 Flash-Next PRs have a doc that describes something the code doesn't do.

What Flash-Next did and Opus didn't

• CI: All 4 Claude Code + Flash-Next PRs passed on every platform. Opus medium #2 failed on Windows.
• SECURITY.md: Pi + Flash-Next xhigh #6 added a SECURITY.md note, since the feature writes conversation tokens to disk and turns on llama-server's slot endpoints. Opus #2 and #11 didn't.
• Tests: Pi + Flash-Next #3 and #12 scored a bit higher on tests than the Opus medium PRs #2 and #8, which don't test their own check for requests that arrive during the save.
• Scope: Claude Code + Flash-Next #5, #10 and #13 made no unrelated changes. Opus xhigh #4 left an unused function behind.
• Smaller PRs: The Claude Code + Flash-Next PRs added 867 to 1,223 lines, against 1,373 to 2,729 for Opus. Not scored.
• On by default: 7 of 8 Flash-Next PRs turn the feature on by default, against 2 of 4 Opus PRs. That saves a config step, but I didn't score it either way.

So Opus was ahead on the parts that change what a user gets: cache matching and eviction safety. Flash-Next did better only in some PRs, on CI, docs, tests and scope.

Claude Code vs Pi for Flash-Next

This was surprising for me, since Pi is my daily driver. Both scored the same on the review questions, and Claude Code was ahead on CI.

Claude Code also sent its auto mode tool checks to the same Flash-Next server, 22 to 79 per run, which took 10 to 14% of the run time. Pi sent none. Claude Code + Flash-Next was still faster than Pi in both set 2 pairs (by 6% and 22%). In set 1, Pi was faster in both (by 18% and 3%).

Three of the 4 Pi PRs fail the Windows test jobs, and no pre-push hook covers Windows. With one run per combination per set, I wouldn't read too much into it.

Medium vs xhigh

For Opus, xhigh scored higher in both sets (6.63 against 6.0, and 7.25 against 6.5), and only the xhigh PRs block two evictions at once. But it cost 2.6x to 2.7x as much and took 43.5 minutes against 23.4 in set 2.

For Flash-Next, medium and xhigh scored almost the same (5.28 and 5.41), and xhigh was slower in all 4 pairs. I have been running Flash-Next at xhigh, and on this task it didn't help.

The Reviewers

Flash-Next, Opus and Kimi K3 all scored these PRs, so they can be compared as reviewers too:

• Flash-Next found the "request during the save" problem in 2 of the 8 Flash-Next PRs and no llama.cpp names in generic code. Opus found the first in all 8 and the second in 6.
• Opus missed that 3 PRs left their new config key out of the README.
• Kimi K3 gave the same grade as Opus on 63 of 84 answers and was never stricter. It missed the four docs that describe the wrong behaviour.

In the previous post, I said Flash-Next's reviews found more issues than Opus's. On this task it was the other way around.

Caveats

• Biggest caveat is that the models are not on the same league. Opus is a frontier model run at FP16 or atleast FP8. I ran Flash-Next on my laptop at 4.16 bpw (experts and the rest: 4-bit, the mixing layers: 6-bit, draft head: 8-bit, n-gram lookup table: FP8). Flash-Next at FP8 should show a bit more quality.
• One run per combination per set, so I don't know how much the results vary from run to run.
• The end-to-end test used a 1B model at 15k tokens.
• The two save-race problems were found by reading the code, not reproduced.
• The sets differ in the prompt (do spike vs spike data given) and the llama-server build (CPU against GPU).
• All the scorers are LLMs, and the session that settled their disagreements ran on Opus.
• The machine running Flash-Next ran at 70W TDP ceiling for the entire duration. I could have run it at 90W for slightly better speed but the fan noise and thermal stress was not worth it.

Final Thoughts

With the harness and effort kept the same, Opus 5.5 wrote better code than Flash-Next on this task and finished sooner. This was expected of course from a frontier model. That's different from my previous post, where I liked Flash-Next's PR better than the one from Opus at medium effort.

But Flash-Next was right behind in scores and time, and the gap wasn't that wide. Opus is a frontier model on Anthropic's servers, and Flash-Next has 6B active parameters and runs on my laptop. It finished every run, every PR works, it took 1.1x to 2.1x as long, and it cost me only electricity. What it missed (the save race, cache matching, integration tests, docs) is what a review catches, and that is how I already use it: an orchestrator plans and reviews, and Flash-Next does the implementation.

Two things I'll try in my work. I'll run Flash-Next at medium instead of xhigh, and I'll give Claude Code + Flash-Next a proper try instead of using only Pi, and see how both go.

All 12 PRs are in deepu105/llamastash-b4r2 if you want to read the code yourself.

Here is the entire setup if you want to run a similar bench yourself. Do post results if you do: https://gist.github.com/deepu105/c3826003c9e4d59583bf94536dd652c7

If you like this article, please leave a like or a comment.

You can follow me on Bluesky, Mastodon, and LinkedIn.