Hands-On: Qwen3.8-Flash-Next and a Glimpse into Qwen4's Design Philosophy — llama.cpp Support Status and A100 Benchmarks

Out of nowhere, on August 26, 2026, the Qwen team posted an announcement on ModelScope and Hugging Face: a new model, Qwen3.8-Flash-Next, would be released at 00:00 on August 27. The community was still buzzing over how strong the Qwen3.8 series already was, and the accompanying note describing it as a "next-generation architecture heading toward Qwen4" only added to the excitement.

What exactly is this next-generation architecture? As it happened, I was in the middle of researching hybrid attention structures myself. New techniques keep arriving one after another, so I decided to chase this one down right away.

This article first walks through Qwen3.8-Flash-Next's architecture, then covers its support status in llama.cpp and hands-on benchmarks on an NVIDIA A100 80GB. In the latter half, I also look at real logs from running it through an agent, and cover additional verification of the MTP feature that was released partway through writing this piece.

Model Architecture of Qwen3.8-Flash-Next

Overall Structure, Specs, and New Elements

Qwen3.8-Flash-Next is a model that gets an early application of the structure expected to carry over into the next-generation Qwen4.x.

Figure 1: Architecture diagram of Qwen3.8-Flash-Next

The model overview is as follows.

• Parameter count: 125B (main) - 51B (n-gram embedding) - 4B (MTP) - A6B
• Hidden dimension: 2,560
• Vocabulary size: 248,320
• N-gram Embedding: 2–3 token units, placed in main memory (DRAM)
• Layer count: 48 (structured as 12 × (3 GDN + 1 QSA))
• GDN Heads: 48 for V and 16 for QK, 256 dimensions
• QSA Heads: 24 for Q and 2 for KV, 256 dimensions
• FFN: MoE (512 experts, 10 routed + 1 shared)
• MTP model: single-layer, already multi-step trained¹

The model card is available here.

https://huggingface.co/Qwen/Qwen3.8-Flash-Next

There are three major highlights.

• Qwen Sparse Attention
  Previous models used Gated Full Attention (in practice, Grouped Query Attention with a gate). This has been reworked here into Qwen's own Sparse Attention structure, which predicts tokens in micro-block units of four tokens at a time (labeled "residual stream" in the diagram). By moving to a sparse attention structure, the model is built to selectively process only the important parts of the context, cutting down on computation over long contexts.

• Gated Residual Read/Write
  To enable processing at the micro-block level, there's a gated mechanism that converts micro-blocks into state vectors (and back) while controlling the order, and it's placed on either side of the sections that need to be processed as state vectors.

• N-gram Embedding
  If I had to compare it to something, it behaves similarly to the Per-Layer Embedding found in Google's Gemma 3n, 4-E2B, and 4-E4B. Rather than loading all embedding information into VRAM, it's a mechanism that stores static information in main memory (DRAM) as a simplified retriever table, simpler than MoE.

How QSA and N-gram Embedding Work Together

As explained above, QSA is built on sparse attention. However, with the existing DeepSeek Sparse Attention, important positions are still located using a token-level indexer, which means computational cost keeps rising as context grows longer. The Qwen team's answer to this problem is N-gram Embedding.

Figure 2: DeepSeek Sparse Attention vs. Qwen Sparse Attention

Qwen's attention structure fundamentally processes things at the micro-block level. By having N-gram Embedding work together with QSA, the input token sequence gets aggregated into micro-blocks, context importance gets estimated at the block level, and only the most relevant regions are selected as targets for the attention mechanism. Where DSA processed things token by token, QSA processes them by micro-block instead — my guess is that this lowers the difficulty of the selection problem, enabling higher-quality position identification while also contributing to faster output.

The Intent Behind the New Architecture

Judging from this structure, my guess is that the Qwen team designed this new architecture with the following goals in mind.

• Improving speed
  Up through Qwen3.8, the basic policy was a structure that prioritized accuracy, and in exchange, the Full Attention portion ended up fairly heavy (more heads, higher dimensionality). It also had more "global" layers relative to other models (a 3:1 ratio, versus Gemma's 5:1), which meant output speed tended to lag behind other models.
  To raise speed while keeping quality, they appear to have moved from single-token inference to inferring four tokens at once. However, this carries the same kind of weakness as MTP features: accuracy within each individual token can drop. My guess is that's why they added a gate to the conversion mechanism between micro-blocks and state vectors, to keep that conversion under control.

• Making better use of memory
  Embedding offload was originally a feature built for mobile edge devices. My guess is that N-gram Embedding came about by pulling that offloading a bit closer to the GPU — into main memory instead of storage like an SSD — while piling a large amount of static information there, achieving both low latency and a major reduction in VRAM usage.

In short, this looks like an architecture that takes the series' existing problems head-on.

Let's Try Running It!

Current Support Status in llama.cpp

On August 28, 2026, llama.cpp Pull Request #27742, created by Daniel Han of Unsloth AI, was officially merged. That said, keep in mind its title is "initial Qwen3.8-Flash-Next (qwen4exp) architecture support" — this is still at the "got it running for now" stage. For the time being, it works from 0.3.0-dev build 10660, commit 6c847d onward. I ran my evaluation on a slightly newer version than that:

version: 0.3.0-dev (build 10665, commit ca3d5a3e1)
built with GNU 13.3.0 for Linux x86_64

I downloaded the following model files ahead of time. This time I decided to use the UD-Q2_K_XL version.

https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q2_K_XL

Table 1: Model files used for evaluation

Now let's actually run it with the following command. Keep all the GGUF files together in one place, and pointing to the one named 00001-of-00003 will automatically link up the other files.

build/bin/llama-server --model ../models/qwen3.8-flash-next/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf -t 4 -np 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --port 32211 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 131072 --reasoning on --reasoning-effort xhigh --log-verbosity 4 --mmproj ../models/qwen3.8-flash-next/mmproj-BF16.gguf

Given that this involves running inference on a model as large as 125B+51B-A6B, there was basically no way my own setup could handle it reliably. So I ended up borrowing an NVIDIA A100/80GB instance from Highreso², who provide the GPUSOROBAN GPU instance service I often write about, and used that instead.

Specs of the Server Highreso Lent Me

The specs are as follows.

Table 2: Overview of the borrowed hardware specs

Since this was special-case hardware I was lent for this occasion, the way I connected to it was a bit unusual, so I'll leave those details out of this article.

Back in 2021, the A100's 80GiB of VRAM was a leap beyond anything GPUs had offered before, and by 2024 it was prized as the strongest GPU around for LLM workloads. These days it's fallen behind the newer Hopper and Blackwell architectures, but I wanted to see whether it can still hold its own on the front line.

I also wanted to find out just how far a single one of these cards could push this model's context length.

Checking Resource Consumption

Here's how memory resource consumption looked, with the context length set to 256k.

Table 3: Memory resource consumption

Because of N-gram Embedding, about 27GB worth of weight data ended up placed on the CPU side. Aside from that, things don't look all that different from previous models at first glance — but comparing it against Qwen3.8-27B's memory consumption, the KV cache size turns out to be dramatically smaller.

For Qwen3.8-27B, the KV cache came to 4096.00 MB at a context length of 64k. Scaling that up to match a 256k context length would put this model at 16,384.00 MB.

For Qwen3.8-Flash-Next, adding in the KV cache size for N-gram Embedding at a 256k context length comes to 8,448.00 MB — roughly half of the 27B model. This is likely a change that comes from switching the structure from GQA to QSA. My guess is the cause is the shift in processing unit, from a single token to a four-token residual stream.

For a model release of this scale, I was honestly surprised to see the KV cache memory drop this much. On top of that, 27GB of the model sits in main memory, which should leave a decent amount of headroom.

Pushing Memory to Its Limit

If you ultimately want to run the KV cache in f16, I confirmed that the following configuration supports up to 2 concurrent connections.

build/bin/llama-server --model ../models/qwen3.8-flash-next/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf -t 12 -np 2 --ub 4096 -b 4096 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --port 32211 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 524288 --reasoning on --reasoning-effort xhigh --log-verbosity 4 --mmproj ../models/qwen3.8-flash-next/mmproj-BF16.gguf

Here's how memory resource consumption looked.

Table 4: Memory consumption when pushing VRAM usage to the maximum

Here's the memory usage. I pushed this pretty close to the limit. By shortening the context length a bit, there should still be room to run other models alongside it.

N/A 51C P0 128W / 300W | 80645MiB / 81920MiB | 58% Default

Running an Inference Task

As usual, I had the model go off at length about the geological characteristics of Kyushu.

Output Volume and Speed

The volume of reasoning this model produced at the xhigh level was substantial, as shown below. Let's compare it against Qwen3.8-27B here too.

[Qwen3.8-Flash-Next + NVIDIA A100]
prompt eval time = 1154.61 ms / 378 tokens ( 3.05 ms per token, 327.38 tokens per second)
       eval time = 224353.82 ms / 10057 tokens (22.31 ms per token, 44.82 tokens per second)
      total time = 225508.43 ms / 10435 tokens

[Qwen3.8-27B + NVIDIA RTX A4000]
prompt eval time = 303.63 ms / 64 tokens ( 4.74 ms per token, 210.78 tokens per second)
       eval time = 248375.97 ms / 5771 tokens (43.05 ms per token, 23.23 tokens per second)
      total time = 248679.60 ms / 5835 tokens

Table 5: Token count comparison for the same prompt

The reasoning volume came out to about 1.7x that of Qwen3.8-27B. Speed was also roughly double, so the two cancel out and total output time ends up about the same.

About the Output

The output ended up being quite long.

(The following is the model's raw output, kept in the original Japanese since this example specifically demonstrates the model's native Japanese generation. The topic is the geological characteristics of Kyushu, Japan.)

九州地方の地盤は、一言でいうと「火山活動・活断層・地殻沈降・山地崩壊」の要素が強く結びつき、地域によって地盤の性質がかなり異なるのが特徴です。
同じ県内でも、平野部、台地、山地、海岸、火山周辺ではまったく違う地盤になります。

1. 全体の特徴：日本の中でも「動的な地盤」

九州は日本列島の西側・南側に位置し、大陸縁部の構造と海洋側プレートの沈み込みの影響を強く受けています。

主な背景は次の通りです。

1. フィリピン海プレートなどの沈み込み
   九州南方でフィリピン海プレートが沈み込み、南側・南東方の地震活動に影響しています。
   日向灘・南海トラフ・琉球海溝系に関連する大きな地震の潜在リスクがあります。

2. 九州中部〜東部に見られる「裂割帯」「沈降帯」
   別府湾から大分、阿蘇、熊本方面にかけて、地殻が裂けて沈み込みやすい帯があります。
   これが別府---島原地溝帯や大分---熊本構造線などに代表される地質構造です。

3. 火山活動が非常に活発
   阿蘇山、雲仙普賢岳、霧島連山、桜島、開聞岳など、火山が多い地域です。
   地熱も高く、温泉地帯が各地にあります。

4. 盆地と山地が複雑に分布
   平野部は堆積物で厚く覆われ、山地は急斜面と風化岩盤が発達しています。
   その結果、軟弱地盤と強風化岩盤、崩壊地盤が入り混じる地盤構造になっています。

2. 代表的な地盤タイプ

① 火山性の地盤：シラス・軽石・火山灰・凝灰岩など

九州、特に南部の地盤を語るうえで重要なのがシラスです。

シラス：軽石・火山灰・火山砕屑物などが堆積したもの
主に鹿児島県・宮崎県・熊本県南西部などで見られます。
台地を形成していることも多く、シラス台地と呼ばれます。

地盤としての特徴
特徴：粒子が軽い　内容：軽石・火山灰のため、全体として軽く、多孔質
特徴：透水性が高い　内容：水はけは良いが、雨水で侵食されやすい
特徴：風化・崩壊しやすい　内容：未固結〜弱固結のものは崩れやすい
特徴：農業面では問題　内容：保水力が低く、地下に浸透しやすい
特徴：斜面では不安定　内容：崩壊・地すべり・土石流の原因になりやすい
特徴：地盤として脆弱な場合がある　内容：締固め状態・風化程度・含水状態によって強さが大きく変わる

特に未締固めのシラス、風化が強いシラス、飽和状態のシラスは、地盤として弱い場合があります。
一方で、十分に締まったシラスや古層の堆積物は、条件さえ良ければ支持力を持つこともあります。

ただし、火山碎屑物そのものが災害源になることもあります。
桜島、霧島、阿蘇、雲仙普賢岳などでは、噴火による降灰、軽石流、火砕流、泥流の影響を考慮する必要があります。

② 活断層帯の地盤

九州は活断層の分布が多い地域でもあります。

代表的な地質構造としては、以下のようなものがあります。
別府---島原地溝帯
大分---熊本構造線
宇佐---八代構造線
布田川断層帯
日奈木断層帯
大隅半島周辺の断層帯

地盤としての特徴
特徴：地盤が切れている　内容：断層帯では地層・岩盤がズレて境界ができる
特徴：破砕帯ができる　内容：断層沿いで岩石が細かく砕かれている
特徴：変質しやすい　内容：断層帯では地熱・流体の影響で岩盤が弱くなることもある
特徴：地震時に地表変位の可能性がある　内容：活断層直下では地表面のズレを考慮する必要がある
特徴：斜面不安定になりやすい　内容：断層帯の周辺は谷・崖・崩壊地が形成されやすい

例えば熊本地震（2016年）では、断層帯付近での地表変位、強震動、斜面崩壊、地盤被害などが広く報告されました。
九州で建物を建てる場合、土地選びでは活断層の位置と距離を確認する必要があります。

③ 盆地・平野部の軟弱地盤

九州には、山地に挟まれた盆地や河川沿いの平野が多数あります。

例：
福岡平野
佐賀平野
筑後平野
大分平野
熊本平野
宮崎平野
鹿児島湾岸低地

地盤としての特徴
地盤タイプ：河川堆積物　特徴：砂・シルト・粘土が層状に分布
地盤タイプ：扇状地　特徴：山地から運ばれた砂礫で、比較的堅い層と緩い層が混在
地盤タイプ：軟弱粘性土　特徴：圧縮性が高く、沈下しやすい
地盤タイプ：砂質土　特徴：飽和砂は液状化しやすい
地盤タイプ：埋立地　特徴：港湾・臨海部では締固め不良の埋立土が問題になることがある
地盤タイプ：旧炭鉱地域　特徴：福岡県・佐賀県・長崎県などで過去の地下掘削による沈下の可能性がある

特に河川低地・海岸部・埋立地では、以下のような地盤上の問題が起きやすいです。
液状化
不同沈下
長期的な圧密沈下
地下水位上昇による浮き上がり
埋立土中の腐食性・汚染性
地下掘削跡による陥没リスク

九州の平野部では、「山間部より平野部のほうが地盤が強い」とは限りません。
平野部ほど堆積層が厚く、軟弱な地盤が分布している場合があります。

④ 山地・丘陵の崩壊しやすい地盤

九州山地や丘陵地は、地震・大雨・台風に非常に敏感です。

主な特徴は次の通りです。
急斜面が多い
岩盤が風化・変質している
花崗岩が深く風化して弱くなっている
斜面崩壊土・崩積層が厚く載っている
表土と岩盤の境界が滑り面になりやすい

特に問題になる災害は以下です。
表層はくずれ
斜面崩壊
地すべり
土石流
岩盤崩落

九州では梅雨前線豪雨、台風の大雨、地震動が組み合わさると、山地で大量の土砂災害が発生しやすいです。
山地の住宅地や農地を選ぶ場合は、傾斜・崖の位置・地すべり危険区域・土石流危険区域を必ず確認すべきです。

⑤ 地熱・温泉地盤：変質して弱くなる地盤

九州は日本有数の地熱帯です。

代表的な温泉地・火山地帯としては、
別府
由布院
雲仙普賢岳周辺
阿蘇周辺
霧島・桜島周辺
黒川温泉周辺
などがあります。

地盤としての特徴
地熱地域では、温泉流体や地熱によって岩石が変質することがあります。
現象：地熱が高い　地盤上の影響：地下工事や基礎設計に影響
現象：温泉成分で岩石が変質　地盤上の影響：岩盤が粘土化し、強度低下
現象：酸性・硫化物環境　地盤上の影響：地下構造物や配管の腐食リスク
現象：熱水帯　地盤上の影響：地下温度が高く、施工管理が難しい

「温泉が出る＝地盤が良い」とは限りません。
温泉域では、温泉が湧いているほど地盤が変質し、岩盤や土壌が弱くなっている可能性があります。

3. 地域ごとのイメージ

地域：北部九州：福岡・北九州・佐賀・長崎　地盤の特色：盆地・平野・埋立地・かつての炭鉱地域など　主な留意点：軟弱地盤、液状化、埋立地、坑道沈下、河川氾濫
地域：中部九州：熊本・大分・阿蘇・別府　地盤の特色：裂割帯・火山・温泉・堆積盆地　主な留意点：活断層、地熱、変質地盤、斜面崩壊、熊本地震系断層
地域：南部九州：宮崎・鹿児島　地盤の特色：シラス台地・火山前線・湾岸低地　主な留意点：シラス崩壊、火山噴火、地震、津波、斜面不安定
地域：山地：九州山地　地盤の特色：急斜面・風化岩盤・崩積土　主な留意点：豪雨・地震による地すべり・土石流
地域：沖縄　地盤の特色：沖縄本島では石灰岩、珊瑚礁性地盤、離島地形　主な留意点：台風、高潮、津波、斜面、石灰岩の空洞化など。本土とは別性格

4. 九州の地盤で特に注意したい災害

① 地震

九州は比較的地震活動が高い地域です。

注意点：
活断層による直下型地震
沈み込み帯沿いの大地震
盆地部での震動増幅
山地での揺れによる崩壊
平野部での液状化

「九州は関西ほど地震が少ない」という印象を持たれることもありますが、熊本地震の例もあるように、決して安全とは言い切れません。

② 噴火・火山災害

九州は火山帯が重なる地域です。

留意対象：
阿蘇山
雲仙普賢岳
霧島連山
桜島
開聞岳
姪の浜付近を含む薩摩半島・大隅半島の火山帯

被害としては、
降灰
軽石
火砕流
泥流（ラハール）
山崩れ
噴石
があります。
特に桜島は活動性が継続している火山であり、鹿児島市街地にも深刻な影響を与える可能性があります。

③ 大雨・台風による土砂災害

九州は日本でも雨の多い地域です。

梅雨末期豪雨
台風による集中豪雨
山地での浸透による崩壊
谷部での土石流

山地や台地の縁では、一度雨が入ると一気に壊れることがあります。
特にシラス台地、風化花崗岩、変質岩盤の斜面は危険です。

④ 津波

九州は海に囲まれ、南側はプレート境界の近くにあります。

日向灘
南海トラフ系
琉球海溝系

などに関連して、津波リスクがあります。
湾口部の市街地、河口沿い、海岸平野では、津波浸水と地盤被害の両方を考える必要があります。

5. 地盤調査・土地購入・建設時に見るべきポイント

九州で土地を買う場合や建物を建てる場合は、最低限次のような項目を確認すべきです。

必ず見るべきもの

1. 地盤マップ・地質図
   地形・地質・台地・低地・斜面の位置を見る

2. 活断層の位置
   断層直下は避ける、または影響を評価する

3. 地震ハザードマップ
   震度分布、液状化、斜面危険度をチェック

4. 土砂災害警戒区域
   急傾斜地崩壊危険区域、土石流危険区域、地すべり防止区域

5. 洪水・浸水ハザードマップ
   河川沿い・河口・湾岸低地の危険度

6. 火山ハザードマップ
   降灰、軽石、火砕流、泥流の対象地域

7. 過去の地盤調査
   ボーリング調査、N値、SPT、土質試験

8. 地下水位
   地下水位が高いと、浮き上がり・施工性・腐食性に影響

9. 埋立地・造成地の履歴
   盛土の深さ、造成年月、締固め状態

10. 坑道・鉱山跡・空洞
    北部九州などでは特に要注意

6. 工学的な留意点

住宅・建物設計での考え方
地盤条件：軟弱粘性土が厚い　対策例：表層処理・置換・パイル基礎・地盤改良
地盤条件：砂質土が飽和状態　対策例：液状化対策：締固め、グラベルコンパクションパイル、耐液状化設計
地盤条件：シラス・軽石が厚い　対策例：地質確認、透水性・侵食対策、斜面対策、基礎深さの検討
地盤条件：断層帯付近　対策例：基礎設計・配管設計・構造物の可動性・地表面変位対策
地盤条件：斜面近傍　対策例：崖線管理、擁壁、排水、地すべり対策、表土対策
地盤条件：地熱・温泉域　対策例：温度影響、変質岩盤、腐食性、地質追加調査
地盤条件：旧炭鉱域　対策例：空洞探査、沈下履歴、地盤補強

重要なのは、「周辺が良そうだから安心」とは限らないことです。
九州では、数百メートル離れただけで地質が急変することも珍しくありません。

7. まとめ：九州の地盤を一言で

九州の地盤の最大の特徴は、

「火山と断層が強く、地質が複雑で、山地・台地・盆地ごとに地盤性质がまったく違う」

ことです。

具体的には、

南部ではシラス・軽石・火山砕屑物
中部では別府---島原地溝帯などの裂割帯・断層帯
北部では盆地・平野・埋立地・旧炭鉱域
山地では風化岩盤と崩壊地盤
海岸部では軟弱地盤・津波・液状化

といったように、同じ九州でも地盤災害の型がまったく異なります。

そのため、九州で地盤を考えるときは、

火山性土か否か
活断層に近いか
盆地の軟弱土か
斜面崩壊地か
旧坑道・埋立地か
地熱地盤か

を具体的に確認することが非常に大切です。

Third-Party Evaluation by a Frontier Model

I'm also reproducing part of the diagnostic results from GPT-5.6-Luna. The list of issues it raised was long enough to be organized into its own chapter structure — a sign of just how many fine-grained points it found.

(Kept in the original Japanese, since the evaluation itself concerns the quality of the Japanese-language output above.)

総評

提示文は、九州の地盤について一般向けに説明する導入としては、方向性が大きく外れていません。

特に、次の点は妥当です。

九州では、火山性堆積物、河川堆積物、海成粘土、風化岩盤などが地域ごとに異なる。
九州中部では、火山活動と正断層型の活断層が密に分布する。
南九州では、シラスの侵食と斜面崩壊が重要な問題になる。
平野部や埋立地では、軟弱地盤、圧密沈下、液状化を検討する必要がある。
山地では、地震と豪雨が斜面災害を引き起こす。
土地購入や建築では、地質図だけでなく、地形分類図、活断層情報、ハザードマップ、ボーリング情報を確認する必要がある。

一方で、地質学、地盤工学、地震災害、火山災害、土地利用上の注意が一つの説明に混在しているため、専門的には整理し直したほうがよいです。

また、いくつかの名称に誤記と考えられる箇所があり、断定が強すぎる部分もあります。

そのまま公開する場合は、少なくとも以下の4点を修正すべきです。

1. 「地殻沈降」と「地盤沈下」を区別する。
2. 「裂割帯」「宇佐・八代構造線」「日奈木断層帯」などの用語を確認する。
3. 「温泉が湧くほど地盤が弱い」「活断層直下は避ける」などの一般化を弱める。
4. 「姪の浜付近を含む薩摩半島・大隅半島」という明らかな地理上の誤りを修正する。

Most of the feedback centered on fine points of phrasing, but one thing that caught my attention was how many naming errors there were. I've listed them below — this model may not be especially strong at Japanese output.

Table 6: Typos and errors flagged by the evaluating model

These issues could probably be improved somewhat by running the output through a "Refiner" — a model with strong Japanese ability that handles proofreading and cleanup.

I also looked into the thinking section: the word "wait" appeared inside the <think></think> block only once, and there was almost no sign of circular, repetitive reasoning. That said, in its drive for thoroughness — checking every little detail — it ended up in a "every little bit adds up" situation, and it took quite a while to produce its answer.

It seems like some tuning is needed here, such as reducing reasoning volume via the Reasoning_Effort setting.

Checking the Vision Feature

For this test, I used the Qwen3.8-Flash-Next architecture diagram that appeared at the start of this article.

Figure 3: The image actually uploaded via the Vision feature (the diagram text is in Japanese, since it's the original Figure 1 image being fed back in as a Vision-capability test)

Here's the transcription it produced. In the original, six spots are marked in red to flag where it seems to have misread something; since the text below is preserved as a plain code block, that coloring doesn't carry over, so the flagged spots are listed here instead:

• "三明治構造"
• "PROPE"
• "GDN（線形・状態ベース）を多用し、QSA（疎フルアテンション）"
• "kNN-LM/RETRO"
• "NTP"
• "レトリオーバble"

(Kept in the original Japanese, since this is a verbatim transcript of the model's Japanese-language description of the diagram.)

図の全体像

この図は、通常の「1トークンずつ処理するCausal LM」ではなく、残差ストリームをマイクロブロック（ブロック状のチャンク）単位で処理する言語モデルアーキテクチャを説明したものです。上部は「1つのブロック内の計算グラフの反復（時間軸に展開したRNN的な見せ方）」、下部は「入力文字列から予測文字列までの全体の計算フロー」を描いており、両者は同一アーキテクチャの異なる視点です。

下部には Input strings → tok → Emb → 各レイヤー → LNR → Soft Max → Sampler → Predicted strings という生成モデルとしての流れが描かれており、推論パイプライン全体（トークナイザ〜サンプリング）まで含めた構成図であることが分かります。

上部：残差ストリームのブロック反復処理

残差ストリームが右から左へ順にブロックを通過し、最後の残差ストリームが最上部の矢印で最初のブロックへループバックしています。これは、シーケンスをマイクロブロックに分割し、同じブロック構造を順次適用し続ける（＝状態をcarryしながら逐次処理する）構造であることを示しています。

各ブロックは GRR / 中央演算 / GRW の三明治構造です。

GRR (Gated Residual Read)・GRW (Gated Residual Write)：注釈のとおり「残差ストリームを制御し、状態ベクトルを示すためのゲート機構」。GRRが読み出し、GRWが書き戻し、その対が1つの「状態ベクトルの更新」をなします（GRR→GRWをまたぐ上の矢印が「状態ベクトルとして処理」）。要するに残差流への読み書きにゲートを噛ませた、Mamba/DeltaNet的な「読み書き制御のある残差接続」です。

中央演算がブロックごとに GDN → MoE → QSA → MoE と変わっていく点がポイントです。GDNは線形アテンション系（Gated DeltaNet）のレイヤー、QSAは注釈にある通り Qwen Sparse Attention（マイクロブロック単位でフルアテンションを可能にし、GQAの高速化を狙う仕組み）、MoEは混合エキスパートのMLPです。

つまり「どのブロックがどの中央演算を持つか」が反復内で異なり、線形アテンション層と疎なフルアテンション層とMoEを交互配置するハイブリッド構造になっています。

下部：全体の推論フロー

1. Input strings → tok → Emb：トークン化とトークン埋め込み。

2. 加算ノード（⊕）：埋め込みに対して PROPE（おそらく回転系位置エンコーディング系の機構。破線矢印で各レイヤーにも注入されている＝レイヤー間で位置情報を補正/注入している）と、N-gram Embedding の参照結果を足し合わせて残差ストリームを生成します。

3. 破線枠内の本体レイヤー：[GDN|MoE] × 3 → [QSA|MoE] と続く。GDN（線形・状態ベース）を多用し、QSA（疎フルアテンション）は稀に混ぜる、というQwen3-Next系のハイブリッド層構成に一致します。

4. 末尾側は GRR + NTP Module（Next Token Prediction用ヘッド）、さらに LNR → Soft Max → Sampler で Predicted strings を出力。最終段にもGRRがあるのは、予測直前に残差ストリームをゲート経由で読み出す構造であることを示します（「strings」複数＝ブロック単位の複数トークン予測/並列生成を意図している可能性があります）。

5. N-gram Embedding（On Main memory, 黄色）：VRAM外にオフロード可能なレトリーバ用テーブル。2〜3トークン単位（n-gram）で索引化され、静的な知識情報を蓄積し、必要に応じて参照する。入力トークン系列からキーを引き当ててホストメモリ（RAM/ディスク）から知識ベクトルを引いてくる、kNN-LM/RETRO的な外部メモリ機構です。黄色で強調されているのは「重みパラメータとは別系統のメモリ」だからです。

色分けの意味（推定）

濃い青（GDN/QSA）＝シーケンス間の情報集約の中心となる演算（注意機構系）。
薄い青（GRW, tok, LNR, Softmax, Sampler, NTP）＝残差書き戻しと入出力/予測経路。
灰色（MoE, GRR, Emb, PROPE）＝補助・ゲート・位置情報系。
黄色＝唯一の「オフチップ（メインメモリ）知識源」。

この図が主張している設計思想

1. トークン単位からブロック（マイクロブロック）単位への転換：残差ストリームを固定長のブロックで区切り、GRR/GRWによる状態ベクトル更新で逐次処理することで、長いシーケンスを状態を保持しながらストリーム的に処理できる（RNN的な計算量・メモリ振る舞い）構造にしています。

2. 疎アテンション（QSA）による高速化：フルアテンションをブロック粒度で許可し、線形アテンション（GDN）で補うことで、GQAや通常のフルアテンションよりスループットを出そうという意図。

3. 静的知識のVRAM外化：n-gram索引で引ける「変わらない知識」はパラメータに焼かず、メインメモリのテーブルへオフロードする。これによりパラメータ/VRAMを圧縮しつつ知識容量を拡張する（sparse memory的な発想）。MoEが「選択」を担い、N-gramテーブルが「検索」を担う、二層の疎構造と読むこともできます。

留意点

GDN・MoE・QSA・GQAは実在の研究概念（Qwen3-Next系列のハイブリッド注意力＋MoE）と整合しますが、GRR/GRW、N-gram Embedding（On Main memory）、PROPE、LNR といった命名は標準的な公開アーキテクチャ名ではなく、この図独自の提案要素、あるいは何らかの技術ノート/発表資料内の仮称と見られます（PROPEは位置エンコーディング系、LNRは最終段の正規化系と推測される程度）。

したがってこの図は「実装済みの公式アーキテクチャの仕様図」というより、Causal LMをブロック逐次型＋外部メモリ検索型に変える提案アーキテクチャの概念図と読むのが妥当です。

要約すると、「残差ストリームをマイクロブロック単位でゲート制御しながら反復処理するハイブリッドアテンション（GDN/QSA）＋MoE言語モデル」を描き、その上から「静的知識をメインメモリ上のn-gramテーブルに退避したレトリオーバbleな知識源」と「NTP/Softmax/Samplerによる生成ヘッド」を接続した全体構成図、ということになります。

As an experiment, I fed this description to GPT-IMAGE-2 and had it turn it into a single illustrative sketch.

What kind of picture would come out of this, I wondered...

Here's what came out.

Figure 4: The image GPT-IMAGE-2 produced from Qwen's generated text (left as-is in Japanese, since it's a direct demonstration of the model's own Japanese-language output)

What came out was a reasonably solid image. The arrows pointing the wrong way are just a quirk on GPT-IMAGE-2's end, so please forgive that. Still, it goes to show that the text Qwen produced was detailed enough to reproduce faithfully.

Trying It Out with an Agent Package

I've been running various evaluations on the inference engine I built this time, and apparently one of the engineers at Highreso tried using Hermes Agent to build a clock app with it.

Figure 5: The finished app and the agent's screen (UI is in Japanese; this is an unmodified screenshot of the actual session)

Figure 6: Qwen3.8-Flash-Next usage as shown in the agent tool (UI is in Japanese; this is an unmodified screenshot of the actual dashboard)

Here, I want to cross-reference this against the logs I collected myself to check just how many tokens were actually used.

Checking the Trend in Output Token Volume and Speed

First, I checked the relationship between token volume and speed.

Since I had 3 concurrent connections running this time, you can see that sessions producing especially large token volumes have their speed roughly cut in half. This suggests Hermes Agent is firing multiple simultaneous requests at the inference engine it's pointed at.

Figure 7: Output token volume and speed over time (note that concurrent connections were allowed)

Perhaps because requests are being distributed this way by design, the token volume within any single session didn't end up all that large — the maximum output token count topped out at 29,606 tokens.

How Much of the Context Window Got Used?

Next, I lined up context usage on a per-case basis to see how much of the context window got used. The maximum came to 45,816 — less than I expected. However, since Hermes Agent's own "Token Input/Output" field showed "1.1M/85.4k," my guess is that context compression is kicking in somewhere along the way.

Figure 8: Context usage over time

This might just come down to the fact that what was being built was a clock app with a simple look and feel.

Checking the Correlation Between Prompt Processing Time and Speed

Checking the correlation between prompt-reading time and speed produced a nearly linear graph, confirming that speed stayed consistent across every session. The horizontal axis is time, and the vertical axis is token count.

Figure 9: Relationship between prompt-reading time and token count

The average came to 554.671, confirming that prompt-reading does take a bit of time. Normally, speed tends to decay as context grows, but the fact that it stayed consistent even past 40,000 tokens of context feels like a sign of just how capable QSA is.

Let's look a little closer. Below is prompt-processing speed lined up on a per-case basis.

Figure 10: Distribution of prompt-reading speed

What stands out is the upper band of the graph: in most cases, speed declines gently from around 600 tps as context grows. With GQA, this would degrade rapidly every time context increases, so it looks like the new attention structure keeps that effect very small.

Right in the Middle of Writing, MTP Support Arrived!

Trying Out the MTP Feature

Actually, this happened right around September 2, 2026, while I was in the middle of writing this document. I caught word that an MTP model had been released by Unsloth, and sure enough, checking Hugging Face confirmed the model really had been released.

https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/MTP

This time I implemented it using what's described as the standard version, mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf, and watched how it behaved for about two days.

The MTP model used here isn't a separate model like Dflash — it's the type that's built directly into the main model. In diagram form, it corresponds exactly to the area circled by the red dashed line below.

Figure 11: The MTP layer within the model

It's a model sized at 4B with an interesting structure of just a single layer, positioned as layer 48 (counting from 0) within the overall model. It sits right behind layer 47, the final attention layer, and is implemented as a branch off what would otherwise be a single line running into the decode circuit.

As a result, the specification of the tensors flowing through it is exactly the same as the base model. What I've been able to confirm is that micro-blocks get fed directly into the MTP layer as-is.

Given this structure, Qwen3.8-Flash-Next's MTP feature looks like it was implemented as an inherent part of the model, rather than being bolted on.

Implementation in llama.cpp

As for implementing this in llama.cpp, it's a feature that hasn't been merged into mainline yet.

A pull request was submitted once by Daniel Han of Unsloth, but it looks like full support for Qwen Sparse Attention was judged to need to come first, and as a result this PR — along with everything else related to MTP — has been put back into draft status.
models: Qwen3.8-Flash-Next MTP- #28243

https://github.com/ggml-org/llama.cpp/pull/28243

So I used a different branch instead — specifically the qwen4exp/mtp branch provided by Daniel Han of Unsloth, the same person who opened the pull request above.

git clone --branch qwen4exp/mtp https://github.com/danielhanchen/llama.cpp

I built this package in a directory separate from the original package and used that binary to enable the MTP feature, so the build options and cmake command are the same as before.

It can then be run with the following command. This time, to improve prompt-reading speed, I raised the batch size to 4096, which means the configuration consumes a larger compute buffer.

build/bin/llama-server --model ~/models/qwen3.8-flash-next/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf -t 12 -np 1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 -ub 4096 -b 4096 --port 32211 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt -fa on --cache-ram 0 -c 262144 --reasoning on --reasoning-effort xhigh --log-verbosity 4 --mmproj ~/models/qwen3.8-flash-next/mmproj-BF16.gguf --model-draft ~/models/qwen3.8-flash-next/mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf --spec-type draft-mtp --spec-draft-n-max 4

Next, let's check actual memory consumption.

Resource Consumption

Resource consumption came out as follows.

Table 7: Memory resource consumption with the MTP model enabled

Running the MTP model appears to require roughly 6GiB of additional memory, at least for the 4-bit quantized model used here.

What Happened Between September 3 and 4?

For this measurement, I used the logs as-is from the two days (September 3–4) right after implementing this setup, during which roughly 2–3 users were using the model. Here's what the people trying it out did during that window:

• Used Dify to try out technical consultations
• Used Hermes Agent to have it build a few browser games
• Used Hermes Agent to try out technical consultations
• Asked questions about OSS package specifications and requested configuration examples

The focus was basically on running AI agents and generating activity that consumes a lot of tokens, while also measuring how much context got used along the way.

The results are shown in the table below.

The "without MTP" column shows values from doing similar things before this was implemented. I'm including it as a point of comparison, but on the speed side, there were periods allowing 2 concurrent sessions when averaging, so treat those averages as a rough reference only — comparing max speed is probably more meaningful.

Table 8: Various metrics with and without MTP enabled

And here's the acceptance rate data.

Table 9: Acceptance rate while running MTP

Taken together, I think the following can be said:

• MTP's effect provided a certain amount of improvement (roughly 10 tok/s), though not a huge jump. That said, the effect was somewhat larger during code output, and that difference shows up in the max values.
• The maximum context volume in a single instance was 114.4k tokens, meaning a maximum context length of 128k was sufficient. It's possible that even more tokens had actually accumulated in reality, but it's likely that Hermes Agent's token-compression technology did its job.
• The average acceptance rate came out lower than expected. I think this stems from a mismatch — the base model is 2-bit quantized, while the MTP model is 4-bit quantized. If you want to enable the MTP feature, you should match the quantization bit depth between the two.

This measurement was still limited in scope, but even so, speed came out ahead of the no-MTP case in most instances, almost without exception, so it may well be worth enabling. There doesn't appear to have been much of a penalty from MTP.

More importantly, llama.cpp still hasn't fully caught up with this model overall — a lot of the gaps appear to be papered over with existing techniques, and full support for the attention structure in particular is still something everyone is waiting on.

Whether full support arrives before Qwen4.x ships is worth keeping an eye on. The relevant pull request is below.

qwen4exp : enable sparse flash attention in build_attn_qsa- #28349

https://github.com/ggml-org/llama.cpp/pull/28349

Conclusion

I've introduced Qwen3.8-Flash-Next, a model that gets an early implementation of a next-generation architecture. Its design reflects the Qwen team's stance of continuing to devise and implement their own improvements rather than simply reusing existing mechanisms.

Neural network structures in recent AI models have been growing more complex. They've moved away from the simpler designs of the past, and analyzing their architecture is getting harder.

This trend isn't unique to Qwen — it's common across Chinese AI vendors as a whole. There's also Anthropic's claim that "Chinese companies' AI relies on distillation techniques," but separately from that, this could also be read as one sign that they're pursuing independent technical progress of their own.

Movement among American vendors has been intense too — Meta, for instance, has rolled out its new Muse series. Open-model development is expected to keep accelerating going forward.

New models packed with new techniques keep arriving one after another, to the point where the pace of architecture analysis can't keep up. At the same time, local LLMs keep getting more capable, and we've reached a point where consumer-grade GPUs can produce answers of practically usable quality.

Parameter counts overall are trending upward, but how smaller models evolve from here is something worth watching closely.

Going forward, I'll keep providing information useful for deciding when to use an external API-based LLM versus a local one.

References

On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Qwen Team
https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf

DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
DeepSeek-AI
https://arxiv.org/pdf/2512.02556

Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency
https://qwen.ai/blog?id=qwen3.8-flash-next

model: add Qwen3.8-Flash-Next (qwen4exp)- #27742
https://github.com/ggml-org/llama.cpp/pull/27742

Qwen3.8-Flash-Next: How to Run Locally
https://unsloth.ai/docs/models/qwen3.8-next

Hugging Face/transformers library -- models/qwen4_exp
https://github.com/huggingface/transformers/tree/main/src/transformers/models/qwen4_exp

1. Although the model itself is equipped with an MTP feature, as of August 2026 it isn't yet supported in llama.cpp. The current state still looks like trial and error. ↩

2. Highreso Co., Ltd. A cloud provider offering the GPUSOROBAN service and compute batch cluster services. https://highreso.jp/ ↩