Muse Spark Open Benchmark: Open-Weight Models as Coding Agents

I asked for this review to come only from runs we already had, and in those runs none of the open models we ran this month kept up with the hosted ones as a coding agent. Meta's open-weights Muse Spark Open came out on 15 October with 816B parameters, 42B of them active. It's a different model from the API-only Muse Spark 1.3. It scored 0.091 and 0.0877 out of 1 on our Forge 2.0 agentic coding build, where the best build on the board scores 0.8523. Its Gauntlet 7.2 attempt never got a verdict. Google's Gemma 4, Granite 4.2 30B, LFM2.5-24B-A2B, Xing 4.0 and Nemotron 3.5 Bolt ran locally on our Macs and did no better, and neither did our own Qwen3.8-27B fine-tunes. Of those local runs, only LFM2.5's is on the board.

Open weights had a good autumn on paper. Meta's card puts Muse Spark Open at 76.4 on Terminal-Bench 2.1, and Xing's September card claims 75.0 on SWE-bench Verified. All of these models ship a tool-calling chat template. So we pointed them at the two benchmarks we built for agent work: a payments console on the Gauntlet and a Jira Forge app upgrade on Forge 2.0. Most of them fell over long before the scoring got interesting.

Maybe you pay for a hosted coding agent and wonder whether an open model on your own hardware could do the job. The result I'd look at first is the plain Qwen3.8-27B base. It scored 0.2576 on the same Forge build on a 64 GB MacBook, nearly three times what Meta's 816B model managed through OpenRouter. (It ran on a newer goose test build with thinking on, so that isn't a clean comparison either.) Where each model broke is more useful than the scores, because the same breaks kept coming back. A big file never arrives. A model reads for an hour and writes nothing. A tool call comes back as prose, or an app lints clean and shows wrong numbers. Some of those were the models and some were ours, and I've marked which was which.

Muse Spark Open benchmark results: Forge 2.0 and Gauntlet 7.2

These numbers are from the live boards as I read them on 20 October; the board itself is the current word. A pilot is a run we kept off the board, and the table says why.

On Forge 2.0, 0.0617 is what the unchanged starter app scores, so a model sitting exactly on it wrote nothing. Xing 4.0 and Granite 4.2 30B sit on it, and Bolt sits just under it. Both scored Gemma 4 builds sit under it too: one for a missing migration, one for a name declared twice.

Meta's 76.4 on Terminal-Bench 2.1 comes from Artificial Analysis's own run at xhigh effort in the Terminus 2 harness. The other independent numbers are on different scales. Artificial Analysis gives it 39 on its Intelligence Index (the median for comparable models is 19), and Vals AI's page shows 43.33% accuracy, with 6.57% on the newer, harder Terminal-Bench 4.0. We ran it at "medium" effort in our own harness.

What we found

1. Muse Spark Open stopped early, and nothing forced it to. It had 300 calls, used 58 and 44, and its reasoning said "we have limited calls" and "time limited". The prompt gave it 300 calls. It also said time was virtual.
2. Its Gauntlet attempt died on the serving route, as far as the logs show: four silences of about two minutes, four "Upstream idle timeout exceeded". Its siblings, served by Meta, wrote the same file in one call.
3. A tool call that arrives as text ends the run. It happened to LFM2.5 on its first move, and to our own fine-tune because of a flag we left off.
4. Xing 4.0 spent 43 calls failing to read the task. Granite 4.2 30B and our v2 fine-tune spent 40 and 106 calls reading. None of the three wrote a file.
5. Nemotron 3.5 Bolt, the one local model that built a full app, never looked at a total. Lint passed and the widget said "Committed 0.0002195".
6. Gemma 4 scored twice in eleven runs, both times under the unchanged starter: one missing migration, and one name declared twice. Six of the other nine ended on our engine or were cancelled by us.
7. Our v2 fine-tune cut its tool-call slips and failed the agent. The gates checked one step at a time, and I signed off on them.

The big file that never arrived

The Gauntlet asks for a payments console in 150 calls: two services that sync from a vendor and take its webhooks, plus a 3D view. Most of the work sits in one Python service, app/ledgerd.py, and a good build writes most of it in one go.

Muse Spark Open got there fast. In its first minute it wrote the transport layer (91 lines) and the notifier service (90 lines). Then it started on the ledger, four times. Three of the attempts opened with one line of reasoning, "Now ledgerd - big." or "Now ledgerd - huge."; the other streamed a short plan for six seconds. Then nothing reached our side for about two minutes until the connection dropped:

The provider failed mid-answer (Server error: Upstream idle timeout exceeded).
Resending — attempt 1 of 3. The unfinished answer was discarded.

Our agent, a fork of the open-source goose, resent the same request three times and got the same silence: 121,970, 127,589, 125,370 and 121,134 milliseconds. Then the run ended, and app/ledgerd.py in the tree is still the 23-line stub we hand each model. It cost 19 cents. I wouldn't call that a model result.

All 13 Muse Spark Open calls went to one host, Modal, and the error is an upstream idle timeout. The model didn't refuse. It hadn't run out of budget. I think the route sends nothing while the model builds a long tool call. A big file write then looks idle and trips a timeout at around two minutes. I can't prove that from the files. I can't tell whether OpenRouter or Modal raised it. On Forge 2.0, on the same host, its longest completed call took 80 seconds. A ledger the size its siblings wrote would need closer to four minutes.

Its siblings did the same thing and got through. Muse Wonder wrote an 845-line ledgerd.py in one write call of 45,841 characters and finished the Gauntlet at 0.7638, 9th of 44. Muse Spark 1.3 wrote a 1,173-line one in one call of 54,910 characters, after a single 168-second call, and finished at 0.5826. Both were served by Meta's own endpoint, where Muse Spark 1.3 survived calls of 394 seconds. (Both also ran earlier in October on older goose builds, and Muse Spark 1.3's run sent no effort setting.)

Our own v1 fine-tune lost the same file to a different limit, on our own engine, two weeks earlier. On 6 October it streamed one write call for 50 minutes: 44,238 characters, 1,021 lines, a complete-looking payments backend. Our engine stopped it at exactly 32,768 tokens, its default output cap when the client sends none, and goose sends none. Only the content arrived, with no path:

Error: Failed to parse arguments: missing required field `path`.
Arrived: `content`. Nothing was changed.

My guess is that it writes the content before the path, so the cut took the path with it. Its next words were "I've already written the ledgerd backend." That was half true. A 20,170-byte version from earlier in the run was on disk, and the 44,000-character one was lost. Its next write, a much smaller file, also arrived without a path. So I doubt the cap explains all of it. We raised the cap in our serve script the same morning. On the relaunch, v1 wrote a 900-line ledger in 16 minutes. Then it spent one call, 99 minutes long, trying to "clean up a buggy nested method" it had left as a placeholder returning "Drafts not yet wired." We stopped it there.

An agent that writes whole files in one call is betting that everything between it and the disk stays patient. Meta's endpoint was patient. The OpenRouter route to Modal wasn't, and neither was our 32,768 default. If you serve an open model yourself, check your output cap and your proxy's idle timeout before you blame the weights.

Our side contributed to that failure too. We didn't pin the OpenRouter route to a host, our benchmark runner doesn't retry a run that ends without a verdict, and our harness filed the stop as "model_finished". So Muse Spark Open has no Gauntlet score on our board.

Muse Spark Open on Forge 2.0: it stopped before it had tested anything

Forge 2.0 hands the model a working Jira Forge app (a sprint scope widget) and a contract for version 2. It asks for a new storage entity and a migration of the v1 rows. It also wants an admin page, a signed CI web trigger, a custom field, a JQL function and a rate budget the app must respect. The model gets 300 calls and an offline dev kit that runs the app's functions and events, and renders its UI Kit pages, without a live site. The prompt tells it to "plan to implement, test, and finish well inside" the 300 calls, and also says "Time is virtual: nothing is graded on wall-clock time."

About 33 calls into build 1, Muse Spark Open wrote this in its reasoning:

Need to continue implementing other modules quickly but we have limited calls.
We need to prioritize delivering something that will score.

Build 2, a minute in: "Need to continue building but time limited." Build 1 stopped at 58 calls after 7 minutes 25 seconds, and build 2 stopped at 44 calls after 6 minutes 9 seconds, both times with the model's own summary and no error. Over 80% of the budget was left unused both times. I don't know where that came from.

The rush shows in the code. Build 1's event handler calls a function that does nothing:

// src/index.js:93-94, Muse Spark Open, Forge 2.0 build 1
async function mirrorToLedger(body, cfg) {
  // no-op for now: actual ledger writes happen via sync.js which we can extend to write both entities

The comment is accurate, I'll give it that. Build 1 also wrote scopeLedger.js, dosing.js and admin.js, and nothing imports them. The scorer found 0 of 170 event rows in the new ledger, which is what a no-op produces. Build 2 did better there, with 132 of 170 event rows. But it copied the v1 rows with estimate: 0, boardId: '' hard-coded for all of them (src/migration.js:67-69), and its own summary admits it.

The admin page is where build 1 went wrong. The dev kit said that the admin function "does not load: Error: Unable to establish a connection with the Custom UI bridge." Lint then warned that the function's render export didn't exist. It fixed the warning and left the problem:

// src/admin-ui.jsx:29
export const render = () => {};

Its reasoning a few seconds later: "The harness's own UI Kit host may be different and may pass despite dev tool error." It didn't. Build 2 wrote layout: native where the contract says render: native. It never checked its page. Neither build's admin page ever rendered, so the admin panel scored zero. In build 2 I think that also zeroed the CI trigger. No panel means no "Rotate CI secret" button and no secret. The contract says every request then gets 401, and all of them got 401. Build 1's trigger had a separate bug. It returned { statusCode, body } where the contract asks for { outputKey }. It answered 500 to everything. With that many requirements at zero, the reliability multiplier (it scales the score down for every group of failing checks, never below 0.25) hit its floor. You can see it on the run page: 0.3638 tests earned, times 1.0 for criticals, times 0.25.

The best build on this benchmark used fewer calls than Muse Spark Open's build 1. jev-router, a router that in our runs sent nearly all its calls to OpenAI's GPT-6.1 Sol (our write-up), scored 0.8523 in 43 calls. It ran the offline dev kit in 26 of them, 70 times in all: the scheduled reconcile, event replays, UI Kit checks, the CI sender. Muse Wonder used 126 and 117 calls, ran the scheduled job 13 and 11 times, and declared the admin page the way the contract says, render: native pointing at src/frontend/admin.jsx. Its admin actions scored 1.0 in both builds. I'd copy jev-router's habit before anything else.

Our board groups runs that share the same task, kit, scorer and goose build into comparison groups, and the Muse runs are in different ones. They ran on 14, 15 and 19 October with identical task and scorer hashes, so my guess is the goose build differs. None of the ten Muse Wonder Forge requests we kept carried a reasoning-effort field, so it likely ran at its provider's default, while Muse Spark Open got "medium". jev-router sits in a fourth group. Muse Spark Open lists at the same OpenRouter price as Muse Spark 1.3 ($1.25 in, $4.25 out per million tokens; Muse Wonder is $4 and $20). But Spark Open's calls hit zero cache on Modal while the siblings cached heavily on Meta's host.

Meta says the open release is "derived from the Muse Spark model we released in August, with additional safety post-training". Its PyTorch post adds that it "shares the same base model as Muse Spark 1.3". So the base is an unlikely culprit. Was it the safety training, the effort setting or the route? I can't tell from two builds. The card doesn't say.

Local open-weight models on our Macs: Granite 4.2 30B benchmarks, LFM2.5-24B-A2B, Xing 4.0, Nemotron Bolt

These ran on our own MLX engine (a fork of Rapid-MLX) through the Goose Swarm app, the same app that drives our cloud runs. One setting matters for every Forge run in this section. The app sent no reasoning setting, and our engine switches thinking off on any request with tools unless told otherwise. So Xing, Granite and Bolt all ran Forge 2.0 with thinking off, while Muse Spark Open got "medium". I didn't connect it to these runs until we reviewed them, though we'd written the behaviour down in September for our training data.

LFM2.5-24B-A2B: 0.0187 on Gauntlet 7.2, after one call written as prose

Liquid's model has 24B parameters with 2.45B active, 30 short-convolution layers and 10 attention layers, and it always reasons before answering. On the Mac Studio it decoded at 85.7 tokens a second with a peak of 27.0 GB, the fastest local model we ran. Its whole run took 14 seconds.

After about a thousand tokens of reasoning, its answer was this, as text (home folder redacted):

[shell(command='ls -la /Users/<user>/Library/Application Support/Goose/benchmark/runs/build/openai-cloud-4ef38a6e-...-r0') 2>/dev/null || echo "ls failed"')]

Goose saw no tool call. It took the turn as a final answer, and the run was over after 2 of 150 calls, before it had read a line of the task. The string isn't a valid call even by LFM's own Pythonic format (a quote closes too early and a parenthesis dangles). It also passes a path with a space in it without quotes, so ls would have failed anyway. The engine had LFM's tool parser switched on. Its boot log shows it picked that format from the model's own files. Which layer turned it into text, the model leaving out its tool-call tokens or the parser rejecting a malformed call, the logs can't tell me.

Claude Haiku 5.5's first move on the same benchmark was a proper shell call with the path quoted: cd "/Users/<user>/Library/Application Support/..." && ls -la && ... cat SB7-CONTRACT.md. Haiku went on to 0.661. The October roundup already flagged LFM's score as no verdict on the model yet, and one run doesn't change that.

Xing 4.0: 40 tool calls, 35 errors, and the contract never read

Xing 4.0 is China Telecom's 29B-A4B model (successor to TeleChat), with multi-head latent attention and hyper-connections. mlx-lm doesn't support its architecture yet, so we ported it into our engine. The port matched the reference on a small test model but hadn't been tried on the real weights before this run. On Forge 2.0 Xing spent its first call asking for a tool that doesn't exist:

read {"path":"FORGE2-CONTRACT.md"}
-32002: Tool 'read' not found ... All 5 available tools, closest first: [read_image, write, edit, tree, shell]

Then bash eight times, read_image on a Markdown file, and 22 edit calls on the contract with placeholder text ("APPENDED", "APPENDED2", "APPENDED3"), apparently trying to read it through the error previews. Its own words at 21:57: "Let me read the files through the read_image tool in a different way:". Its one working shell call came at 21:54, and the next call went straight back to read_image. It wrote no files.

I don't blame the parser for this one. The engine parsed those calls correctly and logged that the model asked for read, which the request never offered, so those names are the model's. My guess is that read and bash are what popular coding harnesses call their tools, and it reached for them by reflex. Cloud models do this too: Mistral Large 4 lost 48 calls to "Tool 'bash' not found" on the same benchmark. jev-router's call 2 on the same task was cat STARTER.md FORGE2-CONTRACT.md RATE-MODEL.json BROWSER-TESTING.md, and it wrote its first new file less than five minutes in.

The run is a pilot because our engine cut it short. The memory guard capped Xing at 35,328 tokens on a 96 GB Mac. It priced the compressed attention cache at about 25 times what it needs. The session died on a failed compaction at call 43. We fixed the pricing that same night. A re-run on the fixed engine called read_image on text files 121 times without writing anything before we cancelled it.

Granite 4.2 30B: it read everything, then listed node_modules

IBM's Granite 4.2 30B is a plain dense transformer (64 layers, and no Mamba layers anywhere in its config). It was the most careful reader in the set. After a bad start (four read_image calls on text files, then a cat that broke on the space in "Application Support"), it said "the cat command had path issues with spaces. Let's use shell with proper quoting" and read the contract, the rate model, the manifest and all nine source files cleanly. Its notes were correct: "The manifest doesn't declare a jira:adminPage yet."

Then Granite ran find <run folder> | grep -v node_modules | grep -v build and got "(no output)", because our run folder lives under .../runs/build/, so the filter erased its own workspace. I'd put that one on our folder layout. And after goose compacted its context, the model's own summary said "Ready to begin v2 implementation", and its next call was tree depth 3. That lists the dependency folder, about 65,000 tokens, into a window our engine had capped at 69,888. Compaction failed. The run ended at call 40, with no write calls in 41 minutes. A second build hit the same cap on its second call, 16 seconds in.

The cap was our bug. The guard priced Granite's cache at double its size, and we fixed it about an hour later. The node_modules folder in the run directory is our harness too. Would Granite have built anything with the right window? A re-run on a 64 GB MacBook, where it only gets 52,480 tokens, made 106 calls and no writes before we stopped it. Our own log says that run measured the Mac's memory more than Granite. It also prefilled at only 171 to 275 tokens a second on the Studio (another session was rendering video at the time, which may have slowed it). Each 70,000-token tree dump cost six or seven minutes.

Nemotron 3.5 Bolt: a whole app, with its totals divided by a million

Bolt is NVIDIA's 74B-A9B compression of Nemotron 3.5 Super VL, a hybrid of 40 Mamba-2 blocks, 40 MoE blocks and 8 attention blocks. We ran it at 4-bit (43 GB) on a 128 GB MacBook Pro, with a 262,144-token window, at a median 30.6 tokens a second. Of the four models in this section, it's the only one that behaved like an agent. It used 292 of 300 calls over 2 hours 16 minutes. It read carefully and fixed its own syntax error ("The error is at line 272"). It left a v2 that lints, builds and would deploy.

Then we looked at the numbers. They were wrong. Bolt's ledger sums raw story points:

// src/ledger.js:118, Nemotron 3.5 Bolt
const estimate = member ? Number(member.estimate) || 0 : 0;

The v1 views it left unchanged expect micro-points and divide by 1,000,000. So the widget the scorer opened said:

Committed 0.0002195   Added 0.000057   Removed 0.0001365   Creep NaN.NaN%

A second bug at src/sync.js:351 divides twice, so a 5-point issue writes "added +0.000005" into the custom field. The admin page is declared without render: native, which zeroed six checks. It ran the dev kit 18 times, and not one of the outputs it read contained a total. In its last 23 minutes it checked only build and lint, then declared itself done: "Build and lint pass ... All the v2 surfaces are present in the manifest." Score 0.0542: 0.3615 tests earned, times 0.6 for corrupted v1 rows, times the 0.25 floor.

jev-router kept v1's units (committed: fromMicro(totals.committed), views.js:20-23) and declared the admin page with render: native. Then it ran the migration in the dev kit, read the storage dump, and reported "The first upgrade run migrated all 88 seeded v1 rows". Bolt never read its own storage back. I think that habit is what separated the two. The run is a pilot because build 1 scored under our 0.1 bar, so under our rule build 2 was cancelled and nothing was published. I dropped the whole Nemotron family the same day and had its weights deleted.

Speed and memory on the Mac

Decode speed didn't predict much. The fastest model made one tool call and the slowest finisher built the most. In these four runs, one each, what separated them was the context a model really had and whether it read only what it needed. A dense 30B at 8-bit needs more than 64 GB for this kind of work. On 64 GB it gets 52,480 tokens, too few to hold the contract and the code at once. On the 96 GB Studio, after our fix, it gets its full 131,072.

Gemma 4 as a coding agent: eleven runs of the 31B, 26B-A4B and 12B on our Macs, two scores

Google released the Gemma 4 31B (dense) and 26B-A4B (mixture of experts) in April and a 12B in June. Its launch post calls the family "purpose-built for advanced reasoning and agentic workflows", and the card promises "native function-calling support". So I'd call this a fair test of what Google pitches. What Google doesn't publish is a repository-level coding-agent number. Its card reports LiveCodeBench and tau2 (76.9 for the 31B), and its technical report puts Terminal Bench Hard at 36.0 for the 31B and 14.0 for the 26B-A4B. It's a general family pitched at agents, not a model tuned and measured as a coding agent. We gave it Forge 2.0 eleven times on three Macs on 18 and 19 October. Two runs got a score, 0.058 for the 31B and 0.0237 for the 26B-A4B, and both are under the 0.0617 that the starter app we hand every model scores when nothing is changed. The other nine ended without a verdict, and most of those nine were our doing.

Gemma 4 31B: seven checks better than the starter, and still below it

Our cleanest run was the 31B on the 128 GB MacBook Pro, on the app release that carried our cache fixes. Its prompt cache reused a median 95% of each prompt and no tool call failed to parse. It read the contract and the source files and ran lint. Then it made the decision that sank it:

// src/ledger.js:5-6, Gemma 4 31B, Forge 2.0 (128 GB MacBook Pro)
export const CHANGES = 'scope-ledger';
export const V1_CHANGES = 'scope-change';

That points the live ledger at the new entity. V1_CHANGES is never used, and no migration moves the old rows across. Our scorer found "migrated v1 payload corrupted at 371 checkpoint row(s)". It counts that as a critical defect, which multiplies the score by 0.6. Bolt failed that check too. It hit the same 371 rows. Apart from that, it beat the starter. It scored higher than the starter on seven checks, four of them in full, and lost none the starter passes. Its test score before the multipliers was 0.3727 against the starter's 0.2468. Without the critical defect it would have scored 0.0967, still under our 0.1 bar.

Most of its 39 minutes went into typing: three big writes took about 24 minutes at around 7 tokens a second. Its second sync.js write was identical to the first, and goose said so ("This exact call already returned this same output"). Its last write, the 176-line index.js, came out byte for byte the same as our starter. Then, at 29 of 300 calls, it stopped and wrote a plan: "Next steps: Implement the live migration from scope-change to scope-ledger (§9)", followed by six more sections it hadn't built. It never ran the bundler. Why stop there? I don't know, and the transcript doesn't say. Goose puts the current time in the turns it sends, so the model could see 38 minutes go by. But the task says nothing is graded on wall-clock time, and it had 271 calls left.

Its thinking was off. Our app sent no reasoning setting, so our engine switched it off, and each stored thought is a single newline. Gemma 4's thinking is opt-in anyway (a <|think|> token in the system prompt), but Google's technical report says its benchmark numbers are "in thinking mode unless explicitly stated".

Our plain Qwen3.8-27B base got this right on the same benchmark, in one build on the 64 GB MacBook M3 Max with its thinking on: it migrated 59 of 59 v1 rows. jev-router, the cloud router that ran mostly GPT-6.1 Sol, ran its migration in the dev kit and read the storage back before it claimed anything.

Gemma 4 26B-A4B: one name declared twice, nine functions gone

On the 64 GB MacBook M3 Max the 26B-A4B ran with thinking on (that Mac's goose build sends the reasoning setting) and decoded at a median 18.9 tokens a second. It made 68 calls in 76 minutes. About halfway through, at 04:37, it wrote this:

// src/index.js:178 and :249, Gemma 4 26B-A4B, Forge 2.0 (64 GB MacBook)
const adminResolver = new Resolver();
// ...
export const adminResolver = adminResolver.getDefinitions();

Bundling stops on that, with The symbol "adminResolver" has already been declared. All nine of the app's functions live in that file, so no function loaded. Checks the unchanged starter passes went to zero. The starter gets 170 event rows exact; this build got none, and its 130-row backfill went from complete to nothing. So it scored 0.0237. It had no critical defect. (The v1 rows weren't corrupted, because no v1 row ever reached the new ledger.)

It never ran the bundler. It never saw that error. Its last 25 minutes went on a manifest lint error instead, "jira:jqlFunction required properties are 'arguments, function, name, operators, types, key' or ...". It rewrote the whole manifest six times between 04:49 and 05:07. The first switched the argument and its types from string to number. The rest changed little beyond key order and quoting (one added a response block to the webtrigger and the next took it out again), and two of them were byte-identical. The error stayed. From its thinking:

04:52  I'm going crazy. I'll just rewrite the `jira:jqlFunction` block with the requested order.
04:56  Wait! I have them all!
05:03  Wait, let's look at the order again. Maybe it matters? Unlikely for YAML.

Its manifest needed required: true on the argument and types: [issue], as in the Qwen base build on the same Mac. That build also named its resolver instance something else and exported the definitions under the name the manifest expects:

// src/index.js:190 and :206, Qwen3.8-27B base, Forge 2.0 build that scored 0.2576
const adminModule = new Resolver();
export const adminResolver = adminModule.getDefinitions();

All nine functions loaded there. Gemma isn't alone in this one. Another 31B run on the same MacBook and a local pilot of Muse Glimmer 30B (0.0325) wrote the same pair. I'd guess they copy the starter's export const uiResolver = resolver.getDefinitions() line and reuse the export name for the instance.

To its credit, it caught a bug of ours. Goose's edit tool reported five edits as successes that had changed nothing, three of them in a row early on. It noticed after the first one ("Ah, I failed to actually edit anything because the before and after were the same"), sent the same no-op twice more, and after the third gave up on the edit tool: "I'll just use write to replace the whole file." Our engine's repetition guard then cut ten looping replies and reported each one as hitting the output limit, and goose ended the session after the third cut reply in a row, at 68 of 300 calls. Goose's repeat check also appended its notice to a cat output, and the model read that as a corrupted file.

The nine Gemma 4 runs that never got a verdict

All three Studio runs hit the same goose bug. With thinking off, Gemma 4 writes a reasoning block that is only a newline on almost every turn (58 of 59 on the Studio 26B-A4B). Goose took identical reasoning as a sign that tool calls came from one reply, and merged the later turns into a single assistant message, so the 26B-A4B's last request held 47 tool calls in one message. That defeats the prompt cache. Our engine re-read most of each prompt on every call: everything past the first 2,600 to 2,850 tokens on the 31B and the 12B, and everything past 23,417 on the 26B-A4B. The 31B's wait for a first token grew from 46 to 174 seconds. Then the 26B-A4B and the 31B each answered with a single token. Goose sent the identical request three more times and got one token back each time. I can't prove the merge caused that. I still count it as ours. On the 64 GB MacBook, where turns weren't merged and that Mac's goose turned thinking on, Gemma had no one-token replies in 76 requests, so I can't tell the merge from the thinking setting. We fixed the merge and the resend in goose 3.0.119 on 19 October, and the 128 GB MacBook Pro run above, on 3.0.119 according to our run ledger, has no merged turns. Left alone, I think the 12B's loop could have run for hours. Our engine let a local reply grow to the whole context window, and goose sent no cap. Since 3.0.122 our engine stops long exact repeats.

Gemma's own slips show even in the broken runs, but I'll put goose's part first. The Studio 26B-A4B made 22 edits and 19 changed nothing, and goose reported each as a success, in a history it had merged. The model also declared const MAX_RETRY_AFTER = 900; three times at the top of index.js and imported a ./ci.js it never wrote. And the Studio 31B made the same scope-ledger switch as the MacBook Pro one.

I dropped the Gemma 4 family on the morning of 19 October, right after that MacBook Pro run, and had its weights deleted from all three Macs. The run folders stay as evidence. None of the eleven runs matched Google's setup: all used 8-bit weights in our own harness, and five of them had thinking switched off by our engine. That includes the cleanest, the 31B on the MacBook Pro, and it still passed more of the checks than the starter before one migration decision cost it the score. I'm left with two pilots and no verdict on the model.

Fine-tuning for tool calling: our v2 fine-tune cut its tool-call slips and failed the agent

These are our models, and I signed off on both releases. Neither has produced a usable agent run. Since September we've trained Qwen3.8-27B on Atlassian work and published the results on Hugging Face. Our first one was released as v0.5, and I call it v1 below. The second, v2-goose, came out on 5 October, trained fresh from the base to work as a goose agent. The fine-tuning tutorial shows how we train them.

Tool calls came back as plain text: --enable-auto-tool-choice was off

v2-goose's first Gauntlet attempt on 5 October started right. Its first move was to read the contract, in well-formed Qwen tool-call format:

<tool_call>
<function=shell>
<parameter=command>
ls -la && cat SB7-CONTRACT.md
</parameter>
</function>
</tool_call>

It came back from our engine as plain text three times, and on its fourth turn the model wrote [Calling tool: unparseable_tool_call({})], copying the placeholder name it had seen in its own history. Our serve script caused it. Rapid-MLX only parses streamed tool calls when it gets --enable-auto-tool-choice next to --tool-call-parser, and we had copied our flags from a release script that left it off. Our pre-flight check had used a non-streamed request, which parses through a fallback, so it passed. We added the flag that afternoon. With it on, the same model streamed a parsed call on the first try. If your local model's tool calls arrive as text in vLLM or a Rapid-MLX server, check that flag first.

The second attempt ran 81 calls. The model was debugging its own blank table ("viz.brush is not a function", 0 rows rendered) and asked for a screenshot. Our engine answered "Model ... is serving text-only; image input is unsupported", and the run ended. The checkpoint has a full vision tower (333 vision tensors). The cloud-hosted base reads images fine. We had served it text-only: our serve script passed a text-only flag copied from the release script, and the speculative-decoding head we ship forces that lane anyway.

Back office built, shop never opened

The third Gauntlet attempt ran 135 calls, and it shows both sides of the model. The back end is good. All 12,290 payments synced through the paged walk. It resumed after a dropped connection, answered forged webhooks with 401, kept sends idempotent and survived a SIGKILL. Currency totals were exact to the minor unit with no cross-currency sums. The front end stayed exactly as we'd shipped it. web/app.js in the tree is still our one-line starter, "// Implement live data, page journeys, custom controls, and cross-view state here." Its final summary didn't match the disk. It claimed "four files with the required ids" in web/, a DECISIONS.md that doesn't exist, and "150/150 turns" at turn 134, none of which our tree checks found on disk.

"Map keys must be unique", read and ignored

On our older Forge 1.0 benchmark the same model wrote a 385-line resolver, three UI files and a Rovo skill. Its manifest.yml had a duplicate key: under a queue consumer at line 54, so the manifest didn't parse and nothing could deploy. It ran lint six times. It got the same crash each time. At turn 124 the validator told it exactly: "Map keys must be unique ... line 54, column 5". It rewrote the manifest and kept the duplicate. Then it ended with "The linter and forge-dev both crash in this environment", and claimed built bundles in a folder with zero files. That one is mostly the model; the caveat is that our serve script had thinking off that day.

77 calls of reading on Forge 2.0

On 19 October we gave v2-goose Forge 2.0 on the Studio. It made 106 calls in 31 minutes, all of them reads: the contract, the v1 source, the dev kit's internals, the manifest schema. After call 77, at about 103,000 tokens, goose compacted the context to about 11,000, and the summary the model wrote for itself said:

Done so far: full recon. Read the whole contract, STARTER.md, RATE-MODEL.json, all v1 source ...
## Next step
Write the v2 implementation. Plan, in order: 1. manifest.yml: add jira:adminPage ...

It went back to ls -la; cat STARTER.md, then tree, then cat FORGE2-CONTRACT.md, and read everything again. We stopped it at call 106 under our kill rule, and the run ledger says why: no source edits in 31 minutes. Our engine had thinking switched off for this run.

The plain Qwen3.8-27B base, in our own 8-bit conversion, first changed an app file at call 43 in the build that scored 0.2576 and at call 34 in the one that scored 0.1128, on the same benchmark. Those base runs used a goose test build with a new compaction, and the base kept its thinking on. On the shipped compaction the same base also re-read its files for about 33 calls before writing, so the restart after compaction isn't evidence against v2. What does count against v2 is 77 calls of reading before the compaction, where the base was writing by call 43 at the latest.

What the gates measured

v2 shipped because it beat v1 on the tool-call slip rates above all:

All seven rows are single-step checks. I didn't see that as a problem then. The pipeline did have one end-to-end check, a real task run in the Goose app, and it was marked not run, to be done after publication. The goose "FINAL" check behind those slip rates takes a recorded session, cuts it at one point and scores the model's next action. Its own code admits it "UNDER-counts versus live goose". Our training rows were mostly short (99% under 16,313 tokens, the longest 65,536), while the benchmark runs reached 87,000 to 103,000-token prompts over 80 to 300 calls. I don't think anything in our mix taught it to resume from its own summary or fix the line a validator points at. Nor to come back to the half of the product it skipped. We never checked the 527 goose sessions in it for that, though.

And seven gates failed. The 30-question Forge gate failed. The app-build gate fell below v1, and so did a thinking-format check. The 8-bit build failed its own accuracy check on goose contexts up to 16,384 tokens (a KL of 0.400 against a limit of 0.067). It looped once more than v1 in 160 answers, missed four style marks on the voice check, and failed the goose gate on one metric: in the two cases where it should have asked the user first, it never did. We waived all seven to ship on 5 October, for a deadline I set myself.

People kept downloading v1. Its repos (Q8, Q6, Q4, GGUF and LoRA) have 1,075 downloads in the last 30 days. v2-goose's four repos, up for 15 of those days and with no GGUF build, have 179 (Hugging Face's counts on 20 October).

Four more Qwen3.8-27B fine-tunes and merges we tried, with their caveat

Before the models in the 10-19 article, we ran four other Qwen3.8-27B derivatives on a 64 GB MacBook in mid-October. BAAI's AREX-2 (a long-horizon agent fine-tune) scored 0.0617 in both app runs with no files written, and 0.0014 in a run our engine ended. bytkim's Qwen3.8-27B-pi, tuned for the Pi agent harness, wrote nothing in one run. In the other it wrote four files and then drafted the rest of its v2 as prose. I cancelled it at 166 calls. JetBrains' Qwen3.8-3.6-27B-blend, a 50/50 weight merge with no training, scored 0.0321, 0.0522, 0.0938 and 0.0042. Abacus.AI's Smaug-Mini wrote nothing once (0.0617) and a manifest with an unquoted colon at line 79 the other time (0.0014).

Our own ledger says those first-round scores "are not a fair verdict on these models". Our engine dropped the reasoning from each past turn, and two more serving bugs turned up afterwards, and both affected all four models. The plain Qwen base, run as a control on a partly fixed engine that still had one of those bugs, also wrote no app code in 62 calls before we cancelled it. On the fully fixed engine the same base wrote 17 files before our memory guard forced us to stop it. Only the JetBrains blend got a re-run on a partly fixed engine, so I'm not ranking them.

What the runs say about each model

I'd sum up Muse Spark Open as fast, confident and too quick to stop. Two builds and one aborted Gauntlet attempt are thin, though, and the Gauntlet loss belongs to the route more than the model. LFM2.5-24B-A2B is very fast on a Mac and couldn't drive this harness in its one run. Xing 4.0 reaches for tools that aren't there. Granite 4.2 30B reads and never writes. Bolt works like an agent and doesn't check its numbers. In two pilots, one build per size, Gemma 4 wrote and rewrote more than it tested, and neither ran the bundler on its own code. The Xing, Granite and Bolt pilots ran with thinking off, as did the Gemma 31B pilot, so I'd re-run them before calling any of it a verdict. I wouldn't trust any of these local models without reading a transcript first.

Our serving cost v1 its first Gauntlet attempt, and on the relaunch it stalled on one 99-minute call. Our v2 has better single-step tool manners and doesn't finish anything. Both are ours to fix.

Muse Spark Open weights: 816B-A42B, what runs where, and the licences

Muse Spark Open has 816B parameters in 256 experts, 8 of them active per token (42B active). It mixes 46 sliding-window attention layers with 16 full ones, reads a 1M context, and its reasoning can't be turned off. Meta's card asks for at least 700 GB of GPU memory at NVFP4. Hugging Face's own launch post ran a 2-bit quant on a 512 GB M3 Ultra at 18 tokens a second, using 304 GB. Our 96 GB Studio wasn't going to run it, so it ran through OpenRouter, which is where I'd point anyone without a 512 GB Mac.

All read on 20 October. Meta's repo is gated and our account isn't approved. So the Muse licence text comes from the gate prompt on the repo page and the config from the unsloth and mlx-community mirrors. The licence has no user-count threshold like Llama's, and the $20M clause applies only to offering the model to others as a service. Using it inside your own product is exempt.

Where we were wrong

Most of our mistakes are already in the sections above: the unpinned route, the missing retry, the tool-call flag, the text-only serving, the output cap, the memory guard and the node_modules folder. For Gemma 4 add goose's merged turns and identical resends, an edit tool that reported no-op edits as successes, a repeat notice pasted into a file listing, an older engine that re-read every prompt, a memory estimate about nine times too high, no cap on a local reply, and loop stops reported as output limits. We fixed the merge, the resend, the no-op edits, the re-read and the estimate in 3.0.119 on 19 October. Since 3.0.122 on 20 October a loop stop is reported as a loop, and our engine also stops exact repeats up to 1,024 tokens long. Two belong here because they cut across everything. Our engine ran the Xing, Granite and Bolt Forge runs and v2's 19 October run with thinking switched off on tool calls, because the app sent no reasoning setting. Our fine-tunes also ran with thinking off on 5 and 6 October. So did Gemma 4 on the Studio and the 128 GB MacBook Pro. Meanwhile Muse Spark Open got "medium" and the plain base kept its thinking. And I never ran v2 against the base on the same engine and the same goose build, so I can't fully separate the model from our serving. I should have caught both.

The tool-call flag, the vision lane and the guard pricing are fixed, and so is the output cap in our fine-tune serve script. The app's engine still defaults to 32,768, and the route pin and the verdict retry aren't fixed yet. The node_modules folder still sits in the run directory, though since 3.0.119 goose's tree names it without listing it. In the Studio and 128 GB MacBook runs the app also sent no reasoning setting to the local engine.

What I'd check before trusting an open-weight model as a coding agent

Run it through one long task in your own harness, the way I'd test any new model, then read the transcript before you look at the score. In particular, look for these:

• Does its first tool call parse? Stream it, the way your agent does, and check that the call arrives as a call.
• Can your stack carry a long single write? Check the output cap, any proxy's idle timeout, and whether the file path arrives at all.
• What context does it really get on your machine, once the memory guard and the KV cache have taken their share? A model that has to compact at 70,000 tokens will re-read your repo.
• Is thinking actually on? Our engine turns it off on tool calls unless asked, and only its log says so. I'd check it on every new model.
• Does it ever run what it wrote? Count the dev-kit or test runs after its first write. jev-router ran it 64 times after its first write, in 22 of its 43 calls. Muse Spark Open's two builds ran it three times (all failing) and not at all.
• Does its final summary match the disk? Ours didn't, twice.

Reproduce it

The quotes above come from the run's own transcript, which goose keeps as SQLite in a hidden runtime folder next to each build (its name starts with a dot). I'd copy it first, so the live file stays untouched:

sqlite3 "file:<runs/build>/.<run>-runtime/goose/data/sessions/sessions.db?mode=ro" ".backup /tmp/s-muse-b1.db"
sqlite3 "file:/tmp/s-muse-b1.db?immutable=1" \
  "SELECT datetime(created_timestamp,'unixepoch','localtime'), substr(json_extract(value,'$.thinking'),1,110)
   FROM messages, json_each(messages.content_json)
   WHERE json_extract(value,'$.type')='thinking' AND json_extract(value,'$.thinking') LIKE '%limited calls%';"

2026-10-19 19:58:35|Need to continue implementing other modules quickly but we have limited calls. We need to prioritize deliverin

And to see which reasoning effort a run really sent, read the first logged request:

head -1 "<runs/build>/.<run>-runtime/goose/state/logs/llm_request.0.jsonl" | grep -o '"effort":"[a-z]*'

"effort":"medium

Deduplicate tool requests by their call id before you count them: compaction re-inserts old history with the same ids. Scores and bills for all published runs are on the benchmark board, and the methodology page explains how a run gets a verdict. For the format side of local serving, our MLX vs GGUF comparison covers the same Macs.

Goose Swarm, the app that ran all of this, isn't production ready yet, and you've just read a list of the reasons. If you run an open model as an agent, what broke first for you, the model or the plumbing around it? Tell me in the comments under this post, especially if you got Muse Spark Open through a long write on a route other than Meta's. In our case it was a bit of both, and more of the plumbing than I'd like.

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.