Mistral Large 4 Benchmark for Coding vs DeepSeek V4.1 Flash

Mistral Large 4 scored 0.5462 on our Gauntlet benchmark, 28th of 44. About a third of what it lost came from a check of ours that read its table the wrong way round. All 12 amounts it put on screen were correct. I went in expecting to write about a slow, expensive model, and on our Forge benchmark that part holds up.

If you pay for a coding agent, this model is pitched at your budget. Mistral released Large 4 on 6 October as a Public Preview, 1.05 trillion parameters with 52 billion active and a 1M-token context, at half its list price for the first two weeks. Their launch post puts it ahead of DeepSeek V4 Pro on their own Coding Agent Index (49.8%), which is the kind of claim that moves a budget, so we ran it through both of our agentic coding benchmarks. The Gauntlet has an agent build a payments console with two services and a 3D view inside 150 model calls, then scores it on 99 checks. Forge 2.0 asks for a v2 upgrade of a working Jira Forge app inside 300 calls and scores it on 105 checks against three seeded Jira sites.

Getting a Gauntlet score took four attempts. The first, on 9 October, ended after 49 calls on OpenRouter's "Upstream idle timeout exceeded". On 18 October the second was ended by Mistral's servers. Early on 19 October the third was ended by a bug in goose, the open-source coding agent we run (our own fork), and the fourth finished that afternoon. Its Forge 2.0 pair scored 0.1086 and 0.3304 for at least $10.95. DeepSeek V4.1 Flash 1017 did better on the same Forge task. It's the 17 October snapshot I wrote about in my DeepSeek piece, and it scored 0.2422 and 0.6618 for at least $0.62 in about a quarter of the time.

On the Gauntlet the low score is mostly our scorer's fault. On Forge the code losses were Mistral's own. Our harness also gave it a smaller context window than DeepSeek. The build trees show where each loss came from. This is one scored Gauntlet run and one Forge pair, in our setup, on a model whose vendor says it can change silently while it's in preview.

Mistral Large 4 benchmark results: Gauntlet 0.5462, Forge 2.0 median 0.2195

I checked these on the live boards on 21 October; if a run page says something else now, trust the run page. All six Mistral runs together cost at least $21.53, with three of the bills only lower bounds.

Six things we found in the Mistral Large 4 builds

1. The x0.6 "wrong money" critical on the Gauntlet is our bug. Our money check compares the amounts on screen with the oldest 50 payments, and Mistral's table opens on the newest.
2. Mistral's money formatter works on integer digits with a per-currency decimals table, the same approach as DeepSeek's.
3. Forge build 1, by my reading, lost most of its gap to build 2 to a throttle it wrote itself: it booked 25 to 34 times the real cost of a list call and paused its own scheduled work each hour, before its v1 migration could run.
4. Neither Forge build ever displayed its admin page, and neither ran a browser self-test.
5. The 18 October attempt opened with one 37,853-word thought that ran 18 minutes, hit a 65,536-token output limit and was thrown away.
6. Mistral's API refused a message order that three other providers' endpoints had answered, through OpenRouter, in our runs two weeks earlier.

Mistral Large 4 on our Gauntlet: 0.5462, about 0.70 without a check of ours that misread its table

Most Gauntlet checks just add to a weighted average, but a handful are critical, and when one of those fails it multiplies the whole score down, by as much as x0.6. Wrong money in a payments console is one of those. The money check is called b_money_rendered. A browser probe reads the amount cell of the first 12 rows of the payments table. Then the scorer checks that the digits match a stored amount and that the decimals match the currency, because amounts are stored as integers in the currency's smallest unit (cents, or fils for the Kuwaiti dinar). A correct cell shows two decimals for EUR and USD, none for yen and three for dinars. Mistral's run scored 0.00 on it, with the detail line "0.00 of 12 cells exponent-correct; JPY/KWD traps 0.00", and the multiplier took the whole run from 0.9104 down to 0.5462.

What Mistral rendered

These are the 12 cells the probe read, verbatim: KWD 111.000, ¥594, ¥199, $4.47, ¥809, €224.00, KWD 82.400, €33.10, KWD 2.420, ¥1,060, $1,840.00, KWD 72.400. Our probe also records each row's id, and joined to the fixture, pay_12287 is stored as KWD 111000, so 111.000 dinars is right, and pay_12286 is JPY 594, which needs no decimals. All 12 are right. So are the probe's 12 extra rows.

Its formatter in web/app.js (lines 7 to 25, with the source's \u escapes shown as characters) never touches a float:

const CUR = {
  EUR: { sym: '€', exp: 2 },
  USD: { sym: '$', exp: 2 },
  JPY: { sym: '¥', exp: 0 },
  KWD: { sym: 'KWD ', exp: 3 },
};
...
const s = String(a).padStart(exp + 1, '0');
const int = s.slice(0, s.length - exp) || '0';
const frac = exp > 0 ? s.slice(s.length - exp) : '';

It planned this before writing it. At 10:58 our time (EEST), 27 minutes in, it checked the contract's example and two real payments in its thinking: "129900 EUR → €1,299.00 ✓; JPY 8564 → ¥8,564 ✓; KWD 8263000 → KWD 8,263.000 ✓". Our scorer's own screenshot of the inspector shows that dinar payment as KWD 8,263.000, and the inspector check (q_payment_context) passed.

Why the check said zero

Here is the scorer, score_sb7.py around line 973:

first_page = c.pack.payments[:50]
by_cur: Dict[str, set] = {}
for row in first_page:
    by_cur.setdefault(row["currency"], set()).add(row["amount_minor"])

payments is stored in date order, so payments[:50] is pay_00000 to pay_00049, the oldest fifty. A rendered cell only passes if its digits equal one of those fifty amounts, so the check assumes, without saying it, that the table opens oldest first. Mistral's opens newest first (sort: '-created_at', web/app.js:49). None of its 12 cells could ever match. Our probe even collects the row ids "so the scorer can join against fixtures", in its own comment. No scorer file reads them.

I re-graded the run with the scorer's own _money_ok function, copied unchanged. It gives 0.0 as written, 1.0 against the 50 rows actually on screen, and 24 of 24 when each cell is joined to its payment by row id.

Was opening newest first a mistake by Mistral? Its contract fixes the API's default order ("created_at (default, ascending by INSTANT)") and says the Date header toggles, but it never says which way the table starts. Mistral saw the gap and decided, at 11:48 our time: "The grader might check the first page shows specific rows? Unlikely to assume an order. I'll default to '-created_at' (newest first) — most console-like." It bet that the grader would not assume an order, and the grader did. You can argue the UI should inherit the API's default. I'd listen. Even then it's a sort-order quibble. The check charged it as wrong money.

Who did better at money

All five models I compared passed with 1.0: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol (the model behind Jev Router's top Forge score), GPT-6.1 Sol Pro and DeepSeek V4.1 Flash 1017. All of them open their table with sort: 'created_at', and so does our own reference app, which is why our release validation never met a descending default. Grade those five tables against the newest 50 payments and four score 0.0, Sol 0.05.

DeepSeek's formatter is close to a copy of Mistral's:

let digits = String(Math.abs(minor));
if (exponent > 0) digits = digits.padStart(exponent + 1, '0');
const whole = exponent > 0 ? digits.slice(0, digits.length - exponent) : digits;

Same idea, same output. It got 1.0 and Mistral got 0.0. Sol and Sol Pro call Intl.NumberFormat and divide the integer by 10**exp first, and Sonnet does the same with Math.pow, which is fine for amounts this size. So on money code it's a tie between Mistral and DeepSeek. The sort order made the difference.

Mistral did leave one real money bug, in its own test. Its browser self-test checks the first amount against /^[€$¥]\d[\d,]*\.\d{2}$|^KWD [\d,]+\.\d{3}$/, which would reject a correct ¥594 because it demands two decimals after a yen sign. It printed "PASS money formatted" four times, and my guess is that its own first row was never a yen payment. I can't see which row it was.

Without our false critical: about 0.70 and 11th of 44

The score path for this run goes like this. Its checks earned 0.9104. Our false critical multiplied that by 0.6, to 0.5462. Separately, a run that fails some of the 3D checks gets a ceiling on its score, and Mistral's partial failures on picking, the coasting camera and the overview's legibility set it at 0.709. The ceiling formula keeps a score just under the cap, so without the false critical the run would come out at about 0.7048. That's my arithmetic, not a re-score. The board still says 0.5462. It would sit 11th of 44, two places above DeepSeek 1017's 0.6957. The 3D failures, which are Mistral's own, cost it about 0.2 on top.

My audit of that check turned up a second flaw. The yen and dinar "traps" score 0 when no yen or no dinar row happens to be among the cells the probe reads (up to 12). A run whose visible amounts are all right can still pay x0.92. Four ranked runs paid it that way: Elorian Glimpse, Step 5 Preview, Qwen3.8 Omni Flash and GLM-5.3. Across every Gauntlet 7.2 build I could reach, Mistral is the only one the sort assumption hit.

Is Mistral Large 4 good for coding? Forge 2.0 builds of 0.1086 and 0.3304

Forge 2.0 tests the same habits harder: the agent reads a long spec, upgrades a working Forge app under a Jira rate model, and has to prove it all in a local sandbox (our Forge emulator kit) before the calls run out. My piece on the September DeepSeek V4.1 Flash covered this same task, with medians of 0.33 for it and 0.67 for Opus 5.5 in an earlier comparison group. Both Mistral builds and DeepSeek 1017's pair are in the same comparison group on our board. That means the same task, seeds, scorer and engine build. All four asked for the same effort setting.

A Forge 2.0 score is the earned score times a critical multiplier times a reliability factor, where reliability starts at 1 and loses up to 10% for each group of checks whose worst check failed, with a floor of 0.25. Build 1 sat on that floor (its unfloored product was 0.2194) and build 2 came out at 0.4718. Neither triggered a critical. My reading is that most of the 0.22 gap comes from build 1's throttle and the order it ran its migration in.

Build 1 throttled itself and never migrated

The task gives each Jira site a rate budget of 2,400 points an hour, and background work may use 70% of it, so 1,680. Build 1 wrote a gate for that in src/dose.js (lines 76 to 88):

if (dose.bg + estimatedCost > limit) {
  const untilMs = hourStartOf(now) + 3600000;
  await kvs.set(PAUSE_KEY, { untilMs });
  throw new QuotaPaused(untilMs);

The gate is fine. The estimates feeding it were not. With no points header from the site, it charged its own counter from guesses. It booked 51 points for a board or sprint list page (src/config.js:43, cost: 1 + 50), 105 for a search and 260 for a changelog bulk fetch. On the scoring site its first scheduled run had counted 1,567 points after 21 seconds. The next estimated call would have taken it past 1,680. So it paused for the rest of the hour. Yet the site's own ledger for that run says it had spent 715 points over 27 requests, with list pages costing 1.5 to 2 points each.

That would only have been a nuisance if the migration ran first. It ran last. reconcile() in src/index.js calls reconcileAll, then saveClosed, then migrateV1Rows, and the pause in the first step skipped the rest. Its rescan had no cursor. Each hour it started over and paused in the same place. Our scorer's verdict reads "0/130 historical changes v1 never recorded" and "h2: 0 of 59 v1 rows in scope-ledger", and it traces 15 checks to that one root cause.

On the development site, which our spec gives the same shape on another seed, the gate tripped too, twice. At 21:19 our time it wrote "The reconcile hit my own dose gate". After a reset one run got through and it reported "migration complete (88/88)".

Build 2 got the same thing right. I think it was more luck than design. Its cost counter under-counted instead (its own words near the end: "my costOf under-counts returned records"). It ran over the background share in 4 of 31 hours, and on our scoring sites it never paused early. In its last minutes its own dev run did pause early. At call 300 of 300 it spotted the ordering problem: "I'm at the call budget (300/300) — this is my final turn". Then it moved the migration to the top with the comment "it is the graded deliverable and needs the fewest points". It never ran that version. It still scored 59 of 59 migrated and 130 of 130 backfilled. DeepSeek 1017's build 2 had that ordering from the start (src/sync.js:409, "Migrate v1's rows first").

Neither build displayed its admin page

Both Mistral builds score 0.00 on the admin panel tier. That one failure takes down the panel labels, the CI web trigger (no secret can exist before the panel's Rotate button creates one) and the admin boot check with it. Build 1's src/frontend/admin.jsx ends with export default AdminPage; and never calls ForgeReconciler.render, so UI Kit had nothing to draw. It found that out at 21:39, on the 298th of its 300 calls, when its only admin-page test printed "(nothing rendered)". Build 2 does call ForgeReconciler.render(<App />) at line 197, and the non-admin view displayed its permission message, but the admin view came back as an empty tree. The probe recorded no error that explains it. And build 2 never ran the kit's UI Kit check on that page.

DeepSeek 1017's build 2 renders the same way (admin.jsx:186) and ran that UI Kit check 14 times before it finished. The habit shows in the counts too: DeepSeek's two builds invoked their resolvers in the sandbox 45 and 30 times, Mistral's 3 and 4, and neither Mistral build ran a browser self-test at all. Build 1 also sent 37 of its 355 tool calls, about one in ten, to a bash tool that doesn't exist (goose's is called shell), and build 2 sent 19 of 344.

Build 2 had good moments. It had the highest UI-function (0.9957) and reconcile (0.9974) tiers of the four builds in this comparison group. Both are small tiers. Both also beat an Opus 5.5 build from an earlier group. In its last ten minutes it noticed that v1's React bundles were 244 KB against a 150 KB boot budget. It rewrote all three Custom UI pages in plain DOM, to between 102 and 106 KB each.

Mistral Large 4 vs DeepSeek V4.1 Flash 1017: 27 minutes to the first file against 2

Time is where the two models differ most. Mistral's first app-code write came 32, 35 and 27 minutes into its three Gauntlet attempts of 18 and 19 October, and about 49 and 55 minutes into its Forge builds. DeepSeek 1017 wrote its first app file 2.4 minutes into its Gauntlet run and about 4.5 minutes into each Forge build, and it finished both Forge builds with 29 to 37 calls to spare. Mistral ran out of calls in both.

Its median call took 4.5 to 4.8 seconds on Forge against DeepSeek's 4.0. The typical call barely differs. The slow tail is what costs it. Each Mistral run had calls that took 11 to 18 minutes, and in the Gauntlet attempts the calls over five minutes add up to more than half of all the time the model spent answering. DeepSeek's slowest call in any of its three runs took 71 seconds, and across our runs Mistral generated between 55 and 70 tokens a second while DeepSeek managed 108 to 155.

The 18-minute thought that was thrown away

The 18 October attempt that Mistral's servers later ended opened with one thinking call of 37,853 words that ran 1,081 seconds. It was coherent (under 1% of its 12-word sequences repeat). It covered the money rule, the 3D tower heights and the order of the services. Then it stopped at exactly 65,536 output tokens. goose marked the reply as cut off, and nothing in it was carried out. Its next thought: "Let me continue. I was planning extensively. Time to write code." goose sent no max_tokens and our model limits list 262,144 output tokens, so the 65,536 cap came from the serving side. Whether that's OpenRouter or Mistral, our logs don't show.

Reasoning was 57% to 81% of everything it generated, depending on the run. On Forge both builds also compacted their context three times, once the prompt had grown to between about 170,000 and 210,000 tokens. After most compactions it re-read files it had already read ("I need to re-read the v1 sources and my v2 files"). Build 2's own compaction summary says it plainly: "I've spent ~66% of context on exploration; no code".

Those compactions are partly ours. goose gave Mistral a usable window of 262,144 tokens. It took the smallest Mistral endpoint's 524,288 (the EU one) and reserved half of it for output, on a model whose card says 1M, while DeepSeek got 655,360. At about 200,000 tokens a call, DeepSeek would have compacted too under Mistral's limit, and under its own it never did. The summary calls ran about 9 to 11 minutes per Forge build, inside the times charged to Mistral.

What did the long thinking buy? On the Gauntlet, quite a lot: its plans got the money rule right, and its decisions file is careful (a rejected draft is terminal, and any further action on it answers 409 invalid_state). On Forge it bought almost nothing the tests could see. A build that reads for 50 minutes and finds its admin bug on call 298 has no calls left to fix it.

Unexpected role 'tool' after role 'user': the 400 that was our bug

The third Gauntlet attempt ended at 02:05 our time on 19 October, 97 calls in, on this answer from Mistral's API, passed through OpenRouter:

{"object":"error","message":"Unexpected role 'tool' after role 'user'","type":"invalid_request_message_order","param":null,"code":"3230","raw_status_code":400}

Mistral had asked for two tools in parallel, read_image on a screenshot and shell. OpenAI's chat format only accepts text in a tool message ("For tool messages, only type text is supported", in OpenAI's API spec). So since goose 1.0 the formatter has put a placeholder in the tool result and sent the picture as a separate user message right after it. With one tool call, or when the image comes back last, that's harmless. When an image result comes back before another tool result, the request goes out as assistant (two tool calls), tool result, user message holding the image, tool result. Mistral refused the second tool result because a user message stood between it and its call.

Who is right? Mistral's function-calling docs show the order only as a table ("assistant fc.1, fc.2 → tool r.1 → tool r.2 → assistant"), and the error type invalid_request_message_order isn't in their error glossary. Anthropic states the rule in words: "You cannot include any messages between the assistant's tool use message and the user's tool result message." Our request broke the documented order. I count this one as ours.

Three of our runs from 3 and 4 October sent the same order, and all three were answered and scored: Claude Sonnet 5.5 (served by Claude on AWS), GPT-6.1 Sol Pro (OpenAI) and MiMo V2.6 Flash (Xiaomi). All three went through OpenRouter, and I only see what goose sent to OpenRouter, so whether OpenRouter reordered anything for those hosts our logs can't tell. Mistral refused it the first time. Mistral's own coding CLI has an open pull request for the same failure, where a steered user message ends up between a tool call and its result.

We fixed goose 17 minutes after the 400 (commit a0247fffc2, "keep parallel tool results contiguous when one carries an image"). Inside each run of tool results the image messages now move to after the last result, merged into one user message, and a test replays the exact refused request. It shipped in Goose Swarm 3.0.119, our desktop app, on 19 October. The scored fourth attempt most likely ran on 3.0.119, and it never called read_image, so the fix wasn't exercised in a Mistral run.

The second attempt's ending wasn't ours. At 19:40 our time on 18 October Mistral returned "Internal server error" (type unreachable_backend, a 503). goose resent three times with backoff, the last one failed the same way, and the run ended after 30 answered calls and at least $1.18. The 9 October attempt ended on OpenRouter's side. Three times, while it was writing its main service file, OpenRouter dropped the reply with "Upstream idle timeout exceeded", and goose gave up after the third resend. Our harness refuses to score a session the provider ended, because grading a half-built tree would publish an outage as the model's score.

What a Mistral Large 4 build costs: about 18 times DeepSeek's, at sale prices

Its Forge pair billed at least $10.95 for a median of 0.2195. DeepSeek 1017's pair billed at least $0.62 for 0.452, which works out at about $50 per median point against $1.36.

Those Mistral prices are the launch sale: Mistral's changelog says "Launch pricing: 50% off for 2 weeks" from 6 October, which by my arithmetic runs out around the 20th, though OpenRouter still showed the sale prices on the 20th. Its model page lists the regular rates as $1.36 in, $4.18 out and $0.14 cached, and at those rates the same pair would cost roughly twice as much, around $22, or about 35 times DeepSeek's at the off-peak rate it ran at.

Where Mistral's money went:

DeepSeek carried twice the context into each call, inside a window two and a half times larger. Nearly all of it was billed as cache reads at a price 23 times lower than Mistral's. At list prices, fresh prompt tokens make up about half of each Mistral Forge bill, though they're only 13% to 15% of its prompt tokens. My guess is that the three compactions per build and the re-reads after them are a large part of that uncached slice. On the Forge builds the compaction summaries also ran on Mistral itself, because our host pin caught the helper calls meant for a small Gemini model. We fixed that while build 2 was still running. It shipped in 3.0.119, after the pair. The fourth Gauntlet attempt's four Gemini helper calls are inside its $4.93.

What the architecture might explain

Mistral publishes less than I'd like. The model card says "a granular Mixture-of-Experts architecture" with "52B active parameters and 1.05T total parameters, and a 1.6B vision encoder", a 1M context and "Status: Public Preview". The Hugging Face placeholder says 49 billion active per token, 52 billion counting embeddings and output layers, with weights expected on 31 October. The launch post defers the architecture, "additional benchmarks, and our post-training methodology" to the weights release, and says "the reinforcement learning run behind this preview is still in flight". Expert counts, the attention design and a licence aren't published. There is no tech report yet.

On reasoning, Mistral's reasoning guide documents two values for this model, high and none. We sent medium. We do that for all our cloud runs. It's a valid value in Mistral's API schema, OpenRouter lists only high and none and defaults to high, and nobody documents what medium becomes. The 57% to 81% reasoning share suggests it was not thinking minimally, but that's an inference from our token counts.

I have two guesses about the behaviour. A preview whose RL run is "still in flight" may not yet be tuned for long agentic loops, where a long think costs calls and minutes. Our Forge logs look like that. Thorough reading, late writing, no time left to test. And 49B active parameters per token (52B with embeddings) is about three times the 16B at decode on the V4.1 Flash card, which the 1017 snapshot updates. That fits the gap in tokens per second we measured, though serving hardware and load matter as much, and our numbers can't separate them. Nothing in the cards explains the admin page or the throttle. Those are decisions in the code. I'd judge it on the code.

Mistral's own lifecycle page adds one more caveat: Public Preview models "allow silent updates". A pinned mistral-large-4-0 today may not be the model we measured on 18 and 19 October.

Where we were wrong

1. Our Gauntlet money check assumed the table opens oldest first and charged Mistral x0.6 for correct money. Joining each cell to its payment by row id fixes it; it isn't shipped, and the board still shows 0.5462.
2. The same check scores a currency that isn't on screen as a failed test. Four other ranked runs paid x0.92 for it.
3. We capped Mistral's context at 262,144 tokens on a 1M model, against 655,360 for DeepSeek, so its six Forge compactions and their minutes and dollars are partly ours.
4. goose put a user message between two tool results whenever an image came back before another result. Mistral refused it, and that ended a run. Fixed in 3.0.119.
5. Under a Mistral host pin our helper calls went to Mistral too, so its Forge bills include summary work meant for a cheaper model. Fixed in 3.0.119.
6. We asked for reasoning effort medium, which Mistral doesn't document for this model, so the effort it really ran at was the provider's choice.

Should you put Mistral Large 4 behind your agent?

From these runs, not yet for long agentic builds like our Forge task. It read for close to an hour and ran out of calls. It also cost about 18 times DeepSeek V4.1 Flash 1017 at sale prices. On the Gauntlet, once our check is fixed, it sits around 11th of 44, which is respectable for a preview that's still in training. I'd look again when the weights and the architecture write-up land (Mistral says 31 October), with the effort set to high or none and a full context window.

If you try it in your own setup, test four things. First, check whether your agent ever puts a user message between tool results (images, steering, injected reminders), because Mistral rejects that request outright. Through OpenRouter, the hosts behind three of our earlier runs answered it. Set the reasoning effort to a value Mistral documents. Watch the time to first write and the slowest calls rather than the median, since the 11- to 18-minute calls are where its time went. And read the failing check before you believe a critical, ours included.

Goose Swarm isn't production-ready yet. We run these boards with it because we know what it's made of. I'd like to hear from anyone running Large 4 in a real agent loop: did you get the long first thought too, and did none make it usable, or just cheaper? Tell me in the comments.

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.