The best translation model is not one model: gpt-5.6-sol leads in 7 of the 9 languages we judged, claude-fable-5 edges it in Korean, gemini-3.7-flash ties it in Korean and passes it in Italian at the same measured price, and the cost of translating the same million characters spans $0.26 to $104 depending on which API you pick. We generated 4,210 translations with 9 models across 9 languages and put 8,455 blind pairwise verdicts on them; this post is the resulting matrix.
TL;DR
- Default pick: gpt-5.6-sol defended 7 of 9 languages in 8,455 blind verdicts; Korean goes to claude-fable-5 (52%) and Italian to gemini-3.7-flash (52%).
- By content: literary belongs to gemini-3.7-flash (60% vs baseline); UI strings tie across all 9 models with zero broken placeholders, so the cheapest wins; best sub-$4 pick is qwen3.8-max.
- Cost spans $0.26 (deepseek-v4-flash) to $104 (gemini-3.1-pro) per 1M characters; the $104 is mostly 1.17M reasoning tokens the model refuses to turn off.
- Judges preferred frontier models over human post-edits 83-100% of the time.
How did we test translation quality?
Three ingredients: a parallel corpus, one production-grade prompt, and a blind judging panel. Everything below, including the raw verdicts, is in the public benchmark repo.
Corpus. 52 English segments across six domains, translated by every model into all 9 languages. Four domains (news, social, speech, literary; 8 seeded-sampled segments each) come from WMT24++ (Apache-2.0), the evaluation set of WMT, the annual machine-translation shared task. Every WMT24++ segment carries a human post-edited reference (a professional translation reviewed and corrected by a second linguist), and the set covers the exact dialect of every locale we ship: zh_CN, zh_TW, ja, ko, fr, de, es_MX, pt_BR, it. Two domains are ours: 10 technical-documentation paragraphs from posts published in the last 60 days, which doubles as a contamination control since no model's training data can contain them, and 10 synthetic UI strings built from known localization hazards (context-free "Archive", {placeholder} integrity, plural forms). Sampling is seeded and the segment ids are published, so nobody can accuse the corpus of being curated toward a result.
Judging. Every candidate translation was compared blind against the same segment translated by a fixed baseline, gpt-5.6-sol, the model that currently translates this blog. Three judge families (claude-sonnet-5, gpt-5.6-luna, gemini-3.1-pro) scored independently under a rubric derived from MQM, the industry's standard translation-error typology, priority-ordered so an accuracy error outweighs any amount of polish:
1. Accuracy mistranslation, omission, addition, hallucination
2. Terminology domain terms as a native engineer would keep them
3. Fluency grammar, natural reading
4. Style register appropriate to the text type
5. Locale numbers, dates, units, punctuation width
6. Markup inline code, placeholders, links preserved exactly
The severity-graded design follows the spirit of Error Span Annotation, the protocol WMT has used for human evaluation since 2024, adapted here to pairwise LLM judging; corpus and references come from the WMT24++ paper. Each pair was judged twice with the presentation order swapped; when a judge contradicted itself across orders, the verdict became a tie. Judges are averaged with equal weight, never pooled into one vote, and per-judge numbers are reported so family bias stays visible. A second arm compared the three frontier-tier models (gpt-5.6-sol, claude-fable-5, gemini-3.1-pro) against WMT24++'s human references. Placeholder integrity was graded by script, not by judges.
Which model translates best, per language?
The winner first: gpt-5.6-sol is not just the baseline of this table, it is the measured champion. Every challenger below is scored against it, none clears 50% in more than one language, and in the separate human-reference arm sol's translations were preferred over WMT24++'s professional post-edits in 86-100% of verdicts, the strongest showing of the three frontier models tested there. The table reads as win rate against sol, ties excluded, three judges averaged; 50% means parity.
| vs baseline | zh | zh-TW | ja | ko | fr | de | es | pt | it |
|---|---|---|---|---|---|---|---|---|---|
| gemini-3.7-flash | 26% | 35% | 43% | 50% | 40% | 37% | 39% | 8% | 52% |
| claude-fable-5 | 19% | 15% | 38% | 52% | 39% | 32% | 31% | 20% | 47% |
| gemini-3.1-pro | 17% | 22% | 24% | 45% | 30% | 21% | 38% | 14% | 25% |
| qwen3.8-max | 26% | 21% | 18% | 28% | 36% | 17% | 32% | 14% | 25% |
| kimi-k3 | 27% | 15% | 23% | 23% | 16% | 24% | 10% | 0% | 24% |
| claude-sonnet-5 | 3% | 6% | 9% | 32% | 25% | 27% | 28% | 10% | 9% |
| deepseek-v4-flash | 20% | 6% | 0% | 24% | 22% | 19% | 12% | 7% | 11% |
| glm-5.2 | 9% | 3% | 10% | 6% | 7% | 8% | 11% | 7% | 12% |
| Pick per language | sol | sol | sol | fable-5 | sol | sol | sol | sol | 3.7-flash |
Four things stand out. The baseline defends 7 of 9 languages outright, which retroactively validates the blind judging round that picked it for our own pipeline. Korean is the contested language: fable-5 edges past (52%) and 3.7-flash reaches parity, so if Korean is your main market the ranking genuinely differs. The Chinese open-weight models lose their home turf: Qwen at 26% and GLM at 9% in Simplified Chinese. And GLM-5.2's collapse is task-specific, not general: the same model held 6/6 schema keywords in our structured-outputs study days earlier; translation is simply not what it is good at.
Which content is still hard to translate?
Per content type, the recommendation plus the strongest alternative to it, with the alternative's win rate against the baseline (the field median is across all eight challengers):
| Domain | Recommended model | Strongest alternative | Field median |
|---|---|---|---|
| literary | gemini-3.7-flash (60% vs baseline) | gpt-5.6-sol | 8% |
| news | gpt-5.6-sol | gemini-3.1-pro (48%) | 28% |
| speech | gpt-5.6-sol | gemini-3.7-flash (43%) | 25% |
| social | gpt-5.6-sol | qwen3.8-max (27%) | 12% |
| techdoc | gpt-5.6-sol | fable-5 / 3.7-flash (30%) | 15% |
| ui | cheapest (deepseek-v4-flash) | any model, verdicts mostly tie (77% top) | 61% |
One control deserves its own sentence: the techdoc rows come from paragraphs no model can have trained on, and the Gemini models drop hardest there (gemini-3.1-pro averages 33% across the public WMT domains and scores 10% on the fresh paragraphs). That gap is consistent with benchmark familiarity, but also with the baseline's home advantage on technical prose it translates in production, so we flag it rather than conclude it.
Read as recommendations per content type: for news, social, speech, and technical documentation the baseline stands, and social is where it stands tallest, since slang, fragments, and implied context punish every challenger (three of eight scored under 10%). Literary prose is the one category with a different answer: gemini-3.7-flash beats the baseline outright at 60%, so fiction-like content has a cheaper, better home. And UI strings invert the logic entirely: verdicts mostly tie because ten words of button copy leave little to disagree about, and every one of the nine models preserved every {seconds}, %d, and {workspace_name} intact (0 broken of 27 each). When quality is indistinguishable, price decides: translate UI strings with the cheapest model you have. Placeholder breakage, the classic localization bug, appears to be solved at the model level in 2026.
What does the same translation cost on each model?
From $0.26 to $104.37 per million output characters (total measured spend divided by total output, aggregated over all nine languages), a 400x spread, and the most expensive model is not the best one:
| Model | $/1M output chars | Reasoning tokens burned | Win rate range |
|---|---|---|---|
| deepseek-v4-flash | $0.26 | 0 | 0-24% |
| glm-5.2 | $2.48 | 0 | 3-12% |
| qwen3.8-max | $3.18 | 0 | 14-36% |
| claude-sonnet-5 | $8.36 | 0 | 3-32% |
| kimi-k3 | $8.37 | 0 | 0-27% |
| gpt-5.6-sol (baseline) | $13.70 | 0 | n/a |
| gemini-3.7-flash | $14.46 | 505K | 8-52% |
| claude-fable-5 | $39.81 | 0 | 15-52% |
| gemini-3.1-pro | $104.37 | 1,171,770 | 14-45% |
The $104 row is a thinking tax, not a quality premium. Translation needs no reasoning, and every model that lets us pin thinking to zero ran clean; the current Gemini generation refuses every off-switch spelling, so gemini-3.1-pro burned 1.17M reasoning tokens across 468 translations and came out 7.6x more expensive than the baseline while losing to it in all nine languages. Even its flash sibling carried a 505K-token burn that pushed its cost above the baseline's.
The value picks follow directly. If quality must stay near the frontier, gemini-3.7-flash is the strongest challenger: 35-52% against the baseline in seven of nine languages (50% is parity; Simplified Chinese at 26% and Portuguese at 8% are its weak spots), the only outright domain win (literary, 60%), at a measured cost within 6% of the baseline's, because its mandatory reasoning burn eats the flash discount. If cost dominates, deepseek-v4-flash translates for 53x less than the baseline and never broke a placeholder; you give up real quality (0% win rate in Japanese is a floor, not a rounding error), which is a defensible trade for internal content and a poor one for anything customer-facing.
So which model do you actually pick?
The whole matrix folds into one decision table:
| You are optimizing for | Pick | Evidence |
|---|---|---|
| Best average quality, many languages | gpt-5.6-sol | defends 7 of 9 languages against every challenger |
| Korean | claude-fable-5, or 3.7-flash on a budget | 52% vs baseline, the only language a challenger wins; flash at parity (50%) |
| Italian | gemini-3.7-flash | 52% vs baseline at the baseline's price |
| Literary or fiction-like content | gemini-3.7-flash | 60% vs baseline, the only domain won outright |
| News, social, speech, tech docs | the baseline | no challenger above 48% in any of these domains |
| UI strings | deepseek-v4-flash | verdicts tie across models and placeholders are safe everywhere, so the $0.26 model wins by price |
| Budget multilingual, under $4/1M chars | qwen3.8-max | best win rates of the sub-$4 tier in all 9 languages |
| Absolute cost floor, internal content | deepseek-v4-flash | $0.26/1M chars, 53x under the baseline; accept the CJK quality gap |
| Never for translation | gemini-3.1-pro, glm-5.2 | a 7.6x mandatory thinking tax; a task-specific quality collapse |
Do LLMs beat human translators now?
Our judges preferred the machine over WMT24++'s human post-edited references in 83-100% of verdicts for all three frontier models, in every language, with a single exception: claude-fable-5 in Simplified Chinese at 44%. Two readings of that number, and honesty requires both. The strong one: WMT24++'s own authors concluded LLMs are now the best MT systems in all 55 languages they cover, and our panel agrees. The cautious one: LLM judges share stylistic priors with LLM translators, and a fluency-polished machine translation may be exactly what another model prefers; the references are post-edits, not literary gold. What the number safely establishes is that frontier machine translation is no longer distinguishable from professional human reference quality by any automated panel we can build, which moves the interesting question to price, where the gap is measured in orders of magnitude.
How noisy are LLM judges, honestly?
Noisy enough that unguarded single-pass judging would be malpractice, and the guards are cheap. When we showed the same pair twice with the order swapped, judges contradicted themselves outright (a hard win-to-loss flip) 9% of the time for claude-sonnet-5, 13% for gemini-3.1-pro, and 19% for gpt-5.6-luna. Our protocol converts every such contradiction into a tie, so position bias cancels instead of accumulating, and the per-judge win rates published alongside the pooled number let you check that no single family drives a conclusion. The three families disagreed on magnitude in places (sonnet-5 was the harshest judge of its own Claude sibling), but on direction they agreed in every language, which is the property the conclusions rest on.
FAQ
Which LLM should I use for translation?
gpt-5.6-sol is the best default for multilingual translation: it defended 7 of 9 languages against every challenger in 8,455 blind verdicts. For near-frontier quality at the same price, gemini-3.7-flash, which ties or wins in Korean and Italian. For bulk internal translation where cost dominates, deepseek-v4-flash at $0.26 per million characters, accepting a real quality gap in CJK languages.
Are LLMs better than human translators?
Automated judges now prefer frontier LLM translations over human post-edited references 83-100% of the time in our nine languages, consistent with WMT24++'s own findings. That claim is scoped to reference-quality post-edits and machine judges; it does not cover literary translation with editorial intent, and LLM judges may share stylistic preferences with LLM translators.
Should translation use a reasoning model?
No. Translation gains nothing from thinking in our data, and models that cannot turn it off pay for it: gemini-3.1-pro burned 1.17M reasoning tokens on 468 short translations, making it 7.6x the baseline's cost while losing every language. Pin reasoning_effort to none (or the model's off-spelling) for translation workloads.
Measured 2026-08-26 to 2026-08-29 through the Synthorai gateway: 9 models, 9 languages, 52 segments in 6 domains, 4,210 translations, 8,455 blind pairwise verdicts by 3 judge families under an MQM-derived rubric with order-swapping. Corpus, code, and every raw verdict are in the public repo; absolute numbers come from this single batch, so re-measure before relying on any row.
Related, same series: cheapest LLM by language, thinking controls across 13 models, structured outputs measured, Gemini 3.7 Flash cost.
