Five Models Agreed and Were Still Wrong: A QA Playbook for AI-Translated i18n Files

You added forty new strings, ran your AI translation step, and every locale file came back green. Then a Spanish-speaking user asked why your startup description sounds like a gift box.

AI translation is good enough to ship by default, which is exactly why unchecked output is risky. Tools like Interceptor fill missing keys while you develop, and that speed is a real win. The cost is that nobody looks at the output.

This playbook covers what we learned running developer-flavored text through multiple models at MachineTranslation.com, an AI translation tool built by Tomedes, a professional translation company. It ends with a five-step QA pass you can run on your own locale files.

Key Takeaways

• Agreement between AI models measures the models, not the text. In one test, five models agreed on a wrong translation of "AI wrapper".
• Developer text fails in four recognizable ways: surrender, fragmentation, false consensus, and register match.
• Documents processed in fragments show 28% more terminology inconsistency than documents processed whole.
• Identical quality scores can hide different terms. Five models all scored 9.4 and still used three different French terms for "recognition".
• Review effort should follow disagreement and locale data, not assumptions about which languages are "easy".

Table of contents

1. Agreement measures the models, not the text
2. Four ways AI translation fails on developer text
3. A five-step QA pass for translated i18n files
4. Run the test on your own strings
5. Where to spend review time by locale
6. Q&A

Agreement measures the models, not the text

High agreement between models tells you the models share a habit. It does not tell you the translation is correct.

When MachineTranslation.com ran "AI wrapper" from English to Spanish, five models agreed on "envoltura de IA". That phrase means a physical wrapping, not the software pattern. ChatGPT and Claude both scored it 9.5 out of 10.

We reproduced the same failure for this article. A fresh run of "Our startup is an AI wrapper." returned "envoltorio de IA", which has the same problem.

The Hebrew idiom test showed the mechanism. When the dominant cluster of models was wrong, consensus found the centre of that cluster rather than the correct answer. Divergence between models was the real signal.

"Agreement is a signal about the models. Correctness is a signal about the text." Rachelle Garcia, AI Lead at Tomedes

Four ways AI translation fails on developer text

Developer text fails in four patterns, and each one needs a different check. We named them from a four-term experiment with current tech slang, run from English to Spanish.

"Slang is the stress test. 'Vibe coding' produced six translations from six models." William Mamane, CMO of Tomedes

A five-step QA pass for translated i18n files

A useful QA pass is small enough to run on every pull request. These five steps catch most of the problems above.

1. Lock placeholders and syntax. Every {token} in the source must survive in the translation. Remember that plural categories differ by language, so a target file needs the categories its language defines. See Unicode's plural rules and the language plural chart.
2. Give every key context. One-word strings are the riskiest, and a translator or model cannot guess whether "Post" is a noun, a verb, or an HTTP method. A description field per key fixes most of this, as covered in this i18n vs l10n guide.
3. Translate whole files, not fragments. MachineTranslation.com's internal analysis found documents processed in fragments show 28% more terminology inconsistency than documents processed whole. Batching strings one at a time is convenient for your pipeline and costly for your terminology.
4. Queue disagreements for review. Review the keys where models split, not a random sample. A quality score will not find them: in a financial test, five models all scored 9.4 and used three different French terms for "recognition".
5. Send high-stakes strings to a human. Billing, legal, consent, and error messages are the strings where being wrong costs something.

Here is a minimal check for step 1. It flags any placeholder that disappeared in translation:

const tokens = (s) =>
  new Set([...s.matchAll(/\{\s*([A-Za-z_]\w*)\s*[,}]/g)].map((m) => m[1]));

function missingTokens(source, translated) {
  const t = tokens(translated);
  return [...tokens(source)].filter((name) => !t.has(name));
}

missingTokens(
  "Your {count} files are ready. Tap {action} to keep them.",
  "Tus {cantidad} archivos están listos. Toca {action} para conservarlos."
); // ["count"]

This catches renamed or dropped tokens, including inside ICU plural strings. It does not check whether the target language has the right plural categories, so keep step 1's CLDR links handy.

"Translating strings one at a time is convenient for the pipeline and costly for terminology." Rachelle Garcia, AI Lead at Tomedes

Run the test on your own strings

You can test the failure modes yourself in a few minutes, with any translation tool. We ran three quick checks through MachineTranslation.com while writing this.

• Your {count} files are ready. Tap {action} to keep them. into Spanish kept both placeholders intact.
• Post into German returned "Beitrag", a noun that suits a forum post but not an HTTP POST or a "Post" button. That is the context problem from step 2.
• Our startup is an AI wrapper. into Spanish reproduced the false consensus described above.

If your strings live in XLIFF, MachineTranslation.com accepts XLIFF and MQXLIFF files and returns the same format. On a paid account, the MachineTranslation.com connector sets up in about a minute in Claude, ChatGPT, or Cursor, so you can run these checks without leaving your editor.

"Where models split is where a human should look first." William Mamane, CMO of Tomedes

Where to spend review time by locale

Model agreement varies by language pair, and it does not follow how well-resourced a language is. English to Spanish, a heavily resourced pair, landed near the bottom of the agreement ranking, while English to Hindi and English to Tagalog landed near the top.

MachineTranslation.com measured average agreement across its ten highest-volume language pairs over three months. It excluded its own consensus feature from the comparison, because that feature is built from the other models' outputs and would win by construction.

Lower agreement means more disagreement to review. Prioritize your QA time by this data, not by gut feeling about which languages are tricky.

Agreement also has a blind spot. In one test, "coger" returned 100% agreement even though one of eight models flagged that the word is neutral in Spain but vulgar slang in Argentina and Uruguay. Consensus scores measure convergence, not whether the output accounts for regional meaning.

Q&A

Is high model agreement enough to ship a translation?
No. Agreement shows the models converged, and five models converged on a wrong "AI wrapper" translation. Use disagreement to decide what to review, and never use agreement alone to approve.

Should I translate i18n strings one at a time?
No. Fragmented processing showed 28% more terminology inconsistency than whole-document processing. Translate whole files with a shared glossary where you can.

Do placeholders survive AI translation?
Usually, but check every time. A simple token comparison like the one above catches renamed or dropped placeholders before they reach production.

Can I run these checks inside my editor?
Yes, on a paid account. The MachineTranslation.com connector works in Claude, ChatGPT, and Cursor.

Conclusion

AI translation belongs in your i18n pipeline, but not unchecked. Lock placeholders, add context, translate whole files, review disagreements, and send the high-stakes strings to a human.

If you keep your strings in XLIFF, you can try this check on your own files in a few minutes.

Disclosure: this article was written with AI assistance and reviewed by humans for accuracy. #ABotWroteThis