The same prompt costs 1.64x more in Hindi than in English — one tokenizer ago it was 5.28x

You pay for an LLM by the token, not by the word. So here's a question nobody asks until the invoice arrives: does the same message cost the same in every language?

It doesn't. I measured it on one small text, and the answer ranges from "a bit more" to "five times more", depending on the language and on which tokenizer your model uses.

Two terms first. A token is a chunk of text the model reads and bills — often a whole English word, sometimes a piece of one. A tokenizer is the fixed dictionary that cuts your text into those chunks; each model family ships with its own.

Why language changes the count

A tokenizer is built by scanning a huge pile of training text and giving the most frequent character sequences their own token. That pile has historically been mostly English. So "shipping" is one token, while a Russian or Hindi word of similar length gets chopped into three or four pieces, because the dictionary has fewer ready-made chunks for it.

Same meaning, more pieces, bigger bill. Every language you serve carries its own multiplier — call it the language tax.

The measurement

I wrote a typical system prompt for a support bot — 192 English words, rules about refunds, tone, what not to reveal. A system prompt is the hidden instruction text sent at the start of every request to set the bot's behaviour. Then I translated it myself into Spanish, German, Russian, Japanese and Hindi, saved each as a .txt file and counted tokens with OpenAI's tiktoken library in two encodings: o200k_base, used by GPT-4o and everything newer, and cl100k_base, used by GPT-4 and GPT-3.5.

# pip install tiktoken
import tiktoken

new = tiktoken.get_encoding("o200k_base")   # GPT-4o and newer
old = tiktoken.get_encoding("cl100k_base")  # GPT-4, GPT-3.5

base = None
for lang in ["en", "es", "de", "ru", "ja", "hi"]:
    text = open(f"{lang}.txt", encoding="utf-8").read()
    n_new, n_old = len(new.encode(text)), len(old.encode(text))
    base = base or (n_new, n_old)
    print(f"{lang}: {n_new} ({n_new/base[0]:.2f}x)  "
          f"old {n_old} ({n_old/base[1]:.2f}x)")

Results (tiktoken 0.14.0, October 15, 2026):

Two things jump out. The tax is still there: Japanese and Hindi cost about two-thirds more than English. And the newer tokenizer cut it drastically. Hindi went from 5.28x to 1.64x; Russian from 2.25x to 1.33x. The bigger dictionary finally has room for non-Latin scripts. English didn't change at all — 224 in both — so the whole gain went to everyone else.

Notice Japanese has the fewest characters (566 vs. 1,125 in English) and still the most tokens. Characters are not a proxy for cost.

What it does to a bill

Take a support bot on gpt-5.4-mini — official price $0.75 per million input tokens and $4.50 per million output tokens. Assume 300,000 calls a month, each with 2,000 input tokens and 300 output tokens if it were in English. Now scale both sides by the multipliers above, as if the whole conversation ran in that language:

• English: $855 a month.
• Russian: $1,137 (+$282). On the old tokenizer's ratio it would be $1,928.
• Japanese: $1,443 (+$588).
• Hindi: $1,405 (+$550). On the old ratio: $4,512.

Is the model worse at Hindi because it sees more tokens? Not necessarily — the tax here is on money and on context space, not a quality verdict. But a long Hindi chat also fills the context window about 1.6 times faster.

Should you write the system prompt in English?

Partly, and the honest number is smaller than people expect. The system prompt is only the part you control; the user's messages and the model's replies stay in the user's language, and that's where most tokens live.

With my 224-token prompt, switching a Russian prompt to English saves 74 tokens per call — 22 million tokens a month, or $16.65. Not worth a migration. With a 3,000-token prompt (long rules plus product docs), the same switch saves about 297 million tokens and $223 a month for Russian, $434 for Hindi. That's worth an afternoon.

A pattern that works: instructions in English, plus one explicit line about the reply language.

You are the support assistant for Parcelly. ...
Always reply in the same language the user writes in.

Translate the prompt when it's long and stable, and test the replies afterwards: rules about tone and politeness sometimes lose nuance in translation. Don't translate few-shot examples of the answers themselves — those show the model how to write in the target language. For our own Russian-language blog, the old 2.25x ratio was a real cost line, so we keep pipeline instructions in English and content in Russian.

The other lever is the price per token, which this tax multiplies. We resell the same models 10–25% below official prices — gpt-5.4-mini at $0.6375/$3.825 instead of $0.75/$4.50 — so the Russian bot above would cost about $966 instead of $1,137. That trims the bill by a fixed percentage; it doesn't remove the 1.33x. Translate first, then shop for price.

What this doesn't solve

This is one 192-word text, translated by me — a different translator or domain will shift the ratios by a few points. I only measured OpenAI's tokenizers; Anthropic and Google use their own and need their own count. And the money estimate assumes the whole conversation scales by the same ratio, which is an approximation, not a measurement of real traffic.

Run the script on your own system prompt in your users' language. It takes a minute, and it tells you what your multiplier actually is.