Restoring Japanese mojibake: brute-forcing encoding pairs with TextDecoder

Paste a Japanese email into the wrong decoder and こんにちは comes back as
縺薙ｓ縺ｫ縺｡縺ｯ. Save the same word to a file and open it in a Western
editor and you get 日本語. Annoying? Yes. Hopeless? No — in the
overwhelming majority of cases the bytes are still there, untouched.
Mojibake is a decoding mistake, not corruption, and you can undo it with
nothing but the TextDecoder that is already built into your browser.

This post walks through the whole restoration pipeline: the encoding pairs
that actually matter, how to get the bytes back, what to do when
TextEncoder can't help you, how to rank the candidate readings, and the
two nasty edge cases — the lone 縺・ at the end of old emails, and text
that was garbled twice.

Bytes in, wrong interpretation out

Every case of classic mojibake follows the same pattern:

1. Text is saved as encoding A (bytes on disk / bytes in the email).
2. Something reads those bytes as encoding B.
3. The reader prints whatever characters those bytes mean in B.

Nothing destroyed the bytes. The UTF-8 bytes of こんにちは are
E3 81 93 E3 82 93 E3 81 AB E3 81 A1 E3 81 AF — fifteen bytes. Read as
Shift_JIS (1.5 bytes per kana character on average), they turn into eight
different, perfectly valid characters: 縺薙ｓ縺ｫ縺｡縺ｯ. The information
survived; only the interpretation is wrong.

So restoration is a search problem: recover the bytes, re-read them with
every plausible charset, and rank the readings by how much they look like
real Japanese.

The nine pairs that matter

Japan has more legacy encodings than anyone would like: UTF-8, Shift_JIS,
EUC-JP, UTF-16 — plus Windows-1252/Latin-1, the fallback every Western
tool reaches for. Going through who-saved-what and who-misread-it, nine
pairings cover practically everything you will meet in the wild:

If you only ever memorize one row, make it the first one — a UTF-8 CSV
double-clicked open in Excel is probably the single most common way
Japanese text gets mangled on earth.

Step 1: getting the bytes back

When the text was misread through a single-byte charset such as
Windows-1252, the original bytes are trivially recoverable — every
character you see is one original byte:

const bytes = Uint8Array.from(
  [...garbled],
  (ch) => ch.charCodeAt(0) & 0xff,
);

Now throw every decoder at it. TextDecoder accepts all the labels we
need: utf-8, shift_jis, euc-jp, utf-16le, utf-16be, and
windows-1252:

const labels = [
  "utf-8", "shift_jis", "euc-jp",
  "utf-16le", "utf-16be", "windows-1252",
];
const candidates = labels.map((label) => {
  try {
    return new TextDecoder(label).decode(bytes);
  } catch {
    return ""; // invalid sequences for this encoding
  }
});

Step 2: the TextEncoder problem

The recipe above works when the reading was wrong. But in the Japanese
→ Japanese cases (saved as Shift_JIS, misread as EUC-JP — or the
reverse), restoration needs the opposite move: take the characters you
see and re-encode them back through the wrong charset to recover the
bytes.

And here the platform says no: TextEncoder only ever produces UTF-8.
There is no new TextEncoder("shift_jis").

The workaround is to build the reverse mapping yourself. Decode every
possible byte sequence with TextDecoder("shift_jis") once, and you
have a complete byte → character table for Shift_JIS; invert it and you
can map characters back to bytes. It sounds heavy — a 2-byte code point
has 65,536 combinations — but the table is built lazily and the whole
thing runs in milliseconds. This is also how you recover bytes from
Shift_JIS-style mojibake where several characters were merged, not just
re-mapped one-to-one.

Step 3: which candidate is right?

Brute force returns many "readable" results. The wrong ones tend to be
assembled from rare characters. 縺, 繧, 繝 — the first characters of
classic UTF-8-as-Shift_JIS garbage — are all JIS level-2 kanji,
characters that appear almost never in daily Japanese. The correct
reading, meanwhile, is made of level-1 characters and everyday kana.

That gives you a cheap, dependency-free ranking function: score each
candidate by how frequently its characters occur in normal Japanese
text, adding weight for common kanji and kana, subtracting for JIS
level-2 rarities. A frequency table built from a modest corpus is
enough; the right reading almost always floats to the top with a visible
margin.

The lone 縺・ at the end

Old Japanese mail software truncated messages at fixed byte lengths —
sometimes right in the middle of a multi-byte character. The victim is
always the last character: its leading byte survived, the continuation
bytes were cut. The result ends with something like 縺・ and every
decoder refuses it or prints a replacement mark.

These bytes can be resurrected. A cut multi-byte character leaves a
known fragment; enumerate the byte sequences that would continue it
(there are only a handful of legal continuations), re-decode, and pick
the completion that best fits the surrounding text. It is a guess in
the strict sense — but an extremely well-educated one, and in practice
it reads naturally in the overwhelming number of cases.

When text was garbled twice

Copy a mojibake sentence, save it in another tool, misread it again —
now you have double garbling, where the second wrong decoding was
applied to the output of the first. The fix is what you would expect:
run the whole pair search once, then run it again on the first
round's output. What you would not expect is the cost: the second
pass multiplies the work ninefold, so a sane implementation runs it
only on short texts — pasted sentences, never whole files.

Putting it together

The nine pairs, the byte-recovery tricks, naturalness ranking, the
trailing-character inference and the second pass all run in one page at
henkanya.com — free, no sign-up, and entirely
in your browser: the text you paste never leaves your machine. It also
handles the encodings that don't fit the brute-force model
(=?ISO-2022-JP?B?…?= mail headers, old JIS mail with lost ESC
sequences, Unicode escapes, HTML numeric references) and converts CSV
or text files to UTF-8 with BOM so that Excel opens them without
decorating your data with mojibake of its own.

If you want to see the pipeline fail — or better, save your afternoon —
paste your favorite garbage string and check whether the top candidate
is the sentence you remember.