Storing Legacy Encodings as UTF-8 Inside the Index: 5.4x to 38.8x Over ripgrep -E, and the Line-Number Bug That Nearly Stopped It

The decode ran on every query.

I build a search tool for very large text files, and one of its two commands keeps a persistent index. On a 3 GB Shift-JIS file, that index removed the second read of the file but not the second decode — every query still turned 3 GB of CP932 bytes into characters before it could decide anything.

This post is about moving the index to UTF-8: what it bought, what it cost, and the one decoding detail that quietly corrupted line numbers until we caught it.

Up front: the numbers

Measured on a Mac (Apple M4, 10 cores, 32 GB, external USB SSD), on OpenStreetMap XML (2,892 MB) converted with iconv to each encoding, against ripgrep 15.2.0 with -E <encoding> so it decodes the same text. Cold means the page cache was purged and ten seconds elapsed; hot is the run straight after. Two runs combined:

And the loss, which matters more than the wins: the pass that builds the index is slower than ripgrep — 4.93 s vs 3.35 s on Shift-JIS, 5.01 s vs 3.32 s on EUC-JP, 7.76 s vs 6.53 s on UTF-16LE. For a file you search once, rg -E is the right tool and nothing here changes that.

Output was compared against ripgrep on 7 patterns across 3 encodings — 21 queries, byte-identical, same exit codes.

Why the index kept the source encoding in the first place

Not an oversight. The index records the byte offset of every line in the source file, and several features depend on those offsets lining up: opening the original file in the viewer and injecting an index a CLI run produced, and restoring the original file from the index.

Convert the text and every offset moves. A two-byte Shift-JIS character is three bytes in UTF-8, so line 400,000 no longer begins at the same byte. An index with shifted offsets injected into a view of the source file shows the wrong lines.

So the change was not "call a converter". It was: convert, and then make sure a converted index can never reach a code path that expects source offsets. In practice that meant giving the converted format its own version number, refusing to read older ones, and making the verification step (length plus timestamp, with the header timestamp written as 0) fail by construction for every injection path. The converted index is always opened standalone.

The part that is useful without my tool: three things that bite when you move text to UTF-8

This is the section I would have wanted to read before starting.

1. A naive decoder can eat your newlines. The default CP932 decoder in .NET treats 0x82 0x0A — the lead byte of a two-byte character followed by a line feed — as one character, and the newline disappears. Decode a damaged or truncated Shift-JIS file as a single stream and you silently end up with fewer lines than the file has, with every line number after that point wrong. Our fix: when the newline count of a decoded region does not match the source, decode that region line by line. If you are writing anything that reports line numbers, test with a file whose last character is cut in half.

2. "Round-trips cleanly" is not one property, it is two. Character-identical and byte-identical are different promises. Characters with two code points — the NEC-selected IBM extensions in CP932, where the same glyph exists at ED40 and FA5C — come back with whichever one your decoder normalises to. We ended up promising character-identical only, and saying so in the help text. Dropping the byte-comparison pass on the way in also took index construction from 9.67 s to 5.33 s, so the weaker promise paid for itself. If someone downstream verifies archives by hash, they need to know which promise they have.

3. Replacement characters are a UI decision, not a correctness one. The spec said use U+FFFD for undecodable bytes. But the viewer already displayed "・" for them, because that is what the platform decoder does. Storing U+FFFD in the index would have changed search output for existing users. We stored "・" instead, and the output stayed byte-identical to the previous release. Pick whichever you like, but pick it once, and make the decode path and the display path agree.

A fourth, smaller one: UTF-16 at scale found a latent bug. Searching a 6 GB UTF-16 file in its original encoding failed with Specified argument was out of the range, because the carry-over bytes of a character split across a read boundary could exceed the read window. Files up to about 1,000 MB were fine, which is exactly why it survived so long. If you buffer-and-decode, write a test at a size larger than your buffer arithmetic assumes.

What the conversion actually bought

On a hot cache, the per-query numbers for the Shift-JIS file land on top of the numbers for the same content stored as UTF-8: 0.15 s vs 0.15 s for a plain string, 0.33 s vs 0.34 s case-insensitive, 0.30 s vs 0.32 s for a regex, 0.53 s vs 0.54 s for an inverted match.

That is the whole point, stated as a general rule: an index can absorb the per-query transformation, not just the per-query read. Decoding, newline counting, normalisation, case folding — these are not reads, they are "turning bytes into characters", and most of them can be done once at build time and stored.

The price is that your offsets stop matching the source. Which is a design problem, not a performance one: either you collapse to a single path, or you support both and test the product of every combination. I wrote the option to keep the source encoding, then deleted it, for exactly that reason.

What still does not work

• EUC-JP supplementary kanji (JIS X 0212) are not decoded; they come back as "・".
• Restoring to the original encoding got slower first: 7.88 s against 3.44 s for the previous build, which never converted at all — the old build 2.29x faster. Rebuilt as 8 MB chunks decompressed and converted in parallel and written in order, with chunk boundaries aligned to UTF-8 character boundaries (each chunk reads three extra bytes ahead; continuation bytes at the head belong to the previous chunk). That brought it to 1.03 s hot; in the release measurement it was 3.52 s cold, which includes writing 3 GB to disk.
• Everything above is measured on one machine. The Windows and Linux builds pass the same correctness tests, but I do not compare seconds across operating systems.

I make this tool — uvf is the free build, uvp is the paid one — so treat the ratios as the author's own measurements and the method as the part worth checking.

• Full write-up with every pattern and timing: https://uvp.y42u.net/en/blog/uwview-fs01-sjis-utf8-index-en/
• Source: https://github.com/amru195704/UwView