Erasing text from manga in the browser, without an inpainting model (and where that ran out)

imgtranslator.com translates the text in an image and redraws it in place: a menu, a product photo, a manga page. OCR and translation run on a server. Erasing the original text and typesetting the translation happen on a canvas in your browser.

That erasing step is the subject of this post. The obvious tool for it is an inpainting model, which I deliberately didn't use for the first two months. A generative fill can quietly change things that weren't text, like a price on a ticket or a face behind a speech bubble, and nothing in its output tells you it did. So I wrote the eraser by hand, as pixel rules.

What follows is how the second version of those rules worked, how I measured it when there's no ground truth for a real photo, and the point where more rules stopped helping, which I can show with numbers.

The input

For each line of text, OCR gives me a box. Sometimes it also gives per-character boxes, depending on the engine. The eraser gets the image and the boxes and has to make the text inside each box disappear, while leaving everything else alone.

"Everything else" is the hard part. In manga it includes the speech bubble's outline 2px from the text, the neighbouring column of text, screentone, and line art drawn right through the lettering.

Version 1: find the text colour, grow the strokes

The first eraser read the background from the strips above and below the box, decided the text colour was whatever contrasted most with that, seeded from pixels of that colour, and grew the strokes outward. Then it filled the strokes in from their surroundings.

On clean UI screenshots and product photos it was fine. Then I ran it over 166 real uploads (2,115 boxes) and sorted the failures:

• 68% were the growth flooding the window. Black text touches a bubble outline or line art, and growing from one floods into the other. A bubble's white gets read as the text colour.
• 45% of all boxes were vertical columns. In a vertical manga column, "above and below the box" is the artwork, so the text colour was wrong before anything started.

Neither was a tuning problem. The model of "text = the contrasting colour next to the background above and below" doesn't describe a manga page.

Version 2: what's inside the box that isn't around it?

The second eraser asks a different question. Text is whatever colour is common inside the box and rare just outside it. That's true for black on white, white with a black outline, coloured title lettering, and horizontal or vertical text. It needs no text colour up front.

1. Cluster the colours of the box plus a narrow ring around it (k-means, k=6, clusters closer than 22 RGB units merged). The ring goes all the way round, and other OCR boxes are excluded from it, so a neighbouring column doesn't count as "background".
2. A cluster is text when its share inside the box is at least 3× its share in the ring, and it covers at least 1.5% of the box. Anti-aliasing and outlines form their own clusters and are enriched too, so they come along automatically.
3. Mask the text-cluster pixels near the box. A connected run of text colour that's mostly outside the box and reaches the edge of the window is structure (a bubble outline, a panel border, line art) and is left alone.
4. Fill the mask by push-pull interpolation: build a pyramid that averages only known pixels on the way down, then fills holes from coarser levels on the way back up. It's smooth, fast, and has no opinions.
5. Judge the result the way a viewer would: compare edge energy inside the box after the erase with the edge energy of the background around it. When it fails, cover the box with a plain rounded patch and write the translation on that.

The second row in that figure is worth a look. On screentone, the dots are black and white too, so the letters' own colours are already common in the ring. Nothing reaches 3×, and the sound effect isn't erased as text. Keep that one in mind for the end of the post.

Two details that each took a failed version to learn:

• The ring has to be narrow (0.3× the glyph size). A wide ring reached past a button into the page behind it. The button's own colour then looked far commoner inside the box than around it, and was erased as "text". On synthetic tests, panels came out 89–99% covered.
• Black-and-white manga needs a second route. On a dense B/W page the anti-aliasing splits into four or five grey clusters, none enriched on its own, and the ring is 13–26% dark from other ink. So there's a separate "far enough from the background colour" route for dark ink, kept only as blobs that sit mostly inside the box. The two routes do connected components separately; merging them first threw lettering drawn on the art away as line art.

The result

Here are both versions on a panel I drew for this post, run with the production code and nothing changed. v1 leaves half the dialogue behind. Both of them report all five boxes as clean.

And on real uploads:

On 327 synthetic cases (real images as backgrounds, rendered text on top, so the answer is known), leftover original text went from 0.130 to 0.091, and damage to the surroundings from 0.137 to 0.067. Vertical text in speech bubbles went from 0.303 leftover to 0.

It lost in two places: horizontal CJK on busy backgrounds (0.079 → 0.082) and outlined horizontal text (0.201 → 0.241). I shipped it anyway, because the categories it won are most of the traffic.

The one-line bug that drew a dotted line in every bubble

51% of real boxes overlap another box, because in manga the gap between columns is narrower than the boxes OCR draws around them. Push-pull was using the neighbouring column's grey anti-aliasing as "known background", so every erased bubble had a faint dotted grey line where two columns met. The self-check called those boxes clean.

The fix was to stop treating ink inside other boxes as a fill source. The patch count didn't move (82 → 82), but 68 of 166 images changed. The synthetic tests couldn't see this bug at all. Only looking at real output did.

Measuring when there's no answer key

The table above uses the eraser's own verdict. That's a problem, because the verdict is part of what I was trying to improve.

For real images there's no ground truth. So I built a labeller. It crops each box before and after the erase and asks a headless claude -p run (Opus) to rate it as one of four labels:

• clean
• faint: visible only up close
• trace: visible at normal viewing
• damage: a smear, the wrong colour, or screentone wiped flat

It runs locally, on my machine, as an offline batch job.

The first thing it measured was v2 itself:

About 400 boxes (19%) shipped with a problem while the eraser judged them clean. Of the 96 boxes it did patch, 36 had actually been erased fine. Its main self-check signal, post-erase edge energy, scored an AUC of 0.53 against the labels, which is a coin flip.

Then I measured the labeller. I labelled the same 296 crops twice:

• The two runs agreed on the exact label 85% of the time, and on "bad vs. not bad" 93% of the time.
• My original method compared versions on a weighted sample of boxes. With that much labeller noise, its margin of error was about ±0.017 in utility (clean = 1, faint = 0.6, everything else = 0). That was larger than most of the differences I'd been acting on.

The fix was to label only boxes whose output visibly changed between two versions, label both versions, and compare them as pairs. On around 400 changed boxes that brings the error down to about ±0.003. Several earlier "improvements" didn't survive the switch.

Four things that made it worse

With a ruler I trusted, I tried the obvious next steps. Baseline utility was 0.708.

1. Texture fill. After the smooth fill, copy high-frequency detail from a nearby clean region, so erased text on screentone isn't a smooth smudge. Result: 0.516. The "clean region" next to manga text usually has a bubble edge, another glyph or line art in it, and copying that added new strokes. A second version that also widened the mask on textured backgrounds scored 0.443.
2. Block matching (multi-scale PatchMatch) instead of push-pull. Of 375 changed boxes, 44 improved and 45 got worse, and it was 8× slower (0.13 s → 0.94 s per image). Where the mask had missed some ink, it found that ink and copied it in. So the fill wasn't the bottleneck.
3. A learned stroke mask. A small U-Net predicts the text pixels in a window, including outline and shadow. I trained it only on synthetic text over wallpapers and procedurally generated screentone and line art, never on user uploads. On synthetic validation, IoU went from 0.84 to 0.88. On real boxes, utility went down by 0.065 when it replaced v2's mask, and by 0.007 when the two masks were combined. It fixed about 60 of the 409 boxes v2 had ruined, and broke more clean ones than that. The higher its synthetic score, the worse it did on real pages. The gap is in the data, not the model.
4. Patch more aggressively. If the self-check is weak, just patch more boxes? Patching every box scored 0.643. Patches look worst exactly where erasing does too, on textured backgrounds: 50% of patches looked fine there versus 67% of erases. The best threshold-based rule I could fit scored 0.712, or 0.704 under 5-fold cross-validation. That's no improvement.

That's the evidence behind the conclusion I wrote in the project notes on 2026-09-27: hand-written rules have reached their end. Every remaining lever (flat backgrounds, the fill, the patch decision) measured at zero. The bad boxes left are large lettering on textured art, and they need either training data that looks like real pages or a fill that understands content.

What I did instead

For manga sound effects, I stopped erasing them. Splitting 65 vertical pages by background showed that dialogue inside bubbles erased cleanly. Almost all the bad boxes were sound effects drawn directly on the art. Scanlation groups solved this years ago: leave the original lettering and put a small translation next to it. On black-and-white pages that's now the default. On 84 lines from 150 held-out pages, it was good on about 67, neutral on 8 and bad on 9.

On colour pages the same rule mostly caught small bubbles and translucent ones (12 good, 21 bad), so it stays off there until I can detect bubbles reliably.

I tested inpainting properly, scoped to patched boxes only. LaMa (the Apache-2.0 ONNX export), with a DBNet text-detection mask:

• Synthetic cases: leftover text went from 0.239 to 0.099 on the hardest categories (outlined, shadowed and Latin text), with damage from 0.101 to 0.022. It won 166 cases and lost 4.
• Real boxes: about 15 of the 22 patched boxes I checked by eye came out better.
• Speed: on a small CPU server it took 10.4 s per box (p50). Going from 1 to 2 threads didn't help at all, probably because the two vCPUs are hyperthreads of one core.

That rules out "click and wait". The next step is running it in the background for patched boxes only, and swapping the result in when it's ready. Boxes v2 already erases cleanly stay with v2, because on those LaMa was flat or occasionally worse.

What I'd keep from this

• Sort failures by cause before tuning anything. "68% are flooding, 45% of boxes are vertical" told me v1's model was wrong, which no amount of threshold tuning would have shown.
• Your self-check is a model too, and probably a bad one. Mine had an AUC of 0.53. Measure it before you build on it.
• Measure the labeller's noise before reading its results. An LLM judge that agrees with itself 85% of the time can't tell apart versions that differ by 1%. Paired comparison on changed items is far cheaper than more labels.
• Synthetic wins don't transfer by default. Two of my four failed ideas looked clearly better on synthetic data and clearly worse on real pages.
• Sometimes the fix is in the product, not the algorithm. Keeping the original sound effects took one rule, and the result looks like what manga readers already expect.

The eraser is about 1,100 lines of TypeScript on Uint8ClampedArray, with no dependencies. If you're working on the same problem and want the numbers behind any row above, ask in the comments.