I run a free video upscaler that works entirely in the browser: no upload, the video never leaves your machine. For sources up to 720p it runs a small super-resolution model (IMDN_RTE, about 100k parameters) through onnxruntime-web on WebGPU. Above 720p it switches to AMD's FSR 1.0 shaders, because at that point speed matters more than squeezing out a few extra pixels.
It worked fine on phone footage. Then I fed it anime, and it looked bad.
The problem: the model learned from photos
IMDN_RTE learned on DIV2K, a set of photographs. Photos have soft gradients, texture, noise. Anime has none of that: it's flat colour fills separated by dark, hard ink lines. So when the model enlarges a cel-shaded frame, it does what it learned to do on photos. It keeps the edges smooth. The lines come out a little blurry and a little grey, and the whole frame looks like a slightly out-of-focus scan.
My first instinct was the obvious one: throw a sharpening filter on top. That made it worse. A normal unsharp mask overshoots on both sides of an edge, so every black line gets a thin white glow next to it. On live action you barely notice. On anime, with its perfectly flat fills, those halos jump out immediately.
So I wrote a dedicated "anime mode" pass that runs on the GPU right after the upscale. It only touches luma (brightness) and leaves the colour channels alone, so it can't create colour fringes. It does three things.
Step 1: sharpen, but never overshoot
I still use an unsharp mask, but I clamp the result to the darkest and brightest luma values already present in the pixel's own small neighbourhood:
let Ls = clamp(L + u.sharpen * (L - blurred), localMin, localMax);
That one clamp is the whole trick. The edge gets steeper, but a pixel can never become brighter or darker than something that's already next to it. No overshoot means no halo.
Step 2: darken the lines, not the edges
Anime lines should be dark. But "darken edges" is the wrong rule, because not every edge is a line. A white shirt against a blue wall is an edge between two flat regions, and if I darken it, I get a fake outline around everything.
I tried a Difference-of-Gaussians darken first (the approach Anime4K uses). It did exactly that: every colour boundary got a dark rim, and the result looked like a cheap cartoon filter.
What works better is a morphological black top-hat: take a grayscale closing of the image (max filter, then min filter) and subtract the original. The closing erases any dark stroke thinner than the window, so closing(L) - L is positive only on thin dark lines. A boundary between two wide regions survives the closing, so it gets zero response. I add that response back as extra darkness, and only real ink lines get darker.
Step 3: thin the lines
Upscaled lines also come out too thick. For this I borrowed Anime4K's "thin lines" idea: compute the Sobel gradient, blur it into a smooth "edge field", then for each pixel sample the image a tiny step down that field's slope, away from the centre of the line. Lines shrink a bit, the transitions between flat regions get steeper, and flat areas don't move at all because the field is zero there.
The details that made it actually work
Everything scales with the upscale factor. A line that's 2 px wide in the source is 8 px wide after a 4x upscale. So every blur radius, the top-hat window and the warp distance multiply by k = scale / 2. With fixed radii, the pass looked great at 2x and did nothing at 4x.
Separable passes. The Gaussian blurs and the min/max filters for the top-hat all split into a horizontal pass and a vertical pass. The whole thing ends up as 9 compute passes plus a final warp, ping-ponging between two rgba16float textures.
Bands instead of full frames. A 720p video at 4x becomes 5120×2880. Two full-size half-float textures at that resolution cost about 236 MB, which is a great way to crash a laptop GPU. So I process the frame in horizontal bands of roughly 4 megapixels, with extra rows above and below each band. That halo equals the sum of every vertical kernel radius in the chain, so each band's output matches a full-frame run exactly, and the seams stay invisible.
A reference before shaders. I wrote the whole chain in numpy first, tuned it on real anime frames, and only then ported it to WGSL. The GPU output matches the numpy reference to f16 precision. Debugging a 10-pass shader chain by staring at video is miserable; diffing against a known-good array is not.
On the FSR path, it replaces a pass. FSR normally runs EASU (the upscale) then RCAS (its own sharpener). In anime mode I drop RCAS and run my pass instead, since mine already sharpens without halos, and sharpening twice just brings the halos back.
Things I'd tell you if you're trying this
- Don't reach for a bigger model first. The fix here wasn't more AI, it was about 350 lines of plain image processing that knows what anime looks like.
- Work in luma only whenever you can. You avoid a whole class of colour bugs for free.
- Clamp every sharpening step to its local range. It's one line and it kills halos.
- Test at every scale factor. Anything with a radius in it needs to scale with the output.
If you want to see it on your own clips, the tool is here: squishyfile.com/video-upscaler. Tick "Anime / cartoon" before you hit start. It needs a browser with WebGPU (recent Chrome or Edge works); without WebGPU the upscale still runs, just on the CPU and without the anime pass.
I'm happy to go deeper on any of the passes in the comments.
