ffmpeg.wasm to WebCodecs: the 12x Gap, What Broke, and What Still Can't Move

When we built the local-processing mode of our media toolbox (meicut.com) — 20+ tools that trim, compress, convert, and filter video and audio entirely in the browser — the first version bet everything on ffmpeg.wasm. It felt like the obvious choice. Within weeks, the hot paths moved to WebCodecs hardware encoding, ffmpeg.wasm survives as a fallback plus the tools that sit at hard boundaries — audio processing, speed change, AVI/GIF output, reverse — and we have a precise map of where each technology's real boundaries are.

This is the full story: why we started with WASM, the three technology-inherent problems that forced the migration, how the WebCodecs pipeline works, and the decision framework we now use to route each tool.

Why we chose ffmpeg.wasm first (and it was the right call)

Our backend already processed everything server-side with the ffmpeg CLI. That made ffmpeg.wasm almost unfairly attractive:

1. Zero redesign of processing semantics. Our server code was full of carefully-tuned filter graphs — -ss/-t input-side seeking for cuts, even-dimension rounding for crops, maskedmerge + geq three-plane masks for mosaic regions. With ffmpeg.wasm, the browser executor could reuse these line by line. Local and cloud outputs were identical by construction.

2. Complete capability coverage. ffmpeg handles every container, codec, and filter we needed: atempo for audio speed, MP3/FLAC encoding, AVI/GIF muxing, complex filter chains. WebCodecs, by contrast, is a codec primitive — it encodes and decodes raw frames. It doesn't demux your MKV, it doesn't resample your audio, and it definitely doesn't do WSOLA time-stretching.

3. 28 tools, one UI. Because the processing semantics came for free, we could share the entire frontend between cloud and local modes and ship every tool at once.

We knew WebCodecs existed. We also knew its caveats: uneven browser support, no MP3/FLAC encoders, and opaque hardware encoder behavior. Betting on WASM first let us ship. That sequencing mattered — if we'd started with WebCodecs, we'd have spent weeks just on demuxing before shipping a single tool.

Problem 1: Slow, and it sounded like a jet engine

ffmpeg.wasm's x264 is software encoding compiled to WASM — no SIMD in practice, no hardware acceleration. Compressing a high-resolution 2.5K clip made laptop fans spin up like we were mining crypto. Competitors using WebCodecs hardware encoding were silent and an order of magnitude faster. Our benchmark: same 10.5 MB / 720p / 10 s file, medium quality, median of 3 runs on the same machine:

(The output sizes differ because the two paths use different rate-control strategies — more on that below. Both were visually fine; the point is the 12× speed gap and the fan noise.)

For a free tool whose entire pitch is "it just works in your browser", CPU-bound software encoding is a UX ceiling you can't design around.

Problem 2: Outputs that came out bigger

Pure CRF rate control targets quality, not size. Feed it an already-lean source (3.30 Mbps) and CRF 28 happily produces a larger file (3.91 Mbps). Users compress a video to make it smaller; an output that grows is a bug report waiting to happen, no matter how defensible the encoding theory is.

CRF-driven inflation is the flavor we first hit, and it is squarely a WASM-path failure. But "outputs that come out bigger" is not exclusive to WASM: the hardware encoder in WebCodecs uses target-bitrate semantics and will happily spend the whole budget — without a per-pixel ceiling, a low-res "smaller" encode can land at 5 Mbps just because the target said so. The dual ceiling in the migration section is what stops that on both paths.

Problem 3: The 32 MB engine bootstrap is a reliability surface

The WASM engine is a ~32 MB download (core JS + WASM + worker), fetched on first use. Around that download sits an entire bootstrap chain — progress reporting, caching, ff.load() initialization, pthread startup for the multi-threaded core — and we learned that every link in that chain can fail silently:

• Our progress denominator was an estimate (34,135,000 bytes); the real download was 32,849,385 bytes — 96.2% of it. So the bar froze at "96%" even on healthy loads, and users reasonably read that as a hang. (Fix: exact byte counts per engine file, updated every time the core package is bumped.)

• On some machines, ff.load() never resolved — no error, no rejection, just silence. We added a 60 s watchdog that terminates the attempt and falls back to the single-threaded core.

• We ultimately traced one production-wide outage of this bootstrap to a server configuration bug of our own making — a missing HTTP response header that Chrome requires before it will construct a module worker. But the larger lesson stood: the engine bootstrap is infrastructure, with the failure modes of infrastructure — while WebCodecs has no bootstrap at all. The browser already contains everything it needs.

That contrast is what turned "WASM is slow" into "the hot paths should not depend on a downloadable engine at all".

The WebCodecs migration

The pipeline

input file
  → mediabunny (demux; we use it for muxing too — MP4 with faststart)
  → VideoDecoder (hardware) → optional per-frame canvas processing
  → VideoEncoder (hardware H.264)
  → audio: copied passthrough when container-compatible, transcoded otherwise

First interaction runs immediately — no warmup, no download progress, no load watchdog.

hwOrSw: hardware first, always a way back

Every migrated tool is wrapped in the same pattern:

async function hwOrSw<T>(hw: () => Promise<T>, sw: () => Promise<T>): Promise<T> {
  try {
    return await hw()
  } catch (e) {
    if (e instanceof LocalCancelledError) throw e // user cancel: never "fall back" into more work
    console.warn('[local] hardware path failed, falling back to wasm:', e)
    return sw()
  }
}

Any failure in the hardware path — unsupported codec, decode error, encode error — silently lands on the proven WASM executor. A localStorage flag forces all-software for production debugging.

The stall watchdog

Individual files can freeze hardware decoders/encoders silently. We carried over the lesson from the engine-load watchdog: bounded waits, never infinite hangs. If progress doesn't advance for 30 seconds, we abandon the hardware attempt and fall back.

Rate control: the dual ceiling that stopped inflation

Hardware encoders use target bitrate semantics — they spend the whole budget, unlike CRF which backs off on easy content. And we needed a guarantee: output must be smaller than input. Two caps, take the minimum:

target = min(
  sourceVideoBitrate × qualityRatio,      // 0.8 / 0.6 / 0.4 — guarantees smaller
  outputPixels × fps × bpp                 // quality ceiling: 0.2 / 0.15 / 0.1
)

The first term kills the reverse-inflation bug. The second is the subtle one: without a bits-per-pixel ceiling, a 480p "smaller" compression would cheerfully burn 5 Mbps of hardware-encoder budget because the target said so.

Preflight with real dimensions

H.264 level support varies with resolution/bitrate/framerate. Asking VideoEncoder.isConfigSupported() about a hypothetical config misses machines that encode 480p fine but choke on 4K. We probe with the actual output dimensions before committing to the hardware path.

The honest hard-boundary list: what stays on WASM

A full audit of every tool produced the list WebCodecs advocates rarely publish — what it genuinely cannot do today:

Everything else — compress, mute, rotate, flip, format conversion (MP4/MOV/MKV/WebM), single-segment trim, center crop, mosaic blur, sketch effect — runs on hardware.

The decision framework we ended up with

Is the capability expressible as decode → optional canvas → encode?
├─ No (audio processing, exotic containers, complex filter graphs)
│    → ffmpeg.wasm, and that's final
└─ Yes
     → Does the browser support it with THESE dimensions/codec?
          ├─ Yes → WebCodecs hardware path
          │         (30 s stall watchdog, any error → wasm)
          └─ No  → ffmpeg.wasm

Three principles held up across the whole migration:

1. Bounded waits everywhere. Both paths have watchdogs (60 s engine load, 30 s stall). Silent infinite hangs are the defining failure mode of in-browser media processing; every wait must have a ceiling and a fallback.

2. Fallbacks are architecture, not error handling. hwOrSw isn't a try/catch — it's the product. Users on odd hardware get slower results, never failed ones.

3. Match the technology to the capability, not the other way round. WASM-first got us coverage and correctness; WebCodecs-second got us speed where it matters. Neither was a mistake — the mistake would have been treating either one as the whole answer.

The result: the tools people actually hammer (compress, convert, trim) run at hardware speed with no engine download, the long tail still works everywhere ffmpeg works, and the 32 MB WASM engine only ships to browsers that genuinely need it.

This pipeline powers the local-processing mode at meicut.com, where you can try both paths on your own files.