I build browser extensions that dub online course videos (Udemy, Coursera, YouTube) into the viewer's language. Going in, I assumed the interesting engineering would be the AI: the translation, the voices, the pipeline. It wasn't. The AI parts turned out to be mostly plumbing between APIs.
The part that actually ate my time was dumber and much harder: playing generated audio, in sync, inside a video player I don't own and can't modify.
One aside first, because someone always asks: I don't run speech-to-text. The platforms already ship a caption track, so I read that instead of transcribing the audio. Cheaper, faster, cleaner on technical vocabulary. Good decision, one line to explain, and not what this post is about. This post is about everything downstream of "I already have the text."
Problem 1: getting a second audio track to coexist with a player you don't control
You can't just play a new <audio> element and hope for the best. The original video is still sitting there with its own audio, its own controls, its own event handling. Your track has to duck the source audio, follow every play, pause, seek and rate change the user triggers on the native controls, and never break those controls in the process.
So the dub can't be something that plays alongside the video. It has to be slaved to the video element. You listen to the player's state and drive your audio from it, never the reverse. The player is the clock. You're a follower that can't desync from it, including when the user does something you didn't plan for.
Problem 2: translated speech is never the same length
A four-second English line runs six seconds in German. Do nothing and the dub drifts further out with every sentence until it's comically wrong by the end of the lecture.
There's no clean fix. Every option is a tradeoff. You can time-stretch the generated audio to fit the caption's time slot, which costs naturalness. You can let each line start on its caption timestamp and absorb the slack in the gaps between lines, which works when there's breathing room and falls apart on dense back-to-back speech. In practice it's a mix, tuned against the caption timing rather than the original audio, because the captions are the one timeline you can trust.
Problem 3: the user scrubs
The moment you've got playback synced, the user drags the scrubber to 12:30 and everything you lined up is wrong. Every seek has to re-resolve which caption line you're now in, and where inside that line's audio you should be. Same for a 10-second rewind, a chapter jump, a speed change. Each one is a re-alignment event, not just a play event.
Problem 4: three players, three DOMs
Udemy, Coursera and YouTube each expose their video element, their captions and their state differently, and they change their markup whenever they feel like it. The only structure that stayed sane: a platform-agnostic core (timeline, sync, audio scheduling, voice handling) with a thin per-platform adapter at the edge, whose whole job is "find the video element, find the caption track, read the player state." When a platform breaks something, you fix one adapter instead of the engine.
What I'd tell someone building something similar
- Treat the caption track as the source of truth for time, not the original audio. It's already aligned to the video, and it's the thing you're speaking.
- Slave your playback to the player, and never assume it'll behave. Seek, speed, pause, tab-switch and buffering are all things the player does to you.
- Push every platform-specific detail out to the edges. The core shouldn't know what Udemy is.
- The AI is the easy part. Voices and translation are a solved-ish, buy-it problem. Living inside someone else's player is where the real work is.
If you're building anything that overlays timed media on a player you don't control (dubbing, live annotation, alternate audio tracks) I'm happy to compare notes in the comments.
The extensions, if you want to see the result: Udemy · Coursera · YouTube
