The first time I watched a dubbed scene with two people talking, I almost laughed. The AI was giving every line the same flat voice. I couldn't tell who was talking. The film lost all of its tension, because half of a movie is just listening to people interrupt each other.
Fixing that meant solving a tiny question that turns out to be the whole battle: how does the dubber decide who speaks?
It can't watch the screen. It only hears audio. So it has to guess from sound alone.
The naive answer, and why it failed
My first idea was speaker diarization — the technique that splits an audio stream into "this person, then that person." It works great on clean podcast recordings. It falls apart on movie audio, which has music, effects, and two people occasionally talking at the same time. The labels got swapped mid-scene, and the dub kept handing lines to the wrong person. Worse, diarization is another model you have to run, and on a laptop every millisecond counts.
I backed away from the fancy approach and asked a dumber question: what actually separates most speakers, at a glance?
Pitch.
Men and women tend to sit in different vocal ranges. It's not a perfect rule, but it doesn't need to be. It needs to be right often enough to keep a scene readable.
Reading pitch off the incoming stream
Whisper already hands me timestamps for each spoken segment, so I know when someone talks. The missing piece was who. Before I translate a segment, I run the raw audio through a quick pitch estimator. I average the fundamental frequency across the segment and compare it against a threshold.
Below it, I route the line to the male voice. Above it, the female voice. And I relax the threshold with a little hysteresis, so a speaker who drifts up and down doesn't flip-flop every sentence.
avg_freq = estimate_pitch(segment_samples)
voice = 'female' if avg_freq > threshold else 'male'
if avg_freq < threshold - drift:
voice = 'female'
elif avg_freq > threshold + drift:
voice = 'male'
That pushed roughly a full step of difference into the no-reversal zone. A scene stops flickering between voices and just settles.
Making the two voices feel real
Two different voices on paper is cheap. Two believable ones is not. I run two Kokoro TTS voices and pick at render time, which keeps latency low because both stay warm. And I do something simple but effective: the original audio ducks under the dub while a line plays, so the reader's ear is mostly hearing my voices, not fighting the original soundtrack.
Is it perfect? No. Deep male and deep female voices occasionally brush the same range, and then the tool guesses. A whispered line can sit entirely on the wrong side. But for watching a film, it keeps who-is-talking clear, which is what subtitles give you and what a plain dub mostly doesn't.
That's the kind of feature that never shows up on a spec sheet. Nobody buys a dubbing tool because it detects pitch. But it's the difference between a dub you watch and a dub you tolerate.
If you've got a laptop and a movie you keep meaning to watch, the whole thing is one small local model and a free hour. I sell my own build for $19 at https://symshah.gumroad.com/l/livedub — it runs entirely on your machine. No uploads, no subscription, and it finally knows who's talking.
