The complaint that keeps showing up in every voice-agent beta test I've watched is not about the wrong answer. It's about the timing of the right answer.
"The AI cut me off." "It answered while I was still thinking." "I said 'um' and it started talking." Latency looks like a stopwatch problem, but users experience it as a manners problem, which is a much harder thing to fix.
The 200ms vs 700ms gap the field is trying to close
The ACL IWSDS 2025 survey on turn-taking in spoken dialogue puts a concrete number on the mismatch. In natural human conversation, a listener starts their reply roughly 200ms after the speaker's turn ends. Current dialogue agents take 700 to 1000ms to do the same thing.
Three to five times slower. If human dialogue is a tennis rally, current voice AI is playing chess by post.
Closing that gap is not one problem, it's two. And they pull in opposite directions.
- Too slow to reply: awkward silence, dropped rapport, users start repeating themselves.
- Too fast to reply: the agent talks over the user, interrupts mid-word, kills trust.
Both fail modes come from the same root cause: silence is not the same as end-of-turn.
Why plain VAD is not enough
Most current systems still decide the user is done talking with a Voice Activity Detector. VAD is cheap and fast, and it answers exactly one question: is there voice in this audio frame, yes or no.
That question is not the one the agent actually needs answered.
- The 0.3–0.5s pause a speaker takes mid-sentence looks identical to end-of-turn.
- The silence after "uh" or "let me think" looks identical to end-of-turn.
- A cough or a sigh looks like voice.
You can tune the silence threshold, but you cannot tune your way out of the fact that VAD does not know what a sentence is. VoiceInfra's production data lands on 300–500ms of trailing silence as the least-bad setting: shorter and the agent chops off natural pauses, longer and the delay becomes obvious. The comfortable range depends on the use case. Call-center flows tolerate 400–500ms because callers pause to think; command interfaces want 200–300ms because utterances are short; long-form narration wants 500–600ms because the pauses inside the story are longer.
The threshold is a compromise, not a fix. The real fix has to know what the words mean.
Semantic endpointing: Deepgram Flux and the "one model does both" bet
The interesting move in 2026 is folding transcription and turn detection into the same model. Deepgram's Flux, launched as their first ASR built for voice agents rather than for general transcription, does exactly that. The model outputs turn boundaries directly from audio, using the same joint architecture it uses to output text. Because the same weights see both the acoustic signal and the emerging transcript, the model has a shot at answering the harder question: "is this utterance semantically complete?"
Flux exposes three knobs that make the tradeoff explicit rather than hidden inside a silence threshold:
-
eot_threshold— how confident the model has to be before it commits to end-of-turn. -
eager_eot_threshold— a lower bar for tentative end-of-turn, useful for starting inference speculatively. -
eot_timeout_ms— the maximum silence duration before the model forces end-of-turn even if the confidence never crosses the threshold.
You can also switch modes per-turn with a Configure message, so a barge-friendly interaction ("agent, wait") and a monologue turn ("read me the terms") do not have to share a threshold.
The pattern is a hint at where the field is going. Turn-taking will stop being a pipeline stage after ASR and start being a joint output of the ASR itself.
Krisp's 6M-parameter turn model: same idea, edge-shaped
Not everyone can afford a cloud round-trip on every silence check. Krisp ships two small audio-only models trained to handle the acoustic cues a naive VAD collapses — a 9M-parameter turn prediction model (~30MB) that scores end-of-turn from prosody and pausing patterns, and a 6M-parameter interruption model (~24MB) that separates real barge-ins from backchannels like "yeah" and "mhm". Together they cover the same failure modes VAD misclassifies:
- intentional speech vs a thinking pause
- filler words (um, uh, well) mid-sentence
- backchannels vs interruptions
At that size it runs on-device in real time, which matters for privacy-sensitive deployments and for wearables where the round-trip to the cloud is itself the biggest source of latency.
The Krisp and Flux paths look opposite. One shrinks the model, the other gives the ASR the extra job. But they are attacking the same VAD failure mode from opposite sides.
Graceful abort: what to do when you're wrong
No detector, semantic or acoustic, is going to be right every time. The interesting question is what your pipeline does when it decides "the user is done" and turns out to be wrong.
Twilio's graceful abort pattern is the cleanest version I've seen written up. When STT signals end-of-turn early, the LLM starts generating a response. If fresh audio arrives before the response reaches the speaker, the pipeline kills the generation in flight and swallows the partial output. The window for this is the few hundred milliseconds it takes STT → LLM → TTS to produce audible speech. If you can revoke the guess inside that window, the user never hears it.
The economics matter here: you are paying for LLM tokens on turns that get thrown away. In exchange, you get to be aggressive on end-of-turn detection without punishing the user when you're aggressive-and-wrong. For most conversational products that tradeoff is worth it.
Barge-in: hearing the user through your own voice
The other half of turn-taking is the reverse case: the user interrupts while the agent is still speaking. Detecting that seems trivial until you realize the AI's own audio is leaking into the microphone, and any naive VAD will happily flag that leak as a barge-in and abort the agent's own utterance.
Sparkco's write-up on duplex barge-in handling is the clearest description of the fix I've read. Three moving parts:
- Full-duplex audio — keep the mic hot the entire time the agent is speaking, never gate it on TTS output.
- Echo cancellation — subtract the speaker output (as a reference signal) from the mic input, so what remains is the user's voice minus the agent's.
- Nuisance rejection — filter environmental noise and short transients that a bare VAD would misclassify as speech onsets.
[agent is speaking]
speaker out ──► reference signal ─┐
▼
mic in ─────► echo canceller ────► user voice only ──► VAD / turn model
Without echo cancellation you get the failure mode users describe as "the AI got startled by its own voice and stopped talking." The dog scared of its own reflection.
The bot-feel paradox and the 200-300ms delay trick
Now the counterintuitive move. Once you've closed the acoustic and semantic gaps and the agent can reply in 300ms, it turns out that replying in 300ms feels worse. The response arrives before the user has finished processing their own sentence, and it registers as robotic rather than sharp.
The fix is to inject 200–300ms of intentional delay before the agent starts speaking, while continuing to run the LLM in the background. You get the "thinking for a moment" cue humans read as attention, without paying real latency for it. The tokens are already streaming; you're just holding the TTS start.
It's a UX trick, not a technical one. But it's the piece of the puzzle that gets forgotten when a team spends a quarter shaving milliseconds off ASR and then wonders why user ratings didn't move.
What actually closes the 500ms gap
There is no single component that takes you from 700ms to 200ms. You get there by stacking:
| Layer | What it buys you |
|---|---|
| Semantic endpointing (Flux-class) | Stops chopping mid-sentence, stops waiting for silence that already means end-of-turn. |
| Small on-device turn + interruption models (Krisp-class) | Score end-of-turn and separate backchannels from real barge-ins without a round-trip. |
| Graceful abort | Lets you be aggressive on early end-of-turn without punishing the user when wrong. |
| Duplex barge-in with AEC | Lets you leave the mic hot without the agent interrupting itself. |
| 200–300ms intentional delay | Buys back the "thinking" cue that pure speed removes. |
The move that unlocks the stack is admitting that end-of-turn is a language problem, not a silence problem. Once you accept that, VAD-only pipelines look the way pre-BERT NLP pipelines look now: perfectly reasonable for their era, obviously incomplete in retrospect.
The gap to 200ms is not going to close by tuning thresholds. It closes when the model that hears the audio is the same model that understands the sentence.
The full 300ms-UX playbook (latency budgets by pipeline stage, when to pick Whisper vs Deepgram vs Piper, and how to design the wait states so users forgive the last 200ms you can't remove) is written up in Voice AI 300ms UX. Chapter 9 covers turn-taking end-to-end; chapters 4 and 5 cover the latency anatomy that decides whether closing the gap is even possible for your stack.
References
- ACL IWSDS 2025. "A Survey of Recent Advances on Turn-taking Modeling in Spoken Dialogue."
- Krisp AI. "Audio-only, 6M weights Turn-Taking model for Voice AI Agents." 2025.
- Twilio. "Core Latency in AI Voice Agents." 2025.
- Deepgram. "Flux: End-of-Turn Detection Parameters." 2026.
- Sparkco. "Optimizing Voice Agent Barge-in Detection." 2025.
- VoiceInfra. "Voice AI Prompt Engineering: Complete Technical Guide." 2025.
