Browser Voice Interaction AI Pitfall Guide 2026 — 16 Common Traps with AEC, getUserMedia, and Headless Modes

📝 Originally published (in Japanese) at forge.workstyle.tech.

When building voice-based AI interactions in the browser (avatars, voice bots, streaming AI), you’ll inevitably hit pitfalls stemming from audio physics and browser implementation quirks. This article compiles 16 traps I encountered during product development, organized in a symptom → cause → solution lookup format. No need to read from top to bottom—jump straight to the symptom you’re facing.

Echo and Self-Response Issues

1. Avatar Responds to Its Own Voice (Despite echoCancellation: true)

• Symptom: TTS audio is picked up by the mic, and STT recognizes it as user speech, creating a self-response loop.
• Cause: AEC (Acoustic Echo Cancellation) requires a reference signal (the "sound to cancel"). Only the browser's official playback paths (<audio> / WebRTC receiver tracks) serve as references. Custom playback via Web Audio API does not reliably function as a reference.
• Solution: Return TTS audio from the server as a WebRTC remote track and play it via an <audio> element. This eliminates echoes without text-matching workarounds (tested: 99 seconds of continuous speech with speakers on, zero false user turn detections).

2. Echoes Are Gone, but Speaking Simultaneously with the Avatar Distorts My Voice and Causes Misrecognition

• Symptom: Only during dual speech, proper nouns get mangled (e.g., "社員数" → "シャインズ"), especially at word beginnings.
• Cause: Fundamental AEC trade-off. To cancel echoes, AEC suppresses/distorts near-end (user) audio during dual speech.
• Solution: Mitigate in three layers: ① Increase mic Opus bitrate and enable FEC (see Pitfall 13) ② Provide vocabulary hints to STT (see separate article: use "recent avatar speech" as initial_prompt, not a dictionary) ③ Instruct LLM: "Input is STT transcription with potential errors. Interpret unnatural words as phonetically similar terms and add confirmation prompts."

3. Can’t Suppress Audio from Other Apps (Music, Videos)

• Symptom: Audio/lyrics from a YouTube video opened by the agent keep getting transcribed by STT.
• Cause: Browser AEC can only reference audio played by the same tab/app. Audio from other processes is indistinguishable from human speech to the mic.
• Solution: No technical silver bullet. Combine OS speaker separation (e.g., macOS "Voice Isolation"; request via voiceIsolation: true—ignored on unsupported systems), headphones, and downstream noise rejection (LLM/tool-based filtering).

"Only the Beginning Is Unheard" Issues

4. Initial Speech at Session Start Isn’t Recognized

• Symptom: First 10–20 seconds of speech go unanswered. Works normally afterward.
• Cause: Two factors: (a) AEC convergence—AEC only learns during actual playback, so unconverged dual speech suppresses the user's voice. (b) autoGainControl ramp-up—Gain increases gradually, making initial speech too quiet.
• Solution: (a) Play short greetings/sound effects before the main session to train AEC (loading screens are perfect for this). (b) Set autoGainControl: false. If STT is server-side, volume fluctuations are handled well, and disabling AGC has minimal downsides. Never disable echoCancellation.

5. Can’t Diagnose "No Response" Issues

• Symptom: Unclear whether the issue is mic off, preprocessing loss, or server-side—leads to guesswork.
• Solution: Add two observability points: ① Server-side logging of input audio energy/probability (10Hz. Distinguishes "complete silence" from "distorted audio"). ② Level meter in UI for the actual transmitted stream (createMediaStreamSource to read the same stream as transmission—doesn’t affect transmission).

6. Starting Mic Track with enabled=false Causes "No Response" Complaints

• Symptom: Implementation follows spec ("mic off until button press"), but users expect "always listening."
• Cause: Not a bug—mismatch between design contract and user experience expectations. A silent track looks indistinguishable from "quiet room" to the server.
• Solution: Align defaults with product promises. If "always listening" is a selling point, auto-enable on initialization completion (not during connection—that breaks mid-init conversations). Keep the button as a mute toggle.

Chromium Implementation Pitfalls

7. Remote Audio Connected to WebAudio Remains Silent (Lip Sync Doesn’t Work)

• Symptom: WebRTC receiver stream connected to createMediaStreamSource for analysis/processing, but no data flows.
• Cause: Chromium doesn’t send remote MediaStream to WebAudio until a media element starts consuming it (long-standing behavior).
• Solution: Attach srcObject to a muted <audio> element and call play(). The goal is to trigger playback, not produce sound.

const a = new Audio();
a.muted = true;
a.srcObject = remoteStream;
a.play().catch(() => {});
audioElRef.current = a; // Prevent GC by holding reference

8. MediaStreamAudioSourceNode Silently Dies

• Symptom: Input to processing graph stops after working for a while. No errors.
• Cause: Chrome may garbage collect unreferenced nodes, silently stopping input.
• Solution: Always retain references to nodes (and the above <audio> element) via ref or similar.

9. Volume Slider Doesn’t Work (New Path Only)

• Symptom: GainNode worked for local audio but stops working after switching to remote track playback.
• Cause: <audio> playback bypasses Web Audio GainNodes.
• Solution: Manipulate audioElement.volume / .muted. During transition periods, apply changes to both paths.

Headless Environment (Streaming/Automation) Pitfalls

10. Headless Chromium Connection Dies at Startup

• Symptom: Server never receives connection offer (offer).
• Cause: Initial getUserMedia throws an exception in headless environments (no mic).
• Solution: For mic-less use cases (streaming renderers), send a synthesized silent track via WebAudio. SDP and server pipelines behave identically to real mics.

const ctx = new AudioContext();
const dest = ctx.createMediaStreamDestination();
const keep = ctx.createConstantSource();
keep.offset.value = 0;  // Continuously stream "silent" samples (see Pitfall 11)
keep.connect(dest);
keep.start();
pc.addTrack(dest.stream.getAudioTracks()[0], dest.stream);

11. MediaStreamAudioDestinationNode Without Input May Stop

• Symptom: Silent track connection works initially but RTP stops after a while, causing disconnection. If connected to MediaRecorder, the muxer halts, stopping video too.
• Cause: Some implementations stop frame generation for destinations without input sources.
• Solution: As above, connect a ConstantSource(0) to keep rendering active.

12. Autoplay Policy Blocks Playback and AudioContext Startup

• Symptom: No sound in headless mode / AudioContext remains suspended.
• Solution: Launch flag --autoplay-policy=no-user-gesture-required + ctx.resume(). In regular browsers, always include a user interaction-triggered resume() to "unlock" audio.

Quality and Tuning Pitfalls

13. Default WebRTC Mic Bitrate Is Surprisingly Low

• Symptom: Switching from WS+raw PCM to WebRTC (Opus) reduced STT accuracy.
• Cause: Default Opus bitrate is ~30kbps. Lossy compression artifacts become critical under limiting conditions like dual speech.
• Solution: Modify answer SDP fmtp line to maxaveragebitrate=128000;useinbandfec=1 (answer-side fmtp controls sender encoder). FEC also helps with packet loss over TURN.

14. VAD Silence Wait Dominates Response Latency

• Symptom: Perceived 2-second response delay. Profiling shows neither LLM nor TTS is the bottleneck.
• Cause: Silence wait for end-of-speech detection (stop_secs) accounts for >1 second. Shortening this causes mid-sentence cuts (another failure mode).
• Solution: No silver bullet. Run STT concurrently during speech to preempt finalization, use turn detection models, and prioritize barge-in speed (instant interruption) for better perception.

15. "False Regression" in STT/TTS Latency Right After Deployment

• Symptom: Post-release tests fail. Wasted time debugging code.
• Cause: Model lazy loading/prewarming (e.g., serial TTS speaker model loads) monopolizes GPU/event loop, causing inference APIs to wait tens of seconds.
• Solution: Wait for prewarming completion before testing (check log counts for "load start" vs "complete"). Integrate warmup checks into E2E harnesses.

16. Server Pipeline Idle Timeouts

• Symptom: Sessions auto-terminate after 5 minutes of silence in valid use cases (streaming, monitoring).
• Cause: Default pipeline framework timeouts (e.g., Pipecat's 300s idle timeout) assume conversational use. "5 minutes inactive = abandoned" logic triggers self-cancellation.
• Solution: Separate settings by workload (e.g., cancel_on_idle_timeout=False for streaming routes). Treat framework defaults as potential hazards when use cases shift.

Checklist (Save for Later)

• [ ] TTS playback uses WebRTC remote track + <audio> (AEC reference path)
• [ ] Pre-convergence "training sounds" are played for AEC
• [ ] autoGainControl disabled / echoCancellation enabled
• [ ] Input energy logging + transmit level meter implemented
• [ ] Remote streams consumed by muted <audio> before WebAudio
• [ ] Node/element references retained (GC prevention)
• [ ] Headless paths use synthesized silent track + ConstantSource(0)
• [ ] Opus bitrate/FEC configured
• [ ] Testing waits for prewarming completion
• [ ] Idle timeouts reviewed per workload

Hope this saves fellow travelers in this swamp at least an hour of debugging!