Anatomy of a Voice Agent: VAD, STT, LLM, TTS and Why WebRTC Matters

Talking to a voice agent feels simple. You speak, it answers. But behind that half-second exchange sits a pipeline of models and network decisions, and every one of them can make the conversation feel natural or painfully robotic.

In this post, I'll break down the four core components of a voice agent, where latency hides, and why the choice of transport protocol matters more than most people expect.

The pipeline: four core components

Almost every voice agent follows the same high-level flow:

🎙️ User audio → VAD → STT → LLM → TTS → 🔊 Agent audio

1. Voice Activity Detection (VAD)

VAD answers one question: is the human speaking right now?

It sounds trivial, but it's foundational. Without VAD, you'd be streaming silence to your speech-to-text provider (and paying for it), and your agent would have no reliable signal for when to listen versus when to respond. A bad VAD is also the most common reason agents interrupt users mid-sentence.

2. Speech-to-Text (STT)

STT turns raw audio into text the LLM can reason about. Accuracy here sets the ceiling for everything downstream: if the transcript is wrong, the smartest LLM in the world will still give a wrong answer.

When choosing a provider, test with the languages, accents, and audio conditions your real users have, not just clean studio samples.

3. Large Language Model (LLM)

This is the brain. The LLM decides what to say next, and it's where your prompts, conversation state, planning logic, and tool calls (booking an appointment, looking up an order) live.

For voice, the key metric isn't total generation time. It's time to first token, because you can start speaking before the full response is ready.

4. Text-to-Speech (TTS)

TTS converts the LLM's reply into audio. This is the part users actually hear, so voice quality, pacing, and personality matter a lot for how your product feels. Again, the metric to watch is time to first audio byte, not total synthesis time.

The supporting cast

Beyond the core four, production agents rely on components that make them better conversational partners:

• Noise suppression filters out keyboard clicks, traffic, and fans.
• Background voice cancellation (BVC) removes other people talking nearby, so the agent doesn't respond to a TV or a coworker.
• End-of-turn detection goes beyond VAD. A pause doesn't always mean someone is done talking ("My account number is... uh..."). Smarter turn detection uses context to decide when it's actually the agent's turn.

Latency: the real enemy

In natural conversation, people start replying to each other remarkably quickly. For a voice agent to feel natural, a good target is around 500ms from the moment the user stops speaking to the moment the agent starts talking.

Here's roughly how the budget breaks down:

Notice the gap. The best case just squeezes under the target, while a typical setup is 2–4x too slow. That difference is what separates an agent that feels like a conversation from one that feels like a phone menu.

How low-latency agents close the gap

Stream everything. Don't wait for one stage to finish before starting the next. Stream partial transcripts into the LLM, stream LLM tokens into TTS, and stream audio back to the user as soon as the first chunk is ready.

Run work in parallel. While the user is still speaking, you can already be warming up connections, fetching context, or running speculative processing.

Avoid blocking I/O. A single synchronous database call or slow tool invocation in the hot path can blow your entire latency budget.

Choose providers deliberately. The most accurate STT or the most capable LLM isn't always the right choice. Pick the one that balances quality and speed for your use case, and measure it under realistic conditions.

Why WebRTC for voice?

Here's the part that surprises people: you can optimize every model in the pipeline and still end up with a laggy agent because of the network layer.

Real-time audio has very different needs than text. Let's compare the options.

HTTP (over TCP)

Great for request/response workloads like REST APIs, but a poor fit for live speech. TCP guarantees in-order delivery, so one lost packet holds up everything behind it (head-of-line blocking). HTTP also has no built-in understanding of audio: no timestamps, no jitter handling.

WebSockets (over TCP)

A step up: persistent and bidirectional, so you're not opening a new connection per message. But it still runs on TCP, so under packet loss you get the same retransmission stalls. On a shaky mobile connection, audio freezes and then arrives in a burst.

WebRTC (over UDP)

WebRTC was purpose-built for real-time media, and it shows:

• Opus codec: efficient compression that sounds good even at low bitrates.
• Per-packet timestamps: the receiver knows exactly when each chunk of audio belongs.
• Jitter buffering: smooths out packets that arrive unevenly.
• Adaptive bitrate: quality scales down gracefully when the network degrades instead of stalling.

The key insight: for live voice, a late packet is often worse than a lost one. UDP lets WebRTC skip missing data and keep the conversation moving, where TCP would stop and wait.

In practice, this means your agent delivers its first audio faster and stays responsive even on flaky networks, which is exactly where real users are.

Wrapping up

A voice agent is more than an LLM with a microphone. It's a pipeline where:

1. VAD, STT, LLM, and TTS each contribute to the experience and to latency.
2. Supporting components like noise suppression and turn detection make it feel human.
3. Streaming and parallelism are what get you under the ~500ms bar.
4. WebRTC keeps the transport layer from undoing all that work.

In the next post, I'll build a working voice agent using LiveKit's WebRTC infrastructure and measure latency at each stage.

What's been the hardest part of building voice agents for you: latency, turn-taking, or something else? Let me know in the comments. 👇