It's 7:46 PM on a Tuesday. A homeowner's kitchen sink is leaking and she's calling the first plumber in her search results. The office closed an hour ago. She hears it ring twice, then someone picks up, asks what's wrong, and books her a morning appointment. She hangs up relieved.
She didn't talk to a person. She talked to AI receptionist software, and she probably didn't notice the difference. But what actually happened between the ring and the booked appointment? If you're a developer, there's a real pipeline behind that 90-second call worth pulling apart.
This post walks through the technical pipeline that fires when an AI agent picks up a phone call: telephony handoff, audio stream capture, speech recognition, intent classification, slot filling, and backend actions like booking and CRM updates. We'll look at real latency numbers, where the architecture differs from text chatbots, and where voice AI still falls apart.
What happens in the first 500 milliseconds?
The phone rings. Here's what fires before the caller hears a single word back.
The call comes in through a SIP trunk or a cloud telephony provider like Twilio, Vonage, or Telnyx. The provider routes it to the AI system's endpoint, which opens a bidirectional audio stream. Most systems use WebSocket connections to pipe raw audio in both directions with minimal buffering. If you want to see what that webhook-driven call control layer looks like in practice, there's a solid walkthrough on routing phone calls to an AI agent with the Telnyx Voice API.
The AI agent needs to respond fast enough that the caller doesn't think the line is dead. In normal human conversation, people expect a response within about 200–600ms. Anything over a second feels broken. So the system has to start processing audio almost immediately, usually with a streaming architecture that begins transcription on the first audio frame rather than waiting for the caller to finish their sentence.
A simplified version of what that looks like:
Incoming SIP INVITE
→ Telephony gateway accepts, opens RTP stream
→ Audio chunks (20ms frames) forwarded via WebSocket
→ ASR engine begins streaming transcription
→ TTS-generated greeting queued and played back
→ Total time to first response: ~300–500ms
That greeting is usually pre-cached. The system isn't generating "Hi, thanks for calling" on the fly. It's playing a pre-rendered audio clip while the real processing pipeline warms up behind it.
How does the AI understand what the caller is saying?
This is where automatic speech recognition (ASR) does the heavy lifting. The audio stream gets fed into a speech-to-text engine (Deepgram, Google Cloud Speech-to-Text, Whisper, or one of the newer real-time models), which outputs a rolling transcript.
Modern ASR engines hit word error rates around 5–8% on clean audio with American English, according to benchmark comparisons published by Deepgram and Google in 2023–2024. That number climbs with background noise, accents, or speakerphone calls. A caller standing next to a running dishwasher is a harder problem than a caller sitting in a quiet office.
The transcript isn't the end of the story. Raw words need to become meaning. That's the natural language understanding (NLU) layer. It takes "yeah I need someone to come look at my sink, it's leaking under the cabinet" and extracts:
{
"intent": "book_appointment",
"service_type": "plumbing",
"issue": "leak_under_sink",
"urgency": "routine",
"preferred_time": null
}
The NLU step is where most of the interesting engineering happens, and where voice agents diverge from text chatbots. In text, you get a clean, complete message. In voice, you get partial utterances, mid-sentence corrections, filler words ("um," "like," "so basically"), and crosstalk. The system has to handle all of that without asking the caller to repeat themselves constantly.
How does it decide what to do next?
Once the system has an intent and a set of extracted slots, it enters dialogue management. This is the part that decides what to say next.
If the caller said they want an appointment but didn't mention a time, the agent needs to ask for one. If they asked about pricing, it needs to pull from a knowledge base. If they said "I need to talk to someone right now," it needs to route to a human.
This works through a state machine or, in more modern setups, a prompted LLM that maintains a dialogue state. The state tracks what's been said, what's still needed, and what the next action should be.
The voice-specific challenge here is turn-taking. In a text chat, turns are explicit: the user sends a message, the bot responds. On a phone call, people interrupt, talk over each other, trail off, and pause mid-thought. The system needs to handle:
- Barge-in detection: the caller starts talking while the AI is still speaking. The system needs to stop its own audio and listen.
- Silence detection: how long to wait before assuming the caller is done. Too short and you cut them off. Too long and the call feels sluggish. Most systems tune this to 700–1200ms depending on context.
- Backchanneling: the little "mm-hm" and "got it" sounds that signal active listening. Some systems generate these; most don't bother yet.
This is where latency really matters. Every step in the pipeline (ASR, NLU, dialogue decision, response generation, TTS) adds delay. The total round-trip from "caller finishes talking" to "AI starts responding" needs to stay under about 1–1.5 seconds to feel conversational. That's tight, especially if you're running a large language model for response generation.
What about booking, routing, and CRM updates?
The AI agent isn't just talking. It's doing things.
When the dialogue manager determines the caller wants to book an appointment, it fires API calls to whatever calendar system the business uses: Google Calendar, Calendly, Outlook. It checks availability, proposes a slot, and confirms it. That booking hits the calendar as a real event with the caller's details attached.
The same goes for CRM updates. A new caller gets a contact record created in HubSpot, Salesforce, Zoho, or whatever the business runs. The call transcript, extracted intent, and outcome all get logged. If the business uses Zapier or a similar automation layer, downstream workflows trigger automatically: follow-up email, technician dispatch notification, Slack message to the owner.
AgentZap published a complete guide to AI receptionists covering setup, integration paths, and use cases across industries if you want the full implementation picture.
For the developer, the interesting bit is that all of this has to happen while the call is still active. The AI confirms the appointment to the caller in real time. It can't say "I'll book that for you" and then fail silently on the backend. Error handling here is critical. If the calendar API times out, the system needs a fallback ("Let me have someone confirm that appointment with you by text, what's a good number?") rather than dead silence.
Where does it still break?
Plenty of places. If you're evaluating this tech or building on it, here's where to look for cracks.
Accents and dialects remain a real problem. ASR accuracy drops significantly for non-standard speech patterns. A 2020 study by Koenecke et al. published in PNAS tested five major commercial ASR systems and found word error rates of roughly 0.35 for Black speakers versus 0.19 for white speakers, nearly double. Non-native English speakers and strong regional accents hit similar walls, though fewer large-scale studies have quantified it as precisely.
Background noise kills accuracy. A caller on a construction site, in a car with the windows down, or in a crowded restaurant is going to generate a messy transcript. Noise-cancellation preprocessing helps, but it's not magic.
Multi-intent turns are tricky. "I want to reschedule my Thursday appointment and also ask about your weekend rates" contains two separate intents. Many systems handle the first one and drop the second.
Emotional callers are hard. Someone calling in a panic because their basement is flooding needs a different interaction than someone casually shopping for quotes. Sentiment detection exists but it's crude. Most systems don't adjust their tone, pacing, or escalation behavior based on caller emotion in any meaningful way.
And sometimes the right answer is just to hand off to a human. The best systems know when to do this. The worst ones keep looping through their dialogue tree while the caller gets increasingly frustrated.
FAQ
Can AI receptionist software handle multiple calls at once?
Yes. Each call runs as an independent session, so the system can process dozens or hundreds of concurrent calls without degradation. It's limited by compute resources, not attention. That's one of the clearest advantages over a human receptionist.
What languages do AI phone agents support?
It depends on the ASR and TTS engines in the stack. Most commercial systems support English, Spanish, French, German, and Mandarin at production quality. Less common languages have spottier support and higher error rates. Multilingual mid-call switching (the caller starts in English and shifts to Spanish) is still rough for most platforms.
How does an AI receptionist book appointments?
It extracts the caller's preferred time through conversation, checks availability via the business's calendar API (Google Calendar, Outlook, Calendly), proposes open slots, and confirms the booking. The appointment appears on the calendar as a standard event with the caller's info attached.
Is AI phone answering reliable enough for healthcare or legal?
For intake and scheduling, generally yes. For anything involving clinical triage, legal advice, or sensitive PHI/PII, most businesses route those calls to a human. HIPAA and similar regulations don't prohibit AI from handling calls, but they impose strict requirements on data handling, storage, and disclosure that the system must be built to meet.
What's the latency on a typical AI-answered call?
Expect 300–500ms for the initial greeting (pre-cached), and 800ms–1.5 seconds of round-trip latency during active conversation. That's ASR plus NLU plus response generation plus TTS. Anything under a second feels natural. Over 1.5 seconds starts feeling sluggish.
Does the caller know they're talking to AI?
Sometimes. A well-tuned system with low latency and natural TTS can pass unnoticed in short, transactional calls (booking, hours, basic info). Longer or more complex calls tend to reveal the limitations: repetitive phrasing, inability to handle tangents, unnatural pauses. Some jurisdictions require disclosure, and many businesses choose to disclose upfront regardless.
How does AI receptionist software integrate with existing phone systems?
Through SIP trunking or call forwarding. The business either points their existing phone number's forwarding rules to the AI system, or the AI provider sets up a SIP trunk that sits in front of the existing PBX. Most setups take under an hour to configure and don't require changing the business's phone number.
What happens when the AI can't handle a request?
The system should detect that it's stuck (repeated misunderstandings, caller frustration signals, or an intent outside its trained scope) and route to a human or take a message. The specific fallback depends on how the business configures it. A good implementation logs why the handoff happened so the system can be improved.
The bottom line
That plumber's customer? She got a 9 AM appointment booked while the business owner was eating dinner. The call took 94 seconds. On the backend, a calendar event was created, a CRM record was written, and a notification went out to the on-call tech.
None of that requires new science. It's a pipeline (telephony, ASR, NLU, dialogue management, API integrations, TTS) running against a clock where every millisecond of latency matters. The pieces are all known technology. The hard part is making them work together fast enough that a person on the other end of the line doesn't notice they're talking to software.
If you're building in this space or evaluating it for a business, the tech works for straightforward, transactional calls today. Complex, emotionally charged, or multi-issue calls still need a human on the other end.
