Engineering an AI Voice Agent That Safely Turns Calls Into CRM Actions

How we used Twilio, ElevenLabs, OpenAI, FastAPI, CRM APIs, deterministic validation, human escalation, and observability to build a production AI voice workflow.

Building an AI voice agent that can hold a conversation is becoming increasingly accessible.

Building one that can safely trigger real business operations is a different engineering problem.

In a recent production workflow, we needed an inbound voice agent to do more than answer a phone call.

After understanding the caller, the system needed to reliably determine things such as:

• who was calling,
• why they were calling,
• whether the request was urgent,
• whether a CRM record already existed,
• what follow-up should happen,
• who should own that follow-up,
• whether a human should take over,
• and what should happen if any part of the workflow failed.

The architecture eventually combined:

• Twilio Voice
• ElevenLabs Conversational AI
• OpenAI
• FastAPI
• CRM APIs
• notification workflows
• human escalation
• application and workflow logs

The most important architectural decision was not which LLM or voice model to use.

It was deciding where AI stopped and deterministic software took control.

The actual problem was after the conversation

The original workflow was familiar.

A caller speaks with someone.

That person understands the request.

Then somebody has to:

1. write notes,
2. identify the caller,
3. classify the request,
4. update or create a CRM record,
5. create a follow-up task,
6. assign an owner,
7. notify someone if necessary,
8. and make sure nothing gets lost.

The voice conversation is only the first part of that process.

For automation to create meaningful operational value, we needed to transform:

conversation

into:

structured context
    ->
validated business intent
    ->
controlled application action
    ->
ownership
    ->
observable workflow state

That required treating the voice AI as one component inside a larger software system.

High-level architecture

A simplified view looked like this:

Caller
  |
  v
Twilio Voice
  |
  v
ElevenLabs Conversational AI
  |
  v
Conversation / transcript context
  |
  v
OpenAI structured interpretation
  |
  v
FastAPI validation and orchestration
  |
  +----------------------------+
  |                            |
  v                            v
CRM actions               Human escalation
  |                            |
  v                            v
Task / ownership          Notification / callback
  |
  v
Workflow logs

The important detail is the layer between AI interpretation and business systems:

AI output
   |
   v
FastAPI
   |
   v
CRM / operational side effects

We did not want probabilistic model output directly controlling CRM actions.

AI interprets. Software decides whether to act.

An LLM can be very good at understanding a statement such as:

I spoke with someone yesterday about pricing. I still have a few questions and would prefer somebody to call me tomorrow morning.

From this, the AI might infer:

{
  "intent": "sales_follow_up",
  "urgency": "medium",
  "preferred_callback": "tomorrow_morning",
  "existing_interaction": true,
  "requires_human": true
}

That is useful.

But it is not enough to immediately call:

crm.create_task(...)

The backend still needs to answer questions such as:

Is the intent valid?

Are required fields present?

Does the contact already exist?

Is the callback time usable?

Which team owns this request?

Should this become a task or a transfer?

Is the confidence sufficient?

Did the caller explicitly request a human?

Is this workflow allowed to execute automatically?

That is why we treated FastAPI as a control boundary between probabilistic interpretation and deterministic execution.

A simplified structured output model

A production schema might conceptually look like:

from pydantic import BaseModel
from typing import Optional, Literal


class CallIntent(BaseModel):
    caller_name: Optional[str]
    phone: str

    intent: Literal[
        "new_sales",
        "existing_customer",
        "support",
        "billing",
        "appointment",
        "other"
    ]

    service_interest: Optional[str]

    urgency: Literal[
        "low",
        "medium",
        "high"
    ]

    preferred_callback: Optional[str]

    requires_human: bool

    summary: str

The point of a model like this is not merely type safety.

It establishes the contract between:

language interpretation

and:

business logic

Without a contract, the LLM effectively produces prose.

With a contract, the application receives data it can inspect.

Why transcripts alone were not enough

A transcript might contain several hundred words.

The CRM usually does not need several hundred words.

It needs specific operational context.

For example:

{
  "caller_name": "Daniel Harris",
  "call_reason": "Pricing and implementation inquiry",
  "intent": "new_sales",
  "service_interest": "AI workflow automation",
  "urgency": "medium",
  "preferred_callback": "Tomorrow morning",
  "next_action": "Create CRM lead and callback task",
  "requires_human": false
}

Now the system can reason deterministically about the next action.

A transcript is useful for:

• review,
• debugging,
• quality checks,
• historical context.

Structured data is useful for:

• routing,
• CRM updates,
• task creation,
• prioritization,
• automation.

That distinction was fundamental.

Validating before creating side effects

A simplified FastAPI-style service layer might look something like this:

def process_call(call: CallIntent):

    if not call.phone:
        return escalate(
            reason="missing_phone"
        )

    if call.requires_human:
        return escalate(
            reason="human_requested"
        )

    if call.urgency == "high":
        return escalate(
            reason="urgent_request"
        )

    contact = crm.find_contact_by_phone(
        call.phone
    )

    if not contact:
        contact = crm.create_contact(
            name=call.caller_name,
            phone=call.phone
        )

    task = crm.create_task(
        contact_id=contact.id,
        intent=call.intent,
        summary=call.summary,
        priority=call.urgency,
        callback_time=call.preferred_callback
    )

    log_workflow_success(
        contact_id=contact.id,
        task_id=task.id
    )

    return task

This is intentionally simplified, but it demonstrates the architectural idea.

The LLM does not decide whether a CRM task successfully exists.

The application does.

Separate understanding from execution

We found it useful to think about the system as two different worlds.

Probabilistic layer

Good uses of AI included:

natural-language understanding
intent classification
conversation summarization
information extraction
sentiment/context interpretation
suggested next action

Deterministic layer

Application code remained responsible for:

schema validation
CRM lookup
deduplication
authorization
routing rules
task creation
ownership
notifications
workflow status
error handling
logging

Some areas involved both.

For example:

Intent classification
    -> AI interpretation
    -> backend validation

Structured fields
    -> AI extraction
    -> Pydantic validation

Human escalation
    -> AI/context signal
    -> workflow rule

This hybrid architecture was significantly safer than trying to make the model the entire application.

Human handoff should be part of the architecture

A common anti-pattern in AI agents is treating human escalation as failure.

For production voice workflows, it is often exactly the opposite.

Human involvement may be the correct state when:

caller explicitly asks for a person
confidence is low
information is incomplete
the request is urgent
the conversation becomes sensitive
billing/account concerns appear
the opportunity is commercially important
the system cannot safely determine the next action

Our workflow therefore treated escalation as a first-class output.

Conceptually:

if confidence < MIN_CONFIDENCE:
    escalate_to_human()

elif caller_requested_human:
    escalate_to_human()

elif urgency == "high":
    create_priority_callback()

else:
    continue_automated_workflow()

The objective was never:

Keep AI talking for as long as possible.

It was:

Automate repeatable work.

Escalate judgment.

Preserve context.

Preserve context during handoff

A poor escalation looks like this:

AI: Let me transfer you.

Human: Hi, how can I help?

Now the caller repeats everything.

A better workflow passes useful context forward:

{
  "caller": "Daniel Harris",
  "reason": "Pricing question",
  "service": "AI workflow automation",
  "summary": "Caller previously discussed implementation and has follow-up pricing questions.",
  "urgency": "medium",
  "preferred_callback": "Tomorrow morning",
  "escalation_reason": "Requested human follow-up"
}

Now the human can continue from where the automation stopped.

That is a much better system experience.

Design the failure paths before the happy path is finished

Voice demos tend to focus on:

caller speaks
AI understands
everything works

Production systems need more branches.

We explicitly considered scenarios such as:

Caller information is incomplete

Possible response:

Attempt to collect the missing information.

If it still cannot be obtained, preserve the partial context and create a human follow-up.

AI confidence is insufficient

Do not force a classification.

uncertain interpretation
    ->
human review / escalation

CRM operation fails

Do not pretend the call completed successfully.

AI workflow succeeds
CRM request fails
    ->
persist workflow error
    ->
retry or notify

Caller requests a person

Treat that as a valid workflow outcome.

Urgent or sensitive call

Bypass routine automation rules.

Idempotency matters

Voice workflows can trigger retries.

Webhooks can be delivered more than once.

CRM APIs can time out after completing an operation.

That means this is dangerous:

crm.create_contact()
crm.create_task()

without identifying whether the operation already happened.

A production implementation should consider an idempotency key such as:

call_id

or another stable workflow identifier.

Conceptually:

existing = workflow_store.get(call_id)

if existing and existing.completed:
    return existing.result

This prevents accidental duplicate:

contacts
tasks
notes
notifications

when integrations retry.

Treat CRM writes as controlled side effects

We also found it useful to explicitly think of CRM changes as side effects.

The application should know:

what was requested
what was attempted
what succeeded
what failed

For example:

{
  "call_id": "CA_12345",
  "intent": "new_sales",
  "crm_contact_action": "created",
  "crm_task_action": "created",
  "notification_action": "sent",
  "escalation": false,
  "workflow_status": "completed"
}

Compare that with:

{
  "call_id": "CA_12346",
  "intent": "support",
  "crm_contact_action": "found_existing",
  "crm_task_action": "failed",
  "notification_action": "sent",
  "escalation": true,
  "workflow_status": "requires_review"
}

Now the workflow is inspectable.

Observability is part of the product

An AI agent that works 95% of the time but gives you no visibility into the remaining 5% becomes painful very quickly.

We retained context around:

call start/end
caller identity
detected intent
structured extraction
summary
CRM operation status
task status
escalation
notifications
errors

That allowed questions such as:

What happened to call X?

Was a CRM record created?

Was a follow-up task assigned?

Who owns it?

Was human escalation triggered?

Did an integration fail?

to be answered without reconstructing the workflow manually.

A simple workflow state model

One way to think about the call lifecycle is:

RECEIVED
   |
   v
IN_CONVERSATION
   |
   v
INTERPRETED
   |
   v
VALIDATING
   |
   +----------------------+
   |                      |
   v                      v
EXECUTING              ESCALATED
   |
   v
COMPLETED

Failure states can occur between stages:

VALIDATION_FAILED

CRM_FAILED

NOTIFICATION_FAILED

REQUIRES_REVIEW

This is much easier to reason about than a boolean:

success = True

Why this was not just an IVR replacement

A traditional IVR works well for deterministic menus:

Press 1 for sales.

Press 2 for support.

Press 3 for billing.

A caller may instead say:

I spoke to someone yesterday about the service. I have another question before I decide, and could somebody call me tomorrow morning?

A conversational system can extract:

existing sales interaction

unresolved question

human follow-up required

preferred callback window

without forcing the caller to understand the company's internal menu structure.

But natural-language understanding does not remove the need for deterministic workflow rules.

That is why I think of production voice AI as:

Conversational AI
+
Application engineering
+
Workflow automation
+
Human judgment

not simply:

A smarter IVR

Technology responsibilities

The stack made more sense when described by responsibility rather than vendor.

The important part is not that all these technologies were used.

The important part is that each one had a defined responsibility.

Tested results

For the call scenarios supported by this implementation, testing showed:

~70-80% of standard calls
-> moved through the structured AI-assisted workflow

~60-70% reduction
-> in manual post-call note-taking

~1-2 minutes
-> for CRM follow-up tasks to appear after the call

These are implementation-specific results.

They are not universal benchmarks for every voice AI project.

Different call types, workflows, integrations, business rules, and escalation requirements will produce different outcomes.

The more interesting result was architectural.

A routine conversation could become an owned and observable business workflow without requiring a human to manually connect every step.

What I would do differently from a typical AI prototype

If I were starting another voice-agent project tomorrow, I would not begin with the voice.

I would first document:

What happens after a sales call?

What happens after a support call?

What data must be captured?

What does the CRM require?

Which actions are safe to automate?

Which actions require deterministic validation?

When must a human intervene?

What happens if an API fails?

How will retries work?

What needs to be logged?

How do we know the workflow actually completed?

Only then would I design the conversational layer.

Some engineering rules I would keep

1. Do not let an LLM directly own transactional business logic

Use AI for interpretation.

Use software for execution.

2. Structured output is more useful than transcription

Keep transcripts for context.

Create schemas for operations.

3. Design escalation before launch

Human handoff is a feature.

Not an embarrassment.

4. Model failure states explicitly

If something fails, the system should know what failed.

5. Make downstream actions idempotent

Retries are normal.

Duplicate CRM objects should not be.

6. Persist workflow state

If the system cannot answer:

What happened to this call?

then observability is incomplete.

7. Keep the AI boundary narrow

The less ambiguous the AI's responsibility is, the easier the overall system becomes to reason about.

The bigger lesson

There is a lot of attention right now on models, agents, prompting, speech quality, latency, and realism.

All of those matter.

But once an AI agent interacts with a real business, ordinary software engineering becomes extremely important again.

You still need:

validation
state
APIs
authorization
idempotency
retries
error handling
observability
deterministic rules
human escalation

An impressive conversation is a demo.

A system that can safely convert that conversation into a controlled business action is a product.

That distinction became the most important lesson from this implementation.

Full case study

We documented the complete implementation, including the architecture, call-to-CRM workflow, human escalation, failure handling, observability decisions, and tested outcomes here:

https://www.zestminds.com/ai-voice-agent-automation-case-study

If you are building something similar, I would be interested in how you are handling the boundary between probabilistic agent decisions and deterministic application logic.

That boundary is becoming one of the most interesting parts of production AI engineering.