Tried `livekit/agents` Today: A Fast Path to Realtime Voice AI

livekit/agents is a framework for building low-latency voice, video, and multimodal AI agents on LiveKit’s realtime infrastructure. It picked up +256 GitHub stars today, which makes sense: it removes much of the glue code between WebRTC media streams, STT, LLM reasoning, TTS, turn detection, and tool calls.

The useful design choice is that the agent runtime is provider-flexible. You can keep realtime orchestration in LiveKit while pointing the reasoning layer at an OpenAI-compatible gateway.

import os
from livekit.agents import Agent, AgentSession
from livekit.plugins import openai

llm = openai.LLM(
    model="claude-fable-5",
    api_key=os.environ["B_LOST_API_KEY"],
    base_url="https://b-lost.com/v1",
)

assistant = Agent(
    instructions=(
        "You are a concise voice assistant. "
        "Confirm intent before executing sensitive actions."
    ),
    llm=llm,
)

session = AgentSession()
# Connect `session` to your LiveKit room, then run `assistant`.

A practical deployment shape:

Layer	Responsibility	Swap cost
LiveKit	WebRTC rooms, audio/video transport	Low
`livekit/agents`	Turn-taking, tools, agent lifecycle	Low
STT/TTS provider	Speech recognition and synthesis	Medium
`claude-fable-5` gateway	Reasoning and tool planning	Low

For cost planning, B-Lost’s relay advertises 0.8× official list pricing (20% off). The exact effective cost still depends on input/output token mix, audio duration, retries, and tool-call verbosity—so I would measure per completed conversation rather than only cost per 1M text tokens.

Metric to benchmark	Why it matters
TTFT	Determines whether a voice agent feels interruptible
End-of-speech → first audio	Best user-perceived latency metric
Tokens / completed task	Captures prompt and tool overhead
Tool-call success rate	More useful than generic code benchmarks

If your workflow can use native Anthropic /v1/messages, Prompt Caching is especially relevant: cache hits can receive up to a 90% discount, useful for repeated system prompts, policy blocks, and long tool schemas. For OpenAI-compatible routing, verify cache behavior per endpoint rather than assuming it is automatically applied.

The main attraction: LiveKit Agents lets teams benchmark providers independently without rebuilding the realtime stack each time.