Scaling Voice Agents to 10K+ Calls: Jarvis’s 3 Biggest Challenges: Part 1

I Fixed Routing. Then I Found Out Routing Was Never the Real Problem

From "the pods keep dying" to "oh, it's the architecture"

Where We Left Off

In my last post I replaced a Redis session-affinity lookup with Istio consistent hashing. WebRTC drops stopped. We saved money. I felt clever.

Then we tried to actually scale Jarvis.

Jarvis is our real-time voice agent. It brokers a Daily.co WebRTC room per call and runs a pipecat pipeline: user speaks → VAD → STT → LLM → TTS → bot speaks. The LLM lives in Stark, our orchestrator. Jarvis is the audio edge. One pod served several calls at once, like any normal web service.

Here is what "normal web service" thinking got us:

jarvis-xxxxx   RESTARTS: 1   LAST STATE: OOMKilled
  heap:        pressing against the memory limit
  OS threads:  in the hundreds
  live calls:  a handful

A handful of calls. Hundreds of threads. Consistent hashing had nothing to do with it.

This post is the autopsy: three problems, what each one did to CPU and memory, and the one variable they all share. Part 2 is the rebuild with Agones and Karpenter, one pod per call.

What One Call Costs

Per call: one WebRTC peer, two vendor WebSockets, one SSE stream, one ONNX model scoring audio about 30 times a second, and a pile of resampling. All inside one Python process, under one CPU limit, sharing one memory limit.

That limit was small. A few hundred millicores and well under a gigabyte, because that's what a web service pod looks like. Everything below is a fight over it.

Here is what that fight looked like in prod, back when jarvis was a plain Deployment. Three pods sharing calls, two days, real data:

Two things to hold onto. The CPU panel sits near zero and jumps in short spikes, and every spike is a call doing something: someone talking, a call starting, a TTS burst going out. The memory panels are a staircase. We'll come back to both.

Problem 1: VAD Isn't CPU-Hungry. It's CPU-Sensitive.

VAD decides when the user has stopped talking. In pipecat that's Silero, an ONNX model scoring every short chunk, plus a stop_secs timer: once the score drops below confidence for stop_secs seconds, the turn ends and the transcript goes to the LLM.

@dataclass
class VAD:
    confidence: float   # how sure the that this is speech
    start_secs: float   # how long speech must persist before "user started"
    stop_secs:  float   # how long silence must persist before "user stopped"
    min_volume: float   # below this, don't even ask the model

The catch: stop_secs is wall-clock time, but the chunks arrive through a process that can be throttled. Late chunks look like silence.

What happened on our pods:

• Loading Silero costs tens of ms of CPU per call. Under the pod's CPU limit that stretched to several times longer in wall time, on the start-call path.
• Look at the CPU panel again. Between turns a pod sits near zero. When someone talks it spikes, and the spike is short. The kernel doesn't bill the average. It bills 100 ms slices against the quota.
• When that happens, the logs look like this:

[VAD] user stopped speaking      (stop_secs of "silence": really stop_secs of late chunks)
[STT] flush → LLM                "I was going to say that"
[VAD] user started speaking      (the chunks finally arrive)
[STT] flush → LLM                "we should probably"
→ two LLM turns for one sentence. User hears two replies.

We called these "duplicate messages". The first fix is in git history and I'm not proud of it:

bump STTToTextProcessor response timeout (band-aid for duplicate messages)

Over the next few months we also loosened confidence and min_volume and shortened stop_secs, because VAD "wasn't firing". Some of that was real. Some of it was us tuning a model to compensate for a starved process.

CPU: VAD's own cost is small. Its sensitivity to throttling is total. It's the canary.
RAM: a few MB per analyzer. Not the memory problem.

Problem 2: STT/TTS Traffic Is Where the CPU Goes

My first theory was Python. Single-threaded, GIL, two calls step on each other, done. I had that paragraph written. Then I remembered we'd run TTS to a bunch of devices off one process without anything choking.

So I measured it. Two threads of pure Python loops take turns, textbook GIL. But the audio work a call does, resampling and VAD inference, is C and C++ underneath. It drops the GIL and runs on two cores just fine. The Daily client's threads don't ask Python at all.

Python wasn't the bottleneck. The pod's CPU budget was. Parallel threads don't get around a budget. They spend it faster.

Where a call's CPU goes:

None of it is heavy alone. Each one is a spike on that CPU panel. Put calls together on one pod and it goes like this:

Our load tester Voldemort, exists because of this table. One Locust user is one whole call: real STT, mock TTS and LLM with measured vendor latencies baked in. An instant mock makes a pod look several times more capable than it is.

CPU: the bulk of the in-call burst.
RAM: a bounded working set per live call. If this were the only memory cost we'd never have OOMed.

So why did we OOM?

Problem 3: The Daily.co Thread Leak

Python asyncio code doesn't make hundreds of threads. Something native does.

daily-python wraps Daily's C++ WebRTC client. Every call creates one, and every client starts roughly ten OS threads. You'll see names like SoupSignalling and mediasoup-manag in /proc/<pid>/task/*/comm. Those threads and the client's audio buffers are freed by exactly one thing: daily.release().

In pipecat, release() runs at the very end of pipeline teardown, after an EndFrame has flowed through every processor. Our teardown under load:

end-call arrives
  → stop_pipeline()
    → queue EndFrame
    → wait for runner ... timeout ...
    → task.cancel()          ← "fallback if the pipeline does not finish"
  → drop transport reference
  → gc.collect()

task.cancel() cancels the Python coroutine. It does not call release() on the C++ client. The Python object gets collected. The ten threads and the buffers don't. They were never Python's to collect.

And once we dropped our reference, our "active transports" gauge went back to zero. Our metric said clean. The kernel said hundreds of threads.

Back to the screenshot from the top. CPU is quiet. Memory is a staircase, and there isn't a single step down in two days. Every step is a call that ended and left something behind. Three pods, same slope, so not one bad pod or one bad user.

The Fixes That Weren't

Every one of these felt like progress at the time:

That last row is my previous post. Hashing was correct. It only mattered because one pod held state for many calls. We solved routing to a stateful pod without asking why the pod was stateful for more than one call.

The actual leak fix is small:

# daily.release() is the only thing that reclaims the native client's
# threads and buffers. Never cancel it.
release = spawn(transport.cleanup(), name=f"daily-release-{session_id}")
for timeout in (_RELEASE_TIMEOUT_SECONDS, _RELEASE_GRACE_SECONDS):
    try:
        await asyncio.wait_for(asyncio.shield(release), timeout=timeout)
        return
    except TimeoutError:
        logger.warning("[MEMORY] Daily release slow, still waiting")

Shield the release. Prune the transport entry only after it completes, so the gauge stays honest. That stopped the bleeding. It didn't change the fact that one pod was juggling several realtime pipelines on a web-service CPU limit.

The Comparison That Changed the Architecture

This reframed it to:
"What if a pod only ever served one call? Like a game server. One match, then it exits."

Then nothing leaks across calls, because there is no across calls. VAD never competes with another call's TTS, because there are no neighbours. CPU per pod is one call's spikes and nothing else. Memory per pod is one call's working set, and then the process is gone. Routing is "the pod I was given".

The problem collapses from "make a multi-tenant realtime Python process behave" to "hand out fresh processes fast enough". That's a scheduling problem, and the game industry solved it years ago.

If You're In My Shoes

If you run a native SDK in Python on Kubernetes and you see threads creeping up while your metrics say clean, memory that only goes up, or endpointing that's flaky only under load, don't start with the model params. Count the threads.

# 1. threads, not memory
kubectl exec <pod> -- ls /proc/1/task | wc -l

# 2. what the threads are
kubectl exec <pod> -- sh -c 'cat /proc/1/task/*/comm | sort | uniq -c | sort -rn | head'

# 3. real heap vs cache
kubectl exec <pod> -- grep -E '^(anon|file) ' /sys/fs/cgroup/memory.stat

For CPU, watch throttled periods, not the usage average. The average is what the spikes look like after Grafana flattens them:

rate(container_cpu_cfs_throttled_periods_total{container="<app>"}[5m])
  / rate(container_cpu_cfs_periods_total{container="<app>"}[5m])

And one more, because container_oom_events_total read zero in our cluster even for pods that were genuinely OOMKilled:

max_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[14d]) > 0

Part 2: how we rebuilt Jarvis as a one-pod-one-call fleet on Agones with Karpenter supplying nodes, the warm-pool formula N = x + 3√x, the three ways a pod can leak out of the pool, and the arithmetic to 10,000 concurrent calls.

It's not as clever as consistent hashing. It's a lot more boring. In the best way.