Read isn't heard: detecting when streaming TTS actually finishes

Our voice agent kept reopening the mic before it had finished talking. The last word was still coming out of the speaker when the app started listening for the user again.

I spent a good part of this year on a 3D AI companion for phones: you talk, an LLM answers, ElevenLabs streams the voice back as Ogg/Opus, and a character on screen says it with lip sync, gaze and expression. Getting the first word out fast was the easy part. This bug took three rewrites, and it came down to one question:

When has the audio actually been heard?

That one moment drives everything after it:

• the microphone reopens, so the user can answer
• the subtitle comes down
• the mouth closes and the body goes back to idle
• the next turn is allowed to start

Too early, and the mic catches the tail of the character's own voice, the classic way a voice agent ends up treating its own words as a new user turn. Too late, and the user is talking to a character that isn't listening yet. Neither shows up in the logs.

Wrong answer 1: estimate it from the text

The first version guessed. Count the words, weight the punctuation:

float duration = words * 0.33f       // seconds per word
               + longPauses * 0.5f   // . ! ?
               + shortPauses * 0.2f; // , ; :

Start a timer, end the turn at duration + 1s.

That holds in a demo and drifts everywhere else. Voices read at different speeds. We sent Chinese at 1.2× speed, so it needed its own correction. And LLM text is full of numbers and abbreviations the voice model reads however it likes. Padding just moves the failure: too little clips long replies, too much leaves a dead gap after short ones.

The estimate still earns its place as a progress bar before any audio arrives. It can't decide when the turn ends.

Wrong answer 2: done when the stream is done

Next: stop guessing, count samples.

// main thread (DownloadHandlerScript.ReceiveData): each decoded chunk
totalDecoded += pcm.Length;

// PCMReaderCallback: each block Unity pulls from the clip
totalRead += samplesRead;

bool finished = downloadComplete && totalRead >= totalDecoded;

This kills a trap on its own: the HTTP stream closing means nothing. TTS servers generate faster than real time, so a ten-second reply can be fully downloaded while the character is on its second sentence. OpenAI's Realtime API documents the same gap. conversation.item.truncate exists for audio "sent to the client but not yet played". And two weeks ago Pipecat, one of the most widely used voice-agent frameworks, had an issue filed for logging "Bot stopped speaking" about four seconds before playback ended.

Counting read against decoded is much better, and progress becomes honest too. But on some phones the last syllable was still getting cut. A 0.15-second grace period helped. A magic number that works on some phones and not others means the model is wrong.

Turning point: stop reading logs, record the audio

Every log line said "done." So I built a capture you can arm for one session, in editor and development builds only. It writes three files:

1. the raw Ogg bytes, as downloaded
2. the decoded PCM, as WAV
3. what came out of the AudioSource, recorded in OnAudioFilterRead, as WAV

Line them up in any audio editor. If a word is in (1) but not (2), it's the decoder. If it's in (2) but not (3), it's timing. Ours was in (2) and missing from (3).

Wrong answer 3: done when Unity has read every sample

A streaming AudioClip is fed by a PCMReaderCallback. When that callback takes your last sample, the sample has not been played. It has entered a pipeline:

your buffer → PCMReaderCallback → AudioSource (+ filters) → mixer → DSP buffers → OS / device → speaker
                    ↑                                                                      ↑
            "all samples read"                                                    "user hears it"

Unity reads streamed clips ahead, in blocks. People have measured how far ahead. How much audio is in flight behind the reader depends on the platform, which is exactly why a fixed grace period worked on one phone and not another.

The same gap exists outside Unity. In the browser, an AudioWorklet consuming your last sample isn't playback either (see AudioContext.outputLatency). On native iOS and Android, the output buffer sits after your render callback.

What held up: follow the last frame to the output

Each utterance is a session with explicit states:

The session exposes a PlaybackCompleted task that the turn logic awaits. Cancel and fail resolve it too, so nothing waits forever on audio that will never play. LiveKit Agents hit that exact deadlock this year, when wait_for_playout() ignored interruption.

Ending the turn takes three steps. The counters cross threads, so they use Interlocked.

1. Mark the last frame at the reader. When the download is complete and the reader has consumed every decoded sample, record the frame index of the final sample and move to Draining. Short replies can get here straight from Buffering.

// PCMReaderCallback (a Unity audio thread)
int frames = data.Length / channels;
long frameStart = Interlocked.Add(ref readerFrames, frames) - frames;
int got = buffer.Read(data);
totalRead += got;

if (downloadComplete && totalRead >= totalDecoded && state is Buffering or Playing)
{
    Interlocked.Exchange(ref finalFrame, frameStart + got / channels);
    state = Draining;
}

2. Confirm it after the AudioSource. OnAudioFilterRead inserts a filter into this AudioSource's own chain. It sees the source's output, already resampled to the output rate and channel layout, before the mixer sums it. Count frames there too. When that count passes finalFrame, the last sample has made it through the source.

void OnAudioFilterRead(float[] data, int channels)
{
    long outFrames = Interlocked.Add(ref outputFrames, data.Length / channels);
    long last = Interlocked.Read(ref finalFrame);
    if (last > 0 && outFrames >= last && !tailConfirmed)
    {
        tailConfirmedAt = AudioSettings.dspTime;
        tailConfirmed = true;
    }
}

One caveat: the reader counts frames at the clip's rate, and the filter counts at AudioSettings.outputSampleRate. Our clip is 48 kHz Opus. If your output rate differs, or you change pitch, scale first: finalFrame * outputRate / clipRate.

3. Wait out Unity's mixer buffer. Unity will tell you its size:

AudioSettings.GetDSPBufferSize(out int length, out int count);
double drain = (double)length * count / AudioSettings.outputSampleRate;
endAt = tailConfirmedAt + drain;   // e.g. 1024 × 4 / 48000 ≈ 85 ms

When AudioSettings.dspTime passes endAt, stop the source, settle the session, reopen the mic.

Treat this as a lower bound. It covers Unity's buffering, not the OS mixer or a Bluetooth headset, which can add a few hundred milliseconds. For our purpose that was fine: with headphones on, the mic can't hear the tail anyway.

The mobile catch. On some phones, OnAudioFilterRead was late, and in some cases never fired, once the stream went quiet. If step 2 waits forever, the subtitle stays up forever. So step 2 has a 2-second timeout. By then every decoded sample has been handed to Unity, so the timeout moment becomes the confirmation point and the drain is added on top. The worst case is a couple of seconds of silence, not a stuck turn. It logs a warning, so you can see which devices take that path.

Two more that bit us

A fixed ring buffer drops audio without telling you. The first buffer was a 20-second ring. The stream outruns playback, so a long reply could fill it. Write returned how many samples it accepted, and nothing checked. The replacement starts at 10 seconds, doubles as needed, and is capped at 120 seconds of unplayed audio (5,760,000 samples at 48 kHz mono). Past the cap it reports overflow and fails the session. A reply with a missing middle is worse than an error you can see.

Callbacks from a dead session. The user interrupts, a new reply starts, and a chunk from the old reply lands a moment later. If it writes into shared state, you get two sentences glued together. Every callback carries its session and checks it's still current. And callers cancel only the session they own, instead of a global "stop everything" that used to kill audio from other features.

void OnChunk(Session s, byte[] chunk)
{
    if (!ReferenceEquals(current, s) || s.IsTerminal) return;
    // decode, write
}

The buffer and the session state machine are unit-tested without a device. The tail detection is verified with the three-file capture.

How others handle it

• LiveKit Agents makes the audio sink report the end itself: the sink calls on_playback_finished() with a playback position. That's the same "confirm at the output" idea.
• OpenAI's realtime console counts samples inside an AudioWorklet. That's the browser version of step 2, without a drain step.
• Pipecat decides the bot stopped speaking when its output queue goes idle for a short window. That is why the issue above sees the event arrive early.

If you're building this

• Don't end a turn on a timer or on the stream closing.
• "Read by the audio callback" is not "heard." Confirm after the source, then add the buffer.
• Build the capture before you need it.

References: Unity AudioClip.Create · ElevenLabs stream speech · RFC 7845 (Ogg Opus) · RFC 3533 (Ogg) · Voice AI & Voice Agents primer