CuePilot listens to a live support call, turns it into text, and puts the best response on the agent's screen in under 200 milliseconds.
That number is the whole product. A suggestion that arrives after the agent has already stumbled through an answer is worse than no suggestion, because now it is a distraction.
So the hard part was never the AI. It was removing delay from a chain of steps where every single step wanted to be slow.
What is the chain you are racing against?
Every real-time voice product fights the same four steps.
Each stage adds delay, and the obvious way to build each stage is far too slow for a live conversation.
The target is not "fast on average". It is fast enough, every time, that the agent trusts it mid-sentence. A system that is quick most of the time and occasionally slow gets ignored, because people cannot rely on it.
Why does streaming change everything?
Because the alternative is waiting, and waiting happens four times.
The single most important decision was to never let one step finish before the next begins. Audio moves from the browser to the backend while the person is still talking. Transcription runs on that stream as it arrives, so the reasoning step sees text building up rather than receiving a finished block at the end of a sentence. The suggestion is pushed back over the same open connection the moment it exists.
The moment you catch yourself saying "and then we send the whole thing to the next step", you have added a delay you will spend weeks trying to remove later.
We chose FastAPI for the backend because its async, connection-native design fits this shape rather than fighting it. Transcription runs on a separate worker with its own hardware, so it never becomes the thing everything else waits for.
Where did the milliseconds actually come from?
A series of small, unremarkable wins. There was no single clever trick.
Speed over size. For the reasoning step we chose based on how fast a model responds, not how capable it is on paper. Inside a 200 millisecond budget, that difference is not a nice-to-have. It is the entire margin.
Connections stay open. Opening a new connection for each request quietly destroys real-time performance. Ours are pooled and reused.
Shared per session, not per person. Sharing connections across a session cut overhead and kept things steady when many agents were live at the same time.
Nothing computed twice. Context, setup and warm models are prepared once and reused. Every repeated calculation is delay you are paying for again and again.
None of these are interesting on their own. Together they are the difference between a demo and something people rely on during a real conversation with a real customer.
Measure the slowest one percent of responses, not the average. Users do not experience your average. They remember the one time it was late while they were talking to someone.
What was the hardest part?
Staying fast when many people use it at once.
Making one call fast is a weekend project. Keeping it fast while dozens of agents are on live calls is the actual work, and it is where real-time systems quietly fall apart. A pipeline that feels instant for one user can collapse into lag for everyone the moment real load arrives.
Most of the hardening went into two things. First, making the system slow down gently under pressure rather than falling off a cliff. Second, making failures visible, because a real-time system you cannot observe is a real-time system you cannot fix.
When something goes wrong during a live call, nobody has time to reproduce it. The logs have to have already answered the question.
What would we do differently?
Build the test set earlier.
When a system generates suggestions live, it is very tempting to judge quality by watching it work. That is exactly the trap we warn clients about, and we were slower to escape it than we should have been.
A prepared set of real call snippets with known-good responses lets you change models and prompts with confidence instead of caution. We got there eventually. Getting there sooner would have been cheaper, and would have made every model decision faster to make.
Live output is convincing precisely because it is live. A fixed set of real examples with known-good answers is the only way to tell an improvement from a coincidence.
What transfers to other products?
Almost all of it, because very little of this was specific to voice.
Stream instead of batching. Reuse instead of reopening. Measure instead of watching. Pick the one number that defines whether the product is useful, and be ruthless about protecting it.
That thinking applies the same way to live translation, a voice assistant, a collaborative editor, or anything where a late answer is the same as a wrong one.
You can read the full CuePilot case study for more on the product itself, and we wrote separately about the guardrails any AI feature needs before it meets real users.
The short version
Real-time AI is mostly not about AI. It is about refusing to wait, four times over.
Stream every stage, hold connections open, compute nothing twice, and pick models for speed when speed is the product. Then spend most of your effort on staying fast under load, and build the test set before you think you need it.
