323 requests, 8 test conditions, one production deployment. The slow first word is not the model's fault.
The 30-second version
Through my app, the median time to first token (TTFT) is 2,938.6 ms. Sent straight to the model provider, a comparable request takes 664.4 ms. While, that leaves 2,274.2 ms that sits between the user and the model. Switching providers moves TTFT by roughly 350 to 490 ms. Opening a fresh connection adds about 169 ms.
The application layer is the biggest lever by a wide margin.
Under load it gets worse. With 5 users at once, the median first-word delay is 13.6 seconds, about 4.3 times the single-user number. I make sure every app request carries a unique ID, so in the next step I can match each one to my server logs and see exactly which stage is slow.
Why the first word matters more than the last
When a chat app streams its answer, people do not judge it by how long the whole answer takes. They judge it by how long they stare at a blank screen before the first word appears. That delay is called time to first token (TTFT).
A 3-second blank screen feels like the app froze. A half-second blank screen feels instant, even if the full answer takes ten seconds to finish. So if you build with LLMs, TTFT is the
number that decides whether your product feels alive.
Vicquant, my AI financial advisor, streams its answers. And its TTFT was bothering me.
How I measured it
Vicquant runs in production: a FastAPI backend on Render, a Postgres database on Neon, and models served through OpenRouter. I ran every test against the live site, from a client over the real internet. No mocks, no simulated numbers.
The trick of this benchmark is that I changed one thing at a time. Each condition isolates one possible cause of delay:
where n is the sample size.
Every one of the 223 requests routed through the app carries a unique x-request-id header in the form bench_. I verified that all 223 of them are present in my results file.
That ID is a join key: it lets me match each client-side measurement to the exact server-side log line for the same request, instead of guessing from averages. This idea was given to me by Israel Negrete Lepe on this channel. He has been helpful. Thank you Israel.
The 223 Requests Routed Through the App are:
A1 (Through app, reused connection): 120
A2 (Through app, fresh connection): 40
D (Through app under concurrency: 20 + 20 + 20): 60
E (Through app cold starts): 3
Sum: 120 + 40 + 60 + 3 = 223
The headline numbers
Notice the p90 column too. Through the app, 1 in 10 requests waited 4.6 seconds or more for the first word. Medians hide the worst experiences, so I report both.
Finding 1: My app adds 2,274 ms
Subtract the direct provider call (B, 664.4 ms) from the through-app call (A1, 2,938.6 ms) and you get 2,274.2 ms. That is everything my application puts between the user and the model ( the web framework, authentication checks, loading chat history, assembling the prompt, and buffering the streamed chunks ).
An honest caveat; That 2,274 ms is an upper bound on pure application code time. A request through my app also takes one extra network hop to Render that the direct call does not. The network baseline (631 ms to reach the server) hints at how big that hop is, so the true code-only cost is smaller. Splitting the two cleanly is exactly what the request-ID join against server logs is for.
Finding 2: Fresh connections cost little
A fresh connection (A2) was 169.3 ms slower than a reused one (A1). That is the price of the TCP and TLS handshake. It sounds meaningful, but look at the confidence intervals: A1 is
2,815 to 3,107 ms and A2 is 2,900 to 3,285 ms. They overlap.
So the honest reading is: connection setup is a small cost, and this experiment cannot prove it is bigger than noise.
Finding 3: The provider matters, but it is not the main
story
OpenRouter routes your request to one of several upstream providers, and they are not equally fast. With default routing, 23 of 30 requests landed on Groq (76.7%), 5 on Novita
(16.7%) and 2 on DeepInfra (6.7%).
Novita first (C1): 21 Novita, 9 DeepInfra. Median 1,150.9 ms, which is +486.5 ms over default routing.
DeepInfra only (C2): 29 DeepInfra, 1 Cloudflare. Median 1,013.2 ms, which is +348.8 ms over default routing, with the tightest confidence interval of the three.
Choosing the provider is worth a few hundred milliseconds. Fixing my own application layer is worth more than two seconds. Chasing the fastest provider while ignoring the app would have been optimizing the wrong thing.
Finding 4: Concurrency is where it really hurts
Here is the pattern that stands out. Going from 1 to 3 users added about 3.1 seconds per extra user. Going from 1 to 5 added about 2.6 seconds per extra user. Each extra person at the door adds roughly one full request's worth of waiting. That is how a queue behaves, not how a system that handles requests in parallel behaves.
My app currently runs on a single Uvicorn worker. That is the prime suspect, and it is the main target for the optimization round. (A note on the table: "app share" here is TTFT minus
the 631 ms network baseline, so it still includes the model provider's own time.)
Finding 5: Cold starts are a small penalty
Three cold-start runs gave 3,967.2 ms, 3,322.7 ms and 3,395.6 ms. Compared with the 2,938.6 ms steady-state median, a cold start costs roughly 0.4 to 1.0 seconds extra. With only three runs this is directional, not final. But it tells me cold starts are not what is making the app slow day to day.
Keeping the experiment clean
Exact join keys: 223 of 223 app requests contain their bench_ ID, ready to join against the server's TIMELINE_TRACE logs (t_in when the request arrives, t_first_sse whenthe first streamed chunk leaves).
No leftover test data: all benchmark accounts were deleted from the Neon database. A query for @bench.invalid emails returns 0 rows.
Raw data kept: every row and a statistical report are stored locally and deliberately kept out of Git.
What I think is happening (hypotheses, not conclusions)
I want to be careful here. The benchmark tells me where the time is, not yet why. My current suspects:
The Work carried out before the model is called. Auth validation, history lookup and prompt assembly all happen first. If any of them is slow or blocks, the user waits for it.
A single worker acting like a queue. The near-linear slowdown under concurrency is exactly what serialization looks like.
Streaming that is not truly progressive. If chunks are buffered before they leave the server, the first word is delayed even when the model was fast.
How I will test them
I'll Join request IDs to server logs. Split each request into time spent before the model call versus time spent waiting on the model, using t_in and t_first_sse.
Time each stage before the model call individually, so I know which one to fix first.
Add workers and re-run Condition D. If serialization is the cause, c = 3 and c = 5 should drop sharply while c = 1 barely moves. If c = 1 does not move, the cost lives in the code path itself.
Set a target before I look: my goal is a c = 5 slowdown below 1.5x, instead of today's 4.3x.
Takeaways you can use today
Measure TTFT, not just total time. It is what users feel.
Isolating one variable per experiment. Direct vs through-app is what exposed the real gap.
Tag every test request with a unique ID so you can join client numbers to server logs later.
Report confidence intervals and p90, not only medians, and say out loud when intervals overlap.
Test concurrency. A single-user benchmark can make a fragile app look healthy.
In the next part I will publish the before-and-after numbers once the fixes are in. If you were running this benchmark, what would you measure next? Tell me in the comments.
