A few days ago I published latency numbers for my AI finance app that were wrong. The headline was "all queries under one second." A reader asked what the benchmark actually measured, and my own audit revealed that 35 of the 75 data points had been simulated rather than measured end-to-end. I retracted the numbers immediately.
This is the replacement: measured through the real, live production app, across six benchmark sessions over three consecutive days, with zero simulated data.
The Short Version of what i worked on in the past few days
Typical case: Time to first token (TTFT) through the deployed app averaged 2.3 to 3.6 seconds (the median of each session). The exact same model call made directly to the inference provider took about 0.6 to 1.8 seconds.
The surprise: With several people using the app concurrently, it slowed down dramatically. At 5 simultaneous users, median time to first token jumped to 9 to 22 seconds.
What I can't explain yet: An overhead of about 1.9 seconds sits outside the model inference window, and the "fast" chat path was unexpectedly slower on average than the document-retrieval (RAG) path.
Earlier tests on my laptop against a remote database clocked ~13 seconds to first token. Removing four sequential database round trips and repairing missing HTTP streaming chunk headers eliminated that initial lag. Those older numbers came from a local environment and different code; the numbers below represent the audited, production-verified measurements.
How I Measured
Target Deployment: The live app hosted on Render, a Neon Postgres database in us-east-2, and meta-llama/llama-3.1-8b-instruct routed through OpenRouter using the production provider hierarchy (Groq first, then Novita, then DeepInfra).
Client Vantage Point: An automated streaming client on a mobile cellular broadband connection, connecting through the public HTTPS URL.
Metric Definitions:
Time to First Token (TTFT): Elapsed duration from client request transmission (t_client_send) until the first content token arrives at the client.
Completion: Elapsed duration from client request transmission to the arrival of the final token / stream termination.
Test Protocol: Six discrete sessions across three days (two sessions per day, October 13–15), with 323 requests per session (1,938 total requests):
120 steady-state requests on a warm, reused connection.
40 requests on fresh, newly established TCP/TLS connections.
30 direct OpenRouter requests across each of three provider policies (Default, Groq-excluded, Groq+Novita excluded).
60 concurrency load-test requests (20 each at $c=1, 3, 5$).
3 cold-start probes following server idle intervals.
10 baseline network probe pings.
Hygiene & Cost: Test accounts used reserved ephemeral domains (@bench.invalid) and were purged by exact ID from Neon at session conclusion (verified 0 residual rows). Total OpenRouter inference cost across all 1,938 requests was $0.0108 USD.
Result 1: The App Adds About 1.9 Seconds to the Raw Model Call
Note: that the difference column isn't all "app." Some of it is the extra hop from my laptop to Render, and the network probe (a lightweight endpoint, so it isn't a clean subtraction) suggests that hop costs a few hundred milliseconds or more on my connection. I did not trace the server side in these sessions, so I won't pin the rest on any single cause. My candidates are database round trips, request middleware, and streaming overhead, but those are guesses.
Two other patterns I can't explain yet:
Slower on the second and third day; Day 1 medians were 2.3 to 2.7 seconds, and days 2 and 3 were 3.0 to 3.6. The network probe didn't rise by a comparable amount. I'm checking whether the deployed code or the data changed between days.
The "fast" path was slower. The fast-chat path had a mean session median of about 3.4 seconds, and the document (RAG) path about 2.7 seconds. I expected the opposite. One would expect document retrieval, vector matching, and reranking to add latency, not decrease it.
Result 2: a fresh connection costs 60 to 714 ms
On a brand-new connection, first token arrived 60 to 714 ms later than on a reused connection, with a mean of about 330 ms. Browsers reuse connections, so most turns skip this, but a first-time visitor pays it. While browsers reuse active connections during an ongoing chat session, first-time visitors must absorb this initial handshake overhead.
Result 3: Groq-First Provider Order Delivers ~0.4s Faster TTFT; It is about 0.4 seconds faster to the first token
By isolating provider endpoints in our routing policy, we measured the upstream first-token latency differences:
Important Context: This test evaluated first-token arrival only. In full generation benchmarks, DeepInfra exhibits lower tokens-per-second output throughput, leading to substantially longer full-response completion times (~10–14 seconds). Provider selection involves a trade-off between instant token streaming initiation and sustained output throughput.
*Result 4: Concurrency Bottlenecks Emerge Quickly; running simultaneous users hurt a lot
*
Running concurrent client requests through the deployed app exposed an event loop / worker concurrency wall:
Median TTFT (seconds) across Concurrency Tiers (20 requests per tier per session):
With 3 clients the median first token was 1.6 to 3.9 times slower than with 1, and with 5 clients it was 2.7 to 4.4 times slower. That's roughly what you'd see if the server handled requests largely one after another: with k requests arriving together, the median wait is about (k+1)/2 times a single request, which is 2 times for 3 clients and 3 times for 5. It's consistent with that picture, but it doesn't prove it. I haven't measured whether something is blocking the event loop. Day 3 was worse at every level, and I don't know why.
For a small beta, a single user sees about 3 seconds. Three or more people chatting at the same moment is the current limit, and it's the first thing I'm fixing.
When multiple clients query the system concurrently, response times degrade in near-linear proportion to the concurrency level. A pattern characteristic of single-worker event loop blocking or synchronous I/O operations occurring within asynchronous route handlers.
Result 5: Cold Starts
The first request after an idle period took 5.4, 3.6, 3.2, 3.8, 5.8 and 6.6 seconds across the six sessions. Those are single runs, so I'm reporting ranges, not statistics. On day 2, session 2, the second and third requests were also slow (7.0 and 6.2 seconds), and I have no explanation for that.
Pooled Aggregate Percentiles (All 720 Steady-State App Requests)
Rather than evaluating session medians alone, pooling all 720 steady-state requests ($A1$) collected across identical code and conditions yields the following distribution:
p50 (Median): 2,842 ms
p90: 5,420 ms
p95: 7,140 ms
p99: 10,680 ms
What I Can and Cannot Claim
Network Specificity: All telemetry was measured from a remote cellular client connecting over public international routing to a US-East server deployment. Users on direct fiber or closer to the server host will experience lower transport and TLS handshake latencies.
Server-Side Dissection: Because server tracing was turned off during benchmark execution, the exact millisecond distribution between application middleware, template compilation, database session acquisition, and upstream proxy dispatch remains unverified.
Quality vs. Speed: These benchmarks isolate response velocity only. They make no assessment of response grounding, semantic relevance, or financial calculation accuracy.
What I Learned
Benchmark the User Path, Not the Bare Model: Direct model calls bypass the database, session handling, serialization, and network routing. Measuring the raw model told me how fast OpenRouter was, but told me nothing about the experience of someone using the app.
Concurrency Reveals What Single-User Testing Hides: A 2.8-second single-user response is usable; a 16-second response under modest concurrency is unacceptable. Concurrency profiling must be an integral component of testing from day one.
Transparency Builds Trust: Retracting incorrect numbers is painful, but publishing verified, unvarnished data is the only standard that matters.
What I'm doing next
1.Trace the server side to find where the extra 1.9 seconds goes, and why the fast path is slower than the document path.
2.Fix the concurrency behavior, then re-run this exact harness and publish before and after numbers.
3.Pooled percentiles with confidence intervals, grouped by network and code version.
4.Repeat the provider quality comparison with enough runs to see past the noise.
5.Tomorrow: a second article on a blind spot in my quality metrics which a reader . They measured the pipeline accurately and still missed that 18 of 20 document questions never reached it.
If you've tackled single-worker event loop concurrency bottlenecks in production FastAPI/Uvicorn applications, what was your primary culprit?
