Find Your API's p99 Latency Spikes with a Zero-Dependency Python Benchmark

Your API's median latency looks fine, but users keep reporting the occasional request that hangs for a second. Averages hide that. Percentiles show it, and you don't need k6, Locust or a JVM to measure them. ApiBench is one Python file, standard library only, that fires concurrent requests at an endpoint and reports p50/p90/p95/p99. It exits non-zero when your p99 or error-rate budget is blown, so it can gate a CI job.

All the numbers in this post come from runs on my machine against a small local test server. They show how the tool behaves; they aren't benchmarks of anything you run.

Install and first run

Unzip the download and check it runs (any recent Python 3, nothing to pip install):

python3 apibench.py --demo -n 100 -c 5

--demo starts a built-in mock server on 127.0.0.1:8765 that sleeps 15 ms per request, and benchmarks it:

 Completed Reqs:    100 in 0.332s
 SLA Gatekeeper:    [PASS]
 Throughput:        301.55 req/sec (14.36 KB/s)
 Success Rate:      100/100 (100.00%)
 Median (p50):         16.17 ms
 p99:                  18.06 ms  (Target SLA: < 250.0 ms)

The handler sleeps 15 ms, the median is 16 ms, and the p99 sits right next to it: about a millisecond of client and loopback overhead on top of the work. That's what a healthy tail looks like. The rest of this post is about the unhealthy kind.

A real investigation: where does a 1-second p99 come from?

I wrote a minimal JSON endpoint with Python's ThreadingHTTPServer and hit it with 500 requests from 20 workers, with a 50 ms p99 budget:

python3 apibench.py http://127.0.0.1:9000/health -n 500 -c 20 --sla-p99 50 --json r.json --markdown r.md
echo $?

 Completed Reqs:    500 in 1.236s
 SLA Gatekeeper:    [FAIL]
 Throughput:        404.58 req/sec (4.74 KB/s)
 Success Rate:      500/500 (100.00%)

--- Latency Profile (Wall Time) ---
 Min:                   0.90 ms
 Mean:                 28.71 ms
 Median (p50):          1.72 ms
 p90:                   2.98 ms
 p95:                   4.86 ms
 p99:                1023.13 ms  (Target SLA: < 50.0 ms)
 Max:                1212.12 ms
 StdDev:              164.09 ms

Exit code: 2. Every request succeeded, the median is under 2 ms, and yet the slowest 1% take about a second. A mean of 28 ms describes no request that actually happened.

A delay of about one second with no errors is the signature of TCP SYN retransmission. When a server's listen backlog is full, the kernel drops the new connection attempt, and the client retries it after the initial retransmission timeout of one second. socketserver defaults to request_queue_size = 5, and ApiBench opens a fresh connection for each request, so 20 workers overflow it. To test that theory, I changed one line in the server:

class S(http.server.ThreadingHTTPServer):
    request_queue_size = 128

Same command, same machine:

(This table comes from a second pair of runs, so the backlog-5 column differs slightly from the output above.)

The median went up a little, because the server now accepts more connections at once and they compete for the CPU. The tail collapsed, though, and throughput went up almost tenfold because clients stopped sleeping in retransmission. That's why you gate on p99 and not on the average.

Gating CI on latency and errors

The exit code is the whole integration:

• 0: p99 ≤ --sla-p99 (default 250 ms) and error rate ≤ 1%
• 2: either budget was exceeded
• 1: bad invocation, such as a missing URL

python3 apibench.py "$STAGING_URL/health" -n 300 -c 10 --sla-p99 200 --markdown apibench.md

Any status outside 200–399 counts as an error. I ran it against an endpoint that returns a random 503 about 10% of the time:

 Success Rate:      269/300 (89.67%)
 Error Rate:        10.33%
 p99:                  15.59 ms  (Target SLA: < 250.0 ms)

--- Status Code Breakdown ---
 Status 200  ->    269 ( 89.7%)
 Status 503  ->     31 ( 10.3%)

--- Failure Diagnostics ---
  - [31x] HTTP Status 503

Latency passed, errors didn't: exit 2. A slow endpoint fails the other way. Against a handler that sleeps 300 ms, p99 came out at 304.99 ms against a 250 ms budget, and the exit code was again 2.

--json writes every metric (latency_p99_ms, error_rate_pct, status_codes, overall_sla_passed and so on), which is useful for tracking trends between builds. --markdown writes a table you can post as a PR comment.

Other options: -m POST -d '{"q":1}' for a request body, -H 'Authorization: Bearer …' (repeatable) for headers, and -t for the per-request timeout. HTTPS works with the default certificate verification.

How it works

Each request is timed around a raw http.client call, so the measurement covers connect, send, the server's work and reading the full body:

start_time = time.perf_counter()
conn = http.client.HTTPConnection(host, port, timeout=self.config.timeout_seconds)
conn.request(self.config.method, path, body=body_bytes, headers=self.config.headers)
resp = conn.getresponse()
content = resp.read()
latency = (time.perf_counter() - start_time) * 1000.0

A ThreadPoolExecutor with -c workers runs -n of these. Percentiles use linear interpolation between the closest ranks rather than picking the nearest sample:

k = (len(sorted_values) - 1) * (percentile / 100.0)
f = math.floor(k); c = math.ceil(k)
if f == c:
    return sorted_values[int(k)]
return sorted_values[int(f)] * (c - k) + sorted_values[int(c)] * (k - f)

The gate is just p99 <= sla_p99_ms and error_rate <= sla_max_error_rate.

Limitations

• A new connection for every request. There's no keep-alive, so the numbers include the TCP handshake (and the TLS handshake on HTTPS). That's realistic for clients that don't pool connections, but pessimistic compared with a browser or a pooled HTTP client.
• Threads, not an event loop. It's fine for tens or low hundreds of concurrent workers. If you need tens of thousands of virtual users, use a distributed load tester.
• Fixed request count, not a duration or a ramp. Each run sends -n identical requests, with no scenario scripting.
• The "Efficiency Score" and cost projection in the report are simple formulas based on your SLA and the --cost-per-million you pass in. Use the percentiles and the exit code to make decisions.

FAQ

Can I benchmark production with it? Technically yes, but every request is real traffic. Point it at staging or a health endpoint, and keep -n and -c modest.

Why do my numbers differ from wrk or k6? Those tools reuse connections. ApiBench doesn't, so its latencies include the connection setup.

Does it need anything installed? No, only the Python standard library.

ApiBench is a one-time purchase that includes the single-file source and README: get ApiBench ($29). Details are on the ApiBench product page.