What Actually Breaks at 1 Million Requests per Second?

One million HTTP requests every second sounds like a server-sizing problem: add more CPU, launch more machines, and place a load balancer in front of them.

The reality is much less comfortable.

At 1M operations per second, a one-in-a-million event may happen every second.

At that scale, the application is only one part of the system. The network can fill up before the CPU does. A database query that looks harmless with a few thousand rows can become disastrous with ten million. The machines generating the load can become the bottleneck. Even a managed load balancer may need capacity reserved in advance.

I recently watched Cododev's experiment, “Let’s Handle 1 Million Requests per Second, It’s Scarier Than You Think!”. Instead of treating it as a recipe to copy, I used it as a case study in how bottlenecks move as a system scales.

TL;DR: There is no magic stack for one million RPS. Define the workload, measure the system, remove the current bottleneck without weakening correctness, and repeat.

A note on attribution: the infrastructure, code, tests, and results discussed below belong to Cododev. I did not run this benchmark. This article is my explanation of what I learned from it.

The benchmark journey at a glance

These numbers describe one experiment, not a universal framework ranking. The useful story is how the limiting resource changed at every stage.

01. RPS is not a complete metric

RPS measures throughput, but the number is almost meaningless without the workload behind it.

Returning a tiny cached response is very different from:

• parsing a request body
• validating and transforming data
• generating a 30 KB JSON response
• reading from PostgreSQL
• writing durable data
• calling several downstream services

Payload size alone changes the problem. At one million responses per second, a 30 KB response represents roughly 30 GB of application data every second, before accounting for protocol overhead. A route can therefore become network-bound even while CPU capacity remains available.

Throughput is also different from concurrency. A useful approximation is:

concurrent requests ≈ requests per second × average response time

At 1M RPS and 1 ms average latency, about 1,000 requests are in flight. At 100 ms, that becomes roughly 100,000. Connection limits, memory usage, timeouts, and queue depth now look completely different.

So a serious benchmark needs more than one large RPS number. At minimum, I would want to know the route logic, response size, latency percentiles, error rate, connection count, test duration, protocol settings, hardware, and cost.

Throughput without workload context is a headline, not a capacity plan.

02. Every improvement revealed a new limit

The video begins locally with a route that returns a tiny JSON response. On the test machine, Express handled roughly 18–20K RPS. Fastify reached about 66K, while a small experimental Node.js framework used in the video reached about 73K.

Then the route was made more realistic. It parsed inputs, performed some CPU work, and returned around 30–32 KB of JSON. Throughput fell to roughly 8K RPS in one Node.js process. Running multiple processes to use more CPU cores raised it to roughly 36–50K RPS.

That first progression already contains two lessons:

1. Framework overhead matters when the handler does almost nothing.
2. The shape of the request matters far more than the name of the framework.

Moving the test to very large AWS machines made the next constraint visible. The simple route could reach millions of requests per second, but the larger response pushed enormous amounts of data. The network, rather than the CPU, became the limiting resource.

The bottleneck had moved.

03. The database changed the problem

Adding PostgreSQL made the target much harder. In the demonstrated setup, direct writes reached tens of thousands per second rather than one million. Increasing provisioned storage performance improved the result, but it also increased the monthly estimate and did not remove every limit.

The read tests were even more instructive. One version selected a random record by making the database do work proportional to the size of the table. With millions of rows, that approach became painfully slow. Versions that used an indexed identifier performed dramatically better.

The important lesson is not the exact query or its exact RPS. It is the amplification effect:

small inefficiency × huge request volume = a large operational problem

At 100 requests per second, an unnecessary scan may be hidden. At one million requests per second, it can dominate the system and the bill. Algorithmic complexity, indexes, query plans, and data access patterns stop being academic topics.

Scaling the database hardware is still a valid option, but it is not a substitute for understanding the work being scaled. Multiplying an inefficient operation across more machines can produce an expensive, inefficient system.

04. Redis helped, but durability did not disappear

The next stage moved hot reads and writes into Redis. A single Redis process introduced its own throughput ceiling, so the experiment used Redis Cluster to shard the keyspace across multiple primary nodes, with replicas for redundancy. That allowed the demonstrated route to cross one million operations per second.

The conceptual architecture looked something like this:

clients
   |
load balancer
   |
stateless application instances
   |
Redis Cluster (partitioned hot path)
   |
queue or controlled batches
   |
PostgreSQL (durable system of record)

This design is powerful when the business rules allow it, but it creates new questions:

• What happens to acknowledged writes if Redis fails before a batch reaches PostgreSQL?
• Can the background writer safely retry without creating duplicates?
• How is ordering preserved where it matters?
• What applies backpressure when the durable database falls behind?
• How are failed batches replayed and audited?
• Is eventual consistency acceptable for this data?

Redis can remove work from the synchronous request path, but it cannot remove the need to define durability and consistency. A benchmark can prove throughput for a chosen operation. It cannot decide the product's data-loss policy.

Moving data out of the request path changes when work happens. The work and its failure modes do not disappear.

05. C++ broke through the CPU-heavy route

For the larger PATCH route, adding more Node.js processes eventually stopped producing enough improvement. The implementation was then rewritten in C++ using Drogon, with RapidJSON replacing the framework's default JSON handling.

That detail matters. The first C++ version was not automatically faster just because it was C++. The JSON implementation was itself a bottleneck. Only after measuring and changing that hot path did the test reach its goal.

According to the creator's C++ benchmark repository, the final version sustained about 1M RPS for the tested 30 KB route while moving roughly 40 GB/s over the network. In the video, a 60-second run averaged about 1M RPS and briefly reached around 1.2M RPS.

This does not prove that Node.js, Java, Go, or any other language cannot handle one million requests per second. It proves that this particular C++ implementation performed best for this particular CPU- and network-heavy route on this particular machine. Change the handler, payload, runtime configuration, hardware, or protocol and the result can change too.

My takeaway is to optimize the narrow hot path when the measurements justify it. A system does not need to be written entirely in one language. Most services benefit more from development speed and maintainability; a small number of extreme endpoints may justify specialized native code.

06. The load generator is part of the benchmark

Producing one million requests per second is also an engineering problem. One client machine could not generate enough traffic, so the final test distributed the work across 60 load-generating instances.

Over a 30-minute run, the reported aggregate was approximately:

Those totals are more useful than a one-second peak because they show sustained behavior. Even so, a production readiness test would go further: inspect p95 and p99 latency, verify the response contents, include TLS and realistic connection behavior, test cold starts and failovers, and watch for coordinated omission in the load generator.

The test also exposed a less obvious dependency. Adding a network load balancer initially reduced throughput because its consumed capacity reached the available limit. Additional capacity had to be planned with the cloud provider.

Managed does not mean unlimited.

07. The four resource budgets

The cleanest mental model I took from the experiment is to track four separate budgets:

Optimizing the resource that is not currently limiting throughput usually adds complexity without improving the result. Measurement decides where to work next.

08. Before calling a system “1M-RPS ready”

A benchmark headline is a starting point. For a production claim, I would also ask for:

• a precise request and response contract
• realistic traffic distribution instead of one uniformly hot endpoint
• latency percentiles and timeout rates, not only averages
• correctness checks on returned and persisted data
• a sustained soak test, not only a short burst
• tests with TLS, authentication, logging, and observability enabled
• dependency failure, failover, retry, and backpressure tests
• capacity headroom instead of operation at the absolute limit
• a cost per million successful requests

That last metric matters. The fastest architecture is not automatically the best architecture. If expected traffic is 5K RPS, designing for a constant 1M may make the system harder to operate and more expensive without helping a single user.

09. My biggest takeaway

There is no “1M RPS architecture.” There is only a specific workload moving through a chain of finite resources.

In this experiment, the limiting factor moved from framework overhead to CPU utilization, then to network bandwidth, database work, Redis sharding, JSON processing, load generation, and managed load-balancer capacity. Each solved bottleneck exposed the next one.

That is what makes performance engineering interesting and a little scary. At one million operations per second, a one-in-a-million event is no longer rare. It is something the system may encounter every second.

The goal is not to guess the perfect architecture in advance. It is to define the workload, measure the system, find the current bottleneck, improve it without breaking correctness, and repeat.

Define → Measure → Find the bottleneck → Improve → Verify → Repeat

That lesson applies long before a system reaches one million requests per second.

Source and further reading

• Cododev: Let’s Handle 1 Million Requests per Second, It’s Scarier Than You Think!
• C++ implementation and benchmark notes

_{Writing note: I used AI to help structure and edit this article. I reviewed the technical claims against the original video and its source repository.}