I Broke the Thundering Herd, One Worker Took 99% of Connections

Sharing one listening socket across several workers has a classic price: a connection arrives, the kernel wakes everyone waiting on that socket, one of them takes the connection, the rest go back empty-handed. We call it the thundering herd, and since Linux 4.5 it is supposedly solved by a single flag you pass to epoll_ctl — EPOLLEXCLUSIVE. The man page says so too, though what it actually says is that the flag is useful for avoiding thundering herd problems in certain scenarios.

I wanted to measure it. Eight workers, one socket, 500 connections, five repetitions. The herd really did break up: empty wakeups dropped from 869 to 1, and the workers' total CPU time was cut in half. Then I looked at the distribution table. In the median of the five runs, one worker had handled 497 of the 500 connections — and in all five runs, six workers had never woken up at all.

This piece is about that table. There is already a post on cache stampede and thundering herd on this blog, but that one lives at the application layer — clients asking for the same key at the same moment. The herd here is one floor down: processes sitting in the same socket's wait queue. Both the fix and the side effect of the fix are different.

The rig

I ran the measurement on this blog's own server: 18 vCPUs, Ubuntu kernel 6.8.0-147-generic, Python 3.12.3, load average around 5 at the time. Eight worker processes are created with fork, each builds its own epoll instance, registers the shared listening socket, and then blocks in epoll_wait. The parent dictates the registration order: no child calls epoll_ctl(ADD) until its turn comes. That detail matters later. The whole rig is three Python scripts, 305 lines.

The client side is plain: 500 connections, 2 ms apart, one at a time. Each worker counts three things — how many times it woke with an event, how many connections it accepted, and how many times it woke only to get EAGAIN from accept. That last one is the herd's invoice: you woke up, you ran, the plate was empty.

Four setups went through the rig. No flag: eight workers share one socket, nothing else. Flagged: they all register with EPOLLEXCLUSIVE. SO_REUSEPORT: no shared socket at all, each worker listens on the same address and port with its own. Half-migrated: half the workers registered with the flag and half without — the "we are moving to the flag but we are not done" state. Five runs each. The numbers below are the mean of those five, with the lowest and highest run in parentheses; the individual distributions I show in the text always come from one single run. The CPU figures are Python's figures — do not carry the absolute values over to a C server; the ratios are what matter.

The no-flag row shows 1369 wakeups for 500 connections: 2.7 workers get up per connection, and 63% of those trips are wasted. The flag brings the count down to one per connection — 498 to be exact, meaning a couple of wakeups accepted two connections at once. So far the advertising holds.

The problem is in the last column. Let me say up front: that 98.9% is not a discovery, it is the arithmetic of the setup. Work time is zero and connections arrive 2 ms apart, so the first worker is always back in epoll_wait before the next one lands. What happens to the monopoly under load comes shortly. The genuinely odd thing is inside the no-flag run, and I missed it at first.

The no-flag secret: the last registrant wins

Here is the no-flag distribution (one run, in registration order, left to right):

[6, 9, 22, 35, 44, 78, 107, 199]

If the herd means "everyone wakes, one grabs it", then who grabs it should come down to race conditions, and over a long run eight workers should end up with roughly similar shares. Instead the worker that registered last took thirty-three times as many connections as the one that registered first. Same direction, similar magnitude, in every run.

With EPOLLEXCLUSIVE on, the opposite happens: now the first registrant takes everything. Same machine, same program; the single difference of one flag flips the distribution end to end. A symmetry that sharp is not an accident. The answer is in the code.

Two lines, two ends

When epoll starts watching a file, fs/eventpoll.c adds its wait entry like this:

if (epi->event.events & EPOLLEXCLUSIVE)
    add_wait_queue_exclusive(whead, &pwq->wait);
else
    add_wait_queue(whead, &pwq->wait);

The two functions write to two different ends. add_wait_queue puts the entry at the head of the queue; add_wait_queue_exclusive puts it at the tail, via __add_wait_queue_entry_tail. The waking side, __wake_up_common in kernel/sched/wait.c, always walks the queue from the head and stops at the first exclusive entry that answers "yes, I woke up":

if (ret && (flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
    break;

The rest follows on its own. Without the flag, entries go to the head, so the queue is the reverse of registration order; the last registrant sits in front, gets woken first, and starts the accept race one step ahead. That is the 199-versus-6 gap. With the flag, entries go to the tail, so the queue matches registration order; the walk from the head hits the same first worker every time and stops there. That is the 98.9%.

In other words, "exclusive" is not a distribution policy. It is a stopping condition. The kernel tells you how many wakeups it will perform; it says nothing about which one.

(The measurement ran on a 6.8 kernel while the quotes above come from mainline. The four pieces that matter — ep_ptable_queue_proc, add_wait_queue_exclusive, the stopping line in __wake_up_common, and the ewake block coming up next — are identical in v6.8 and mainline.)

A control: does the winner follow registration order?

That explanation makes a testable prediction. If the winner is decided by epoll_ctl(ADD) order rather than by the order in which processes were created, then reversing the registration order should move the winner.

So I wired it that way. The fork order stayed fixed (0 through 7) and I reversed the registration order: the last process created became the first to register the socket. The result:

registration order: [7, 6, 5, 4, 3, 2, 1, 0]
accepted:           [500, 0, 0, 0, 0, 0, 0, 0]

The winner is worker 7 — born last, registered first. Running the same test without the flag, the winner was again the last registrant (226 against 3). What decides is not the process id, the creation order, or timing; it is the position in the socket's wait queue.

This is useful to know in practice: if one worker on a server is consistently doing many times the work of its peers, the place to look is not the worker, it is the startup order.

The kernel skips a busy worker — without breaking the order

So what happens when the first worker is drowning? There is a subtlety inside ep_poll_callback: for exclusive entries, the "I woke up" answer is only given if someone is actually waiting on that epoll instance (waitqueue_active). If the worker is outside epoll_wait, that is, busy with work in hand, the callback returns zero; __wake_up_common does not count it and moves on to the next entry.

Adding fake work after each accept puts numbers on it:

That table is what finally made the mechanism intuitive for me. The kernel does not treat the eight workers as a pool; starting from the head of the queue, it calls up the next one only as long as the work is not keeping up. The numbers land exactly where queueing theory says they should: serving a connection in 10 ms while one arrives every 2 ms needs about five workers in steady state — and in the measurement the first five carry 495 of the 500. A 5 ms service time needs about two and a half; the first three carry 489. The eighth worker did not wake once in any of these four runs; even at 10 ms, with seven workers busy at the same time, its turn never came.

The table explains one more thing: that 99% up top is not a fixed number, it is a product of low load. Running the same measurement three more times while the server was busier, the first worker's share fell as low as 88.6% — under load that worker spends more time outside epoll_wait, so the turn passes to the second. But across all eighteen zero-work runs of the flag, the one in front was always the first registrant, with a share somewhere between 88.6% and 100%. Load softens the monopoly; it does not remove it.

This is not bad design — it genuinely minimises wakeups, keeps a hot process hot, and is kind to cache locality. But read it as "load balancing" and it will mislead you. It is not a balancer; it is an overflow chain.

Kill the front worker and the monopoly moves next door

Once the mechanism is clear, the first production question is obvious: what happens if the worker at the head of the queue crashes? The rig answers that too — a SIGKILL to the first registrant at connection 250:

killed: registration rank 0 · surviving workers, in registration order
accepted: [250, 0, 0, 0, 0, 0, 0]

The next worker in line took every one of the remaining 250 connections, with zero empty wakeups. The monopoly does not break; it shifts by one seat. Three runs out of three. The operational consequence matters more than it looks: when you restart the crashed worker, it registers the socket again and lands at the tail of the queue. A single crash permanently rearranges your load distribution — yesterday's first is now last.

One legacy worker brings the herd back

There is an easily skipped sentence in the man page: if the same file is registered in several epoll instances, some with the flag and some without, the event goes to all of the non-exclusive ones and to at least one of the exclusive ones.

The "half-migrated" row in the table is exactly that: four of the eight workers registered with the flag, four without — the state of a setup in the middle of a migration. The distribution came out like this:

registration order: 0(E)  1     2(E)  3     4(E)  5     6(E)  7
accepted:            37   57     0    76     0   128     0   202

Wakeups are back up to 1294 and empty wakeups to 795; barely different from the no-flag row. On top of that, three of the four flagged workers accepted nothing. The cause is the queue layout again: non-exclusive entries are added at the head, so they all sit in front of the wake list (and in reverse among themselves: 7, 5, 3, 1), the exclusive ones are behind them, and the walk stops at the first of those. The measured ordering — 202 > 128 > 76 > 57 > 37, then three zeroes — is exactly what the queue layout predicts. A migration left half-finished gives you the bad half of both worlds.

nginx hit this wall in 2022

After finishing the measurement I went to look at what nginx does, since it is the best-known user of this flag. Its change log tells the story in two lines. 1.11.3 (July 2016): accept_mutex became off by default and nginx started using EPOLLEXCLUSIVE on Linux. 1.21.6 (January 2022): a bug fix — with EPOLLEXCLUSIVE on Linux, client connections were unevenly distributed among worker processes.

To see what went missing in between, it is enough to read the accept_mutex documentation: with that directive on, worker processes accepted connections by turn. The docs still say there is no need to enable it on systems that support EPOLLEXCLUSIVE — but when the flag took over the lock's herd-suppression job, it did not take over the turn-taking. The turn came back six years later, in user space.

The fix in the source is called ngx_reorder_accept_events, and its comment matches my table word for word: Linux usually notifies the process that added the socket to epoll first, so "most of the connections are handled by the first worker process". The remedy is to re-add the socket periodically. The interval is visible in the code: a DEL + ADD every 16 accepts (more often as a worker approaches its connection limit). Re-adding moves the entry to the tail of the queue, and the turn passes on.

I tried the same thing in my own rig — re-register every 16 accepts:

[64, 64, 64, 64, 64, 64, 64, 52]

Identical numbers in all three runs; nothing but the arithmetic of 16-accept rounds. Zero empty wakeups, still one wakeup per connection, worker CPU at 335 ms — indistinguishable from the flagged run's 294-380 ms band. So the entire gain from the flag survives and the distribution is fixed too. That a forty-line user-space workaround on top of a kernel behaviour lands this cleanly honestly surprised me.

Which raises a question: if it is this simple, why doesn't the kernel do it? It was proposed. In 2015, alongside EPOLLEXCLUSIVE, a second flag named EPOLLROUNDROBIN was suggested to spread wakeups across waiters in turn. One flag went in; the other did not. There is no EPOLLROUNDROBIN in today's mainline tree — eleven years later, still none.

One epoll, eight threads: same kernel, different outcome

To find out whether the problem lives in the idea of exclusive wakeups or somewhere else, I set the rig up once more: this time a single epoll instance shared by eight threads, no flag.

[95, 59, 62, 44, 76, 42, 57, 65]

No monopoly. At 1.6 wakeups per connection it is also below the 2.7 of the process version, though not zero.

The reason is in the source again. ep_poll re-adds its wait entry on every loop and removes it on wakeup; the removal is done by ep_autoremove_wake_function, and the comment inside ep_poll states the rationale plainly: when there are several waiters, each new wakeup should reach the next one. So the kernel does rotate inside a single epoll instance. Where it does not rotate is where separate epoll instances sit permanently in the socket's queue.

The monopoly is not caused by the flag; it is caused by the entry being permanent. nginx's fix makes exactly that temporary.

The flag's small print

While measuring I also tried, one by one, the things the flag refuses, because the list of restrictions in the man page is long and most of them show up in your code as a quiet EINVAL:

The second row is the one that will hurt in practice: a descriptor you added with EPOLLEXCLUSIVE can never be changed with EPOLL_CTL_MOD again. If you want to update the set of events you care about, the only path is delete and re-add. That also explains why nginx's rotation trick is a DEL + ADD: there is no other door.

How to check this on your own server

The one actionable output of this article is a question: are accepted connections spread evenly across your workers? There are two ways to look.

On nginx the cheapest is the access log: put $pid in your log_format (the docs define it as exactly that, the PID of the worker process) and count PIDs over one log file. The distribution shows up right there.

The way that works regardless of the server is to ask the kernel. I tried it on VPS3:

bpftrace -e 'kretprobe:inet_csk_accept /retval != 0/ { @accepts[comm, pid] = count(); }'

My lab run came out of it as these lines:

@accepts[python3, 3259746]: 4
@accepts[python3, 3259745]: 196

In a four-worker run, 196 of 200 connections had gone to a single process. The same dump also listed the server's own services with their own counts — so this is not a lab-only trick but a measurement you can run directly on a live machine. A one-minute run is enough to tell you whether the imbalance is there.

Which one to pick

The decision frame this measurement leaves behind:

• With a single worker the flag means nothing. nginx only turns it on when worker_processes is greater than one.
• If you care about CPU and not about the spread between workers — for instance, your workers are already loaded enough to saturate — the flag alone is enough. Under load the queue opens downward by itself.
• If you want an even spread at low or medium load, the flag alone will not get you there. Either do what nginx does (periodic DEL+ADD) or use SO_REUSEPORT.
• If you pick SO_REUSEPORT, know its two blind spots. The first is distribution: the kernel picks the socket with a hash derived from the connection's four-tuple (inet_ehashfn: destination address, local port, source address, source port), with no regard for how busy a worker is. In my rig the shares settled in the 14-15% band because every connection was an exact copy of every other; in real traffic, if one long-lived connection jams a worker, new connections in that queue wait even while the worker next door sits idle. If you want to take the choice back, the door is open: SO_ATTACH_REUSEPORT_CBPF/EBPF lets you pick the socket in the group yourself (on TCP since Linux 4.6). The second blind spot is sneakier, and it has a measurement of its own: when you swap workers, the connections sitting in the closing listener's queue die silently unless tcp_migrate_req is on — and that switch is off by default.
• If you are migrating, migrate in one go. Running flagged and unflagged workers side by side brought the herd back in full.

What the docs promised, and what I read

The real lesson I took from this is not about the flag. For EPOLLEXCLUSIVE the man page says that one or more of the epoll descriptors will receive an event. For years I read that sentence as "one of them gets it, problem solved"; what the sentence actually gives you is only a count. It makes no promise about which one. The promise it never made, I had added myself.

This trap is common in kernel interfaces: the documentation tells you a quantity and we read a policy. The distance between "the minimum number of wakeups" and "a fair share" was, in my rig, 497 out of 500 connections. Without measuring I would never have noticed; the server was running, requests were being answered, and six workers simply never woke.

The question you can put to your own setup is simple: have you ever compared the number of accepted connections across your worker processes? If they are not even, that is not a bug — it is most likely exactly what the kernel promised.

Official Sources

• epoll_ctl(2) — EPOLLEXCLUSIVE, its restrictions and the mixed-registration rule
• fs/eventpoll.c — ep_ptable_queue_proc, ep_poll_callback and ep_autoremove_wake_function
• kernel/sched/wait.c — add_wait_queue_exclusive and __wake_up_common
• include/uapi/linux/eventpoll.h — the EPOLLEXCLUSIVE bit definition
• nginx CHANGES — the 1.11.3 and 1.21.6 entries
• nginx Core functionality — the accept_mutex directive
• nginx HTTP Core — the $pid embedded variable
• socket(7) — taking over socket selection with SO_ATTACH_REUSEPORT_CBPF/EBPF