Brotli on NGINX thread pools: ngx_mbrotli vs ngx_brotli

Summary

• With google/ngx_brotli, each response is compressed inside the NGINX worker’s event loop, so every other request on that worker waits. While large responses were compressed at Brotli levels 6 and 11, the p99 latency of small uncompressed requests rose from 1.9 ms to 17–514 ms.
• ngx_mbrotli with brotli_thread_pool compresses on NGINX thread pools. In the same test, the p99 of small requests stayed at 1.7–3.3 ms, the same as with no compression.
• One NGINX worker with a 4-thread pool served 3.8–15 times as many Brotli responses per second as google/ngx_brotli on 85 KB–2.2 MB files. The gain was largest at levels 9 and 11.
• The cost is memory. At 8 connections on 1–2.2 MB responses, a worker peaked at 138–995 MiB, 11–26 times as much as google/ngx_brotli. Responses longer than one 1 MiB segment came out 0.9–5.2% larger.
• Measured on 20 October 2026 with NGINX 1.30.6, one worker, an Intel Core i7-13700KF under WSL2. Every response was decoded and checked against the original by SHA-256.

ngx_mbrotli is an NGINX module for on-the-fly Brotli compression, written in Rust on top of the mbrotli encoder. It is a drop-in replacement for google/ngx_brotli, with the same directives, defaults and $brotli_ratio variable. Unlike google/ngx_brotli, it can compress on NGINX thread pools, so that compressing a large response does not stop the worker from serving anything else.

This post measures what that changes. One NGINX worker served large files compressed on the fly while a second stream of small uncompressed requests went to the same worker. With google/ngx_brotli, the 99th percentile latency of the small requests rose from 1.9 ms to as much as 514 ms. With ngx_mbrotli on a 4-thread pool it stayed at 1.7–3.3 ms.

p99 latency of small requests while large responses are compressed

Large requests arrive 8 at a time. ms, logarithmic scale.

No compression: 1.9 ms.

Small uncompressed requests (1.9 KB, 1,000 per second) on the same worker as the large compressed ones. wrk2 with coordinated omission corrected, median of three 30-second runs. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

The numbers come from a reduced run under WSL2 on 20 October 2026. They are good for comparing the modules with each other, less so as absolute figures for your hardware. How it was measured has the details and what was not measured.

Why compression delays other requests

An NGINX worker process serves thousands of connections from one thread. Its event loop waits for sockets to become ready, runs a short piece of work for each one, and goes back to waiting. That works because each piece of work is short. A filter that runs for 100 ms holds every other connection of that worker for 100 ms.

Compression is such a filter. google/ngx_brotli and the stock gzip module compress in the event loop. When a 1 MB JavaScript bundle is compressed at Brotli level 6, it takes about 16 ms of CPU time, and at level 11 about 1.2 s. A small request that arrives on the same worker meanwhile waits for compression to end. When several large responses arrive at once, the worker compresses them back to back, and the small request waits for all of them.

NGINX has had a way out since version 1.7.11, released on 24 March 2015 with “experimental thread pools support” (CHANGES). NGINX itself uses thread pools for blocking file I/O (aio threads). ngx_mbrotli uses them for compression.

How ngx_mbrotli compresses on a thread pool

With brotli_thread_pool, the module cuts a response into segments of brotli_parallel_segment_size, 1 MiB by default. Each segment becomes an NGINX thread task (ngx_thread_task_t) and is compressed on one of the pool’s threads. Up to brotli_parallel_tasks segments of one response, 4 by default, are compressed at once. A response that fits in one segment is compressed by one thread, in one piece.

When a task is done, its completion handler runs in the event loop. It passes the compressed bytes down the filter chain in order and wakes the connection, as a write event would. While tasks are in flight, the request stays marked as blocked and buffered, so NGINX neither finalizes it early nor frees memory a thread is still using. Meanwhile the event loop serves other connections.

Segments are compressed independently: Brotli back-references do not cross segment boundaries. That makes segments safe to compress in parallel. It also makes a response longer than one segment compress slightly worse, as the numbers below show.

load_module modules/ngx_mbrotli.so;

thread_pool brotli threads=4;

http {
    brotli on;
    brotli_comp_level 6;
    brotli_types text/plain text/css application/javascript application/json
                 image/svg+xml;

    brotli_thread_pool brotli;
}

NGINX must be built with --with-threads; nginx -V shows it. nginx.org’s packages are. The pool named default exists without being declared, with 32 threads per worker, and is shared with aio threads.

How long do other requests wait?

Small uncompressed requests (small.json, 1.9 KB) were sent at 1,000 per second over 32 connections. At the same time, a second stream requested large compressed files at a constant rate of about half of what google/ngx_brotli can sustain. All variants kept up with both streams.

p99 latency of small requests, large responses one at a time

Large requests arrive on one connection at a constant interval. ms, logarithmic scale.

The same test as the chart at the top, with one 30-second run. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

“8 at once” is the large stream’s 8 connections, which wrk2 starts together and which stay in phase. It is what happens when several clients ask for large files at the same moment. In the event loop, a small request then waits for up to 8 compressions. “One at a time” is a single connection at a constant interval, where a small request waits for at most one. It is the lower bound of the effect: the pool cut p99 by 9–40 times there. On the pool, the maximum latency of a small request one at a time was 3.6 ms.

Without a thread pool, ngx_mbrotli lowers p99 by 14–41% against google/ngx_brotli, only because mbrotli compresses faster. It blocks the event loop all the same. The difference in tail latency comes from moving compression off the event loop, not from a faster encoder.

How many requests does one worker serve?

The same worker under closed-loop load, with wrk and 8 connections:

Requests per second, relative to google/ngx_brotli

8 connections. 1 task per response serves within 3% of 4, except one case. Logarithmic scale.

wrk, median of three runs. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

The pool’s 4 threads use the 4 CPUs that google/ngx_brotli, with one worker, cannot. That explains about 4× at levels 4–6. The rest comes from the encoder. At level 9 the C encoder allocates a 32 MiB hash table for every response over 64 KB and frees it afterwards. That cost does not depend on the response size, so mbrotli’s lead is largest on the 85 KB file. At level 11 a 1 MiB segment compresses faster than a whole file, so the pool gains more than its 4 threads.

With 8 connections, 1 and 4 tasks per response served within 3% of each other, with one exception (+17%). The pool’s threads were already busy with different responses.

How fast is one large response?

More tasks per response help when a response is longer than one segment and nothing else competes for the threads:

Time to serve one 2.2 MB response

libraries.json, three 1 MiB segments, one connection. ms, logarithmic scale.

wrk, median of three runs. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

4 tasks served it 2.0–2.3 times as fast as 1 task, and 2.2–5.4 times as fast as google/ngx_brotli. Responses up to 1 MiB fit in one segment, so the number of tasks does not change how fast they are served. react-dom.development.js is a 1 MiB segment and a 31 KB one, so 4 tasks served it only 4–34% faster than 1.

Brotli level 11 on the fly

Level 11 is usually kept for pre-compressed static files, because one response can hold a worker for a second or more. On a thread pool, that time no longer holds up other requests: next to a stream of level-11 responses, small requests kept a p99 of 1.8–2.0 ms. A 2.2 MB response took 371 ms with 4 tasks, against 2,004 ms with google/ngx_brotli.

Level 11 still costs CPU. libraries.json needed 785 ms of CPU time per response on the pool, and the worker served 5.3 such responses per second on 4 CPUs. It fits responses that are large, compressible and requested rarely enough, not a busy API.

What the thread pool costs

Memory. In the event loop, a worker compresses one response at a time and needs one encoder. On a pool, every response being sent holds encoders for its segments until it is sent. Peak memory then grows with the number of responses in flight and with the number of tasks per response.

Peak memory of the worker

VmHWM at 8 connections, MiB, logarithmic scale. The labelled value is the pool with 4 tasks.

Highest of three runs. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

In the event loop, ngx_mbrotli needs 2–5 MiB more than google/ngx_brotli. On the pool, it needs 11–26 times as much.

CPU time. CPU time per request, including the pool’s threads, was within 0.6–1.4 times that of the event loop. At level 11 the pool needed 37–40% less, because 1 MiB segments compress faster than a whole file. At level 9 it needed 22–40% more.

CPU time per request

Worker and pool threads together, 8 connections. ms, logarithmic scale.

Median of three runs. NGINX 1.30.6, one worker on 4 CPUs, a pool of 4 threads; 20 October 2026.

Compression ratio. Responses longer than one segment came out 0.9–5.2% larger, because segments are compressed independently. Responses up to one segment are byte-for-byte what the event loop produces.

How to configure it

The defaults suit most sites: a 1 MiB segment and 4 tasks per response. What to change and why:

• Declare a pool of your own with thread_pool brotli threads=N;. The benchmark used 4 threads on 4 CPUs. Pool sizes were not compared, so start near the number of CPUs a worker can use.
• Keep brotli_parallel_segment_size 1m. With 4m, both files here fit in one segment, so the output is as small as in the event loop and peak memory is 19–42% lower. At levels 4–9, it served between 9% fewer and 14% more requests per second than 1m. At level 11 it served 35–37% fewer, because the whole file compresses slower than 1 MiB segments. With 4m, only responses longer than 4 MiB are split across threads.
• Lower brotli_parallel_tasks to save memory when single responses are rarely longer than a segment. On the 2.2 MB file, 4 tasks needed up to 1.9 times the memory of 1. Under load, 1 and 4 tasks served about the same number of requests.
• Bound encoder memory with brotli_parallel_compressors (4 by default), which limits how many compressors a worker keeps encoders for, and with brotli_retention_policy bounded <size> or release_all, which keep less memory between responses.
• Turn the pool off per location with brotli_thread_pool off;. Compressing a small response in the event loop holds the loop only briefly. In this run the pool still served 1.5–5.4 times as many 1.9 KB responses per second, so there is no measured reason to turn it off for small responses. A size threshold was not measured.

The ngx_mbrotli README documents every directive.

When you do not need it

• Static files are better compressed once, at build time, at level 11, and served with brotli_static on;. Nothing is compressed per request then.
• A CDN that compresses for you takes the work off NGINX. Traffic between the CDN and NGINX may still be worth compressing.
• Small responses only. If no response takes long to compress, the event loop is never held long, and the pool mostly adds memory.

How it was measured

Hardware. Intel Core i7-13700KF, Windows 10.0.26300.9711, WSL 2.7.14.0 with kernel 6.18.33.2-microsoft-standard-WSL2, Ubuntu 22.04.5. WSL2 shows 12 identical cores with 2 threads each and does not tell which are performance cores, so some of the pinned CPUs may be efficiency cores. Traffic went over loopback without TLS, and files were served from tmpfs (/dev/shm).

Build. NGINX 1.30.6, ./configure --with-compat --with-threads
--without-http_rewrite_module, gcc 11.4.0, both modules dynamic. google/ngx_brotli at commit a71f9312 with the Brotli 1.1.0 it pins as a submodule. ngx_mbrotli at commit 8f7508a with mbrotli 0.5.10, built with rustc 1.99.0. This commit predates the 0.4.0 release; the current release is 0.4.5, and the run has not been repeated on it.

Configuration. worker_processes 1, with the worker and its pool pinned to 4 physical cores, and thread_pool cp threads=4. On the pool, brotli_thread_pool cp with brotli_parallel_tasks 1 or the default 4. Everything else was left at its default, including the 1m segment and encoder reuse.

Load. Requests per second and CPU time: wrk 4.1.0 on 4 other physical cores, 2 threads and 8 connections, 4 seconds per run (8 seconds for files over 1 MB, 5 times as long at level 11) after a 1-second warm-up. NGINX was restarted for every run, and each run started once the worker had been idle for 300 ms. Three runs, in an order shuffled between runs; tables give the median. CPU time is the worker’s user and system time from /proc/<pid>/stat, including the pool’s threads. Latency: two wrk2 streams (commit 44a94c1) at once, open loop, with coordinated omission corrected; 8 at once is the median of three 30-second runs, one at a time is a single 30-second run.

Files. jQuery 3.7.1 (jquery.min.js, 85 KB) and React DOM 18.3.1 (react-dom.development.js, 1.03 MB) from cdnjs, the full cdnjs API listing (libraries.json, 2.2 MB) and its first 10 entries (small.json, 1.9 KB), all fetched on 20 October 2026.

Correctness. Before the runs, every response in every combination of module, variant, level and file was decoded with curl’s Brotli decoder and compared with the original by SHA-256. All matched.

Not measured. gzip, more than one worker process, pool sizes other than 4 threads, a response-size threshold for the pool, HTTP/2 and HTTP/3, slow clients, long runs and bare-metal Linux. Runs were 3 per cell and 4–40 seconds long, shorter than a full benchmark.

Install it

ngx_mbrotli has packages for Debian, Ubuntu, RHEL and its derivatives, and Amazon Linux: install it from packages, or build it from source. Existing google/ngx_brotli configuration keeps working; add brotli_thread_pool to move compression off the event loop.

The same measurements for Zstandard, with ngx_mzstd against tokers/zstd-nginx-module, are in zstd on NGINX thread pools.

Questions

Does Brotli compression block the NGINX event loop?

Yes, with google/ngx_brotli and the stock gzip module. The response is compressed by the worker process that serves it, inside its event loop, so the worker’s other connections wait until compression of that buffer ends. With ngx_mbrotli and brotli_thread_pool, compression runs on an NGINX thread pool and the event loop keeps serving other connections.

How do I compress Brotli on an NGINX thread pool?

Install ngx_mbrotli, which accepts google/ngx_brotli’s configuration unchanged, and add brotli_thread_pool default; next to brotli on;. NGINX must be built with --with-threads. To size the pool on its own, declare thread_pool brotli threads=4; in the main context and use brotli_thread_pool brotli;.

How much faster is ngx_mbrotli than google/ngx_brotli?

With a 4-thread pool, one worker served 3.8–4.5 times as many requests at levels 4–6 and 4.3–15 times as many at levels 9–11, on 85 KB–2.2 MB files. Without the pool, in the event loop like google/ngx_brotli, it served 5–64% more requests, and 4.4 times as many at level 9 on an 85 KB file.

How much memory does ngx_mbrotli need on a thread pool?

Much more than in the event loop. Every response being sent holds encoders for its segments. At 8 connections on 1–2.2 MB responses, one worker peaked at 138–995 MiB against 9–39 MiB for google/ngx_brotli. brotli_parallel_tasks 1, brotli_parallel_compressors and brotli_retention_policy bounded <size> keep it lower.

Is Brotli level 11 practical for on-the-fly compression?

It is still expensive: a 2.2 MB JSON file took 785 ms of CPU time at level 11. On a thread pool that time no longer delays other requests. With 4 tasks, one response is served in 371 ms instead of 2,004 ms with google/ngx_brotli, and one worker served 6.4–9.6 times as many level-11 responses per second.

Why not just add more NGINX worker processes?

It was not measured here. A connection stays on the worker that accepted it, so a request that lands on a worker busy compressing still waits for it. More workers make that less likely; they do not shorten the wait.