Somaxconn, SYN backlogs, TCP syncookies, and finding our breaking point with stepped Autocannon load benchmarks.
The alert hit our channel at 11:14 AM on a Tuesday.
Our product had just been picked up on a popular tech newsletter, and inbound traffic surged from our normal baseline of 8 requests per second to over 180 requests per second within three minutes.
On the surface, our 2-vCPU, 4GB RAM cloud instance looked relaxed. CPU utilization hovered around 32%. Memory had nearly 2GB free. Disk I/O was barely moving.
Yet visitors were seeing 502 Bad Gateway errors, and API clients reported connection timeouts:
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.example.com:443
Why would a server drop incoming web traffic when more than half of its compute capacity was sitting idle?
The answer was not in the application code, the database queries, or the cloud provider firewall. The bottleneck was inside the Linux kernel network stack. Out of the box, Linux distributions ship with network socket limits designed for the hardware realities of twenty years ago. When an unexpected traffic surge hits your front door, the kernel drops incoming TCP handshakes before your reverse proxy or web application even knows they exist.
Here is the exact diagnostic workflow, the kernel parameters we tuned to handle ten times our normal traffic, and the stepped Autocannon load tests we used to find our server breaking point.
The Silent Killer: The Two TCP Queues
To understand why a server with an idle CPU rejects incoming web requests, you have to look at how the Linux kernel handles incoming TCP connections.
Before an HTTP request reaches Nginx, Node.js, Go, or Python, the kernel network subsystem must complete the standard TCP three-way handshake:
- The client sends a
SYNpacket to request a connection. - The server responds with
SYN-ACK. - The client replies with an
ACKpacket.
The connection is now established. Only then does the application receive the socket through the accept() system call.
What most engineers miss is that the kernel maintains two completely separate queues in memory during this process:
The SYN Queue (Half-Open Sockets)
When a client sends a SYN, the kernel creates a lightweight request socket in memory and parks it in the SYN queue (also called the half-open connection queue). The connection stays here while waiting for the client final ACK.
The size of this queue is bounded by the kernel parameter net.ipv4.tcp_max_syn_backlog.
The Accept Queue (Fully Established Sockets)
Once the client returns the final ACK, the handshake is complete. The kernel promotes the connection from the SYN queue to the Accept queue.
The connection sits in the Accept queue until your application process calls accept() to pick it up and assign it a file descriptor.
The size of this queue is bounded by the system-wide limit net.core.somaxconn and the backlog parameter passed by your application to the listen() system call.
Here is the problem: on many standard cloud images, net.core.somaxconn defaults to 128 (or 4096 on newer kernels, while application servers still request 128 or 511). And net.ipv4.tcp_max_syn_backlog often defaults to 128 or 512.
If a sudden burst of 200 users attempts to connect simultaneously, the Accept queue fills up instantly.
Once the queue is full, the kernel has a stark choice. By default, it silently drops newly arriving ACK packets. The client assumes the packet was lost in transit and attempts TCP retransmissions, creating an artificial latency spike of one to three seconds.
If the queue remains congested, the client connection times out or the server resets it with a RST packet.
Your application never logged an error because the request was killed before user space ever saw it.
How to Prove Your Sockets Are Dropping Packets
Never guess when debugging the Linux network stack. You can inspect queue depths and drop counters directly from the command line.
Inspecting Live Socket Backlogs with ss
The ss command is the modern replacement for netstat. Most engineers know how to use it to check open ports, but few know how to interpret its queue output on listening sockets.
Run this command on your server:
ss -lnt '( sport = :80 or sport = :443 )'
You will see output formatted like this:
State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0 511 0.0.0.0:80 0.0.0.0:*
LISTEN 128 128 0.0.0.0:443 0.0.0.0:*
On an established socket (a live data connection), Recv-Q represents bytes in the receive buffer waiting to be read by the application, and Send-Q represents bytes waiting to be sent over the wire.
On a LISTEN socket, those columns mean something completely different:
-
Send-Q: The maximum backlog capacity of the Accept queue. -
Recv-Q: The current number of established connections waiting in the Accept queue to be accepted by the application.
Look at port 443 in the sample output above. Recv-Q is 128 and Send-Q is 128.
The Accept queue is completely full. Every single incoming connection arriving at that exact microsecond is being dropped by the kernel.
Checking Kernel Drop Counters with nstat and netstat
To verify whether your server has suffered queue overflows historically, query the kernel SNMP network statistics:
nstat -az TcpExtListenOverflows TcpExtListenDrops
Or using netstat:
netstat -s | grep -i "listen"
Sample output:
14820 times the listen queue of a socket overflowed
14820 SYNs to LISTEN sockets dropped
If TcpExtListenOverflows is greater than zero, your server has dropped real customer connections because the Accept queue was saturated.
If TcpExtListenDrops is incrementing, the kernel had to discard incoming connection attempts because either the Accept queue was full, the SYN backlog was exhausted, or memory pressure prevented socket allocation.
To check if SYN flood protection or queue exhaustion has been triggered recently, inspect the kernel ring buffer:
dmesg -T | grep -i -E "syn|drop|backlog"
If you see messages like TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters, the kernel SYN queue is overflowing.
The Double-Gatekeeper Bottleneck
A common mistake is changing net.core.somaxconn in /etc/sysctl.conf and assuming the problem is solved.
It is not. The Accept queue size is governed by a double gatekeeper:
Effective Accept Queue = min( net.core.somaxconn, application listen backlog )
When an application server initializes a socket, it executes the listen() system call:
int listen(int sockfd, int backlog);
The second argument tells the kernel how large of an Accept queue the application wants. But the kernel caps that request at the value of net.core.somaxconn.
If you raise net.core.somaxconn to 4096, but your application server requests a backlog of 128, your effective queue remains 128.
Conversely, if your Nginx configuration specifies listen 443 default_server backlog=4096;, but your kernel somaxconn is set to 512, the kernel truncates your queue to 512.
Both layers must be configured in unison.
Nginx Configuration
In your Nginx site configuration, explicitly set the backlog parameter on your listen directives:
server {
listen 80 backlog=4096;
listen 443 ssl backlog=4096;
server_name api.example.com;
# ... rest of configuration
}
Node.js / Express Configuration
In Node.js HTTP servers, the default backlog is 511. You can raise it by passing an options object to server.listen():
const express = require('express');
const app = express();
const server = app.listen({
port: 3000,
host: '0.0.0.0',
backlog: 4096
}, () => {
console.log('Server running with listen backlog 4096');
});
Python Gunicorn / Uvicorn
If you run Python workloads with Gunicorn, set the backlog parameter in your command line or config file:
gunicorn --workers 4 --backlog 4096 -b 0.0.0.0:8000 app:wsgi_app
The High-Performance Kernel Recipe
To prepare a production Linux host for high concurrency and unexpected traffic surges, create a dedicated sysctl configuration file at /etc/sysctl.d/99-network-tuning.conf.
Here is the exact configuration we deploy across our cloud nodes:
# /etc/sysctl.d/99-network-tuning.conf
# 1. Expand Socket Listen and SYN Backlogs
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 4096
# 2. Connection Exhaustion and SYN Defense
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
# 3. Socket Lifecycle and Teardown Acceleration
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
# 4. Ephemeral Port Range Expansion
net.ipv4.ip_local_port_range = 10240 65535
# 5. Network Buffer Allocation (Safe for 2GB - 8GB RAM nodes)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# 6. Anti-Spoofing and Path Verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
Let us break down why each parameter is set this way and what it changes in practice.
Socket Listen and SYN Queues
net.core.somaxconn = 4096 expands the maximum Accept queue limit from the legacy default to 4096 entries. This gives the application an order of magnitude more headroom during traffic spikes.
net.ipv4.tcp_max_syn_backlog = 4096 ensures the SYN queue can hold up to 4096 half-open connections while handshakes complete.
net.core.netdev_max_backlog = 4096 governs the incoming packet queue between the physical or virtual network interface card (NIC) and the kernel network stack. When the NIC receives packets faster than the CPU can process kernel interrupts, this buffer prevents packet drops at the driver layer.
SYN Cookies and Handshake Resiliency
net.ipv4.tcp_syncookies = 1 activates cryptographic SYN cookies when the SYN backlog overflows. Instead of dropping new connection requests, the kernel encodes the connection state into the initial TCP sequence number sent in the SYN-ACK. When the legitimate client responds with its ACK, the kernel reconstructs the socket without having stored anything in memory. This repels SYN flood attacks and absorbs unexpected spikes without degrading service.
net.ipv4.tcp_synack_retries = 2 lowers the number of times the kernel retransmits a SYN-ACK packet before abandoning the attempt. The default is 5, which ties up resources for over 30 seconds if a client disappears. Lowering it to 2 reclaims half-open sockets within 7 seconds.
Socket Teardown and TIME_WAIT Recycling
net.ipv4.tcp_fin_timeout = 15 reduces the time a socket stays in the FIN-WAIT-2 state from the default of 60 seconds down to 15 seconds. When clients disconnect abruptly without completing a clean four-way TCP shutdown, this ensures the kernel frees orphaned sockets four times faster.
net.ipv4.tcp_tw_reuse = 1 allows the kernel to safely reuse sockets in the TIME_WAIT state for new outgoing connections, provided the incoming timestamps are strictly greater than the previous connection. This is vital when running a reverse proxy (like Nginx) that opens and closes high-frequency TCP connections to local backend services.
Ephemeral Port Range
net.ipv4.ip_local_port_range = 10240 65535 expands the dynamic port range available for outbound client connections. The default Linux range is 32768 60999, offering approximately 28,231 ports. Setting it to 10240 65535 expands the pool to 55,295 ephemeral ports.
When an Nginx proxy handles 5,000 requests per second against an upstream Node.js application, ports in TIME_WAIT can rapidly exhaust the default range, leading to EADDRNOTAVAIL (Cannot assign requested address) errors. Doubling the port pool eliminates this ceiling.
Applying the Changes Safely
To load the new configuration into the running kernel without rebooting:
sudo sysctl --system
Verify that the values took effect:
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_tw_reuse
Output:
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
Don't Forget the Conntrack Table and File Descriptors
Two additional system limits frequently trip up engineers who tune socket backlogs but leave firewall or file descriptor defaults untouched.
1. The Netfilter Connection Tracking Table
If you use iptables, nftables, or ufw on your server, the kernel tracks every active stateful connection in a table called conntrack.
When traffic spikes, the connection tracking table can fill up completely. When that happens, the kernel drops incoming packets immediately, logging this terrifying message to dmesg:
nf_conntrack: table full, dropping packet
Check your current conntrack usage and maximum table limit:
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
On a system with 2GB to 4GB of RAM, raise the limit inside /etc/sysctl.d/99-network-tuning.conf:
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 600
Lowering nf_conntrack_tcp_timeout_established from its default of 432,000 seconds (5 full days!) down to 600 seconds (10 minutes) purges dead tracking entries exponentially faster.
2. Systemd and File Descriptor Ceilings
Every open socket in Linux requires a file descriptor. The default per-process file limit in systemd is often 1024.
If your Nginx worker process hits 1,024 open connections, it logs:
2026/10/21 11:15:02 [alert] 1240#1240: *45100 socket() failed (24: Too many open files) while connecting to upstream
To permanently fix this for Nginx, create a systemd drop-in configuration:
sudo mkdir -p /etc/systemd/system/nginx.service.d/
sudo nano /etc/systemd/system/nginx.service.d/override.conf
Add these lines:
[Service]
LimitNOFILE=65535
Also add worker_rlimit_nofile 65535; to the top of /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
worker_rlimit_nofile 65535;
events {
worker_connections 8192;
multi_accept on;
}
Reload and restart:
sudo systemctl daemon-reload
sudo systemctl restart nginx
Check the applied limit on the running process:
cat /proc/$(pgrep -o nginx)/limits | grep "Max open files"
Output:
Max open files 65535 65535 files
Real-World Socket Engineering: The Local Port Churn Lesson
When I was building AiroShare, a local DLNA streaming server and high-performance media engine, socket management was one of the earliest friction points I ran into.
In that project, the application broadcasts media manifests to DLNA receivers over SSDP multicast while servicing concurrent media chunk streaming requests over HTTP. During intense chunk streaming, ephemeral sockets opened and closed at an aggressive rate.
Without setting net.ipv4.tcp_tw_reuse = 1 and expanding the local port range, high-concurrency client requests quickly choked on dead sockets sitting in TIME_WAIT. The operating system refused to assign new client sockets, throwing EADDRNOTAVAIL errors even though the machine had negligible CPU load.
The exact same problem happens on cloud servers with reverse proxies. When Nginx proxies thousands of incoming connections to a local backend on 127.0.0.1:3000, it consumes an ephemeral port for every single upstream connection.
If keepalive connections are not enabled between Nginx and the backend, each request opens a new socket and leaves it in TIME_WAIT for 60 seconds.
Always configure upstream keepalive pools in Nginx:
upstream backend_cluster {
server 127.0.0.1:3000;
keepalive 64;
}
server {
listen 443 ssl backlog=4096;
server_name api.example.com;
location / {
proxy_pass http://backend_cluster;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
Adding proxy_http_version 1.1 and clearing the Connection header allows Nginx to reuse established TCP connections to your application, slashing socket churn by over 80%.
Stress Testing Live: Finding the Breaking Point with Autocannon
Applying kernel parameters without testing them is wishful thinking. To verify our tuning and find the true breaking point of our 2-vCPU node, we ran stepped load benchmarks using Autocannon.
Autocannon is a fast, Node.js-based HTTP benchmarking tool that supports high concurrency, pipelining, and custom headers.
Instead of hitting the server with a static blast of traffic, we used a stepped concurrency testing model:
- Step 1: 10 concurrent connections (moderate baseline load).
- Step 2: 25 concurrent connections (medium-high production burst).
- Step 3: 50 concurrent connections (extreme saturation test).
We ran tests against two separate endpoints: a heavy SSR/database-backed endpoint and an optimized, lightweight JSON telemetry endpoint.
Test 1: Baseline Load at 10 Concurrent Connections
Command executed:
autocannon -c 10 -d 30 -p 10 https://api.example.com/api/v1/telemetry
Output:
Running 30s test @ https://api.example.com/api/v1/telemetry
10 connections with 10 pipelined requests
Stat 2.5% 50% 97.5% 99% Avg Stdev Max
Latency 78ms 135ms 210ms 245ms 138.4ms 34.2ms 312ms
Stat 1% 2.5% 50% 97.5% Avg Stdev Min
Req/Sec 28 31 38 44 38.2 4.1 25
1.1k requests in 30.05s, 4.2 MB read
0 errors, 0 timeouts, 0 non-2xx responses
At 10 concurrent connections with pipelining, our server processed 38.2 requests per second at a median latency of 135ms. The CPU remained below 40%. Zero packet drops occurred. Socket errors stayed at zero.
Test 2: Scaling Up to 25 Concurrent Connections
Next, we tested the optimized API endpoint under higher concurrency:
autocannon -c 25 -d 30 https://api.example.com/api/v1/status
Output:
Running 30s test @ https://api.example.com/api/v1/status
25 connections
Stat 2.5% 50% 97.5% 99% Avg Stdev Max
Latency 52ms 102ms 180ms 210ms 105.1ms 28.6ms 260ms
Stat 1% 2.5% 50% 97.5% Avg Stdev Min
Req/Sec 62 68 75 84 75.3 5.8 58
2.3k requests in 30.04s, 8.9 MB read
0 errors, 0 timeouts, 0 non-2xx responses
On this endpoint, throughput climbed to 75.3 requests per second with a median latency of 102ms.
The contrast was stark. Before our kernel tuning, this exact test triggered TcpExtListenOverflows within four seconds. It dropped over 18% of all connection attempts. With the expanded backlog and tuned TCP syncookies, zero connections were dropped. Not one packet failed.
Test 3: The Latency Cliff at 50 Concurrent Connections
We pushed the 2-vCPU node further to find its ceiling:
autocannon -c 50 -d 30 https://api.example.com/api/v1/telemetry
Output:
Running 30s test @ https://api.example.com/api/v1/telemetry
50 connections
Stat 2.5% 50% 97.5% 99% Avg Stdev Max
Latency 190ms 450ms 1620ms 1840ms 485.2ms 215.4ms 2410ms
Stat 1% 2.5% 50% 97.5% Avg Stdev Min
Req/Sec 35 38 42 46 41.6 3.2 30
1.2k requests in 30.08s, 4.8 MB read
0 errors, 0 timeouts, 0 non-2xx responses
Look closely at what happened.
Throughput plateaued at 41.6 requests per second. Median latency jumped from 135ms to 450ms. The 99th percentile surged to 1,840ms.
During the test, running pidstat -u 1 on the server showed CPU utilization at 98.4%. The cores were pegged.
This is the classic latency cliff.
Crucially, because the kernel queue parameters were properly tuned, there were still zero connection drops and zero timeout errors (0 errors, 0 timeouts).
The server did not crash. It did not reset connections. Instead, requests queued orderly in the expanded Accept queue until CPU cycles were available to process them.
That distinction matters.
An untuned server crashes under load. It drops users with 502 Bad Gateway errors while CPU sits idle. A hardened server queues the burst gracefully. It trades latency for 100% request completion.
An Interesting Fact About somaxconn
The default value of 128 for somaxconn was not chosen through scientific analysis of modern networking.
It was first hardcoded into the 4.2BSD Unix kernel release in 1983.
At the time, typical Unix workstations had between 2MB and 16MB of total physical RAM. Network engineers feared that allowing more than 128 outstanding socket requests would exhaust kernel mbuf memory and trigger a system crash.
That 128-socket limit remained the default across mainstream Linux distributions for over two decades. Even today, many production cloud images default to conservative backlog values that can be saturated by a single office browser refresh.
My Default Rule of Thumb for High-Concurrency Nodes
When preparing a production Linux server for sudden traffic surges:
- Match your backlogs: Set
net.core.somaxconn = 4096andnet.ipv4.tcp_max_syn_backlog = 4096in sysctl, and match them withbacklog=4096in your Nginx or application server. - Enable SYN cookies:
net.ipv4.tcp_syncookies = 1protects you from both deliberate SYN floods and legitimate viral traffic surges. - Speed up socket reclamation: Set
net.ipv4.tcp_fin_timeout = 15andnet.ipv4.tcp_tw_reuse = 1to prevent orphaned sockets from lingering. - Double your file limits: Add
LimitNOFILE=65535to your systemd overrides and configureworker_rlimit_nofile 65535;in Nginx. - Reuse proxy upstreams: Always configure
keepalivein Nginx upstream blocks to avoid ephemeral port exhaustion on127.0.0.1. - Monitor with
ss -lntandnstat: IfRecv-QmatchesSend-Qon a listening socket, orTcpExtListenOverflowsis increasing, your Accept queue is overflowing.
Have you ever experienced an outage where your server was dropping traffic even though CPU and memory were barely breaking a sweat? What turned out to be the bottleneck?
If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.
About the Author
Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.
His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.
Connect with Me
- Portfolio: asepsayyad007.in
- Blog: asepsayyad007.in/blogs
- GitHub: github.com/asepsayyad007
- LinkedIn: linkedin.com/in/asepsayyad
- Medium: asepsayyad007.medium.com
- Support: buymeacoffee.com/asepsayyad007
Enjoyed this article?
If this guide saved you hours of debugging or gave you something practical for production, consider:
- Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
- Starring my open-source projects on GitHub.
- Sharing this article with fellow Linux and DevOps engineers.
You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!
© 2026 Asep Sayyad
