nginx under load: workers, connections, and the kernel underneath

Someone says "nginx is slow." You have a terminal and no idea whether the problem is nginx, the kernel underneath it, or the thing nginx is proxying to. This is the reference I wanted when that happened to me: the mental model, the arithmetic that sets your ceiling, and the order to check things in so you stop guessing.

At the end I audit a real reverse proxy on my own box and find three things wrong with it.

Where this design came from

nginx exists because of one specific problem, and knowing it makes every directive below make sense.

In 1999 Dan Kegel wrote up the C10K problem: how do you serve ten thousand concurrent connections on a single box? At the time the answer was "you don't," and the reason was architectural. Apache's classic multi-processing modules gave you one process per connection (prefork) or one thread per connection (worker). That model is easy to reason about and completely fine when connections are few and short. But the cost scales with the number of open connections, not with the amount of work — and on the public internet most connections are idle most of the time, waiting on a slow client, a keepalive timeout, or a dial-up link.

Ten thousand connections meant ten thousand processes or threads. Each one carries its own stack, its own scheduler entry, and its own memory footprint measured in megabytes rather than kilobytes. Long before you run out of CPU you run out of RAM, and the scheduler spends its time context-switching between threads that have nothing to do.

Igor Sysoev inverted it. nginx runs a fixed, small number of processes — one per core — and each one runs an event loop over thousands of sockets. Cost now scales with active work instead of open connections. Ten thousand idle keepalive connections are a few megabytes and no scheduler pressure at all, because an idle connection is just a file descriptor in an epoll set, not a thread waiting to be woken.

Apache closed a lot of this gap with the event MPM in 2.4, which hands idle keepalive connections off to a small dedicated pool instead of pinning a thread to each. It is a genuine improvement and the comparison is no longer the massacre it was in 2004. But a request actually in flight still occupies a worker thread, so the shape of the trade-off is unchanged: Apache's model is per-request-thread with an event-driven waiting room bolted on, nginx's is event-driven throughout.

The practical consequence, and the reason this matters for the rest of the article: nginx is cheap at holding connections and expensive at blocking. That is exactly the profile you want in a reverse proxy or TLS terminator sitting in front of an application server that does use a thread-per-request model — nginx absorbs ten thousand slow clients so your backend only ever sees the small number of requests that are genuinely doing work.

It is also why worker_connections exists as a directive at all. You are not capping a process pool. You are pre-sizing an event loop, telling one worker how many descriptors it should be prepared to hold at once.

The model you need in your head

nginx runs one master process and N worker processes.

The master does almost nothing interesting. It reads the config, binds the listening sockets, spawns workers, and handles signals. It does not serve traffic. Notably, the master opens the listening socket and the workers inherit it — which is why a reload can swap workers without dropping the socket.

Each worker is single-threaded and event-driven. It is not one-thread-per-connection. A worker sits in epoll_wait() holding thousands of connections, wakes up when any of them has something to say, does a small amount of non-blocking work, and goes back to sleep. This is why nginx holds 10,000 idle connections on almost no memory, and why one blocking operation in a worker stalls every connection that worker owns.

That last point is the source of most surprising nginx behaviour. A slow disk read, a synchronous DNS lookup, a blocking third-party module — while that happens, thousands of unrelated requests sit still. aio threads exists to push file I/O onto a thread pool for exactly this reason.

Three consequences worth internalising:

• CPU work parallelises across workers, not within one. One worker = one core, maximum.
• A single expensive request cannot be preempted. Event loops are cooperative.
• Load balance between workers is the kernel's job, not nginx's. More on that under reuseport.

The arithmetic that sets your ceiling

Two directives decide the maximum number of connections nginx will hold:

worker_processes  auto;   # one worker per CPU core
events {
    worker_connections  1024;   # per worker, and this default is the trap
}

The ceiling is the product:

max connections = worker_processes × worker_connections

worker_connections counts every connection a worker owns, not just the client-facing ones. This is where the arithmetic usually goes wrong, because "two per client" gets repeated as though it were a law, and it is only the common case.

What one client request actually costs:

So a fairly ordinary setup — auth_request in front of a proxy_pass to a resolved hostname — can hold four connections for one user, not two. WebSockets and gRPC streams are still two, but they hold both ends for the entire life of the session rather than milliseconds, which hurts a ceiling far more than a higher count would.

The general form:

max clients ≈ (worker_processes × worker_connections) / connections_per_client

Use 2 as a floor for any proxy, and count your own middleware before trusting it. The number you want is whatever your busiest path actually opens.

The default worker_connections 1024 has shipped for years and is far too low for anything under real load. On a 16-core box that is 16,384 total, or roughly 8,000 proxied clients — and people are routinely surprised by that number because nobody ever changed a default they never read.

When you hit it, nginx tells you plainly in error.log:

[alert] 1234#0: 768 worker_connections are not enough

That message is a gift. Most limits fail with something far more cryptic.

File descriptors: the limit that bites first

Every connection is a file descriptor. Every open log file, every cached file handle, every upstream socket. So the fd limit has to be larger than your connection ceiling, and it is the constraint people forget because it lives outside nginx.

There are three places a limit can come from, and they override each other in this order:

1. The kernel's global cap — fs.file-max. Rarely the problem on modern systems.
2. The process limit — RLIMIT_NOFILE. This is the one that bites.
3. nginx's own directive — worker_rlimit_nofile, which calls setrlimit() for workers.

Where RLIMIT_NOFILE comes from depends on how nginx starts:

• systemd ignores /etc/security/limits.conf entirely. It uses LimitNOFILE= in the unit.
  This is the single most common cause of "I raised the limit and nothing changed" — the file you edited was never consulted.
• Docker takes it from the daemon's default ulimits or --ulimit nofile=.
• worker_rlimit_nofile in nginx.conf overrides both for workers, up to the hard limit.

Never trust the config. Ask the running process:

# what the worker actually has, not what you configured
cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

# how many it is using right now
ls /proc/$(pgrep -f 'nginx: worker' | head -1)/fd | wc -l

The failure mode is unmistakable once you know it:

[crit] accept4() failed (24: Too many open files)

Errno 24 is EMFILE. nginx is alive, healthy, and refusing connections. A rule of thumb that has never let me down: set worker_rlimit_nofile to at least twice worker_connections, and make sure the hard limit is above that.

What the kernel does before nginx sees anything

A connection passes through two kernel queues before a worker ever touches it. If you tune nginx without understanding these, you tune the wrong thing.

1. The SYN queue holds half-open connections between SYN and the final ACK. Sized by net.ipv4.tcp_max_syn_backlog. Overflow here shows up as SYN cookies or dropped handshakes.

2. The accept queue holds fully established connections waiting for the application to call accept(). Sized by min(listen backlog, net.core.somaxconn).

That min() is where people get caught. nginx's listen ... backlog=N is capped silently by somaxconn. Ask for 65535 with somaxconn at 4096 and you get 4096, with no warning.

You can see both queues directly:

# on a LISTEN socket: Recv-Q = connections waiting to be accepted
#                     Send-Q = the accept queue's maximum
ss -lnt

# has the accept queue ever overflowed? this counter is the smoking gun
nstat -az | grep -E 'ListenOverflows|ListenDrops'

A non-zero and climbing ListenOverflows means the kernel completed handshakes that nginx never accepted. The client saw a connection succeed and then hang. That is nginx being too slow to accept — workers busy, blocked, or too few — not a network problem, and no amount of somaxconn will fix the cause. It only buys you a bigger waiting room.

The other kernel knobs that matter under real load:

SO_REUSEPORT

By default the master creates one listening socket and every worker calls accept() on it. The kernel wakes workers to compete for each new connection, and the distribution is uneven — under high connection rates you get one hot worker and fifteen bored ones.

One word fixes it:

listen 443 ssl reuseport;

Now each worker gets its own listening socket and its own accept queue, and the kernel hashes incoming connections across them. No contention, far better balance across cores. On connection-heavy workloads this is one of the largest single-line wins available.

Two things to know before you paste it in:

• Put reuseport on one listen directive per address:port. Repeating it across server
  blocks for the same socket is a config error.
• Reloads become slightly lossy. Each worker owns a private accept queue; when workers are
  replaced, connections sitting in an old worker's queue can be dropped rather than handed over. For most sites this is invisible. If you reload constantly under heavy load, know the trade.

Related, and usually best left alone: accept_mutex has defaulted to off since nginx 1.11.3, and reuseport makes it irrelevant. multi_accept on tells a worker to drain all pending connections per wakeup — it can raise throughput and can also raise latency variance. Measure, don't assume.

What to expect, in orders of magnitude

Precise numbers from someone else's hardware are worthless, but orders of magnitude stop you chasing the wrong bottleneck. Modern x86, per core:

• Static file from page cache, keepalive on: tens of thousands of requests/sec per core. nginx
  is rarely the limit here; your NIC or the kernel's packet path is.
• New TCP connection per request (no keepalive): an order of magnitude worse. Connection setup
  and teardown dominate. This is the single biggest self-inflicted wound in benchmarks.
• TLS handshakes: the expensive part, and the algorithm matters enormously. ECDSA P-256 is
  several times cheaper than RSA-2048 per handshake. Session resumption skips the asymmetric work entirely.
• Bulk TLS transfer: cheap on any CPU with AES-NI. Handshakes cost, streaming does not.

The practical takeaways: if you are TLS-terminating, your handshake rate is your throughput, and enabling session resumption plus moving to an ECDSA certificate will beat every sysctl in this article. If your benchmark opens a fresh connection per request, you are measuring connection setup, not nginx.

Triage: the order that finds it fastest

Work outside in. Each step rules out a layer.

Step 0 — make nginx observable. If you have not done this, do it before you need it:

# a timing-aware log format: this single change ends most "is it nginx or the app?" arguments
log_format timing '$remote_addr $status $request_time '
                  'upstream=$upstream_response_time connect=$upstream_connect_time '
                  'header=$upstream_header_time "$request"';

# and the built-in counters
location = /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

$request_time is the whole request. $upstream_response_time is what your backend took. If those two are close, the problem is not nginx. That is usually the entire investigation.

Step 1 — how many connections do you actually have?

ss -s                                          # totals, incl. timewait
ss -tan state established '( sport = :443 )' | wc -l
curl -s localhost/nginx_status

stub_status returns something like Active connections: 291 and a line of three numbers: accepts handled requests.

If accepts and handled differ, nginx dropped connections — it hit worker_connections or ran out of file descriptors. Those two numbers should be identical forever. It is the fastest single check on the whole box, and almost nobody looks at it.

Step 2 — are you hitting a limit?

grep -c 'worker_connections are not enough' /var/log/nginx/error.log
grep -c 'Too many open files' /var/log/nginx/error.log
cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

Step 3 — is the kernel dropping before nginx?

nstat -az | grep -E 'ListenOverflows|ListenDrops'
ss -lnt                      # Recv-Q climbing on a LISTEN row = you are not accepting fast enough

Step 4 — are you CPU bound, and is it one worker or all of them?

top -H -p $(pgrep -d, -f 'nginx: worker')

All workers pegged → genuinely CPU bound, likely TLS. One worker pegged while others idle → uneven distribution: reach for reuseport. That asymmetry is the tell.

Step 5 — the upstream side. Covered next, because it deserves its own section.

The upstream side: where reverse proxies die

Every outbound connection to an upstream consumes an ephemeral port from net.ipv4.ip_local_port_range. The usual range is 32768–60999, which is about 28,000 ports.

A port is not reusable immediately after close — it sits in TIME_WAIT for 60 seconds. So a proxy opening a fresh upstream connection per request tops out around 28,000 / 60 ≈ 470 connections per second to a single upstream (ip, port) pair, no matter how much CPU you have.

Then you get this, and it looks like a mystery:

connect() to 10.0.0.5:8080 failed (99: Cannot assign requested address)

That is not a network error. That is ephemeral port exhaustion.

The fix is not a sysctl. It is keepalive to your upstream, and it needs three lines that must all be present — the two proxy_* lines are the ones everybody forgets, and without them the keepalive directive silently does nothing:

upstream backend {
    server 10.0.0.5:8080;
    keepalive 64;               # persistent connections held per worker
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;          # required — 1.0 cannot keep alive
        proxy_set_header Connection "";  # required — strips the inbound "close"
    }
}

Widening ip_local_port_range and setting tcp_tw_reuse treat the symptom. Connection reuse removes the problem.

One more ceiling if you are behind Docker or any NAT: every connection consumes a conntrack entry. When nf_conntrack_count approaches nf_conntrack_max, the kernel starts dropping packets and logs nf_conntrack: table full, dropping packet. It presents as random, unexplained connection failures under load.

sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count

What not to tune

• tcp_tw_recycle — do not look for it. It broke clients behind NAT and was removed from
  Linux in 4.12. Any guide still recommending it was written for a kernel you are not running.
• Raising somaxconn to "fix" overflows — it enlarges the waiting room. If ListenOverflows
  climbs, workers are not accepting fast enough; find out why.
• worker_processes above your core count — workers then compete for the same cores and you
  add context switching for nothing. auto is right almost always.
• Copy-pasted sysctl blocks from blog posts, applied wholesale. Every knob in this article
  fixes a specific, observable symptom. If you cannot name the counter that made you change it, do not change it.

Worked example: auditing a live reverse proxy

Theory is cheap. Here is my own nginx-proxy container, which has been running untouched for five months, on a 16-core host.

# nginx version: nginx/1.29.2
worker_processes  auto;
events { worker_connections  1024; }

auto on 16 cores gives 16 workers, which I confirmed rather than assumed:

$ docker top nginx-proxy -o pid,args
nginx: master process nginx -g daemon off;
nginx: worker process        ← ×16

Finding 1 — the connection ceiling is a stock default nobody chose. 16 × 1024 = 16,384 connections; as a reverse proxy that is roughly 8,000 concurrent clients. Not a crisis for a homelab, but it is the number I would hit first under load, and it is there by accident rather than decision.

Finding 2 — file descriptors are not the constraint, by luck.

$ grep 'Max open files' /proc/<worker>/limits
Max open files    1048576    1048576    files
$ ls /proc/<worker>/fd | wc -l
25

A million fds, 25 in use. That headroom comes from the container image's defaults, not from anything in my config — worker_rlimit_nofile is not set. On a systemd host with a stock LimitNOFILE, the same config would fail long before worker_connections was reached. The constraint here is worker_connections, which is the healthy way round — but only by accident.

Finding 3 — nginx is unobservable. No stub_status, and the log format is stock main with no timings. I cannot answer "did nginx drop connections?" or "was it nginx or the upstream?" without changing the config first — which is exactly the wrong time to be editing config.

The kernel side, for contrast, is healthy:

net.core.somaxconn              4096
net.ipv4.tcp_max_syn_backlog    2048
net.ipv4.ip_local_port_range    32768  60999      # 28,232 ports
net.ipv4.tcp_tw_reuse           2                 # loopback only — the modern default
nf_conntrack_max                262144            # in use: 385

TcpExtListenOverflows           0
TcpExtListenDrops               0

Zero overflows since boot: nothing has ever waited in the accept queue. tcp_max_syn_backlog (2048) sitting below somaxconn (4096) is worth evening out, and netdev_max_backlog is at the stock 1000, which only matters at packet rates I will never see here.

And the trap I hadn't thought about until I looked. I checked the listening sockets on the host:

$ ss -lntp '( sport = :80 or sport = :443 )'
LISTEN 0 4096   0.0.0.0:80    users:(("docker-proxy",...))
LISTEN 0 4096   100.96.60.8:443  users:(("tailscaled",...))

Neither of those is nginx. Port 80 is held by docker-proxy and 443 by tailscaled. nginx's own listening socket lives inside the container's network namespace:

$ docker exec nginx-proxy netstat -lntp
tcp  0  0 0.0.0.0:80   LISTEN   1/nginx: master pro

So every accept-queue number I could have read on the host describes docker-proxy's queue, not nginx's. If you run nginx in a container and debug queues from the host, you are measuring the wrong socket. Use nsenter into the container's netns, or run the tools inside it.

That one is worth the price of admission. It is invisible until it wastes an afternoon.

The one-screen checklist

# 1. what is the ceiling?
nginx -T | grep -E 'worker_processes|worker_connections'

# 2. did nginx drop anything? (accepts must equal handled, forever)
curl -s localhost/nginx_status

# 3. what limit does the worker really have?
cat /proc/$(pgrep -f 'nginx: worker'|head -1)/limits | grep 'open files'

# 4. did the kernel drop before nginx?
nstat -az | grep -E 'ListenOverflows|ListenDrops'
ss -lnt

# 5. one hot worker, or all of them?
top -H -p $(pgrep -d, -f 'nginx: worker')

# 6. running out of ephemeral ports?
ss -s
grep 'Cannot assign requested address' /var/log/nginx/error.log

# 7. nginx or the backend?  ($request_time vs $upstream_response_time)
tail -f /var/log/nginx/access.log

If you take one thing: accepts must equal handled in stub_status. The moment they diverge, nginx is refusing work, and you have exactly two suspects — worker_connections and file descriptors.