Linux Networking Part 5: Latency, Queues, and a Real nginx Hunt

"The site is slow."

Three words, and almost no information. Slow where? Slow for whom? Slow by how much, compared to what?

Latency work is mostly attribution — cutting the total time into pieces until one piece is obviously the problem. Do that first and the fix is usually easy. Skip it and you'll spend a day tuning sysctls that were never involved.

Final part of the series. We'll build the attribution method, cover the queues where latency actually accumulates, and then work a full nginx investigation using everything from parts 1 through 4.

Latency is queueing

Almost every unexplained delay in a networked system is something waiting in a queue.

Actual transmission time is physics and it's small — light through fibre crosses a continent in tens of milliseconds and that number never changes. When something takes 800ms that should take 20ms, that extra 780ms was spent waiting: in a NIC ring buffer, a qdisc, a socket buffer, an accept queue, a thread pool, a connection pool, a disk queue.

So the question is never "why is the network slow." It's "which queue is it sitting in?" Every queue we've met in this series is a candidate:

Ten candidates. The job is elimination, and the fastest elimination tool is a timing breakdown.

Split the total time first

Before touching the server, measure from the client. curl will decompose a request for you, and this is the most useful thirty seconds in the whole investigation.

curl -w '
  dns:          %{time_namelookup}s
  tcp connect:  %{time_connect}s
  tls:          %{time_appconnect}s
  pre-transfer: %{time_pretransfer}s
  first byte:   %{time_starttransfer}s
  total:        %{time_total}s
' -o /dev/null -s https://example.com/

These are cumulative from the start of the request, so you subtract to get each phase:

• time_namelookup — DNS. If this is 2 seconds, your problem is DNS and nothing in this post applies.
• time_connect − time_namelookup — the TCP handshake. This is one round trip. It's your cleanest measurement of raw network RTT to that server, and it involves zero application code.
• time_appconnect − time_connect — the TLS handshake. Typically one or two additional round trips.
• time_starttransfer − time_pretransfer — time to first byte: the server thinking. This is where application latency lives.
• time_total − time_starttransfer — transferring the body. Large here with a small body means throughput problems.

The interpretation is mechanical:

• Connect time is high, TTFB is low → the network path is slow. The server is fine. Go look at RTT, routing, and loss.
• Connect time is low, TTFB is high → the network is fine. The server is slow. Stop reading tcpdump.
• Connect time is erratic — fast usually, occasionally terrible → queueing. Accept queue, or loss causing retransmitted SYNs.
• Everything is fine but users still complain → it's not this path. Different region, different client, or a specific endpoint.

Run it in a loop to see distribution rather than one sample:

for i in $(seq 20); do
  curl -w '%{time_connect} %{time_starttransfer} %{time_total}\n' \
    -o /dev/null -s https://example.com/
done

Averages hide everything. What you're looking for is the shape: are all 20 slow, or are 18 fast and 2 catastrophic? Those are different problems. Consistent slowness is a capacity or path issue. Occasional catastrophic outliers are queueing or loss — a retransmission timeout, which is a delay of seconds, not milliseconds.

Full list of curl timing variables is under -w in man curl.

Path latency vs application latency

If curl points at the network, narrow it further.

ping -c 20 203.0.113.7

Look at min/avg/max/mdev on the summary line. A tight spread means a stable path. A wide one means variable queueing somewhere along it.

mtr -rwc 100 203.0.113.7

mtr combines traceroute and ping — 100 probes to every hop, reporting loss and latency per hop. It's the right tool for "where along the path does it go bad."

One critical caveat that trips up almost everyone: loss reported at an intermediate hop is usually meaningless. Routers deprioritise generating ICMP replies for their own address; they'll happily show 40% "loss" while forwarding your actual traffic flawlessly. What matters is loss at the final hop, and loss that persists from a hop onward through all subsequent hops. Loss at hop 4 that disappears at hops 5 through 10 is an artifact. Loss that starts at hop 4 and continues to the end is real.

I've watched engineers escalate to a transit provider over hop-4 ICMP rate limiting more than once. Know this rule and you'll never be that person.

And remember the better measurement from part 2 — real RTT on the real connection carrying real traffic:

ss -tin state established dst 203.0.113.7

rtt: versus minrtt: is the tell. If minrtt is 2ms and current rtt is 180ms, the path can do 2ms and something is queueing right now. ping gives you a synthetic probe that may take a different path or priority; ss -ti gives you the truth about the connection you actually care about.

Throughput: window over RTT

One formula explains most "why is my transfer slow over a long distance" questions:

max throughput ≈ window size / round-trip time

A 64KB window over a 100ms RTT caps you at roughly 5 Mbit/s — regardless of whether the link is 10Gbit. The sender fills the window, then waits a full round trip for acknowledgement before it can continue.

That product — bandwidth × RTT — is the bandwidth-delay product, and it's the amount of data that must be in flight to keep a link busy.

Linux autotunes receive buffers by default (net.ipv4.tcp_moderate_rcvbuf), and the tunables are net.ipv4.tcp_rmem and net.ipv4.tcp_wmem — each three values: minimum, default, maximum. net.core.rmem_max and net.core.wmem_max cap what an application can request explicitly.

Do not go tuning these because a blog told you to. The defaults are sane for typical deployments, autotuning works, and badly-chosen values make things worse while consuming memory. This matters when you have genuinely high bandwidth-delay product paths — cross-continent bulk transfer, replication between distant regions. It does not matter for a web server serving clients 20ms away, and turning these knobs there is cargo cult. Read man 7 tcp before changing anything, and measure before and after.

If your traffic crosses long distances, the congestion control algorithm is worth a look:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control

CUBIC is the long-standing default. BBR takes a different approach — modelling bandwidth and RTT rather than treating loss as the congestion signal — and often performs considerably better on lossy long-distance paths. It's not universally better, and it's a change worth testing on your own traffic rather than adopting on faith.

Two classic latency traps

The 40ms mystery

If you have request/response traffic that's consistently slow by a suspiciously round ~40ms, you may be hitting the interaction between Nagle's algorithm and delayed ACK.

Nagle's algorithm buffers small writes, waiting to coalesce them into a full segment or until outstanding data is acknowledged. Delayed ACK, on the other side, holds back acknowledgements briefly hoping to piggyback them on outgoing data.

Put them together and you can get a standoff: the sender waits for an ACK before sending its small segment, and the receiver waits for data before sending its ACK. The delayed-ACK timer eventually fires and breaks the deadlock — after a fixed delay.

The signature is that suspicious consistency. Real network problems are noisy; this is metronomic. The application-level fix is TCP_NODELAY (disabling Nagle), which most modern servers and libraries already set. If you're seeing this pattern, check whether your application sets it.

Ephemeral port exhaustion

The thing that looks like a network outage on a perfectly healthy box.

Every outbound connection needs a local source port from net.ipv4.ip_local_port_range — by default roughly 28,000 ports. And per part 2, closed connections hold their port in TIME_WAIT for 60 seconds.

So a service making outbound connections at a high rate, to a single destination IP and port — a database, a cache, an internal API — can run out. New connections fail. CPU is low, memory is fine, the network is healthy, and nothing works.

ss -tan state time-wait | wc -l
sysctl net.ipv4.ip_local_port_range

Thousands of TIME_WAIT entries all pointing at one destination is the fingerprint.

The tempting fixes are widening the port range and enabling net.ipv4.tcp_tw_reuse. Both help, both are treating symptoms, and one of them has caveats you should read in tcp(7) before deploying.

The real fix is connection reuse. If your service opens a fresh connection for every request to the same backend, that's the bug. Connection pooling and HTTP keepalive eliminate the problem entirely instead of raising the ceiling on it. And as established in part 2: never tcp_tw_recycle. It's gone from the kernel and it broke NAT'd clients when it existed.

Disk latency wearing a network costume

Worth saying plainly, because it's a common misdiagnosis: application latency backs up into the network stack.

Your app is blocked on a slow disk read. It isn't calling read() on its sockets. Recv-Q climbs. It isn't calling accept(). The accept queue fills. Clients hang and time out.

Every symptom points at the network. The cause is a disk.

iostat -x 1

The columns that matter: r_await and w_await — average time in milliseconds for read and write requests, including queue time — and aqu-sz, the average queue depth. Column names vary between sysstat versions; check the header your box prints.

A word on %util: on spinning disks it meant something. On SSDs and NVMe, which service many requests in parallel, it does not mean saturation. A device at 100% %util with 0.2ms await is completely fine. Use await, not %util. People still make capacity decisions on that number and they're making them wrong.

For deeper work, biolatency from the BCC tools gives you a latency histogram rather than an average — and averages hide the tail that your users are actually experiencing.

Check CPU too, and specifically softirq time:

mpstat -P ALL 1

Per-CPU breakdown. The %soft column is software interrupt time — network processing. One core at 100% %soft while the rest idle is the imbalance from part 3, and it causes both drops and latency.

Case study: nginx is slow and sometimes returns 502

Now the whole series at once. Symptom: users report intermittent slowness, occasional 502 Bad Gateway. Load is up but not extreme. Nobody knows anything else.

Step 1 — Where is the time going?

for i in $(seq 20); do
  curl -w '%{time_connect} %{time_starttransfer} %{time_total}\n' \
    -o /dev/null -s https://oursite.example/
done

Say the result is: 16 requests with connect ~0.02s and TTFB ~0.05s, and 4 requests with connect ~1.02s.

Read that carefully. It isn't uniform slowness. Connect time is occasionally about a second — and 1 second is exactly the first SYN retransmission interval from part 4.

Hypothesis: some connections aren't getting through on the first SYN. Not application slowness. Something at connection setup.

Step 2 — Is the listener keeping up?

ss -lnt

State   Recv-Q  Send-Q  Local Address:Port
LISTEN  487     511     0.0.0.0:443

487 of 511. The accept queue is nearly full. Confirm it's overflowing:

nstat -az | grep -i listen

TcpExtListenOverflows is climbing.

Confirmed. Handshakes complete, connections sit in the accept queue, nginx isn't picking them up fast enough, and the queue overflows. Dropped final ACKs mean SYN-ACK retransmissions and that ~1 second penalty. Exactly the part-4 pattern.

Note what we have not done: no tcpdump, no sysctl tuning. Two commands.

Step 3 — Rule out the layers below

Be rigorous. Prove there's nothing else:

ip -s link show eth0             # errors, overruns — clean
cat /proc/net/softnet_stat       # column 2 zero, column 1 even across CPUs
nstat -az | grep -i retrans      # not moving
conntrack -S                     # no drops (or n/a on this box)

All clean. The wire is fine, softirq is balanced, the path isn't lossy, conntrack isn't full. The network is exonerated, and now you can say so with evidence rather than opinion.

Step 4 — Why isn't nginx accepting?

The accept queue is a symptom. Raising somaxconn and the backlog would give it a deeper queue to fall behind in. Find the actual cause.

Check the error log first, always:

tail -100 /var/log/nginx/error.log

Three messages nginx produces that answer this question immediately:

• worker_connections are not enough — you've hit the worker_connections limit. Every connection nginx handles, client-facing and upstream, consumes one.
• accept4() failed (24: Too many open files) — file descriptor limit. Check worker_rlimit_nofile in nginx and the process limit (LimitNOFILE under systemd; cat /proc/<pid>/limits shows what the running process actually has, which is the number that counts).
• upstream timed out — nginx is waiting on a backend. Workers tied up waiting can't accept new connections.

That third one is the common answer, and it's a nice reminder that "nginx is slow" is very often "the thing behind nginx is slow."

Step 5 — Split nginx time from upstream time

nginx exposes exactly the variables you need in log_format:

• $request_time — total time nginx spent on the request.
• $upstream_connect_time — time to establish the upstream connection.
• $upstream_header_time — time until upstream response headers arrived.
• $upstream_response_time — total upstream time.

If they aren't in your access log format, add them. Then:

• $upstream_response_time ≈ $request_time → nginx is a passenger. The backend is slow. Go there.
• $request_time ≫ $upstream_response_time → nginx or the client link is the delay. Slow clients, TLS overhead, or nginx resource limits.
• $upstream_connect_time high → connecting to the backend is slow. Backend accept queue full, or backend port exhaustion. Run this same investigation one layer down.

That recursion is the point. The method doesn't change; you just move a layer.

Exact variable semantics are in the nginx log module docs and the upstream module docs. Read them there, not here.

Step 6 — Check upstream connection reuse

If $upstream_connect_time is high and the backend looks healthy, check whether nginx is opening a new connection per request to it. By default in a proxy_pass setup, it does.

The keepalive directive in an upstream block enables a pool of persistent connections. Making it actually work also requires proxy_http_version 1.1 and clearing the Connection header — the exact configuration is documented in the upstream module docs linked above, and it's worth reading rather than pasting.

Without it, a high-traffic proxy burns through source ports and produces exactly the TIME_WAIT pile-up described earlier. Check:

ss -tan state time-wait | awk '{print $5}' | sort | uniq -c | sort -rn | head

If one backend dominates that list, connection reuse is your fix.

Step 7 — The fix, in order

1. Fix the actual cause — the slow upstream, the FD limit, the exhausted worker connections. This is the fix.
2. Enable upstream keepalive if nginx is proxying at volume.
3. Then, and only then, consider raising the accept queue — net.core.somaxconn and the backlog parameter on listen, together, since the kernel takes the minimum. A deeper queue absorbs bursts. It does not create capacity.

Order matters. Do 3 first and you've hidden the symptom and moved the failure to a worse place — instead of failing fast, connections now wait a long time and then fail.

What that investigation actually cost

Six commands and a log file. No packet capture, no kernel tuning, no guessing.

The path was: measure from outside → localise to a layer → prove the layers below are clean → find the cause → fix the cause, not the symptom.

That's the method. The commands are interchangeable. The method isn't.

The umbrella

Since this is the last part, here's the whole thing compressed.

Everything is a queue. Networking is producers, consumers, and buffers between them. Every problem is either "a queue is full" (drops) or "a queue is deep" (latency). Find the queue.

Every tool reads one layer. ethtool and ip -s link read the wire. /proc/net/softnet_stat reads packet processing. nstat reads the IP and TCP stack. ss reads sockets and the application boundary. tcpdump reads what's actually on the wire. Knowing which layer a tool reads is knowing when to run it.

Drops and retransmits are different evidence. Drop counters mean this box discarded something. Retransmits mean the path lost something. They point in opposite directions.

Localise before you tune. Every sysctl in every tuning guide is a guess until you know which layer is failing. Measure, localise, then change one thing and measure again.

The application is part of the network stack. Recv-Q, accept queues, CLOSE_WAIT, port exhaustion — every one of these is caused by application behaviour and diagnosed with network tools. "The network is slow" is, more often than not, "our code stopped reading its sockets."

Where to go next

Two directions, both worth it.

Down, into eBPF. bpftrace and the BCC tools let you ask questions counters can't answer: which connection retransmitted, how long each socket lived, exactly where in the kernel a packet was dropped. tcpretrans, tcplife, tcpconnect, biolatency. This is the current frontier of Linux observability and it's the clearest line between competent and genuinely strong. Start at the BCC repo and bpftrace.

Sideways, into the protocol. Read TCP/IP Illustrated, Volume 1 (Stevens; the second edition is updated by Fall). Nothing else builds the same depth. Then read man 7 tcp end to end — it is dense, unglamorous, and better than most books.

And the unfashionable advice: capture traffic for things that are working. Take a tcpdump of a healthy request. Watch a normal handshake. Read ss -ti on a connection that's fine. You cannot recognise broken until you know exactly what healthy looks like, and 3am is a terrible time to be learning it.

Next post is the cheat sheet — every command in this series in one page, plus a learning path.

Compiled by AI. Proofread by caffeine. ☕