Linux Networking Part 2: Reading Sockets with ss

You have nginx running. Someone says it's broken. You need to know, in the next thirty seconds: who is connected, from where, to which port, and whether anything is stuck.

One command answers all of that. Most people run it wrong.

This is part 2 of the series. Part 1 built the map — the seven-stop path from wire to application. ss reads stops 6 and 7: the socket layer and how well your application is keeping up with it.

Why ss and not netstat

ss ships with iproute2 — the same package as ip. It reads socket state from the kernel over a netlink socket.

netstat comes from net-tools, which has been effectively unmaintained for years and isn't installed by default on most modern distros. It works by parsing /proc/net/tcp as text.

On an idle box the difference is invisible. On a box with 200,000 sockets, netstat -an takes ten seconds and burns CPU you can't spare during an incident. ss returns immediately. When you're the one holding the pager, that gap is the whole argument.

Use ss. Recognise netstat in old docs. Don't write new runbooks with it.

The flags that matter

Six letters cover 90% of real use:

-t   TCP sockets
-u   UDP sockets
-a   all sockets (listening AND established)
-l   listening sockets only
-n   numeric — do not resolve ports or hostnames
-p   show the process holding the socket (needs root for other users' sockets)

Always pass -n. Without it, ss does a DNS lookup for every peer address. On a busy host that turns an instant command into a thirty-second hang — and if DNS is the thing that's broken, your diagnostic tool hangs on the outage you're diagnosing. Skipping -n is the single most common way people make a bad day worse.

The two commands you'll type most:

# What is this box listening on?
ss -lntp

# Who is connected right now?
ss -antp

Reading the output

State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port   Process
LISTEN  0       511      0.0.0.0:80            0.0.0.0:*           users:(("nginx",pid=1234,fd=6))
ESTAB   0       0        10.0.0.5:80           203.0.113.7:51234   users:(("nginx",pid=1235,fd=12))
ESTAB   4096    0        10.0.0.5:80           203.0.113.9:51402   users:(("nginx",pid=1235,fd=13))

Six columns. Each one earns its place.

State — where this socket sits in the TCP state machine. Full list below.

Local Address:Port — your side. 0.0.0.0:80 means "port 80 on every IPv4 address on this machine." 127.0.0.1:80 means loopback only — nothing outside the box can reach it, ever. That one line has explained more "but the service is running!" tickets than anything else in this post. [::]:80 is the IPv6 equivalent of 0.0.0.0.

Peer Address:Port — the other side's IP and port. This is your answer to "who is connecting to me and on which port." No extra tooling needed.

Recv-Q and Send-Q — these mean two different things depending on the state, and this is the part everyone gets wrong.

Recv-Q / Send-Q on an ESTABLISHED socket

• Recv-Q — bytes sitting in the kernel receive buffer that your application has not read() yet.
• Send-Q — bytes your application has written that the kernel has not yet got acknowledged by the peer.

So in that third line above, Recv-Q 4096 says: 4KB arrived from 203.0.113.9, and nginx hasn't picked it up.

The diagnostic split is clean:

A brief spike in either is normal — that's what buffers are for. Sustained and growing is the signal. Watch it over time rather than judging a single snapshot:

watch -n1 'ss -ant state established | head -30'

Recv-Q / Send-Q on a LISTENING socket

Completely different meaning. Here's where the real gold is.

• Recv-Q — the number of established connections currently sitting in the accept queue, waiting for your application to call accept().
• Send-Q — the maximum size of that accept queue.

So:

LISTEN  0    511    0.0.0.0:80

means: queue capacity 511, currently zero waiting. Healthy.

LISTEN  511  511    0.0.0.0:80

means: the accept queue is completely full. New connections are being dropped right now. Your NIC counters are clean, the network is fine, and clients are hanging or getting resets.

That one line is worth the whole post. Check it before you touch tcpdump.

The two queues nobody separates

When a client connects to your server, the connection passes through two kernel queues. Confusing them makes overload impossible to diagnose.

client SYN
    |
    v
[ SYN queue ]  ---- half-open. Handshake in progress. State: SYN-RECV.
    |             sized via net.ipv4.tcp_max_syn_backlog
    |  (handshake completes)
    v
[ accept queue ] -- fully established, waiting for the app.
    |               sized via min(listen() backlog, net.core.somaxconn)
    |  (app calls accept())
    v
your application

SYN queue full — usually a SYN flood, or a genuine connection storm. If net.ipv4.tcp_syncookies is enabled (it is by default on most distros), the kernel falls back to SYN cookies instead of dropping, which is exactly what you want.

Accept queue full — your application is too slow to accept(). This is far more common and far more misdiagnosed. The kernel completed the handshake, the connection is established and ready, and your app never picked it up.

What happens when the accept queue overflows depends on a sysctl:

• net.ipv4.tcp_abort_on_overflow = 0 (the default) — the kernel silently ignores the client's final ACK. The client thinks it's connected, waits, retransmits. The user experiences a hang. Nothing in your logs mentions it.
• net.ipv4.tcp_abort_on_overflow = 1 — the kernel sends a RST. The client fails fast with "connection reset by peer."

Default 0 is why this is so hard to spot. The failure mode is designed to be invisible and hope the app catches up.

Two places to see it. First, live:

ss -lnt          # Recv-Q at or near Send-Q on a listener

Second, cumulative:

nstat -az | grep -i listen

TcpExtListenOverflows counts accept-queue overflows. TcpExtListenDrops counts all drops on listening sockets. If ListenOverflows is climbing, stop looking at the network. Full detail on nstat in part 3.

For nginx specifically the queue size comes from the backlog parameter on the listen directive, capped by net.core.somaxconn. Raising one without the other does nothing — the kernel takes the minimum. Check the current values on your box with sysctl net.core.somaxconn and confirm the nginx side against the nginx listen documentation rather than trusting any blog's numbers, including this one.

And a caveat worth saying plainly: a full accept queue is a symptom. Raising the backlog buys you a deeper queue to fall behind in. Find out why the app stopped accepting.

The TCP states, and what each one means when it piles up

You'll see these in the State column. Individually they're routine. In bulk, each one is a specific diagnosis.

Two of these deserve their own section.

CLOSE_WAIT is always your fault

CLOSE_WAIT means the remote end sent a FIN and the kernel acknowledged it — and your application has not called close() on that file descriptor.

The kernel cannot fix this. It has done its job and is waiting on you. CLOSE_WAIT sockets do not time out. They sit there until the process closes the descriptor or dies.

Thousands of CLOSE_WAIT sockets on one process means that process is leaking file descriptors, and it will eventually hit its ulimit -n and start failing to accept anything at all — which looks, from outside, exactly like a network outage.

ss -antp state close-wait

If that returns a large number pointing at one PID, you have found a bug in that application. Not in the network. Restarting clears it; only a code fix stops it coming back.

TIME_WAIT is usually not your fault

TIME_WAIT appears on whichever side closed the connection first. The socket waits out a fixed interval — 60 seconds on Linux, compiled in, not tunable by sysctl — so that delayed packets from the old connection can't be misdelivered into a new one reusing the same 4-tuple.

Servers usually let clients close, so servers usually don't accumulate them. A load balancer or proxy that closes its own upstream connections does, and 20,000 TIME_WAIT entries there can be completely healthy.

ss -tan state time-wait | wc -l

It only becomes a real problem when you're making a very high rate of outbound connections to the same destination and exhausting source ports. That's part 5.

One hard rule: do not enable net.ipv4.tcp_tw_recycle. It broke connections from clients behind NAT, and it was removed from the kernel in 4.12 — so on any modern system the sysctl doesn't even exist. Any tuning guide that recommends it was written before 2017 and should be closed. net.ipv4.tcp_tw_reuse is a different and much safer setting; read tcp(7) before touching it.

Filtering — get to the answer directly

Piping ss into grep works, and it's how most people do it. ss has a real filter language that's faster and doesn't produce false matches on a port number that happens to appear in an IP address.

# Everything talking to one host
ss -antp dst 203.0.113.7

# Everyone connected to my port 443
ss -antp '( dport = :443 or sport = :443 )'

# Only established connections
ss -antp state established

# Established, to one specific client
ss -antp state established dst 203.0.113.7

# How many connections per client IP — the "who is hammering me" one-liner
ss -ant state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

That last one is the command to run when traffic spikes and you want to know whether it's one bad actor or genuine load. It answers the question in a second.

And for a fast overview:

ss -s

Totals by protocol and state. Good for "is this box holding 400 connections or 400,000?" before you decide where to dig.

Looking inside a connection

This is where ss stops being a listing tool and becomes a diagnostic one:

ss -tin

-i dumps the kernel's internal TCP state per connection. The output is dense. The fields worth knowing:

• rtt:12.5/6.25 — smoothed round-trip time and its variance, in milliseconds. This is measured RTT on the real connection, not a synthetic ping. It reflects the actual path this traffic takes, including any middleboxes.
• minrtt — the lowest RTT ever seen on this connection. Compare with rtt: if current RTT is far above minrtt, you're queueing somewhere.
• retrans:0/12 — retransmits currently outstanding / total on this connection. A non-zero total is direct evidence of loss on the path.
• cwnd — congestion window, in segments. How much the kernel is willing to have in flight. A small cwnd on a fast link means the stack has recently detected loss and backed off.
• bytes_sent / bytes_acked / bytes_received — lifetime totals for this connection.
• send 1.2Mbps / delivery_rate — the stack's own estimate of throughput.
• lastsnd / lastrcv / lastack — milliseconds since the last send, receive, and ack. Large values on a supposedly active connection mean it's idle or stuck.
• mss / pmtu — segment size and path MTU. Relevant when you suspect MTU problems, which get their own treatment in part 4.

Field names vary a little between kernel and iproute2 versions, and new ones get added. man ss on the box you're standing on is the authority.

The practical use: pick a slow connection and inspect it.

ss -tin state established dst 203.0.113.7

If rtt is 200ms and retrans is climbing, the path is bad and you can say so with evidence. If rtt is 0.4ms and retrans is 0 and the user still says it's slow, the network is exonerated — go look at the application. Being able to make that call confidently, in one command, is most of what this series is for.

What did we just build?

A complete picture of stops 6 and 7 on the map:

• Who's connected — the Peer Address column.
• Is my app keeping up — Recv-Q on established sockets, Recv-Q vs Send-Q on listeners.
• Is my app leaking — CLOSE_WAIT counts.
• Is the path lossy — retrans and rtt from ss -ti.

Notice what's missing. ss tells you nothing about packets dropped at the NIC, in softirq processing, or by conntrack — all of which happen below the socket layer and never reach a socket to be counted. If ss looks clean and things are still broken, that's your signal to go down the ladder.

Which is part 3: proving packet loss with counters.

Primary sources: man ss, man 7 tcp.

Compiled by AI. Proofread by caffeine. ☕