Linux Networking Part 7: The Cheat Sheet

The whole series on one page. Print it, pin it, put it in your runbook.

Parts 0 through 6 explain why each of these works. This is the lookup table for when you don't have time to read.

Glossary

Start here: three questions

Decide which one you're asking before you type anything.

1. Can they reach me? → connectivity. nc -zv, ss -lnt, tcpdump, firewall rules, ip route get
2. Are we losing traffic? → loss. nstat, ip -s link, /proc/net/softnet_stat, ethtool -S
3. Why is it slow? → latency. curl -w, ss -ti, mtr, iostat -x, mpstat

Mixing these up costs hours. Answer this first, every time.

The ladder

Every tool reads one layer. Know which.

 [7] application ......... ss Recv-Q, app logs, app metrics
 [6] socket buffers ...... ss -tin
 [5] TCP/IP stack ........ nstat
 [4] routing ............. ip route get, ip rule, rp_filter
 [3] netfilter/conntrack . iptables -L -n -v, nft list ruleset, conntrack -S
 [2] driver/softirq ...... /proc/net/softnet_stat, /proc/interrupts
 [1] NIC hardware ........ ip -s link, ethtool -S, ethtool -g
 [0] layer 2 ............. ip neigh, bridge fdb show
     the wire

The flowchart

                  "it doesn't work"
                          |
              +-----------+-----------+
              |                       |
        fails INSTANTLY          HANGS then times out
     ("connection refused")     ("connection timed out")
              |                       |
        packet ARRIVED           packet may not have arrived
              |                       |
     +--------+--------+         tcpdump on the server
     |                 |              |
  nothing         firewall     +------+------+
  listening       REJECT       |             |
     |                    SYN visible?   nothing at all?
  ss -lntp                    |             |
  check bind addr        LOCAL problem   UPSTREAM problem
  (127.0.0.1?)           firewall -Z     routing / secgroup
                         rp_filter       ip route get
                         conntrack       ip neigh
                         accept queue


              "it works but it's slow"
                          |
                    curl -w breakdown
                          |
      +-------------------+-------------------+
      |                   |                   |
  high DNS          high CONNECT          high TTFB
      |                   |                   |
  dig / getent      network path OR      the APPLICATION
  resolvectl        accept queue         (and what's behind it)
                         |                     |
                   ss -lnt Recv-Q       app logs, upstream time
                   ss -ti rtt/retrans   iostat -x, mpstat
                   mtr (final hop only)

The 60-second triage

On an unfamiliar box, in this order:

nstat                              # run twice, 30s apart — what is moving?
ip -s link show eth0               # is the wire healthy?
ss -lnt                            # are listeners keeping up? (Recv-Q vs Send-Q)
cat /proc/net/softnet_stat         # is one CPU drowning? col 2 = drops
conntrack -S                       # if this box does NAT or containers

Five commands. Most of the search space eliminated.

First contact — is it even reachable?

Bottom-up. Each step assumes the previous one passed.

ip -br addr                       # interface up? right address? right prefix?
ip -br link
ip route get 10.0.0.50            # route, interface, AND source IP
ping -c 5 10.0.0.50               # see caveat below
getent hosts db.internal          # resolves the way the APP sees it (NSS)
dig +short db.internal            # what DNS itself says
nc -zv -w 3 10.0.0.50 5432        # is the port actually open?
curl -vI https://example.com/     # TLS + HTTP

Ping failing means almost nothing — ICMP is blocked by policy everywhere. Ping succeeding is useful; ping failing is not evidence the host is down. Test the port.

nc failure modes are the diagnosis:

• Refused instantly → packet arrived, nothing listening. Check bind address.
• Hangs then times out → something is silently dropping. Firewall.

DNS — the split that solves the hard cases:

dig +short name       # queries DNS directly, BYPASSES /etc/hosts and NSS
getent hosts name     # goes through NSS — same path as your application
dig @8.8.8.8 name     # is it my resolver, or the record itself?
dig +trace name       # follow delegation from the root
resolvectl status     # systemd-resolved config
grep ^hosts /etc/nsswitch.conf
cat /etc/resolv.conf

dig works but getent fails → not DNS. It's /etc/hosts, nsswitch.conf, or the resolver library.

Who owns the port / what is the process doing:

ss -lntp
lsof -i :443
lsof -i -P -n
fuser 443/tcp
strace -f -e trace=network -p <pid>     # is it even TRYING to connect?

TLS:

openssl s_client -connect example.com:443 -servername example.com
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Always pass -servername or you'll get the wrong certificate and debug a problem you created.

Bandwidth (not latency):

iperf3 -s                    # server
iperf3 -c <ip>               # client
iperf3 -c <ip> -R            # reverse direction — always test both
iperf3 -c <ip> -P 4          # parallel streams

One stream slow, four streams fast = window/RTT ceiling, not a slow link.

traceroute caveats: * * * means the router declined to reply, not a broken hop. Intermediate-hop loss is usually ICMP rate limiting. Only the final hop, or loss persisting through every subsequent hop, is real. traceroute -T -p 443 gets further through firewalls than the UDP default.

Symptom → command

ss — sockets

ss -lntp                    # what am I listening on, and which process
ss -antp                    # every TCP connection with process
ss -s                       # summary counts by protocol and state
ss -tin                     # per-connection TCP internals (rtt, cwnd, retrans)

ss -antp dst 203.0.113.7               # one peer
ss -antp '( dport = :443 or sport = :443 )'
ss -antp state established
ss -antp state close-wait              # app leaking descriptors
ss -tan state time-wait | wc -l

# Top clients by connection count
ss -ant state established | awk '{print $5}' | cut -d: -f1 \
  | sort | uniq -c | sort -rn | head

Always -n. Without it, ss does a DNS lookup per address.

Reading the queue columns

ESTABLISHED socket:

• Recv-Q = bytes waiting for the app to read() → app is behind
• Send-Q = bytes waiting to be acked → network or peer is behind

LISTENING socket:

• Recv-Q = connections waiting for accept()
• Send-Q = maximum accept queue size
• Recv-Q at Send-Q = overflowing right now

ss -ti fields

TCP states

Counters

nstat                       # delta since last run — use this during incidents
nstat -az                   # everything, including zeros
nstat -d 1                  # sample every second

nstat -az | grep -i retrans
nstat -az | grep -i listen
nstat -az | grep -i Udp

Names vary by kernel version. nstat -az on your box is authoritative.

Interface and driver

ip -s link show eth0        # errors, dropped, overrun/missed
ip -s -s link show eth0     # more detail

ethtool -S eth0 | grep -i -E 'drop|err|miss|discard|fifo'
ethtool -g eth0             # ring buffer sizes (current vs max)
ethtool -k eth0             # offloads — GRO/TSO make tcpdump show huge packets
ethtool eth0                # link speed, duplex

tc -s qdisc show dev eth0   # transmit-side drops

errors = physical layer. Cable, SFP, port, duplex.
overrun / missed = ring buffer overflow. The box can't keep up.
dropped = kernel discarded it. Usually no buffer or no handler.

ethtool -S counter names are driver-specific. Grep, don't memorise.

Softirq

cat /proc/net/softnet_stat      # hex, one line per CPU
watch -d cat /proc/net/softnet_stat
cat /proc/interrupts | grep -i eth
mpstat -P ALL 1                 # %soft column

• Column 1 = processed. Wildly uneven across CPUs = IRQ imbalance.
• Column 2 = dropped (backlog full). Non-zero is a real, invisible drop.
• Column 3 = time_squeeze. NAPI budget exhausted.

tcpdump

tcpdump -i eth0 -nn -c 100 'host 203.0.113.7 and port 443'
tcpdump -i eth0 -nn -w /tmp/cap.pcap 'host 203.0.113.7'
tcpdump -r /tmp/cap.pcap -nn | head -50

tcpdump -nn 'tcp[tcpflags] & tcp-syn != 0'    # who is connecting
tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0'    # who is being reset
tcpdump -nn 'host X and not port 22'          # exclude your own SSH

Key flags: -nn (no resolution), -c N (stop after N), -w/-r (file), -e (ethernet header), -A/-X (payload), -S (absolute seq), -tttt (readable timestamps).

tcpdump sees inbound packets before netfilter. So: SYN visible but app never sees it = local firewall. SYN not visible at all = never arrived.

Flags

The dot is ACK. That's the whole notation.

Patterns

Window scale trap: if you didn't capture the handshake, every win value printed is wrong. Recapture from connection start.

Latency

curl -w '\ndns %{time_namelookup}\nconnect %{time_connect}\ntls %{time_appconnect}\nttfb %{time_starttransfer}\ntotal %{time_total}\n' \
  -o /dev/null -s https://example.com/

ping -c 20 X                # min/avg/max/mdev
mtr -rwc 100 X              # per-hop loss and latency
ss -tin state established dst X    # real RTT on the real connection
iostat -x 1                 # r_await / w_await, NOT %util
mpstat -P ALL 1             # %soft

Values are cumulative — subtract to get each phase.

• High connect, low TTFB → network
• Low connect, high TTFB → application
• Erratic connect → queueing or loss

mtr caveat: loss at intermediate hops is usually ICMP rate limiting, not real. Only trust loss at the final hop, or loss that persists through every subsequent hop.

iostat caveat: on SSD/NVMe, %util does not mean saturation. Use await.

Throughput ceiling ≈ window / RTT. 64KB over 100ms RTT ≈ 5 Mbit/s no matter how fast the link.

Historical data

sar -n DEV 1                    # throughput
sar -n EDEV 1                   # errors and drops
sar -n TCP,ETCP 1               # connection rates, retransmits
sar -n SOCK 1                   # socket counts

sar -n EDEV -f /var/log/sa/sa13
sar -n DEV -s 03:00:00 -e 04:00:00 -f /var/log/sa/sa13

Log path varies by distro (/var/log/sa or /var/log/sysstat). On Debian/Ubuntu the collector is often disabled by default — check /etc/default/sysstat.

Enable this on every production box. Every other tool tells you about now; incidents are about twenty minutes ago.

Routing, ARP, firewalls, containers

ip route get 10.0.0.50        # ASK the kernel. Gives via, dev, AND src.
ip route                      # the table — most specific prefix wins
ip rule show                  # which table gets consulted, lowest number first
ip route show table 100

sysctl -a 2>/dev/null | grep '\.rp_filter'    # 0=off 1=strict 2=loose
nstat -az | grep IPReversePathFilter          # proof rp_filter is dropping

ip neigh show                 # REACHABLE / STALE / INCOMPLETE
ip neigh flush dev eth0
arping -I eth0 192.0.2.1
sysctl net.ipv4.neigh.default.gc_thresh3

ip route get over reading the table. It runs the real decision and reports the source IP that will be used — which is what the far end's firewall sees.

INCOMPLETE in ip neigh = ARP sent, nobody answered. Below every other layer in this series.

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v          # container hosts: check nat BEFORE filter
iptables -Z                       # zero, reproduce, then look
nft list ruleset
conntrack -L ; conntrack -E ; conntrack -S

Zero, reproduce, look. iptables -Z, trigger the problem, iptables -L -n -v — whichever counter moved is the rule that handled your packet. Stop reasoning about rule order; measure it.

ip netns list                              # often EMPTY on Docker — not a bug
docker inspect -f '{{.State.Pid}}' <name>
nsenter -t <pid> -n ss -lntp
nsenter -t <pid> -n ip -br addr
nsenter -t <pid> -n tcpdump -nn -i eth0

nsenter -t <pid> -n ip link show eth0      # shows eth0@ifN
ip link | grep '^N:'                       # N = the host-side veth
bridge link ; bridge fdb show

nsenter -t <pid> -n <cmd> runs your host tools inside the container's namespace. You never need to install debugging tools in an image.

eth0@ifN inside the container → interface index N on the host. That's which veth to capture on.

Namespaces are independent stacks: host ss, tcpdump, iptables and every counter in part 3 do not cover container traffic.

ss -xlp                       # listening unix sockets + process
ss -xp

Unix sockets have no packets — tcpdump will never show them. (13: Permission denied) is a file-permission problem, not a network one.

Config worth checking

sysctl net.core.somaxconn                  # accept queue ceiling
sysctl net.ipv4.tcp_max_syn_backlog        # SYN queue
sysctl net.core.netdev_max_backlog         # softirq backlog
sysctl net.ipv4.ip_local_port_range        # outbound port pool
sysctl net.ipv4.tcp_syncookies
sysctl net.ipv4.tcp_congestion_control
sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count
cat /proc/<pid>/limits                     # the FD limit that actually applies

Rules on tuning:

• Accept queue = min(listen() backlog, net.core.somaxconn). Raising one alone does nothing.
• Never net.ipv4.tcp_tw_recycle. Removed in kernel 4.12; broke NAT'd clients before that. Any guide recommending it predates 2017.
• Buffer tuning (tcp_rmem/tcp_wmem) matters for high bandwidth-delay paths. It's cargo cult for a web server 20ms from its clients.
• A full accept queue is a symptom. A deeper queue is somewhere deeper to fall behind.

nginx specifics

tail -100 /var/log/nginx/error.log

Add to log_format and split the time:

• $request_time — total
• $upstream_connect_time — connecting to backend
• $upstream_header_time — until backend headers
• $upstream_response_time — total backend time

$upstream_response_time ≈ $request_time → the backend is the problem, run the same investigation one layer down.

Use keepalive in upstream blocks for high-volume proxying, or you'll exhaust source ports. Configuration details in the nginx upstream docs.

Learning path

Honest sequencing. Don't skip.

Stage 0 — the floor (a few days)

Glossary above, cold. The seven-step reachability order: address → route → ping → DNS → port → TLS → HTTP. Learn that ping failing proves nothing and nc -zv is the real test. Learn why dig and getent can disagree. Read man ip, man dig, man nc.

Stage 1 — the model (a week)

The 4-tuple. Socket buffers. The TCP handshake and teardown. Stream vs datagram. Draw the seven-layer ladder from memory. Capture a healthy request with tcpdump and read every field. You cannot recognise broken without knowing healthy.

Stage 2 — the socket layer (a month of using it)

Live in ss. ss -lntp and ss -antp until they're muscle memory. Learn what Recv-Q and Send-Q mean in both contexts. Learn the states, especially CLOSE_WAIT. Read man ss properly — it's short.

Stage 3 — evidence (a month)

nstat, ip -s link, ethtool -S, /proc/net/softnet_stat. The goal is one skill: make a falsifiable claim. Not "we have packet loss" but "ListenOverflows went 0 → 14,000 in ten minutes while interface errors stayed at zero." Learn the difference between a drop and a retransmit until it's automatic.

Stage 4 — the wire (ongoing)

tcpdump filters, then the eight patterns above. Capture on both ends of a connection at once and compare — that's the highest-value habit in this entire series. Then Wireshark for anything complex.

Stage 5 — attribution (ongoing)

curl -w decomposition. Path vs application. iostat, mpstat, and understanding that application and disk latency surface as network symptoms. Work a real production problem end to end.

Stage 6 — routing, L2 and containers (a month)

ip route get and ip rule. rp_filter and asymmetric routing. ip neigh states. Reading firewall rule counters with the zero-reproduce-look technique. Then namespaces: nsenter -t <pid> -n, veth pairs, and where container traffic gets NAT'd. If you run containers, this stage is not optional.

Stage 7 — eBPF

bpftrace, and the BCC tools: tcpretrans, tcplife, tcpconnect, biolatency. Move from "0.3% of connections retransmitted" to "this connection retransmitted, at this moment." This is the line between competent and strong.

Reading, in order

1. man 7 tcp — dense, unglamorous, better than most books
2. man ss, man tcpdump, man pcap-filter
3. TCP/IP Illustrated, Volume 1 (Stevens / Fall) — nothing else builds the same depth
4. Documentation/networking/ in the kernel source
5. BCC and bpftrace

The five things to remember if you forget everything else

1. Everything is a queue. Full queue = drops. Deep queue = latency. Find the queue.
2. Drops and retransmits point in opposite directions. Drops mean this box discarded something. Retransmits mean the path lost something.
3. Fast failure and slow failure are different diagnoses. Refused = it arrived. Timeout = it didn't.
4. ss -lnt before tcpdump. Recv-Q at Send-Q on a listener answers more incidents than any packet capture.
5. Localise, then tune. Every sysctl is a guess until you know which layer is failing.
6. The application is part of the network stack. Most "network problems" are an app that stopped reading its sockets.

Compiled by AI. Proofread by caffeine. ☕