Linux Networking Part 4: Reading tcpdump

Counters tell you that something is wrong. tcpdump tells you what happened.

Most engineers can start a capture. Far fewer can look at forty lines of output and say "the firewall is dropping this" or "the server's accept queue is full" without opening Wireshark. That skill is smaller than it looks. There are maybe eight patterns worth recognising, and once you know them, most captures answer themselves in ten seconds.

Part 4 of the series. This one is about the wire.

Where tcpdump sits, and why it matters

tcpdump taps traffic via AF_PACKET, close to the driver. The consequence is worth learning cold:

On receive, tcpdump sees packets before netfilter. So if iptables or nftables is going to drop that packet, tcpdump still shows it arriving.

That gives you the cleanest diagnostic split in Linux networking:

• tcpdump shows the SYN arriving, but your application never sees the connection → the packet reached the box and something in the kernel ate it. Firewall rules, conntrack, or the socket layer.
• tcpdump shows nothing at all → the packet never got here. Routing, upstream firewall, wrong interface, the client isn't actually sending.

One capture, and you've cut the problem in half.

The invocation

tcpdump -i eth0 -nn port 443

The flags that matter:

-i eth0     interface. -i any captures on all of them.
-nn         no DNS resolution, no port-name resolution. ALWAYS use this.
-c 100      stop after 100 packets.
-w file.pcap   write raw packets to a file.
-r file.pcap   read a file back.
-v / -vv    more detail (TTL, IP ID, checksum validation).
-e          show the ethernet header — MAC addresses and VLAN tags.
-A          print payload as ASCII.
-X          print payload as hex and ASCII.
-S          absolute sequence numbers instead of relative.
-tttt       human-readable absolute timestamps.

Three habits worth building:

Always -nn. Single -n skips host resolution; double skips port-name resolution too. Without it, tcpdump does a DNS lookup per address and your capture output stalls — during a network incident, possibly forever. It also renders port 443 as "https", which is noise when you're pattern-matching.

Always bound the capture. -c 500, or -w to a file with a size limit. An unbounded capture on a busy interface fills a terminal faster than you can read and can fill a disk.

For anything non-trivial, capture to a file. Read it on the box with tcpdump -r, or pull it down and open it in Wireshark. Live terminal output is for quick yes/no questions.

# The pattern for real work
tcpdump -i eth0 -nn -w /tmp/capture.pcap 'host 203.0.113.7 and port 443'
# ... reproduce the problem ...
# Ctrl-C
tcpdump -r /tmp/capture.pcap -nn | head -50

On modern versions the default snapshot length already captures full packets, so the old -s 0 habit is unnecessary. Harmless, just dated.

Two gotchas that will confuse you

-i any doesn't give you an ethernet header. It uses a special cooked-capture mode. If you need MAC addresses or VLAN tags, capture on the real interface.

Offloads make packets look impossible. As covered in part 3: GRO merges incoming segments before tcpdump sees them, TSO means outgoing segments are split after. You will see a 40,000-byte "packet" on a 1500-byte MTU link. Nothing is wrong. You're seeing a merged buffer, not a frame that existed on the wire. Remember this before you spend an hour on it.

Capture filters

The filter syntax is BPF, and it's the same language libpcap uses everywhere. It's worth learning properly because it's the difference between 40 relevant packets and 40,000 irrelevant ones.

# By host
tcpdump -nn host 203.0.113.7
tcpdump -nn src host 203.0.113.7
tcpdump -nn dst host 203.0.113.7

# By port
tcpdump -nn port 443
tcpdump -nn dst port 443
tcpdump -nn portrange 8000-8100

# By network
tcpdump -nn net 10.0.0.0/8

# By protocol
tcpdump -nn tcp
tcpdump -nn udp port 53
tcpdump -nn icmp

# Combined — quote it so the shell leaves it alone
tcpdump -nn 'host 203.0.113.7 and port 443 and tcp'
tcpdump -nn 'port 80 or port 443'
tcpdump -nn 'host 203.0.113.7 and not port 22'

That last one — excluding your own SSH session — is one you'll use every single time you capture on a box you're logged into. Otherwise you're watching yourself watch the traffic.

Filtering on TCP flags

This is the power move, and it's how you answer real questions in one line:

# Only SYNs — who is trying to connect to us?
tcpdump -nn 'tcp[tcpflags] & tcp-syn != 0'

# Only RSTs — who is getting rejected?
tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0'

# Connection setup and teardown only, no data
tcpdump -nn 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'

The named constants are tcp-syn, tcp-ack, tcp-fin, tcp-rst, tcp-push, tcp-urg. Full syntax is in man pcap-filter, which is genuinely one of the better man pages on the system.

That middle command deserves its own callout. "Show me every reset on this box" is often the fastest route to a root cause, because RSTs are where things go wrong and someone is usually being explicit about it.

Reading a line

Here's a full handshake. This is the thing to be able to read at a glance:

14:23:45.123456 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 111 ecr 0,nop,wscale 7], length 0
14:23:45.123789 IP 10.0.0.5.443 > 203.0.113.7.51234: Flags [S.], seq 5000, ack 1001, win 65160, options [mss 1460,sackOK,TS val 222 ecr 111,nop,wscale 7], length 0
14:23:45.145123 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [.], ack 5001, win 502, length 0

Field by field:

• 14:23:45.123456 — timestamp, microsecond resolution. The deltas between lines are your latency measurement. In this capture, SYN to SYN-ACK is 333 microseconds — the server responded instantly. Client ACK comes 21ms later, which is the network round trip. Right there you've measured RTT without running ping.
• IP — IPv4. You'll see IP6 for v6.
• 203.0.113.7.51234 > 10.0.0.5.443 — source, then destination. Note tcpdump uses a dot before the port, not a colon. It reads oddly at first, then never again.
• Flags [S] — the TCP flags. The whole game. Decoded below.
• seq 1000 — sequence number. Relative to the connection start by default, which is what you want. Use -S for absolute.
• ack 1001 — the next byte expected from the peer.
• win 64240 — the receive window. How much the sender of this packet is willing to accept.
• options [...] — negotiated at handshake. mss is max segment size, sackOK means selective ACK is supported, wscale 7 is the window scale factor.
• length 0 — payload bytes. Zero for pure control packets.

The flag notation

This trips up everyone once. Decode it once, properly:

The dot is the ACK flag. That's the whole trick. [S] is a bare SYN, [S.] is SYN with ACK set. Once you see the dot as "ACK," the notation is obvious forever.

The window scale trap

win 502 on that third line looks alarmingly small. It isn't. The handshake negotiated wscale 7, so the real window is 502 × 2⁷ = 64,256 bytes.

The trap: if you start a capture on a connection that's already established, tcpdump never saw the handshake, so it can't apply the scale factor. Every window value it prints will be wrong by a large multiple. If window sizes look absurd, check whether you captured the handshake. Whenever you suspect a windowing problem, restart the capture and reproduce the connection from scratch.

The patterns

This is the part to keep. Eight signatures, each with a diagnosis.

1. SYN with no response — a firewall is dropping

14:23:45.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:46.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:48.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:52.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0

Same sequence number, retransmitted, roughly doubling intervals — 1s, 2s, 4s. That's TCP's exponential backoff.

Diagnosis: something is silently discarding the packet. A DROP firewall rule, a security group, a network ACL. Silence is the signature of DROP; a REJECT would send something back.

Client-side symptom: a long hang, then "connection timed out."

Now combine it with where you captured:

• Ran this on the server and you see the SYNs arriving but no SYN-ACK leaving? The packet reached the box. A local firewall rule is eating it. Check iptables -L -n -v / nft list ruleset and look at the counters — the rule that's dropping will have a rising packet count.
• Ran it on the server and you see nothing? The packet never arrived. Upstream: routing, cloud security group, network firewall.

Two captures, problem localised.

2. SYN then RST — nothing is listening

14:23:45.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:45.100 IP 10.0.0.5.443 > 203.0.113.7.51234: Flags [R.], seq 1, ack 1001, length 0

Immediate reset. Diagnosis: no socket is listening on that port, or a firewall configured to REJECT rather than DROP.

Client symptom: instant "connection refused."

The speed is the tell. Refused is fast; dropped is slow. If a user says "it fails immediately," you already know it's this class of problem before you capture anything. Confirm with ss -lnt from part 2 — and check the bind address, because a service bound to 127.0.0.1 produces exactly this from outside.

3. Handshake completes, then repeated SYN-ACK — the accept queue is full

14:23:45.100 IP client.51234 > server.443: Flags [S], seq 1000, length 0
14:23:45.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0
14:23:45.120 IP client.51234 > server.443: Flags [.], ack 5001, length 0
14:23:46.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0
14:23:48.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0

The client completed the handshake. The server keeps retransmitting its SYN-ACK as if it never got the final ACK.

Diagnosis: the server's accept queue was full when that ACK arrived. With net.ipv4.tcp_abort_on_overflow = 0 — the default — the kernel discards the ACK rather than resetting, and retransmits the SYN-ACK hoping the application catches up.

This is the packet-level signature of the problem from part 2. Confirm it in two seconds:

ss -lnt                            # Recv-Q at Send-Q on that listener
nstat -az | grep -i ListenOverflow # rising

The reason this matters: from the client's point of view it looks exactly like network loss. Same hang, same retransmissions. Teams have blamed their network provider for weeks over an application that was too slow to accept().

4. Duplicate ACKs and retransmission — genuine loss

IP server > client: Flags [P.], seq 1000:2000, length 1000
IP server > client: Flags [P.], seq 2000:3000, length 1000
IP client > server: Flags [.], ack 2000, length 0
IP client > server: Flags [.], ack 2000, length 0
IP client > server: Flags [.], ack 2000, length 0
IP server > client: Flags [P.], seq 2000:3000, length 1000

The client keeps acking 2000 — it's saying "I'm still missing the byte at 2000." After the third duplicate ACK the server fast-retransmits.

Diagnosis: real packet loss on the path. This is the pattern that justifies a call to the network team, and combined with a rising TcpRetransSegs from part 3, it's a solid case.

Note who's retransmitting. Loss on the path from server to client shows up as the server retransmitting.

5. Zero window — the receiver is drowning

IP client > server: Flags [.], ack 50000, win 0, length 0

win 0 means "stop sending, my buffer is full." The sender must wait and periodically probe.

Diagnosis: the receiving application isn't reading its socket fast enough. Same underlying condition as a climbing Recv-Q in part 2, seen from the other side.

Users experience this as a transfer that stalls, resumes, stalls. It is not a network problem — the network delivered everything perfectly. The receiver couldn't cope.

6. RST mid-connection — someone hung up

IP client > server: Flags [P.], seq 1000:1500, length 500
IP server > client: Flags [R.], seq 5000, ack 1500, length 0

An established connection, working fine, suddenly reset.

Common causes, roughly in order of frequency:

• The application crashed or the process was killed.
• A load balancer or proxy hit an idle timeout and tore the connection down.
• A firewall or NAT device dropped the connection from its state table — often because the connection was idle longer than the device's timeout. Then the next packet has no matching state and gets reset.
• The application deliberately reset, e.g. closing a socket with unread data in its receive buffer.

The idle-timeout case is the one to look for, because it has a fingerprint: connections die after a suspiciously round interval of inactivity. If everything breaks after exactly 350 seconds or exactly 5 minutes of idle, that's a middlebox timeout, not a bug in your code. TCP keepalives on a shorter interval than the middlebox timeout are the standard fix.

7. FIN vs RST — how it ended

# Graceful
IP client > server: Flags [F.], seq 2000, ack 5000, length 0
IP server > client: Flags [.], ack 2001, length 0
IP server > client: Flags [F.], seq 5000, ack 2001, length 0
IP client > server: Flags [.], ack 5001, length 0

Four packets: FIN, ACK, FIN, ACK. Both sides closed properly, all data delivered. This is what healthy looks like.

An RST instead means abrupt termination, and any data still in flight is discarded. Some RSTs are perfectly normal — plenty of software resets deliberately to avoid TIME_WAIT. A change in the FIN-to-RST ratio is the interesting signal, not the presence of resets.

8. Small packets fine, large packets stall — MTU blackhole

The signature: the TCP handshake completes instantly, small requests work, and any transfer over a few kilobytes hangs and retransmits forever.

Diagnosis: an MTU problem, with ICMP blocked.

Here's the mechanism, because this one is genuinely subtle. When a packet is too large for a link and has the "don't fragment" bit set, the router is supposed to send back an ICMP "fragmentation needed" message telling the sender the correct MTU. Path MTU Discovery depends on it.

Some network administrator, at some point, decided to "block ICMP" for security. So that message never arrives. The sender has no idea its packets are too big, keeps sending them at full size, and they keep vanishing. Small packets fit and get through, which is why the handshake works and everything looks superficially fine.

Confirming it:

# Send a large packet with DF set. Syntax varies between ping implementations.
ping -M do -s 1472 203.0.113.7

# The path MTU the kernel currently believes in
ip route get 203.0.113.7

# Trace where the MTU changes
tracepath 203.0.113.7

1472 bytes of payload plus 28 bytes of headers is exactly 1500. If that fails but -s 1400 succeeds, you have your answer. tracepath is specifically built to find where along the path the MTU drops.

This is endemic with VPNs, tunnels, and overlay networks, all of which reduce the effective MTU. And "we blocked ICMP for security" is the cause often enough that it's worth checking early. Check ping and tracepath syntax on your distro — flags differ between the iputils and BSD variants.

A workflow, not a command

When someone hands you "the connection doesn't work":

Capture on the server, filtered to the client:

tcpdump -i any -nn -c 100 'host <client-ip>'

Then read it against three questions:

1. Do you see the SYN? No → the packet never arrived. Look upstream: routing, cloud security groups, network firewalls. Stop capturing on this box.
2. Do you see a response? No → the packet arrived and something local ate it. Check local firewall rule counters and conntrack.
3. Does the handshake complete? Look at what happens next — repeated SYN-ACKs mean accept queue, an immediate RST means nothing listening, dup ACKs mean loss.

If you can, capture on both ends simultaneously. A packet that leaves the client and never appears on the server has been dropped in between, and you've proven it. That comparison is the single most powerful thing in this post, and it's the one people forget to do.

When to stop and open Wireshark

tcpdump is for quick answers on the box. For anything involving many connections, throughput analysis, or protocol decode above TCP, capture to a file and open it in Wireshark.

Wireshark's "Expert Information" and its TCP stream graphs will find things you'd miss reading text. tshark gives you the same engine on the command line. And Wireshark's display filters are a different, richer language than the capture filters above — that's a common source of confusion, not a mistake on your part.

Capture on the server, analyse on your laptop. That's the normal workflow, and it's not cheating.

What did we just build?

The ability to look at a capture and say what happened, in a sentence, without guessing.

The eight patterns cover the large majority of what you'll meet:

Part 5: latency. Where it hides, how to attribute it, and a full nginx walkthrough that uses everything in the series at once.

Primary sources: man tcpdump, man pcap-filter, and wireshark.org.

Compiled by AI. Proofread by caffeine. ☕