Linux Networking Part 3: Proving Packet Loss

"We're seeing packet loss."

That sentence starts more wasted investigations than any other in infrastructure. It's not a diagnosis. It's a feeling. Packets can be discarded at six different places between the wire and your application, and each one has a different cause, a different counter, and a different fix.

This post is about replacing the feeling with a number.

Part 1 gave us the ladder. Part 2 covered the socket layer. Now we go down — stops 1 through 5, the layers ss can't see, where packets die before any socket ever hears about them.

The rule

A drop counter tells you where. A retransmit counter tells you the path lost something. They are not the same measurement and they are almost never both true at once.

If a router three hops away discards your packet, your NIC's drop counter stays at zero forever. Your NIC never saw the packet. The only evidence you will ever get is that TCP had to send it again.

Conversely, if your NIC's ring buffer overflows, that's not "the network" — that's your machine failing to keep up with traffic that arrived perfectly.

Get this backwards and you'll open a ticket with your network team about a problem that lives entirely inside your own kernel. It happens constantly.

nstat — start here

nstat reads the kernel's network counters from /proc/net/snmp and /proc/net/netstat and, unlike netstat -s, it's built to show deltas.

That matters more than it sounds. A counter reading 4,000,000 retransmits is meaningless — is that from the last minute or from 400 days of uptime? What you need is the rate right now.

# Absolute values, including zeros
nstat -az

# Just what changed since the last time you ran it
nstat

# Sample every second and print what moved
nstat -d 1

nstat with no arguments stores a snapshot and prints the difference on the next run. So the workflow during an incident is: run it once to set the baseline, wait, run it again. What prints is what's happening now.

To find a specific counter:

nstat -az | grep -i retrans
nstat -az | grep -i listen
nstat -az | grep -i drop

The counters worth knowing by name

Loss on the path:

• TcpRetransSegs — total segments retransmitted. The headline number for "is the network losing our traffic."
• TcpExtTCPSynRetrans — retransmitted SYNs specifically. If this is climbing, connection setup is failing, which points at a firewall or an overloaded listener rather than general congestion.
• TcpExtTCPTimeouts — retransmission timeouts fired. Worse than a fast retransmit: it means TCP waited out the full timer with no acknowledgement at all.
• TcpExtTCPLostRetransmit — a retransmission was itself lost. Bad path, or serious congestion.

Some retransmission is normal on the public internet. What you want is the ratio: retransmits against TcpOutSegs. A fraction of a percent is background noise. A few percent is a real problem. Ten percent and users are already complaining.

Listener overload:

• TcpExtListenOverflows — accept queue was full when a handshake completed. Covered in part 2; this is the cumulative proof.
• TcpExtListenDrops — all drops on listening sockets, overflows included.

Receive-side pressure:

• TcpExtTCPBacklogDrop — the socket was locked by its owning process, so incoming packets went to the backlog queue, and that filled. Your application is holding the socket too long.
• TcpExtPruneCalled / TcpExtRcvPruned — the kernel is under receive-buffer memory pressure and discarding queued data. Rare, and when it shows up it's usually the real answer.
• TcpExtTCPZeroWindowDrop — data arrived while the receive window was zero. The receiver said "stop" and the sender didn't.

Connections dying:

• TcpAttemptFails — outbound connection attempts that failed.
• TcpEstabResets — established connections terminated by RST.
• TcpOutRsts — resets we sent. A jump here usually means we're rejecting connections — to a closed port, or from an app slamming connections shut.

UDP, which has its own vocabulary:

• UdpNoPorts — datagrams arrived for a port nothing was listening on.
• UdpInErrors — receive errors.
• UdpRcvbufErrors — the important one. The socket receive buffer was full and datagrams were thrown away. UDP has no retransmission, so this is permanent, silent data loss. If you run DNS, syslog, or statsd, check this counter. People run these services for years without knowing it exists, quietly losing metrics.
• UdpSndbufErrors — same on the send side.

Counter names differ slightly across kernel versions, and new ones appear. nstat -az on the machine in front of you is the authoritative list. Don't trust a name from a blog post — including this one — if grep doesn't find it locally.

Interface counters — layers 1 and 2

nstat covers the IP and TCP layers. For the wire itself:

ip -s link show eth0

Two blocks, RX and TX. The exact column headers vary between iproute2 versions, so read the header your box prints rather than assuming. The ones you care about:

• errors — malformed frames. Checksum failures, framing errors. Real errors here mean physical problems: bad cable, bad SFP, duplex mismatch, dying port. Non-zero and climbing is a hardware conversation.
• dropped — the kernel received the frame and threw it away. Not a hardware error. Usually no buffer space, or no protocol handler.
• overrun / missed — the ring buffer overflowed. The NIC had packets and the kernel didn't collect them in time. This is the classic "box can't keep up" signal.
• carrier — link-layer problems. Flapping link.

For more detail:

ip -s -s link show eth0

Same data, broken out further.

ethtool — the driver's own numbers

ip -s link shows the kernel's generic view. The driver keeps its own, more detailed counters:

ethtool -S eth0

These counter names are driver-specific. An Intel ixgbe card, a Mellanox mlx5, and a virtio NIC in a VM all name things differently. Do not memorise names from a blog; grep what your card actually reports:

ethtool -S eth0 | grep -i -E 'drop|err|miss|discard|fifo'

Anything non-zero and climbing there is worth understanding. If a counter name isn't obvious, look it up in that specific driver's documentation — guessing at semantics from the name is how people end up chasing counters that are harmless on their hardware.

Ring buffers

If you're seeing overruns, this is the knob:

ethtool -g eth0

Prints the hardware maximum and the current setting for RX and TX ring sizes. Vendors frequently ship a default well below the maximum.

Raising the ring size (ethtool -G, syntax in man ethtool) gives the kernel more slack to absorb bursts. It is a genuine fix for burst-driven overruns. It is not a fix for sustained overload — a bigger queue that's permanently full just adds latency to the drops. And it's not free: bigger rings mean more memory and can hurt cache behaviour. Change it deliberately, measure before and after.

Offloads — and why your tcpdump will lie

ethtool -k eth0

Shows offload features. Two matter for troubleshooting:

• GRO / LRO (receive) — the NIC or driver merges multiple incoming segments into one big buffer before the stack sees it.
• TSO / GSO (send) — the stack hands down one huge buffer and the NIC splits it into wire-sized packets.

These are good for performance and terrible for your sanity, because tcpdump captures after GRO and before TSO. You will see 40KB "packets" on a link with a 1500-byte MTU and think you've found something profound. You haven't. You're seeing merged buffers.

Know this now so part 4 doesn't confuse you. If you need to see true wire packets, offloads can be disabled temporarily — check man ethtool for the exact syntax on your version, and remember it will cost throughput while it's off.

The file almost nobody knows

cat /proc/net/softnet_stat

One line per CPU, all values in hex. This is stop 2 on the ladder — packet processing in software interrupt context.

The first three columns are the ones to read:

1. processed — packets this CPU handled.
2. dropped — packets dropped because the per-CPU backlog queue was full. Governed by net.core.netdev_max_backlog.
3. time_squeeze — the number of times the NAPI poll loop ran out of budget with work still pending.

What to look for:

Column 2 non-zero — you are dropping packets in softirq processing. The interface counters may look fine. Nothing at the socket layer will tell you. This is a real, invisible, easily-missed drop.

Column 3 climbing — softirq processing is being cut off before it finishes. Under heavy load some of this is expected; growing steadily under normal load is not.

Wildly uneven columns 1 across CPUs — this is the big one. If CPU 0 processed 400 million packets and CPUs 1-31 processed almost nothing, all your network interrupt handling is landing on a single core. That core saturates, drops packets, and your monitoring shows 3% overall CPU usage because it's averaging across 32 cores.

This is one of the most common serious misconfigurations in Linux networking, and the only place it's visible is this hex file and /proc/interrupts. Multi-queue NICs with proper IRQ affinity spread the load. Check /proc/interrupts for how many queues your card exposes and where they're landing.

Since column 1 is a lifetime total, compare over an interval rather than trusting the absolute numbers:

watch -d cat /proc/net/softnet_stat

-d highlights what changes. Which column is moving tells you the story.

Conntrack — the silent killer

If your box does NAT, runs containers, or has any stateful firewall rules, the kernel keeps a table of every tracked connection. That table has a hard maximum.

sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count
conntrack -S

The first two are ceiling and current usage. When count approaches max, new connections get dropped. Not slowed. Dropped.

The symptom is brutal to diagnose: intermittent connection failures, no application errors, clean interface counters, healthy CPU. conntrack -S shows per-CPU statistics including drops and insert failures. The kernel also logs to dmesg when the table fills:

dmesg | grep -i conntrack

If you see "table full, dropping packet", you've found it. On container hosts this is a rite of passage. The conntrack-tools package provides the conntrack command; on some systems the sysctl path is net.nf_conntrack_max instead — check what exists on yours.

Outbound drops — the qdisc

Everything so far has been receive-side. Transmit has its own queue, and its own drops:

tc -s qdisc show dev eth0

Look for dropped, overlimits, and requeues in the statistics. If you have traffic shaping configured, or you're hitting the queue length limit, this is where it shows. On a box with no shaping and drops here, you're sending faster than the link can carry.

sar — the only tool that knows about yesterday

Every command so far tells you about now. Incidents are about twenty minutes ago, or last Tuesday at 03:00.

sar comes from the sysstat package and, when its collector is enabled, records system statistics to disk continuously.

Live:

sar -n DEV 1        # per-interface throughput and packet rates
sar -n EDEV 1       # per-interface ERRORS and drops
sar -n TCP,ETCP 1   # TCP connection rates, retransmits, resets
sar -n SOCK 1       # socket counts by protocol, including TIME_WAIT

Historical — this is the point of the whole tool:

sar -n DEV -f /var/log/sa/sa13     # data for the 13th of this month
sar -n EDEV -s 03:00:00 -e 04:00:00 -f /var/log/sa/sa13

The log directory path varies by distro (/var/log/sa on RHEL-family, /var/log/sysstat on some Debian-family systems), and on Debian/Ubuntu the collector is often disabled by default — installing sysstat isn't enough, you have to enable it. Check /etc/default/sysstat.

Strong opinion, and I'll die on this hill: enable sysstat on every production box you own, today. It is a few megabytes a day. The first time an incident resolves itself before you can log in and you can still say exactly what the retransmit rate was at 02:47, it pays for every byte. Every hour spent debugging blind is an hour you chose in advance by not enabling this.

Which counter answers which question

The table to actually use:

Which connection is retransmitting?

Counters give you totals. They can't tell you which connection is suffering — and that's usually the question you actually need answered.

Two ways up:

Per-connection, from part 2:

ss -tin state established

The retrans: field is per-connection. Slower to scan, but it points at a specific peer.

Live tracing. The BCC tools (package name is usually bcc-tools or bpfcc-tools depending on distro) include tcpretrans, which prints every retransmission as it happens with source, destination, and TCP state. There's also tcpconnect, tcpaccept, and tcplife.

That's the jump from "we have 0.3% retransmits" to "10.0.0.7 is retransmitting to 10.0.0.99 on port 5432." One of those gets fixed today. The tooling here moves fast, so get current syntax from the BCC repository rather than an old post.

What did we just build?

The ability to make a falsifiable claim. Not "we're seeing packet loss" but:

"TcpExtListenOverflows went from 0 to 14,000 in ten minutes while interface errors stayed at zero and retransmits didn't move. The network is fine. Our application stopped accepting connections."

That's a sentence that ends an argument between two teams. That's the entire job.

The order I'd work in, on a box I've never seen before:

1. nstat twice, thirty seconds apart — what's moving?
2. ip -s link — is the wire healthy?
3. ss -lnt — are the listeners keeping up?
4. /proc/net/softnet_stat — is one CPU drowning?
5. conntrack -S if the box does NAT or containers.

Five commands, under a minute, and you've eliminated most of the search space.

Part 4: when the counters aren't enough and you have to look at the actual packets. tcpdump, and reading SYN/ACK/RST like a sentence.

Primary sources: man ip, man ethtool, man nstat, man sar, and Documentation/networking/ in the kernel tree.

Compiled by AI. Proofread by caffeine. ☕