Most engineers learn Linux networking backwards. They memorise commands first, then spend years never quite knowing which one to reach for. So they run netstat -an, stare at 4000 lines, and guess.

The fix is not more commands. The fix is a map. Once you know the path a packet takes from the wire to your application, every tool slots into a specific point on that path, and "which command do I use?" answers itself.

This is part one of a series on troubleshooting connectivity, packet loss, and latency on Linux. This post is all model, almost no commands. That is deliberate. The commands in parts 2 through 5 will be useless to you without it.

What a socket actually is

Start here, because almost everything else is downstream of it.

A socket is a kernel data structure. That's it. It is not a connection, it is not a port, and it is not a file — though the kernel deliberately makes it look like a file so you can read() and write() it.

When your process calls socket(), the kernel allocates that structure and hands you back a file descriptor — an integer that indexes into your process's file descriptor table. Everything you do afterward, you do through that integer.

The important part: the socket owns two buffers.

  • A receive buffer. The kernel puts incoming data here. Your app drains it by calling read()/recv().
  • A send buffer. Your app fills it by calling write()/send(). The kernel drains it onto the network.

Hold onto that, because it explains the single most useful pair of numbers in Linux networking:

  • Recv-Q filling up means data arrived and your application is not reading it fast enough. That is an application problem.
  • Send-Q filling up means your application wrote data and the kernel cannot get it onto the network fast enough. That is a network problem — or a receiver problem.

Two columns. Two completely different on-call pages. We'll read them for real in part 2.

Stream, datagram, raw

When you create a socket you pick a type. The three that matter:

SOCK_STREAM — TCP. A reliable, ordered, bidirectional byte stream. Emphasis on byte stream: TCP has no concept of messages. If the sender writes 100 bytes then 100 bytes, the receiver might read 200 bytes at once, or 37 then 163. Every framing bug in every network protocol ever written comes from someone forgetting this. TCP guarantees the bytes arrive, in order, exactly once. It guarantees nothing about how they're grouped.

SOCK_DGRAM — UDP. Messages, preserved as units. Send a 100-byte datagram, and the receiver reads exactly 100 bytes or nothing. No ordering, no delivery guarantee, no retransmission. Which is a feature: DNS, QUIC, and most video use it precisely because they'd rather handle loss themselves than wait for TCP's retransmit.

SOCK_RAW — you build the packet headers yourself. This is what ping and tcpdump-adjacent tooling use. Needs privilege. You will almost never write one, but knowing it exists explains why some tools need root.

There's also SOCK_SEQPACKET — reliable and ordered like TCP, but message-preserving like UDP. Real, used by SCTP and Unix domain sockets, rare in practice. Mentioned so the name isn't a mystery when you meet it.

The 4-tuple — the one thing to memorise

A TCP connection is uniquely identified by four values:

source IP : source port  ->  destination IP : destination port

That's the 4-tuple. The kernel uses it as a lookup key. Every packet that arrives gets matched against this key to decide which socket it belongs to.

Three consequences fall straight out of this, and all three show up in production:

One. Your web server can hold 50,000 connections on port 443. Port 443 isn't "used up" by the first one. Each connection has a different client IP or client port, so each is a distinct tuple. People find this confusing for years. It's just a composite key.

Two. A client machine making outbound connections to a single destination IP and port is limited by its own source ports — roughly 28,000 by default (net.ipv4.ip_local_port_range). Hit that ceiling and new connections fail while the box looks completely idle. This is ephemeral port exhaustion, and it's one of the great "the server is fine, why can't we connect" mysteries. Part 5.

Three. TIME_WAIT sockets hold a tuple hostage for 60 seconds after close. Thousands of them are usually normal. Sometimes they're not. Part 2.

The path a packet takes

Here is the whole journey, inbound. Every troubleshooting tool you'll ever use plugs in at one of these stops.

   the wire
      |
 [1]  NIC hardware  ........ ring buffer, offloads, link errors
      |
 [2]  driver / softirq ..... NAPI polling, per-CPU processing
      |
 [3]  netfilter / conntrack  iptables, nftables, connection tracking
      |
 [4]  routing ............. which interface, which next hop
      |
 [5]  TCP/IP stack ........ checksums, reassembly, TCP state machine
      |
 [6]  socket receive buffer  the kernel's holding pen for your app
      |
 [7]  your application ..... read() / recv() / accept()

Read it once more. That ladder is the whole series.

[1] The NIC. Packets land in a ring buffer — a fixed-size queue in memory the card writes into. If the kernel doesn't drain it fast enough, the card overwrites or discards. That's a drop, and crucially it happens before anything else can see it. Nothing above this layer will tell you it happened. You need ethtool and ip -s link counters, which is part 3.

[2] Driver and softirq. The kernel processes received packets in software interrupt context, on a specific CPU. If that CPU is saturated — and it can be saturated while 31 other cores idle — you drop packets. /proc/net/softnet_stat is the file that tells you. Almost nobody knows it exists.

[3] netfilter and conntrack. Firewall rules run here. Connection tracking, which is what makes NAT and stateful rules possible, keeps a table with a maximum size. When that table fills, new connections get dropped — silently, as far as the application is concerned. If you run containers or NAT, this bites eventually.

[4] Routing. Does the kernel even know how to reach that destination? Which interface goes out? Asymmetric routing and policy routing live here.

[5] The TCP stack. The state machine, retransmissions, window management, congestion control. Where "packet loss" in the classic sense becomes visible as retransmits.

[6] The socket buffer. Data has arrived and is correct. It's sitting in the kernel, waiting. If it sits too long, it's because of [7].

[7] Your application. It has to actually call read(). Or, for a listening socket, accept(). If it doesn't — because it's blocked on a database query, or the thread pool is exhausted, or it's stuck in GC — the queues back up beneath it and the symptom looks exactly like a network problem. It isn't.

Outbound is the same ladder in reverse, with a different key failure: the qdisc (queueing discipline) between the stack and the driver, where traffic shaping and its own drops live.

The three questions

Every connectivity investigation is one of three questions. Knowing which one you're asking saves an hour.

"Can they reach me at all?" — connectivity. Is the port listening, is the packet arriving, is a firewall eating it, is routing sane. Tools: ss -lnt, tcpdump, ip route get, firewall rules.

"Are we losing traffic?" — loss. Not "does it feel slow" but counters, at a specific layer. Tools: nstat, ip -s link, ethtool -S, /proc/net/softnet_stat.

"Why is it slow?" — latency. Which almost always means: something is queueing. Find the queue. Tools: ss -ti, ping/mtr, sar, and reading Recv-Q/Send-Q.

Mixing these up is the classic failure. Someone reports slowness, and an engineer spends two hours in tcpdump proving packets arrive — which was never in doubt. The question was never connectivity.

Why "packet loss" is usually the wrong phrase

When someone says "we have packet loss," ask where.

  • Dropped at the NIC because the ring buffer overflowed? That's your box being overwhelmed. Fix locally.
  • Dropped in the network between two hosts? That shows up as TCP retransmissions, not as a drop counter on your interface. Your interface never saw the packet.
  • Dropped because the accept queue was full? The client sees a hang or a reset. Your NIC counters are pristine. Nothing is wrong with the network at all.
  • Dropped by a firewall rule? Nothing anywhere reports "loss." The packet was deliberately discarded and your counters are clean.

Four completely different problems, one phrase. This is why the ladder matters more than the tools: it turns "packet loss" into a location.

What did we just build? A map with seven stops and a rule: before you run anything, decide which stop you're investigating. From here on, every tool in this series is introduced as "this reads layer N".

Which tools are worth your time

Blunt opinion, since you'll meet all of these:

Learn these. ss (sockets and TCP state — replaces netstat entirely and is dramatically faster on busy hosts), ip (addresses, routes, link counters), nstat (kernel network counters, with deltas), tcpdump (what's actually on the wire), ethtool (NIC-level truth), sar (historical data — the only one of these that answers "what happened at 3am").

Know they exist. netstat, ifconfig, route — deprecated, from the net-tools package, often not even installed on modern distros. Read them in old runbooks; don't write new ones with them. netstat -s is the one genuinely worth keeping around, and nstat does the same job better.

The upgrade path. bpftrace and the BCC tools (tcpretrans, tcpconnect, tcplife). These let you ask questions the counters can't answer — which connection retransmitted, not just how many. This is the line between senior and everyone else. Later in the series.

One opinion I'll defend: sar is the most underrated tool on this list. Every other tool tells you about now. Incidents are about 20 minutes ago. If sysstat isn't collecting on your production boxes, you are choosing to investigate every future incident with one hand tied.

Next

Part 2: ss properly. Who is connected to you right now, from which IP, on which port, in which TCP state — and what those two queue columns are trying to tell you.

Reference for anything here: the socket(2), tcp(7), and ip(7) man pages are the primary sources and are better written than most blog posts, this one included.

Compiled by AI. Proofread by caffeine. ☕