Linux Networking Part 6: Routing, ARP, Firewalls and Containers

Five parts in, we've covered sockets, counters, packets and latency. There's a gap, and it's the one that eats the most time on modern infrastructure: the packet reaches the box and disappears.

No socket sees it. No drop counter moves. tcpdump shows it arriving and then nothing. It was routed somewhere unexpected, discarded by a firewall rule, dropped by reverse path filtering, or delivered into a network namespace you didn't know existed.

This part covers stops 3 and 4 on the ladder — netfilter and routing — plus the layer 2 and container material the series has been quietly assuming.

Routing: ask, don't read

Everyone starts with ip route. Wrong instinct. Start with:

ip route get 10.0.0.50

10.0.0.50 via 192.0.2.1 dev eth0 src 192.0.2.2 uid 0
    cache

This asks the kernel to run its actual routing decision and report the result. No interpreting a table, no working out which prefix is more specific. Three answers:

• via — the next hop
• dev — the outbound interface
• src — the source IP that will appear on the packet

That third one carries more weight than people give it. On a multi-homed box, "which address do we come from" determines whether the far end's firewall accepts you. It's not a guess and it's not always the interface's primary address.

The table itself, when you need it:

ip route

default via 192.0.2.1 dev eth0
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.2

Reading it: most specific prefix wins. A /32 beats a /24 beats /0. default is 0.0.0.0/0 — the fallback. proto kernel means the kernel added it automatically when the address was configured. scope link means the destination is directly reachable without a gateway.

Multiple routing tables

Linux doesn't have one routing table. It has many, and a rule list deciding which to consult.

ip rule show

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Lowest priority number is evaluated first. main is what ip route shows by default. To see another:

ip route show table local
ip route show table 100

This matters on multi-homed hosts, VPN setups, and anything using policy routing — where the source address, mark, or interface picks a different table. If ip route get returns something that contradicts ip route, a rule sent it to another table. Check ip rule show before assuming the kernel is wrong.

Asymmetric routing and rp_filter

The failure that produces no error anywhere.

A packet arrives on eth1. The kernel checks: if I wanted to reply to this source address, would I send it out eth1? If the answer is no, reverse path filtering may discard the packet. Silently.

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.default.rp_filter
sysctl -a 2>/dev/null | grep '\.rp_filter'

Three values:

• 0 — off
• 1 — strict. The reply must go back out the same interface.
• 2 — loose. Any route to that source is enough.

Strict mode plus asymmetric routing equals silently dropped traffic. It's designed as an anti-spoofing measure and it does that job well; it also breaks legitimate multi-homed and policy-routed setups.

The counter that proves it:

nstat -az | grep -i IPReversePathFilter

If that number is climbing, rp_filter is eating your packets. This is one of the very few drops that shows up nowhere else — not in interface counters, not at the socket layer, not in firewall rule counters. Know the counter's name and you'll find it in a minute instead of a day.

Note that the effective setting is per-interface, and the all value interacts with the per-interface one. Check net.ipv4.conf.<iface>.rp_filter for the interface in question, and read the behaviour in the kernel's Documentation/networking/ip-sysctl.rst before changing it.

ARP and the neighbour table

Layer 2, which the series has skipped so far. When a host sends to an IP on its own subnet, it needs that host's MAC address. ARP is the discovery protocol; the results live in the neighbour table.

ip neigh show

192.0.2.1 dev eth0 lladdr 02:fc:00:00:00:05 REACHABLE

The states tell a story:

INCOMPLETE is the diagnostic. It means: I broadcast "who has this IP?" onto the local segment and nobody answered. The host is off, not on this segment, or something is filtering ARP.

That's a fundamentally different failure from "connection refused" or "connection timed out at the TCP layer," and it's below everything else in the series. No amount of ss or nstat will show it.

ip neigh flush dev eth0        # force re-resolution
arping -I eth0 192.0.2.1       # probe directly (package: iputils-arping or arping)

Duplicate IPs

If two hosts claim the same address, ARP replies alternate and connectivity becomes maddeningly intermittent — works, then doesn't, then works. arping will show replies from two different MAC addresses for one IP. Once you've seen it, the symptom is unmistakable; before that it looks like haunted hardware.

Neighbour table overflow

On large flat networks the table can fill:

sysctl net.ipv4.neigh.default.gc_thresh1
sysctl net.ipv4.neigh.default.gc_thresh2
sysctl net.ipv4.neigh.default.gc_thresh3
dmesg | grep -i 'neighbour table overflow'

gc_thresh3 is the hard maximum. Hit it and entries get evicted aggressively, causing intermittent failures to hosts that were fine a moment ago. The dmesg line is unambiguous when it happens. Common on hosts with thousands of neighbours — large L2 segments, or busy container hosts.

Reading firewall rules

I've said "check your firewall rules" three times in this series without showing you how. Here's how, and the technique matters more than the syntax.

iptables

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v --line-numbers

• -L list, -n numeric (always — otherwise it resolves every address), -v verbose, --line-numbers for reference.
• -t nat shows the NAT table. Forgetting this table exists causes a lot of confusion, because your filter rules look fine while NAT is rewriting everything.

-v is the whole point. It adds two columns at the front:

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source       destination
 4521  312K ACCEPT     tcp  --  *      *       0.0.0.0/0    0.0.0.0/0    tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0    0.0.0.0/0    tcp dpt:443
  892 53400 DROP       all  --  *      *       0.0.0.0/0    0.0.0.0/0

Those pkts counters are per-rule hit counts, and they turn firewall debugging from reading into measuring.

Look at that output: the rule accepting 443 has zero hits. The DROP at the bottom has 892. Traffic to 443 isn't matching the accept rule — wrong interface, wrong protocol, or an earlier rule caught it. You didn't have to reason about it. The counters said so.

The technique that makes this decisive:

iptables -Z              # zero all counters
# ... now reproduce the problem ...
iptables -L -n -v        # which rule moved?

Zero, reproduce, look. Whichever counter incremented is the rule that handled your packet. This is the single most useful firewall debugging move there is and most people never learn it.

Also check the chain policy — the (policy DROP) in the chain header. A default-DROP chain with no matching accept rule discards silently, which produces exactly the SYN-with-no-reply pattern from part 4.

nftables

Modern distros use nftables, often with iptables as a compatibility shim over it. If iptables -L looks empty but traffic is clearly being filtered, look here:

nft list ruleset

nftables doesn't count by default — you add counter to a rule to get statistics. If the ruleset was written without counters, add them to the rules you're investigating, or use logging.

Logging what gets dropped

When counters aren't enough, make the kernel tell you. Both iptables (LOG target) and nftables (log statement) can log matched packets to the kernel log, where dmesg or journalctl -k picks them up.

This is precise but dangerous on a busy box — an unrated log rule can flood your logs and your disk in minutes. Always rate-limit it, always remove it afterwards. Exact syntax is in man iptables-extensions and the nftables wiki; it's worth reading properly rather than pasting.

conntrack, revisited

Part 3 covered the table filling up. The other half is inspecting it:

conntrack -L                 # current tracked connections
conntrack -L -p tcp --dport 443
conntrack -E                 # live event stream
conntrack -S                 # per-CPU statistics

conntrack -E streams connection events as they happen — NEW, UPDATE, DESTROY. When you need to see whether the kernel is even creating state for a connection, this shows you live.

The failure to look for: a connection idle longer than conntrack's timeout gets its state removed, and the next packet arrives with no matching entry. Depending on rules, it's dropped or reset. That's the "connections die after exactly N minutes idle" pattern from part 4, seen from the kernel's side.

Containers and network namespaces

The modern gap. If you run containers and you've ever captured on the host and seen nothing, this is why.

The concept

A network namespace is an independent copy of the entire network stack — its own interfaces, routing table, ARP table, firewall rules, socket list, and counters. Containers each get one.

Consequences that surprise people:

• ss -lntp on the host does not show sockets inside containers.
• tcpdump on the host interface does not see container-internal traffic.
• iptables -L on the host is not the container's rule set.
• Every counter in part 3 is per-namespace.

Everything in this series still works. You just have to run it in the right namespace.

Getting into a namespace

ip netns list

Frequently empty on Docker hosts, and that's not a bug. ip netns only lists namespaces registered under /var/run/netns/, and Docker doesn't register them there. The namespaces exist; this command just can't see them.

The universal method — enter the namespace of a running process:

# Get the container's main PID
docker inspect -f '{{.State.Pid}}' <container>

# Run any network command inside its namespace
nsenter -t <pid> -n ss -lntp
nsenter -t <pid> -n ip -br addr
nsenter -t <pid> -n ip route
nsenter -t <pid> -n tcpdump -nn -i eth0

nsenter -t <pid> -n <command> is the key that unlocks the whole series inside containers. -n means "enter the network namespace only." The command runs with the host's filesystem and your host tools — so you get tcpdump, ss, and nstat inside a container that has none of them installed. That last point is worth pausing on: you do not need to install debugging tools in your images.

For Kubernetes, get the PID via crictl inspect on the node; the nsenter step is identical.

veth pairs — matching both ends

A container connects to the host through a veth pair: two virtual interfaces acting as a pipe. One end is eth0 inside the container, the other sits on the host, usually attached to a bridge.

Inside a container, ip link show gives you something like:

12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> ...

That @if13 is the interface index of the peer on the host. So:

# Inside the namespace — note the @ifN
nsenter -t <pid> -n ip link show eth0

# On the host — find index 13
ip link | grep '^13:'

Now you know exactly which host-side interface carries that container's traffic, and you can tcpdump it directly.

That trick — reading the peer index off eth0@ifN — turns "which of these forty veth interfaces belongs to my container" from guesswork into one command. It's the sort of thing that's obvious once and invisible before.

Bridges

ip -br link show type bridge
bridge link
bridge fdb show

bridge fdb show is the forwarding database — which MAC addresses the bridge has learned on which ports. The layer 2 equivalent of a routing table.

Where container traffic actually gets NAT'd

Docker publishes ports by inserting NAT rules. To see them:

iptables -t nat -L -n -v

You'll find DOCKER chains doing DNAT from the host port to the container's address. When a published port doesn't work, this is where to look — and the per-rule counters tell you whether packets are reaching the DNAT rule at all.

A rule of thumb worth internalising: on container hosts, check the nat table before the filter table. More container networking problems are NAT problems than firewall problems.

Unix domain sockets

Not networking in the wire sense, but they carry a lot of production traffic — nginx to php-fpm, most database local connections, container runtime APIs — and the series would have a hole without them.

A Unix domain socket is an endpoint identified by a filesystem path, not an IP and port. Communication never leaves the kernel: no IP, no TCP, no checksums, no packets.

ss -xlp          # listening unix sockets, with process
ss -xp           # all unix sockets

The "address" column is a path like /run/php/php8.2-fpm.sock.

Three things to know:

They don't appear in tcpdump. There are no packets. If nginx talks to php-fpm over a Unix socket, no amount of capturing will show you those requests. People lose real time to this.

Permissions matter. They're files, subject to file permissions. connect() to unix:/... failed (13: Permission denied) in an nginx log means the socket file's ownership or mode doesn't allow the nginx worker user. Not a network problem at all.

They're faster than loopback TCP — no protocol overhead — but only work on one machine. That's the tradeoff behind "should php-fpm listen on a socket or 127.0.0.1:9000."

What did we just build?

The layers the rest of the series assumed. When a packet arrives and vanishes, you can now find it:

Three habits from this post, if nothing else:

1. ip route get instead of reading the table. Ask the kernel what it will do.
2. Zero, reproduce, look. iptables -Z then check which counter moved. Stop reasoning about rule order; measure it.
3. nsenter -t <pid> -n. Every tool in this series, inside any container, without installing anything.

That's the series. The cheat sheet post collects all of it into one page, with a glossary and a learning path.

Primary sources: man ip-route, man ip-rule, man ip-neighbour, man iptables, man nsenter, nftables wiki, and Documentation/networking/ip-sysctl.rst in the kernel tree.

Compiled by AI. Proofread by caffeine. ☕