Somebody says "the app can't reach the database." You have a terminal and no idea where to start.
This post is the on-ramp. Vocabulary, then the seven commands that answer "is it reachable, is the port open, is DNS working, is TLS working, and who owns this socket." No kernel counters, no packet analysis — those come later in the series. Just the ground floor, done properly.
If you're already comfortable with dig and nc, skim the glossary and skip to part 1. If you're not, this is the post that makes the rest of the series readable.
Glossary
Get these straight once and a lot of confusion evaporates.
Interface — a network device as the kernel sees it. eth0, ens3, lo, docker0. Might be physical hardware, might be entirely virtual.
Loopback (lo, 127.0.0.1) — the interface that talks to the machine itself. Traffic on it never reaches a wire. A service bound to 127.0.0.1 is unreachable from anywhere else, permanently, no matter what your firewall says.
IP address — identifies a host. 192.0.2.10.
Subnet / CIDR — 192.0.2.0/24 means the first 24 bits are the network part, leaving 256 addresses. Hosts in the same subnet reach each other directly over layer 2; anything else goes via a gateway.
Gateway / default route — where packets go when the destination isn't in a directly-connected subnet. "The way out."
Port — a 16-bit number (1–65535) identifying which service on a host. Not a physical thing.
Well-known vs ephemeral ports — below 1024 are the reserved ones (22 SSH, 53 DNS, 80 HTTP, 443 HTTPS) and binding them traditionally requires privilege. The high range is the ephemeral pool the kernel picks from for outbound connections.
Socket — the kernel structure representing one endpoint of a communication. Part 1 goes deep.
The 4-tuple — source IP : source port → destination IP : destination port. The unique identity of a TCP connection. The single most useful concept in this entire series.
Listening vs established — a listening socket waits for connections. An established socket is one live connection. One listener on port 443 can have 50,000 established connections beneath it.
TCP — reliable, ordered, connection-oriented. Handshake first, retransmits what's lost. A byte stream, not messages.
UDP — fire and forget. No handshake, no retransmission, no ordering. Messages preserved as units. Fast, and lossy by design.
ICMP — the control protocol. Error and diagnostic messages: ping, "destination unreachable", "fragmentation needed". Not TCP or UDP, and it has no ports. Blocking it wholesale breaks things people don't expect.
Frame / packet / segment / datagram — the same data at different layers. Frame on the wire (layer 2, has MAC addresses). Packet at the IP layer (layer 3). Segment is TCP's unit, datagram is UDP's (layer 4). People use them interchangeably; being precise helps when reading man pages.
MAC address — layer 2 hardware address, 02:fc:00:00:00:05. Only meaningful within a single network segment.
ARP — how a host discovers the MAC address for an IP on its local segment. Part 6.
MTU — the largest packet an interface will send, typically 1500 bytes on Ethernet. Tunnels and VPNs reduce it, and that causes some of the nastiest bugs in networking.
MSS — the largest TCP payload per segment. Roughly MTU minus IP and TCP headers.
RTT — round-trip time. How long a packet takes to get there and back.
TTL — a hop counter in the IP header. Each router decrements it; at zero the packet is discarded and an ICMP error is returned. This is the mechanism traceroute exploits.
NAT — rewriting addresses in flight, so many private hosts share one public IP. Your home router does it. So does Docker.
DNS — names to addresses.
TLS — the encryption layer under HTTPS. A separate handshake after the TCP handshake, which is why it gets its own timing measurement.
The order to check things
Work bottom-up. Each step assumes the one before it passed.
1. Is the interface up and does it have an address? ip -br addr
2. Do I have a route to the destination? ip route get <ip>
3. Does the host respond at all? ping
4. Does the name resolve? dig / getent
5. Is the port open? nc -zv
6. Does TLS work? openssl s_client
7. What is the server actually doing? curl -v
Most people start at 5 and work backwards in a panic. Starting at 1 takes ninety seconds and eliminates half the possibilities.
1. Interfaces and addresses
ip -br addr
ip -br link
-br is "brief," and it's the flag that makes ip pleasant to use. Compare:
lo UNKNOWN 127.0.0.1/8
eth0 UP 192.0.2.2/24
Two lines instead of twenty. ip -br link shows the same interfaces with MAC addresses and state.
What to look for: is the interface UP, does it have the address you expect, is the prefix length (/24) right. A wrong prefix length is a classic — the host thinks half the network is local and never sends those packets to the gateway.
ifconfig does a similar job and is from the deprecated net-tools package. It's often not installed. Use ip.
2. Routing — the one command
ip route get 8.8.8.8
8.8.8.8 via 192.0.2.1 dev eth0 src 192.0.2.2 uid 0
cache
This is the best routing command on Linux and it's underused. It doesn't dump the table for you to interpret — it asks the kernel to make the actual decision and report it. You get:
via 192.0.2.1— the next hop it will usedev eth0— the interface it will leave bysrc 192.0.2.2— the source IP it will use, which is the one your firewall rules on the far end need to allow
That source address is the part people miss. On a multi-homed box, "which IP do we appear to come from" is not a guess — this command tells you.
Full table if you need it:
ip route
Policy routing and multiple tables are part 6.
3. ping — and its limits
ping -c 5 192.0.2.1
ping sends ICMP echo requests. -c 5 stops after five, which you want — otherwise it runs forever.
Read the summary line: min/avg/max/mdev. A tight spread means a stable path. mdev (mean deviation) high means jitter, which means queueing.
The critical caveat, and it's the most common junior mistake: a failed ping does not mean the host is down.
Plenty of hosts and networks drop ICMP by policy. Cloud security groups routinely block it by default. A host can be perfectly healthy, serving traffic on port 443, and completely silent to ping.
So: ping succeeding is useful information. Ping failing is almost none. Never report "the host is down" based on ping alone. Test the actual port instead — step 5.
Useful variants:
ping -c 10 -i 0.2 host # faster interval, better sample
ping -M do -s 1472 host # don't fragment, 1472+28 = 1500 — MTU testing
Flags differ between the iputils and BSD implementations. Check man ping on your box.
4. traceroute — and why it lies
traceroute 8.8.8.8
The mechanism, because it explains the weird output: traceroute sends packets with TTL=1, then TTL=2, then 3. Each router that decrements TTL to zero discards the packet and returns an ICMP "time exceeded" — revealing its address. Increment until you reach the destination.
Two things that confuse everyone:
Stars (* * *) do not mean a broken hop. They mean that router chose not to send an ICMP reply. Many are configured that way. Traffic still flows straight through. A hop of stars in the middle with the destination reachable at the end is completely normal.
Loss at an intermediate hop is usually fake. Routers deprioritise generating ICMP for their own address — it's handled by a slow control-plane CPU, not the forwarding hardware. A router showing 40% "loss" may be forwarding your real traffic flawlessly. Only loss at the final hop, or loss that persists from a hop through every hop after it, is real.
I have watched engineers escalate to a transit provider over hop-4 ICMP rate limiting. Know this and you won't be that person.
By default traceroute uses UDP to high ports, which firewalls often block. TCP mode usually gets further because it looks like real traffic:
traceroute -T -p 443 example.com # may need root
mtr combines traceroute and ping, probing continuously — better for spotting intermittent problems. mtr -rwc 100 host gives a report of 100 probes per hop.
5. DNS — properly
The gap that causes the most wasted hours, because DNS failures don't look like DNS failures.
How a Linux host resolves a name
Applications do not just query DNS. They call getaddrinfo(), which follows the Name Service Switch configuration:
grep '^hosts' /etc/nsswitch.conf
hosts: files dns
That means: check /etc/hosts first, then DNS. Other entries you'll meet are mdns4_minimal, resolve (systemd-resolved), and myhostname.
Then, for the DNS step:
cat /etc/resolv.conf
Nameservers, and a search domain list that gets appended to unqualified names.
On systemd-resolved systems, /etc/resolv.conf often just points at a local stub (127.0.0.53) and the real configuration lives elsewhere:
resolvectl status
The distinction that solves the hard cases
dig +short example.com # queries DNS directly
getent hosts example.com # goes through NSS — the same path your app uses
dig bypasses /etc/hosts and NSS entirely. It talks to a nameserver directly.
So when dig returns the right answer but your application can't resolve the name, the problem is not DNS. It's /etc/hosts, nsswitch.conf, or the resolver library. That divergence between dig and getent is the fastest way to find it, and almost nobody checks both.
dig, the parts you need
dig example.com # full answer with sections
dig +short example.com # just the addresses
dig @8.8.8.8 example.com # ask a specific server — bypass yours
dig MX example.com # a specific record type
dig -x 192.0.2.10 # reverse lookup
dig +trace example.com # follow delegation from the root
Two of these earn their keep:
dig @8.8.8.8 — if your resolver gives a stale or wrong answer and a public one gives the right answer, your resolver or its cache is the problem. One command, problem localised.
dig +trace — walks the delegation chain from the root servers down. This is how you find "the nameservers at the registrar don't match the ones in the zone," which produces intermittent failures that look like black magic.
Also watch the Query time: line in full dig output. A resolver taking 2 seconds is your latency problem, and no amount of TCP tuning will help.
host and nslookup do similar jobs. dig gives more detail and is what you'll see in every serious runbook. It ships in dnsutils or bind-utils depending on distro.
6. Is the port actually open?
nc -zv 192.0.2.10 443
-z— scan only, don't send data-v— tell me the result
Bound the wait, or a filtered port hangs:
nc -zv -w 3 192.0.2.10 443
This, not ping, is how you test whether a service is reachable. It tests the exact thing that matters: can a TCP connection be established to that port.
Read the failure mode, because it tells you why:
- "Connection refused", instantly — the packet reached the host and nothing is listening. The path is fine, the service is down or bound to the wrong address.
- Hangs, then times out — something is silently dropping. A firewall with a DROP rule, a security group, or the host is unreachable. See part 4 for confirming which.
Fast failure and slow failure are different diagnoses. That distinction alone will save you hours.
A range:
nc -zv -w 2 192.0.2.10 20-25
UDP (-u) is much less reliable — no handshake means no confirmation, so "success" often means "no error came back." Don't trust a UDP scan result the way you trust a TCP one.
nc implementations differ. OpenBSD netcat, traditional netcat, and Ncat (from nmap) have overlapping but not identical flags. Check nc -h on the box. If nc isn't installed, curl -v telnet://host:port works as a fallback, and socat is the more powerful sibling worth knowing about.
7. Which process owns this port?
Three ways, all worth knowing:
ss -lntp # the primary tool — part 2 covers it fully
lsof -i :443 # everything using port 443
lsof -i -P -n # all network connections, unresolved
fuser 443/tcp # just the PID
lsof -i is genuinely useful because it thinks in files, so it also shows you which files that process has open — often the fastest way to find a config file a service actually loaded. Use -P and -n to skip port-name and hostname resolution.
And when you need to see what a running process is actually doing at the syscall level:
strace -f -e trace=network -p <pid>
You'll see the socket(), connect(), accept(), sendto() calls in real time, with their return values. This is how you answer "is it even trying to connect, and where?" — a question no packet capture answers, because the packet may never be sent.
strace pauses the traced process on every syscall, so it is genuinely slow. Fine on a struggling service, dangerous on a hot one. Use it deliberately.
8. TLS
Connection works, HTTPS doesn't. Look at the TLS layer directly:
openssl s_client -connect example.com:443 -servername example.com
-servername sets SNI. Leave it off and you'll often get the wrong certificate, because one IP serves many sites and the server needs the name to pick. Then you spend twenty minutes confused about a certificate mismatch you created yourself.
What to read in the output: the certificate chain, Verify return code at the bottom (0 (ok) is what you want), and the negotiated protocol and cipher.
Certificate dates specifically:
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -dates -subject -issuer
Expired certificates cause a startling proportion of "the network is broken" incidents.
Quicker, for the common case:
curl -vI https://example.com/
-v shows the TLS handshake and certificate summary, -I fetches headers only. If curl fails but nc -zv to port 443 succeeds, the TCP layer is fine and your problem is TLS or HTTP — not the network.
9. Bandwidth
ping measures latency. It tells you nothing about throughput. For that:
# On the server
iperf3 -s
# On the client
iperf3 -c <server-ip>
iperf3 -c <server-ip> -R # reverse — test the other direction
iperf3 -c <server-ip> -P 4 # 4 parallel streams
Two things worth knowing:
Test both directions. Asymmetric performance is common and single-direction tests miss it entirely.
A single TCP stream often can't fill a fast long-distance link. That's the window-over-RTT ceiling from part 5, not a broken network. If one stream gives you 50 Mbit/s and four parallel streams give you 200, you're seeing a per-connection window limit, not a bandwidth limit. Diagnosing that as "the link is slow" is a classic mistake.
iperf3 needs to be installed on both ends and it needs a port open between them.
Putting it together
"The app can't reach the database." Ninety seconds:
ip -br addr # do I have an address?
ip route get 10.0.0.50 # do I have a route, and from which source IP?
getent hosts db.internal # does the name resolve the way the app sees it?
dig +short db.internal # and does DNS itself agree?
nc -zv -w 3 10.0.0.50 5432 # is the port actually open?
Five commands and you'll usually know. And crucially, you'll know which layer to escalate to instead of saying "the network is broken."
Then read the failure:
- No route → routing or interface configuration. Part 6.
getentfails butdigworks → NSS,/etc/hosts, or resolver config. Not DNS.- Both fail → DNS. Try
dig @<other-server>to see if it's your resolver. ncrefused instantly → reached the host, nothing listening. Check the service and its bind address withss -lntp(part 2).nchangs → something is dropping silently. Firewall. Part 4 proves it.
What did we just build?
The floor. Seven questions, in an order, each with one command and a way to read the answer.
The three things worth carrying out of this post:
- Ping failing means almost nothing. Test the port, not ICMP.
digandgetentanswer different questions. When they disagree, you've found the bug.- Fast failure and slow failure are different diagnoses. Refused means it arrived. Timeout means it didn't.
Part 1 goes under the hood: what a socket really is, and the seven-stop path a packet takes from the wire to your application.
Primary sources: man ip, man dig, man nc, man ping, man traceroute, man nsswitch.conf, man getaddrinfo.
Compiled by AI. Proofread by caffeine. ☕