Vocabulary, in one line each

Rack numbers

• 19 in / 482.6 mm — mounting face width (EIA-310). The cabinet is wider.
• 1U = 1.75 in = 44.45 mm. Holes in uneven groups of three per U.
• Cabinet width: 600 mm, or 800 mm when you want side cable channels.
• Cabinet depth: 600 / 800 / 1000 / 1200 mm. Datacenter servers need 800+.
• 600 mm deep = network cabinet. Will not take a server.
• 0U = mounts vertically in the side channel. PDUs. Costs you no U.
• Heavy at the bottom. Always.
• U1 is usually the bottom, but not universally. Check your rails and write it down.
• Multi-U devices are referenced by their lowest U.

Power

Usable capacity at 80 % of breaker rating (US NEC continuous-load rule; elsewhere the same practical outcome via PDU rating and contract):

Three-phase: kW = 1.732 × V × A.

The redundancy rule: in a 2N cabinet, 40 % per feed is full. Either feed must be able to carry the whole cabinet at 80 %. Running both feeds at 70 % means a single feed failure trips the survivor — that is swing load, and it turns one failure into a total outage.

IEC connectors — odd is socket, next even up is the matching inlet:

C15/C16 is the heat-rated variant with a notch — not interchangeable with C13.

NEMA input plugs (PDU → whip): L6-30P (208 V 30 A 1Ø, most common), L6-20P, L21-30P (3Ø wye), L15-30P (3Ø delta). L = twist-locking.

PDU tiers: basic → metered (total draw) → monitored (per outlet) → switched (per-outlet relays). Get switched and monitored; it is the only remote power cycle that works when the BMC is wedged.

Cables and optics

Reach:

Form factors — Q means four lanes:

Breakout: 1×100G → 4×25G, 1×40G → 4×10G. Because Q is four lanes.

Fibre — the jacket colour is reliable:

Transceiver naming: SR = short reach, multimode. LR = long reach, singlemode, 10 km. ER/ZR = further. A trailing 4 means four lanes, so eight fibres, so MPO rather than LC.

Connectors: LC duplex (2 fibres, the common one), MPO/MTP (12 or 24 fibres, for parallel optics and trunks), SC (older, carrier hand-offs).

Copper: Cat5e = 1 Gb, Cat6 = 10 Gb to 55 m, Cat6A = 10 Gb to 100 m. 10GBASE-T is hotter, slower-latency and more power-hungry than DAC — which is why it survives mainly for BMC and cheap kit.

Choosing: same cabinet → DAC. Adjacent → AOC. Through a patch panel → transceivers and fibre, always (you cannot patch an AOC).

Transceivers are vendor-coded; specify the switch model when ordering. DOM/DDM exposes Rx power per module — scrape it, it is the best early warning of a dying link.

Out-of-band

BMC="-I lanplus -H 10.20.0.37 -U admin -P $PW"

ipmitool $BMC chassis power status|on|off|soft|cycle|reset
ipmitool $BMC chassis identify 300     # light the blue ID LED, 300s
ipmitool $BMC sel elist                # hardware event log — run this first
ipmitool $BMC sdr list                 # all sensors
ipmitool $BMC lan print 1              # BMC IP/MAC/VLAN, cipher suites
ipmitool $BMC sol activate             # serial over LAN  (escape: ~. )
ipmitool $BMC chassis bootdev pxe      # override next boot
ipmitool $BMC mc reset cold            # reboot the BMC, NOT the server

# Redfish equivalent
curl -sk -u "admin:$PW" https://$BMC/redfish/v1/Systems/1 | jq .PowerState
curl -sk -u "admin:$PW" -X POST -H 'Content-Type: application/json' \
  -d '{"ResetType":"ForceRestart"}' \
  https://$BMC/redfish/v1/Systems/1/Actions/ComputerSystem.Reset

Serial console, set up in advance:

# /etc/default/grub
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0"
# then
update-grub && systemctl enable --now serial-getty@ttyS0.service

The reboot ladder:

1. ssh host reboot — needs a working OS.
2. ipmitool ... chassis power cycle — needs a reachable BMC. Does not remove standby power.
3. Switched PDU, both outlets off, 30 seconds — removes standby power. Clears a wedged BMC.
4. Remote hands ticket.

Security — non-negotiable: dedicated management VLAN, no internet route, reach via jump host or VPN. Disable cipher suite 0 (no authentication at all). IPMI 2.0's RAKP exchange leaks password hashes pre-authentication by design, so an exposed BMC is an offline cracking target. Change default credentials. Prefer Redfish; disable IPMI-over-LAN if unused. Patch BMC firmware.

Dedicated management port beats shared/NC-SI. Shared puts your recovery path in the same failure domain as the thing that breaks.

Network placement

• ToR — switch per cabinet, short DAC to servers, fibre uplinks out. The default.
• EoR / MoR — chassis at the row end or middle, long structured cabling. Better when cabinets are sparse or you want intra-row traffic on one switch.
• Leaf-spine — every leaf to every spine, no leaf-leaf, no spine-spine. Every server three hops from every other. Scales by adding spines.
• Oversubscription = downstream ÷ upstream. 48×25G ÷ 8×100G = 1.5:1. 3:1 is common. Know yours and graph the uplinks — that is where unexplained slowness lives.
• Two ToR switches per cabinet, plus a cheap 1 Gb management switch for BMCs.
• Check switch airflow direction when ordering — port-side intake vs exhaust. The wrong one inhales its neighbour's exhaust.

ethtool eno1 | grep -E 'Speed|Duplex|Link detected'
cat /proc/net/bonding/bond0
lldpctl                 # which switch, which port — learn this one

Connectivity

The chain: your router → demarc panel → cross-connect → MMR → carrier / cloud / IX.

• A cross-connect is a cable, not a service. Install fee plus monthly rent. Takes days.
• Order it with an LOA-CFA from the far end, naming cabinet, panel and port. They expire.
• Direct Connect: dedicated = a whole port (AWS: 1/10/100/400 Gbps), you order the cross-connect. Hosted = a partner's slice (AWS: 50 Mbps–25 Gbps), no cross-connect for you.
• Transit = reach everything, paid. Peering = direct to one network, usually free, only their customers. IX = one cross-connect, hundreds of peering sessions.
• 95th percentile billing: five-minute samples, top 5 % discarded, pay on the highest remainder. Roughly the top 36 hours a month are free — bursts are free, sustained growth is not. A commit is a minimum you pay regardless; ~10 % of port capacity is a common shape.
• "Redundant" means physically diverse. Ask what the two paths share: panel, room, building, conduit.

The remote hands ticket

SUBJECT: [action] on [device] at [cabinet] / [U]

LOCATION   facility · cage · cabinet · U position · FRONT or REAR
           · ID LED lit?
DEVICE     hostname · make/model · service tag · where the label is
STEPS      numbered · every port and outlet named · nothing assumed
DO NOT     what must not be touched
VERIFY     how you will confirm
IF DIFFERENT   stop and call me — explicit permission to stop
PHOTOS     rear cabling · both PDU outlets · front of rack
URGENCY    production impact · target time

Rules that do the work:

• Say FRONT or REAR every time. Left and right are as viewed from the rear.
• Light the ID LED before filing. It removes identity ambiguity for free.
• Both cords, different PDUs — state it twice on any install.
• Distinguish the BMC port from the data ports explicitly.
• Label both ends with the far end's identity.
• Ask for photos. Always. It is your only durable record.
• Never ask them to diagnose, and never ask them to log in.
• Verify the result, not the report — lldpctl, link counters, DOM levels, sel elist, cabinet power draw.

Notify the facility of inbound shipments in advance with tracking numbers and cage marking, and say what should happen to the packaging.

The five facts about your own cage

If you can answer these without looking, you have closed the gap this series was written for.

1. Cabinet identifier — room, row, position — and whether U1 is top or bottom.
2. Free U, and usable post-to-post depth.
3. Power: how many feeds, what rating, what you are actually drawing.
4. Uplinks: what the ToR connects to, spare ports, oversubscription ratio.
5. Cross-connects: how many leave the cage, where each terminates, and what your "redundant" pair physically shares.

Where to verify this

Standards and vendor documentation beat blog posts, including this one. The primary sources behind this series:

• EIA-310 / rack dimensions — rack unit reference, 19-inch rack standards
• Rack power and A/B feeds — ServerCentral on sizing rack power, Belden on dual power feeds
• DAC / AOC reach — FS 100G DAC/AOC, DAC selection guide
• Fibre grades and reach — multimode fibre types, OM3 vs OM4 distance limits
• BMC / IPMI — out-of-band management overview, dedicated vs shared IPMI port
• BMC security — Rapid7's penetration tester's guide to IPMI, runZero BMC research, internet-exposed BMCs leaking hashes
• Switch placement and fabric — ToR vs EoR vs MoR, Arista EoR/MoR white paper
• MMR, demarc, cross-connect — meet-me room explained, cross-connect guide
• LOA-CFA and Direct Connect — AWS: requesting cross connects at Direct Connect locations, AWS: download the LOA-CFA, AWS Direct Connect layer 1 explained
• Transit, peering, IX, 95th percentile — Equinix on peering vs transit, Kentik on transit vs peering, 95th percentile explained
• Labelling and structured cabling — TIA-606 labelling guide
• Remote hands vs smart hands — datacenters.com comparison, TechTarget definition

Numbers vary by vendor and by facility contract. Where a figure here matters to a purchase or a design, confirm it against the specific part and the specific site.

That is the series. If it worked, the next facility email you get should read as sentences rather than acronyms — and the next ticket you write should come back done, first time.

Every ambiguity you leave becomes a phone call, a delay, or the wrong cable being unplugged. This part is how to not do that.

The one principle

Write instructions that can be followed by someone competent who knows nothing about your systems.

That is the whole discipline. Not dumbed down — the technician is skilled, often far more skilled with hardware than you are. But context-free. They know racks, cables, screws and connectors. They do not know that srv-web-11 is the one that matters, or that "the storage box" means the 4U at the bottom.

The two failure modes that follow from ignoring this are worth naming, because they are the ones that actually happen. Ambiguity, where the instruction is unclear and the technician either guesses or calls you. And under-specification, where you assume a step is obvious and it is not, so the job comes back half-done.

The ticket template

Every remote hands request should have these sections. Not all of them will have content every time, but the headings stop you forgetting one.

SUBJECT: [action] on [device] at [cabinet] / [U position]

LOCATION
  Facility:      DC1, 123 Example Road
  Cage / suite:  Cage 14
  Cabinet:       ROW-C RACK-07   (label on door: "C07")
  U position:    U37   (numbered from the bottom; U1 is the lowest)
  Front or rear: REAR
  Identifying:   ID LED is lit and flashing blue, front and rear

DEVICE
  Hostname:      srv-web-11
  Make/model:    Dell PowerEdge R650, 1U
  Service tag:   ABC1234
  Asset label:   on the pull-out tab, front right

WHAT I NEED DONE
  1. ...
  2. ...
  3. ...

WHAT I DO NOT WANT TOUCHED
  ...

HOW I WILL VERIFY
  ...

IF SOMETHING IS DIFFERENT FROM THIS DESCRIPTION
  Stop and contact me. Do not improvise. I am on <phone> until <time>.

ESCALATION / URGENCY
  Production impact: yes/no. Target: within N hours.

Four of those sections do disproportionate work.

Front or rear. Say it explicitly, every time. Almost all work is at the rear; almost all descriptions unconsciously assume the front. And when you say left or right, say as viewed from the rear — a rear-facing observer's left is the front-facing observer's right, and this exact confusion unplugs the wrong power cord.

The ID LED. From Part 4: light it before you file the ticket. ipmitool ... chassis identify 300 turns "the third one down" into "the one flashing blue". If the BMC is unreachable, fall back on the service tag or asset label, and say where the label physically is.

What I do not want touched. The most valuable line in the ticket, and the one people leave out. "Do not disturb any other cable in this cabinet. If a cable must be moved to gain access, photograph it first and put it back exactly as found."

Stop and contact me. Explicitly give permission to stop. Without it, a conscientious technician facing a situation that does not match your description will try to make it work, and that is where damage happens.

Standard job: rack and cable a new server

This is the big one and it is a smart hands job, not remote hands — it needs judgement and hardware skill. Everything in Parts 1 through 5 shows up here.

SUBJECT: Install and cable 1x Dell R650 in ROW-C RACK-07 at U20-U20

LOCATION
  Cage 14, ROW-C RACK-07, U20 (1U). Numbered from the bottom.

HARDWARE
  Shipment: tracking 1Z999..., delivered <date>, marked "ATTN: Cage 14"
  Contents: 1x Dell PowerEdge R650 (1U) + ReadyRails sliding rail kit
            2x C13-C14 power cords (1x blue, 1x red)
            2x 2m SFP28 passive DAC
            1x 2m Cat6A patch lead (grey)
  Service tag on the box: ABC1234

MECHANICAL
  1. Install the sliding rails at U20, front and rear posts.
     Square-hole rack; cage nuts as required by the kit.
  2. Rack the server at U20. Confirm the front bezel sits flush and the
     rails travel freely through their full extension.
  3. Fit the cable management arm supplied with the kit.
  4. Leave a service loop on all rear cables so the server can be pulled
     fully forward without unplugging anything.

POWER   (two cords — they must go to DIFFERENT PDUs)
  5. Blue cord: PSU1 -> PDU-A, outlet A14.
  6. Red cord:  PSU2 -> PDU-B, outlet B14.
     ** Both cords must NOT be in the same PDU. **
  7. Engage the retention clips on both outlets.

DATA
  8. DAC 1: server NIC port 1 (left port as viewed from the REAR)
            -> switch sw-c07-01, port Et14
  9. DAC 2: server NIC port 2 (right port as viewed from the REAR)
            -> switch sw-c07-02, port Et14

MANAGEMENT
 10. Cat6A grey: the RJ45 labelled "iDRAC" on the server rear
            -> switch mgmt-sw-01, port 14
     This is the small dedicated port, NOT one of the four LOM ports.

LABELLING
 11. Label both ends of every cable with the far end, e.g.
     server end of DAC1:  "-> sw-c07-01 Et14"
     switch end of DAC1:  "-> srv-web-14 p1"

POWER ON
 12. Power the server on at the front button. Confirm both PSU LEDs are
     green and the front status LED is not amber.

PHOTOS PLEASE
 13. Rear of the server showing all five cables seated.
 14. PDU-A outlet A14 and PDU-B outlet B14, labels visible.
 15. Front of the rack showing the server at U20 with the bezel on.

DO NOT
  Do not disturb any other equipment or cabling in this cabinet.
  If anything does not match the above, stop and call me.

The things that make that ticket work: every port has a name, every cable has a colour, the A/B constraint is stated twice, the management port is distinguished from the data ports explicitly (Part 4's dedicated-versus-shared distinction, expressed as a physical instruction), and the photos let you verify all of it without trusting anybody's memory.

Ask for photos on every install. It costs the technician thirty seconds and it is the only durable record of how the cabinet was left.

Standard job: patch a port

Short, but the same rules apply.

SUBJECT: Patch panel PP-01 port 14 -> switch sw-c07-01 port Et22

Cage 14, ROW-C RACK-07. REAR of cabinet.

  1. Take a 1m LC-LC duplex singlemode (yellow) patch lead from the
     spares bin in this cabinet.
  2. Connect patch panel PP-01, port 14  ->  switch sw-c07-01, port Et22.
  3. Clean both ends before insertion.
  4. Label both ends.
  5. Photograph the panel and the switch port.

Do not remove or move any existing lead. If port 14 is already occupied,
STOP and tell me what is in it — do not unplug it.

That last paragraph exists because "patch port 14" and "port 14 already has something in it" is a very common collision, and without an instruction the technician has to guess whether the existing lead is stale or live.

The fibre-cleaning line is not fussiness. From Part 3: a fingerprint on a ferrule is enough to kill a 100 Gb link.

Standard job: reseat

The most common ticket there is, and the least likely to go wrong — provided you say which end.

SUBJECT: Reseat DAC on srv-web-11 NIC port 1

Cage 14, ROW-C RACK-07, U37. REAR. ID LED is lit (flashing blue).

  1. Locate the DAC in server NIC port 1 (left as viewed from the REAR).
     It should be labelled "-> sw-c07-01 Et14".
  2. Unplug it at the SERVER end only. Wait 10 seconds. Plug it back in
     firmly until it clicks.
  3. Tell me the link LED state on the server port before and after.
  4. If there is still no link, repeat at the SWITCH end (sw-c07-01,
     port Et14) and tell me again.

I am watching the switch port; I will confirm within 30 seconds of each step.

Note the sequencing — one end at a time, with a report between. If you ask for both ends at once and it comes back working, you have learned nothing about which end was faulty.

Standard job: power cycle at the PDU

This is rung 3 of the ladder from Part 4, and it has a specific trap.

SUBJECT: Full power cycle of srv-db-03 at the PDU (BMC unresponsive)

Cage 14, ROW-C RACK-07, U12. REAR.
The BMC is not answering, so I cannot do this myself and the ID LED
is NOT lit. Identify by the service tag XYZ5678 on the front pull-tab.

This server is DUAL-CORDED. BOTH cords must be removed for this to work.

  1. Confirm the two cords for this server: PDU-A outlet A06 (blue)
     and PDU-B outlet B06 (red). Both are labelled "srv-db-03".
  2. Unplug BOTH cords.
  3. Wait 30 seconds. (Less than this and standby power does not drain,
     which is the whole point of the exercise.)
  4. Plug both back in, blue to A06 and red to B06.
  5. Confirm both PSU LEDs come up green.

Do not use the front power button. I need standby power removed.

Two things there are the whole reason this ticket exists in this form. Both cords, because removing one from a redundant machine changes nothing. And thirty seconds, because the objective is draining standby power to clear the BMC, and a quick out-and-in does not do it.

Standard job: read me the state

The cheapest and most underused request. When you have no out-of-band access at all, a technician's eyes are a diagnostic instrument.

SUBJECT: Report physical status of srv-db-03

Cage 14, ROW-C RACK-07, U12. Service tag XYZ5678.

Please report, front and rear:
  1. Front status LED: off / green / flashing amber / solid amber?
  2. Any text on the front LCD panel? Photograph it if present.
  3. Both PSU LEDs at the rear: green / amber / off? Which is which?
  4. Both NIC link LEDs: lit? flashing? dark?
  5. Are the fans audibly spinning?
  6. Photograph the front and the rear.

No action needed — I only need the observations.

An amber PSU LED and a photographed LCD reading a fault code will frequently tell you exactly what is wrong before you have opened a hardware support case, and it costs one short ticket.

What not to ask for

Three categories, and knowing them saves you a rejected ticket and some embarrassment.

Do not ask them to diagnose. "Can you see why it's not working?" is not a task. Give them observations to make and steps to take. They do not know what "working" looks like for your system.

Do not ask them to touch your data or your configuration. They do not log into your servers. If a job needs a login, that is your side of the demarc — figuratively as well as literally. Where a technician does need console access during, say, a firmware recovery, that is arranged explicitly with credentials scoped to the task, not assumed.

Do not ask for anything ambiguous about identity. "The server that's broken", "the new one", "the third from the top". Use the U position, the hostname, the service tag, and the ID LED.

And one thing to be careful with rather than avoid: anything that risks other equipment. Moving a bundle to get access, removing a blanking panel, sliding a machine forward on rails in a densely cabled cabinet. Those are fine to ask for, but say explicitly what may be disturbed and ask for before-and-after photographs.

Shipping hardware in

A related process that goes wrong more often than the technical work.

Most facilities require advance notice of an inbound shipment, with a ticket giving the tracking number, the expected date, the number of packages and the destination cage. Without it, the loading dock may refuse delivery or the boxes may sit in a receiving area nobody told you about.

Mark the boxes with your company name and cage number, not just an address. Say in the ticket whether you want them unboxed and staged, or left sealed. And ask them to record the service tags of what arrived and photograph any physically damaged packaging before it is opened — that is the evidence you need if a machine turns out to be dead on arrival.

Storage of packaging and empty boxes is usually charged or prohibited. Say what should happen to the cardboard.

Verify everything

Every ticket ends with your verification, and the point is to check the result, not the report.

• After a rack and cable: BMC pings on the management VLAN, lldpctl shows the expected switch and port on both NICs, both PSUs report present and drawing power, and the photographs show A and B outlets.
• After a patch: the link comes up and optical Rx power is in range. From Part 4 — if you are graphing DOM, you can see the level, not just the up/down.
• After a reseat: the switch port counter stops incrementing errors, not merely that the link is up.
• After a power cycle: the machine boots and ipmitool ... sel elist shows the power event where you expect it.

And after any physical work in a cabinet, glance at the cabinet's total power draw on the PDUs. An unexpected change means something was plugged in or unplugged that you did not ask about.

The one-line summary

Write for a skilled person who knows nothing about your systems. Cabinet, U position, front or rear, and the ID LED. Name every port and every outlet. State the A/B constraint explicitly. Say what must not be touched, give explicit permission to stop, ask for photographs, and verify the result yourself rather than trusting the report.

Next up: Part 8 — the cheat sheet, which is every number, connector and command in this series on one page.

The chain, end to end

Almost all confusion here dissolves once you can draw the path. It is four links.

  ┌── YOUR CAGE ───────────────────────┐
  │                                     │
  │  server ── ToR switch ── your edge  │
  │                          router     │
  │                             │       │
  │                        ┌────┴────┐  │
  │                        │ DEMARC  │  │  ← the boundary of responsibility.
  │                        │ PANEL   │  │    Your side / their side.
  │                        └────┬────┘  │
  └─────────────────────────────┼───────┘
                                │
                    ╔═══════════╧═══════════╗
                    ║      CROSS-CONNECT     ║  ← a physical cable the
                    ║  (a cable, not a       ║    facility installs and
                    ║   service)             ║    charges rent on.
                    ╚═══════════╤═══════════╝
                                │
  ┌── MEET-ME ROOM (MMR) ───────┼──────────────────────────────┐
  │  restricted room. Every carrier, cloud on-ramp and         │
  │  tenant terminates here on fibre distribution frames.      │
  │                                                             │
  │   [carrier A]  [carrier B]  [AWS]  [Azure]  [the IX]  ...  │
  └─────────────────────────────────────────────────────────────┘

Your responsibility ends at the demarc. The cross-connect is the facility's to install and maintain. The far end belongs to whoever you connected to.

Demarc

The demarcation point is a boundary of responsibility, expressed as a physical object — normally a patch panel in or beside your cage.

The reason it matters operationally: when a circuit is down, the very first thing that happens is an argument about which side the fault is on, and every conversation you have with the facility and the carrier will be framed by it. "Is the light present at the demarc?" is the question that resolves it. If light is arriving at your panel and your router sees nothing, it is yours. If no light is arriving, it is theirs.

This is why "please check for light at the demarc panel and read me the port label" is one of the most valuable remote hands requests in existence. It is a five-minute task that assigns ownership of an outage.

Meet-Me Room

The MMR is the building's switchboard: a secured room containing fibre distribution frames, patch panels and optical termination equipment, where every carrier, cloud provider and tenant in the facility lands their network.

You do not get access. Nobody gets casual access — the MMR is typically the most access-controlled space in the building, because it contains everyone's connectivity at once.

The property that makes a facility valuable is the contents of its MMR. A carrier-neutral facility is one that lets any carrier terminate there and does not force you to buy connectivity from the landlord. The number and identity of carriers, clouds and exchanges present in the MMR is, in practice, the main thing that distinguishes one datacenter from another once power and cooling are adequate. When someone says a facility is "well connected", that is what they mean.

Cross-connect

A cross-connect is a physical cable installed by the facility between two termination points in the building — nearly always from your demarc panel to some other party's termination in the MMR.

The things to internalise:

It is a cable, not a circuit and not a service. There is no bandwidth associated with it. It carries whatever the two ends agree to run over it. Ordering a cross-connect and expecting connectivity is like ordering a patch lead and expecting an ISP.

It has an install fee and a recurring monthly rental. Cross-connect rent is a meaningful line item, and it is why consolidating onto fewer, bigger connections is a real cost lever.

It takes days, not minutes. A human has to route and patch it. Plan lead time into any project that needs one, and plan more if the facility is busy.

It comes in flavours. Singlemode fibre is the default for anything modern. Multimode and copper exist, usually for legacy hand-offs. Specify which you need, because the far end dictates it.

A and Z. The two ends of a circuit are the A-side and the Z-side. This is unrelated to the A/B of power from Part 2 — same letters, different domain. Context disambiguates: power has A and B, circuits have A and Z.

LOA-CFA, the document that makes it happen

To order a cross-connect, you supply a Letter of Authorization / Connecting Facility Assignment.

The other party issues it. It says, in effect: this customer is authorised to connect to us, and here is the exact cabinet, patch panel and port to land it on. The facility will not patch anything without it, because they have no other proof the far end consented to being connected to.

A good LOA-CFA identifies the party, the location, the cabinet, the panel and the port. That precision is the point — a technician in the MMR needs to find one specific port among thousands.

Two operational details worth knowing. They expire. AWS, for example, issues an LOA-CFA for a Direct Connect port and the authority lapses if the cross-connect is not completed in time; you re-download it from the console to renew. And you will normally need two: one from the far end for their side, and often one from yourself authorising the facility to land it on your panel.

The realistic sequence for a cloud on-ramp:

1. Request a port from the provider in their console. They allocate it and issue an LOA-CFA.
2. Download the LOA-CFA.
3. Open a cross-connect order with the facility, attaching the LOA-CFA, specifying your cabinet, your panel, your port, and the media type.
4. Facility schedules a technician. Days pass.
5. Cross-connect is installed. You confirm light and the provider confirms their side.
6. Configure the logical layer — VLAN, BGP session, addressing.

Steps 1 to 5 are paperwork and waiting. Step 6 is the part that feels like engineering and takes twenty minutes. Budget accordingly, and start the paperwork long before you think you need to.

Direct connect to a cloud

A direct connection to a cloud provider is a cross-connect whose far end is the provider's router rather than a carrier's. The brand names: AWS Direct Connect, Azure ExpressRoute, Google Cloud Interconnect.

The benefit is that traffic to that cloud leaves your cabinet, crosses the MMR on your fibre, and lands on the provider without traversing the public internet. You get predictable latency, no exposure to internet-path weather, generally lower egress pricing, and a private path for things you would rather not route publicly.

Two purchasing models, and the difference matters:

Dedicated. An entire physical port that is yours. In AWS's case 1, 10, 100 or 400 Gbps. You order the cross-connect yourself, using the LOA-CFA process above. You get full control, the option of MACsec encryption on your own port, and the highest availability commitments. You also need to be physically present in a facility where the provider has a presence.

Hosted. A logical slice of a partner's existing port, sold to you in much smaller increments — AWS hosted connections run from 50 Mbps up to 25 Gbps. The partner already has the cross-connect, so you order none. It is faster to provision, available at sizes a dedicated port cannot offer, and it is the right answer if you need 500 Mbps rather than 10 Gbps.

Choose dedicated if you need the bandwidth, the control or the encryption. Choose hosted if you need a modest amount of private connectivity quickly, or if you are not in the right building.

Redundancy here is a physical question, not a logical one. Two connections that share a cross-connect, a panel, a room or a building are not redundant. If the direct connection matters, the second one should land on a different provider device, ideally in a different facility, and you should be able to say which physical elements the two paths share. Ask specifically; "we have two connections" is not an answer.

Getting to the internet: transit, peering, IX

Three distinct things, frequently muddled.

IP transit is the commercial service of "reach the entire internet through me". You buy it from a carrier, you announce your prefixes to them over BGP, and they carry your traffic anywhere. It is the default and it is what most people mean by "our internet connection".

Peering is a direct BGP relationship with another network, over which you exchange only traffic destined for each other's customers. It is not a route to the internet — it is a shortcut to that specific network. Because both sides benefit, peering is settlement-free in the overwhelming majority of cases.

An Internet Exchange (IX) is shared infrastructure — a large layer-2 switching fabric in a colocation facility — where many networks connect once and then peer with many others. You take one cross-connect to the IX, and from that single port you can establish BGP sessions with hundreds of networks. That leverage is the entire value proposition.

   TRANSIT                     PEERING                 IX
   ───────                     ───────                 ──
   you ──► carrier ──► the     you ◄──► them           you ──┐
           (paid, reaches      (direct, usually        them ─┤ IX
            everywhere)         free, only their       more ─┤ fabric
                                own customers)         more ─┘
                                                       one cross-connect,
                                                       many BGP sessions

In practice a mature network buys transit from two carriers for full reachability and resilience, and peers at an IX to move the large, cheap, latency-sensitive volume — which is why an IX presence pays for itself once your traffic is big enough.

Billing is worth understanding because it is unlike anything else you buy. IP transit is normally billed on the 95th percentile: usage is sampled at five-minute intervals across the month, the samples are sorted, the top 5 % are discarded, and you pay on the highest remaining sample. Roughly the top 36 hours of every month are free.

The consequences are genuinely useful. Short bursts are free — a release, a backup window, a traffic spike, provided it stays under about 36 hours in aggregate. Sustained increases cost immediately. And a commit is a minimum monthly volume you agree to pay for regardless of use, in exchange for a lower per-megabit rate; a common shape is a commit around 10 % of port capacity. So a large port with a small commit is a legitimate way to buy burst headroom cheaply, and scheduling bulk transfers so they do not extend your 95th percentile is a real and effective cost optimisation.

What to know about your own connectivity

Five questions. If you cannot answer them, that is your homework, and every one of them is answerable by asking rather than by touching anything.

1. How many cross-connects leave your cage, and where does each one terminate?
2. For each, which physical panel and port is it on at your end? This is what you quote in a ticket.
3. Which are transit, which are peering, which are cloud on-ramps?
4. What do your two "redundant" paths physically share? Same panel? Same room? Same building? Same conduit into the street?
5. What is your 95th percentile, and what is your commit?

Question 4 is the one that catches people. Diversity on paper is common; diversity in conduit is rare and expensive; the difference only becomes visible when a digger finds the one duct both circuits happened to use.

The one-line summary

Demarc is where your responsibility ends. The MMR is the building's switchboard and its tenant list is what makes a facility valuable. A cross-connect is a rented physical cable, ordered with an LOA-CFA from the far end, and it takes days. Direct connect is a cross-connect to a cloud — dedicated for a whole port, hosted for a slice. Transit reaches everywhere and is billed at the 95th percentile; peering is a free shortcut to one network; an IX is one cross-connect that gets you hundreds of them.

Next up: Part 7 — directing remote hands, which is where all six previous parts turn into a ticket that works the first time.

This part is where switches physically sit, why that choice was made, and what it means for the cabling in your cabinet.

The three placements

There are three ways to answer "where does the server plug in", and they are named after where the switch is.

Top of Rack (ToR). A switch in each cabinet, usually at the top, serving only the servers in that cabinet. Server-to-switch links are short DAC runs inside the cabinet. The switch's uplinks leave the cabinet on fibre.

End of Row (EoR). One or two large chassis switches in a dedicated cabinet at the end of a row, serving every server in that row. Every server has a long cable running from its cabinet, through overhead or underfloor trays, to that cabinet.

Middle of Row (MoR). Identical to EoR but positioned in the centre of the row, which halves the worst-case cable length.

  TOP OF RACK                      END OF ROW
  ───────────                      ──────────
  ┌────┐ ┌────┐ ┌────┐             ┌────┐ ┌────┐ ┌────┐ ┌──────┐
  │ SW │ │ SW │ │ SW │             │    │ │    │ │    │ │ ████ │  ← chassis
  ├────┤ ├────┤ ├────┤             │srv │ │srv │ │srv │ │ ████ │    switches
  │srv │ │srv │ │srv │             │srv │ │srv │ │srv │ │ ████ │
  │srv │ │srv │ │srv │             │srv │ │srv │ │srv │ │ ████ │
  │srv │ │srv │ │srv │             │srv │ │srv │ │srv │ │      │
  └─┬──┘ └─┬──┘ └─┬──┘             └─┬──┘ └─┬──┘ └─┬──┘ └───▲──┘
    │      │      │                  └──────┴──────┴────────┘
    └──────┴──────┴─► spine          long structured cabling runs
    short DAC inside                 (Cat6A or fibre, through trays,
    each cabinet; only               via patch panels at both ends)
    fibre uplinks leave

ToR won, for most people, for practical rather than architectural reasons. Cabling inside a cabinet is cheap, short, and done once when the cabinet is built. Cabling from every cabinet to a row-end chassis is expensive, has to be planned before anything is installed, and fills trays. ToR also means a new cabinet is a self-contained unit you can build and test in isolation.

EoR and MoR still make sense in three cases: when you have too few servers per cabinet for a whole switch to be worthwhile, when you want server-to-server traffic within a row to stay inside one switch (which removes a hop and can remove oversubscription entirely), and when you have standardised on structured Cat6A cabling and 10GBASE-T. A single big switch is also one thing to manage rather than forty.

A note on the name: a "top of rack" switch is not required to be at the top. Placing it at the top is the convention because that is where cable trays and vertical managers converge, but you will see them mid-cabinet in high-density layouts to halve the average cable run to servers above and below.

What a ToR cabinet actually contains

Working down from the top of a typical cabinet:

  U42  ░ blanking ░
  U41  patch panel   ← fibre to the MMR / cross-connects (Part 6)
  U40  sw-01         ← ToR switch A   48x25G SFP28 + 8x100G QSFP28
  U39  sw-02         ← ToR switch B   (the redundant pair)
  U38  mgmt-sw-01    ← 1G copper switch for BMC / out-of-band (Part 4)
  U37  ░ blanking ░
  U36  srv-01
  U35  srv-02
   ..  ...

Four things in that stack are worth naming.

Two ToR switches, not one. Each server has one link to each switch. Lose a switch — or reboot one for a firmware upgrade — and every server stays up on the other. A cabinet with a single ToR switch has a single point of failure that takes out forty machines at once, which is a decision you can make deliberately for a stateless tier and should never make by accident.

A separate management switch. Cheap, 1 Gb, copper, carrying only BMC and console traffic on its own VLAN. It exists so that your out-of-band path does not depend on the production switches you might be trying to fix. This is Part 4's argument expressed in hardware.

A patch panel. Structured fibre terminates here rather than plugging directly into the switch, so that when a switch is replaced the building's cabling is not disturbed. Anything leaving the cabinet permanently should pass through it.

Blanking panels above and below the active kit, for the airflow reason from Part 1.

One airflow detail specific to switches, because it bites people: switches are built in two airflow directions. Port-side exhaust (also called front-to-back relative to the ports) and port-side intake (back-to-front). Since a switch's ports face the rear of the cabinet where the servers' cables are, but the cold aisle is at the front, you need the variant that pulls cold air from the front and exhausts out the rear like the servers do. Order the wrong one and you have installed a device that inhales its neighbours' exhaust. Vendors sell both and the part numbers differ by a suffix.

Leaf and spine

Above the cabinet, modern datacenter networks are almost universally leaf-spine, also called a Clos fabric or a fat tree.

Every leaf — in practice, every ToR switch — connects to every spine switch. Leaves never connect to each other. Spines never connect to each other. Servers connect only to leaves.

            ┌────────┐   ┌────────┐   ┌────────┐   ┌────────┐
   SPINE    │  sp-1  │   │  sp-2  │   │  sp-3  │   │  sp-4  │
            └─┬─┬─┬─┬┘   └┬─┬─┬─┬─┘   └┬─┬─┬─┬─┘   └┬─┬─┬─┬─┘
              │ │ │ └──────┼─┼─┼─┼──────┼─┼─┼─┼──────┘ │ │ │
              │ │ └────────┼─┼─┼─┼──────┘ │ │ │        │ │ │
              ... every leaf connects to every spine ...
            ┌─┴─────┐   ┌──┴────┐   ┌──┴────┐   ┌──┴────┐
   LEAF     │ leaf1 │   │ leaf2 │   │ leaf3 │   │ leaf4 │
            └───┬───┘   └───┬───┘   └───┬───┘   └───┬───┘
              servers     servers     servers     servers

The property that matters: every server is exactly the same number of hops from every other server — leaf, spine, leaf. Three hops, always, whether the destination is in the next cabinet or across the hall. Latency becomes predictable, and capacity scales by adding spines rather than by replacing a bigger box with an even bigger box.

This replaced the old three-tier access/aggregation/core design, which was built for traffic that mostly flowed north-south — in and out of the datacenter. Modern workloads are overwhelmingly east-west: service to service, node to node, replica to replica. Leaf-spine is the shape you build when most of your traffic never leaves the building.

Oversubscription, and the number to ask for

Your ToR switch has, say, 48 ports of 25 Gb facing the servers and 8 ports of 100 Gb facing the spines.

• Downstream: 48 × 25 = 1200 Gb/s
• Upstream: 8 × 100 = 800 Gb/s

That is an oversubscription ratio of 1200:800, or 1.5:1. If every server in the cabinet transmitted at full line rate simultaneously, the uplinks could not carry it.

This is deliberate and normal. Uplink optics are expensive, and it is vanishingly unlikely that forty-eight machines all saturate their NICs in the same millisecond. A 3:1 ratio is common and unremarkable; 2:1 is comfortable; 1:1 (non-blocking) is bought only for workloads that genuinely need it — distributed storage rebuilds, HPC, large-scale training.

Why you should care: oversubscription is where "the network is slow" comes from when every individual link looks healthy. Your NIC shows 25 Gb, the switch port shows no errors, and yet throughput collapses during the nightly backup — because sixteen other machines in the cabinet are also backing up and you are all sharing 800 Gb of uplink.

So: know your cabinet's oversubscription ratio, and know it as a number. It is a fair question to ask whoever runs the fabric, and if the answer is "we don't know", that is itself the finding. Then graph uplink utilisation, not just server NIC utilisation. The server-side graph will look fine right up until it does not.

How your server attaches

Two links, one to each ToR switch. What happens on top of them is a Linux question, and it is the one part of this series that lands squarely in your territory.

Active/backup bonding (mode=1) is the simple option. One link carries traffic, the other stands by. It needs nothing from the switches, works with any two switches anywhere, and costs you half your bandwidth.

LACP (mode=4, 802.3ad) uses both links at once. It requires both switch ports to be in the same link aggregation group — which, when the ports are on two different physical switches, requires those switches to pretend to be one. That is MLAG (multi-chassis link aggregation): Arista's MLAG, Cisco's vPC, and vendor equivalents. The pair keeps state in sync over a peer link and presents a single logical switch to your server.

# what your NIC actually negotiated — check this before blaming the fabric
ethtool eno1 | grep -E 'Speed|Duplex|Link detected'

# bond state, including which slave is active and whether LACP came up
cat /proc/net/bonding/bond0

# who is on the other end — LLDP is the single most useful command here
lldpctl                        # or: lldpcli show neighbors detail

Learn lldpctl. With LLDP running on the switches, it tells you the neighbour switch's name, the exact port your cable is in, and often the port description. That turns "which switch port is this server in" from a remote hands ticket with a torch into one command. When you are writing a ticket about a specific link, lldpctl output is the authoritative identifier to quote.

A third option worth knowing about: EVPN/VXLAN with a routed fabric, where the leaf is a layer-3 boundary and there is no MLAG at all — each server link is a separate routed path, and the host runs BGP. It is cleaner and increasingly common at scale, and it changes your side of the picture from bonding to routing.

Where oversubscription meets your cabling

One last practical link back to Part 3. The uplinks from a ToR switch to the spines are the runs that leave the cabinet, which means they are fibre, through the patch panel, and they are the reason your cabinet has a fibre panel at all. Server links stay inside the cabinet, which is why they are DAC.

So the cabling in a cabinet has a shape you can now predict entirely:

• Inside the cabinet, rear: forty short DAC cables, servers to the two ToR switches.
• Inside the cabinet, rear: forty short Cat6A leads, BMCs to the management switch.
• Leaving the cabinet: a handful of fibre uplinks through the patch panel, and two power whips.

Which means: most of what looks like a wall of cables is short, local and identical, and only a few strands actually go anywhere. That was the thing that made the rear of a cabinet stop looking like chaos to me.

The one-line summary

Top of rack won: short DAC inside the cabinet, fibre uplinks out, two switches per cabinet plus a cheap 1 Gb management switch. Above that, leaf-spine makes every server three hops from every other. Know your oversubscription ratio and graph your uplinks, because that is where unexplained slowness lives. And run LLDP, so you never have to ask a human which port a cable is in.

Next up: Part 6 — leaving the cage, where the demarc, the Meet-Me Room and the cross-connect finally connect to each other.

There is a second computer inside your server. It has its own CPU, its own firmware, its own network stack and its own IP address, it runs on standby power whenever the machine is plugged in, and it does not care whether the operating system is running, halted, or absent. Understanding it properly is the single highest-leverage thing in this series.

BMC, and its four brand names

The chip is a BMC — Baseboard Management Controller. In practice it is usually an ASPEED AST2500 or AST2600 soldered to the motherboard, running a small embedded Linux of its own.

Every vendor brands it differently, which is why one thing has four names:

They are the same category of thing. When someone says iDRAC and you run HPE, mentally substitute iLO and carry on.

IPMI is the old wire protocol for talking to a BMC — a UDP-based standard from the late nineties, at version 2.0 since 2004. Redfish is its modern replacement: HTTPS, JSON, REST, sane authentication. New tooling should use Redfish. Old tooling and every scrap of documentation on the internet uses ipmitool, so you need both.

What it can actually do

The list is longer than most people assume, and each item is a ticket you did not open.

• Power on, power off, hard reset, and graceful shutdown — independent of the OS.
• Serial-over-LAN: the machine's console, including the bootloader and kernel messages, over the network.
• Virtual media: mount an ISO from your laptop as if it were a physical drive, and reinstall the OS.
• Boot device override: "next boot, come up on PXE" without touching BIOS by hand.
• Full sensor telemetry: every temperature, fan RPM, voltage and PSU status in the chassis.
• The System Event Log (SEL): a hardware event history that survives OS reinstalls and often contains the actual cause of a mystery reboot.
• The identify LED: light a blue LED on the front and rear of the chassis on demand.

That last one deserves its own paragraph, because it is the best remote hands trick in this whole series. Instead of writing "the third server from the top, I think, the one with the blue sticker", you light the LED and write "the machine with the flashing blue ID LED, at U37". The technician cannot pick the wrong box. It converts an ambiguous instruction into an unambiguous one, for free.

# light it
ipmitool -I lanplus -H 10.20.0.37 -U admin -P "$PW" chassis identify 300   # 300 seconds
# turn it off again
ipmitool -I lanplus -H 10.20.0.37 -U admin -P "$PW" chassis identify 0

The commands worth knowing

ipmitool is in every distro's repos. -I lanplus selects IPMI 2.0 over the network and is required for essentially everything remote.

BMC="-I lanplus -H 10.20.0.37 -U admin -P $PW"

ipmitool $BMC chassis power status      # on or off
ipmitool $BMC chassis power on
ipmitool $BMC chassis power off         # abrupt — like holding the button
ipmitool $BMC chassis power soft        # ACPI graceful shutdown
ipmitool $BMC chassis power cycle       # off, pause, on
ipmitool $BMC chassis power reset       # hard reset, no power removal

ipmitool $BMC sel elist                 # hardware event log, human-readable
ipmitool $BMC sel clear

ipmitool $BMC sdr list                  # every sensor and its reading
ipmitool $BMC sensor list

ipmitool $BMC lan print 1               # the BMC's own IP, MAC, VLAN, gateway
ipmitool $BMC chassis bootdev pxe       # override next boot
ipmitool $BMC sol activate              # attach to the serial console
ipmitool $BMC mc reset cold             # reboot the BMC itself, not the server

Two of those are worth calling out. sel elist is the first thing to run after an unexplained reboot — a machine that vanished without a trace in the OS logs will frequently have a clean, dated entry in the SEL saying it overheated or lost a PSU. And mc reset cold reboots the BMC without touching the running server, which is the fix when the BMC's web UI has wedged but the workload is fine. That distinction matters: you can repair your management path without an outage.

To escape from sol activate, the sequence is ~. — the same escape as ssh. Which means if you are inside ssh and inside SOL, you need ~~. to leave only the inner one. This catches everyone once.

The Redfish equivalent, for anything new:

curl -sk -u "admin:$PW" https://10.20.0.37/redfish/v1/Systems/1 | jq .PowerState
curl -sk -u "admin:$PW" -X POST \
  -H 'Content-Type: application/json' \
  -d '{"ResetType":"ForceRestart"}' \
  https://10.20.0.37/redfish/v1/Systems/1/Actions/ComputerSystem.Reset

Dedicated versus shared: the cable question

This is the thing you asked about directly, and it is a real decision with a real consequence.

On the back of the server there is usually a small RJ45 physically separate from the data NICs, labelled BMC, IPMI, iDRAC, iLO or MGMT. That is the dedicated management port. It is almost always 1 Gb copper, and it is wired to a separate management switch.

The alternative is shared mode — also called LOM (LAN on Motherboard) or NC-SI, which is the sideband protocol that lets the BMC borrow one of the main NICs. One physical cable carries both your production traffic and your management traffic, usually separated by VLAN.

   DEDICATED                              SHARED / NC-SI
   ┌─────────────────────┐                ┌─────────────────────┐
   │ server              │                │ server              │
   │  ┌──────┐           │                │  ┌──────┐           │
   │  │ BMC  ├── mgmt ───┼──► mgmt switch │  │ BMC  ├──┐        │
   │  └──────┘   RJ45    │                │  └──────┘  │ NC-SI  │
   │  ┌──────┐           │                │  ┌──────┐  │        │
   │  │ NIC  ├── data ───┼──► ToR switch  │  │ NIC  │◄─┘        │
   │  └──────┘           │                │  └──┬───┘           │
   └─────────────────────┘                └─────┼───────────────┘
                                                └── data + mgmt ──► ToR
   two cables, two failure domains         one cable, one failure domain

The argument for dedicated is the entire point of out-of-band. If the BMC shares a NIC with the OS, then a failed NIC, a bad driver, a misconfigured bond, an unplugged cable or a switch port error-disabling takes out both your production path and the thing you were going to use to fix it. You have built out-of-band management that goes out of band at exactly the moment you need it.

The argument for shared is cost: half the cables, half the switch ports, less structured cabling. In a large fleet that is real money.

My view, stated as a view: use the dedicated port. The saving is one RJ45 patch lead and one port on a cheap 1 Gb switch, and what you buy with it is that your recovery path does not share a failure domain with the thing that breaks. That is the whole product. If budget forces shared mode, at least know that you have made that trade, and make sure something else — a serial console server, or a switched PDU — gives you an independent path.

There is a third setting on many boards, failover, where the BMC prefers the dedicated port and falls back to shared if the link drops. It sounds ideal and is occasionally surprising, because the BMC can silently move to a different network and a different address. If you use it, know it is on.

Never, ever expose a BMC

This is not a stylistic preference. It is the one hard security rule in this series.

A BMC is a privileged computer with full physical-equivalent control of your server, running vendor firmware that is patched slowly and often not at all, sitting underneath the operating system. An attacker who owns the BMC owns the machine in a way that survives an OS reinstall, and no host-based control can see them.

The protocol makes it worse. IPMI 2.0's RAKP key exchange hands out a password hash to anyone who asks for a valid username, before authentication. That is not a bug in an implementation — it is how the protocol was specified, and it means an exposed BMC is an offline cracking target. Separately, cipher suite 0 is an option that disables authentication entirely: supply a valid username, no password, get in. Historical scans found it enabled by default on a majority of internet-facing BMCs.

Research published this year found tens of thousands of BMCs still reachable from the internet, with a large fraction of them still handing over crackable hashes to unauthenticated requests. The problem has not aged out. It got bigger, because the fleet got bigger.

So, the rules:

1. BMCs live on a dedicated management VLAN with no route to the internet. Not a firewall rule on a shared VLAN. A separate segment.
2. Reach them through a jump host or a VPN, never directly.
3. Disable cipher suite 0. Check with ipmitool ... lan print 1 and look at the cipher suite list.
4. Change the default credentials. Supermicro's ADMIN/ADMIN and equivalents are the first thing anyone tries.
5. Prefer Redfish over IPMI where the hardware supports it, and turn IPMI-over-LAN off entirely if nothing needs it.
6. Patch BMC firmware. It is the piece of software in your estate least likely to have been updated and most likely to be catastrophic.

CISA's guidance on this is blunt and worth quoting in spirit: use a physically separate management network where you can, default-deny, and permit management access only from dedicated administrative systems.

The serial console, which still matters

SOL over IPMI is convenient but it depends on the BMC being alive and on the network. A console server (sometimes a terminal server) is the independent alternative: a box with 16, 32 or 48 serial ports, cabled to the serial port of every device in the row, reachable over the management network.

It is old technology and it is the thing that saves you when everything else is gone, because the serial console works when the network stack is dead, when the machine is stuck in the bootloader, and when a firmware update has bricked the BMC. It is also the only way to manage a lot of network equipment during an upgrade gone wrong.

Cabling vocabulary you will meet: many devices use RJ45 serial with the Cisco-standard rollover pinout, and the light blue flat cable is a near-universal visual convention. Others use DB9, and newer servers increasingly use USB or a micro-USB service port on the front bezel.

On the Linux side, none of this is exotic:

# make sure the kernel actually talks to the serial port — otherwise the
# console server shows you a blank screen and you blame the hardware
# /etc/default/grub:
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0"

# then
update-grub          # or grub2-mkconfig -o /boot/grub2/grub.cfg
systemctl enable --now serial-getty@ttyS0.service

That last pair is worth doing on every machine you own, in advance. A serial console you configured after the outage started is not a serial console.

Three different things "reboot it" can mean

This is the escalation ladder, and knowing which rung you are on stops you from asking for the wrong thing.

  1. OS-level         ssh host reboot
     ─────────        Clean. Needs a working OS and network.

  2. BMC-level        ipmitool ... chassis power cycle
     ──────────       Works when the OS is dead. Needs a reachable BMC.
                      NOTE: does NOT remove standby power. The BMC keeps
                      running, and so does anything else on standby.

  3. Power-level      switched PDU: outlet off, wait, outlet on
     ────────────     Works when the BMC is dead. Removes standby power
                      entirely, which is what actually clears a wedged BMC
                      or a confused PSU. Both cords for a dual-corded box.

  4. Human            remote hands ticket.
     ──────           Everything else has failed, or the fault is physical.

The gap between rung 2 and rung 3 is the one people miss. chassis power cycle does not de-energise the board — the BMC and standby rails stay up. When a machine is in a genuinely stuck state, or when the BMC itself has stopped answering, you need real power removal. That is what a switched PDU buys you, and it is why Part 2 argued for switched-and-monitored units. Without one, rung 3 is a ticket.

If you are cutting power at the PDU on a dual-corded server, both outlets must go off, and a real drain needs perhaps thirty seconds rather than two. Half a power-cycle on a redundant machine is a no-op that looks like a failed fix.

What to have in place before you need it

A short checklist. Each of these is boring to do in advance and impossible to do during an incident.

• Every BMC has a static or reserved address, is in DNS, and is in your documentation next to its server's name and U position.
• Every BMC is on the management VLAN, with default credentials changed and cipher 0 disabled.
• Serial console configured in GRUB and a getty enabled on every host.
• Switched PDU, with outlet-to-server mapping documented — including which outlet is A and which is B for each machine.
• BMC reachability is monitored. A BMC that died three months ago is discovered at the worst possible moment otherwise. It is an ICMP check and an authenticated Redfish poll; both are cheap.
• Optical Rx power and PSU status scraped into metrics, so degradation shows up as a trend rather than an outage.

The one-line summary

The BMC is a separate computer that can power, console and reinstall your server without the OS. Give it its own port, its own VLAN, and no path to the internet. Learn sel elist for post-mortems and chassis identify for tickets. Keep a switched PDU, because a BMC-level power cycle does not remove standby power and a wedged BMC does not answer.

Next up: Part 5 — where the switches live, and why your server's NIC probably terminates two feet above it.

The three ways to connect two ports

Every high-speed link in a datacenter is one of three physical things, and the choice is almost entirely about distance.

DAC — Direct Attach Copper. A fixed length of twinaxial copper with the transceiver moulded permanently onto both ends. One part number, one item, no separate cable. Cheapest, lowest power, lowest latency. Also the shortest.

AOC — Active Optical Cable. Same idea — ends permanently attached — but the middle is fibre and the ends contain lasers. Longer reach, more money, more power draw, and still a single sealed unit you cannot repair.

Transceiver plus fibre. Two removable modules plus a separate fibre patch lead. The most expensive per link and the most flexible: you can change the fibre without changing the optics, run through patch panels, and reach anything from metres to kilometres.

  DAC                 [==transceiver==]━━━━copper━━━━[==transceiver==]
                       (moulded on)                    (moulded on)
                       ← up to ~3-7 m passive, ~10 m active →

  AOC                 [==transceiver==]────fibre────[==transceiver==]
                       (moulded on)                   (moulded on)
                       ← up to ~100 m →

  Transceiver+fibre   [module] ◄─┐                 ┌─► [module]
                                 └── fibre lead ───┘
                       ← metres to tens of kilometres →
                        (and the only one that survives a patch panel)

How far each one goes

These are the numbers worth memorising, because they decide the answer nine times out of ten.

Passive DAC is typically good to about 3 m, with 5 m and sometimes 7 m available depending on speed and vendor. It contains no electronics at all — it is wire.

Active DAC puts signal conditioning in the ends and reaches roughly 5 to 10 m.

AOC reaches up to about 100 m over the multimode fibre inside it.

Notice how neatly that maps onto physical layout. Inside one cabinet, DAC. Between adjacent cabinets, active DAC or AOC. Across the room or between rooms, transceivers and structured fibre. The cable type is a proxy for "how far away is the other end", and if someone tells you the link type you can infer the topology.

The trap: DAC lengths are exact and unforgiving. You are buying a specific physical length with the ends attached. Order 1 m when you needed 2 m and it does not reach; order 5 m when you needed 2 m and you have three metres of stiff twinax to hide in a cabinet that has nowhere to put it. Twinax is thick and does not bend tightly. Measure the real cable path — down the rack, through the manager, back up — not the straight-line distance.

Form factors, which encode speed

The connector shape tells you the speed class. This is the whole naming system.

The Q stands for quad: a QSFP carries four lanes in the space of one SFP. That is the whole trick — QSFP28 is 100 Gb because it is four lanes of 25 Gb, and QSFP+ is 40 Gb because it is four lanes of 10 Gb.

Which leads directly to the useful consequence: breakout cables. Because a QSFP28 is physically four 25 Gb lanes, you can buy a cable with one QSFP28 on one end and four SFP28s on the other, and split a single 100 Gb switch port into four 25 Gb server ports. Written 1x100G → 4x25G, or 4x25G breakout. This is extremely common and it is how you get high port counts out of a switch that only has QSFP cages. The same trick gives you 1x40G → 4x10G.

The physical compatibility rule: an SFP-family module fits any SFP-family cage, and a QSFP-family module fits any QSFP-family cage — but a QSFP will not fit an SFP cage. They are different sizes. Speed negotiation within a family is a separate question and often does not happen automatically, so a 25 Gb module in a 10 Gb switch port may or may not work depending on the switch.

Copper RJ45, which still exists

Not everything is optics. 10GBASE-T runs 10 Gb over ordinary RJ45 on Cat6A to 100 m, and it is the reason some servers still have familiar-looking Ethernet ports.

It has real drawbacks in a datacenter: noticeably higher power per port, higher latency than DAC, and more heat. Which is why it lost the server-to-switch fight to SFP28 DAC. But it is still everywhere for exactly two jobs, and both matter to you:

• Out-of-band management. BMC ports are almost always 1 Gb RJ45. Part 4.
• Anything cheap or old — small switches, appliances, PDUs, console servers, environmental sensors.

Cable categories in the order you will meet them: Cat5e (1 Gb), Cat6 (10 Gb to 55 m), Cat6A (10 Gb to 100 m). In a cabinet, use Cat6A and stop thinking about it.

Fibre: the two families

If a link uses transceivers and separate fibre, the fibre is one of two fundamentally different things and they are not interchangeable.

Multimode (MMF) has a wide core — 50 microns — that lets light take multiple paths. Cheap optics, short distances. Datacenter grades are OM3, OM4 and OM5. Jacket colour is aqua for OM3 and OM4, lime green for OM5.

Singlemode (SMF) has a 9-micron core so light takes essentially one path. Expensive optics, enormous distances. The datacenter grade is OS2, and the jacket is yellow.

The colour of the jacket genuinely tells you the type, and this is one of the few places in the building where a visual convention is reliable enough to act on. Aqua is multimode. Yellow is singlemode. If someone sends you a photo of a patch panel, you can read the fibre type off it.

Reach, which is the whole reason the distinction exists:

Two things to take from that table. Multimode reach collapses as speed rises — the same OM4 fibre that carried 10 Gb for 400 m carries 100 Gb for 100 m. And singlemode does not care, which is why long runs, cross-connects and anything leaving the building are singlemode.

The transceiver naming follows: SR is short reach (multimode), LR is long reach (singlemode, 10 km), ER and ZR go further still. The trailing digit in SR4 means four lanes, which means it needs eight fibres rather than two.

Fibre connectors

LC is the small duplex connector — two fibres, one for transmit and one for receive, clipped together. It is what an SR or LR transceiver takes and it is the overwhelming majority of what you will see.

MPO (also branded MTP) is a rectangular multi-fibre connector carrying 12 or 24 fibres in one ferrule. It exists for the parallel optics — a 40GBASE-SR4 or 100GBASE-SR4 needs eight fibres, so it takes an MPO rather than an LC. MPO is also how high-density trunk cabling gets run between patch panels: one trunk, then a cassette that fans it out to LC.

SC is the older, larger square connector. Still found on carrier hand-offs and in the MMR.

Two things about fibre worth knowing even though you will never handle it. Polarity — transmit must land on receive at the far end, and with MPO trunks there are three standardised polarity methods (A, B and C) that a cabling team has to keep consistent; getting it wrong gives you a dark link with no obvious cause. And cleanliness — a fingerprint on a ferrule end-face is enough to kill a 100 Gb link, so fibre gets cleaned before every insertion and capped when unplugged. If a new fibre link is dark and everything on paper is right, "please clean and reseat both ends" is a legitimate and very common first remote hands request.

Transceiver coding, the annoying part

Transceivers carry a small EEPROM identifying the vendor. Some switches — Cisco historically being the notable one — refuse modules that do not identify as their own. Third-party vendors sell the same optics coded for a given switch brand, at a fraction of the price.

This is worth knowing for two reasons. When you order optics you must specify the switch they are going into, not just the speed. And if a link is dead and the switch log says something about an unsupported or unrecognised transceiver, that is a coding problem, not a hardware fault, and no amount of reseating will fix it.

On the good side, transceivers support DOM/DDM — Digital Optical Monitoring — which exposes real-time transmit power, receive power and temperature per module. That is how you tell a dirty connector from a dead laser from a fibre someone stood on, without going near the building. Your switch will show it. If you are not scraping optical Rx power into your metrics, you are missing the single best early warning of a degrading link.

Choosing, in one flowchart

   Both ends in the SAME cabinet?
        │
        ├── yes ──► DAC. Passive if ≤3 m, active up to ~10 m.
        │           Cheapest, coolest, lowest latency. Measure the real path.
        │
        └── no ──►  Does it pass through a patch panel?
                        │
                        ├── yes ──► Transceivers + fibre. Always.
                        │           (An AOC cannot be patched — the ends are moulded on.)
                        │
                        └── no ──►  Under ~100 m, point to point?
                                        │
                                        ├── yes ──► AOC is simple and cheap enough.
                                        │
                                        └── no ──►  Transceivers + singlemode (OS2, LR).

Cabling hygiene that is actually your problem

Most of this is done by people with more skill than me, but three of them are decisions you make, not them.

Order the right length, not the nearest length. Slack has to live somewhere, and in a rear cabinet channel there is nowhere for it to live. Cut-to-length or correctly-chosen patch leads are the difference between a cabinet you can work in and one where every change risks disturbing something else.

Leave a service loop on anything on sliding rails. Enough slack to pull the machine fully forward. Covered in Part 1; it belongs here too, because it is a cabling decision.

Label both ends, with the far end's identity on the label. → sw-c07-01 Et1/14 on the server end, → srv-web-11 eno1 on the switch end. No tracing, ever.

And one convention that costs nothing: use cable colour as a type signal. Whatever scheme you like — say, one colour for production data, one for out-of-band management, one for cross-connects — as long as it is written down and consistent. It makes a photograph from remote hands into something you can actually diagnose from.

The one-line summary

Same cabinet, DAC. Adjacent cabinets, AOC. Through a panel or across the room, transceivers and fibre — aqua multimode for short, yellow singlemode for long. Q means four lanes, which is why 100 Gb breaks out into four 25 Gb. Order by exact length and exact switch model, and label both ends with the far end.

Next up: Part 4 — out-of-band access, which is the difference between fixing a machine from your laptop and filing a ticket.

Power is the part of the datacenter that punishes vagueness hardest, because the failure mode is not a warning in a dashboard — it is a breaker opening and half a cabinet going dark instantly. This part is the arithmetic, the connectors, and the two mistakes that account for most cabinet-level outages.

A-side and B-side, properly

Your cabinet receives two feeds. They are called A and B, and the entire point is that they share nothing upstream: different breakers, different distribution panels, ideally different UPS systems and different generator paths.

   utility ──┬── UPS-A ── panel A ── breaker A ── whip A ──┐
             │                                              │
             │                                       ┌──────v──────┐
   generator ┤                                       │   PDU-A     │
             │                                       │  (0U, left) │
             │                                       └──────┬──────┘
             │                                              │
             │                                     ┌────────v────────┐
             │                                     │  server         │
             │                                     │  PSU1 ─── A     │
             │                                     │  PSU2 ─── B     │
             │                                     └────────^────────┘
             │                                              │
             │                                       ┌──────┴──────┐
             │                                       │   PDU-B     │
             │                                       │ (0U, right) │
             │                                       └──────^──────┘
             │                                              │
             └── UPS-B ── panel B ── breaker B ── whip B ───┘

The whip is the flexible armoured cable that drops from the overhead busway or comes up through the floor and terminates in a socket at your cabinet. The PDU plugs into it.

Colour convention is near-universal and worth trusting: A is usually blue, B is usually red, on the PDU bodies, the whips and often the power cords themselves. It is not a standard, but every facility I have seen documentation for does something like it, and it makes "both cords are in the same side" visible from three metres away instead of invisible forever.

The 80 % rule, which is where the capacity goes

In the US, the National Electrical Code treats anything drawing current for three hours or more as a continuous load. Every server is a continuous load. A continuous load may not exceed 80 % of the breaker's rating.

So a 30 A circuit gives you 24 A of usable continuous draw. Not 30. The last 6 A exists so the breaker does not sit permanently on the edge of tripping from thermal drift.

This is a US code framing — elsewhere the equivalent constraint comes through IEC practice, the PDU's own rating and the facility's contract rather than the NEC — but the practical outcome is the same everywhere: you do not get to use the number on the label. Ask your facility what they will actually let you draw, and get it in writing. Many PDUs will alarm, and some facilities will bill or shut you off, well before the breaker itself opens.

In kilowatts, for the feeds you are most likely to meet:

Three-phase looks like magic until you see the formula: kW = 1.732 × volts × amps. The 1.732 is the square root of three. It is the reason a three-phase 30 A feed carries roughly 73 % more than a single-phase 30 A feed on the same amperage number, and it is why dense cabinets are three-phase.

208 V rather than 120 V is the norm in datacenters and is slightly better than 120 V for the same equipment, because server PSUs run marginally more efficiently at the higher voltage and you get more watts per amp of breaker.

Redundancy costs you half

Here is the part that surprised me most, and it is the single most useful thing in this post.

If the point of A and B is that either one can carry the cabinet alone, then the total cabinet load must fit inside one feed's 80 % limit — not inside the sum of both.

Two 30 A feeds do not give you 48 A of usable capacity. They give you 24 A of usable capacity, delivered over two paths.

In normal operation the load splits roughly evenly, so each feed carries about 12 A — 40 % of its rating. That looks like waste right up until the moment one side drops.

The thing that goes wrong when people ignore this has a name: swing load. Both feeds are running at, say, 70 %. The A-side breaker opens. Every dual-corded server instantly moves its entire draw to the B side, which is now asked for 140 % of its rating. The B breaker opens too. You have just converted a single-feed failure into a total cabinet outage, and you did it by using capacity you were sold but could not actually have.

   Designed correctly (2N):          Designed wrong:
   A: 40%  ████                      A: 70%  ███████
   B: 40%  ████                      B: 70%  ███████

   A fails ↓                          A fails ↓

   A:  0%                             A:  0%
   B: 80%  ████████  holds            B: 140% ██████████████  TRIPS
                                          → whole cabinet down

If you take one number from this series, take this one: in a 2N cabinet, target 40 % per feed and treat 40 % as full.

There is a softer variant. If your fleet is genuinely tolerant of losing a fraction of nodes — stateless web tier, three-way replicated storage — some operators deliberately run primary/primary, loading both feeds to 80 % and accepting that a feed failure kills half the cabinet. That is a legitimate engineering choice and it doubles your usable power. It is only legitimate if it is a decision, written down, with the blast radius understood. Drifting into it by accident is the failure mode described above.

PDU types, and which one you want

PDUs come in four escalating tiers, and the difference is entirely about what you can see and do remotely.

Basic. A metal strip with outlets. No intelligence, no network port, no measurement. Cheap. You have no idea what you are drawing.

Metered. Reports total current draw for the PDU, either on a local display or over the network. This is the minimum I would accept, because without it the 40 % rule above is unenforceable — you cannot manage a number you cannot see.

Monitored. Per-outlet measurement. Now you can tell which server is drawing what, spot a PSU that has quietly failed over, and do capacity planning that is not guesswork.

Switched. Per-outlet relays, so you can power-cycle an individual outlet over the network. This is the one that matters operationally, for a reason covered properly in Part 4: a switched PDU is the only remote reset that works when the BMC itself is wedged. It is the difference between "I can fix this from my laptop at 3 a.m." and "I need to open a remote hands ticket and wait."

Modern units combine tiers — "switched and monitored" is common — and most speak SNMP, plus increasingly a REST API. Get them into Prometheus. Cabinet power draw is a genuinely useful metric and almost nobody graphs it.

The connectors

This is the vocabulary you asked about, and it is smaller than it looks. The IEC 60320 family covers the cords between the PDU and your equipment, and there are really only four pieces.

The pairing rule is trivial once you see it: odd numbers are sockets, the next even number up is the matching inlet. A C13-to-C14 cord runs from the PDU to an ordinary 1U server. A C19-to-C20 cord runs from the PDU to something hungry.

Two practical notes. First, locking variants exist — C13/C14 with a latch, or the C15/C16 heat-rated version — and they are worth specifying, because the most embarrassing outage in this entire domain is a power cord walking out of a socket a few millimetres while someone works in the cabinet next to it. Many datacenter PDUs have a built-in retention clip. Use it. Second, C15 is not C13 with a different name: it is rated for higher temperatures and has a notch, and a C15 plug will not seat in a C13 socket.

The other family is NEMA, which is the plug on the input side of the PDU, going into the facility's whip. You will see:

• L6-30P — the twist-lock plug for a 208 V, 30 A single-phase feed. Extremely common.
• L6-20P — same but 20 A.
• L21-30P — five-wire twist-lock for 208 V three-phase wye, 30 A.
• L15-30P — four-wire, three-phase delta.

The L means locking: you push and twist, and it cannot be pulled out. You will never handle one of these. You will absolutely have to quote one when ordering a PDU, because a PDU with the wrong input plug is a paperweight — and the whip is fixed infrastructure that the facility installed, so the PDU is what has to match.

What to actually do with all this

Four things, in order of how much trouble they save.

Know your budget in kilowatts, not amps. Vendors quote server power in watts. Facilities quote feeds in amps. Convert everything to kW once, at 80 %, and keep it in the same document as the rack elevation.

Add up nameplate, then measure. Nameplate PSU ratings are worst case and are usually far above real draw — a 750 W PSU in a 1U web server might pull 150 W idle and 300 W under load. Plan on nameplate so you never trip a breaker; measure real draw with a metered PDU so you know how much headroom you truly have. Those two numbers can differ by a factor of three, and the gap is where your remaining capacity lives.

Audit the cording. For every dual-PSU machine, confirm one cord is in A and one in B. This is a perfect remote hands job and it is worth doing once as an explicit ticket, with photos, rather than assuming. Ask for a photo of each PDU's outlet strip with the labels visible.

Know which machines are single-corded. Some things — cheap switches, older appliances, small NAS units — have one PSU. In a 2N cabinet these are your true single points of failure, and the mitigation is either a transfer switch (a small unit with two inputs and one output that switches automatically) or accepting the risk deliberately. Either is fine. Not knowing which machines they are is not.

The one-line summary

Two feeds, sized so either can carry the whole cabinet: 40 % each is full. C13/C14 for normal kit, C19/C20 for hungry kit, odd is socket and even is inlet. Get switched-and-monitored PDUs so you can see draw per outlet and power-cycle without a ticket.

Next up: Part 3 — cables, optics and transceivers, which is where DAC, AOC, SFP28 and QSFP28 stop being noise and start being a decision you can make.

This part is the physical frame: what the numbers mean, what holds equipment in, why depth and weight matter more than height, and how to read the diagram that describes it all.

Nineteen inches, and why everything is that wide

The standard is EIA-310. It fixes the width of the equipment's mounting face at 19 inches (482.6 mm) and the vertical pitch at 1.75 inches (44.45 mm), which is one U.

Two things follow that are worth saying out loud, because they trip people up.

19 inches is the front panel width, not the cabinet width. The mounting rails are 19 inches apart at the face; the cabinet around them is wider. Colocation cabinets are typically 600 mm or 800 mm wide externally. That extra 200 mm on an 800 mm cabinet is not wasted — it is the vertical cable management channel down each side, and it is the difference between a rear that you can work in and a rear that is a rat's nest.

U is a pitch, not a box. Equipment is built slightly under its nominal U height so it can slide in without binding. A 1U server is about 44 mm tall, not exactly 44.45. This is why you cannot stack equipment to fill an arbitrary gap: mounting holes only exist at U boundaries.

     ← ─────────  19" / 482.6 mm mounting face  ───────── →
    ┌─┬──────────────────────────────────────────────────┬─┐
    │o│                                                  │o│  ┐
    │o│                 1U of equipment                  │o│  │ 44.45 mm
    │o│                                                  │o│  ┘
    ├─┼──────────────────────────────────────────────────┼─┤
    │o│                                                  │o│
    │o│                 1U of equipment                  │o│
    │o│                                                  │o│
    └─┴──────────────────────────────────────────────────┴─┘
      ^                                                  ^
      └── mounting rail, punched with holes in            ┘
          repeating groups of three per U

The holes come in groups of three per U, unevenly spaced — a wide gap, then two closer together. That uneven pattern is how you find a U boundary by eye. If a technician mounts something one hole out of alignment, everything above it in the rack is now offset by a third of a U and nothing else will line up. This is a real and common failure, and it is why "start from a marked U" is standard practice.

Square holes, cage nuts, and threaded holes

Three kinds of rail exist and you should know which you have.

Square holes are the modern datacenter default. The hole is a plain square with no thread. You clip a cage nut into it — a small square nut in a springy sheet-metal cage — and then screw into that. The advantage is flexibility: any thread size, any equipment, replaceable if you strip one.

Round threaded holes (usually M6 or 10-32) are older and common in telecom and small office racks. You screw directly into the rail. Simpler, but a stripped thread damages the rack rather than a 20-cent part.

Tool-less rails clip straight into square holes with no screws at all. Most modern 1U and 2U servers ship with these, and they are why racking a server is now a two-minute job rather than a twenty-minute one.

Cage nuts have a correct orientation — they insert from the rear of the rail with the spring clips facing the installer — and they are the single most-cursed component in the building because installing them by hand hurts. There is a dedicated tool. Technicians have one. This is not your problem, but if a ticket comes back saying a cage nut was missing at a given position, now you know what was being asked for.

Rails, and the depth problem

A server does not sit on a shelf. It mounts on a pair of rails that bolt to the front and rear posts, and the server slides onto them.

This means the distance between the front and rear posts must match what the rail kit supports. Rail kits have an adjustment range — commonly something like 610–915 mm — and if your cabinet's post-to-post depth falls outside it, the rails do not fit. Full stop. No amount of effort at the far end fixes it.

Cabinet depths you will see are 600, 800, 1000 and 1200 mm externally, with most datacenter cabinets in the 800–1070 mm range. A 600 mm deep cabinet is a network cabinet — switches and patch panels are rarely deeper than about 400 mm — and it will not take a modern server at all.

Depth is the dimension that actually constrains you, not height. A GPU server or a high-density storage chassis can be long, and once you add the rear cable bend radius and a cable management arm you can run out of cabinet before you run out of U. When you spec a machine for a cage you did not design, the chassis depth and the rail kit's supported range are the two numbers to check first.

Two kinds of rail behave differently. Sliding rails let the server pull forward for service without unracking it, which is what you want for anything with disks you will swap or lids you will open. Static rails are fixed; servicing means unbolting the machine and taking it out, with a second person if it is heavy. Sliding rails usually need a cable management arm and a service loop — deliberate slack in every rear cable so the machine can travel without unplugging itself. If a machine is racked on sliders with cables cut to exact length, it can only be serviced by disconnecting it, which quietly defeats the point.

Weight, and why the bottom of the rack is not free real estate

Cabinets have a static load rating, often in the region of 1000–1500 kg, and floors have their own limit, which in a raised-floor room can be the tighter of the two.

The operational rule is simple and universal: heavy things go at the bottom. A full 4U disk shelf mounted at the top of a cabinet makes it top-heavy, and a top-heavy cabinet on castors is dangerous to move and can tip when a sliding-rail server is pulled fully out of it. Technicians know this. It is worth knowing too, so that when you plan an elevation you put the storage low and the 1U compute high, rather than the other way around.

Blanking panels are not cosmetic

An empty U in the middle of a populated cabinet is a hole between the cold aisle and the hot aisle. Hot exhaust air flows back through it to the front of the rack, gets sucked into the next server's intake, and that server now runs hot for no reason a monitoring dashboard will ever explain.

Blanking panels are blank sheets of metal or plastic that fill unused U positions and stop that recirculation. They cost very little and they are the single highest-return thing in a cabinet.

Two related airflow items in the same family: brush strips or grommets where cables pass through the cabinet floor or roof, and side panels fitted so air cannot bypass between adjacent cabinets. All three do the same job — force air to go through the equipment rather than around it.

If you are ever asked to approve a rack build and you see empty unblanked U positions in the elevation, that is a legitimate thing to comment on.

0U and the vertical space

You will see PDUs described as 0U. That means they mount vertically in the cabinet's side channel rather than horizontally in the 19-inch mounting space, so they consume no U at all. A 42U cabinet with two 0U PDUs still has 42U free.

That side channel is also where vertical cable managers live. It is the reason to pay for an 800 mm wide cabinet rather than a 600 mm one when the cabinet holds servers.

Reading a rack elevation

A rack elevation is the map of what is mounted where. It is what you send when you ask for something to be installed, and what you consult when you ask for something to be touched. It looks like this:

  CABINET  DC1 / ROW-C / RACK-07                 (front view)
  ┌────┬─────────────────────────────────────────────────────┐
  │ 42 │  ░░░ blanking ░░░                                   │
  │ 41 │  patch panel  PP-01   (24 x LC duplex, to MMR)      │
  │ 40 │  switch  sw-c07-01    Arista 7050SX  48x10G+4x40G   │
  │ 39 │  switch  sw-c07-02    Arista 7050SX  48x10G+4x40G   │
  │ 38 │  ░░░ blanking ░░░                                   │
  │ 37 │  srv-web-11    Dell R650   1U                       │
  │ 36 │  srv-web-12    Dell R650   1U                       │
  │ 35 │  srv-web-13    Dell R650   1U                       │
  │ .. │  ...                                                │
  │ 04 │  ┌───────────────────────────────────────────┐      │
  │ 03 │  │  srv-stor-01   Supermicro 4U  36-bay      │      │
  │ 02 │  │  (heavy — bottom of rack)                 │      │
  │ 01 │  └───────────────────────────────────────────┘      │
  └────┴─────────────────────────────────────────────────────┘
     0U left channel:  PDU-A   0U right channel:  PDU-B

Some conventions to note.

A U position is a location, and it is how you name things to remote hands. "The server at U37" is unambiguous in a way that "the third one down" never is.

Which end is U1 is not universally standardised. The common convention numbers from the bottom up, so U1 is the lowest position — and most cabinets are physically labelled that way on the rail. But it is not a guarantee: some manufacturers and some organisations number from the top down, and NetBox has carried a long-running discussion about supporting both because real deployments do both. So do not assume. Look at how your own cabinet's rails are printed, and match that. If a cabinet is unlabelled, agree a convention with the facility in writing before the first install, and put it in the elevation.

A multi-U device occupies a range and is referenced by its lowest U. The 4U storage box above lives at U1–U4 and you call it "U1".

Front and rear are separate views. Some things — PDUs, rear-mounted patch panels, cable managers — appear only in the rear elevation. If your documentation only has a front view, half the cabinet is undocumented, and that half is the half with all the cables in it.

If you keep this in NetBox or an equivalent DCIM, keep it accurate rather than complete. An elevation that is 60 % filled in but true is worth more than one that is 100 % filled in and eight months stale, because remote hands will act on it.

Labelling

The standard here is ANSI/TIA-606 (currently at revision D), which sets out how to identify cabling infrastructure, and it dovetails with TIA-942, the datacenter standard that divides a facility into functional areas — entrance room, main distribution area, horizontal distribution area, equipment distribution area, and so on.

You almost certainly do not need to implement TIA-606 formally. What you need is the idea behind it: a hierarchical identifier that reads from the outside in. Room, row, cabinet, panel, port. Something like DC1-C07-PP01-14 tells anyone in the building exactly where to stand and what to touch, and it sorts sensibly in a spreadsheet.

The two rules that pay for themselves regardless of standards: label both ends of every cable, and make the label say where the other end is. A cable tail that reads → sw-c07-01 Gi1/0/14 means nobody ever has to trace it by hand.

The five numbers to know about your own cabinet

If you take one thing from this part, make it this list. These are the facts about your cage that you should be able to answer without looking:

1. Cabinet identifier — room, row, position, in whatever scheme the facility uses.
2. Height in U, and how many are free — plus whether U1 is at the top or the bottom.
3. Usable depth, post to post, so you can check a rail kit before you buy a server.
4. Power — how many feeds, at what voltage and amperage. Part 2.
5. Uplinks — what the cabinet's switches connect to, and how many ports are spare. Part 5.

I could not answer three of those five for my own kit when I started writing this. That is exactly the gap this series exists to close.

Next up: Part 2 — power, A-side and B-side, where the 80 % rule turns a 30 A feed into rather less than 30 A.

That is the normal shape of this job now. You are a competent Linux engineer, you own machines that live in a building you have visited maybe twice, and every physical act on those machines happens through a stranger's hands and a ticketing system. It works fine right up until the moment someone says "the B-side feed is down on the cabinet, can you confirm your dual-corded gear is holding" and you realise you are nodding at words rather than understanding them.

This series is me closing that gap, written down. It is not a datacenter design course — I am not going to teach you to size a CRAC unit or argue about Tier ratings, because neither of us is ever going to do that. It is the subset a Linux engineer needs in order to describe their own hardware accurately, read a facility's paperwork, and write a remote hands ticket that gets the job done on the first attempt.

Part 0 is vocabulary. If you already know what a demarc is, skip to Part 1.

The five nested boxes

Almost every term in this series is a location. The whole model is five boxes inside each other, and knowing which box a word belongs to is most of the battle.

┌─ FACILITY ────────────────────────────────────────────────┐
│  the building. Has an address, a loading dock, a security  │
│  desk, generators, chillers, and a Meet-Me Room.           │
│                                                            │
│  ┌─ DATA HALL ─────────────────────────────────────────┐  │
│  │  one room full of rows. Also called a "suite".      │  │
│  │                                                      │  │
│  │  ┌─ CAGE ────────────────────────────────────────┐  │  │
│  │  │  your fenced-off floor space. Locked. Yours.  │  │  │
│  │  │  (If you rent single cabinets instead, you    │  │  │
│  │  │   have no cage — skip this box.)              │  │  │
│  │  │                                                │  │  │
│  │  │  ┌─ RACK / CABINET ────────────────────────┐  │  │  │
│  │  │  │  the 19-inch steel frame. 42U-48U tall. │  │  │  │
│  │  │  │                                          │  │  │  │
│  │  │  │  ┌─ SERVER ──────────────────────────┐  │  │  │  │
│  │  │  │  │  the thing you actually ssh into. │  │  │  │  │
│  │  │  │  └───────────────────────────────────┘  │  │  │  │
│  │  │  └──────────────────────────────────────────┘  │  │  │
│  │  └────────────────────────────────────────────────┘  │  │
│  └──────────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────┘

Your ssh session reaches the innermost box. Everything else in this series is one of the outer four, and every one of them can take your service down without a single thing being wrong inside Linux.

Rack and cabinet

Used interchangeably in conversation. Strictly, a rack is an open frame — four posts, no sides — and a cabinet is an enclosed rack with doors and side panels. Colocation is almost always cabinets, because the doors are lockable and the enclosure is what makes front-to-back airflow work.

The number you will see everywhere is U (or RU, rack unit): the vertical unit of mounting space, standardised at 1.75 inches / 44.45 mm. A "1U server" is one unit tall, a "2U server" is two. A typical colo cabinet is 42U or 48U. Part 1 pulls this apart properly.

Front and back

This is the one that surprised me by how much it matters, and it is the answer to "what do I expect at the front versus the back of the rack".

The front is where the cold air is and where nothing is plugged in. Front of a server: bezel, drive bays, power button, status LEDs, a service tag, sometimes a small LCD or a USB/serial port for a technician. That is essentially it.

The back is where every cable is. Network, power, out-of-band management, console. All of it.

        COLD AISLE                              HOT AISLE
   (air conditioning in)                    (hot exhaust out)
            │                                       │
            v                                       v
    ┌───────────────────┐                   ┌───────────────────┐
    │  FRONT of rack    │                   │   REAR of rack    │
    │                   │  ═══ airflow ══>  │                   │
    │  · drive bays     │                   │  · NIC ports      │
    │  · power button   │                   │  · BMC/IPMI port  │
    │  · status LEDs    │                   │  · PSU inlets     │
    │  · service tag    │                   │  · PDUs (A and B) │
    │  · blanking panels│                   │  · patch panels   │
    └───────────────────┘                   └───────────────────┘

Servers pull cold air in the front and blow hot air out the back. Rows are arranged so that all the fronts face each other across one aisle (the cold aisle) and all the backs face each other across the next (the hot aisle). That is hot aisle / cold aisle containment, and it is why you cannot simply turn a cabinet around to make cabling easier.

Practical consequence for you: almost every remote hands task happens at the back. If your ticket says "the port on the left" without saying whether you mean facing the front or the back, you have written an ambiguous ticket. Say rear, and say left-or-right as viewed from the rear.

A-side and B-side

Your cabinet gets two independent power feeds, traced back through separate breakers, separate distribution panels, and ideally separate UPS systems. They are called the A-side and the B-side. Each feeds its own PDU (power distribution unit — the metal strip of outlets running down the inside of the cabinet).

A server with two power supplies has one cord in the A PDU and one in the B PDU. Lose an entire side — a tripped breaker, a UPS in maintenance — and the server keeps running on the other.

Two things I did not appreciate until I went looking. First, this only protects you if the machine genuinely has two PSUs and both are genuinely plugged into different sides; a single-PSU box in a dual-fed cabinet has no redundancy at all, and it is depressingly easy to have one cabinet in which somebody cabled both cords of a server into the same PDU. Second, redundancy costs you capacity: if either side must be able to carry the whole cabinet alone, neither side can normally run above roughly half its usable rating. Part 2 does the arithmetic, including the 80 % rule that makes "usable" smaller than you think.

Demarc

The demarcation point, universally shortened to demarc, is the boundary of responsibility. On one side, the facility's or carrier's equipment and problem. On the other, yours.

Physically it is normally unglamorous: a patch panel, in or near your cage, where the facility's cabling ends and you are allowed to plug in. When a circuit breaks, the first question anyone asks is which side of the demarc the fault is on, because that decides whose job it is to fix.

The word comes from telephony, where the demarc was the box on the outside of the building. Same idea, indoor version.

Meet-Me Room and cross-connect

The Meet-Me Room (MMR) is a secure room in the facility where all the carriers, cloud on-ramps and other tenants terminate their networks. Nobody gets casual access to it. It is the building's switchboard.

A cross-connect is a physical cable the facility installs for you between two points in the building — typically from your cage's demarc panel to some other party's termination in the MMR. It is not a service or a circuit. It is literally a piece of fibre or copper that a technician runs and patches, which is why it carries an install fee and a monthly rental and why it takes days rather than seconds.

To order one you supply an LOA-CFA: Letter of Authorization / Connecting Facility Assignment. The other party issues it, and it is the document that says "this customer is permitted to connect to us, at this exact cabinet, panel and port." Without it the facility will not patch anything, because they have no proof the far end consented. AWS, for example, issues an LOA-CFA for a Direct Connect port and it expires if the cross-connect is not completed in time.

Two more terms that ride along: the A-side and Z-side of a circuit. Confusingly this is not the A/B of power — here A is simply one end of the circuit and Z is the other. Context tells you which meaning is in play: power has A and B, circuits have A and Z.

Direct connect

A direct connection to a cloud provider is a cross-connect whose far end happens to be AWS, Azure or Google rather than a carrier. The commercial names are AWS Direct Connect, Azure ExpressRoute and Google Cloud Interconnect. Traffic to that cloud leaves your cabinet, crosses the MMR on your fibre, and lands on the provider's router without ever touching the public internet.

You will meet two flavours. A dedicated connection is a whole physical port that is yours, in AWS's case 1, 10, 100 or 400 Gbps, and you order the cross-connect yourself. A hosted connection is a slice of somebody else's port sold to you by a partner, available at much smaller sizes, with no cross-connect for you to order because the partner already has one. Part 6 covers which you actually want.

DAC, AOC, transceivers

A DAC is a Direct Attach Copper cable: a fixed length of twinaxial copper with the transceiver moulded onto both ends. You cannot unplug the ends. You buy it as one unit — "a 3-metre SFP28 DAC" — and it is the cheap, low-power, low-latency way to connect a server to a switch in the same rack. Cheap and reliable, but short: passive DACs are typically good to about 3 m and stretch to 5 or 7 m; active ones with signal conditioning built in reach roughly 10 m.

An AOC is an Active Optical Cable: same idea, ends moulded on, but the middle is fibre and the ends contain lasers. Much longer reach — up to about 100 m — for more money and more power.

A transceiver (a module, an "optic", an SFP) is the removable part on its own: you plug the transceiver into the switch and a separate fibre patch lead into the transceiver. That is what you use when the run is long, goes through a patch panel, or crosses to another rack.

The form factor names encode speed. SFP+ is 10 Gb, SFP28 is 25 Gb, QSFP+ is 40 Gb, QSFP28 is 100 Gb. Part 3 covers the whole family, plus which fibre goes with which and why the colour of the cable tells you something real.

IPMI, BMC, iDRAC, iLO

Your server contains a second, tiny computer that runs whether or not the main machine is powered on. It is called the BMC — Baseboard Management Controller. It has its own processor, its own firmware, its own network stack, and its own IP address, and it can power the server on and off, show you the console, mount an ISO, and read every temperature and fan in the chassis.

IPMI is the old standard protocol for talking to it. Redfish is the modern HTTPS/JSON replacement. The vendors each brand their BMC differently, which is why the same thing has four names: Dell calls it iDRAC, HPE calls it iLO, Lenovo XCC, Supermicro just IPMI.

The cable question you asked about: on the back of the server there is usually a small RJ45 labelled BMC, IPMI, iDRAC or iLO, physically separate from the data NICs. That is the dedicated management port and it is the one you want. The alternative is shared mode, where the BMC borrows one of the main NIC ports over a sideband channel called NC-SI. Shared saves you a switch port and a cable; it also means that whatever takes out the main NIC takes out your way of fixing the main NIC. Part 4 argues that one out properly, along with why a BMC must never be reachable from the internet.

Remote hands and smart hands

Remote hands is the basic, usually contract-included tier: a technician follows precise instructions. Press this button. Reseat that cable. Read me the LEDs. Tell me what the LCD says. They are not expected to diagnose anything, and you should not expect them to.

Smart hands is the chargeable tier for work that needs judgement or real hardware skill: racking and cabling a new server, replacing a PSU or a disk, receiving and unboxing a shipment, testing a circuit.

The distinction matters commercially, but the operational lesson is the same for both: the technician cannot see your monitoring, does not know your naming scheme, and is standing in a loud cold room in front of forty identical black boxes. Every ambiguity in your ticket becomes either a phone call or a mistake. Part 7 is entirely about writing tickets that survive that environment.

What is coming

• Part 1 — the rack itself: U numbering, rails, cage nuts, depth, weight, blanking panels, and how to read a rack elevation.
• Part 2 — power: A/B feeds, PDU types, C13 through C20, the 80 % rule, and why redundancy halves your capacity.
• Part 3 — cables and optics: DAC, AOC, transceivers, multimode versus singlemode, LC and MPO, and picking a length.
• Part 4 — out-of-band: BMC, IPMI, Redfish, serial consoles, switched PDUs, and the three different things "reboot it" can mean.
• Part 5 — where the switches live: top-of-rack versus end-of-row, leaf-spine, uplinks, oversubscription.
• Part 6 — leaving the cage: demarc, MMR, cross-connects, direct connects, transit, peering and IXs.
• Part 7 — directing remote hands: the ticket template, the standard jobs, and how to verify the work.
• Part 8 — the cheat sheet: every number and connector in this series on one page.

A note on how this was written, since the blog's rule is not to claim experience I do not have: this series is built from vendor and standards documentation, facility operator guidance, and the parts of the job I do touch — the tickets, the paperwork and the Linux side of the machine. Where something is a genuine hardware-handling skill, I have said so and pointed at the people who have it, rather than pretending otherwise.

Next up: Part 1 — anatomy of a rack, which starts with the fact that a "42U cabinet" does not give you 42U of usable space.

Parts 0 through 6 explain why each of these works. This is the lookup table for when you don't have time to read.

Glossary

Start here: three questions

Decide which one you're asking before you type anything.

1. Can they reach me? → connectivity. nc -zv, ss -lnt, tcpdump, firewall rules, ip route get
2. Are we losing traffic? → loss. nstat, ip -s link, /proc/net/softnet_stat, ethtool -S
3. Why is it slow? → latency. curl -w, ss -ti, mtr, iostat -x, mpstat

Mixing these up costs hours. Answer this first, every time.

The ladder

Every tool reads one layer. Know which.

 [7] application ......... ss Recv-Q, app logs, app metrics
 [6] socket buffers ...... ss -tin
 [5] TCP/IP stack ........ nstat
 [4] routing ............. ip route get, ip rule, rp_filter
 [3] netfilter/conntrack . iptables -L -n -v, nft list ruleset, conntrack -S
 [2] driver/softirq ...... /proc/net/softnet_stat, /proc/interrupts
 [1] NIC hardware ........ ip -s link, ethtool -S, ethtool -g
 [0] layer 2 ............. ip neigh, bridge fdb show
     the wire

The flowchart

                  "it doesn't work"
                          |
              +-----------+-----------+
              |                       |
        fails INSTANTLY          HANGS then times out
     ("connection refused")     ("connection timed out")
              |                       |
        packet ARRIVED           packet may not have arrived
              |                       |
     +--------+--------+         tcpdump on the server
     |                 |              |
  nothing         firewall     +------+------+
  listening       REJECT       |             |
     |                    SYN visible?   nothing at all?
  ss -lntp                    |             |
  check bind addr        LOCAL problem   UPSTREAM problem
  (127.0.0.1?)           firewall -Z     routing / secgroup
                         rp_filter       ip route get
                         conntrack       ip neigh
                         accept queue


              "it works but it's slow"
                          |
                    curl -w breakdown
                          |
      +-------------------+-------------------+
      |                   |                   |
  high DNS          high CONNECT          high TTFB
      |                   |                   |
  dig / getent      network path OR      the APPLICATION
  resolvectl        accept queue         (and what's behind it)
                         |                     |
                   ss -lnt Recv-Q       app logs, upstream time
                   ss -ti rtt/retrans   iostat -x, mpstat
                   mtr (final hop only)

The 60-second triage

On an unfamiliar box, in this order:

nstat                              # run twice, 30s apart — what is moving?
ip -s link show eth0               # is the wire healthy?
ss -lnt                            # are listeners keeping up? (Recv-Q vs Send-Q)
cat /proc/net/softnet_stat         # is one CPU drowning? col 2 = drops
conntrack -S                       # if this box does NAT or containers

Five commands. Most of the search space eliminated.

First contact — is it even reachable?

Bottom-up. Each step assumes the previous one passed.

ip -br addr                       # interface up? right address? right prefix?
ip -br link
ip route get 10.0.0.50            # route, interface, AND source IP
ping -c 5 10.0.0.50               # see caveat below
getent hosts db.internal          # resolves the way the APP sees it (NSS)
dig +short db.internal            # what DNS itself says
nc -zv -w 3 10.0.0.50 5432        # is the port actually open?
curl -vI https://example.com/     # TLS + HTTP

Ping failing means almost nothing — ICMP is blocked by policy everywhere. Ping succeeding is useful; ping failing is not evidence the host is down. Test the port.

nc failure modes are the diagnosis:

• Refused instantly → packet arrived, nothing listening. Check bind address.
• Hangs then times out → something is silently dropping. Firewall.

DNS — the split that solves the hard cases:

dig +short name       # queries DNS directly, BYPASSES /etc/hosts and NSS
getent hosts name     # goes through NSS — same path as your application
dig @8.8.8.8 name     # is it my resolver, or the record itself?
dig +trace name       # follow delegation from the root
resolvectl status     # systemd-resolved config
grep ^hosts /etc/nsswitch.conf
cat /etc/resolv.conf

dig works but getent fails → not DNS. It's /etc/hosts, nsswitch.conf, or the resolver library.

Who owns the port / what is the process doing:

ss -lntp
lsof -i :443
lsof -i -P -n
fuser 443/tcp
strace -f -e trace=network -p <pid>     # is it even TRYING to connect?

TLS:

openssl s_client -connect example.com:443 -servername example.com
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Always pass -servername or you'll get the wrong certificate and debug a problem you created.

Bandwidth (not latency):

iperf3 -s                    # server
iperf3 -c <ip>               # client
iperf3 -c <ip> -R            # reverse direction — always test both
iperf3 -c <ip> -P 4          # parallel streams

One stream slow, four streams fast = window/RTT ceiling, not a slow link.

traceroute caveats: * * * means the router declined to reply, not a broken hop. Intermediate-hop loss is usually ICMP rate limiting. Only the final hop, or loss persisting through every subsequent hop, is real. traceroute -T -p 443 gets further through firewalls than the UDP default.

Symptom → command

ss — sockets

ss -lntp                    # what am I listening on, and which process
ss -antp                    # every TCP connection with process
ss -s                       # summary counts by protocol and state
ss -tin                     # per-connection TCP internals (rtt, cwnd, retrans)

ss -antp dst 203.0.113.7               # one peer
ss -antp '( dport = :443 or sport = :443 )'
ss -antp state established
ss -antp state close-wait              # app leaking descriptors
ss -tan state time-wait | wc -l

# Top clients by connection count
ss -ant state established | awk '{print $5}' | cut -d: -f1 \
  | sort | uniq -c | sort -rn | head

Always -n. Without it, ss does a DNS lookup per address.

Reading the queue columns

ESTABLISHED socket:

• Recv-Q = bytes waiting for the app to read() → app is behind
• Send-Q = bytes waiting to be acked → network or peer is behind

LISTENING socket:

• Recv-Q = connections waiting for accept()
• Send-Q = maximum accept queue size
• Recv-Q at Send-Q = overflowing right now

ss -ti fields

TCP states

Counters

nstat                       # delta since last run — use this during incidents
nstat -az                   # everything, including zeros
nstat -d 1                  # sample every second

nstat -az | grep -i retrans
nstat -az | grep -i listen
nstat -az | grep -i Udp

Names vary by kernel version. nstat -az on your box is authoritative.

Interface and driver

ip -s link show eth0        # errors, dropped, overrun/missed
ip -s -s link show eth0     # more detail

ethtool -S eth0 | grep -i -E 'drop|err|miss|discard|fifo'
ethtool -g eth0             # ring buffer sizes (current vs max)
ethtool -k eth0             # offloads — GRO/TSO make tcpdump show huge packets
ethtool eth0                # link speed, duplex

tc -s qdisc show dev eth0   # transmit-side drops

errors = physical layer. Cable, SFP, port, duplex.
overrun / missed = ring buffer overflow. The box can't keep up.
dropped = kernel discarded it. Usually no buffer or no handler.

ethtool -S counter names are driver-specific. Grep, don't memorise.

Softirq

cat /proc/net/softnet_stat      # hex, one line per CPU
watch -d cat /proc/net/softnet_stat
cat /proc/interrupts | grep -i eth
mpstat -P ALL 1                 # %soft column

• Column 1 = processed. Wildly uneven across CPUs = IRQ imbalance.
• Column 2 = dropped (backlog full). Non-zero is a real, invisible drop.
• Column 3 = time_squeeze. NAPI budget exhausted.

tcpdump

tcpdump -i eth0 -nn -c 100 'host 203.0.113.7 and port 443'
tcpdump -i eth0 -nn -w /tmp/cap.pcap 'host 203.0.113.7'
tcpdump -r /tmp/cap.pcap -nn | head -50

tcpdump -nn 'tcp[tcpflags] & tcp-syn != 0'    # who is connecting
tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0'    # who is being reset
tcpdump -nn 'host X and not port 22'          # exclude your own SSH

Key flags: -nn (no resolution), -c N (stop after N), -w/-r (file), -e (ethernet header), -A/-X (payload), -S (absolute seq), -tttt (readable timestamps).

tcpdump sees inbound packets before netfilter. So: SYN visible but app never sees it = local firewall. SYN not visible at all = never arrived.

Flags

The dot is ACK. That's the whole notation.

Patterns

Window scale trap: if you didn't capture the handshake, every win value printed is wrong. Recapture from connection start.

Latency

curl -w '\ndns %{time_namelookup}\nconnect %{time_connect}\ntls %{time_appconnect}\nttfb %{time_starttransfer}\ntotal %{time_total}\n' \
  -o /dev/null -s https://example.com/

ping -c 20 X                # min/avg/max/mdev
mtr -rwc 100 X              # per-hop loss and latency
ss -tin state established dst X    # real RTT on the real connection
iostat -x 1                 # r_await / w_await, NOT %util
mpstat -P ALL 1             # %soft

Values are cumulative — subtract to get each phase.

• High connect, low TTFB → network
• Low connect, high TTFB → application
• Erratic connect → queueing or loss

mtr caveat: loss at intermediate hops is usually ICMP rate limiting, not real. Only trust loss at the final hop, or loss that persists through every subsequent hop.

iostat caveat: on SSD/NVMe, %util does not mean saturation. Use await.

Throughput ceiling ≈ window / RTT. 64KB over 100ms RTT ≈ 5 Mbit/s no matter how fast the link.

Historical data

sar -n DEV 1                    # throughput
sar -n EDEV 1                   # errors and drops
sar -n TCP,ETCP 1               # connection rates, retransmits
sar -n SOCK 1                   # socket counts

sar -n EDEV -f /var/log/sa/sa13
sar -n DEV -s 03:00:00 -e 04:00:00 -f /var/log/sa/sa13

Log path varies by distro (/var/log/sa or /var/log/sysstat). On Debian/Ubuntu the collector is often disabled by default — check /etc/default/sysstat.

Enable this on every production box. Every other tool tells you about now; incidents are about twenty minutes ago.

Routing, ARP, firewalls, containers

ip route get 10.0.0.50        # ASK the kernel. Gives via, dev, AND src.
ip route                      # the table — most specific prefix wins
ip rule show                  # which table gets consulted, lowest number first
ip route show table 100

sysctl -a 2>/dev/null | grep '\.rp_filter'    # 0=off 1=strict 2=loose
nstat -az | grep IPReversePathFilter          # proof rp_filter is dropping

ip neigh show                 # REACHABLE / STALE / INCOMPLETE
ip neigh flush dev eth0
arping -I eth0 192.0.2.1
sysctl net.ipv4.neigh.default.gc_thresh3

ip route get over reading the table. It runs the real decision and reports the source IP that will be used — which is what the far end's firewall sees.

INCOMPLETE in ip neigh = ARP sent, nobody answered. Below every other layer in this series.

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v          # container hosts: check nat BEFORE filter
iptables -Z                       # zero, reproduce, then look
nft list ruleset
conntrack -L ; conntrack -E ; conntrack -S

Zero, reproduce, look. iptables -Z, trigger the problem, iptables -L -n -v — whichever counter moved is the rule that handled your packet. Stop reasoning about rule order; measure it.

ip netns list                              # often EMPTY on Docker — not a bug
docker inspect -f '{{.State.Pid}}' <name>
nsenter -t <pid> -n ss -lntp
nsenter -t <pid> -n ip -br addr
nsenter -t <pid> -n tcpdump -nn -i eth0

nsenter -t <pid> -n ip link show eth0      # shows eth0@ifN
ip link | grep '^N:'                       # N = the host-side veth
bridge link ; bridge fdb show

nsenter -t <pid> -n <cmd> runs your host tools inside the container's namespace. You never need to install debugging tools in an image.

eth0@ifN inside the container → interface index N on the host. That's which veth to capture on.

Namespaces are independent stacks: host ss, tcpdump, iptables and every counter in part 3 do not cover container traffic.

ss -xlp                       # listening unix sockets + process
ss -xp

Unix sockets have no packets — tcpdump will never show them. (13: Permission denied) is a file-permission problem, not a network one.

Config worth checking

sysctl net.core.somaxconn                  # accept queue ceiling
sysctl net.ipv4.tcp_max_syn_backlog        # SYN queue
sysctl net.core.netdev_max_backlog         # softirq backlog
sysctl net.ipv4.ip_local_port_range        # outbound port pool
sysctl net.ipv4.tcp_syncookies
sysctl net.ipv4.tcp_congestion_control
sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count
cat /proc/<pid>/limits                     # the FD limit that actually applies

Rules on tuning:

• Accept queue = min(listen() backlog, net.core.somaxconn). Raising one alone does nothing.
• Never net.ipv4.tcp_tw_recycle. Removed in kernel 4.12; broke NAT'd clients before that. Any guide recommending it predates 2017.
• Buffer tuning (tcp_rmem/tcp_wmem) matters for high bandwidth-delay paths. It's cargo cult for a web server 20ms from its clients.
• A full accept queue is a symptom. A deeper queue is somewhere deeper to fall behind.

nginx specifics

tail -100 /var/log/nginx/error.log

Add to log_format and split the time:

• $request_time — total
• $upstream_connect_time — connecting to backend
• $upstream_header_time — until backend headers
• $upstream_response_time — total backend time

$upstream_response_time ≈ $request_time → the backend is the problem, run the same investigation one layer down.

Use keepalive in upstream blocks for high-volume proxying, or you'll exhaust source ports. Configuration details in the nginx upstream docs.

Learning path

Honest sequencing. Don't skip.

Stage 0 — the floor (a few days)

Glossary above, cold. The seven-step reachability order: address → route → ping → DNS → port → TLS → HTTP. Learn that ping failing proves nothing and nc -zv is the real test. Learn why dig and getent can disagree. Read man ip, man dig, man nc.

Stage 1 — the model (a week)

The 4-tuple. Socket buffers. The TCP handshake and teardown. Stream vs datagram. Draw the seven-layer ladder from memory. Capture a healthy request with tcpdump and read every field. You cannot recognise broken without knowing healthy.

Stage 2 — the socket layer (a month of using it)

Live in ss. ss -lntp and ss -antp until they're muscle memory. Learn what Recv-Q and Send-Q mean in both contexts. Learn the states, especially CLOSE_WAIT. Read man ss properly — it's short.

Stage 3 — evidence (a month)

nstat, ip -s link, ethtool -S, /proc/net/softnet_stat. The goal is one skill: make a falsifiable claim. Not "we have packet loss" but "ListenOverflows went 0 → 14,000 in ten minutes while interface errors stayed at zero." Learn the difference between a drop and a retransmit until it's automatic.

Stage 4 — the wire (ongoing)

tcpdump filters, then the eight patterns above. Capture on both ends of a connection at once and compare — that's the highest-value habit in this entire series. Then Wireshark for anything complex.

Stage 5 — attribution (ongoing)

curl -w decomposition. Path vs application. iostat, mpstat, and understanding that application and disk latency surface as network symptoms. Work a real production problem end to end.

Stage 6 — routing, L2 and containers (a month)

ip route get and ip rule. rp_filter and asymmetric routing. ip neigh states. Reading firewall rule counters with the zero-reproduce-look technique. Then namespaces: nsenter -t <pid> -n, veth pairs, and where container traffic gets NAT'd. If you run containers, this stage is not optional.

Stage 7 — eBPF

bpftrace, and the BCC tools: tcpretrans, tcplife, tcpconnect, biolatency. Move from "0.3% of connections retransmitted" to "this connection retransmitted, at this moment." This is the line between competent and strong.

Reading, in order

1. man 7 tcp — dense, unglamorous, better than most books
2. man ss, man tcpdump, man pcap-filter
3. TCP/IP Illustrated, Volume 1 (Stevens / Fall) — nothing else builds the same depth
4. Documentation/networking/ in the kernel source
5. BCC and bpftrace

The five things to remember if you forget everything else

1. Everything is a queue. Full queue = drops. Deep queue = latency. Find the queue.
2. Drops and retransmits point in opposite directions. Drops mean this box discarded something. Retransmits mean the path lost something.
3. Fast failure and slow failure are different diagnoses. Refused = it arrived. Timeout = it didn't.
4. ss -lnt before tcpdump. Recv-Q at Send-Q on a listener answers more incidents than any packet capture.
5. Localise, then tune. Every sysctl is a guess until you know which layer is failing.
6. The application is part of the network stack. Most "network problems" are an app that stopped reading its sockets.

Compiled by AI. Proofread by caffeine. ☕

No socket sees it. No drop counter moves. tcpdump shows it arriving and then nothing. It was routed somewhere unexpected, discarded by a firewall rule, dropped by reverse path filtering, or delivered into a network namespace you didn't know existed.

This part covers stops 3 and 4 on the ladder — netfilter and routing — plus the layer 2 and container material the series has been quietly assuming.

Routing: ask, don't read

Everyone starts with ip route. Wrong instinct. Start with:

ip route get 10.0.0.50

10.0.0.50 via 192.0.2.1 dev eth0 src 192.0.2.2 uid 0
    cache

This asks the kernel to run its actual routing decision and report the result. No interpreting a table, no working out which prefix is more specific. Three answers:

• via — the next hop
• dev — the outbound interface
• src — the source IP that will appear on the packet

That third one carries more weight than people give it. On a multi-homed box, "which address do we come from" determines whether the far end's firewall accepts you. It's not a guess and it's not always the interface's primary address.

The table itself, when you need it:

ip route

default via 192.0.2.1 dev eth0
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.2

Reading it: most specific prefix wins. A /32 beats a /24 beats /0. default is 0.0.0.0/0 — the fallback. proto kernel means the kernel added it automatically when the address was configured. scope link means the destination is directly reachable without a gateway.

Multiple routing tables

Linux doesn't have one routing table. It has many, and a rule list deciding which to consult.

ip rule show

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Lowest priority number is evaluated first. main is what ip route shows by default. To see another:

ip route show table local
ip route show table 100

This matters on multi-homed hosts, VPN setups, and anything using policy routing — where the source address, mark, or interface picks a different table. If ip route get returns something that contradicts ip route, a rule sent it to another table. Check ip rule show before assuming the kernel is wrong.

Asymmetric routing and rp_filter

The failure that produces no error anywhere.

A packet arrives on eth1. The kernel checks: if I wanted to reply to this source address, would I send it out eth1? If the answer is no, reverse path filtering may discard the packet. Silently.

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.default.rp_filter
sysctl -a 2>/dev/null | grep '\.rp_filter'

Three values:

• 0 — off
• 1 — strict. The reply must go back out the same interface.
• 2 — loose. Any route to that source is enough.

Strict mode plus asymmetric routing equals silently dropped traffic. It's designed as an anti-spoofing measure and it does that job well; it also breaks legitimate multi-homed and policy-routed setups.

The counter that proves it:

nstat -az | grep -i IPReversePathFilter

If that number is climbing, rp_filter is eating your packets. This is one of the very few drops that shows up nowhere else — not in interface counters, not at the socket layer, not in firewall rule counters. Know the counter's name and you'll find it in a minute instead of a day.

Note that the effective setting is per-interface, and the all value interacts with the per-interface one. Check net.ipv4.conf.<iface>.rp_filter for the interface in question, and read the behaviour in the kernel's Documentation/networking/ip-sysctl.rst before changing it.

ARP and the neighbour table

Layer 2, which the series has skipped so far. When a host sends to an IP on its own subnet, it needs that host's MAC address. ARP is the discovery protocol; the results live in the neighbour table.

ip neigh show

192.0.2.1 dev eth0 lladdr 02:fc:00:00:00:05 REACHABLE

The states tell a story:

INCOMPLETE is the diagnostic. It means: I broadcast "who has this IP?" onto the local segment and nobody answered. The host is off, not on this segment, or something is filtering ARP.

That's a fundamentally different failure from "connection refused" or "connection timed out at the TCP layer," and it's below everything else in the series. No amount of ss or nstat will show it.

ip neigh flush dev eth0        # force re-resolution
arping -I eth0 192.0.2.1       # probe directly (package: iputils-arping or arping)

Duplicate IPs

If two hosts claim the same address, ARP replies alternate and connectivity becomes maddeningly intermittent — works, then doesn't, then works. arping will show replies from two different MAC addresses for one IP. Once you've seen it, the symptom is unmistakable; before that it looks like haunted hardware.

Neighbour table overflow

On large flat networks the table can fill:

sysctl net.ipv4.neigh.default.gc_thresh1
sysctl net.ipv4.neigh.default.gc_thresh2
sysctl net.ipv4.neigh.default.gc_thresh3
dmesg | grep -i 'neighbour table overflow'

gc_thresh3 is the hard maximum. Hit it and entries get evicted aggressively, causing intermittent failures to hosts that were fine a moment ago. The dmesg line is unambiguous when it happens. Common on hosts with thousands of neighbours — large L2 segments, or busy container hosts.

Reading firewall rules

I've said "check your firewall rules" three times in this series without showing you how. Here's how, and the technique matters more than the syntax.

iptables

iptables -L -n -v --line-numbers
iptables -t nat -L -n -v --line-numbers

• -L list, -n numeric (always — otherwise it resolves every address), -v verbose, --line-numbers for reference.
• -t nat shows the NAT table. Forgetting this table exists causes a lot of confusion, because your filter rules look fine while NAT is rewriting everything.

-v is the whole point. It adds two columns at the front:

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source       destination
 4521  312K ACCEPT     tcp  --  *      *       0.0.0.0/0    0.0.0.0/0    tcp dpt:22
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0    0.0.0.0/0    tcp dpt:443
  892 53400 DROP       all  --  *      *       0.0.0.0/0    0.0.0.0/0

Those pkts counters are per-rule hit counts, and they turn firewall debugging from reading into measuring.

Look at that output: the rule accepting 443 has zero hits. The DROP at the bottom has 892. Traffic to 443 isn't matching the accept rule — wrong interface, wrong protocol, or an earlier rule caught it. You didn't have to reason about it. The counters said so.

The technique that makes this decisive:

iptables -Z              # zero all counters
# ... now reproduce the problem ...
iptables -L -n -v        # which rule moved?

Zero, reproduce, look. Whichever counter incremented is the rule that handled your packet. This is the single most useful firewall debugging move there is and most people never learn it.

Also check the chain policy — the (policy DROP) in the chain header. A default-DROP chain with no matching accept rule discards silently, which produces exactly the SYN-with-no-reply pattern from part 4.

nftables

Modern distros use nftables, often with iptables as a compatibility shim over it. If iptables -L looks empty but traffic is clearly being filtered, look here:

nft list ruleset

nftables doesn't count by default — you add counter to a rule to get statistics. If the ruleset was written without counters, add them to the rules you're investigating, or use logging.

Logging what gets dropped

When counters aren't enough, make the kernel tell you. Both iptables (LOG target) and nftables (log statement) can log matched packets to the kernel log, where dmesg or journalctl -k picks them up.

This is precise but dangerous on a busy box — an unrated log rule can flood your logs and your disk in minutes. Always rate-limit it, always remove it afterwards. Exact syntax is in man iptables-extensions and the nftables wiki; it's worth reading properly rather than pasting.

conntrack, revisited

Part 3 covered the table filling up. The other half is inspecting it:

conntrack -L                 # current tracked connections
conntrack -L -p tcp --dport 443
conntrack -E                 # live event stream
conntrack -S                 # per-CPU statistics

conntrack -E streams connection events as they happen — NEW, UPDATE, DESTROY. When you need to see whether the kernel is even creating state for a connection, this shows you live.

The failure to look for: a connection idle longer than conntrack's timeout gets its state removed, and the next packet arrives with no matching entry. Depending on rules, it's dropped or reset. That's the "connections die after exactly N minutes idle" pattern from part 4, seen from the kernel's side.

Containers and network namespaces

The modern gap. If you run containers and you've ever captured on the host and seen nothing, this is why.

The concept

A network namespace is an independent copy of the entire network stack — its own interfaces, routing table, ARP table, firewall rules, socket list, and counters. Containers each get one.

Consequences that surprise people:

• ss -lntp on the host does not show sockets inside containers.
• tcpdump on the host interface does not see container-internal traffic.
• iptables -L on the host is not the container's rule set.
• Every counter in part 3 is per-namespace.

Everything in this series still works. You just have to run it in the right namespace.

Getting into a namespace

ip netns list

Frequently empty on Docker hosts, and that's not a bug. ip netns only lists namespaces registered under /var/run/netns/, and Docker doesn't register them there. The namespaces exist; this command just can't see them.

The universal method — enter the namespace of a running process:

# Get the container's main PID
docker inspect -f '{{.State.Pid}}' <container>

# Run any network command inside its namespace
nsenter -t <pid> -n ss -lntp
nsenter -t <pid> -n ip -br addr
nsenter -t <pid> -n ip route
nsenter -t <pid> -n tcpdump -nn -i eth0

nsenter -t <pid> -n <command> is the key that unlocks the whole series inside containers. -n means "enter the network namespace only." The command runs with the host's filesystem and your host tools — so you get tcpdump, ss, and nstat inside a container that has none of them installed. That last point is worth pausing on: you do not need to install debugging tools in your images.

For Kubernetes, get the PID via crictl inspect on the node; the nsenter step is identical.

veth pairs — matching both ends

A container connects to the host through a veth pair: two virtual interfaces acting as a pipe. One end is eth0 inside the container, the other sits on the host, usually attached to a bridge.

Inside a container, ip link show gives you something like:

12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> ...

That @if13 is the interface index of the peer on the host. So:

# Inside the namespace — note the @ifN
nsenter -t <pid> -n ip link show eth0

# On the host — find index 13
ip link | grep '^13:'

Now you know exactly which host-side interface carries that container's traffic, and you can tcpdump it directly.

That trick — reading the peer index off eth0@ifN — turns "which of these forty veth interfaces belongs to my container" from guesswork into one command. It's the sort of thing that's obvious once and invisible before.

Bridges

ip -br link show type bridge
bridge link
bridge fdb show

bridge fdb show is the forwarding database — which MAC addresses the bridge has learned on which ports. The layer 2 equivalent of a routing table.

Where container traffic actually gets NAT'd

Docker publishes ports by inserting NAT rules. To see them:

iptables -t nat -L -n -v

You'll find DOCKER chains doing DNAT from the host port to the container's address. When a published port doesn't work, this is where to look — and the per-rule counters tell you whether packets are reaching the DNAT rule at all.

A rule of thumb worth internalising: on container hosts, check the nat table before the filter table. More container networking problems are NAT problems than firewall problems.

Unix domain sockets

Not networking in the wire sense, but they carry a lot of production traffic — nginx to php-fpm, most database local connections, container runtime APIs — and the series would have a hole without them.

A Unix domain socket is an endpoint identified by a filesystem path, not an IP and port. Communication never leaves the kernel: no IP, no TCP, no checksums, no packets.

ss -xlp          # listening unix sockets, with process
ss -xp           # all unix sockets

The "address" column is a path like /run/php/php8.2-fpm.sock.

Three things to know:

They don't appear in tcpdump. There are no packets. If nginx talks to php-fpm over a Unix socket, no amount of capturing will show you those requests. People lose real time to this.

Permissions matter. They're files, subject to file permissions. connect() to unix:/... failed (13: Permission denied) in an nginx log means the socket file's ownership or mode doesn't allow the nginx worker user. Not a network problem at all.

They're faster than loopback TCP — no protocol overhead — but only work on one machine. That's the tradeoff behind "should php-fpm listen on a socket or 127.0.0.1:9000."

What did we just build?

The layers the rest of the series assumed. When a packet arrives and vanishes, you can now find it:

Three habits from this post, if nothing else:

1. ip route get instead of reading the table. Ask the kernel what it will do.
2. Zero, reproduce, look. iptables -Z then check which counter moved. Stop reasoning about rule order; measure it.
3. nsenter -t <pid> -n. Every tool in this series, inside any container, without installing anything.

That's the series. The cheat sheet post collects all of it into one page, with a glossary and a learning path.

Primary sources: man ip-route, man ip-rule, man ip-neighbour, man iptables, man nsenter, nftables wiki, and Documentation/networking/ip-sysctl.rst in the kernel tree.

Compiled by AI. Proofread by caffeine. ☕

Three words, and almost no information. Slow where? Slow for whom? Slow by how much, compared to what?

Latency work is mostly attribution — cutting the total time into pieces until one piece is obviously the problem. Do that first and the fix is usually easy. Skip it and you'll spend a day tuning sysctls that were never involved.

Final part of the series. We'll build the attribution method, cover the queues where latency actually accumulates, and then work a full nginx investigation using everything from parts 1 through 4.

Latency is queueing

Almost every unexplained delay in a networked system is something waiting in a queue.

Actual transmission time is physics and it's small — light through fibre crosses a continent in tens of milliseconds and that number never changes. When something takes 800ms that should take 20ms, that extra 780ms was spent waiting: in a NIC ring buffer, a qdisc, a socket buffer, an accept queue, a thread pool, a connection pool, a disk queue.

So the question is never "why is the network slow." It's "which queue is it sitting in?" Every queue we've met in this series is a candidate:

Ten candidates. The job is elimination, and the fastest elimination tool is a timing breakdown.

Split the total time first

Before touching the server, measure from the client. curl will decompose a request for you, and this is the most useful thirty seconds in the whole investigation.

curl -w '
  dns:          %{time_namelookup}s
  tcp connect:  %{time_connect}s
  tls:          %{time_appconnect}s
  pre-transfer: %{time_pretransfer}s
  first byte:   %{time_starttransfer}s
  total:        %{time_total}s
' -o /dev/null -s https://example.com/

These are cumulative from the start of the request, so you subtract to get each phase:

• time_namelookup — DNS. If this is 2 seconds, your problem is DNS and nothing in this post applies.
• time_connect − time_namelookup — the TCP handshake. This is one round trip. It's your cleanest measurement of raw network RTT to that server, and it involves zero application code.
• time_appconnect − time_connect — the TLS handshake. Typically one or two additional round trips.
• time_starttransfer − time_pretransfer — time to first byte: the server thinking. This is where application latency lives.
• time_total − time_starttransfer — transferring the body. Large here with a small body means throughput problems.

The interpretation is mechanical:

• Connect time is high, TTFB is low → the network path is slow. The server is fine. Go look at RTT, routing, and loss.
• Connect time is low, TTFB is high → the network is fine. The server is slow. Stop reading tcpdump.
• Connect time is erratic — fast usually, occasionally terrible → queueing. Accept queue, or loss causing retransmitted SYNs.
• Everything is fine but users still complain → it's not this path. Different region, different client, or a specific endpoint.

Run it in a loop to see distribution rather than one sample:

for i in $(seq 20); do
  curl -w '%{time_connect} %{time_starttransfer} %{time_total}\n' \
    -o /dev/null -s https://example.com/
done

Averages hide everything. What you're looking for is the shape: are all 20 slow, or are 18 fast and 2 catastrophic? Those are different problems. Consistent slowness is a capacity or path issue. Occasional catastrophic outliers are queueing or loss — a retransmission timeout, which is a delay of seconds, not milliseconds.

Full list of curl timing variables is under -w in man curl.

Path latency vs application latency

If curl points at the network, narrow it further.

ping -c 20 203.0.113.7

Look at min/avg/max/mdev on the summary line. A tight spread means a stable path. A wide one means variable queueing somewhere along it.

mtr -rwc 100 203.0.113.7

mtr combines traceroute and ping — 100 probes to every hop, reporting loss and latency per hop. It's the right tool for "where along the path does it go bad."

One critical caveat that trips up almost everyone: loss reported at an intermediate hop is usually meaningless. Routers deprioritise generating ICMP replies for their own address; they'll happily show 40% "loss" while forwarding your actual traffic flawlessly. What matters is loss at the final hop, and loss that persists from a hop onward through all subsequent hops. Loss at hop 4 that disappears at hops 5 through 10 is an artifact. Loss that starts at hop 4 and continues to the end is real.

I've watched engineers escalate to a transit provider over hop-4 ICMP rate limiting more than once. Know this rule and you'll never be that person.

And remember the better measurement from part 2 — real RTT on the real connection carrying real traffic:

ss -tin state established dst 203.0.113.7

rtt: versus minrtt: is the tell. If minrtt is 2ms and current rtt is 180ms, the path can do 2ms and something is queueing right now. ping gives you a synthetic probe that may take a different path or priority; ss -ti gives you the truth about the connection you actually care about.

Throughput: window over RTT

One formula explains most "why is my transfer slow over a long distance" questions:

max throughput ≈ window size / round-trip time

A 64KB window over a 100ms RTT caps you at roughly 5 Mbit/s — regardless of whether the link is 10Gbit. The sender fills the window, then waits a full round trip for acknowledgement before it can continue.

That product — bandwidth × RTT — is the bandwidth-delay product, and it's the amount of data that must be in flight to keep a link busy.

Linux autotunes receive buffers by default (net.ipv4.tcp_moderate_rcvbuf), and the tunables are net.ipv4.tcp_rmem and net.ipv4.tcp_wmem — each three values: minimum, default, maximum. net.core.rmem_max and net.core.wmem_max cap what an application can request explicitly.

Do not go tuning these because a blog told you to. The defaults are sane for typical deployments, autotuning works, and badly-chosen values make things worse while consuming memory. This matters when you have genuinely high bandwidth-delay product paths — cross-continent bulk transfer, replication between distant regions. It does not matter for a web server serving clients 20ms away, and turning these knobs there is cargo cult. Read man 7 tcp before changing anything, and measure before and after.

If your traffic crosses long distances, the congestion control algorithm is worth a look:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control

CUBIC is the long-standing default. BBR takes a different approach — modelling bandwidth and RTT rather than treating loss as the congestion signal — and often performs considerably better on lossy long-distance paths. It's not universally better, and it's a change worth testing on your own traffic rather than adopting on faith.

Two classic latency traps

The 40ms mystery

If you have request/response traffic that's consistently slow by a suspiciously round ~40ms, you may be hitting the interaction between Nagle's algorithm and delayed ACK.

Nagle's algorithm buffers small writes, waiting to coalesce them into a full segment or until outstanding data is acknowledged. Delayed ACK, on the other side, holds back acknowledgements briefly hoping to piggyback them on outgoing data.

Put them together and you can get a standoff: the sender waits for an ACK before sending its small segment, and the receiver waits for data before sending its ACK. The delayed-ACK timer eventually fires and breaks the deadlock — after a fixed delay.

The signature is that suspicious consistency. Real network problems are noisy; this is metronomic. The application-level fix is TCP_NODELAY (disabling Nagle), which most modern servers and libraries already set. If you're seeing this pattern, check whether your application sets it.

Ephemeral port exhaustion

The thing that looks like a network outage on a perfectly healthy box.

Every outbound connection needs a local source port from net.ipv4.ip_local_port_range — by default roughly 28,000 ports. And per part 2, closed connections hold their port in TIME_WAIT for 60 seconds.

So a service making outbound connections at a high rate, to a single destination IP and port — a database, a cache, an internal API — can run out. New connections fail. CPU is low, memory is fine, the network is healthy, and nothing works.

ss -tan state time-wait | wc -l
sysctl net.ipv4.ip_local_port_range

Thousands of TIME_WAIT entries all pointing at one destination is the fingerprint.

The tempting fixes are widening the port range and enabling net.ipv4.tcp_tw_reuse. Both help, both are treating symptoms, and one of them has caveats you should read in tcp(7) before deploying.

The real fix is connection reuse. If your service opens a fresh connection for every request to the same backend, that's the bug. Connection pooling and HTTP keepalive eliminate the problem entirely instead of raising the ceiling on it. And as established in part 2: never tcp_tw_recycle. It's gone from the kernel and it broke NAT'd clients when it existed.

Disk latency wearing a network costume

Worth saying plainly, because it's a common misdiagnosis: application latency backs up into the network stack.

Your app is blocked on a slow disk read. It isn't calling read() on its sockets. Recv-Q climbs. It isn't calling accept(). The accept queue fills. Clients hang and time out.

Every symptom points at the network. The cause is a disk.

iostat -x 1

The columns that matter: r_await and w_await — average time in milliseconds for read and write requests, including queue time — and aqu-sz, the average queue depth. Column names vary between sysstat versions; check the header your box prints.

A word on %util: on spinning disks it meant something. On SSDs and NVMe, which service many requests in parallel, it does not mean saturation. A device at 100% %util with 0.2ms await is completely fine. Use await, not %util. People still make capacity decisions on that number and they're making them wrong.

For deeper work, biolatency from the BCC tools gives you a latency histogram rather than an average — and averages hide the tail that your users are actually experiencing.

Check CPU too, and specifically softirq time:

mpstat -P ALL 1

Per-CPU breakdown. The %soft column is software interrupt time — network processing. One core at 100% %soft while the rest idle is the imbalance from part 3, and it causes both drops and latency.

Case study: nginx is slow and sometimes returns 502

Now the whole series at once. Symptom: users report intermittent slowness, occasional 502 Bad Gateway. Load is up but not extreme. Nobody knows anything else.

Step 1 — Where is the time going?

for i in $(seq 20); do
  curl -w '%{time_connect} %{time_starttransfer} %{time_total}\n' \
    -o /dev/null -s https://oursite.example/
done

Say the result is: 16 requests with connect ~0.02s and TTFB ~0.05s, and 4 requests with connect ~1.02s.

Read that carefully. It isn't uniform slowness. Connect time is occasionally about a second — and 1 second is exactly the first SYN retransmission interval from part 4.

Hypothesis: some connections aren't getting through on the first SYN. Not application slowness. Something at connection setup.

Step 2 — Is the listener keeping up?

ss -lnt

State   Recv-Q  Send-Q  Local Address:Port
LISTEN  487     511     0.0.0.0:443

487 of 511. The accept queue is nearly full. Confirm it's overflowing:

nstat -az | grep -i listen

TcpExtListenOverflows is climbing.

Confirmed. Handshakes complete, connections sit in the accept queue, nginx isn't picking them up fast enough, and the queue overflows. Dropped final ACKs mean SYN-ACK retransmissions and that ~1 second penalty. Exactly the part-4 pattern.

Note what we have not done: no tcpdump, no sysctl tuning. Two commands.

Step 3 — Rule out the layers below

Be rigorous. Prove there's nothing else:

ip -s link show eth0             # errors, overruns — clean
cat /proc/net/softnet_stat       # column 2 zero, column 1 even across CPUs
nstat -az | grep -i retrans      # not moving
conntrack -S                     # no drops (or n/a on this box)

All clean. The wire is fine, softirq is balanced, the path isn't lossy, conntrack isn't full. The network is exonerated, and now you can say so with evidence rather than opinion.

Step 4 — Why isn't nginx accepting?

The accept queue is a symptom. Raising somaxconn and the backlog would give it a deeper queue to fall behind in. Find the actual cause.

Check the error log first, always:

tail -100 /var/log/nginx/error.log

Three messages nginx produces that answer this question immediately:

• worker_connections are not enough — you've hit the worker_connections limit. Every connection nginx handles, client-facing and upstream, consumes one.
• accept4() failed (24: Too many open files) — file descriptor limit. Check worker_rlimit_nofile in nginx and the process limit (LimitNOFILE under systemd; cat /proc/<pid>/limits shows what the running process actually has, which is the number that counts).
• upstream timed out — nginx is waiting on a backend. Workers tied up waiting can't accept new connections.

That third one is the common answer, and it's a nice reminder that "nginx is slow" is very often "the thing behind nginx is slow."

Step 5 — Split nginx time from upstream time

nginx exposes exactly the variables you need in log_format:

• $request_time — total time nginx spent on the request.
• $upstream_connect_time — time to establish the upstream connection.
• $upstream_header_time — time until upstream response headers arrived.
• $upstream_response_time — total upstream time.

If they aren't in your access log format, add them. Then:

• $upstream_response_time ≈ $request_time → nginx is a passenger. The backend is slow. Go there.
• $request_time ≫ $upstream_response_time → nginx or the client link is the delay. Slow clients, TLS overhead, or nginx resource limits.
• $upstream_connect_time high → connecting to the backend is slow. Backend accept queue full, or backend port exhaustion. Run this same investigation one layer down.

That recursion is the point. The method doesn't change; you just move a layer.

Exact variable semantics are in the nginx log module docs and the upstream module docs. Read them there, not here.

Step 6 — Check upstream connection reuse

If $upstream_connect_time is high and the backend looks healthy, check whether nginx is opening a new connection per request to it. By default in a proxy_pass setup, it does.

The keepalive directive in an upstream block enables a pool of persistent connections. Making it actually work also requires proxy_http_version 1.1 and clearing the Connection header — the exact configuration is documented in the upstream module docs linked above, and it's worth reading rather than pasting.

Without it, a high-traffic proxy burns through source ports and produces exactly the TIME_WAIT pile-up described earlier. Check:

ss -tan state time-wait | awk '{print $5}' | sort | uniq -c | sort -rn | head

If one backend dominates that list, connection reuse is your fix.

Step 7 — The fix, in order

1. Fix the actual cause — the slow upstream, the FD limit, the exhausted worker connections. This is the fix.
2. Enable upstream keepalive if nginx is proxying at volume.
3. Then, and only then, consider raising the accept queue — net.core.somaxconn and the backlog parameter on listen, together, since the kernel takes the minimum. A deeper queue absorbs bursts. It does not create capacity.

Order matters. Do 3 first and you've hidden the symptom and moved the failure to a worse place — instead of failing fast, connections now wait a long time and then fail.

What that investigation actually cost

Six commands and a log file. No packet capture, no kernel tuning, no guessing.

The path was: measure from outside → localise to a layer → prove the layers below are clean → find the cause → fix the cause, not the symptom.

That's the method. The commands are interchangeable. The method isn't.

The umbrella

Since this is the last part, here's the whole thing compressed.

Everything is a queue. Networking is producers, consumers, and buffers between them. Every problem is either "a queue is full" (drops) or "a queue is deep" (latency). Find the queue.

Every tool reads one layer. ethtool and ip -s link read the wire. /proc/net/softnet_stat reads packet processing. nstat reads the IP and TCP stack. ss reads sockets and the application boundary. tcpdump reads what's actually on the wire. Knowing which layer a tool reads is knowing when to run it.

Drops and retransmits are different evidence. Drop counters mean this box discarded something. Retransmits mean the path lost something. They point in opposite directions.

Localise before you tune. Every sysctl in every tuning guide is a guess until you know which layer is failing. Measure, localise, then change one thing and measure again.

The application is part of the network stack. Recv-Q, accept queues, CLOSE_WAIT, port exhaustion — every one of these is caused by application behaviour and diagnosed with network tools. "The network is slow" is, more often than not, "our code stopped reading its sockets."

Where to go next

Two directions, both worth it.

Down, into eBPF. bpftrace and the BCC tools let you ask questions counters can't answer: which connection retransmitted, how long each socket lived, exactly where in the kernel a packet was dropped. tcpretrans, tcplife, tcpconnect, biolatency. This is the current frontier of Linux observability and it's the clearest line between competent and genuinely strong. Start at the BCC repo and bpftrace.

Sideways, into the protocol. Read TCP/IP Illustrated, Volume 1 (Stevens; the second edition is updated by Fall). Nothing else builds the same depth. Then read man 7 tcp end to end — it is dense, unglamorous, and better than most books.

And the unfashionable advice: capture traffic for things that are working. Take a tcpdump of a healthy request. Watch a normal handshake. Read ss -ti on a connection that's fine. You cannot recognise broken until you know exactly what healthy looks like, and 3am is a terrible time to be learning it.

Next post is the cheat sheet — every command in this series in one page, plus a learning path.

Compiled by AI. Proofread by caffeine. ☕

Most engineers can start a capture. Far fewer can look at forty lines of output and say "the firewall is dropping this" or "the server's accept queue is full" without opening Wireshark. That skill is smaller than it looks. There are maybe eight patterns worth recognising, and once you know them, most captures answer themselves in ten seconds.

Part 4 of the series. This one is about the wire.

Where tcpdump sits, and why it matters

tcpdump taps traffic via AF_PACKET, close to the driver. The consequence is worth learning cold:

On receive, tcpdump sees packets before netfilter. So if iptables or nftables is going to drop that packet, tcpdump still shows it arriving.

That gives you the cleanest diagnostic split in Linux networking:

• tcpdump shows the SYN arriving, but your application never sees the connection → the packet reached the box and something in the kernel ate it. Firewall rules, conntrack, or the socket layer.
• tcpdump shows nothing at all → the packet never got here. Routing, upstream firewall, wrong interface, the client isn't actually sending.

One capture, and you've cut the problem in half.

The invocation

tcpdump -i eth0 -nn port 443

The flags that matter:

-i eth0     interface. -i any captures on all of them.
-nn         no DNS resolution, no port-name resolution. ALWAYS use this.
-c 100      stop after 100 packets.
-w file.pcap   write raw packets to a file.
-r file.pcap   read a file back.
-v / -vv    more detail (TTL, IP ID, checksum validation).
-e          show the ethernet header — MAC addresses and VLAN tags.
-A          print payload as ASCII.
-X          print payload as hex and ASCII.
-S          absolute sequence numbers instead of relative.
-tttt       human-readable absolute timestamps.

Three habits worth building:

Always -nn. Single -n skips host resolution; double skips port-name resolution too. Without it, tcpdump does a DNS lookup per address and your capture output stalls — during a network incident, possibly forever. It also renders port 443 as "https", which is noise when you're pattern-matching.

Always bound the capture. -c 500, or -w to a file with a size limit. An unbounded capture on a busy interface fills a terminal faster than you can read and can fill a disk.

For anything non-trivial, capture to a file. Read it on the box with tcpdump -r, or pull it down and open it in Wireshark. Live terminal output is for quick yes/no questions.

# The pattern for real work
tcpdump -i eth0 -nn -w /tmp/capture.pcap 'host 203.0.113.7 and port 443'
# ... reproduce the problem ...
# Ctrl-C
tcpdump -r /tmp/capture.pcap -nn | head -50

On modern versions the default snapshot length already captures full packets, so the old -s 0 habit is unnecessary. Harmless, just dated.

Two gotchas that will confuse you

-i any doesn't give you an ethernet header. It uses a special cooked-capture mode. If you need MAC addresses or VLAN tags, capture on the real interface.

Offloads make packets look impossible. As covered in part 3: GRO merges incoming segments before tcpdump sees them, TSO means outgoing segments are split after. You will see a 40,000-byte "packet" on a 1500-byte MTU link. Nothing is wrong. You're seeing a merged buffer, not a frame that existed on the wire. Remember this before you spend an hour on it.

Capture filters

The filter syntax is BPF, and it's the same language libpcap uses everywhere. It's worth learning properly because it's the difference between 40 relevant packets and 40,000 irrelevant ones.

# By host
tcpdump -nn host 203.0.113.7
tcpdump -nn src host 203.0.113.7
tcpdump -nn dst host 203.0.113.7

# By port
tcpdump -nn port 443
tcpdump -nn dst port 443
tcpdump -nn portrange 8000-8100

# By network
tcpdump -nn net 10.0.0.0/8

# By protocol
tcpdump -nn tcp
tcpdump -nn udp port 53
tcpdump -nn icmp

# Combined — quote it so the shell leaves it alone
tcpdump -nn 'host 203.0.113.7 and port 443 and tcp'
tcpdump -nn 'port 80 or port 443'
tcpdump -nn 'host 203.0.113.7 and not port 22'

That last one — excluding your own SSH session — is one you'll use every single time you capture on a box you're logged into. Otherwise you're watching yourself watch the traffic.

Filtering on TCP flags

This is the power move, and it's how you answer real questions in one line:

# Only SYNs — who is trying to connect to us?
tcpdump -nn 'tcp[tcpflags] & tcp-syn != 0'

# Only RSTs — who is getting rejected?
tcpdump -nn 'tcp[tcpflags] & tcp-rst != 0'

# Connection setup and teardown only, no data
tcpdump -nn 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'

The named constants are tcp-syn, tcp-ack, tcp-fin, tcp-rst, tcp-push, tcp-urg. Full syntax is in man pcap-filter, which is genuinely one of the better man pages on the system.

That middle command deserves its own callout. "Show me every reset on this box" is often the fastest route to a root cause, because RSTs are where things go wrong and someone is usually being explicit about it.

Reading a line

Here's a full handshake. This is the thing to be able to read at a glance:

14:23:45.123456 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 111 ecr 0,nop,wscale 7], length 0
14:23:45.123789 IP 10.0.0.5.443 > 203.0.113.7.51234: Flags [S.], seq 5000, ack 1001, win 65160, options [mss 1460,sackOK,TS val 222 ecr 111,nop,wscale 7], length 0
14:23:45.145123 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [.], ack 5001, win 502, length 0

Field by field:

• 14:23:45.123456 — timestamp, microsecond resolution. The deltas between lines are your latency measurement. In this capture, SYN to SYN-ACK is 333 microseconds — the server responded instantly. Client ACK comes 21ms later, which is the network round trip. Right there you've measured RTT without running ping.
• IP — IPv4. You'll see IP6 for v6.
• 203.0.113.7.51234 > 10.0.0.5.443 — source, then destination. Note tcpdump uses a dot before the port, not a colon. It reads oddly at first, then never again.
• Flags [S] — the TCP flags. The whole game. Decoded below.
• seq 1000 — sequence number. Relative to the connection start by default, which is what you want. Use -S for absolute.
• ack 1001 — the next byte expected from the peer.
• win 64240 — the receive window. How much the sender of this packet is willing to accept.
• options [...] — negotiated at handshake. mss is max segment size, sackOK means selective ACK is supported, wscale 7 is the window scale factor.
• length 0 — payload bytes. Zero for pure control packets.

The flag notation

This trips up everyone once. Decode it once, properly:

The dot is the ACK flag. That's the whole trick. [S] is a bare SYN, [S.] is SYN with ACK set. Once you see the dot as "ACK," the notation is obvious forever.

The window scale trap

win 502 on that third line looks alarmingly small. It isn't. The handshake negotiated wscale 7, so the real window is 502 × 2⁷ = 64,256 bytes.

The trap: if you start a capture on a connection that's already established, tcpdump never saw the handshake, so it can't apply the scale factor. Every window value it prints will be wrong by a large multiple. If window sizes look absurd, check whether you captured the handshake. Whenever you suspect a windowing problem, restart the capture and reproduce the connection from scratch.

The patterns

This is the part to keep. Eight signatures, each with a diagnosis.

1. SYN with no response — a firewall is dropping

14:23:45.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:46.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:48.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:52.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0

Same sequence number, retransmitted, roughly doubling intervals — 1s, 2s, 4s. That's TCP's exponential backoff.

Diagnosis: something is silently discarding the packet. A DROP firewall rule, a security group, a network ACL. Silence is the signature of DROP; a REJECT would send something back.

Client-side symptom: a long hang, then "connection timed out."

Now combine it with where you captured:

• Ran this on the server and you see the SYNs arriving but no SYN-ACK leaving? The packet reached the box. A local firewall rule is eating it. Check iptables -L -n -v / nft list ruleset and look at the counters — the rule that's dropping will have a rising packet count.
• Ran it on the server and you see nothing? The packet never arrived. Upstream: routing, cloud security group, network firewall.

Two captures, problem localised.

2. SYN then RST — nothing is listening

14:23:45.100 IP 203.0.113.7.51234 > 10.0.0.5.443: Flags [S], seq 1000, length 0
14:23:45.100 IP 10.0.0.5.443 > 203.0.113.7.51234: Flags [R.], seq 1, ack 1001, length 0

Immediate reset. Diagnosis: no socket is listening on that port, or a firewall configured to REJECT rather than DROP.

Client symptom: instant "connection refused."

The speed is the tell. Refused is fast; dropped is slow. If a user says "it fails immediately," you already know it's this class of problem before you capture anything. Confirm with ss -lnt from part 2 — and check the bind address, because a service bound to 127.0.0.1 produces exactly this from outside.

3. Handshake completes, then repeated SYN-ACK — the accept queue is full

14:23:45.100 IP client.51234 > server.443: Flags [S], seq 1000, length 0
14:23:45.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0
14:23:45.120 IP client.51234 > server.443: Flags [.], ack 5001, length 0
14:23:46.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0
14:23:48.101 IP server.443 > client.51234: Flags [S.], seq 5000, ack 1001, length 0

The client completed the handshake. The server keeps retransmitting its SYN-ACK as if it never got the final ACK.

Diagnosis: the server's accept queue was full when that ACK arrived. With net.ipv4.tcp_abort_on_overflow = 0 — the default — the kernel discards the ACK rather than resetting, and retransmits the SYN-ACK hoping the application catches up.

This is the packet-level signature of the problem from part 2. Confirm it in two seconds:

ss -lnt                            # Recv-Q at Send-Q on that listener
nstat -az | grep -i ListenOverflow # rising

The reason this matters: from the client's point of view it looks exactly like network loss. Same hang, same retransmissions. Teams have blamed their network provider for weeks over an application that was too slow to accept().

4. Duplicate ACKs and retransmission — genuine loss

IP server > client: Flags [P.], seq 1000:2000, length 1000
IP server > client: Flags [P.], seq 2000:3000, length 1000
IP client > server: Flags [.], ack 2000, length 0
IP client > server: Flags [.], ack 2000, length 0
IP client > server: Flags [.], ack 2000, length 0
IP server > client: Flags [P.], seq 2000:3000, length 1000

The client keeps acking 2000 — it's saying "I'm still missing the byte at 2000." After the third duplicate ACK the server fast-retransmits.

Diagnosis: real packet loss on the path. This is the pattern that justifies a call to the network team, and combined with a rising TcpRetransSegs from part 3, it's a solid case.

Note who's retransmitting. Loss on the path from server to client shows up as the server retransmitting.

5. Zero window — the receiver is drowning

IP client > server: Flags [.], ack 50000, win 0, length 0

win 0 means "stop sending, my buffer is full." The sender must wait and periodically probe.

Diagnosis: the receiving application isn't reading its socket fast enough. Same underlying condition as a climbing Recv-Q in part 2, seen from the other side.

Users experience this as a transfer that stalls, resumes, stalls. It is not a network problem — the network delivered everything perfectly. The receiver couldn't cope.

6. RST mid-connection — someone hung up

IP client > server: Flags [P.], seq 1000:1500, length 500
IP server > client: Flags [R.], seq 5000, ack 1500, length 0

An established connection, working fine, suddenly reset.

Common causes, roughly in order of frequency:

• The application crashed or the process was killed.
• A load balancer or proxy hit an idle timeout and tore the connection down.
• A firewall or NAT device dropped the connection from its state table — often because the connection was idle longer than the device's timeout. Then the next packet has no matching state and gets reset.
• The application deliberately reset, e.g. closing a socket with unread data in its receive buffer.

The idle-timeout case is the one to look for, because it has a fingerprint: connections die after a suspiciously round interval of inactivity. If everything breaks after exactly 350 seconds or exactly 5 minutes of idle, that's a middlebox timeout, not a bug in your code. TCP keepalives on a shorter interval than the middlebox timeout are the standard fix.

7. FIN vs RST — how it ended

# Graceful
IP client > server: Flags [F.], seq 2000, ack 5000, length 0
IP server > client: Flags [.], ack 2001, length 0
IP server > client: Flags [F.], seq 5000, ack 2001, length 0
IP client > server: Flags [.], ack 5001, length 0

Four packets: FIN, ACK, FIN, ACK. Both sides closed properly, all data delivered. This is what healthy looks like.

An RST instead means abrupt termination, and any data still in flight is discarded. Some RSTs are perfectly normal — plenty of software resets deliberately to avoid TIME_WAIT. A change in the FIN-to-RST ratio is the interesting signal, not the presence of resets.

8. Small packets fine, large packets stall — MTU blackhole

The signature: the TCP handshake completes instantly, small requests work, and any transfer over a few kilobytes hangs and retransmits forever.

Diagnosis: an MTU problem, with ICMP blocked.

Here's the mechanism, because this one is genuinely subtle. When a packet is too large for a link and has the "don't fragment" bit set, the router is supposed to send back an ICMP "fragmentation needed" message telling the sender the correct MTU. Path MTU Discovery depends on it.

Some network administrator, at some point, decided to "block ICMP" for security. So that message never arrives. The sender has no idea its packets are too big, keeps sending them at full size, and they keep vanishing. Small packets fit and get through, which is why the handshake works and everything looks superficially fine.

Confirming it:

# Send a large packet with DF set. Syntax varies between ping implementations.
ping -M do -s 1472 203.0.113.7

# The path MTU the kernel currently believes in
ip route get 203.0.113.7

# Trace where the MTU changes
tracepath 203.0.113.7

1472 bytes of payload plus 28 bytes of headers is exactly 1500. If that fails but -s 1400 succeeds, you have your answer. tracepath is specifically built to find where along the path the MTU drops.

This is endemic with VPNs, tunnels, and overlay networks, all of which reduce the effective MTU. And "we blocked ICMP for security" is the cause often enough that it's worth checking early. Check ping and tracepath syntax on your distro — flags differ between the iputils and BSD variants.

A workflow, not a command

When someone hands you "the connection doesn't work":

Capture on the server, filtered to the client:

tcpdump -i any -nn -c 100 'host <client-ip>'

Then read it against three questions:

1. Do you see the SYN? No → the packet never arrived. Look upstream: routing, cloud security groups, network firewalls. Stop capturing on this box.
2. Do you see a response? No → the packet arrived and something local ate it. Check local firewall rule counters and conntrack.
3. Does the handshake complete? Look at what happens next — repeated SYN-ACKs mean accept queue, an immediate RST means nothing listening, dup ACKs mean loss.

If you can, capture on both ends simultaneously. A packet that leaves the client and never appears on the server has been dropped in between, and you've proven it. That comparison is the single most powerful thing in this post, and it's the one people forget to do.

When to stop and open Wireshark

tcpdump is for quick answers on the box. For anything involving many connections, throughput analysis, or protocol decode above TCP, capture to a file and open it in Wireshark.

Wireshark's "Expert Information" and its TCP stream graphs will find things you'd miss reading text. tshark gives you the same engine on the command line. And Wireshark's display filters are a different, richer language than the capture filters above — that's a common source of confusion, not a mistake on your part.

Capture on the server, analyse on your laptop. That's the normal workflow, and it's not cheating.

What did we just build?

The ability to look at a capture and say what happened, in a sentence, without guessing.

The eight patterns cover the large majority of what you'll meet:

Part 5: latency. Where it hides, how to attribute it, and a full nginx walkthrough that uses everything in the series at once.

Primary sources: man tcpdump, man pcap-filter, and wireshark.org.

Compiled by AI. Proofread by caffeine. ☕

That sentence starts more wasted investigations than any other in infrastructure. It's not a diagnosis. It's a feeling. Packets can be discarded at six different places between the wire and your application, and each one has a different cause, a different counter, and a different fix.

This post is about replacing the feeling with a number.

Part 1 gave us the ladder. Part 2 covered the socket layer. Now we go down — stops 1 through 5, the layers ss can't see, where packets die before any socket ever hears about them.

The rule

A drop counter tells you where. A retransmit counter tells you the path lost something. They are not the same measurement and they are almost never both true at once.

If a router three hops away discards your packet, your NIC's drop counter stays at zero forever. Your NIC never saw the packet. The only evidence you will ever get is that TCP had to send it again.

Conversely, if your NIC's ring buffer overflows, that's not "the network" — that's your machine failing to keep up with traffic that arrived perfectly.

Get this backwards and you'll open a ticket with your network team about a problem that lives entirely inside your own kernel. It happens constantly.

nstat — start here

nstat reads the kernel's network counters from /proc/net/snmp and /proc/net/netstat and, unlike netstat -s, it's built to show deltas.

That matters more than it sounds. A counter reading 4,000,000 retransmits is meaningless — is that from the last minute or from 400 days of uptime? What you need is the rate right now.

# Absolute values, including zeros
nstat -az

# Just what changed since the last time you ran it
nstat

# Sample every second and print what moved
nstat -d 1

nstat with no arguments stores a snapshot and prints the difference on the next run. So the workflow during an incident is: run it once to set the baseline, wait, run it again. What prints is what's happening now.

To find a specific counter:

nstat -az | grep -i retrans
nstat -az | grep -i listen
nstat -az | grep -i drop

The counters worth knowing by name

Loss on the path:

• TcpRetransSegs — total segments retransmitted. The headline number for "is the network losing our traffic."
• TcpExtTCPSynRetrans — retransmitted SYNs specifically. If this is climbing, connection setup is failing, which points at a firewall or an overloaded listener rather than general congestion.
• TcpExtTCPTimeouts — retransmission timeouts fired. Worse than a fast retransmit: it means TCP waited out the full timer with no acknowledgement at all.
• TcpExtTCPLostRetransmit — a retransmission was itself lost. Bad path, or serious congestion.

Some retransmission is normal on the public internet. What you want is the ratio: retransmits against TcpOutSegs. A fraction of a percent is background noise. A few percent is a real problem. Ten percent and users are already complaining.

Listener overload:

• TcpExtListenOverflows — accept queue was full when a handshake completed. Covered in part 2; this is the cumulative proof.
• TcpExtListenDrops — all drops on listening sockets, overflows included.

Receive-side pressure:

• TcpExtTCPBacklogDrop — the socket was locked by its owning process, so incoming packets went to the backlog queue, and that filled. Your application is holding the socket too long.
• TcpExtPruneCalled / TcpExtRcvPruned — the kernel is under receive-buffer memory pressure and discarding queued data. Rare, and when it shows up it's usually the real answer.
• TcpExtTCPZeroWindowDrop — data arrived while the receive window was zero. The receiver said "stop" and the sender didn't.

Connections dying:

• TcpAttemptFails — outbound connection attempts that failed.
• TcpEstabResets — established connections terminated by RST.
• TcpOutRsts — resets we sent. A jump here usually means we're rejecting connections — to a closed port, or from an app slamming connections shut.

UDP, which has its own vocabulary:

• UdpNoPorts — datagrams arrived for a port nothing was listening on.
• UdpInErrors — receive errors.
• UdpRcvbufErrors — the important one. The socket receive buffer was full and datagrams were thrown away. UDP has no retransmission, so this is permanent, silent data loss. If you run DNS, syslog, or statsd, check this counter. People run these services for years without knowing it exists, quietly losing metrics.
• UdpSndbufErrors — same on the send side.

Counter names differ slightly across kernel versions, and new ones appear. nstat -az on the machine in front of you is the authoritative list. Don't trust a name from a blog post — including this one — if grep doesn't find it locally.

Interface counters — layers 1 and 2

nstat covers the IP and TCP layers. For the wire itself:

ip -s link show eth0

Two blocks, RX and TX. The exact column headers vary between iproute2 versions, so read the header your box prints rather than assuming. The ones you care about:

• errors — malformed frames. Checksum failures, framing errors. Real errors here mean physical problems: bad cable, bad SFP, duplex mismatch, dying port. Non-zero and climbing is a hardware conversation.
• dropped — the kernel received the frame and threw it away. Not a hardware error. Usually no buffer space, or no protocol handler.
• overrun / missed — the ring buffer overflowed. The NIC had packets and the kernel didn't collect them in time. This is the classic "box can't keep up" signal.
• carrier — link-layer problems. Flapping link.

For more detail:

ip -s -s link show eth0

Same data, broken out further.

ethtool — the driver's own numbers

ip -s link shows the kernel's generic view. The driver keeps its own, more detailed counters:

ethtool -S eth0

These counter names are driver-specific. An Intel ixgbe card, a Mellanox mlx5, and a virtio NIC in a VM all name things differently. Do not memorise names from a blog; grep what your card actually reports:

ethtool -S eth0 | grep -i -E 'drop|err|miss|discard|fifo'

Anything non-zero and climbing there is worth understanding. If a counter name isn't obvious, look it up in that specific driver's documentation — guessing at semantics from the name is how people end up chasing counters that are harmless on their hardware.

Ring buffers

If you're seeing overruns, this is the knob:

ethtool -g eth0

Prints the hardware maximum and the current setting for RX and TX ring sizes. Vendors frequently ship a default well below the maximum.

Raising the ring size (ethtool -G, syntax in man ethtool) gives the kernel more slack to absorb bursts. It is a genuine fix for burst-driven overruns. It is not a fix for sustained overload — a bigger queue that's permanently full just adds latency to the drops. And it's not free: bigger rings mean more memory and can hurt cache behaviour. Change it deliberately, measure before and after.

Offloads — and why your tcpdump will lie

ethtool -k eth0

Shows offload features. Two matter for troubleshooting:

• GRO / LRO (receive) — the NIC or driver merges multiple incoming segments into one big buffer before the stack sees it.
• TSO / GSO (send) — the stack hands down one huge buffer and the NIC splits it into wire-sized packets.

These are good for performance and terrible for your sanity, because tcpdump captures after GRO and before TSO. You will see 40KB "packets" on a link with a 1500-byte MTU and think you've found something profound. You haven't. You're seeing merged buffers.

Know this now so part 4 doesn't confuse you. If you need to see true wire packets, offloads can be disabled temporarily — check man ethtool for the exact syntax on your version, and remember it will cost throughput while it's off.

The file almost nobody knows

cat /proc/net/softnet_stat

One line per CPU, all values in hex. This is stop 2 on the ladder — packet processing in software interrupt context.

The first three columns are the ones to read:

1. processed — packets this CPU handled.
2. dropped — packets dropped because the per-CPU backlog queue was full. Governed by net.core.netdev_max_backlog.
3. time_squeeze — the number of times the NAPI poll loop ran out of budget with work still pending.

What to look for:

Column 2 non-zero — you are dropping packets in softirq processing. The interface counters may look fine. Nothing at the socket layer will tell you. This is a real, invisible, easily-missed drop.

Column 3 climbing — softirq processing is being cut off before it finishes. Under heavy load some of this is expected; growing steadily under normal load is not.

Wildly uneven columns 1 across CPUs — this is the big one. If CPU 0 processed 400 million packets and CPUs 1-31 processed almost nothing, all your network interrupt handling is landing on a single core. That core saturates, drops packets, and your monitoring shows 3% overall CPU usage because it's averaging across 32 cores.

This is one of the most common serious misconfigurations in Linux networking, and the only place it's visible is this hex file and /proc/interrupts. Multi-queue NICs with proper IRQ affinity spread the load. Check /proc/interrupts for how many queues your card exposes and where they're landing.

Since column 1 is a lifetime total, compare over an interval rather than trusting the absolute numbers:

watch -d cat /proc/net/softnet_stat

-d highlights what changes. Which column is moving tells you the story.

Conntrack — the silent killer

If your box does NAT, runs containers, or has any stateful firewall rules, the kernel keeps a table of every tracked connection. That table has a hard maximum.

sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count
conntrack -S

The first two are ceiling and current usage. When count approaches max, new connections get dropped. Not slowed. Dropped.

The symptom is brutal to diagnose: intermittent connection failures, no application errors, clean interface counters, healthy CPU. conntrack -S shows per-CPU statistics including drops and insert failures. The kernel also logs to dmesg when the table fills:

dmesg | grep -i conntrack

If you see "table full, dropping packet", you've found it. On container hosts this is a rite of passage. The conntrack-tools package provides the conntrack command; on some systems the sysctl path is net.nf_conntrack_max instead — check what exists on yours.

Outbound drops — the qdisc

Everything so far has been receive-side. Transmit has its own queue, and its own drops:

tc -s qdisc show dev eth0

Look for dropped, overlimits, and requeues in the statistics. If you have traffic shaping configured, or you're hitting the queue length limit, this is where it shows. On a box with no shaping and drops here, you're sending faster than the link can carry.

sar — the only tool that knows about yesterday

Every command so far tells you about now. Incidents are about twenty minutes ago, or last Tuesday at 03:00.

sar comes from the sysstat package and, when its collector is enabled, records system statistics to disk continuously.

Live:

sar -n DEV 1        # per-interface throughput and packet rates
sar -n EDEV 1       # per-interface ERRORS and drops
sar -n TCP,ETCP 1   # TCP connection rates, retransmits, resets
sar -n SOCK 1       # socket counts by protocol, including TIME_WAIT

Historical — this is the point of the whole tool:

sar -n DEV -f /var/log/sa/sa13     # data for the 13th of this month
sar -n EDEV -s 03:00:00 -e 04:00:00 -f /var/log/sa/sa13

The log directory path varies by distro (/var/log/sa on RHEL-family, /var/log/sysstat on some Debian-family systems), and on Debian/Ubuntu the collector is often disabled by default — installing sysstat isn't enough, you have to enable it. Check /etc/default/sysstat.

Strong opinion, and I'll die on this hill: enable sysstat on every production box you own, today. It is a few megabytes a day. The first time an incident resolves itself before you can log in and you can still say exactly what the retransmit rate was at 02:47, it pays for every byte. Every hour spent debugging blind is an hour you chose in advance by not enabling this.

Which counter answers which question

The table to actually use:

Which connection is retransmitting?

Counters give you totals. They can't tell you which connection is suffering — and that's usually the question you actually need answered.

Two ways up:

Per-connection, from part 2:

ss -tin state established

The retrans: field is per-connection. Slower to scan, but it points at a specific peer.

Live tracing. The BCC tools (package name is usually bcc-tools or bpfcc-tools depending on distro) include tcpretrans, which prints every retransmission as it happens with source, destination, and TCP state. There's also tcpconnect, tcpaccept, and tcplife.

That's the jump from "we have 0.3% retransmits" to "10.0.0.7 is retransmitting to 10.0.0.99 on port 5432." One of those gets fixed today. The tooling here moves fast, so get current syntax from the BCC repository rather than an old post.

What did we just build?

The ability to make a falsifiable claim. Not "we're seeing packet loss" but:

"TcpExtListenOverflows went from 0 to 14,000 in ten minutes while interface errors stayed at zero and retransmits didn't move. The network is fine. Our application stopped accepting connections."

That's a sentence that ends an argument between two teams. That's the entire job.

The order I'd work in, on a box I've never seen before:

1. nstat twice, thirty seconds apart — what's moving?
2. ip -s link — is the wire healthy?
3. ss -lnt — are the listeners keeping up?
4. /proc/net/softnet_stat — is one CPU drowning?
5. conntrack -S if the box does NAT or containers.

Five commands, under a minute, and you've eliminated most of the search space.

Part 4: when the counters aren't enough and you have to look at the actual packets. tcpdump, and reading SYN/ACK/RST like a sentence.

Primary sources: man ip, man ethtool, man nstat, man sar, and Documentation/networking/ in the kernel tree.

Compiled by AI. Proofread by caffeine. ☕

One command answers all of that. Most people run it wrong.

This is part 2 of the series. Part 1 built the map — the seven-stop path from wire to application. ss reads stops 6 and 7: the socket layer and how well your application is keeping up with it.

Why ss and not netstat

ss ships with iproute2 — the same package as ip. It reads socket state from the kernel over a netlink socket.

netstat comes from net-tools, which has been effectively unmaintained for years and isn't installed by default on most modern distros. It works by parsing /proc/net/tcp as text.

On an idle box the difference is invisible. On a box with 200,000 sockets, netstat -an takes ten seconds and burns CPU you can't spare during an incident. ss returns immediately. When you're the one holding the pager, that gap is the whole argument.

Use ss. Recognise netstat in old docs. Don't write new runbooks with it.

The flags that matter

Six letters cover 90% of real use:

-t   TCP sockets
-u   UDP sockets
-a   all sockets (listening AND established)
-l   listening sockets only
-n   numeric — do not resolve ports or hostnames
-p   show the process holding the socket (needs root for other users' sockets)

Always pass -n. Without it, ss does a DNS lookup for every peer address. On a busy host that turns an instant command into a thirty-second hang — and if DNS is the thing that's broken, your diagnostic tool hangs on the outage you're diagnosing. Skipping -n is the single most common way people make a bad day worse.

The two commands you'll type most:

# What is this box listening on?
ss -lntp

# Who is connected right now?
ss -antp

Reading the output

State   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port   Process
LISTEN  0       511      0.0.0.0:80            0.0.0.0:*           users:(("nginx",pid=1234,fd=6))
ESTAB   0       0        10.0.0.5:80           203.0.113.7:51234   users:(("nginx",pid=1235,fd=12))
ESTAB   4096    0        10.0.0.5:80           203.0.113.9:51402   users:(("nginx",pid=1235,fd=13))

Six columns. Each one earns its place.

State — where this socket sits in the TCP state machine. Full list below.

Local Address:Port — your side. 0.0.0.0:80 means "port 80 on every IPv4 address on this machine." 127.0.0.1:80 means loopback only — nothing outside the box can reach it, ever. That one line has explained more "but the service is running!" tickets than anything else in this post. [::]:80 is the IPv6 equivalent of 0.0.0.0.

Peer Address:Port — the other side's IP and port. This is your answer to "who is connecting to me and on which port." No extra tooling needed.

Recv-Q and Send-Q — these mean two different things depending on the state, and this is the part everyone gets wrong.

Recv-Q / Send-Q on an ESTABLISHED socket

• Recv-Q — bytes sitting in the kernel receive buffer that your application has not read() yet.
• Send-Q — bytes your application has written that the kernel has not yet got acknowledged by the peer.

So in that third line above, Recv-Q 4096 says: 4KB arrived from 203.0.113.9, and nginx hasn't picked it up.

The diagnostic split is clean:

A brief spike in either is normal — that's what buffers are for. Sustained and growing is the signal. Watch it over time rather than judging a single snapshot:

watch -n1 'ss -ant state established | head -30'

Recv-Q / Send-Q on a LISTENING socket

Completely different meaning. Here's where the real gold is.

• Recv-Q — the number of established connections currently sitting in the accept queue, waiting for your application to call accept().
• Send-Q — the maximum size of that accept queue.

So:

LISTEN  0    511    0.0.0.0:80

means: queue capacity 511, currently zero waiting. Healthy.

LISTEN  511  511    0.0.0.0:80

means: the accept queue is completely full. New connections are being dropped right now. Your NIC counters are clean, the network is fine, and clients are hanging or getting resets.

That one line is worth the whole post. Check it before you touch tcpdump.

The two queues nobody separates

When a client connects to your server, the connection passes through two kernel queues. Confusing them makes overload impossible to diagnose.

client SYN
    |
    v
[ SYN queue ]  ---- half-open. Handshake in progress. State: SYN-RECV.
    |             sized via net.ipv4.tcp_max_syn_backlog
    |  (handshake completes)
    v
[ accept queue ] -- fully established, waiting for the app.
    |               sized via min(listen() backlog, net.core.somaxconn)
    |  (app calls accept())
    v
your application

SYN queue full — usually a SYN flood, or a genuine connection storm. If net.ipv4.tcp_syncookies is enabled (it is by default on most distros), the kernel falls back to SYN cookies instead of dropping, which is exactly what you want.

Accept queue full — your application is too slow to accept(). This is far more common and far more misdiagnosed. The kernel completed the handshake, the connection is established and ready, and your app never picked it up.

What happens when the accept queue overflows depends on a sysctl:

• net.ipv4.tcp_abort_on_overflow = 0 (the default) — the kernel silently ignores the client's final ACK. The client thinks it's connected, waits, retransmits. The user experiences a hang. Nothing in your logs mentions it.
• net.ipv4.tcp_abort_on_overflow = 1 — the kernel sends a RST. The client fails fast with "connection reset by peer."

Default 0 is why this is so hard to spot. The failure mode is designed to be invisible and hope the app catches up.

Two places to see it. First, live:

ss -lnt          # Recv-Q at or near Send-Q on a listener

Second, cumulative:

nstat -az | grep -i listen

TcpExtListenOverflows counts accept-queue overflows. TcpExtListenDrops counts all drops on listening sockets. If ListenOverflows is climbing, stop looking at the network. Full detail on nstat in part 3.

For nginx specifically the queue size comes from the backlog parameter on the listen directive, capped by net.core.somaxconn. Raising one without the other does nothing — the kernel takes the minimum. Check the current values on your box with sysctl net.core.somaxconn and confirm the nginx side against the nginx listen documentation rather than trusting any blog's numbers, including this one.

And a caveat worth saying plainly: a full accept queue is a symptom. Raising the backlog buys you a deeper queue to fall behind in. Find out why the app stopped accepting.

The TCP states, and what each one means when it piles up

You'll see these in the State column. Individually they're routine. In bulk, each one is a specific diagnosis.

Two of these deserve their own section.

CLOSE_WAIT is always your fault

CLOSE_WAIT means the remote end sent a FIN and the kernel acknowledged it — and your application has not called close() on that file descriptor.

The kernel cannot fix this. It has done its job and is waiting on you. CLOSE_WAIT sockets do not time out. They sit there until the process closes the descriptor or dies.

Thousands of CLOSE_WAIT sockets on one process means that process is leaking file descriptors, and it will eventually hit its ulimit -n and start failing to accept anything at all — which looks, from outside, exactly like a network outage.

ss -antp state close-wait

If that returns a large number pointing at one PID, you have found a bug in that application. Not in the network. Restarting clears it; only a code fix stops it coming back.

TIME_WAIT is usually not your fault

TIME_WAIT appears on whichever side closed the connection first. The socket waits out a fixed interval — 60 seconds on Linux, compiled in, not tunable by sysctl — so that delayed packets from the old connection can't be misdelivered into a new one reusing the same 4-tuple.

Servers usually let clients close, so servers usually don't accumulate them. A load balancer or proxy that closes its own upstream connections does, and 20,000 TIME_WAIT entries there can be completely healthy.

ss -tan state time-wait | wc -l

It only becomes a real problem when you're making a very high rate of outbound connections to the same destination and exhausting source ports. That's part 5.

One hard rule: do not enable net.ipv4.tcp_tw_recycle. It broke connections from clients behind NAT, and it was removed from the kernel in 4.12 — so on any modern system the sysctl doesn't even exist. Any tuning guide that recommends it was written before 2017 and should be closed. net.ipv4.tcp_tw_reuse is a different and much safer setting; read tcp(7) before touching it.

Filtering — get to the answer directly

Piping ss into grep works, and it's how most people do it. ss has a real filter language that's faster and doesn't produce false matches on a port number that happens to appear in an IP address.

# Everything talking to one host
ss -antp dst 203.0.113.7

# Everyone connected to my port 443
ss -antp '( dport = :443 or sport = :443 )'

# Only established connections
ss -antp state established

# Established, to one specific client
ss -antp state established dst 203.0.113.7

# How many connections per client IP — the "who is hammering me" one-liner
ss -ant state established | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

That last one is the command to run when traffic spikes and you want to know whether it's one bad actor or genuine load. It answers the question in a second.

And for a fast overview:

ss -s

Totals by protocol and state. Good for "is this box holding 400 connections or 400,000?" before you decide where to dig.

Looking inside a connection

This is where ss stops being a listing tool and becomes a diagnostic one:

ss -tin

-i dumps the kernel's internal TCP state per connection. The output is dense. The fields worth knowing:

• rtt:12.5/6.25 — smoothed round-trip time and its variance, in milliseconds. This is measured RTT on the real connection, not a synthetic ping. It reflects the actual path this traffic takes, including any middleboxes.
• minrtt — the lowest RTT ever seen on this connection. Compare with rtt: if current RTT is far above minrtt, you're queueing somewhere.
• retrans:0/12 — retransmits currently outstanding / total on this connection. A non-zero total is direct evidence of loss on the path.
• cwnd — congestion window, in segments. How much the kernel is willing to have in flight. A small cwnd on a fast link means the stack has recently detected loss and backed off.
• bytes_sent / bytes_acked / bytes_received — lifetime totals for this connection.
• send 1.2Mbps / delivery_rate — the stack's own estimate of throughput.
• lastsnd / lastrcv / lastack — milliseconds since the last send, receive, and ack. Large values on a supposedly active connection mean it's idle or stuck.
• mss / pmtu — segment size and path MTU. Relevant when you suspect MTU problems, which get their own treatment in part 4.

Field names vary a little between kernel and iproute2 versions, and new ones get added. man ss on the box you're standing on is the authority.

The practical use: pick a slow connection and inspect it.

ss -tin state established dst 203.0.113.7

If rtt is 200ms and retrans is climbing, the path is bad and you can say so with evidence. If rtt is 0.4ms and retrans is 0 and the user still says it's slow, the network is exonerated — go look at the application. Being able to make that call confidently, in one command, is most of what this series is for.

What did we just build?

A complete picture of stops 6 and 7 on the map:

• Who's connected — the Peer Address column.
• Is my app keeping up — Recv-Q on established sockets, Recv-Q vs Send-Q on listeners.
• Is my app leaking — CLOSE_WAIT counts.
• Is the path lossy — retrans and rtt from ss -ti.

Notice what's missing. ss tells you nothing about packets dropped at the NIC, in softirq processing, or by conntrack — all of which happen below the socket layer and never reach a socket to be counted. If ss looks clean and things are still broken, that's your signal to go down the ladder.

Which is part 3: proving packet loss with counters.

Primary sources: man ss, man 7 tcp.

Compiled by AI. Proofread by caffeine. ☕

The fix is not more commands. The fix is a map. Once you know the path a packet takes from the wire to your application, every tool slots into a specific point on that path, and "which command do I use?" answers itself.

This is part one of a series on troubleshooting connectivity, packet loss, and latency on Linux. This post is all model, almost no commands. That is deliberate. The commands in parts 2 through 5 will be useless to you without it.

What a socket actually is

Start here, because almost everything else is downstream of it.

A socket is a kernel data structure. That's it. It is not a connection, it is not a port, and it is not a file — though the kernel deliberately makes it look like a file so you can read() and write() it.

When your process calls socket(), the kernel allocates that structure and hands you back a file descriptor — an integer that indexes into your process's file descriptor table. Everything you do afterward, you do through that integer.

The important part: the socket owns two buffers.

• A receive buffer. The kernel puts incoming data here. Your app drains it by calling read()/recv().
• A send buffer. Your app fills it by calling write()/send(). The kernel drains it onto the network.

Hold onto that, because it explains the single most useful pair of numbers in Linux networking:

• Recv-Q filling up means data arrived and your application is not reading it fast enough. That is an application problem.
• Send-Q filling up means your application wrote data and the kernel cannot get it onto the network fast enough. That is a network problem — or a receiver problem.

Two columns. Two completely different on-call pages. We'll read them for real in part 2.

Stream, datagram, raw

When you create a socket you pick a type. The three that matter:

SOCK_STREAM — TCP. A reliable, ordered, bidirectional byte stream. Emphasis on byte stream: TCP has no concept of messages. If the sender writes 100 bytes then 100 bytes, the receiver might read 200 bytes at once, or 37 then 163. Every framing bug in every network protocol ever written comes from someone forgetting this. TCP guarantees the bytes arrive, in order, exactly once. It guarantees nothing about how they're grouped.

SOCK_DGRAM — UDP. Messages, preserved as units. Send a 100-byte datagram, and the receiver reads exactly 100 bytes or nothing. No ordering, no delivery guarantee, no retransmission. Which is a feature: DNS, QUIC, and most video use it precisely because they'd rather handle loss themselves than wait for TCP's retransmit.

SOCK_RAW — you build the packet headers yourself. This is what ping and tcpdump-adjacent tooling use. Needs privilege. You will almost never write one, but knowing it exists explains why some tools need root.

There's also SOCK_SEQPACKET — reliable and ordered like TCP, but message-preserving like UDP. Real, used by SCTP and Unix domain sockets, rare in practice. Mentioned so the name isn't a mystery when you meet it.

The 4-tuple — the one thing to memorise

A TCP connection is uniquely identified by four values:

source IP : source port  ->  destination IP : destination port

That's the 4-tuple. The kernel uses it as a lookup key. Every packet that arrives gets matched against this key to decide which socket it belongs to.

Three consequences fall straight out of this, and all three show up in production:

One. Your web server can hold 50,000 connections on port 443. Port 443 isn't "used up" by the first one. Each connection has a different client IP or client port, so each is a distinct tuple. People find this confusing for years. It's just a composite key.

Two. A client machine making outbound connections to a single destination IP and port is limited by its own source ports — roughly 28,000 by default (net.ipv4.ip_local_port_range). Hit that ceiling and new connections fail while the box looks completely idle. This is ephemeral port exhaustion, and it's one of the great "the server is fine, why can't we connect" mysteries. Part 5.

Three. TIME_WAIT sockets hold a tuple hostage for 60 seconds after close. Thousands of them are usually normal. Sometimes they're not. Part 2.

The path a packet takes

Here is the whole journey, inbound. Every troubleshooting tool you'll ever use plugs in at one of these stops.

   the wire
      |
 [1]  NIC hardware  ........ ring buffer, offloads, link errors
      |
 [2]  driver / softirq ..... NAPI polling, per-CPU processing
      |
 [3]  netfilter / conntrack  iptables, nftables, connection tracking
      |
 [4]  routing ............. which interface, which next hop
      |
 [5]  TCP/IP stack ........ checksums, reassembly, TCP state machine
      |
 [6]  socket receive buffer  the kernel's holding pen for your app
      |
 [7]  your application ..... read() / recv() / accept()

Read it once more. That ladder is the whole series.

[1] The NIC. Packets land in a ring buffer — a fixed-size queue in memory the card writes into. If the kernel doesn't drain it fast enough, the card overwrites or discards. That's a drop, and crucially it happens before anything else can see it. Nothing above this layer will tell you it happened. You need ethtool and ip -s link counters, which is part 3.

[2] Driver and softirq. The kernel processes received packets in software interrupt context, on a specific CPU. If that CPU is saturated — and it can be saturated while 31 other cores idle — you drop packets. /proc/net/softnet_stat is the file that tells you. Almost nobody knows it exists.

[3] netfilter and conntrack. Firewall rules run here. Connection tracking, which is what makes NAT and stateful rules possible, keeps a table with a maximum size. When that table fills, new connections get dropped — silently, as far as the application is concerned. If you run containers or NAT, this bites eventually.

[4] Routing. Does the kernel even know how to reach that destination? Which interface goes out? Asymmetric routing and policy routing live here.

[5] The TCP stack. The state machine, retransmissions, window management, congestion control. Where "packet loss" in the classic sense becomes visible as retransmits.

[6] The socket buffer. Data has arrived and is correct. It's sitting in the kernel, waiting. If it sits too long, it's because of [7].

[7] Your application. It has to actually call read(). Or, for a listening socket, accept(). If it doesn't — because it's blocked on a database query, or the thread pool is exhausted, or it's stuck in GC — the queues back up beneath it and the symptom looks exactly like a network problem. It isn't.

Outbound is the same ladder in reverse, with a different key failure: the qdisc (queueing discipline) between the stack and the driver, where traffic shaping and its own drops live.

The three questions

Every connectivity investigation is one of three questions. Knowing which one you're asking saves an hour.

"Can they reach me at all?" — connectivity. Is the port listening, is the packet arriving, is a firewall eating it, is routing sane. Tools: ss -lnt, tcpdump, ip route get, firewall rules.

"Are we losing traffic?" — loss. Not "does it feel slow" but counters, at a specific layer. Tools: nstat, ip -s link, ethtool -S, /proc/net/softnet_stat.

"Why is it slow?" — latency. Which almost always means: something is queueing. Find the queue. Tools: ss -ti, ping/mtr, sar, and reading Recv-Q/Send-Q.

Mixing these up is the classic failure. Someone reports slowness, and an engineer spends two hours in tcpdump proving packets arrive — which was never in doubt. The question was never connectivity.

Why "packet loss" is usually the wrong phrase

When someone says "we have packet loss," ask where.

• Dropped at the NIC because the ring buffer overflowed? That's your box being overwhelmed. Fix locally.
• Dropped in the network between two hosts? That shows up as TCP retransmissions, not as a drop counter on your interface. Your interface never saw the packet.
• Dropped because the accept queue was full? The client sees a hang or a reset. Your NIC counters are pristine. Nothing is wrong with the network at all.
• Dropped by a firewall rule? Nothing anywhere reports "loss." The packet was deliberately discarded and your counters are clean.

Four completely different problems, one phrase. This is why the ladder matters more than the tools: it turns "packet loss" into a location.

What did we just build? A map with seven stops and a rule: before you run anything, decide which stop you're investigating. From here on, every tool in this series is introduced as "this reads layer N".

Which tools are worth your time

Blunt opinion, since you'll meet all of these:

Learn these. ss (sockets and TCP state — replaces netstat entirely and is dramatically faster on busy hosts), ip (addresses, routes, link counters), nstat (kernel network counters, with deltas), tcpdump (what's actually on the wire), ethtool (NIC-level truth), sar (historical data — the only one of these that answers "what happened at 3am").

Know they exist. netstat, ifconfig, route — deprecated, from the net-tools package, often not even installed on modern distros. Read them in old runbooks; don't write new ones with them. netstat -s is the one genuinely worth keeping around, and nstat does the same job better.

The upgrade path. bpftrace and the BCC tools (tcpretrans, tcpconnect, tcplife). These let you ask questions the counters can't answer — which connection retransmitted, not just how many. This is the line between senior and everyone else. Later in the series.

One opinion I'll defend: sar is the most underrated tool on this list. Every other tool tells you about now. Incidents are about 20 minutes ago. If sysstat isn't collecting on your production boxes, you are choosing to investigate every future incident with one hand tied.

Next

Part 2: ss properly. Who is connected to you right now, from which IP, on which port, in which TCP state — and what those two queue columns are trying to tell you.

Reference for anything here: the socket(2), tcp(7), and ip(7) man pages are the primary sources and are better written than most blog posts, this one included.

Compiled by AI. Proofread by caffeine. ☕

This post is the on-ramp. Vocabulary, then the seven commands that answer "is it reachable, is the port open, is DNS working, is TLS working, and who owns this socket." No kernel counters, no packet analysis — those come later in the series. Just the ground floor, done properly.

If you're already comfortable with dig and nc, skim the glossary and skip to part 1. If you're not, this is the post that makes the rest of the series readable.

Glossary

Get these straight once and a lot of confusion evaporates.

Interface — a network device as the kernel sees it. eth0, ens3, lo, docker0. Might be physical hardware, might be entirely virtual.

Loopback (lo, 127.0.0.1) — the interface that talks to the machine itself. Traffic on it never reaches a wire. A service bound to 127.0.0.1 is unreachable from anywhere else, permanently, no matter what your firewall says.

IP address — identifies a host. 192.0.2.10.

Subnet / CIDR — 192.0.2.0/24 means the first 24 bits are the network part, leaving 256 addresses. Hosts in the same subnet reach each other directly over layer 2; anything else goes via a gateway.

Gateway / default route — where packets go when the destination isn't in a directly-connected subnet. "The way out."

Port — a 16-bit number (1–65535) identifying which service on a host. Not a physical thing.

Well-known vs ephemeral ports — below 1024 are the reserved ones (22 SSH, 53 DNS, 80 HTTP, 443 HTTPS) and binding them traditionally requires privilege. The high range is the ephemeral pool the kernel picks from for outbound connections.

Socket — the kernel structure representing one endpoint of a communication. Part 1 goes deep.

The 4-tuple — source IP : source port → destination IP : destination port. The unique identity of a TCP connection. The single most useful concept in this entire series.

Listening vs established — a listening socket waits for connections. An established socket is one live connection. One listener on port 443 can have 50,000 established connections beneath it.

TCP — reliable, ordered, connection-oriented. Handshake first, retransmits what's lost. A byte stream, not messages.

UDP — fire and forget. No handshake, no retransmission, no ordering. Messages preserved as units. Fast, and lossy by design.

ICMP — the control protocol. Error and diagnostic messages: ping, "destination unreachable", "fragmentation needed". Not TCP or UDP, and it has no ports. Blocking it wholesale breaks things people don't expect.

Frame / packet / segment / datagram — the same data at different layers. Frame on the wire (layer 2, has MAC addresses). Packet at the IP layer (layer 3). Segment is TCP's unit, datagram is UDP's (layer 4). People use them interchangeably; being precise helps when reading man pages.

MAC address — layer 2 hardware address, 02:fc:00:00:00:05. Only meaningful within a single network segment.

ARP — how a host discovers the MAC address for an IP on its local segment. Part 6.

MTU — the largest packet an interface will send, typically 1500 bytes on Ethernet. Tunnels and VPNs reduce it, and that causes some of the nastiest bugs in networking.

MSS — the largest TCP payload per segment. Roughly MTU minus IP and TCP headers.

RTT — round-trip time. How long a packet takes to get there and back.

TTL — a hop counter in the IP header. Each router decrements it; at zero the packet is discarded and an ICMP error is returned. This is the mechanism traceroute exploits.

NAT — rewriting addresses in flight, so many private hosts share one public IP. Your home router does it. So does Docker.

DNS — names to addresses.

TLS — the encryption layer under HTTPS. A separate handshake after the TCP handshake, which is why it gets its own timing measurement.

The order to check things

Work bottom-up. Each step assumes the one before it passed.

1. Is the interface up and does it have an address?   ip -br addr
2. Do I have a route to the destination?              ip route get <ip>
3. Does the host respond at all?                      ping
4. Does the name resolve?                             dig / getent
5. Is the port open?                                  nc -zv
6. Does TLS work?                                     openssl s_client
7. What is the server actually doing?                 curl -v

Most people start at 5 and work backwards in a panic. Starting at 1 takes ninety seconds and eliminates half the possibilities.

1. Interfaces and addresses

ip -br addr
ip -br link

-br is "brief," and it's the flag that makes ip pleasant to use. Compare:

lo               UNKNOWN        127.0.0.1/8
eth0             UP             192.0.2.2/24

Two lines instead of twenty. ip -br link shows the same interfaces with MAC addresses and state.

What to look for: is the interface UP, does it have the address you expect, is the prefix length (/24) right. A wrong prefix length is a classic — the host thinks half the network is local and never sends those packets to the gateway.

ifconfig does a similar job and is from the deprecated net-tools package. It's often not installed. Use ip.

2. Routing — the one command

ip route get 8.8.8.8

8.8.8.8 via 192.0.2.1 dev eth0 src 192.0.2.2 uid 0
    cache

This is the best routing command on Linux and it's underused. It doesn't dump the table for you to interpret — it asks the kernel to make the actual decision and report it. You get:

• via 192.0.2.1 — the next hop it will use
• dev eth0 — the interface it will leave by
• src 192.0.2.2 — the source IP it will use, which is the one your firewall rules on the far end need to allow

That source address is the part people miss. On a multi-homed box, "which IP do we appear to come from" is not a guess — this command tells you.

Full table if you need it:

ip route

Policy routing and multiple tables are part 6.

3. ping — and its limits

ping -c 5 192.0.2.1

ping sends ICMP echo requests. -c 5 stops after five, which you want — otherwise it runs forever.

Read the summary line: min/avg/max/mdev. A tight spread means a stable path. mdev (mean deviation) high means jitter, which means queueing.

The critical caveat, and it's the most common junior mistake: a failed ping does not mean the host is down.

Plenty of hosts and networks drop ICMP by policy. Cloud security groups routinely block it by default. A host can be perfectly healthy, serving traffic on port 443, and completely silent to ping.

So: ping succeeding is useful information. Ping failing is almost none. Never report "the host is down" based on ping alone. Test the actual port instead — step 5.

Useful variants:

ping -c 10 -i 0.2 host      # faster interval, better sample
ping -M do -s 1472 host     # don't fragment, 1472+28 = 1500 — MTU testing

Flags differ between the iputils and BSD implementations. Check man ping on your box.

4. traceroute — and why it lies

traceroute 8.8.8.8

The mechanism, because it explains the weird output: traceroute sends packets with TTL=1, then TTL=2, then 3. Each router that decrements TTL to zero discards the packet and returns an ICMP "time exceeded" — revealing its address. Increment until you reach the destination.

Two things that confuse everyone:

Stars (* * *) do not mean a broken hop. They mean that router chose not to send an ICMP reply. Many are configured that way. Traffic still flows straight through. A hop of stars in the middle with the destination reachable at the end is completely normal.

Loss at an intermediate hop is usually fake. Routers deprioritise generating ICMP for their own address — it's handled by a slow control-plane CPU, not the forwarding hardware. A router showing 40% "loss" may be forwarding your real traffic flawlessly. Only loss at the final hop, or loss that persists from a hop through every hop after it, is real.

I have watched engineers escalate to a transit provider over hop-4 ICMP rate limiting. Know this and you won't be that person.

By default traceroute uses UDP to high ports, which firewalls often block. TCP mode usually gets further because it looks like real traffic:

traceroute -T -p 443 example.com     # may need root

mtr combines traceroute and ping, probing continuously — better for spotting intermittent problems. mtr -rwc 100 host gives a report of 100 probes per hop.

5. DNS — properly

The gap that causes the most wasted hours, because DNS failures don't look like DNS failures.

How a Linux host resolves a name

Applications do not just query DNS. They call getaddrinfo(), which follows the Name Service Switch configuration:

grep '^hosts' /etc/nsswitch.conf

hosts:          files dns

That means: check /etc/hosts first, then DNS. Other entries you'll meet are mdns4_minimal, resolve (systemd-resolved), and myhostname.

Then, for the DNS step:

cat /etc/resolv.conf

Nameservers, and a search domain list that gets appended to unqualified names.

On systemd-resolved systems, /etc/resolv.conf often just points at a local stub (127.0.0.53) and the real configuration lives elsewhere:

resolvectl status

The distinction that solves the hard cases

dig +short example.com          # queries DNS directly
getent hosts example.com        # goes through NSS — the same path your app uses

dig bypasses /etc/hosts and NSS entirely. It talks to a nameserver directly.

So when dig returns the right answer but your application can't resolve the name, the problem is not DNS. It's /etc/hosts, nsswitch.conf, or the resolver library. That divergence between dig and getent is the fastest way to find it, and almost nobody checks both.

dig, the parts you need

dig example.com                    # full answer with sections
dig +short example.com             # just the addresses
dig @8.8.8.8 example.com           # ask a specific server — bypass yours
dig MX example.com                 # a specific record type
dig -x 192.0.2.10                  # reverse lookup
dig +trace example.com             # follow delegation from the root

Two of these earn their keep:

dig @8.8.8.8 — if your resolver gives a stale or wrong answer and a public one gives the right answer, your resolver or its cache is the problem. One command, problem localised.

dig +trace — walks the delegation chain from the root servers down. This is how you find "the nameservers at the registrar don't match the ones in the zone," which produces intermittent failures that look like black magic.

Also watch the Query time: line in full dig output. A resolver taking 2 seconds is your latency problem, and no amount of TCP tuning will help.

host and nslookup do similar jobs. dig gives more detail and is what you'll see in every serious runbook. It ships in dnsutils or bind-utils depending on distro.

6. Is the port actually open?

nc -zv 192.0.2.10 443

• -z — scan only, don't send data
• -v — tell me the result

Bound the wait, or a filtered port hangs:

nc -zv -w 3 192.0.2.10 443

This, not ping, is how you test whether a service is reachable. It tests the exact thing that matters: can a TCP connection be established to that port.

Read the failure mode, because it tells you why:

• "Connection refused", instantly — the packet reached the host and nothing is listening. The path is fine, the service is down or bound to the wrong address.
• Hangs, then times out — something is silently dropping. A firewall with a DROP rule, a security group, or the host is unreachable. See part 4 for confirming which.

Fast failure and slow failure are different diagnoses. That distinction alone will save you hours.

A range:

nc -zv -w 2 192.0.2.10 20-25

UDP (-u) is much less reliable — no handshake means no confirmation, so "success" often means "no error came back." Don't trust a UDP scan result the way you trust a TCP one.

nc implementations differ. OpenBSD netcat, traditional netcat, and Ncat (from nmap) have overlapping but not identical flags. Check nc -h on the box. If nc isn't installed, curl -v telnet://host:port works as a fallback, and socat is the more powerful sibling worth knowing about.

7. Which process owns this port?

Three ways, all worth knowing:

ss -lntp                # the primary tool — part 2 covers it fully
lsof -i :443            # everything using port 443
lsof -i -P -n           # all network connections, unresolved
fuser 443/tcp           # just the PID

lsof -i is genuinely useful because it thinks in files, so it also shows you which files that process has open — often the fastest way to find a config file a service actually loaded. Use -P and -n to skip port-name and hostname resolution.

And when you need to see what a running process is actually doing at the syscall level:

strace -f -e trace=network -p <pid>

You'll see the socket(), connect(), accept(), sendto() calls in real time, with their return values. This is how you answer "is it even trying to connect, and where?" — a question no packet capture answers, because the packet may never be sent.

strace pauses the traced process on every syscall, so it is genuinely slow. Fine on a struggling service, dangerous on a hot one. Use it deliberately.

8. TLS

Connection works, HTTPS doesn't. Look at the TLS layer directly:

openssl s_client -connect example.com:443 -servername example.com

-servername sets SNI. Leave it off and you'll often get the wrong certificate, because one IP serves many sites and the server needs the name to pick. Then you spend twenty minutes confused about a certificate mismatch you created yourself.

What to read in the output: the certificate chain, Verify return code at the bottom (0 (ok) is what you want), and the negotiated protocol and cipher.

Certificate dates specifically:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Expired certificates cause a startling proportion of "the network is broken" incidents.

Quicker, for the common case:

curl -vI https://example.com/

-v shows the TLS handshake and certificate summary, -I fetches headers only. If curl fails but nc -zv to port 443 succeeds, the TCP layer is fine and your problem is TLS or HTTP — not the network.

9. Bandwidth

ping measures latency. It tells you nothing about throughput. For that:

# On the server
iperf3 -s

# On the client
iperf3 -c <server-ip>
iperf3 -c <server-ip> -R          # reverse — test the other direction
iperf3 -c <server-ip> -P 4        # 4 parallel streams

Two things worth knowing:

Test both directions. Asymmetric performance is common and single-direction tests miss it entirely.

A single TCP stream often can't fill a fast long-distance link. That's the window-over-RTT ceiling from part 5, not a broken network. If one stream gives you 50 Mbit/s and four parallel streams give you 200, you're seeing a per-connection window limit, not a bandwidth limit. Diagnosing that as "the link is slow" is a classic mistake.

iperf3 needs to be installed on both ends and it needs a port open between them.

Putting it together

"The app can't reach the database." Ninety seconds:

ip -br addr                          # do I have an address?
ip route get 10.0.0.50               # do I have a route, and from which source IP?
getent hosts db.internal             # does the name resolve the way the app sees it?
dig +short db.internal               # and does DNS itself agree?
nc -zv -w 3 10.0.0.50 5432           # is the port actually open?

Five commands and you'll usually know. And crucially, you'll know which layer to escalate to instead of saying "the network is broken."

Then read the failure:

• No route → routing or interface configuration. Part 6.
• getent fails but dig works → NSS, /etc/hosts, or resolver config. Not DNS.
• Both fail → DNS. Try dig @<other-server> to see if it's your resolver.
• nc refused instantly → reached the host, nothing listening. Check the service and its bind address with ss -lntp (part 2).
• nc hangs → something is dropping silently. Firewall. Part 4 proves it.

What did we just build?

The floor. Seven questions, in an order, each with one command and a way to read the answer.

The three things worth carrying out of this post:

1. Ping failing means almost nothing. Test the port, not ICMP.
2. dig and getent answer different questions. When they disagree, you've found the bug.
3. Fast failure and slow failure are different diagnoses. Refused means it arrived. Timeout means it didn't.

Part 1 goes under the hood: what a socket really is, and the seven-stop path a packet takes from the wire to your application.

Primary sources: man ip, man dig, man nc, man ping, man traceroute, man nsswitch.conf, man getaddrinfo.

Compiled by AI. Proofread by caffeine. ☕

At the end I audit a real reverse proxy on my own box and find three things wrong with it.

Where this design came from

nginx exists because of one specific problem, and knowing it makes every directive below make sense.

In 1999 Dan Kegel wrote up the C10K problem: how do you serve ten thousand concurrent connections on a single box? At the time the answer was "you don't," and the reason was architectural. Apache's classic multi-processing modules gave you one process per connection (prefork) or one thread per connection (worker). That model is easy to reason about and completely fine when connections are few and short. But the cost scales with the number of open connections, not with the amount of work — and on the public internet most connections are idle most of the time, waiting on a slow client, a keepalive timeout, or a dial-up link.

Ten thousand connections meant ten thousand processes or threads. Each one carries its own stack, its own scheduler entry, and its own memory footprint measured in megabytes rather than kilobytes. Long before you run out of CPU you run out of RAM, and the scheduler spends its time context-switching between threads that have nothing to do.

Igor Sysoev inverted it. nginx runs a fixed, small number of processes — one per core — and each one runs an event loop over thousands of sockets. Cost now scales with active work instead of open connections. Ten thousand idle keepalive connections are a few megabytes and no scheduler pressure at all, because an idle connection is just a file descriptor in an epoll set, not a thread waiting to be woken.

Apache closed a lot of this gap with the event MPM in 2.4, which hands idle keepalive connections off to a small dedicated pool instead of pinning a thread to each. It is a genuine improvement and the comparison is no longer the massacre it was in 2004. But a request actually in flight still occupies a worker thread, so the shape of the trade-off is unchanged: Apache's model is per-request-thread with an event-driven waiting room bolted on, nginx's is event-driven throughout.

The practical consequence, and the reason this matters for the rest of the article: nginx is cheap at holding connections and expensive at blocking. That is exactly the profile you want in a reverse proxy or TLS terminator sitting in front of an application server that does use a thread-per-request model — nginx absorbs ten thousand slow clients so your backend only ever sees the small number of requests that are genuinely doing work.

It is also why worker_connections exists as a directive at all. You are not capping a process pool. You are pre-sizing an event loop, telling one worker how many descriptors it should be prepared to hold at once.

The model you need in your head

nginx runs one master process and N worker processes.

The master does almost nothing interesting. It reads the config, binds the listening sockets, spawns workers, and handles signals. It does not serve traffic. Notably, the master opens the listening socket and the workers inherit it — which is why a reload can swap workers without dropping the socket.

Each worker is single-threaded and event-driven. It is not one-thread-per-connection. A worker sits in epoll_wait() holding thousands of connections, wakes up when any of them has something to say, does a small amount of non-blocking work, and goes back to sleep. This is why nginx holds 10,000 idle connections on almost no memory, and why one blocking operation in a worker stalls every connection that worker owns.

That last point is the source of most surprising nginx behaviour. A slow disk read, a synchronous DNS lookup, a blocking third-party module — while that happens, thousands of unrelated requests sit still. aio threads exists to push file I/O onto a thread pool for exactly this reason.

Three consequences worth internalising:

• CPU work parallelises across workers, not within one. One worker = one core, maximum.
• A single expensive request cannot be preempted. Event loops are cooperative.
• Load balance between workers is the kernel's job, not nginx's. More on that under reuseport.

The arithmetic that sets your ceiling

Two directives decide the maximum number of connections nginx will hold:

worker_processes  auto;   # one worker per CPU core
events {
    worker_connections  1024;   # per worker, and this default is the trap
}

The ceiling is the product:

max connections = worker_processes × worker_connections

worker_connections counts every connection a worker owns, not just the client-facing ones. This is where the arithmetic usually goes wrong, because "two per client" gets repeated as though it were a law, and it is only the common case.

What one client request actually costs:

So a fairly ordinary setup — auth_request in front of a proxy_pass to a resolved hostname — can hold four connections for one user, not two. WebSockets and gRPC streams are still two, but they hold both ends for the entire life of the session rather than milliseconds, which hurts a ceiling far more than a higher count would.

The general form:

max clients ≈ (worker_processes × worker_connections) / connections_per_client

Use 2 as a floor for any proxy, and count your own middleware before trusting it. The number you want is whatever your busiest path actually opens.

The default worker_connections 1024 has shipped for years and is far too low for anything under real load. On a 16-core box that is 16,384 total, or roughly 8,000 proxied clients — and people are routinely surprised by that number because nobody ever changed a default they never read.

When you hit it, nginx tells you plainly in error.log:

[alert] 1234#0: 768 worker_connections are not enough

That message is a gift. Most limits fail with something far more cryptic.

File descriptors: the limit that bites first

Every connection is a file descriptor. Every open log file, every cached file handle, every upstream socket. So the fd limit has to be larger than your connection ceiling, and it is the constraint people forget because it lives outside nginx.

There are three places a limit can come from, and they override each other in this order:

1. The kernel's global cap — fs.file-max. Rarely the problem on modern systems.
2. The process limit — RLIMIT_NOFILE. This is the one that bites.
3. nginx's own directive — worker_rlimit_nofile, which calls setrlimit() for workers.

Where RLIMIT_NOFILE comes from depends on how nginx starts:

• systemd ignores /etc/security/limits.conf entirely. It uses LimitNOFILE= in the unit.
  This is the single most common cause of "I raised the limit and nothing changed" — the file you edited was never consulted.
• Docker takes it from the daemon's default ulimits or --ulimit nofile=.
• worker_rlimit_nofile in nginx.conf overrides both for workers, up to the hard limit.

Never trust the config. Ask the running process:

# what the worker actually has, not what you configured
cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

# how many it is using right now
ls /proc/$(pgrep -f 'nginx: worker' | head -1)/fd | wc -l

The failure mode is unmistakable once you know it:

[crit] accept4() failed (24: Too many open files)

Errno 24 is EMFILE. nginx is alive, healthy, and refusing connections. A rule of thumb that has never let me down: set worker_rlimit_nofile to at least twice worker_connections, and make sure the hard limit is above that.

What the kernel does before nginx sees anything

A connection passes through two kernel queues before a worker ever touches it. If you tune nginx without understanding these, you tune the wrong thing.

1. The SYN queue holds half-open connections between SYN and the final ACK. Sized by net.ipv4.tcp_max_syn_backlog. Overflow here shows up as SYN cookies or dropped handshakes.

2. The accept queue holds fully established connections waiting for the application to call accept(). Sized by min(listen backlog, net.core.somaxconn).

That min() is where people get caught. nginx's listen ... backlog=N is capped silently by somaxconn. Ask for 65535 with somaxconn at 4096 and you get 4096, with no warning.

You can see both queues directly:

# on a LISTEN socket: Recv-Q = connections waiting to be accepted
#                     Send-Q = the accept queue's maximum
ss -lnt

# has the accept queue ever overflowed? this counter is the smoking gun
nstat -az | grep -E 'ListenOverflows|ListenDrops'

A non-zero and climbing ListenOverflows means the kernel completed handshakes that nginx never accepted. The client saw a connection succeed and then hang. That is nginx being too slow to accept — workers busy, blocked, or too few — not a network problem, and no amount of somaxconn will fix the cause. It only buys you a bigger waiting room.

The other kernel knobs that matter under real load:

SO_REUSEPORT

By default the master creates one listening socket and every worker calls accept() on it. The kernel wakes workers to compete for each new connection, and the distribution is uneven — under high connection rates you get one hot worker and fifteen bored ones.

One word fixes it:

listen 443 ssl reuseport;

Now each worker gets its own listening socket and its own accept queue, and the kernel hashes incoming connections across them. No contention, far better balance across cores. On connection-heavy workloads this is one of the largest single-line wins available.

Two things to know before you paste it in:

• Put reuseport on one listen directive per address:port. Repeating it across server
  blocks for the same socket is a config error.
• Reloads become slightly lossy. Each worker owns a private accept queue; when workers are
  replaced, connections sitting in an old worker's queue can be dropped rather than handed over. For most sites this is invisible. If you reload constantly under heavy load, know the trade.

Related, and usually best left alone: accept_mutex has defaulted to off since nginx 1.11.3, and reuseport makes it irrelevant. multi_accept on tells a worker to drain all pending connections per wakeup — it can raise throughput and can also raise latency variance. Measure, don't assume.

What to expect, in orders of magnitude

Precise numbers from someone else's hardware are worthless, but orders of magnitude stop you chasing the wrong bottleneck. Modern x86, per core:

• Static file from page cache, keepalive on: tens of thousands of requests/sec per core. nginx
  is rarely the limit here; your NIC or the kernel's packet path is.
• New TCP connection per request (no keepalive): an order of magnitude worse. Connection setup
  and teardown dominate. This is the single biggest self-inflicted wound in benchmarks.
• TLS handshakes: the expensive part, and the algorithm matters enormously. ECDSA P-256 is
  several times cheaper than RSA-2048 per handshake. Session resumption skips the asymmetric work entirely.
• Bulk TLS transfer: cheap on any CPU with AES-NI. Handshakes cost, streaming does not.

The practical takeaways: if you are TLS-terminating, your handshake rate is your throughput, and enabling session resumption plus moving to an ECDSA certificate will beat every sysctl in this article. If your benchmark opens a fresh connection per request, you are measuring connection setup, not nginx.

Triage: the order that finds it fastest

Work outside in. Each step rules out a layer.

Step 0 — make nginx observable. If you have not done this, do it before you need it:

# a timing-aware log format: this single change ends most "is it nginx or the app?" arguments
log_format timing '$remote_addr $status $request_time '
                  'upstream=$upstream_response_time connect=$upstream_connect_time '
                  'header=$upstream_header_time "$request"';

# and the built-in counters
location = /nginx_status {
    stub_status;
    allow 127.0.0.1;
    deny all;
}

$request_time is the whole request. $upstream_response_time is what your backend took. If those two are close, the problem is not nginx. That is usually the entire investigation.

Step 1 — how many connections do you actually have?

ss -s                                          # totals, incl. timewait
ss -tan state established '( sport = :443 )' | wc -l
curl -s localhost/nginx_status

stub_status returns something like Active connections: 291 and a line of three numbers: accepts handled requests.

If accepts and handled differ, nginx dropped connections — it hit worker_connections or ran out of file descriptors. Those two numbers should be identical forever. It is the fastest single check on the whole box, and almost nobody looks at it.

Step 2 — are you hitting a limit?

grep -c 'worker_connections are not enough' /var/log/nginx/error.log
grep -c 'Too many open files' /var/log/nginx/error.log
cat /proc/$(pgrep -f 'nginx: worker' | head -1)/limits | grep 'open files'

Step 3 — is the kernel dropping before nginx?

nstat -az | grep -E 'ListenOverflows|ListenDrops'
ss -lnt                      # Recv-Q climbing on a LISTEN row = you are not accepting fast enough

Step 4 — are you CPU bound, and is it one worker or all of them?

top -H -p $(pgrep -d, -f 'nginx: worker')

All workers pegged → genuinely CPU bound, likely TLS. One worker pegged while others idle → uneven distribution: reach for reuseport. That asymmetry is the tell.

Step 5 — the upstream side. Covered next, because it deserves its own section.

The upstream side: where reverse proxies die

Every outbound connection to an upstream consumes an ephemeral port from net.ipv4.ip_local_port_range. The usual range is 32768–60999, which is about 28,000 ports.

A port is not reusable immediately after close — it sits in TIME_WAIT for 60 seconds. So a proxy opening a fresh upstream connection per request tops out around 28,000 / 60 ≈ 470 connections per second to a single upstream (ip, port) pair, no matter how much CPU you have.

Then you get this, and it looks like a mystery:

connect() to 10.0.0.5:8080 failed (99: Cannot assign requested address)

That is not a network error. That is ephemeral port exhaustion.

The fix is not a sysctl. It is keepalive to your upstream, and it needs three lines that must all be present — the two proxy_* lines are the ones everybody forgets, and without them the keepalive directive silently does nothing:

upstream backend {
    server 10.0.0.5:8080;
    keepalive 64;               # persistent connections held per worker
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;          # required — 1.0 cannot keep alive
        proxy_set_header Connection "";  # required — strips the inbound "close"
    }
}

Widening ip_local_port_range and setting tcp_tw_reuse treat the symptom. Connection reuse removes the problem.

One more ceiling if you are behind Docker or any NAT: every connection consumes a conntrack entry. When nf_conntrack_count approaches nf_conntrack_max, the kernel starts dropping packets and logs nf_conntrack: table full, dropping packet. It presents as random, unexplained connection failures under load.

sysctl net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_count

What not to tune

• tcp_tw_recycle — do not look for it. It broke clients behind NAT and was removed from
  Linux in 4.12. Any guide still recommending it was written for a kernel you are not running.
• Raising somaxconn to "fix" overflows — it enlarges the waiting room. If ListenOverflows
  climbs, workers are not accepting fast enough; find out why.
• worker_processes above your core count — workers then compete for the same cores and you
  add context switching for nothing. auto is right almost always.
• Copy-pasted sysctl blocks from blog posts, applied wholesale. Every knob in this article
  fixes a specific, observable symptom. If you cannot name the counter that made you change it, do not change it.

Worked example: auditing a live reverse proxy

Theory is cheap. Here is my own nginx-proxy container, which has been running untouched for five months, on a 16-core host.

# nginx version: nginx/1.29.2
worker_processes  auto;
events { worker_connections  1024; }

auto on 16 cores gives 16 workers, which I confirmed rather than assumed:

$ docker top nginx-proxy -o pid,args
nginx: master process nginx -g daemon off;
nginx: worker process        ← ×16

Finding 1 — the connection ceiling is a stock default nobody chose. 16 × 1024 = 16,384 connections; as a reverse proxy that is roughly 8,000 concurrent clients. Not a crisis for a homelab, but it is the number I would hit first under load, and it is there by accident rather than decision.

Finding 2 — file descriptors are not the constraint, by luck.

$ grep 'Max open files' /proc/<worker>/limits
Max open files    1048576    1048576    files
$ ls /proc/<worker>/fd | wc -l
25

A million fds, 25 in use. That headroom comes from the container image's defaults, not from anything in my config — worker_rlimit_nofile is not set. On a systemd host with a stock LimitNOFILE, the same config would fail long before worker_connections was reached. The constraint here is worker_connections, which is the healthy way round — but only by accident.

Finding 3 — nginx is unobservable. No stub_status, and the log format is stock main with no timings. I cannot answer "did nginx drop connections?" or "was it nginx or the upstream?" without changing the config first — which is exactly the wrong time to be editing config.

The kernel side, for contrast, is healthy:

net.core.somaxconn              4096
net.ipv4.tcp_max_syn_backlog    2048
net.ipv4.ip_local_port_range    32768  60999      # 28,232 ports
net.ipv4.tcp_tw_reuse           2                 # loopback only — the modern default
nf_conntrack_max                262144            # in use: 385

TcpExtListenOverflows           0
TcpExtListenDrops               0

Zero overflows since boot: nothing has ever waited in the accept queue. tcp_max_syn_backlog (2048) sitting below somaxconn (4096) is worth evening out, and netdev_max_backlog is at the stock 1000, which only matters at packet rates I will never see here.

And the trap I hadn't thought about until I looked. I checked the listening sockets on the host:

$ ss -lntp '( sport = :80 or sport = :443 )'
LISTEN 0 4096   0.0.0.0:80    users:(("docker-proxy",...))
LISTEN 0 4096   100.96.60.8:443  users:(("tailscaled",...))

Neither of those is nginx. Port 80 is held by docker-proxy and 443 by tailscaled. nginx's own listening socket lives inside the container's network namespace:

$ docker exec nginx-proxy netstat -lntp
tcp  0  0 0.0.0.0:80   LISTEN   1/nginx: master pro

So every accept-queue number I could have read on the host describes docker-proxy's queue, not nginx's. If you run nginx in a container and debug queues from the host, you are measuring the wrong socket. Use nsenter into the container's netns, or run the tools inside it.

That one is worth the price of admission. It is invisible until it wastes an afternoon.

The one-screen checklist

# 1. what is the ceiling?
nginx -T | grep -E 'worker_processes|worker_connections'

# 2. did nginx drop anything? (accepts must equal handled, forever)
curl -s localhost/nginx_status

# 3. what limit does the worker really have?
cat /proc/$(pgrep -f 'nginx: worker'|head -1)/limits | grep 'open files'

# 4. did the kernel drop before nginx?
nstat -az | grep -E 'ListenOverflows|ListenDrops'
ss -lnt

# 5. one hot worker, or all of them?
top -H -p $(pgrep -d, -f 'nginx: worker')

# 6. running out of ephemeral ports?
ss -s
grep 'Cannot assign requested address' /var/log/nginx/error.log

# 7. nginx or the backend?  ($request_time vs $upstream_response_time)
tail -f /var/log/nginx/access.log

If you take one thing: accepts must equal handled in stub_status. The moment they diverge, nginx is refusing work, and you have exactly two suspects — worker_connections and file descriptors.

You know the feeling. ps aux scrolls past. Something is listening on port 5000 and you have no idea what put it there. You start the ritual: lsof -i :5000, then ps -p <pid>, then systemctl status, then docker ps, stitching the story together in your head. witr does that stitching for you.

What it is

witr — short for "Why Is This Running?" — is a single-binary CLI (with a TUI) that traces any process, port, container, or file back to the exact chain that started it. Point it at a name, a PID, a port, or a container, and it hands you a causal ancestry chain: systemd → pm2 → node, the working directory, the git branch, the bound sockets. One command instead of five.

The tools we already have — ps, top, lsof, ss, systemctl, docker ps — are great at showing what is running. They leave the why to you. witr makes causality explicit.

Why it's worth your time

The value isn't a prettier ps. It's the ancestry chain. When a process shows up and you don't recognize it, the real question is never "what is this" — it's "who started this, and what's keeping it alive." That answer is normally spread across a supervisor, maybe a container runtime, maybe a stray SSH session, maybe cron. witr collapses all of that into one narrative.

For anyone running a self-hosted box with a dozen services layered through systemd, Docker, and PM2, this is the tool you didn't know you were manually emulating every time something misbehaved.

Hands on

Install is a single binary, and it's already packaged nearly everywhere. On Debian sid / Ubuntu 26.04+ (and Kali, Devuan, Raspbian) it's in the official repos:

sudo apt install witr

Homebrew, on macOS or Linux:

brew install witr

Or the install script for the freshest release:

curl -fsSL https://raw.githubusercontent.com/pranshuparmar/witr/main/install.sh | bash

(That last one pipes a script to bash — read it first if that makes you nervous. It should.)

Now the good part. Ask it about a name:

witr node

Target      : node
Process     : node (pid 14233)
User        : pm2
Command     : node index.js
Started     : 2 days ago (Mon 2025-02-02 11:42:10 +05:30)

Why It Exists :
  systemd (pid 1) → pm2 (pid 5034) → node (pid 14233)

Source      : pm2
Working Dir : /opt/apps/expense-manager
Git Repo    : expense-manager (main)
Sockets     : 127.0.0.1:5001 (TCP | LISTENING)

That "Why It Exists" line is the whole pitch. It read the supervisor, walked the chain, and told you PM2 is keeping this alive — plus which repo and branch it's running from. That last detail alone has saved me a cd and a git status.

Chase a port instead, ancestry only:

witr --port 5000 --short

systemd (pid 1) → PM2 v5.3.1: God (pid 1481580) → python (pid 1482060)

Or get the full tree, including children:

witr --pid 143895 --tree

Every target flag — --pid, --port, --file, --container — is repeatable and mixable. --container searches across Docker, Podman, nerdctl, k8s/crictl, Incus, LXC, LXD, and FreeBSD jails, so witr --container redis finds it regardless of runtime.

Two things worth knowing. First, --json gives you machine-readable output, and witr returns real exit codes (0 clean, 1 warnings, 2 not found, 3 permission denied) — which means it slots into scripts and CI, not just interactive poking. Second, run witr with no arguments and you get the TUI: live tabs for processes, ports, containers, and file locks, with an ancestry side panel and the ability to send signals right from the UI.

The warnings deserve a mention too — it flags things like a process listening on 0.0.0.0, running as root, holding dangerous capabilities, or showing LD_PRELOAD injection indicators. Not a security scanner, but a useful nudge.

Honest verdict

I like this one. It's not solving an impossible problem — everything witr reports, you could dig out by hand. But "could" is doing a lot of work in that sentence, and the whole point is that you stop doing it by hand. The ancestry chain is genuinely the right abstraction for the question, and the tool is honest about uncertainty rather than pretending it always knows.

The caveats are inherent, not sloppy: process ancestry is a best-effort, real-time walk, so very short-lived processes can vanish before witr catches them. Detection is "best effort with explicit uncertainty" by design — it won't lie to you, but it also won't always have a clean answer. And realistically, some of the deeper output needs root to see everything (that 3 exit code exists for a reason).

Who's it for? Anyone who administers a box with more than a couple of moving parts. If your whole world is one Docker Compose file, you'll reach for it occasionally. If you're running a real self-hosted stack, it earns a spot in your PATH.

Go try it

Install it, then just run witr with no arguments and poke around the TUI for two minutes. After that, next time something's on a port you don't recognize, type witr --port <n> and watch it tell you the whole story. That's the moment it clicks.

Project: github.com/pranshuparmar/witr — and there's a browser sandbox if you want to try it before installing anything.

Grammarly sends everything you type to their servers. LanguageTool needs a 16GB n-gram dataset just to get started. There's a third option that most people haven't heard of yet — and it's better than both for most use cases.

What It Is

Harper is an offline grammar checker that runs entirely on your device. No cloud round-trips, no telemetry, no subscription. It ships as a Language Server Protocol binary called harper-ls, which means it integrates directly into any editor that speaks LSP: Neovim, VSCode, Helix, Emacs, Zed. Written in Rust. Responds in under 10ms. Made by Automattic — the people behind WordPress — and fully open source under Apache 2.0.

Why It's Worth Your Time

The competition got this wrong in two different ways.

Grammarly built a cloud product with surveillance-grade data collection baked in. Their business model depends on knowing what you write. They claim they don't sell your data, but that's a different thing from not using it. Every keystroke in every document goes to their servers. The round-trip latency makes revisions tedious. The suggestions lack context and are often just wrong.

LanguageTool went the opposite direction and built something technically powerful but impractical: the full offline version requires gigabytes of n-gram data and significant RAM just to run. It's overkill for most people and slow even when you have it set up.

Harper is neither. Single binary. Fraction of LanguageTool's memory footprint. Completely private.

What makes it genuinely interesting for developers: it doesn't just check .md files. It checks the English inside your code comments — Python docstrings, Rust doc comments, Go comments, Shell scripts, JavaScript, and about 30 more languages. Your inline documentation is fair game too, which is where most developer writing actually happens.

Hands On

Install on Debian/Ubuntu (via Cargo):

cargo install harper-ls --locked

Make sure ~/.cargo/bin is in your $PATH. You may need build-essential on Debian-based distros.

Arch Linux:

sudo pacman -S harper

Homebrew (macOS/Linux):

brew install harper

No Rust? No problem. Pre-built binaries are available on the GitHub releases page.

Neovim + mason.nvim: harper-ls is in the Mason registry — search and install it from there.

VSCode: There's an official extension in the marketplace.

Once installed, Harper creates a plain-text user dictionary at ~/.config/harper-ls/dictionary.txt. When it flags a term you use intentionally — a product name, a technical acronym, a brand — you add it once and Harper stops flagging it. No UI required, just edit the file.

LSP integration means code actions work natively. Hover over a flagged word and your editor offers: fix the spelling, ignore for the session, or add to the user dictionary. It catches capitalization errors, repeated words, wrong a/an usage, standard spelling mistakes. No LLM involved — these are deterministic rules, which means it's fast and predictable.

You can also select a dialect: American, British, Australian, Canadian, or Indian English. Useful if you write for specific audiences.

A sample LSP config for Neovim (using nvim-lspconfig):

require('lspconfig').harper_ls.setup {
  settings = {
    ["harper-ls"] = {
      linters = {
        SpellCheck = true,
        RepeatedWords = true,
        LongSentences = true,
        AnA = true,
      },
      diagnosticSeverity = "hint",
      dialect = "American",
    }
  }
}

This gives you grammar hints in your editor as you type — not blocking errors, just quiet nudges.

Honest Verdict

English-only for now. That's a real limitation — if you write in Arabic, French, or any other language, you'll constantly fight false positives. The team says other languages are planned, but there's no timeline.

The grammar rules are also narrower than Grammarly's full suite. Don't expect it to restructure awkward sentences or suggest better phrasing. Harper catches the mechanical stuff: spelling, repeated words, capitalization, a/an. That's the right scope for a tool running locally with no AI backend.

What it does, it does well and without friction. For anyone who writes documentation, technical blog posts, or just cares about the English in their code comments — this is exactly the kind of tool you install once and forget about until you notice it quietly preventing the same three mistakes you always make.

It's not a Grammarly replacement for heavy prose editing. It's a lightweight, always-on safety net that respects your privacy. That's a useful thing to have.

Go Try It

Install harper-ls and hook it into your editor. Write something. Watch the underlines appear in under 10ms. That response time alone will tell you why this thing exists.

→ writewithharper.com | GitHub

Every time I need to move a file between two machines, I open a terminal and immediately regret my life choices. scp requires SSH access on both ends. rsync does too. Spinning up a quick Python HTTP server means knowing the sender's IP and punching through a firewall. croc sidesteps all of it.

What It Is

croc is a command-line file transfer tool written in Go. The sender runs croc send file.tar.gz, gets a short code phrase back, and shares it with whoever needs the file. The recipient runs croc <code-phrase> on their end. The transfer happens — end-to-end encrypted, no port-forwarding needed, across NAT, on any platform. Windows, Linux, Mac, doesn't matter.

It's not magic. The trick is a relay server that handles the connection handshake, so you don't need a direct path between machines. But your content is encrypted using PAKE (Password-Authenticated Key Exchange) before it ever leaves your machine — the relay sees ciphertext, not your files.

Why It's Worth Your Time

What separates croc from similar tools like magic-wormhole (which directly inspired it):

Transfers resume. Drop a connection halfway through a 2 GB backup and it picks up where it stopped. That alone is worth the install.

Multi-file and folder support. Send a whole directory: croc send /path/to/folder. Exclude what you don't want: --exclude "node_modules,.venv".

Self-hostable relay. This is the part I care about most for self-hosters. By default, croc uses a public relay. But you can run your own in one command, and then your traffic never touches a third-party server.

Tor proxy. If you need it: --socks5 "127.0.0.1:9050".

Piping works. cat archive.tar.gz | croc send and croc --yes <code> > output.tar.gz both work cleanly. Good for scripts.

Hands On

Installation

On Arch Linux:

pacman -S croc

On Fedora/RHEL:

dnf install croc

Via the install script (works on most Linux distros):

curl https://getcroc.schollz.com | bash

Or build from source if you have Go 1.22+:

go install github.com/schollz/croc/v10@latest

Always verify the install script against the official repo before piping to bash.

Sending a file

croc send backup.tar.gz