Data Center Part 8: The cheat sheet

Everything in this series on one page. Bookmark this one and skip the rest.

Vocabulary, in one line each

Rack numbers

• 19 in / 482.6 mm — mounting face width (EIA-310). The cabinet is wider.
• 1U = 1.75 in = 44.45 mm. Holes in uneven groups of three per U.
• Cabinet width: 600 mm, or 800 mm when you want side cable channels.
• Cabinet depth: 600 / 800 / 1000 / 1200 mm. Datacenter servers need 800+.
• 600 mm deep = network cabinet. Will not take a server.
• 0U = mounts vertically in the side channel. PDUs. Costs you no U.
• Heavy at the bottom. Always.
• U1 is usually the bottom, but not universally. Check your rails and write it down.
• Multi-U devices are referenced by their lowest U.

Power

Usable capacity at 80 % of breaker rating (US NEC continuous-load rule; elsewhere the same practical outcome via PDU rating and contract):

Three-phase: kW = 1.732 × V × A.

The redundancy rule: in a 2N cabinet, 40 % per feed is full. Either feed must be able to carry the whole cabinet at 80 %. Running both feeds at 70 % means a single feed failure trips the survivor — that is swing load, and it turns one failure into a total outage.

IEC connectors — odd is socket, next even up is the matching inlet:

C15/C16 is the heat-rated variant with a notch — not interchangeable with C13.

NEMA input plugs (PDU → whip): L6-30P (208 V 30 A 1Ø, most common), L6-20P, L21-30P (3Ø wye), L15-30P (3Ø delta). L = twist-locking.

PDU tiers: basic → metered (total draw) → monitored (per outlet) → switched (per-outlet relays). Get switched and monitored; it is the only remote power cycle that works when the BMC is wedged.

Cables and optics

Reach:

Form factors — Q means four lanes:

Breakout: 1×100G → 4×25G, 1×40G → 4×10G. Because Q is four lanes.

Fibre — the jacket colour is reliable:

Transceiver naming: SR = short reach, multimode. LR = long reach, singlemode, 10 km. ER/ZR = further. A trailing 4 means four lanes, so eight fibres, so MPO rather than LC.

Connectors: LC duplex (2 fibres, the common one), MPO/MTP (12 or 24 fibres, for parallel optics and trunks), SC (older, carrier hand-offs).

Copper: Cat5e = 1 Gb, Cat6 = 10 Gb to 55 m, Cat6A = 10 Gb to 100 m. 10GBASE-T is hotter, slower-latency and more power-hungry than DAC — which is why it survives mainly for BMC and cheap kit.

Choosing: same cabinet → DAC. Adjacent → AOC. Through a patch panel → transceivers and fibre, always (you cannot patch an AOC).

Transceivers are vendor-coded; specify the switch model when ordering. DOM/DDM exposes Rx power per module — scrape it, it is the best early warning of a dying link.

Out-of-band

BMC="-I lanplus -H 10.20.0.37 -U admin -P $PW"

ipmitool $BMC chassis power status|on|off|soft|cycle|reset
ipmitool $BMC chassis identify 300     # light the blue ID LED, 300s
ipmitool $BMC sel elist                # hardware event log — run this first
ipmitool $BMC sdr list                 # all sensors
ipmitool $BMC lan print 1              # BMC IP/MAC/VLAN, cipher suites
ipmitool $BMC sol activate             # serial over LAN  (escape: ~. )
ipmitool $BMC chassis bootdev pxe      # override next boot
ipmitool $BMC mc reset cold            # reboot the BMC, NOT the server

# Redfish equivalent
curl -sk -u "admin:$PW" https://$BMC/redfish/v1/Systems/1 | jq .PowerState
curl -sk -u "admin:$PW" -X POST -H 'Content-Type: application/json' \
  -d '{"ResetType":"ForceRestart"}' \
  https://$BMC/redfish/v1/Systems/1/Actions/ComputerSystem.Reset

Serial console, set up in advance:

# /etc/default/grub
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0"
# then
update-grub && systemctl enable --now [email protected]

The reboot ladder:

1. ssh host reboot — needs a working OS.
2. ipmitool ... chassis power cycle — needs a reachable BMC. Does not remove standby power.
3. Switched PDU, both outlets off, 30 seconds — removes standby power. Clears a wedged BMC.
4. Remote hands ticket.

Security — non-negotiable: dedicated management VLAN, no internet route, reach via jump host or VPN. Disable cipher suite 0 (no authentication at all). IPMI 2.0's RAKP exchange leaks password hashes pre-authentication by design, so an exposed BMC is an offline cracking target. Change default credentials. Prefer Redfish; disable IPMI-over-LAN if unused. Patch BMC firmware.

Dedicated management port beats shared/NC-SI. Shared puts your recovery path in the same failure domain as the thing that breaks.

Network placement

• ToR — switch per cabinet, short DAC to servers, fibre uplinks out. The default.
• EoR / MoR — chassis at the row end or middle, long structured cabling. Better when cabinets are sparse or you want intra-row traffic on one switch.
• Leaf-spine — every leaf to every spine, no leaf-leaf, no spine-spine. Every server three hops from every other. Scales by adding spines.
• Oversubscription = downstream ÷ upstream. 48×25G ÷ 8×100G = 1.5:1. 3:1 is common. Know yours and graph the uplinks — that is where unexplained slowness lives.
• Two ToR switches per cabinet, plus a cheap 1 Gb management switch for BMCs.
• Check switch airflow direction when ordering — port-side intake vs exhaust. The wrong one inhales its neighbour's exhaust.

ethtool eno1 | grep -E 'Speed|Duplex|Link detected'
cat /proc/net/bonding/bond0
lldpctl                 # which switch, which port — learn this one

Connectivity

The chain: your router → demarc panel → cross-connect → MMR → carrier / cloud / IX.

• A cross-connect is a cable, not a service. Install fee plus monthly rent. Takes days.
• Order it with an LOA-CFA from the far end, naming cabinet, panel and port. They expire.
• Direct Connect: dedicated = a whole port (AWS: 1/10/100/400 Gbps), you order the cross-connect. Hosted = a partner's slice (AWS: 50 Mbps–25 Gbps), no cross-connect for you.
• Transit = reach everything, paid. Peering = direct to one network, usually free, only their customers. IX = one cross-connect, hundreds of peering sessions.
• 95th percentile billing: five-minute samples, top 5 % discarded, pay on the highest remainder. Roughly the top 36 hours a month are free — bursts are free, sustained growth is not. A commit is a minimum you pay regardless; ~10 % of port capacity is a common shape.
• "Redundant" means physically diverse. Ask what the two paths share: panel, room, building, conduit.

The remote hands ticket

SUBJECT: [action] on [device] at [cabinet] / [U]

LOCATION   facility · cage · cabinet · U position · FRONT or REAR
           · ID LED lit?
DEVICE     hostname · make/model · service tag · where the label is
STEPS      numbered · every port and outlet named · nothing assumed
DO NOT     what must not be touched
VERIFY     how you will confirm
IF DIFFERENT   stop and call me — explicit permission to stop
PHOTOS     rear cabling · both PDU outlets · front of rack
URGENCY    production impact · target time

Rules that do the work:

• Say FRONT or REAR every time. Left and right are as viewed from the rear.
• Light the ID LED before filing. It removes identity ambiguity for free.
• Both cords, different PDUs — state it twice on any install.
• Distinguish the BMC port from the data ports explicitly.
• Label both ends with the far end's identity.
• Ask for photos. Always. It is your only durable record.
• Never ask them to diagnose, and never ask them to log in.
• Verify the result, not the report — lldpctl, link counters, DOM levels, sel elist, cabinet power draw.

Notify the facility of inbound shipments in advance with tracking numbers and cage marking, and say what should happen to the packaging.

The five facts about your own cage

If you can answer these without looking, you have closed the gap this series was written for.

1. Cabinet identifier — room, row, position — and whether U1 is top or bottom.
2. Free U, and usable post-to-post depth.
3. Power: how many feeds, what rating, what you are actually drawing.
4. Uplinks: what the ToR connects to, spare ports, oversubscription ratio.
5. Cross-connects: how many leave the cage, where each terminates, and what your "redundant" pair physically shares.

Where to verify this

Standards and vendor documentation beat blog posts, including this one. The primary sources behind this series:

• EIA-310 / rack dimensions — rack unit reference, 19-inch rack standards
• Rack power and A/B feeds — ServerCentral on sizing rack power, Belden on dual power feeds
• DAC / AOC reach — FS 100G DAC/AOC, DAC selection guide
• Fibre grades and reach — multimode fibre types, OM3 vs OM4 distance limits
• BMC / IPMI — out-of-band management overview, dedicated vs shared IPMI port
• BMC security — Rapid7's penetration tester's guide to IPMI, runZero BMC research, internet-exposed BMCs leaking hashes
• Switch placement and fabric — ToR vs EoR vs MoR, Arista EoR/MoR white paper
• MMR, demarc, cross-connect — meet-me room explained, cross-connect guide
• LOA-CFA and Direct Connect — AWS: requesting cross connects at Direct Connect locations, AWS: download the LOA-CFA, AWS Direct Connect layer 1 explained
• Transit, peering, IX, 95th percentile — Equinix on peering vs transit, Kentik on transit vs peering, 95th percentile explained
• Labelling and structured cabling — TIA-606 labelling guide
• Remote hands vs smart hands — datacenters.com comparison, TechTarget definition

Numbers vary by vendor and by facility contract. Where a figure here matters to a purchase or a design, confirm it against the specific part and the specific site.

That is the series. If it worked, the next facility email you get should read as sentences rather than acronyms — and the next ticket you write should come back done, first time.