Your server's NIC cable probably travels about two feet. That fact shapes more of the datacenter than anything else in this series.
This part is where switches physically sit, why that choice was made, and what it means for the cabling in your cabinet.
The three placements
There are three ways to answer "where does the server plug in", and they are named after where the switch is.
Top of Rack (ToR). A switch in each cabinet, usually at the top, serving only the servers in that cabinet. Server-to-switch links are short DAC runs inside the cabinet. The switch's uplinks leave the cabinet on fibre.
End of Row (EoR). One or two large chassis switches in a dedicated cabinet at the end of a row, serving every server in that row. Every server has a long cable running from its cabinet, through overhead or underfloor trays, to that cabinet.
Middle of Row (MoR). Identical to EoR but positioned in the centre of the row, which halves the worst-case cable length.
TOP OF RACK END OF ROW
─────────── ──────────
┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌──────┐
│ SW │ │ SW │ │ SW │ │ │ │ │ │ │ │ ████ │ ← chassis
├────┤ ├────┤ ├────┤ │srv │ │srv │ │srv │ │ ████ │ switches
│srv │ │srv │ │srv │ │srv │ │srv │ │srv │ │ ████ │
│srv │ │srv │ │srv │ │srv │ │srv │ │srv │ │ ████ │
│srv │ │srv │ │srv │ │srv │ │srv │ │srv │ │ │
└─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └─┬──┘ └───▲──┘
│ │ │ └──────┴──────┴────────┘
└──────┴──────┴─► spine long structured cabling runs
short DAC inside (Cat6A or fibre, through trays,
each cabinet; only via patch panels at both ends)
fibre uplinks leave
ToR won, for most people, for practical rather than architectural reasons. Cabling inside a cabinet is cheap, short, and done once when the cabinet is built. Cabling from every cabinet to a row-end chassis is expensive, has to be planned before anything is installed, and fills trays. ToR also means a new cabinet is a self-contained unit you can build and test in isolation.
EoR and MoR still make sense in three cases: when you have too few servers per cabinet for a whole switch to be worthwhile, when you want server-to-server traffic within a row to stay inside one switch (which removes a hop and can remove oversubscription entirely), and when you have standardised on structured Cat6A cabling and 10GBASE-T. A single big switch is also one thing to manage rather than forty.
A note on the name: a "top of rack" switch is not required to be at the top. Placing it at the top is the convention because that is where cable trays and vertical managers converge, but you will see them mid-cabinet in high-density layouts to halve the average cable run to servers above and below.
What a ToR cabinet actually contains
Working down from the top of a typical cabinet:
U42 ░ blanking ░
U41 patch panel ← fibre to the MMR / cross-connects (Part 6)
U40 sw-01 ← ToR switch A 48x25G SFP28 + 8x100G QSFP28
U39 sw-02 ← ToR switch B (the redundant pair)
U38 mgmt-sw-01 ← 1G copper switch for BMC / out-of-band (Part 4)
U37 ░ blanking ░
U36 srv-01
U35 srv-02
.. ...
Four things in that stack are worth naming.
Two ToR switches, not one. Each server has one link to each switch. Lose a switch — or reboot one for a firmware upgrade — and every server stays up on the other. A cabinet with a single ToR switch has a single point of failure that takes out forty machines at once, which is a decision you can make deliberately for a stateless tier and should never make by accident.
A separate management switch. Cheap, 1 Gb, copper, carrying only BMC and console traffic on its own VLAN. It exists so that your out-of-band path does not depend on the production switches you might be trying to fix. This is Part 4's argument expressed in hardware.
A patch panel. Structured fibre terminates here rather than plugging directly into the switch, so that when a switch is replaced the building's cabling is not disturbed. Anything leaving the cabinet permanently should pass through it.
Blanking panels above and below the active kit, for the airflow reason from Part 1.
One airflow detail specific to switches, because it bites people: switches are built in two airflow directions. Port-side exhaust (also called front-to-back relative to the ports) and port-side intake (back-to-front). Since a switch's ports face the rear of the cabinet where the servers' cables are, but the cold aisle is at the front, you need the variant that pulls cold air from the front and exhausts out the rear like the servers do. Order the wrong one and you have installed a device that inhales its neighbours' exhaust. Vendors sell both and the part numbers differ by a suffix.
Leaf and spine
Above the cabinet, modern datacenter networks are almost universally leaf-spine, also called a Clos fabric or a fat tree.
Every leaf — in practice, every ToR switch — connects to every spine switch. Leaves never connect to each other. Spines never connect to each other. Servers connect only to leaves.
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
SPINE │ sp-1 │ │ sp-2 │ │ sp-3 │ │ sp-4 │
└─┬─┬─┬─┬┘ └┬─┬─┬─┬─┘ └┬─┬─┬─┬─┘ └┬─┬─┬─┬─┘
│ │ │ └──────┼─┼─┼─┼──────┼─┼─┼─┼──────┘ │ │ │
│ │ └────────┼─┼─┼─┼──────┘ │ │ │ │ │ │
... every leaf connects to every spine ...
┌─┴─────┐ ┌──┴────┐ ┌──┴────┐ ┌──┴────┐
LEAF │ leaf1 │ │ leaf2 │ │ leaf3 │ │ leaf4 │
└───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘
servers servers servers servers
The property that matters: every server is exactly the same number of hops from every other server — leaf, spine, leaf. Three hops, always, whether the destination is in the next cabinet or across the hall. Latency becomes predictable, and capacity scales by adding spines rather than by replacing a bigger box with an even bigger box.
This replaced the old three-tier access/aggregation/core design, which was built for traffic that mostly flowed north-south — in and out of the datacenter. Modern workloads are overwhelmingly east-west: service to service, node to node, replica to replica. Leaf-spine is the shape you build when most of your traffic never leaves the building.
Oversubscription, and the number to ask for
Your ToR switch has, say, 48 ports of 25 Gb facing the servers and 8 ports of 100 Gb facing the spines.
- Downstream: 48 × 25 = 1200 Gb/s
- Upstream: 8 × 100 = 800 Gb/s
That is an oversubscription ratio of 1200:800, or 1.5:1. If every server in the cabinet transmitted at full line rate simultaneously, the uplinks could not carry it.
This is deliberate and normal. Uplink optics are expensive, and it is vanishingly unlikely that forty-eight machines all saturate their NICs in the same millisecond. A 3:1 ratio is common and unremarkable; 2:1 is comfortable; 1:1 (non-blocking) is bought only for workloads that genuinely need it — distributed storage rebuilds, HPC, large-scale training.
Why you should care: oversubscription is where "the network is slow" comes from when every individual link looks healthy. Your NIC shows 25 Gb, the switch port shows no errors, and yet throughput collapses during the nightly backup — because sixteen other machines in the cabinet are also backing up and you are all sharing 800 Gb of uplink.
So: know your cabinet's oversubscription ratio, and know it as a number. It is a fair question to ask whoever runs the fabric, and if the answer is "we don't know", that is itself the finding. Then graph uplink utilisation, not just server NIC utilisation. The server-side graph will look fine right up until it does not.
How your server attaches
Two links, one to each ToR switch. What happens on top of them is a Linux question, and it is the one part of this series that lands squarely in your territory.
Active/backup bonding (mode=1) is the simple option. One link carries traffic, the other stands by. It needs nothing from the switches, works with any two switches anywhere, and costs you half your bandwidth.
LACP (mode=4, 802.3ad) uses both links at once. It requires both switch ports to be in the same link aggregation group — which, when the ports are on two different physical switches, requires those switches to pretend to be one. That is MLAG (multi-chassis link aggregation): Arista's MLAG, Cisco's vPC, and vendor equivalents. The pair keeps state in sync over a peer link and presents a single logical switch to your server.
# what your NIC actually negotiated — check this before blaming the fabric
ethtool eno1 | grep -E 'Speed|Duplex|Link detected'
# bond state, including which slave is active and whether LACP came up
cat /proc/net/bonding/bond0
# who is on the other end — LLDP is the single most useful command here
lldpctl # or: lldpcli show neighbors detail
Learn lldpctl. With LLDP running on the switches, it tells you the neighbour switch's name, the exact port your cable is in, and often the port description. That turns "which switch port is this server in" from a remote hands ticket with a torch into one command. When you are writing a ticket about a specific link, lldpctl output is the authoritative identifier to quote.
A third option worth knowing about: EVPN/VXLAN with a routed fabric, where the leaf is a layer-3 boundary and there is no MLAG at all — each server link is a separate routed path, and the host runs BGP. It is cleaner and increasingly common at scale, and it changes your side of the picture from bonding to routing.
Where oversubscription meets your cabling
One last practical link back to Part 3. The uplinks from a ToR switch to the spines are the runs that leave the cabinet, which means they are fibre, through the patch panel, and they are the reason your cabinet has a fibre panel at all. Server links stay inside the cabinet, which is why they are DAC.
So the cabling in a cabinet has a shape you can now predict entirely:
- Inside the cabinet, rear: forty short DAC cables, servers to the two ToR switches.
- Inside the cabinet, rear: forty short Cat6A leads, BMCs to the management switch.
- Leaving the cabinet: a handful of fibre uplinks through the patch panel, and two power whips.
Which means: most of what looks like a wall of cables is short, local and identical, and only a few strands actually go anywhere. That was the thing that made the rear of a cabinet stop looking like chaos to me.
The one-line summary
Top of rack won: short DAC inside the cabinet, fibre uplinks out, two switches per cabinet plus a cheap 1 Gb management switch. Above that, leaf-spine makes every server three hops from every other. Know your oversubscription ratio and graph your uplinks, because that is where unexplained slowness lives. And run LLDP, so you never have to ask a human which port a cable is in.
Next up: Part 6 — leaving the cage, where the demarc, the Meet-Me Room and the cross-connect finally connect to each other.