Data Center Part 3: Cables, optics and transceivers

"Send me a 3-metre DAC" is a complete, unambiguous, orderable request. "Send me a network cable" is not. This part is about being able to say the first thing.

The three ways to connect two ports

Every high-speed link in a datacenter is one of three physical things, and the choice is almost entirely about distance.

DAC — Direct Attach Copper. A fixed length of twinaxial copper with the transceiver moulded permanently onto both ends. One part number, one item, no separate cable. Cheapest, lowest power, lowest latency. Also the shortest.

AOC — Active Optical Cable. Same idea — ends permanently attached — but the middle is fibre and the ends contain lasers. Longer reach, more money, more power draw, and still a single sealed unit you cannot repair.

Transceiver plus fibre. Two removable modules plus a separate fibre patch lead. The most expensive per link and the most flexible: you can change the fibre without changing the optics, run through patch panels, and reach anything from metres to kilometres.

  DAC                 [==transceiver==]━━━━copper━━━━[==transceiver==]
                       (moulded on)                    (moulded on)
                       ← up to ~3-7 m passive, ~10 m active →

  AOC                 [==transceiver==]────fibre────[==transceiver==]
                       (moulded on)                   (moulded on)
                       ← up to ~100 m →

  Transceiver+fibre   [module] ◄─┐                 ┌─► [module]
                                 └── fibre lead ───┘
                       ← metres to tens of kilometres →
                        (and the only one that survives a patch panel)

How far each one goes

These are the numbers worth memorising, because they decide the answer nine times out of ten.

Passive DAC is typically good to about 3 m, with 5 m and sometimes 7 m available depending on speed and vendor. It contains no electronics at all — it is wire.

Active DAC puts signal conditioning in the ends and reaches roughly 5 to 10 m.

AOC reaches up to about 100 m over the multimode fibre inside it.

Notice how neatly that maps onto physical layout. Inside one cabinet, DAC. Between adjacent cabinets, active DAC or AOC. Across the room or between rooms, transceivers and structured fibre. The cable type is a proxy for "how far away is the other end", and if someone tells you the link type you can infer the topology.

The trap: DAC lengths are exact and unforgiving. You are buying a specific physical length with the ends attached. Order 1 m when you needed 2 m and it does not reach; order 5 m when you needed 2 m and you have three metres of stiff twinax to hide in a cabinet that has nowhere to put it. Twinax is thick and does not bend tightly. Measure the real cable path — down the rack, through the manager, back up — not the straight-line distance.

Form factors, which encode speed

The connector shape tells you the speed class. This is the whole naming system.

The Q stands for quad: a QSFP carries four lanes in the space of one SFP. That is the whole trick — QSFP28 is 100 Gb because it is four lanes of 25 Gb, and QSFP+ is 40 Gb because it is four lanes of 10 Gb.

Which leads directly to the useful consequence: breakout cables. Because a QSFP28 is physically four 25 Gb lanes, you can buy a cable with one QSFP28 on one end and four SFP28s on the other, and split a single 100 Gb switch port into four 25 Gb server ports. Written 1x100G → 4x25G, or 4x25G breakout. This is extremely common and it is how you get high port counts out of a switch that only has QSFP cages. The same trick gives you 1x40G → 4x10G.

The physical compatibility rule: an SFP-family module fits any SFP-family cage, and a QSFP-family module fits any QSFP-family cage — but a QSFP will not fit an SFP cage. They are different sizes. Speed negotiation within a family is a separate question and often does not happen automatically, so a 25 Gb module in a 10 Gb switch port may or may not work depending on the switch.

Copper RJ45, which still exists

Not everything is optics. 10GBASE-T runs 10 Gb over ordinary RJ45 on Cat6A to 100 m, and it is the reason some servers still have familiar-looking Ethernet ports.

It has real drawbacks in a datacenter: noticeably higher power per port, higher latency than DAC, and more heat. Which is why it lost the server-to-switch fight to SFP28 DAC. But it is still everywhere for exactly two jobs, and both matter to you:

• Out-of-band management. BMC ports are almost always 1 Gb RJ45. Part 4.
• Anything cheap or old — small switches, appliances, PDUs, console servers, environmental sensors.

Cable categories in the order you will meet them: Cat5e (1 Gb), Cat6 (10 Gb to 55 m), Cat6A (10 Gb to 100 m). In a cabinet, use Cat6A and stop thinking about it.

Fibre: the two families

If a link uses transceivers and separate fibre, the fibre is one of two fundamentally different things and they are not interchangeable.

Multimode (MMF) has a wide core — 50 microns — that lets light take multiple paths. Cheap optics, short distances. Datacenter grades are OM3, OM4 and OM5. Jacket colour is aqua for OM3 and OM4, lime green for OM5.

Singlemode (SMF) has a 9-micron core so light takes essentially one path. Expensive optics, enormous distances. The datacenter grade is OS2, and the jacket is yellow.

The colour of the jacket genuinely tells you the type, and this is one of the few places in the building where a visual convention is reliable enough to act on. Aqua is multimode. Yellow is singlemode. If someone sends you a photo of a patch panel, you can read the fibre type off it.

Reach, which is the whole reason the distinction exists:

Two things to take from that table. Multimode reach collapses as speed rises — the same OM4 fibre that carried 10 Gb for 400 m carries 100 Gb for 100 m. And singlemode does not care, which is why long runs, cross-connects and anything leaving the building are singlemode.

The transceiver naming follows: SR is short reach (multimode), LR is long reach (singlemode, 10 km), ER and ZR go further still. The trailing digit in SR4 means four lanes, which means it needs eight fibres rather than two.

Fibre connectors

LC is the small duplex connector — two fibres, one for transmit and one for receive, clipped together. It is what an SR or LR transceiver takes and it is the overwhelming majority of what you will see.

MPO (also branded MTP) is a rectangular multi-fibre connector carrying 12 or 24 fibres in one ferrule. It exists for the parallel optics — a 40GBASE-SR4 or 100GBASE-SR4 needs eight fibres, so it takes an MPO rather than an LC. MPO is also how high-density trunk cabling gets run between patch panels: one trunk, then a cassette that fans it out to LC.

SC is the older, larger square connector. Still found on carrier hand-offs and in the MMR.

Two things about fibre worth knowing even though you will never handle it. Polarity — transmit must land on receive at the far end, and with MPO trunks there are three standardised polarity methods (A, B and C) that a cabling team has to keep consistent; getting it wrong gives you a dark link with no obvious cause. And cleanliness — a fingerprint on a ferrule end-face is enough to kill a 100 Gb link, so fibre gets cleaned before every insertion and capped when unplugged. If a new fibre link is dark and everything on paper is right, "please clean and reseat both ends" is a legitimate and very common first remote hands request.

Transceiver coding, the annoying part

Transceivers carry a small EEPROM identifying the vendor. Some switches — Cisco historically being the notable one — refuse modules that do not identify as their own. Third-party vendors sell the same optics coded for a given switch brand, at a fraction of the price.

This is worth knowing for two reasons. When you order optics you must specify the switch they are going into, not just the speed. And if a link is dead and the switch log says something about an unsupported or unrecognised transceiver, that is a coding problem, not a hardware fault, and no amount of reseating will fix it.

On the good side, transceivers support DOM/DDM — Digital Optical Monitoring — which exposes real-time transmit power, receive power and temperature per module. That is how you tell a dirty connector from a dead laser from a fibre someone stood on, without going near the building. Your switch will show it. If you are not scraping optical Rx power into your metrics, you are missing the single best early warning of a degrading link.

Choosing, in one flowchart

   Both ends in the SAME cabinet?
        │
        ├── yes ──► DAC. Passive if ≤3 m, active up to ~10 m.
        │           Cheapest, coolest, lowest latency. Measure the real path.
        │
        └── no ──►  Does it pass through a patch panel?
                        │
                        ├── yes ──► Transceivers + fibre. Always.
                        │           (An AOC cannot be patched — the ends are moulded on.)
                        │
                        └── no ──►  Under ~100 m, point to point?
                                        │
                                        ├── yes ──► AOC is simple and cheap enough.
                                        │
                                        └── no ──►  Transceivers + singlemode (OS2, LR).

Cabling hygiene that is actually your problem

Most of this is done by people with more skill than me, but three of them are decisions you make, not them.

Order the right length, not the nearest length. Slack has to live somewhere, and in a rear cabinet channel there is nowhere for it to live. Cut-to-length or correctly-chosen patch leads are the difference between a cabinet you can work in and one where every change risks disturbing something else.

Leave a service loop on anything on sliding rails. Enough slack to pull the machine fully forward. Covered in Part 1; it belongs here too, because it is a cabling decision.

Label both ends, with the far end's identity on the label. → sw-c07-01 Et1/14 on the server end, → srv-web-11 eno1 on the switch end. No tracing, ever.

And one convention that costs nothing: use cable colour as a type signal. Whatever scheme you like — say, one colour for production data, one for out-of-band management, one for cross-connects — as long as it is written down and consistent. It makes a photograph from remote hands into something you can actually diagnose from.

The one-line summary

Same cabinet, DAC. Adjacent cabinets, AOC. Through a panel or across the room, transceivers and fibre — aqua multimode for short, yellow singlemode for long. Q means four lanes, which is why 100 Gb breaks out into four 25 Gb. Order by exact length and exact switch model, and label both ends with the far end.

Next up: Part 4 — out-of-band access, which is the difference between fixing a machine from your laptop and filing a ticket.