Every remote hands ticket you do not have to open is a win, and out-of-band access is how you avoid most of them.
There is a second computer inside your server. It has its own CPU, its own firmware, its own network stack and its own IP address, it runs on standby power whenever the machine is plugged in, and it does not care whether the operating system is running, halted, or absent. Understanding it properly is the single highest-leverage thing in this series.
BMC, and its four brand names
The chip is a BMC — Baseboard Management Controller. In practice it is usually an ASPEED AST2500 or AST2600 soldered to the motherboard, running a small embedded Linux of its own.
Every vendor brands it differently, which is why one thing has four names:
| Vendor | Brand |
|---|---|
| Dell | iDRAC |
| HPE | iLO |
| Lenovo | XCC (formerly IMM) |
| Supermicro | usually just IPMI |
| Cisco UCS | CIMC |
They are the same category of thing. When someone says iDRAC and you run HPE, mentally substitute iLO and carry on.
IPMI is the old wire protocol for talking to a BMC — a UDP-based standard from the late nineties, at version 2.0 since 2004. Redfish is its modern replacement: HTTPS, JSON, REST, sane authentication. New tooling should use Redfish. Old tooling and every scrap of documentation on the internet uses ipmitool, so you need both.
What it can actually do
The list is longer than most people assume, and each item is a ticket you did not open.
- Power on, power off, hard reset, and graceful shutdown — independent of the OS.
- Serial-over-LAN: the machine's console, including the bootloader and kernel messages, over the network.
- Virtual media: mount an ISO from your laptop as if it were a physical drive, and reinstall the OS.
- Boot device override: "next boot, come up on PXE" without touching BIOS by hand.
- Full sensor telemetry: every temperature, fan RPM, voltage and PSU status in the chassis.
- The System Event Log (SEL): a hardware event history that survives OS reinstalls and often contains the actual cause of a mystery reboot.
- The identify LED: light a blue LED on the front and rear of the chassis on demand.
That last one deserves its own paragraph, because it is the best remote hands trick in this whole series. Instead of writing "the third server from the top, I think, the one with the blue sticker", you light the LED and write "the machine with the flashing blue ID LED, at U37". The technician cannot pick the wrong box. It converts an ambiguous instruction into an unambiguous one, for free.
# light it
ipmitool -I lanplus -H 10.20.0.37 -U admin -P "$PW" chassis identify 300 # 300 seconds
# turn it off again
ipmitool -I lanplus -H 10.20.0.37 -U admin -P "$PW" chassis identify 0
The commands worth knowing
ipmitool is in every distro's repos. -I lanplus selects IPMI 2.0 over the network and is required for essentially everything remote.
BMC="-I lanplus -H 10.20.0.37 -U admin -P $PW"
ipmitool $BMC chassis power status # on or off
ipmitool $BMC chassis power on
ipmitool $BMC chassis power off # abrupt — like holding the button
ipmitool $BMC chassis power soft # ACPI graceful shutdown
ipmitool $BMC chassis power cycle # off, pause, on
ipmitool $BMC chassis power reset # hard reset, no power removal
ipmitool $BMC sel elist # hardware event log, human-readable
ipmitool $BMC sel clear
ipmitool $BMC sdr list # every sensor and its reading
ipmitool $BMC sensor list
ipmitool $BMC lan print 1 # the BMC's own IP, MAC, VLAN, gateway
ipmitool $BMC chassis bootdev pxe # override next boot
ipmitool $BMC sol activate # attach to the serial console
ipmitool $BMC mc reset cold # reboot the BMC itself, not the server
Two of those are worth calling out. sel elist is the first thing to run after an unexplained reboot — a machine that vanished without a trace in the OS logs will frequently have a clean, dated entry in the SEL saying it overheated or lost a PSU. And mc reset cold reboots the BMC without touching the running server, which is the fix when the BMC's web UI has wedged but the workload is fine. That distinction matters: you can repair your management path without an outage.
To escape from sol activate, the sequence is ~. — the same escape as ssh. Which means if you are inside ssh and inside SOL, you need ~~. to leave only the inner one. This catches everyone once.
The Redfish equivalent, for anything new:
curl -sk -u "admin:$PW" https://10.20.0.37/redfish/v1/Systems/1 | jq .PowerState
curl -sk -u "admin:$PW" -X POST \
-H 'Content-Type: application/json' \
-d '{"ResetType":"ForceRestart"}' \
https://10.20.0.37/redfish/v1/Systems/1/Actions/ComputerSystem.Reset
Dedicated versus shared: the cable question
This is the thing you asked about directly, and it is a real decision with a real consequence.
On the back of the server there is usually a small RJ45 physically separate from the data NICs, labelled BMC, IPMI, iDRAC, iLO or MGMT. That is the dedicated management port. It is almost always 1 Gb copper, and it is wired to a separate management switch.
The alternative is shared mode — also called LOM (LAN on Motherboard) or NC-SI, which is the sideband protocol that lets the BMC borrow one of the main NICs. One physical cable carries both your production traffic and your management traffic, usually separated by VLAN.
DEDICATED SHARED / NC-SI
┌─────────────────────┐ ┌─────────────────────┐
│ server │ │ server │
│ ┌──────┐ │ │ ┌──────┐ │
│ │ BMC ├── mgmt ───┼──► mgmt switch │ │ BMC ├──┐ │
│ └──────┘ RJ45 │ │ └──────┘ │ NC-SI │
│ ┌──────┐ │ │ ┌──────┐ │ │
│ │ NIC ├── data ───┼──► ToR switch │ │ NIC │◄─┘ │
│ └──────┘ │ │ └──┬───┘ │
└─────────────────────┘ └─────┼───────────────┘
└── data + mgmt ──► ToR
two cables, two failure domains one cable, one failure domain
The argument for dedicated is the entire point of out-of-band. If the BMC shares a NIC with the OS, then a failed NIC, a bad driver, a misconfigured bond, an unplugged cable or a switch port error-disabling takes out both your production path and the thing you were going to use to fix it. You have built out-of-band management that goes out of band at exactly the moment you need it.
The argument for shared is cost: half the cables, half the switch ports, less structured cabling. In a large fleet that is real money.
My view, stated as a view: use the dedicated port. The saving is one RJ45 patch lead and one port on a cheap 1 Gb switch, and what you buy with it is that your recovery path does not share a failure domain with the thing that breaks. That is the whole product. If budget forces shared mode, at least know that you have made that trade, and make sure something else — a serial console server, or a switched PDU — gives you an independent path.
There is a third setting on many boards, failover, where the BMC prefers the dedicated port and falls back to shared if the link drops. It sounds ideal and is occasionally surprising, because the BMC can silently move to a different network and a different address. If you use it, know it is on.
Never, ever expose a BMC
This is not a stylistic preference. It is the one hard security rule in this series.
A BMC is a privileged computer with full physical-equivalent control of your server, running vendor firmware that is patched slowly and often not at all, sitting underneath the operating system. An attacker who owns the BMC owns the machine in a way that survives an OS reinstall, and no host-based control can see them.
The protocol makes it worse. IPMI 2.0's RAKP key exchange hands out a password hash to anyone who asks for a valid username, before authentication. That is not a bug in an implementation — it is how the protocol was specified, and it means an exposed BMC is an offline cracking target. Separately, cipher suite 0 is an option that disables authentication entirely: supply a valid username, no password, get in. Historical scans found it enabled by default on a majority of internet-facing BMCs.
Research published this year found tens of thousands of BMCs still reachable from the internet, with a large fraction of them still handing over crackable hashes to unauthenticated requests. The problem has not aged out. It got bigger, because the fleet got bigger.
So, the rules:
- BMCs live on a dedicated management VLAN with no route to the internet. Not a firewall rule on a shared VLAN. A separate segment.
- Reach them through a jump host or a VPN, never directly.
- Disable cipher suite 0. Check with
ipmitool ... lan print 1and look at the cipher suite list. - Change the default credentials. Supermicro's
ADMIN/ADMINand equivalents are the first thing anyone tries. - Prefer Redfish over IPMI where the hardware supports it, and turn IPMI-over-LAN off entirely if nothing needs it.
- Patch BMC firmware. It is the piece of software in your estate least likely to have been updated and most likely to be catastrophic.
CISA's guidance on this is blunt and worth quoting in spirit: use a physically separate management network where you can, default-deny, and permit management access only from dedicated administrative systems.
The serial console, which still matters
SOL over IPMI is convenient but it depends on the BMC being alive and on the network. A console server (sometimes a terminal server) is the independent alternative: a box with 16, 32 or 48 serial ports, cabled to the serial port of every device in the row, reachable over the management network.
It is old technology and it is the thing that saves you when everything else is gone, because the serial console works when the network stack is dead, when the machine is stuck in the bootloader, and when a firmware update has bricked the BMC. It is also the only way to manage a lot of network equipment during an upgrade gone wrong.
Cabling vocabulary you will meet: many devices use RJ45 serial with the Cisco-standard rollover pinout, and the light blue flat cable is a near-universal visual convention. Others use DB9, and newer servers increasingly use USB or a micro-USB service port on the front bezel.
On the Linux side, none of this is exotic:
# make sure the kernel actually talks to the serial port — otherwise the
# console server shows you a blank screen and you blame the hardware
# /etc/default/grub:
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0"
# then
update-grub # or grub2-mkconfig -o /boot/grub2/grub.cfg
systemctl enable --now [email protected]
That last pair is worth doing on every machine you own, in advance. A serial console you configured after the outage started is not a serial console.
Three different things "reboot it" can mean
This is the escalation ladder, and knowing which rung you are on stops you from asking for the wrong thing.
1. OS-level ssh host reboot
───────── Clean. Needs a working OS and network.
2. BMC-level ipmitool ... chassis power cycle
────────── Works when the OS is dead. Needs a reachable BMC.
NOTE: does NOT remove standby power. The BMC keeps
running, and so does anything else on standby.
3. Power-level switched PDU: outlet off, wait, outlet on
──────────── Works when the BMC is dead. Removes standby power
entirely, which is what actually clears a wedged BMC
or a confused PSU. Both cords for a dual-corded box.
4. Human remote hands ticket.
────── Everything else has failed, or the fault is physical.
The gap between rung 2 and rung 3 is the one people miss. chassis power cycle does not de-energise the board — the BMC and standby rails stay up. When a machine is in a genuinely stuck state, or when the BMC itself has stopped answering, you need real power removal. That is what a switched PDU buys you, and it is why Part 2 argued for switched-and-monitored units. Without one, rung 3 is a ticket.
If you are cutting power at the PDU on a dual-corded server, both outlets must go off, and a real drain needs perhaps thirty seconds rather than two. Half a power-cycle on a redundant machine is a no-op that looks like a failed fix.
What to have in place before you need it
A short checklist. Each of these is boring to do in advance and impossible to do during an incident.
- Every BMC has a static or reserved address, is in DNS, and is in your documentation next to its server's name and U position.
- Every BMC is on the management VLAN, with default credentials changed and cipher 0 disabled.
- Serial console configured in GRUB and a getty enabled on every host.
- Switched PDU, with outlet-to-server mapping documented — including which outlet is A and which is B for each machine.
- BMC reachability is monitored. A BMC that died three months ago is discovered at the worst possible moment otherwise. It is an ICMP check and an authenticated Redfish poll; both are cheap.
- Optical Rx power and PSU status scraped into metrics, so degradation shows up as a trend rather than an outage.
The one-line summary
The BMC is a separate computer that can power, console and reinstall your server without the OS. Give it its own port, its own VLAN, and no path to the internet. Learn sel elist for post-mortems and chassis identify for tickets. Keep a switched PDU, because a BMC-level power cycle does not remove standby power and a wedged BMC does not answer.
Next up: Part 5 — where the switches live, and why your server's NIC probably terminates two feet above it.