Reinstalling bare-metal servers is one of those jobs that looks trivial until you do it twenty times a day. I have spent the last few years building provisioning automation for hosting providers, and the same failure modes keep showing up: a PXE boot that silently falls back to the local disk, an unattended install that waits forever on a question nobody answered, an IPMI "next boot = PXE" flag that gets cleared by a power event.
This is the checklist I use when auditing a bare-metal reinstall pipeline. It is written from the point of view of someone who has to make it work at 3am, not the point of view of a vendor deck.
1. Replace "boot ISO by hand" with a deterministic chain
The manual version is: mount an ISO, reboot, watch the console, answer questions. It works once. At scale you want a chain where every step is verifiable:
- The controller sets the next boot device to PXE over the BMC.
- The machine gets a DHCP lease from the provisioning network.
- iPXE fetches a boot script for that machine's MAC address.
- The script points at a kernel and initrd, and passes an answer-file location.
- The installer runs unattended and the machine reboots into the installed system.
Every step has a "silent failure" mode. The DHCP lease can come from the wrong scope. The iPXE script can be cached. The answer file can be served publicly, which means anyone can read your root password hash. Build the chain so that each hop is logged and each artifact is scoped to exactly one install task, with a short-lived token rather than a public URL.
2. One answer file per distro, rendered per machine
Nobody wants to maintain four formats, but the installers do not agree with each other:
- Debian: preseed
- Ubuntu Server: cloud-init autoinstall
- CentOS / Rocky / Alma: kickstart
- Windows: autounattend.xml
The trick is to keep a single source of truth for machine facts (hostname, IP, gateway, DNS, admin password, SSH keys, repository mirror) and render the distro-specific file at install time. If you are templating with string replacement, guard against values that contain the delimiter. If you are rendering XML for Windows, escape everything and pick the right firmware branch: BIOS and UEFI need different disk partition blocks, and a mixed fleet will have both.
3. IPMI is the API that actually matters
The provisioning software is only half the story; the other half is the out-of-band controller. What you need to be able to do remotely and idempotently:
- power on / off / reset
- set boot device to PXE for the next boot only
- read the power state back afterwards
- detect when the BMC itself is unreachable
"Set next boot to PXE" is not always a one-shot: some BMCs clear it, some keep it and boot the installer forever. After a successful install, explicitly set the boot device back to disk and verify it, otherwise the next reboot reinstalls the server.
4. Images are artifacts: verify, version, roll back
Pull the ISO from a mirror list (domestic mirrors first is a real requirement in some regions), verify the SHA-256 against the official value, unpack it, and publish atomically. Keep the previous version around so a bad kernel or a broken driver does not take your whole fleet down with it. When a local ISO is uploaded by an operator, verify it too — a corrupted upload that passes "it looks about the right size" is a classic outage cause.
Windows is the exception: ISO and WinPE boot files usually need to be fetched on demand from a verified catalog, because the license terms and file layout make local mirroring messy.
5. The last mile is where installs actually fail
A machine that installed successfully but came up with the DHCP address and a random hostname is not provisioned. Plan for a small first-boot program that runs exactly once, applies the final configuration (static IP, gateway, DNS, hostname, admin password, repositories), reports success to the controller, has its credential revoked and deletes itself.
Two details that bite:
- The temporary password used during install must not survive as the final password.
- The first-boot program has to be idempotent enough to survive a crash and re-run, but strict enough not to be replayable by someone else.
6. What to measure
If you cannot answer these in under a minute, the pipeline is not observable:
- How many installs are running right now, and which task is stuck?
- What is the p50 and p95 time from "reinstall requested" to "OS reachable"?
- How many installs failed, and at which step?
- Which images are in production, and which hosts run the oldest one?
Log the step transitions, not just success/failure. "Failed" tells you nothing; "failed at iPXE fetch, 401, token expired" tells you everything.
7. A note on building versus buying
Most teams start with a pile of scripts and a wiki page, then outgrow it when the install network has more than one site. At that point the choice is between gluing together existing microservices (DHCP, TFTP, an image server, a task runner, an IPMI wrapper) or running one component that already speaks all of them.
We ended up in the second camp and built the thing we wanted to use: a self-hosted platform that keeps DHCP, TFTP, DNS, HTTP image delivery, SMB and the BMC gateway in one Go binary per site, with a cluster console above it. If you are curious, it is at tntcloud.io — the free tier covers three servers, which is enough to test the whole chain end to end before you commit.
Either way, the checklist above is the part that matters. The tooling is just the shortest path to a deterministic reinstall.
Checklist, condensed
- [ ] Boot chain is deterministic and logged at every hop
- [ ] Answer files are rendered per machine, not copy-pasted
- [ ] Answer-file URLs are task-scoped and short-lived
- [ ] BMC power state and boot order are read back after every action
- [ ] Boot device is reset to disk after a successful install
- [ ] Images are SHA-256 verified, versioned and rollback-able
- [ ] First-boot configuration is one-shot, idempotent and self-deleting
- [ ] Install metrics are queryable, not buried in logs
