Installing OpenShift on bare metal has become dramatically easier since Red Hat shipped the Assisted Installer and its offline sibling, the agent-based installer. You boot an ISO, the nodes register themselves, a validation engine tells you what is wrong, and you click install. In theory.

In practice, most bare metal installations do not fail on compute, storage or the pull secret. They stall on DNS, VIPs, bonding and MTU — infrastructure details that nobody owns until the install hangs at 34% with a cryptic pending-for-input state. This article walks through a full install, then spends real time on the validations that block you and how to fix them.

Context: why bare metal installs stall

On a cloud platform, the installer creates the load balancer, the DNS records and the networks itself. On bare metal there is no cloud API, so OpenShift ships its own replacements:

  • keepalived owns two virtual IPs (API and Ingress) and moves them between nodes using VRRP.
  • HAProxy static pods front the API VIP and load-balance to the control plane.
  • CoreDNS static pods resolve the internal cluster names.

All of that lives in the openshift-kni-infra namespace and only works if you provide three things beforehand: resolvable DNS names, two free IPs in the machine network, and layer-2 connectivity between all nodes. Everything in this article is about getting those three right.

A second source of confusion: there are two products with almost the same name.

SaaS Assisted Installer Agent-based installer
Entry point console.redhat.com/openshift/assisted-installer openshift-install agent create image
Internet required Yes (nodes phone home) No, fully disconnected
Config Web UI / REST API install-config.yaml + agent-config.yaml
Engine assisted-service the same assisted-service, embedded in the ISO

They share the same validation engine and the same error messages. This article uses the agent-based installer because it is declarative, reproducible and works in disconnected environments — but every troubleshooting tip applies to both.

Step 0: the prerequisites checklist

Our target: a 3 control plane + 2 worker cluster named ocp1, base domain lab.example.com, machine network 192.168.70.0/24.

DNS records

Three names must resolve before you boot anything:

Record Type Value Who uses it
api.ocp1.lab.example.com A 192.168.70.10 (API VIP) external clients, oc
api-int.ocp1.lab.example.com A 192.168.70.10 (API VIP) nodes, kubelet, operators
*.apps.ocp1.lab.example.com A 192.168.70.11 (Ingress VIP) routes

Reverse (PTR) records for every node are strongly recommended: the node hostname is derived from reverse DNS when you use DHCP. Verify from a machine on the same network:

for n in api api-int; do echo -n "$n -> "; dig +short $n.ocp1.lab.example.com done dig +short console-openshift-console.apps.ocp1.lab.example.com dig +short -x 192.168.70.21 

The classic trap: a wildcard on the cluster domain itself, for example *.ocp1.lab.example.com. That makes api-int, etcd-0 and anything else resolve to a single wrong IP. The assisted-service explicitly rejects it with the dns-wildcard-not-configured validation. The wildcard must be on *.apps.ocp1..., nothing broader. Test it:

# Must return NOTHING (NXDOMAIN) dig +short this-name-does-not-exist.ocp1.lab.example.com 

VIPs and keepalived

The API and Ingress VIPs must be:

  • inside the machineNetwork CIDR,
  • outside any DHCP pool,
  • not assigned to any host, switch or firewall,
  • not answering ARP before the install.

Check with ARP, not ICMP (a firewall may drop pings while the IP is taken):

sudo arping -D -I bond0 -c 3 192.168.70.10 sudo arping -D -I bond0 -c 3 192.168.70.11 

An empty reply set means the address is free. If a device answers, the api-vips-free / ingress-vips-free validation fails before you even start. Also check that your switches do not filter VRRP (IP protocol 112) and multicast 224.0.0.18 between control plane nodes — if they do, keepalived elects two masters and the VIP flaps.

NTP

ntp-synced is the single most frequent failed validation on isolated bare metal. etcd will not form a quorum with clocks more than a few hundred milliseconds apart. Point the nodes at a reachable NTP server through additionalNTPSources (shown below) and make sure UDP/123 is open.

Step 1: install-config.yaml

apiVersion: v1 baseDomain: lab.example.com metadata: name: ocp1 compute: - name: worker replicas: 2 architecture: amd64 controlPlane: name: master replicas: 3 architecture: amd64 networking: networkType: OVNKubernetes clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 machineNetwork: - cidr: 192.168.70.0/24 serviceNetwork: - 172.30.0.0/16 platform: baremetal: apiVIPs: - 192.168.70.10 ingressVIPs: - 192.168.70.11 fips: false pullSecret: '{"auths":{"example.registry.local":{"auth":"REDACTED","email":"you@example.com"}}}' sshKey: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEXAMPLEKEYONLY you@workstation' 

Notes that save hours:

  • apiVIPs / ingressVIPs are lists since 4.12 (the old singular apiVIP is deprecated). Both VIPs must sit in machineNetwork.
  • machineNetwork must match the subnet your NICs actually use. If it does not, every host fails belongs-to-machine-cidr.
  • For a single-node OpenShift install, set controlPlane.replicas: 1, compute.replicas: 0 and omit the VIPs entirely.

Step 2: agent-config.yaml and NMState

This is where bare metal specifics live: which physical host plays which role, which disk to install on, and the exact network configuration applied by NMState before the host even registers.

apiVersion: v1beta1 kind: AgentConfig metadata: name: ocp1 rendezvousIP: 192.168.70.21 additionalNTPSources: - 192.168.70.1 hosts: - hostname: master-0 role: master rootDeviceHints: deviceName: /dev/sda interfaces: - name: eno1 macAddress: 52:54:00:aa:bb:01 - name: eno2 macAddress: 52:54:00:aa:bb:02 networkConfig: interfaces: - name: bond0 type: bond state: up ipv4: enabled: true dhcp: false address: - ip: 192.168.70.21 prefix-length: 24 ipv6: enabled: false link-aggregation: mode: active-backup options: miimon: "100" fail_over_mac: "active" port: - eno1 - eno2 dns-resolver: config: server: - 192.168.70.1 search: - lab.example.com routes: config: - destination: 0.0.0.0/0 next-hop-address: 192.168.70.1 next-hop-interface: bond0 table-id: 254 

Key points:

  • rendezvousIP is the IP of one control plane node (here master-0). That node runs the temporary assisted-service and bootstraps the cluster. It must be a static address you fully control; if it is wrong or unreachable, nothing ever registers and the ISO just sits there.
  • interfaces[].macAddress is the matching key: the agent applies networkConfig to the host whose NIC MAC matches. A typo means the host boots with DHCP (or no network at all).
  • link-aggregation.port is the modern NMState field. On older NMState versions the key is slaves — check nmstatectl show on a RHCOS node if in doubt.
  • Prefer active-backup unless the switch ports are really configured for LACP. Mode 802.3ad against a switch without a port-channel brings the bond down and the host never appears in the inventory.

A VLAN on top of the bond is a common variant:

 - name: bond0.70 type: vlan state: up vlan: base-iface: bond0 id: 70 ipv4: enabled: true dhcp: false address: - ip: 192.168.70.21 prefix-length: 24 

If you add a VLAN, the default route's next-hop-interface must be bond0.70, not bond0.

You can validate the NMState snippet offline before generating the ISO:

sudo dnf install -y nmstate nmstatectl gc nmstate-bond0.yaml # generates the NetworkManager keyfiles, catches schema errors 

Step 3: generate the ISO and boot

mkdir -p ~/ocp1/backup cp install-config.yaml agent-config.yaml ~/ocp1/backup/ cp install-config.yaml agent-config.yaml ~/ocp1/ openshift-install --dir ~/ocp1 agent create image --log-level=debug ls -lh ~/ocp1/agent.x86_64.iso 

The installer consumes and deletes both YAML files from the asset directory — hence the backup copy. Expose the ISO via the BMC virtual media (iDRAC, iLO, Redfish) or write it to a USB key, and boot all five machines. Boot order and UEFI/legacy mode must match on every node; a node booted in legacy BIOS while the others are UEFI will install but fail to come back after the first reboot.

Step 4: follow the installation

openshift-install --dir ~/ocp1 agent wait-for bootstrap-complete --log-level=info openshift-install --dir ~/ocp1 agent wait-for install-complete export KUBECONFIG=~/ocp1/auth/kubeconfig oc get clusterversion oc get nodes -o wide oc get pods -n openshift-kni-infra 

While it runs, the assisted-service REST API on the rendezvous node is the source of truth for validations:

curl -s http://192.168.70.21:8090/api/assisted-install/v2/clusters \ | jq -r '.[0].validations_info | fromjson | to_entries[].value[] | select(.status != "success") | "\(.id): \(.message)"' 

And per host:

ssh core@192.168.70.21 'sudo journalctl -u assisted-service -b --no-pager | tail -n 80' ssh core@192.168.70.22 'sudo journalctl -u agent.service -b --no-pager | tail -n 40' ssh core@192.168.70.21 'sudo agent-gather -O' > agent-gather.tar.xz 

The validation failures you will actually hit

ntp-synced — "Host couldn't synchronize with any NTP server".
UDP/123 blocked, or no NTP source at all. Fix with additionalNTPSources in agent-config.yaml, or verify on the node with chronyc sources -v.

belongs-to-machine-cidr — "Host does not belong to the machine network".
The node got an address from a different subnet (often a second NIC with DHCP), or machineNetwork is wrong. Check ip -brief a on the node and make sure only the intended interface carries a routable address.

api-vips-free / "VIP is already in use".
Someone reused the IP, or a previous failed install left keepalived running on a node. Re-run arping -D and reboot leftovers.

dns-wildcard-not-configured.
Remove the wildcard on *.ocp1.lab.example.com. Only *.apps.ocp1.lab.example.com is allowed.

"No connectivity to the majority of hosts in the cluster".
Usually layer 2: a bond down because of an LACP mismatch, a VLAN missing on one switch port, or an MTU mismatch. If you use jumbo frames, every interface and every switch port must agree. Test from a node:

ping -M do -s 8972 -c 3 192.168.70.22 # 9000 MTU minus 28 bytes of headers 

Any fragmentation error means the path MTU is smaller than advertised — OVN-Kubernetes Geneve traffic will break later even if the install succeeds.

Host never appears in the inventory.
Wrong rendezvousIP, wrong MAC in interfaces, or the node cannot reach port 8090 on the rendezvous host. Log in with the console and run ip a, nmcli con show, curl -v http://<rendezvousIP>:8090/api/assisted-install/v2/versions.

has-min-valid-disks or install on the wrong disk.
rootDeviceHints.deviceName points to a device that does not exist, or leftover RAID/LVM metadata confuses the installer. Prefer stable hints (wwn, serialNumber) over /dev/sdX, which can be reordered across boots:

 rootDeviceHints: wwn: "0x6000c29abcdef0123" 

Wipe stale metadata from the live ISO shell: sudo wipefs -a /dev/sda.

Security best practices

  • Never commit the pull secret or sshKey inline. Keep install-config.yaml in a secrets manager (Vault, SOPS, Sealed Secrets) and template it in CI. The generated ~/ocp1/auth/kubeadmin-password and kubeconfig are cluster-admin credentials — treat the asset directory as a secret, and delete the ISO after installation (it embeds the pull secret).
  • Rotate kubeadmin out. As soon as an identity provider (LDAP, OIDC) is wired in and a real cluster-admin exists, delete the bootstrap account:
 oc delete secret kubeadmin -n kube-system 
  • Restrict the API VIP. api.ocp1... should not be reachable from a user VLAN. Put the VIP on a management network and filter TCP/6443 at the firewall; api-int only needs to be reachable by the nodes.
  • Enable FIPS at install time (fips: true) if you need it — it cannot be turned on afterwards.
  • Encrypt the root disk with TPM v2 or Tang, declared in install-config.yaml so it applies from the first boot:
 platform: baremetal: apiVIPs: - 192.168.70.10 ingressVIPs: - 192.168.70.11 capabilities: baselineCapabilitySet: v4.14 # disk encryption is configured via agent-config.yaml or a Day-1 MachineConfig 

With the agent-based installer, TPM-backed LUKS is declared per host group through a Day-1 MachineConfig placed in ~/ocp1/openshift/ before generating the ISO.

  • Lock down the BMC network. Virtual media means the BMC can mount any ISO on your control plane. Separate VLAN, no default credentials, no internet route.
  • Use a mirror registry with signature verification in disconnected setups, and add imageDigestSources to install-config.yaml rather than disabling signature checks.
  • Limit SSH access to nodes. The sshKey is for break-glass debugging; collect logs through oc adm must-gather once the API is up, not through persistent SSH sessions.

Conclusion: key takeaways

A bare metal OpenShift install is 20% installer and 80% network preparation. If you remember five things:

  1. DNS first. api, api-int and *.apps must resolve, and there must be no wildcard on the cluster domain itself.
  2. VIPs must be free, static and inside machineNetwork — validate with arping -D, not ping.
  3. NMState configuration is matched by MAC address. One typo and the host silently falls back to DHCP or never registers.
  4. Use active-backup bonding unless you have confirmed LACP on the switch ports, and verify the MTU end to end with ping -M do.
  5. The validation engine already tells you what is wrong. Query the assisted-service API on the rendezvous node instead of guessing, and keep agent-gather output for anything you escalate.

Keep install-config.yaml and agent-config.yaml in Git (secrets externalised), and the next cluster becomes a 30-minute job instead of a two-day investigation.