Originally published at tothenoc.substack.com

Why "SSH works" is not the test we use.

We build the same hardened images for x86 and ARM. Same recipe, same profile, same scan.

On x86 they came up "running". On ARM two of them came up "degraded".

SSH worked. The scan was clean. Nothing in the build log was red. If our boot test had only checked SSH, those images would be on the marketplace today.

What failed was rngd, the hardware random number daemon. On x86 it finds a hardware source and is happy. The ARM instance we tested on gives it nothing: no hardware RNG device, nothing it accepts. rngd tries every source it knows, finds none, exits with an error. systemd marks the unit failed. One failed unit and the whole system is "degraded".

This is not our bug alone. The same failure is reported upstream for ARM64 virtual machines on more than one cloud, 100% reproducible.

The fix was not to silence the unit. On a current kernel the random number generator seeds itself; the kernel has had its own jitter entropy collector since 5.4. On ARM rngd had nothing to add, so we took it out: on aarch64 the recipe removes rng-tools. Rebuilt, booted, "running", scan clean.

A week later, a different cause, same symptom, this time on both architectures. A small watcher unit on a system file was triggered in a burst by the cloud guest agent creating users at first boot. More than five starts in ten seconds, systemd hits the start limit, unit failed, system degraded. Again: SSH fine, scan fine.

So the rule stays:

  • boot it on a real instance, every architecture we ship,
  • SSH has to answer,
  • systemctl is-system-running has to say "running", and "degraded" fails the build,
  • then the hardening profile runs again on the live system.

"Degraded" is easy to ignore. It is a word in a status line, not an outage. But it is the first thing a customer's monitoring will see, and it hides the next failed unit behind the one you already know about.

If your fleet has ARM instances, run this once:

systemctl is-system-running systemctl --failed

If the first line says "degraded" and nobody knows why, that is your evening.

The monitoring side of the same problem: Green is not working.

-- Alex