In October our platform team added a Kyverno policy that sets runAsNonRoot: true on every pod that does not set it already. Most of our images ran as an unprivileged user, the change was announced a week ahead, and the announcement came with a list of the images that still ran as root, so that their owners could fix them first.
Twelve days later our invoicing partner asked why our nightly freight settlement file had stopped arriving. kubectl get jobs showed one export job that had been running for twelve days with no completions. Its pod was Pending, and its container was waiting with CreateContainerConfigError: container has runAsNonRoot and image has non-numeric user (exporter), cannot verify user is non-root.
The image did not run as root. Its Dockerfile ended with USER exporter. But the kubelet checks runAsNonRoot against the user the image declares, and it cannot tell from a name what UID that will be, because the name is only resolved inside the container's own /etc/passwd. So it refuses to start the container at all, and asks for a number. The list in the announcement had been made by reading USER lines, and exporter was not root.
A container that cannot be created has not failed. The pod stays Pending, the Job counts it as active, and our alert on failed jobs had nothing to see. The CronJob uses concurrencyPolicy: Forbid, so that two exports can never overlap, and every night at two it found the first job still active and skipped its run. Eleven nights went that way, each recorded as an event and nothing more.
Our services hit the same error, but not for long. Their next rollouts stalled, their pipelines went red, and their owners changed USER to a UID the same day. The export only ever created a pod at two in the morning, and nobody was watching it.
Every Dockerfile we own now ends with a numeric USER, 10001 for the export, and a check in CI rejects a USER line that is not a number. The export's job template has activeDeadlineSeconds of two hours, so a job that cannot start fails instead of waiting. We alert on any pod Pending for more than fifteen minutes, and on any CronJob whose kube_cronjob_status_last_successful_time is more than a day old.
The policy did what it promised. Our export was stopped by a check it could not pass, and the only thing looking at it was a scheduler that saw a job still running.
– Sergey Shinder
