Our EDI poller collects the purchase orders that suppliers drop on our SFTP server and imports them into the ERP. It runs as a single pod with the Recreate strategy, because two pollers would download the same files. Our alert for pods that keep failing was the one most teams write first:
increase(kube_pod_container_status_restarts_total{namespace="integration"}[15m]) > 3
with for set to five minutes.
At ten to eleven on a Friday night in October a change went out with a new setting for the SFTP server's host key, written in the wrong format. The poller read it at startup and exited. The alert paged the engineer on call eight minutes later. Ten minutes after that, before he had his laptop open, it resolved. He looked at the dashboard, saw the pod running, and went back to bed. It crashed about every five minutes until twenty to nine on Monday, when purchasing asked why no supplier orders had arrived since Friday. Two thousand three hundred files were waiting.
The kubelet restarts a container that keeps failing with an exponential back off: ten seconds, then twenty, then forty, doubling up to a cap of five minutes, and it resets the delay only after the container has run for ten minutes without trouble. In its first quarter of an hour our poller was restarted six times. After that it was restarted once every five minutes or so, never more than three times in any fifteen minute window, and three is not more than three. The expression stopped being true at about the eighteenth minute, Prometheus resolved the alert, and the pager sent the resolution the way it sends everything else.
The dashboard had not lied either. A container in back off is started, runs for a few seconds and stops, and a panel that shows the pod's phase at the moment you look will show Running some of the time.
Our rule measured how often the pod was restarted, and the kubelet controls that number. It lowers it on purpose for exactly the containers that have been failing longest.
We now alert on the state instead, the way the kube-prometheus rules do:
max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) >= 1
for fifteen minutes. That stays true for as long as the container keeps failing, however slowly it is being restarted. The poller also exports the age of the oldest file it has not imported, and that pages above forty five minutes on weekdays and two hours at weekends, because what mattered on Monday was the files and not the pod.
Kubernetes restarts a failing container less often the longer it fails, and an alert that counts restarts goes quiet at the same rate.
– Sergey Shinder
