A liveness probe is meant to recover one stuck container. Configured badly, it restarts every healthy pod you have at the same moment, and a 30 second blip becomes an outage.

Here is how that happens, and how to set the three probes so it does not.

Three probes, three jobs

Kubernetes gives a container three probes. They look alike in YAML and do very different things when they fail.

  • Startup probe: has the app finished starting? Until it passes, the other two probes do not run.
  • Readiness probe: can this pod take traffic right now? If it fails, the pod is removed from the Service endpoints. It is not restarted.
  • Liveness probe: is this process stuck beyond recovery? If it fails, the kubelet restarts the container.

The difference between "stop sending traffic" and "kill it" is the whole story.

How a blip becomes an outage

A common setup: one /health endpoint that checks the database, used for both readiness and liveness.

  1. The database has a 30 second failover.
  2. /health starts failing on every pod, because every pod shares that database.
  3. Liveness fails everywhere, so every container is restarted at roughly the same time.
  4. The database comes back. Now all pods are cold-starting together, reconnecting together, and warming caches together.
  5. The startup load slows the app down, probes time out again, and you get a restart loop.

Nothing was wrong with the pods. Restarting them could not fix the database. The probe turned a dependency problem into a self-inflicted one.

What each probe should check

Liveness: only the process itself. Can the event loop or a worker thread answer a trivial request? No database, no cache, no downstream API. If a restart would not fix the failure, liveness should not be testing for it.

Readiness: what this pod needs to serve a request. It is reasonable for readiness to notice that the pod has lost something it cannot work without. Be careful with shared dependencies though: if every pod fails readiness at once, the Service has no endpoints and clients get connection errors instead of your app's own error handling. For a shared dependency, serving a clear error is often better than vanishing.

Startup: slow boots. If the app takes a while to start (migrations, cache warm-up, JIT), use a startup probe with a generous failureThreshold instead of a long initialDelaySeconds on liveness. You get a long allowance at boot and a quick liveness check afterwards.

A starting point

startupProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 5 failureThreshold: 30 # up to 150s to start livenessProbe: httpGet: { path: /livez, port: 8080 } periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 readinessProbe: httpGet: { path: /readyz, port: 8080 } periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 2 

/livez returns 200 if the process can respond at all. /readyz checks what this pod needs before it takes traffic. Treat the numbers as a starting point and tune them against how your app really behaves under load.

Things that catch people out

  • Timeouts under load. The default timeoutSeconds is 1. A busy but healthy pod can miss that, fail liveness and be restarted exactly when you need it most.
  • Tight CPU limits. A throttled container answers probes slowly. If probes start failing during traffic peaks, look at throttling before you blame the app.
  • Liveness on a shared thread pool. If the probe handler queues behind real requests, saturation looks like a hang. That may be what you want, or it may restart pods that were just busy. Decide on purpose.
  • Same endpoint for both probes. It is the shortest path to the outage described above.

The short version

Readiness answers "should this pod get traffic?". Liveness answers "would a restart help?". If the honest answer to the second question is no, keep that check out of the liveness probe.