If you operate Kubernetes, CrashLoopBackOff becomes the error you see most often and diagnose worst.
The name is misleading: CrashLoopBackOff is not the problem. It's the kubelet's restart policy with exponential backoff telling you that your container keeps dying.
The real error is somewhere else, and kubectl describe alone rarely shows it.
This guide is the workflow I use in production environments, ordered from the fastest signals to the deepest ones. Every step produces evidence you can keep — which matters when you need to escalate to another team or even show in front of your customer when the problem is "beyond your control" :-)
Step 1: Read the exit code before anything else
The container's exit code tells you which family of problem you're in:
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}' For pods with multiple containers, list them first with
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].name}' and adjust the index.
The codes that appear constantly in practice:
- Exit 0 in a loop — the process finished successfully but the container expects it to keep running. Common with batch-style entrypoints, wrong PID 1, or an entrypoint that daemonizes and exits.
- Exit 1 — application error. Read the logs (next step).
- Exit 137 — SIGKILL. Almost always OOMKilled. Confirm with:
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' If it says OOMKilled, no amount of log reading will help — raise the memory limit or fix the leak.
NOTE FROM THE FIELD: When I say "fix the leak", I mean, take care of the internal issue that is using and killing your memory. It has been pretty common for me to find bad sizing issues, but also an I/O process that can't write and has nowhere to flush, so memory usage collapses.
- Exit 143 — SIGTERM, graceful shutdown rejected or too slow.
- Exit 126/127 — permission denied or command not found in the entrypoint. Your image and your runtime disagree.
Step 2: Read the previous logs, not the current ones
The current logs are usually empty or useless — the container just restarted. The crash is in the previous container instance:
kubectl logs <pod-name> --previous --tail=100 This single flag is the most underused command in Kubernetes troubleshooting. The stack trace, the connection refused, the missing file — it's almost always in there.
If the logs are empty, the process is dying before writing anything: check the exit code again (126/127 points to the entrypoint), and check whether the log driver of your runtime is actually shipping stdout.
Step 3: The five causes that cover most real cases
In order of how often I find them in production:
1. Configuration injected at the wrong moment. The pod starts before the ConfigMap/Secret exists, or an env var references a key that was renamed. The app crashes on startup, the next attempt finds the resource (if a controller created it late) or crashes forever. Check:
kubectl describe pod <pod-name> | grep -A5 -i "environment\|mounts" 2. A dependency that isn't ready. Databases, Kafka brokers, LDAP servers. The app has no retry logic and dies on first connection refused. The fix is an initContainer or a startup probe — not more replicas.
3. Liveness probe killing a slow starter. The app needs 90 seconds to warm up; the liveness probe starts failing at 30. Kubernetes kills it; the loop begins. The tell: the container dies at a suspiciously regular interval, right after startup. Use a startupProbe for slow applications and keep liveness for steady-state.
4. Running as non-root against root-owned code. Hardened images (or an OpenShift-style cluster with restricted security contexts) run your container as a random UID. Your app tries to write to /tmp or read a root-owned file and gets permission denied. Exit code 1 or 126.
5. OOMKilled. Exit 137. Already covered — but note that a JVM or Python process inside a container with a memory limit needs its heap configured against the limit, not the node's memory.
Step 4: Events and timing
kubectl get events --field-selector involvedObject.name=<pod-name> --sort-by=.lastTimestamp Look at the pattern in time, not just the messages: a container killed exactly every N seconds is a probe or a dependency timeout. A container killed at random intervals is a resource or data-dependent failure.
A reusable evidence collector
When this happens across many pods, I don't type these commands one by one. One loop, saved output, timestamped — everything an escalation needs:
#!/usr/bin/env bash POD=$1 OUT="crashloop-${POD}-$(date +%Y%m%d-%H%M%S)" mkdir -p "$OUT" kubectl get pod "$POD" -o yaml > "$OUT/pod.yaml" kubectl describe pod "$POD" > "$OUT/describe.txt" kubectl logs "$POD" --previous --tail=200 > "$OUT/previous.log" 2>&1 kubectl logs "$POD" --tail=50 > "$OUT/current.log" 2>&1 kubectl get events --field-selector involvedObject.name="$POD" \ --sort-by=.lastTimestamp > "$OUT/events.txt" tar czf "${OUT}.tgz" "$OUT" echo "Evidence package: ${OUT}.tgz" Run it, attach the tarball to the ticket, and you've done in 10 seconds what normally takes 20 minutes of back-and-forth.
Summary
CrashLoopBackOff is a symptom with a short list of real causes. Exit code first, previous logs second, then the five usual suspects: config timing, missing dependencies, aggressive probes, UID permissions, and memory limits. The commands are cheap; the discipline of collecting evidence in order is what separates a 5-minute diagnosis from an afternoon of guessing.



