Follow one failed checkout from the browser to the dependency, so every debugging step gives you a reason for the next one.
Your customer has clicked Pay twice, and you are staring at a Kubernetes dashboard that seems delighted with itself. Every pod is Running, readiness checks are green, and CPU usage looks ordinary. Meanwhile, checkout still fails, which makes all that reassuring green feel a little less reassuring.
Honestly, this is the kind of debugging exercise I want in my practice notes because it exposes a very tempting assumption. A process can answer its health check while the operation a customer needs is broken.
Let’s work through a hypothetical checkout failure and use one request to decide where to look next. By the end, you should have a small investigation you can explain in an interview or adapt to your own application.
Start with the thing your customer cannot finish
Before opening another dashboard, turn the complaint into something you can investigate. Can the customer load checkout but fail when submitting payment? Does every payment method fail? Does the error appear immediately or after a long wait?
Suppose the page loads normally, but submitting a card payment returns a 502 after roughly three seconds. Another payment method succeeds. These observations are invented for our walkthrough, but notice how much they narrow the search compared with a vague report that checkout is broken.
Take the failed request ID and compare its route, timestamp, response code, and downstream spans with a successful checkout from the same period. Suppose the failing trace reaches the checkout application, then spends nearly all its time waiting for one payment provider.
Now we have a specific branch to investigate. We still need to distinguish a provider problem from network access, credentials, or client configuration. A slow span tells us where time went; it does not explain why by itself.
Ask what that green check actually measured
Readiness helps decide whether a pod receives normal Service traffic. Liveness can trigger a container restart. Neither check automatically exercises a full checkout, so a lightweight health endpoint can succeed while the payment path fails.
In our example, that means a successful health check and a failed payment can both be accurate observations. We need the request path to connect them.
Check whether the Service selects the intended pods and whether its EndpointSlices contain the expected backends. Kubernetes’ official Service debugging guide explains how to inspect selectors, endpoints, and ports. Also compare the Service’s target port with the port the application actually serves.
These read-only commands provide a starting point. Replace the placeholders with your namespace and Service name, and keep the failing request’s timestamp nearby.
kubectl get pods -n <namespace> -o wide kubectl describe service <service> -n <namespace> kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<service> kubectl get events -n <namespace> Compare the endpoint addresses with the expected pod IPs and inspect their readiness conditions. Events can help explain scheduling or container problems, but they are not an HTTP request history. To establish which pod handled checkout, use the application logs or traces that actually record that request.
Make each check shrink the search area
I prefer a sequence that leaves me with a smaller set of possibilities after each step. Otherwise, debugging can turn into an impressive collection of terminal tabs with very little progress.
Confirm the failure at the edge. Use an existing trace or reproduce it safely in a test environment. Record the route, response code, latency, and request ID. For payments, avoid creating real charges just to reproduce an error.
Follow routing to the intended workload. Check the gateway or Ingress route, then the Service selector, ports, and EndpointSlices. The existence of healthy pods does not establish that traffic reaches the right ones.
Inspect the pod that handled the request. Compare its logs, version, restart history, and resource pressure around the failure. If a container restarted, previous-container logs may hold a clue that current logs no longer show.
Follow its outbound calls. Check timing, errors, and retry counts for the relevant database, queue, identity provider, or payment API. Use the failing span to choose the next inspection.
Compare a success with a failure. Look for the earliest meaningful difference in route, version, configuration, dependency, or input. Similar successful requests are especially useful because they reduce the number of differences you have to explain.
For our checkout, imagine that both requests reach the intended workload, but only the failing one calls the card provider. That observation moves the investigation toward that integration. If the failures also began after a deployment, compare the provider endpoint, credentials, timeout settings, and network configuration between the working and failing versions.
A different pattern should change the plan. If failures occur only on one replica, inspect that replica’s configuration and connections before blaming a provider shared by every pod. That is the habit worth practising, because evidence should choose the next command.
During a live incident, preserve request IDs and relevant logs when it is quick to do so, but avoid delaying an approved mitigation to collect a perfect evidence bundle. Restoring checkout matters more than finishing the incident detective story.
Fix the signal without creating a restart party
Kubernetes defines startup, readiness, and liveness probes separately. Startup probes give initialization time before the other probes begin. Readiness failures make a pod unready for normal Service routing; liveness failures can restart the container after the configured threshold.
Before changing a probe, decide what action its failure should justify. Restarting every checkout pod during a shared payment-provider outage could add disruption while leaving the original problem untouched. Making readiness depend on that same provider could also remove otherwise useful capacity across the fleet.
For this exercise, add a separate checkout-success signal and, where appropriate, a safe synthetic journey. Track outcomes by payment method so an overall success rate does not hide the broken branch. A test-mode checkout can reveal integration problems, although it still cannot reproduce every production condition.
Let an AI helper earn its next suggestion
An AI assistant can help organise the evidence, but I want more from it than a confident paragraph naming a likely cause. Give it the failed request, a comparable success, and the configuration details you can safely share. Then ask which observation supports its explanation and what would make it change its mind.
Try this first in a controlled staging incident with read only access. Ask for the next diagnostic, what its possible results would mean, and which result would rule out the current theory. Keep production changes such as restarts, scaling, and configuration updates subject to your team’s review process.
That gives you a useful investigation partner while keeping responsibility for changes clear. A suggestion becomes much easier to evaluate when you can see what evidence it depends on.
Practice the explanation before memorising more commands
Close the terminal for a moment and explain our example to a teammate. Checkout loads, card payments fail after a delay, requests reach the intended workload, and the failing branch waits on one provider. Your next step is to distinguish the integration’s possible failure modes using the trace and configuration evidence.
Notice how much more useful that is than reporting that every pod is green. Someone else can pick up the investigation without retracing every command you ran.
For a focused practice run, try this Kubernetes pod troubleshooting question and explain what each check would change about your next step. If you cannot say why the result matters, pause before adding another command.
Which clue would you check first if checkout failed on only one payment method while every pod stayed Ready?
I’m curious whether you’d start with the request trace, a recent deployment, or something else entirely.
~ Emily
