Kubernetes is good at checking itself. The kubelet runs liveness, readiness, and startup probes against each container, restarts the ones that hang, and takes unready ones out of their Service. But those probes never leave the cluster. The kubelet calls the pod's IP directly, so a probe never touches the path your users take: public DNS, the cloud load balancer, TLS termination, the ingress controller, the ingress rule, and the Service in front of your pods.

That path is where a lot of "everything is green and nobody can reach us" outages happen:

  • An ingress rule points at the wrong Service name or port after a rename. The ingress returns 404 or 503 while every pod is Ready.
  • A TLS secret is replaced with the wrong certificate, or the controller falls back to its default self-signed one.
  • The ingress controller itself is unhealthy, and every host behind it returns 502 or 504 at once.
  • external-dns, or a manual change, leaves the public record pointing at an old load balancer.
  • A security group or loadBalancerSourceRanges change blocks internet traffic while in-cluster traffic keeps working.

More probes won't catch these. One check per public host, run from outside the cluster, will.

What to check from outside

  1. A readiness URL that goes through the ingress. Expose https://api.example.com/readyz on the normal ingress path and return a small JSON body only when the app can serve. A 200 there means DNS, TLS, the load balancer, the ingress rule, and the Service all worked for that request.
  2. The body, not only the status. Default backends can answer 200 with the wrong page. Assert on a parsed JSON field ($.status == "ok"), not raw text: "status":"ok" and "status": "ok" are the same JSON and different strings.
  3. The certificate. Make the check fail on an invalid certificate, so a wrong secret or the self-signed fallback shows up as a failure.
  4. The public DNS record. If external-dns manages it or you move between load balancers, check that the name still resolves to what you expect.
  5. Non-HTTP ports. A LoadBalancer Service for Postgres, Redis, or MQTT can be checked with a TCP connect.
  6. More than one region. A regional routing problem looks like a full outage from one place and like nothing from anywhere else.

Alert on confirmed failures, not on rollouts

A rolling deploy, an HPA scale-up, or a node drain can drop a single request. Alert after 2 or 3 failures in a row, set the timeout a little above the readiness endpoint's normal response time, and make sure a flapping check sends one message, not one per change.

What an outside check does not cover

  • CronJobs and workers have no public URL; use a heartbeat (the job pings a URL when it finishes).
  • Certificate expiry ahead of time: keep cert-manager's own alerts on.
  • Internal Services stay with your probes and Prometheus alerts.

A starting checklist

  • [ ] /readyz through the ingress, JSON only when the app can serve
  • [ ] One check per public host, with a body-field assertion and certificate verification on
  • [ ] One DNS check per hostname external-dns manages
  • [ ] One TCP check per non-HTTP port you expose
  • [ ] Alert after 2–3 consecutive failures
  • [ ] Each public host as its own component on your status page

Probes keep your pods healthy. A check from outside tells you whether anyone can reach them.

I build ScaledPapers, which runs API, Website, DNS, and TCP checks from outside your infrastructure and ties them to a status page. The full version of this post, including how each checklist step maps to those checks, is on our blog.