When Kubernetes Health Checks Lie: Readiness, Liveness, Startup Probes and Real Service Availability

https://builder.aws.com/content/3Lu6BGwyRJoJ6dfIsMVaBmwZ4Z9/when-kubernetes-health-checks-lie-readiness-liveness-startup-probes-and-real-service-availability

Why healthy Pods can still deliver broken services, and how to engineer health checks that reflect production reality.

When Kubernetes Health Checks Lie: Readiness, Liveness, Startup Probes and Real Service Availability

A Kubernetes Deployment reports all replicas available. Every Pod is Running. Readiness probes are passing. The cluster dashboard is green.

Customers are still receiving HTTP 503 responses.

This is not necessarily a Kubernetes bug. It is a gap between what Kubernetes considers healthy and what users experience as available.

Kubernetes health probes answer specific operational questions. They do not establish end-to-end service reliability.

The problem begins when platform teams treat those answers as interchangeable.

A liveness probe cannot prove that an application can serve requests. A readiness probe cannot prove that an entire business transaction succeeds. A startup probe cannot guarantee that an application has completed every dependency initialization.

These mechanisms are valuable precisely because their responsibilities are narrow.

The engineering challenge is to preserve those boundaries while building a stronger definition of availability around them.

1. Three probes, three different decisions

Kubernetes provides three probe types:

The distinction matters.

Readiness is a traffic-routing decision.

Liveness is a process-recovery decision.

Startup is a startup-phase protection mechanism.

They should not share one endpoint merely because exposing three endpoints requires additional engineering.

Consider an application that loses access to a database for 20 seconds.

Should Kubernetes restart it?

Not necessarily.

Should Kubernetes continue routing requests to it?

That depends on whether it can still serve useful traffic.

Should the startup probe fail?

If the application has already started successfully, startup probing is no longer the relevant mechanism.

The same dependency failure can require different actions depending on application state and operational intent.

2. Why HTTP 200 does not mean healthy

A common health endpoint looks like this:

@app.get("/health") def health(): return {"status": "ok"} 

The endpoint responds successfully whenever the HTTP server is running.

That proves a limited fact: the process can respond to that particular request.

It does not prove:

• The application can authenticate users.
• Its database queries are succeeding.
• Its message producer can publish events.
• Its worker pool has available capacity.
• Its critical downstream dependencies are reachable.
• Its business transactions complete within acceptable latency.

A Pod can pass this probe while the service is effectively unavailable.

The opposite is also possible.

A health endpoint may fail because an optional dependency is unavailable even though the application continues serving its most important functionality.

A useful health check must therefore answer a precise question rather than attempt to summarize the entire system with one Boolean.

3. Readiness should represent traffic eligibility

Readiness is commonly the most consequential application probe because it influences whether traffic is sent to a Pod through Kubernetes Service routing.

Consider a checkout service.

It requires local configuration, a functioning request handler, and an initialized connection pool.

Those conditions may be appropriate readiness prerequisites.

But suppose its analytics service is unavailable.

If analytics is asynchronous and noncritical, marking checkout unready could be the wrong response.

The checkout service may still process orders successfully.

A dependency-aware readiness policy should distinguish critical and optional dependencies.

checkout-api | +--> HTTP handler Critical | +--> Database pool Critical | +--> Payment provider Context-dependent | +--> Analytics publisher Optional 

A practical readiness contract might be:

Ready when the application can accept its intended class of traffic without immediately failing due to a local condition.

That is deliberately narrower than guaranteeing that every downstream operation will succeed.

For example, readiness might return HTTP 503 when the application has exhausted its request-processing capacity or is still initializing required resources.

It should not automatically return 503 whenever any dependency reports an error.

The distinction prevents local failures from becoming cluster-wide routing events.

4. The dependency amplification trap

Suppose a service has 20 replicas.

Every replica uses the same database.

The database becomes temporarily unavailable.

Each readiness probe checks the database directly.

Database unavailable | v 20 readiness probes fail | v 20 Pods become unready | v No eligible Service endpoints | v All incoming requests fail 

The database failure was already serious.

The readiness policy can make recovery harder by removing all application endpoints simultaneously.

Now imagine the database recovers.

Every replica attempts to reconnect at approximately the same time.

The database experiences a connection storm.

Readiness begins flapping.

The service may oscillate between zero and many eligible endpoints.

This is why dependency checks require careful design.

Consider using bounded timeouts, cached local dependency state, connection-pool health, and application-level circuit breakers rather than running an expensive dependency transaction on every probe.

But cached health introduces another tradeoff: it can be stale.

A readiness check should be cheap, predictable, and locally actionable. It should not become a distributed transaction.

5. Liveness should detect irrecoverable local failure

Liveness is often overused.

Consider:

livenessProbe: httpGet: path: /health port: 8080 periodSeconds: 10 failureThreshold: 3 

If /health performs a database query, a database outage may cause the liveness probe to fail.

Kubernetes then restarts the container.

The replacement container starts, attempts to connect to the same unavailable database, and fails again.

Database outage | v Liveness failure | v Container restart | v Cold startup | v Database still unavailable | v Another restart 

Instead of recovering the database dependency, Kubernetes creates a restart loop.

This can also destroy useful diagnostic state, increase connection attempts, and consume startup resources.

Liveness should generally detect conditions where restarting the local process is a reasonable recovery action.

Examples might include an unrecoverable application deadlock or a critical internal execution loop that has stopped making progress.

Even then, the probe must be designed carefully.

An HTTP server may continue answering /live while background workers are deadlocked.

Conversely, a temporarily saturated request handler may fail a probe even though the application would recover without a restart.

Liveness is a recovery mechanism, not an availability metric.

6. Startup probes prevent premature recovery actions

Some applications require significant startup time.

They may load large configuration sets, initialize caches, perform JVM warm-up, or establish connections before becoming operational.

Without a startup probe, aggressive liveness settings can repeatedly terminate an application before initialization completes.

A startup probe creates a protected startup window.

startupProbe: httpGet: path: /startup port: 8080 periodSeconds: 5 failureThreshold: 24 readinessProbe: httpGet: path: /ready port: 8080 periodSeconds: 5 failureThreshold: 2 livenessProbe: httpGet: path: /live port: 8080 periodSeconds: 10 failureThreshold: 3 

The startup configuration allows approximately two minutes for the startup condition to succeed before the failure threshold triggers a restart, subject to probe scheduling and execution timing.

While the startup probe has not succeeded, Kubernetes does not execute the configured liveness and readiness probes.

Once startup succeeds, those probes take over.

This is preferable to giving liveness an excessively large initial delay because it allows startup completion to be detected dynamically.

However, startup probes should not conceal indefinitely hung initialization.

Their failure budget must reflect a realistic upper bound for startup under expected operating conditions.

7. A production-oriented probe design

For a typical HTTP application, the endpoints can have distinct responsibilities.

/live | +--> Is the local process capable of continuing execution? /ready | +--> Is the application initialized? +--> Can it accept intended traffic? +--> Is required local capacity available? /startup | +--> Has required initialization completed? 

A corresponding Deployment configuration could be:

apiVersion: apps/v1 kind: Deployment metadata: name: checkout-api spec: replicas: 4 selector: matchLabels: app: checkout-api template: metadata: labels: app: checkout-api spec: terminationGracePeriodSeconds: 45 containers: - name: checkout-api image: example.com/checkout-api:v2 ports: - containerPort: 8080 startupProbe: httpGet: path: /startup port: 8080 periodSeconds: 5 failureThreshold: 24 readinessProbe: httpGet: path: /ready port: 8080 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 2 livenessProbe: httpGet: path: /live port: 8080 periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 

This is an illustrative baseline, not a universal production configuration.

The endpoint implementations, probe intervals, failure thresholds, and termination behavior must be validated against actual application behavior.

In particular, do not assume that passing /ready means the application has adequate throughput capacity under peak load.

Readiness determines eligibility, not performance guarantees.

8. Probe timing is part of system behavior

Probe configuration introduces detection latency.

For example:

Readiness interval: 5 seconds Failure threshold: 2 Timeout: 2 seconds 

A failure will not necessarily be detected immediately.

The actual transition depends on probe scheduling, response timing, and the configured threshold.

A rough planning model is:

Detection latency ≈ probe scheduling delay + time needed to accumulate failures 

That matters during a rollout.

A Pod may begin returning application errors before Kubernetes marks it unready.

The load balancer may continue sending requests during the detection interval.

Likewise, recovery requires successful readiness evaluation before traffic eligibility is restored.

Probe settings therefore trade off:

• Failure detection speed
• Sensitivity to transient failures
• Probe overhead
• Readiness flapping
• Recovery latency

Very aggressive probes can turn short-lived resource contention into unnecessary traffic removal.

Very conservative probes can leave unhealthy Pods eligible for too long.

The correct settings should be derived from failure tolerance and measured application behavior.

9. Kubernetes readiness is not the same as load-balancer readiness

On Amazon EKS, another important boundary exists between Kubernetes endpoint readiness and external load-balancer target health.

For example:

Client | v AWS Application Load Balancer | v Target Group | v Kubernetes Pod 

A Pod may become Kubernetes Ready before the ALB has completed its own target health evaluation.

The reverse transition also has propagation timing.

During rolling deployments, this distinction can matter.

A new Pod may be eligible for Kubernetes routing while the external load balancer has not yet established that it can receive traffic.

The AWS Load Balancer Controller supports Pod readiness gates for supported configurations, allowing load-balancer target health to participate in Kubernetes Pod readiness during rollouts.

But readiness gates require appropriate controller configuration and should not be assumed to exist automatically in every EKS deployment.

The important architectural principle is that each traffic-routing layer has its own health state.

Application readiness | v Kubernetes Pod readiness | v EndpointSlice eligibility | v Load-balancer target health | v User request success 

These states are related.

They are not identical.

10. Termination behavior is another health boundary

A Pod becoming unready does not instantly guarantee that no traffic can reach it.

Endpoint updates, load-balancer deregistration, existing connections, and application shutdown all have their own timing.

Consider a rolling deployment.

Old Pod receives termination signal | v Kubernetes begins endpoint updates | v External routing state converges | v Existing requests finish | v Container exits 

If the application exits immediately after receiving SIGTERM, in-flight requests may fail.

A robust shutdown strategy should coordinate application-level graceful termination with the platform's termination grace period and external routing behavior.

This often requires:

• Stop accepting new work when shutdown begins.
• Allow active requests to complete within a bounded interval.
• Stop or drain background consumers appropriately.
• Release resources safely.
• Exit before the configured termination deadline.

For asynchronous workers, termination also needs to account for message acknowledgement semantics.

Killing a worker during message processing can create redelivery or partial side effects.

Health and lifecycle engineering cannot be separated completely.

11. Probe success does not establish an SLO

This is the central distinction.

Kubernetes might report:

Ready replicas: 10/10 

The business service might report:

Successful transactions: 92% 

Both observations can be accurate.

The Pods are eligible to receive traffic.

The application is failing 8% of transactions.

An SLO must be based on service behavior rather than inferred from Kubernetes object state.

For a request-driven service, useful SLIs might include:

Availability: successful eligible requests / total eligible requests Latency: proportion of eligible requests completed within the defined latency threshold Correctness: proportion of transactions producing the expected result 

These definitions need precise eligibility rules and business context.

For example, an intentional HTTP 4xx response caused by invalid client input should not automatically count as service unavailability.

A payment authorization declined by a bank may be a valid business result rather than an application failure.

A health endpoint cannot encode all of these distinctions.

That is why real availability needs application telemetry, not just Kubernetes probes.

12. Build an availability model across layers

A more complete model separates four levels.

The monitoring architecture should reflect those boundaries.

 User Journey | v Synthetic Transactions | v Application SLIs | v Service Routing | v Kubernetes Readiness | v Container Health 

This is not a hierarchy where lower-layer success guarantees upper-layer success.

It is a dependency map for diagnosis.

When user-facing availability drops, engineers should be able to determine whether the failure originated in the application, routing layer, Pod lifecycle, or underlying infrastructure.

13. Test probes through fault injection

Health-check configurations should be tested against failure scenarios, not approved solely by code review.

A useful validation matrix includes:

These tests expose a subtle but important property: probe behavior is part of the application's failure-management design.

A probe should have an expected response for every relevant failure class.

If engineers cannot explain why a particular failure should trigger restart, traffic removal, or neither, the health contract is underspecified.

14. The operational tradeoff

There is no universally perfect readiness or liveness endpoint.

A shallow readiness check may leave partially degraded Pods eligible for traffic.

An overly deep readiness check may cause a shared dependency outage to remove every Pod from routing.

A shallow liveness check may miss internal deadlocks.

An overly aggressive liveness check may create restart storms during recoverable conditions.

The objective is not maximum sensitivity.

It is selecting the right automated action for each failure mode.

That requires understanding:

Failure detected | v Can local restart fix it? | +--> Yes: consider liveness | +--> No | v Can this Pod serve intended traffic? | +--> No: readiness failure | +--> Yes: remain ready and expose degradation through telemetry 

Even this decision tree is simplified.

A shared dependency outage may require circuit breaking, graceful degradation, or traffic shaping rather than making every Pod unready.

The important point is to choose the action intentionally.

15. What senior platform teams should standardize

The strongest platform standard is not a mandatory /health endpoint copied into every application.

It is a documented health contract.

For each service, define:

Startup contract: What initialization must complete, and how long may it reasonably take?

Readiness contract: What local conditions make the Pod unsuitable for intended traffic?

Liveness contract: Which local failures justify restarting the process?

Termination contract: How does the application drain requests, consumers, and other work?

Availability contract: Which externally observable transactions define whether the service is actually working?

Then test those contracts during rollout, dependency failure, saturation, and recovery.

Platform teams can standardize instrumentation, probe configuration patterns, and deployment guardrails.

Application teams must still own the domain-specific meaning of availability.

A generic platform endpoint cannot know whether an order was placed correctly.

Conclusion: Green Pods are not the objective

Kubernetes health checks are necessary infrastructure controls.

They are not business availability guarantees.

Readiness controls traffic eligibility.

Liveness controls restart behavior.

Startup probes protect initialization.

External load balancers introduce additional routing health states.

SLOs measure whether users receive the service they were promised.

Treating all of these as one health signal creates false confidence and sometimes makes incidents worse.

The better engineering question is not:

"Are all Pods healthy?"

It is:

"Which failure conditions can Kubernetes recover automatically, which require traffic management, and which must be detected through user-facing reliability signals?"

A production-ready health strategy connects those answers without confusing them.

A green Kubernetes dashboard is useful evidence. It is not proof that the service works.

Authoritative references

1. Kubernetes: Configure Liveness, Readiness and Startup Probes
2. Kubernetes: Pod Lifecycle
3. Kubernetes: EndpointSlices
4. AWS Load Balancer Controller: Pod Readiness Gate

Technical note: The probe behaviors and Kubernetes examples are based on documented Kubernetes semantics. Application-specific thresholds and failure responses are illustrative and require validation in the target environment.