Postmortem (hypothetical): our health check was so thorough it took every instance out of the load balancer at once

This is a hypothetical postmortem. The company, timings and details are invented to illustrate a real and common failure pattern. It does not describe a real incident or real data.

Summary

For 38 minutes on a weekday afternoon, our fictional company's API returned errors to every request. The underlying cause was a slowdown in the primary database, which on its own would have made the API slow, not unavailable.

What turned slow into down was our health check. The /health endpoint ran a database query. When the database slowed, the query exceeded the health check timeout on every instance at once. The load balancer and the orchestrator both concluded that every instance was unhealthy: the load balancer stopped sending traffic to all of them, and the orchestrator started restarting them. Restarting instances opened new database connections, adding load to the struggling database.

Impact

38 minutes of near-total API unavailability.
Mobile and web clients showed errors; background integrations queued and retried.
No data loss.

Timeline (all times local)

14:02 A large reporting query begins on the primary database, causing lock contention and slower queries across the board.
14:04 API latency rises. Requests are slow but succeeding.
14:05 /health responses start exceeding the 2-second timeout. Each instance needs three consecutive failures to be marked unhealthy.
14:06 All instances fail their health checks within the same 30-second window. The load balancer has no healthy targets and starts returning 503s.
14:06 to 14:20 The orchestrator's liveness probe, which used the same /health endpoint, restarts instances. Each restart opens a fresh connection pool. The database gets busier.
14:21 On-call identifies the reporting query and cancels it.
14:25 Database latency recovers, but instances are mid-restart and the connection storm continues.
14:40 Instances stabilise, pass health checks and return to the load balancer. Service restored.

Root cause

One endpoint was being used to answer three different questions:

Liveness: "Is this process stuck and in need of a restart?"
Readiness: "Should this instance receive traffic right now?"
Dependency health: "Is everything this service depends on working?"

The /health endpoint answered question 3, and both the load balancer (question 2) and the orchestrator (question 1) acted on it. A shared dependency failing made every instance look broken at the same moment, so the protective mechanisms removed all capacity together.

Restarting a healthy process because its database is slow can't help; it can only add load.

What went well

On-call found the long-running query quickly.
External monitoring flagged the outage within a minute, from outside the infrastructure.
Clients with retries and backoff recovered automatically once service returned.

What went wrong

Liveness depended on an external system. A process that was perfectly alive got killed.
Readiness failed all-or-nothing. No instance could stay in the pool, so a slowdown became an outage.
Restarts amplified the problem. Each new instance opened connections to a database already struggling.
The reporting query ran against the primary rather than a replica.

Action items

1. Split the endpoints.

GET /livez -> 200 if the process can serve a trivial request. No dependencies. GET /readyz -> 200 if this instance is ready: started, config loaded, pools initialised. GET /health/deps -> detailed dependency status, for dashboards and humans. Not for probes.

The liveness check answers only "is this process responsive?" It should almost never fail unless the process is genuinely stuck.

2. Keep dependency checks out of liveness entirely, and be very careful with them in readiness. If a dependency shared by every instance is down, removing every instance from traffic doesn't help users. Serving degraded responses or clear errors quickly is usually better than serving nothing.

3. Fail open when everything fails. Many load balancers can be configured so that if every target is unhealthy, traffic is sent to all of them anyway. That turns "every health check failed" into "try anyway", which is often the right call when the cause is shared.

4. Make health checks cheap and bounded. Short internal timeouts, no heavy queries, and results cached for a few seconds so a burst of probes doesn't become a burst of database queries.

5. Limit restart storms. Use startup delays and back-off for restarts, and cap connection pool sizes so a wave of restarts can't overwhelm the database.

6. Move reporting queries to a replica, with a statement timeout on the primary as a backstop.

ALTER ROLE api_user SET statement_timeout = '5s';

7. Monitor from outside, as well as inside. Internal health checks tell the orchestrator what to do. An external check, from the user's side of the internet, tells you whether customers can actually use the service. They answer different questions; you need both.

Lessons

A health check is an instruction, not just a measurement. Whatever consumes it will act on the answer. Design it around the action it triggers.
Correlated failures defeat redundancy. Ten instances that all check the same database will all fail the same check at the same moment.
Automatic remediation needs a brake. Restarting and removing capacity are safe when one instance is broken, and dangerous when the cause is shared.

Checking your service the way a customer would, from outside, is exactly what upordown.online does. It's the view that stays honest when internal checks get confused.