Health Checks

Definition

Three distinct questions, which must be three distinct endpoints:

• Liveness — is this process broken beyond recovery and in need of a restart?
• Readiness — should this instance receive traffic right now?
• Startup — has initialisation finished, so the other probes can begin?

Conflating them is the source of most health-check incidents.

The rule that prevents the classic amplification

Liveness must not test dependencies.

If a liveness probe calls the database, a database slowdown makes every instance report unhealthy. The platform restarts them all. Capacity collapses. The restarted instances cannot start because the database is still slow. A degraded dependency has become a total outage, executed by your own orchestrator on your behalf.

Liveness tests the process: is the event loop responsive, is the thread pool deadlocked. That is all.

Readiness may test dependencies, because failing readiness removes the instance from load balancing without killing it — and that is recoverable. Even so it must be used carefully: if all instances fail readiness simultaneously, you have taken the service down. Some platforms handle this with a "if all are unhealthy, route to all anyway" fallback, which is worth knowing exists.

The other rules

• Cheap. Probes run every few seconds on every instance. A health check that queries the database generates significant load and can itself cause the problem it reports.
• Deregister before shutdown. Fail readiness first, keep serving for at least the longest plausible caller cache (load balancer interval plus DNS TTL plus margin), then stop accepting and drain. Skipping the wait is why deployments produce a burst of errors.
• Distinguish "not ready yet" from "broken". A cold cache or a warming connection pool is a readiness state, not a liveness failure.
• Do not include a deep dependency check in the load balancer's probe unless you have thought through what happens when that dependency is down.

Failure scenarios

• Liveness testing dependencies — the amplification above.
• Health check passing while the service is useless. A shallow check on a process whose thread pool is exhausted still returns 200. This is grey failure, and binary probes cannot see it — outlier detection or latency-aware balancing is required.
• Timeout longer than the probe interval, so probes queue up.
• A health endpoint that is expensive, adding load proportional to fleet size.
• No startup probe, so a slow-starting instance is killed repeatedly in a loop.

Interview question

"Why should a liveness probe not check the database, and what should it check instead?"