practice

Health Check Semantics

The distinction between liveness, readiness and startup checks, and the failure each is intended to address.

Conflating them causes outages, and the confusion is extremely common.

Liveness — is the process broken beyond recovery? Failing it restarts the container. It must therefore check only the process itself. Including a database check in liveness is a classic self-inflicted outage: the database has a blip, every instance fails liveness, every instance restarts simultaneously, and a brief degradation becomes a full outage with a cold start.

Readiness — can this instance serve traffic right now? Failing it removes the instance from the load balancer without restarting. This is where dependency checks belong, because the correct response to an unavailable dependency is to stop receiving traffic, not to restart.

Startup — has initialisation finished? It suspends liveness checking during a slow start, preventing a restart loop for an application that legitimately takes two minutes to warm up.

Two refinements: a deep check that verifies dependencies should be a separate endpoint used by monitoring, not by the orchestrator. And readiness should fail during graceful shutdown before the process stops accepting connections, so in-flight requests complete and no new ones arrive.