Liveness vs Readiness

Liveness asks: is this process broken beyond recovery, such that restarting it is the right action?

Readiness asks: should this process receive new work right now?

They are routinely implemented as one endpoint, and the consequences are severe in opposite directions.

Why they need opposite sensitivities

Liveness must be conservative. Restarting is destructive: it drops in-flight requests, discards warm caches, empties connection pools, and — for a process holding long-lived connections — triggers a reconnect storm. A liveness check that is too sensitive converts a transient problem into a self-inflicted outage, and does so precisely under load, when transient problems are most common.

Readiness must be sensitive. A process that is degraded, saturated or temporarily unable to reach a critical dependency should stop receiving new work immediately, while continuing to serve what it already has. There is no cost to being wrong in the cautious direction.

Conflating them produces the classic failure: a sensitive check wired to liveness causes restarts under load, and restarts under load cause more load.

What each should actually check

Liveness: the process is fundamentally stuck — a deadlock, an unrecoverable state — verified by something cheap and unambiguous. When in doubt, do not restart.

Readiness: progress, not existence. Recent successful work in the last interval; reachability of critical dependencies only; resource headroom in file descriptors, memory and connection slots.

The word critical is load-bearing. A readiness check that fails when an optional dependency is unavailable removes a healthy instance from rotation, turning a degraded feature into reduced capacity — which is a much worse outcome than serving the degraded feature.

Industry example

A gateway tier holding millions of long-lived connections shows why this matters. Such a process can be unable to reach the message backbone, stuck in a long garbage-collection pause, or out of file descriptors, and still answer an HTTP health endpoint with 200 promptly — because answering it requires none of the broken things. The check asserts liveness when the question is whether useful work is happening.

The stateful complication makes it sharper. Removing an unhealthy instance from a load balancer does not move its existing connections, which are pinned; removal only stops new ones arriving. And marking instances unhealthy during a load spike removes capacity from an already-stretched fleet and displaces those connections onto the survivors, accelerating collapse.

So during overload the correct behaviour is usually to shed load rather than fail health checks — refuse new connections while continuing to serve existing ones. A health check that fails under load is a health check that amplifies the incident.

Failure scenarios

• One endpoint serving both, so any sensitivity setting is wrong for one of them.
• Liveness checking dependencies, so a downstream blip restarts the whole fleet.
• Readiness checking optional dependencies, converting feature degradation into capacity loss.
• Health checks that always return 200 because they only prove the HTTP server is listening.
• Checks expensive enough to add load under the conditions where they matter most.

Trade-offs

Two checks means two code paths, two configurations and more to get wrong. A single simple check is appealing for a small stateless service where restarts are cheap and there is little to distinguish.

The distinction earns its cost as soon as restarts are expensive — stateful tiers, long-lived connections, warm caches, slow startup — which describes most systems at scale.

Interview question

"Your service's health check passes while it is unable to process any messages. Walk me through what the check should assert instead — and tell me what you would do differently during a load spike."