A platform's gateway processes pass their health checks while holding connections they can no longer serve. What is wrong with the health check, and what should it assert?

What is wrong

The health check asserts liveness — the process is running — when the question is whether it is doing useful work.

A gateway process holding a hundred thousand connections can be: unable to reach the message backbone, so no messages flow; stuck in a long garbage-collection pause; out of file descriptors so new connections fail; or holding half-open connections whose clients disappeared minutes ago. In every case the HTTP health endpoint returns 200 promptly, because answering it requires none of the things that are broken.

What a health check should assert

Progress, not existence. Concretely:

• Recent successful work. Messages delivered in the last interval, requests completed, jobs processed. A process that has delivered nothing for thirty seconds while holding connections is unhealthy regardless of what its endpoint says.
• Critical dependency reachability, checked cheaply and cached — the message backbone, the session store. Not every dependency: a health check that fails when an optional dependency is down turns a degraded feature into a removed instance, which is a far worse outcome.
• Resource headroom — file descriptors, memory, connection slots. A process at its connection ceiling should stop accepting new ones while continuing to serve existing ones, which is a readiness signal rather than a liveness one.

The distinction that matters

Liveness — should this process be restarted? Should be conservative, because restarting a process holding a hundred thousand connections causes a reconnect storm. A liveness check that is too aggressive turns a transient problem into a self-inflicted outage.

Readiness — should this process receive new work? Should be sensitive. A process that is degraded should stop accepting new connections immediately while continuing to serve what it has.

Conflating them is the classic error: a sensitive check wired to liveness causes restarts under load, and restarts under load cause more load.

The stateful-tier complication

For a tier holding long-lived connections, health checking interacts badly with load balancing in a way worth stating explicitly:

• Removing an unhealthy instance does not move its existing connections — they are pinned. Removal only stops new ones.
• Marking an instance unhealthy during a load spike removes capacity from an already-stretched fleet and displaces its connections onto the survivors, accelerating collapse.

So during overload, the correct behaviour is usually shed load rather than fail health checks — refuse new connections while continuing to serve existing ones. A health check that fails under load is a health check that amplifies the incident.