A load balancer health check is changed from "the process answers" to "the process can reach its database and its cache". The dependency has a brief regional problem. What happens to the fleet, and why do load balancers deliberately fail open?

Second by second

The dependency degrades. Every instance evaluates the same dependency, so every instance fails the check within one check interval. This is the defining property of a deep health check: it is a correlated signal, and it converts a partial dependency problem into a fleet-wide unhealthy state in seconds.

What happens next depends on the load balancer. If it removed every unhealthy target, it would have nothing to send traffic to and the outage would be total, including for the requests that do not touch the failing dependency. So load balancers deliberately do not do that.

AWS documents this behaviour in its Builders' Library article on implementing health checks: Network Load Balancer, Application Load Balancer and Route 53 all fail open when no targets are healthy, and the Network Load Balancer also fails out of an availability zone when every target in that zone reports unhealthy. Failing open is the safety valve that keeps a bad health check from being worse than the fault it detected.

The same article notes that Amazon teams tend to keep fast-acting load-balancer health checks local, and handle deeper dependency checks through centralised systems that can react more carefully - precisely because a fast automated reaction to a correlated signal is dangerous.

Why the other options fail

• Only the affected zone fails assumes the dependency problem is zonal and that the check is zone-aware. A regional cache or a shared database is reached from every zone, so the signal is not zonal even when the fault partly is. This is the answer that feels right and depends on an assumption the stem does not give you.
• Gradual removal in proportion to the fault describes what a shallow check would do under a partial fault. Deep checks are not proportional: they are shared, so they fire together. Proportionality is exactly the property being lost.
• Instances restart and recover confuses liveness with health checking. A restart does not repair a dependency, and if a liveness probe were also deep, the orchestrator would restart the entire fleet in a loop, turning a dependency blip into a self-inflicted cold start with empty caches and a thundering reconnect.

What stops it

1. Keep the load balancer's check shallow. It answers one question: can this instance serve traffic at all. Local resources only.
2. Put dependency awareness somewhere slower and smarter. A centralised health system, or a deployment gate, can notice that a dependency is failing everywhere and decide to do nothing, which is usually the right action.
3. Serve degraded rather than declaring unhealthy. If the cache is unreachable, serve from the origin more slowly; if a non-essential dependency is down, serve without that feature. An instance that can still do 80% of its job should not remove itself from the fleet.
4. Test the check against a dependency failure in a game day, because the behaviour of health checks under dependency loss is not something anyone reasons about correctly on a whiteboard.

When not to keep the check shallow

When the instance genuinely cannot serve anything useful without the dependency and there is a meaningful chance the fault is per-instance rather than shared - a stale local configuration, a corrupted local cache, a connection pool that has failed on this host alone. The test is whether the failure can be independent. If every instance would answer the same way at the same moment, the check belongs somewhere other than the load balancer.