Health Check Depth

A shallow health check answers one question: can this process serve a request at all. A deep check adds the dependencies - the database, the cache, a downstream service - on the reasonable-sounding theory that an instance which cannot reach its database is not really healthy.

The theory ignores a property of the signal. A dependency is shared, so a deep check is a correlated signal: when the dependency degrades, every instance fails the check within one interval. The check stops describing an instance and starts describing the dependency, on a channel designed to remove instances.

Why it matters

If a load balancer honoured that signal, it would remove every target and the outage would be total, including for requests that never touch the failing dependency. A partial dependency problem would become a complete outage, caused by the mechanism meant to protect against it.

This is why load balancers deliberately do not honour it. AWS documents the behaviour in its Builders' Library article on implementing health checks: Network Load Balancer, Application Load Balancer and Route 53 all fail open when no targets are healthy, and the Network Load Balancer also fails out of an availability zone when every target in that zone is unhealthy. Fail-open makes the deep check harmless in the common case and useless in it too, which is the real argument against putting it there.

Implementation patterns

• Shallow at the load balancer. Local resources only: the process is up, the port is accepting, the thread pool is not wedged. Fast interval, fast reaction.
• Deep checks reported, not enforced. Expose dependency status on a separate endpoint that dashboards and a centralised health system consume, where the reaction can be slow, global and reasoned.
• Degrade rather than withdraw. If the cache is unreachable, serve from the origin more slowly; if a non-essential dependency is down, serve without that feature. An instance that can do 80% of its job belongs in the fleet.
• Never make a liveness probe deep. An orchestrator that restarts on a deep probe will restart the entire fleet in a loop during a dependency blip, adding cold caches and a reconnect storm to the original fault.
• Separate readiness from liveness: readiness may consult a dependency needed at startup; liveness must not.
• Exercise the check during a dependency failure in a game day, because nobody predicts this behaviour correctly on a whiteboard.

Industry example

Beyond the AWS guidance, the same article notes that Amazon teams tend to keep fast-acting load-balancer health checks local and handle deeper dependency signals through centralised systems that react more carefully. The reasoning generalises: a fast automated reaction to a correlated signal is dangerous, and the speed that makes health checks valuable for independent faults is exactly what makes them harmful for shared ones.

Failure scenarios

• Fleet-wide unhealthy during a dependency blip, surviving only because the load balancer fails open.
• A deep liveness probe restarting every instance repeatedly, converting a two-minute dependency problem into a twenty-minute cold start.
• Autoscaling reacting to unhealthy instances by replacing them, so a dependency fault becomes a churn of new instances that also cannot reach the dependency.
• A check with a timeout longer than its interval, so checks queue and the instance is marked unhealthy under load rather than under fault.
• A check that performs a real query, adding load to a database that is failing because of load.
• Zone evacuation triggered by a regional dependency, moving traffic to zones with the same problem.

Trade-offs

Shallow checks miss the instance that is genuinely broken in a way only its dependencies reveal: a stale local configuration, a corrupted connection pool, a credential that expired on this host alone. Those are real and they are rarer than the correlated case, and they are better caught by a centralised system that can see that one instance disagrees with its peers - which is a comparison the instance cannot make about itself.

The decision rule is independence. If every instance would answer identically at the same moment, the check does not belong on a per-instance removal channel.

When not to use it

Do not make the load balancer's check deep, essentially ever. There is one narrow exception: a check on a per-instance resource that happens to look like a dependency, such as a local disk or a sidecar that is genuinely per-pod, where the fault really can be independent. Everything shared belongs in a slower system that is allowed to conclude that the right action is to do nothing.

Interview question

Q: A colleague proposes making the load balancer health check verify database connectivity, arguing that an instance which cannot reach the database should not receive traffic. Give them the counter-argument, then tell them where their idea does belong.

What a strong answer covers: that the check becomes a correlated signal and fires fleet-wide within one interval · that load balancers fail open for exactly this reason, so the check has no useful effect · the liveness variant, which is worse because restarts are not idempotent against cold caches · degrading instead of withdrawing · putting the deep signal on a reporting endpoint consumed by a centralised system · and the independence test as the general rule.

Quick check

Quiz: Why do load balancers fail open when all targets are unhealthy? Because the alternative is a total outage from a correlated signal, which is usually worse than sending traffic to instances that may still be able to serve some of it.

Flashcard: What decides whether a check belongs at the load balancer? Whether the fault it detects can be independent per instance. Shared dependencies fail every instance at once.