Fleet Readiness Floor
also called Minimum Healthy Target Threshold, Readiness Fail-Open Guard
A limit on how much of a service the platform may remove from rotation at once, so that a faulty health check or a shared dependency failure cannot take every instance out together.
A cache shared by a service's 200 pods degrades for 90 seconds. Each pod's readiness probe calls the cache, so within a few probe intervals every pod reports not-ready, the load balancer's pool empties, and the whole service returns 503s, including the requests that never touched the cache. The probe turned a partial fault into a total outage because it asked each pod to report a problem that removing the pod cannot fix. A readiness floor is the guard that limits the damage when the probe is wrong or the cause is shared.
The idea has two halves. At the platform level, a rule says never remove more than a set share of targets, or always keep at least N, regardless of what the checks say. At the design level, readiness is restricted to facts that are local to the pod, so a shared dependency cannot fail all pods in one stroke.
Why it matters
Health checks are a control loop with a destructive output: they eject capacity. When the cause is per-instance (a wedged thread pool, a bad host), ejection is right. When the cause is shared (a database, a certificate, a bad configuration push, or the check itself), every instance fails together and the loop removes capacity from a system that is mostly able to serve. The probability that 100% of a large fleet is independently broken is nearly zero, so a mass failure is evidence that the signal is broken, not the fleet.
Implementation patterns
- Local-only readiness. Include warm-up complete, not draining and local pool health. Exclude shared databases, caches and downstream APIs. Report them as metrics and let the application degrade (cached data, partial responses, errors only for requests that need them).
- A floor in the balancer. AWS documents that an Application Load Balancer fails open when all targets in a target group are unhealthy, and offers a minimum healthy target count or percentage setting that changes when the group is treated as failed. Other balancers and service meshes have an equivalent, such as a panic threshold in Envoy, which stops honouring health status when the healthy share falls below a configured percentage.
- A budget in the orchestrator. Pod disruption budgets limit voluntary evictions; a rollout's max unavailable setting limits how much an upgrade may take out at once.
- Alert on the ready share, for example below 70% of expected, rather than on each probe. It fires on the mass-failure signature.
- Fail the check safe. If the check endpoint itself cannot answer (timeout) and nothing else is wrong, prefer treating the instance as ready for a short time rather than ejecting it.
Industry example
An archetypal marketplace runs a shared session cache behind 400 pods, and an incident is caused by a probe that pings that cache. When the cache's latency exceeds the probe timeout, all pods leave rotation within 15 seconds. The documented behaviour of fail-open balancers is exactly the safety net designed for this: traffic goes to every target again, and requests succeed that would have failed. The durable fix was moving the cache check out of readiness and adding the ready-share alert. This example is an archetype; it describes the mechanism, not a specific company's incident.
Failure scenarios
- Floor too low or absent: the fleet empties and the fleet-wide 503 is the incident.
- Floor too high: genuinely broken instances stay in rotation and users see a stable fraction of errors, for example 30% of requests failing.
- Fail-open without alerting: the balancer silently routes to everything, which hides a persistent mass failure for days.
- Probes with the same fate as traffic: the readiness endpoint runs on the application's worker pool, so a saturated pool fails readiness and deepens overload.
- Cascading restarts: liveness checks the same dependency, so the orchestrator restarts everything on top of ejecting it.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Local-only readiness | No mass ejection from shared faults | A dependency fault is found by metrics, not routing |
| Higher floor | Resilience to bad checks | Some broken instances keep serving errors |
| Fail-open | Service continues in total check failure | Possible traffic to genuinely broken instances |
| Deep checks | Early detection of dependency issues | Probe load multiplies with fleet size |
When not to use it
A fleet of three instances with independent failures gains nothing from a floor; eject the broken one. A dependency that is exclusive to each pod, such as a local sidecar proxy, belongs in readiness because its failure is genuinely per-pod. A service that must refuse traffic when a dependency is down for correctness reasons (for example, a payment path that must not run without its ledger) should fail closed on purpose and say so.
Interview question
Q: Every pod flipped to not-ready at once and the service went down. What happened, and what do you change?
What a strong answer covers: a shared cause or a faulty check; readiness as a local-only question; liveness separated from both; load balancer fail-open and minimum-healthy settings; a ready-share alert; shedding or degradation inside the application for dependency faults; and why the probability argument says mass failure indicates a signal problem.
Quick check
Quiz: Why should a shared cache not appear in a pod's readiness probe? If the cache degrades, every pod fails together and the whole service is ejected.
Flashcard: What does a readiness floor limit? — The share of instances that may leave rotation at once.