Health Check & Service Discovery  ·  View 13 of 21  ·  Runtime

Critical Flow — A Replica Fails

Fourteen messages, and the two of them that are the reason a user never sees this.

Editable source SVG draw.io All views
Calling service Envoy sidecar Failing replica Signal ingest Evaluator View assembler xDS stream tier 1. call service B 2. request 3. 503 / timeout 4. local ejection, weight 0 5. retried elsewhere, 200 6. outcome report 7. passive sample 8. score decays past threshold 9. fraction guard: 59% still eligible 10. ineligible + cause 11. delta, version n+1 12. push delta 13. monotonic apply, cache write 14. endpoint gone from the set Critical Flow — A Replica Fails, And Traffic Stops In Five Seconds Steps 4 and 5 are why the user never sees this: the client ejects before the control plane has an opinion. v 1.0 · owner Reliability Architecture

What this flow proves

  • The client ejects and retries before the control plane has an opinion. Local ejection is the first line, not a backstop.
  • The fraction guard is consulted on every removal, including this one — 59% still eligible is why the removal is allowed to proceed.
  • The delta carries the new version, and the client applies it monotonically and writes it to disk before acknowledging it.

Assumptions

  • 5 s to 99% of callers, 15 s to 99.9%, measured from the first failed request rather than from the decision.
  • Local ejection caps the fraction of a set any one client may remove, so one bad caller cannot drain a healthy service.

Risks

  • Between steps 3 and 12 other callers are still sending to the failing replica. The 5-second budget is the real measure of that exposure.
◀  Registry Data Model All views Registration To Eligible  ▶
Solution Architecture v1.0 · Amazon Web Services with an open-source Envoy/xDS data plane · Reliability Architecture · 2026-10 · 21 views · 16 architecture decision records13-withdrawal-propagation