Health Check & Service Discovery  ·  View 06 of 21  ·  People and journeys

Journey — Evacuate a Zone, and Bring It Back

A 16× churn burst, a decision made under pressure, and a way back that has to be one reversible step.

Editable source SVG draw.io All views
SRE on call declared incident Goal — Move traffic out of one availability zone in a minute, and back only when I say so Trigger — Elevated latency across every service in one zone; the cause is below us Done when — Zone carries no traffic, nothing else changed, and the way back is one reversible step 1 · Suspect cross-service 2 · Declare ◆ moment of truth 3 · Shift 4 · Hold 5 · Return ◆ moment of truth What they do See zone-wide latency Declare evacuation Watch tiers re-order Leave it drained Complete shift-back What the plane does Failure domain tagged Topology preference shifts 40k changes / minute Instances still registered Rate-capped return How it feels Calm Tense Exposed Where it hurts Is it the zone or us? Wrong zone named Burst breaks the budget Remaining zones overload Herd hits the cold zone What answers it Failure domains first-class One-step reversible Withdrawal budget held Fraction guard per zone Explicit, capped return Journey — Evacuate a Zone, and Bring It Back v 1.0 · owner Reliability Architecture

What answers the trough

  • Evacuation shifts topology preference rather than removing instances, so the instances stay registered and the return is one step.
  • The withdrawal propagation budget is held through the burst by shedding the addition path instead.
  • Shift-back is rate-capped and, for a manual failover, explicitly completed rather than drifting back on its own.

Assumptions

  • 40,000 instance state changes per minute during a regional evacuation, against 2,500 at steady state.
  • Minimum healthy fraction is evaluated per service per zone, so the remaining zones' guard still protects them.

Open

  • Who is allowed to declare a region unhealthy, and whether that signal may depend on the registry at all — see Question 5 and ADR-14.
◀  Journey — Chase Errors From One Replica All views Layered Architecture  ▶
Solution Architecture v1.0 · Amazon Web Services with an open-source Envoy/xDS data plane · Reliability Architecture · 2026-10 · 21 views · 16 architecture decision records06-journey-evacuate-a-zone