Health Check & Service Discovery  ·  View 16 of 21  ·  Operations

Deployment Architecture

Three regions of the same shape, none of them depending on a global component to route traffic.

Editable source SVG draw.io All views
AWS us-east-1 (home region) AZ a Control plane pods EKS, 6 replicas Prober set a DaemonSet Workloads + sidecars ~20,000 AZ b Control plane pods EKS, 6 replicas Prober set b Workloads + sidecars AZ c Control plane pods EKS, 6 replicas Prober set c Workloads + sidecars Regional state Registry Aurora, 3-AZ Observed state DynamoDB, 7-day TTL Transition log Kinesis + S3 AWS eu-west-1 and ap-south-1 (same shape, independent) Regional control plane Control plane own authority Registry own writes Cross-region view Async replica of peers read-only aggregate Out-of-band failover signal ARC readiness, no registry dependency Route 53 + Cloud Map global DNS surface Service catalogue async replication region health zone sync Deployment — Regional Authority, No Global Request-Path Dependency Application we own Security / platform Data store External / third party event / async synchronous batch A region keeps routing with both peers and every global component unreachable. Cross-region resolution is the only thing that degrades. v 1.0 · owner Reliability Architecture

Decisions

  • Each region's control plane has its own authority and its own registry writes; cross-region resolution is a read-only async aggregate (ADR-14).
  • The failover signal that says a region is unhealthy is out of band and has no dependency on the registry it describes.
  • Probers are per-AZ, so a prober-network failure is scoped to one failure domain rather than to a region.

Assumptions

  • us-east-1, eu-west-1 and ap-south-1 active; ~20,000 workloads per AZ in the home region.
  • Registry on a 3-AZ relational store, observed state on a wide-column store with 7-day TTL, transitions to a log plus object storage.

Risks

  • Regions will sometimes disagree about which region is healthy. Reconciliation after a partition is explicit and logged, never silent.