Chaos Engineering Platform  ·  View 16 of 21  ·  Operations

Deployment Architecture

One control region, one warm replica, six target clusters — and no cross-region hop in the abort path.

Editable source SVG draw.io All views
europe-west1 — control region Control plane (GKE Autopilot, 3 zones) Platform API Cloud Run Run controller GKE Evaluator GKE Sequencer Workflows Safety plane — separate deployment, fewer dependencies Lease issuer Cloud Run, own SA Kill switch own endpoint + IAM Guardrail state Spanner, regional Evidence Evidence warehouse BigQuery Bundle objects Cloud Storage us-east4 — control replica Warm standby Platform API RTO 30 min Lease issuer RTO 60 s Guardrail state Spanner replica Target clusters — 6 regional GKE clusters across 3 regions chaos-system namespace Injection agent DaemonSet, ≤1% CPU Lease watchdog Out-of-band reverter break-glass SA Workload namespaces 900 services 40,000 pods lease, 5 s renew halt ≤ 30 s fault applied Spanner replication renew or not evidence Deployment Architecture — Google Cloud, Three Regions Application we own Security / platform Data store synchronous Guardrail and lease state is regionally partitioned: no cross-region hop sits in the abort path. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Guardrail and lease state is regionally partitioned. Nothing in the abort path crosses a region, because the abort budget is 10 seconds and a cross-region round trip spends too much of it.
  • The safety plane is deployed separately in both regions with its own service account, and its RTO is 60 seconds against the control plane's 30 minutes.
  • The agent runs as a DaemonSet in a dedicated namespace, so its resource envelope is visible to the node's own autoscaling rather than hidden inside a workload.

Assumptions

  • europe-west1 control, us-east4 warm replica, six regional GKE clusters across three regions.
  • Control plane RTO 30 min, abort path RTO 60 s, reporting plane RTO 4 h.
  • Loss of one region leaves no in-flight fault persisting anywhere in that region.

Risks

  • A warm replica that is never exercised is a hope. The self-chaos suite on view 17 has to include failing over the lease issuer.
◀  Scheduling and the Promotion Ladder All views Release Pipeline  ▶
Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records16-deployment-architecture