Chaos Engineering Platform  ·  View 07 of 21  ·  Structure

Container View — Platform Components

What is deployed, in which project or cluster, and which component holds the only privilege that matters.

Editable source SVG draw.io All views
Control project (per region) Definition & governance Platform API Cloud Run Validator Cloud Run Approval workflow Workflows Notifier Pub/Sub fan-out Safety plane Radius calculator Cloud Run Reservation store Spanner Lease issuer signs, 15 s TTL Kill switch own endpoint Orchestration Run controller GKE Scheduler Cloud Scheduler Game-day sequencer Workflows Verification & evidence Signal collector Monitoring API Evaluator 10 s windows Verdict engine Evidence warehouse BigQuery Run store Spanner Target cluster (× 6) Chaos namespace Injection agent DaemonSet Lease watchdog Out-of-band reverter separate identity Workload namespaces Target workloads 900 services Mesh sidecars fault filters Service catalogue SLO store Incident platform Cloud Monitoring resolve radius reserve signed lease fault filter on expiry signals stop renewal evidence Container View — Platform Components Application we own Security / platform Interface / broker Data store External / third party synchronous Only the agent has any privilege inside a target namespace, and it holds no write path to the control plane's state. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • The safety plane is a separate deployment with its own service account and its own datastore, so a control-plane rollout cannot take the abort path with it.
  • The injection agent is the only component with privilege inside a target namespace, and it holds no write path back into control-plane state.
  • The out-of-band reverter runs in the same namespace but under a different identity, so using it is auditable and abnormal by construction.

Google Cloud realisation

  • Cloud Run for the stateless control services, GKE for the run controller and evaluator, Workflows for approvals and game-day sequencing.
  • Spanner for guardrail state — leases, reservations, budgets — because those reads must be strongly consistent and regional.
  • BigQuery plus Cloud Storage for evidence; Pub/Sub for run events.

Assumptions

  • Agent footprint ≤ 1% of a node's CPU and ≤ 64 MB resident when idle. A resilience platform that destabilises nodes has failed.
◀  Layered Architecture All views Integration Surface  ▶
Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records07-platform-components