Chaos Engineering Platform  ·  View 02 of 21  ·  Context and scope

High-Level Architecture

Six stages from a declared experiment to a recorded verdict, and where permission enters.

Editable source SVG draw.io All views
Declare Definition repo versioned, reviewed Validator & dry run Permit Approval & notify Guardrail engine radius, budget, windows Lease issuer 15 s TTL Sequence Run controller state machine Scheduler Inject Injection agent DaemonSet per node Fault adapters 6 classes Observe Signal collector 10 s windows Steady-state evaluator Judge Verdict engine 5 outcomes Evidence store BigQuery Chaos Engineering Platform — High-Level Architecture Data store Application we own Security / platform Interface / broker The evaluator feeds the lease issuer, not the agent: a breach stops renewal and the fault expires. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Permission is its own stage, between declaration and sequencing. A run that has not been permitted cannot be sequenced, and sequencing cannot grant itself permission.
  • The evaluator's output goes to the lease issuer, not to the agent. A breach withholds a renewal; it does not send a stop command that might not arrive.
  • The verdict engine emits one of five outcomes, of which two are failures of the platform rather than of the target.

Assumptions

  • Lease TTL 15 s, renewed every 5 s; steady state evaluated on 10 s rolling windows with signal freshness p95 ≤ 15 s.
  • 200 concurrent runs fleet-wide, 2,000 runs a week at steady state.

Risks

  • Renewal traffic becomes a load-bearing path of its own: 40,000 renewals per second at full fleet saturation.
  • A long experiment is a chain of renewals, so a control-plane restart ends it rather than pausing it.
◀  System Context All views Actors and Their Core Journeys  ▶
Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records02-high-level-architecture