Architecture One-Pager

Chaos Engineering Platform · Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records

Injection authority is a lease held in the data plane, not a command issued from the control plane.

Outages are almost never the failure that was rehearsed. They are the retry storm behind a slow recommendation call, the cache tier that was supposed to be optional, the failover that worked in the runbook and not in the region. A chaos engineering platform exists to find those failures on a Tuesday afternoon instead of on a Friday night — which means deliberately breaking production, in front of the people who own the code, without breaking it for customers. The hard part is not creating a fault. Any script can add packet loss. The hard part is guaranteeing that the fault goes away: when the service starts hurting, when the operator says stop, when the control plane crashes, when the network partitions, and when the telemetry that was supposed to notice goes dark.

The control plane decides that an experiment may run, computes its blast radius from the dependency graph, and issues a signed, target-bound, time-bound lease. An in-cluster agent is the only component that touches a target, and it holds the fault only while that lease is valid and renewed every five seconds. A dedicated high-resolution evaluator watches the target's own SLIs on ten-second windows and, on a breach, simply stops confirming steady state — the renewal is withheld and the fault expires. Nothing in the system tells a target to stop being broken; the permission to be broken runs out.

What it is, and what it is not

• A lease the data plane holds and lets expire — not a revert command the control plane must successfully deliver, which is exactly the delivery it cannot guarantee during the failures that matter most
• A blast radius computed from the dependency graph — not the replica percentage the author typed, which is a lie whenever the affected replicas all talk to one shared cache tier
• Evaluation against the target's own production SLIs — not a metric invented for the experiment, which can pass while the thing customers actually feel is failing
• INCONCLUSIVE as a first-class verdict — not treating an unobserved run as a pass, which converts missing telemetry into false confidence
• An abort path deployed separately, with fewer dependencies than the injection path — not one service that both starts and stops experiments, where a bad rollout removes the brake and the accelerator together
• Findings tracked to remediation as the unit of value — not runs executed as the unit of value, which produces experiments designed not to fail
• A promotion ladder from attended to scheduled to continuous — not cron from day one, which is how an unattended fault meets a peak-traffic window
• A consumer of the service catalogue, dependency graph and SLO store — not a second place where ownership, dependencies and SLIs are defined and then drift

The decisions that are the architecture

1. Authority is leased, not commanded (ADR-01) — A signed 15-second lease, renewed every 5 seconds, is the only thing that keeps a fault alive. Every failure of the platform ends in expiry.
2. The abort path is simpler than the injection path (ADR-02) — Lease issuer and kill switch are a separate deployment with their own identity, their own store and a 60-second RTO against the control plane's 30 minutes.
3. Blindness aborts (ADR-03) — If the platform cannot evaluate the hypothesis, it cannot hold the fault. The verdict is INCONCLUSIVE, never a pass.
4. Blast radius is computed (ADR-05) — Three measures — replica share, request-volume share, downstream closure — and a veto on any run reaching a shared single point of failure.
5. Chaos spends a real budget (ADR-07) — Ten per cent of the service's own monthly SLO error budget, debited per run, refused at exhaustion.
6. No adapter without a proven revert (ADR-09) — Inject, verify, revert, report_state. An adapter whose revert has not been exercised in the last 30 days is ineligible for production.
7. Real traffic, bounded slice (ADR-12) — Shadow and synthetic traffic systematically miss retry storms and cache stampedes, which is the class of failure worth finding.
8. A pass needs a positive observation (ADR-14) — Where a mitigation can be observed — breaker opened, fallback served, retry budget held — the hypothesis must assert it, not merely the absence of harm.
9. The agent verifies its own permission (ADR-15) — Signature and scope are checked in the data plane, so a compromised control plane cannot widen a blast radius.

Why this should still be right in ten years

A chaos platform is unusual in that its central guarantee is about what happens when everything else is broken. That guarantee is not a technology choice, which is why most of this design should outlive the services it tests and all of the Google Cloud products named in it.

• Fail-closed is not a feature, it is the shape. The lease is what makes every failure mode converge on the same safe outcome. No new fault class, cloud migration or scale change alters that, and nothing in the lease mechanism depends on Google Cloud.
• The untested set is the durable product. Coverage — which dependencies have never been exercised — stays useful long after any individual verdict expires. Teams change, services are rewritten, and the gap report keeps answering the same question.
• Adapters are where churn lives, and it is contained. Injection mechanisms will change: eBPF will displace traffic control, meshes will gain and lose features. The capability-declaring adapter contract means that churn never reaches the control plane.
• The hardest part is organisational and the design respects it. Chaos error budgets, ownership-based authorisation and findings tracked to closure exist because the real constraint is remediation capacity, not injection capacity. That will still be true.
• What would break it. A feature request for a fault that cannot be reverted — a destructive data operation, an irreversible schema change — would put the first hole in the fail-closed guarantee. The adapter contract exists so that request has to be refused explicitly rather than accommodated quietly.

Non-functional targets

Every number here is a stated assumption from the requirement document, chosen to be defensible and arguable rather than measured. The view column points at the page where the number is visible in the design rather than merely asserted.

Scope

In scope

• Declarative, versioned experiment definitions with a mandatory falsifiable hypothesis, abort conditions and radius limits.
• Six fault classes: network, dependency response, resource, lifecycle, clock and stateful dependency.
• Computed blast radius with reservation-based mutual exclusion, exclusion windows, chaos error budgets and a global kill switch.
• Lease-based injection with watchdog revert, verified reversion and an out-of-band reverter.
• Steady-state evaluation against the target's own SLIs, five verdict classes and sealed evidence bundles.
• Ad-hoc, scheduled, canary-gated, continuous and game-day execution modes with a promotion ladder between them.
• Findings tracked to remediation, and per-service resilience coverage reporting including the untested set.

Explicitly out of scope

• Load and performance testing. Injecting load and injecting failure need different guardrails and different verdicts, and sharing a platform between them weakens both.
• Security red-teaming. The adversary model is different and the authorisation model would have to be, too.
• Functional test automation and CI unit testing, which belong to the deployment pipeline.
• Owning the service catalogue, dependency graph, SLO definitions or on-call rotations — all read, none authored here.
• Incident management. The platform files findings and receives suspension signals; it does not run the incident.

What a four-week prototype should prove

Four claims in this architecture are cheap to test now and very expensive to discover wrong after thirty teams depend on the platform. Everything else can be built incrementally against a working answer to these.

1. The fail-closed guarantee end to end: kill the lease issuer mid-experiment and measure, from outside, how long the fault survives against the 15-second TTL.
2. The abort budget: from a deliberately induced SLI breach, measure breach-to-reverted at p99 against the 10-second target, including evaluation lag.
3. The dependency closure: compute the downstream reachable set for twenty real services and check, by hand, how often it is wrong and in which direction.
4. Reversion verification: run every adapter a thousand times and count how many reverts need the bounded retry, and how many need the out-of-band reverter.

• Partition the agent from the control plane mid-injection and confirm the fault clears within the TTL with no command delivered.
• Degrade the metrics pipeline during an injection window and confirm the run aborts and records INCONCLUSIVE rather than passing.
• Start two experiments whose computed radii intersect and confirm the second queues rather than running alongside the first.
• Open a real incident touching a downstream service mid-run and confirm injection suspends fleet-wide within 30 seconds.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 16 records across 5 areas, each with the alternatives that lost and what the choice costs.