# Chaos Engineering Platform

**Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records**

The internal platform that goes looking for outages on purpose. Everyone has been on the receiving end of the failure it exists to find: the app that streams perfectly for six weeks and then, on a Friday night with a record audience, shows a spinner — because of a retry storm behind a slow recommendation call, a cache tier that was supposed to be optional, or a failover that worked in the runbook and not in the region. This platform lets a service owner state a falsifiable hypothesis about how their service behaves when a dependency degrades, inject exactly that degradation into a bounded slice of live traffic, watch the service's own steady-state signals, and have the fault removed automatically the moment real customers start to hurt. The operating context is a consumer subscription streaming service — an assumed 140 million subscribers, 900 microservices, 40,000 pods across six regional GKE clusters in three regions — built from Cloud Run and GKE for the control plane, Spanner for guardrail and lease state, Cloud Monitoring for the high-resolution steady-state path, BigQuery for evidence and coverage, Workflows for approvals and game-day sequencing, and open-source fault injectors in the data plane.

The product is not the ability to create a fault — any script can add packet loss. It is **the guarantee that the fault goes away**: when the service starts hurting, when the operator says stop, when the control plane crashes, when the network partitions, and when the telemetry that was supposed to notice goes dark.

The design rests on one rule: **injection authority is a lease held in the data plane, not a command issued from the control plane.** Nothing in the system tells a target to stop being broken; the permission to be broken simply expires.

The decisions that carry the design:

- **Authority is leased, never granted.** A signed, target-bound, 15-second lease, renewed every 5 seconds, is the only reason a fault continues to exist. A partition, a crashed controller, a lost region and a blinded telemetry pipeline all produce the same safe outcome without any of them being handled as a special case (ADR-01).
- **The brake has fewer dependencies than the accelerator.** Lease issuer and kill switch are a separate deployment with their own identity and store, a 60-second RTO against the control plane's 30 minutes, and their own verification schedule — because a bad rollout must not remove both at once (ADR-02).
- **Blindness aborts, and the verdict is INCONCLUSIVE.** If the platform cannot evaluate the hypothesis it cannot hold the fault. A run nobody could observe is never recorded as a pass, which is what stops a degraded metrics pipeline becoming false confidence (ADR-03).
- **Blast radius is computed, not declared.** Three measures — replica share, measured request-volume share, and the downstream closure from the dependency graph — plus a veto on any run reaching a shared single point of failure. "Five per cent of replicas" is a lie when all of them talk to the one cache tier (ADR-05).
- **Chaos spends a real budget, and the owner sets it.** Ten per cent of the service's own monthly SLO error budget, debited at lease issue and settled at abort. Billing deliberate risk in the same currency as accidental risk is what makes the trade-off arguable (ADR-07).
- **The incident feed can veto without a human.** An open incident of configured severity suspends injection and aborts intersecting runs. Stopping is automatic and free; resuming needs a person — the expensive mistake is only in one direction (ADR-08).
- **No adapter ships without a proven revert.** Inject, verify, revert, report_state, plus a non-production proving window. This is the contract that forces an irreversible fault to be refused explicitly rather than accommodated quietly under deadline (ADR-09).
- **Reversion is verified by looking, not by a return value.** A stuck traffic-control rule survives the process that created it. An unconfirmed revert invokes the out-of-band reverter, pages on-call and suspends that fault class fleet-wide (ADR-11).
- **Real traffic on a bounded slice.** Shadow and synthetic load protect customers and systematically miss retry storms, cache stampedes and queue backpressure — which is the class of failure worth finding (ADR-12).
- **A pass needs a positive observation.** Where a mitigation can be seen working — breaker opened, fallback served, retry budget held — the hypothesis must assert it. Nothing going wrong is not the same as something working (ADR-14).
- **The agent verifies its own permission.** Signature and scope are checked in the data plane, so a compromised control plane cannot widen a blast radius (ADR-15).
- **Cost scales with usage, not with estate size.** High-resolution collection only inside the baseline and injection windows, early termination once the hypothesis is decided, and every cent attributed to the team that scheduled the run (ADR-16).

The architecture one-pager (including why the design should still hold up in ten years, and the five risks that would change it) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~12 min) and [docs/decision-record.md](docs/decision-record.md) (~62 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 21 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 16 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..d.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only, no network) |
| `ask.md` | The requirement |

## The twenty-one views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Who runs experiments, which systems are read-only inputs, and why the customer is deliberately absent |
| 02 | High-Level Architecture | Six stages from a declared experiment to a recorded verdict, with permission as its own stage |
| 03 | Actors and Journeys | Six actors including the on-call responder, whose two journeys are not feature requests |
| 04 | Journey — Prove a Dependency Is Optional | The trough is the injection moment, and the whole architecture exists to make that fear unjustified |
| 05 | Journey — Run a Game Day | Time to detect and time to mitigate, which no automated run produces |
| 06 | Layered Architecture | Nine layers, with governance ("may you") and safety ("how far") deliberately separated |
| 07 | Container View | What is deployed where, and the one component with privilege inside a target namespace |
| 08 | Integration Surface | Five read-only inputs, five things it tells or acts on, and nothing it writes back |
| 09 | Data Flow | Six states a run's data passes through, and the two places it is thrown away on purpose |
| 10 | Data Ownership Zones | Four zones graded by who wrote the data and what losing it costs — the hottest store has the cheapest failure |
| 11 | Data Model | Twelve entities; the spine is definition, version, run, lease, applied fault |
| 12 | Critical Flow — One Experiment | Twenty-one messages, with the evaluator never speaking to the agent |
| 13 | Abort and Degradation Paths | Four triggers, one mechanism, and no path where uncertainty keeps a fault alive |
| 14 | Guardrail Evaluation | Seven independent gates across four phases, none waivable by the requester |
| 15 | Scheduling and Promotion | How a definition earns the right to run unattended, and what it never earns |
| 16 | Deployment Architecture | One control region, one warm replica, six target clusters, no cross-region hop in the abort path |
| 17 | Release Pipeline | Gated on proving reversion rather than on passing tests — that gate is the product |
| 18 | Observability | Five signal types across five stages, and the two numbers that are load-bearing |
| 19 | Finding-to-Remediation Loop | Eight steps from an untested dependency to a weakness that stays fixed |
| 20 | Security Trust Zones | Six zones: the control zone cannot inject and the target zone cannot self-authorise |
| 21 | Identity Flow | Eighteen messages before a fault can exist, ending with the agent verifying its own permission |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The script assembles
`specs/views.json`, `specs/manifest.json` and `specs/adr.json` from the authoring parts, then
runs generate → validate → route check → SVG → draw.io → HTML → ADR injection → link check.

Current state of the gates: **0 geometry errors, 0 geometry warnings, 0 routing errors, 8
routing clutter warnings, 408 icons embedded, all relative links resolve.** The eight clutter
warnings are adjacent edge labels on the context, container, hub and abort-path views, where
cutting a label would cost more meaning than the crowding costs legibility.