Chaos Engineering Platform

Chaos Engineering Platform · Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records

Injection authority is a lease held in the data plane, not a command issued from the control plane.

Outages are almost never the failure that was rehearsed. They are the retry storm behind a slow recommendation call, the cache tier that was supposed to be optional, the failover that worked in the runbook and not in the region. A chaos engineering platform exists to find those failures on a Tuesday afternoon instead of on a Friday night — which means deliberately breaking production, in front of the people who own the code, without breaking it for customers. The hard part is not creating a fault. Any script can add packet loss. The hard part is guaranteeing that the fault goes away: when the service starts hurting, when the operator says stop, when the control plane crashes, when the network partitions, and when the telemetry that was supposed to notice goes dark.

The control plane decides that an experiment may run, computes its blast radius from the dependency graph, and issues a signed, target-bound, time-bound lease. An in-cluster agent is the only component that touches a target, and it holds the fault only while that lease is valid and renewed every five seconds. A dedicated high-resolution evaluator watches the target's own SLIs on ten-second windows and, on a breach, simply stops confirming steady state — the renewal is withheld and the fault expires. Nothing in the system tells a target to stop being broken; the permission to be broken runs out.

What it is, and what it is not

• A lease the data plane holds and lets expire — not a revert command the control plane must successfully deliver, which is exactly the delivery it cannot guarantee during the failures that matter most
• A blast radius computed from the dependency graph — not the replica percentage the author typed, which is a lie whenever the affected replicas all talk to one shared cache tier
• Evaluation against the target's own production SLIs — not a metric invented for the experiment, which can pass while the thing customers actually feel is failing
• INCONCLUSIVE as a first-class verdict — not treating an unobserved run as a pass, which converts missing telemetry into false confidence
• An abort path deployed separately, with fewer dependencies than the injection path — not one service that both starts and stops experiments, where a bad rollout removes the brake and the accelerator together
• Findings tracked to remediation as the unit of value — not runs executed as the unit of value, which produces experiments designed not to fail
• A promotion ladder from attended to scheduled to continuous — not cron from day one, which is how an unattended fault meets a peak-traffic window
• A consumer of the service catalogue, dependency graph and SLO store — not a second place where ownership, dependencies and SLIs are defined and then drift

The decisions that are the architecture

1. Authority is leased, not commanded (ADR-01) — A signed 15-second lease, renewed every 5 seconds, is the only thing that keeps a fault alive. Every failure of the platform ends in expiry.
2. The abort path is simpler than the injection path (ADR-02) — Lease issuer and kill switch are a separate deployment with their own identity, their own store and a 60-second RTO against the control plane's 30 minutes.
3. Blindness aborts (ADR-03) — If the platform cannot evaluate the hypothesis, it cannot hold the fault. The verdict is INCONCLUSIVE, never a pass.
4. Blast radius is computed (ADR-05) — Three measures — replica share, request-volume share, downstream closure — and a veto on any run reaching a shared single point of failure.
5. Chaos spends a real budget (ADR-07) — Ten per cent of the service's own monthly SLO error budget, debited per run, refused at exhaustion.
6. No adapter without a proven revert (ADR-09) — Inject, verify, revert, report_state. An adapter whose revert has not been exercised in the last 30 days is ineligible for production.
7. Real traffic, bounded slice (ADR-12) — Shadow and synthetic traffic systematically miss retry storms and cache stampedes, which is the class of failure worth finding.
8. A pass needs a positive observation (ADR-14) — Where a mitigation can be observed — breaker opened, fallback served, retry budget held — the hypothesis must assert it, not merely the absence of harm.
9. The agent verifies its own permission (ADR-15) — Signature and scope are checked in the data plane, so a compromised control plane cannot widen a blast radius.

Why this should still be right in ten years

A chaos platform is unusual in that its central guarantee is about what happens when everything else is broken. That guarantee is not a technology choice, which is why most of this design should outlive the services it tests and all of the Google Cloud products named in it.

• Fail-closed is not a feature, it is the shape. The lease is what makes every failure mode converge on the same safe outcome. No new fault class, cloud migration or scale change alters that, and nothing in the lease mechanism depends on Google Cloud.
• The untested set is the durable product. Coverage — which dependencies have never been exercised — stays useful long after any individual verdict expires. Teams change, services are rewritten, and the gap report keeps answering the same question.
• Adapters are where churn lives, and it is contained. Injection mechanisms will change: eBPF will displace traffic control, meshes will gain and lose features. The capability-declaring adapter contract means that churn never reaches the control plane.
• The hardest part is organisational and the design respects it. Chaos error budgets, ownership-based authorisation and findings tracked to closure exist because the real constraint is remediation capacity, not injection capacity. That will still be true.
• What would break it. A feature request for a fault that cannot be reverted — a destructive data operation, an irreversible schema change — would put the first hole in the fail-closed guarantee. The adapter contract exists so that request has to be refused explicitly rather than accommodated quietly.

Non-functional targets

Every number here is a stated assumption from the requirement document, chosen to be defensible and arguable rather than measured. The view column points at the page where the number is visible in the design rather than merely asserted.

Scope

In scope

• Declarative, versioned experiment definitions with a mandatory falsifiable hypothesis, abort conditions and radius limits.
• Six fault classes: network, dependency response, resource, lifecycle, clock and stateful dependency.
• Computed blast radius with reservation-based mutual exclusion, exclusion windows, chaos error budgets and a global kill switch.
• Lease-based injection with watchdog revert, verified reversion and an out-of-band reverter.
• Steady-state evaluation against the target's own SLIs, five verdict classes and sealed evidence bundles.
• Ad-hoc, scheduled, canary-gated, continuous and game-day execution modes with a promotion ladder between them.
• Findings tracked to remediation, and per-service resilience coverage reporting including the untested set.

Explicitly out of scope

• Load and performance testing. Injecting load and injecting failure need different guardrails and different verdicts, and sharing a platform between them weakens both.
• Security red-teaming. The adversary model is different and the authorisation model would have to be, too.
• Functional test automation and CI unit testing, which belong to the deployment pipeline.
• Owning the service catalogue, dependency graph, SLO definitions or on-call rotations — all read, none authored here.
• Incident management. The platform files findings and receives suspension signals; it does not run the incident.

What a four-week prototype should prove

Four claims in this architecture are cheap to test now and very expensive to discover wrong after thirty teams depend on the platform. Everything else can be built incrementally against a working answer to these.

1. The fail-closed guarantee end to end: kill the lease issuer mid-experiment and measure, from outside, how long the fault survives against the 15-second TTL.
2. The abort budget: from a deliberately induced SLI breach, measure breach-to-reverted at p99 against the 10-second target, including evaluation lag.
3. The dependency closure: compute the downstream reachable set for twenty real services and check, by hand, how often it is wrong and in which direction.
4. Reversion verification: run every adapter a thousand times and count how many reverts need the bounded retry, and how many need the out-of-band reverter.

• Partition the agent from the control plane mid-injection and confirm the fault clears within the TTL with no command delivered.
• Degrade the metrics pipeline during an injection window and confirm the run aborts and records INCONCLUSIVE rather than passing.
• Start two experiments whose computed radii intersect and confirm the second queues rather than running alongside the first.
• Open a real incident touching a downstream service mid-run and confirm injection suspends fleet-wide within 30 seconds.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 16 records across 5 areas, each with the alternatives that lost and what the choice costs.

Chaos Engineering Platform · Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records

The argument these decisions serve is summarised in the Architecture One-Pager.

Sixteen decisions make up this architecture. Everything else across the twenty-one views is either a consequence of one of them or a detail that could be decided differently next month without anybody noticing. Each record states the question as it was actually open, the options that were weighed, what was chosen, how it is realised on Google Cloud, what the choice costs, and the conditions under which a different answer would be right.

Status of this document. This is a design, not a report on a running system. Every rate, latency, cap and retention figure is a stated assumption chosen to be defensible and arguable rather than measured — the operating context is a consumer streaming service of roughly 140 million subscribers, 900 services and 40,000 pods, which is itself an assumption. Where a number is load-bearing it is named in the record that depends on it, so a reviewer can change the number and see which decisions move with it.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on Google Cloud: The concrete mechanism: which service or package, configured how, in which project.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Authority and abort: Where the permission to keep a fault alive lives, and how that permission ends when everything else is broken.

• ADR-01 · Injection authority is a lease held in the data plane, not a command issued from the control plane
• ADR-02 · The abort path is deployed separately and holds fewer dependencies than the injection path
• ADR-03 · Telemetry blindness aborts the run, and the verdict is INCONCLUSIVE rather than a pass
• ADR-04 · The global kill switch is separately authorised and independently reachable

Blast radius and guardrails: How far a run is allowed to reach, who is allowed to say no, and what the run spends to get its answer.

• ADR-05 · Blast radius is a computed downstream closure, not the selector the author declared
• ADR-06 · Intersecting blast radii are resolved by reservation, and the later run queues rather than fails
• ADR-07 · Chaos spends a carved-out share of the service's own error budget
• ADR-08 · The incident feed can veto injection without a human in the loop

Injection and reversibility: How a fault is created, at which layer, and how the platform proves it is gone.

• ADR-09 · Every fault is a capability-declaring adapter, and no adapter ships without a proven revert
• ADR-10 · Faults are injected at the mesh and node layers; in-application libraries are deferred
• ADR-11 · Reversion is verified by re-reading target state, and an unconfirmed revert is an incident

Verification and verdicts: What the run is measured against, what traffic it affects, and what counts as a pass.

• ADR-12 · Production experiments affect real customer traffic on a bounded slice
• ADR-13 · Steady state is evaluated on a dedicated high-resolution path over the owners' own SLIs
• ADR-14 · A pass requires a positive recovery observation, not merely the absence of a breach

Operating it: Who the agent proves it is, and where the money goes.

• ADR-15 · Agents hold per-cluster workload identity and verify their own lease
• ADR-16 · High-resolution collection is windowed, and cost is attributed per team

Technology by capability

Every capability on the twenty-one views, the Google Cloud service chosen for it, the alternative that was weighed, and the reason. Google Cloud was chosen for rotation rather than necessity: chaos engineering belongs to no cloud, and the previous use cases in this practice leaned on Azure, AWS and bare open source. Where the choice is genuinely cloud-independent — the lease, the adapter contract, the verdict model — the record says so, because those are the parts a reader should be able to lift onto any platform.

The decisions, and the alternatives that lost

Authority and abort

Where the permission to keep a fault alive lives, and how that permission ends when everything else is broken.

ADR-01 · Injection authority is a lease held in the data plane, not a command issued from the control plane

Status: Accepted · Shown on views: 02, 12, 13, 16, 21

When a fault must stop, does something have to successfully tell the target to stop — or does the permission to be broken simply run out?

Context. Every published chaos tool starts from a command model: a controller injects, and later the same controller reverts. It is the obvious design and it works on the happy path. Its defect only shows up in the cases the platform exists for. The moments when a fault most needs to stop are exactly the moments when a stop command is least likely to arrive: the controller has crashed, the region is degraded, the network between the control plane and the cluster is the thing being tested. A command model makes the platform's worst failure mode — losing contact with the target — coincide with production's worst outcome, the fault staying. Softer variants do not fix it. A revert queued for retry still depends on a component that may be gone; a finaliser still depends on a controller reconciling; a timeout inside the injection process still depends on that process surviving.

Decision. Injection is held under a signed, target-bound, time-bound lease. The control plane decides that an experiment may run, computes the radius and issues the lease; the in-cluster agent is the only component that touches a target, and it holds the fault only while that lease is valid and renewed. Nothing in the system tells a target to stop being broken. Aborting is the absence of a renewal.

How it is realised on Google Cloud. The lease issuer runs on Cloud Run with its own service account and signs each lease with a Cloud KMS key; claims name the run, the resolved target set, the fault class and an expiry 15 seconds out. The agent — a DaemonSet in the chaos-system namespace — verifies the signature and scope itself and renews every 5 seconds. Guardrail and lease state lives in a regional Spanner instance, so no renewal or expiry decision crosses a region. Lease records are written to the run store as evidence of exactly how long permission was held.

What it buys

• There is no failure mode in which uncertainty keeps a fault alive, and no code path that has to be correct for that to be true.
• The abort budget stops depending on network delivery: the worst case for an unattended fault is the TTL, 15 seconds, by construction.
• Safety separates cleanly from orchestration — one component decides permission, another decides sequence, and permission wins.

What it costs

• Renewal becomes a load-bearing hot path: ~2,000 renewals a second expected and 40,000 at full fleet saturation.
• A long experiment is a chain of renewals, so a control-plane restart ends it rather than pausing it. There is no way to hold a fault steady across a deploy.
• Clock skew between the issuer and the agent becomes a safety concern rather than a nuisance, which is why the clock adapter is bounded to a declared offset.
• Every fault class must be expressible as something an agent can hold and drop — which rules out inherently irreversible faults, deliberately.

Choose differently when. Use a command model when the target environment cannot host an agent at all — a managed service you can only degrade through its own API, or a third-party dependency you can only fail by changing a DNS record. There the revert genuinely is a call, and the honest answer is to accept a longer abort budget and say so in the requirement rather than pretend the TTL applies.

Why it holds up over time. This is the decision least tied to Google Cloud and most likely to be eroded by a feature request rather than by a technology change. Nothing here depends on Cloud Run, Spanner or GKE. What would break it is a request for a fault that cannot expire — a destructive data operation, an irreversible migration — which is why the adapter contract in ADR-09 forces that request to be refused explicitly rather than accommodated quietly.

Lesson. When a system's job is to behave correctly while everything around it fails, prefer a guarantee that holds through absence over one that depends on delivery. A permission that expires needs nothing to work; a command needs everything to work.

ADR-02 · The abort path is deployed separately and holds fewer dependencies than the injection path

Status: Accepted · Shown on views: 06, 07, 16, 17, 20

Do the components that start experiments and the components that stop them ship and fail together?

Context. It is natural to build one service that owns the whole experiment lifecycle. It is also the one coupling this platform cannot afford, because a bad rollout then removes the brake and the accelerator at the same time. The abort path is the only part of the design with no graceful degradation: every other failure class has a safe fallback, and abort-path failure has none. That asymmetry has to show up in the deployment topology, not only in the code.

Decision. Lease issuance, renewal and the kill switch are a separate deployment with their own service account, their own datastore and their own release cadence. The abort path holds strictly fewer dependencies than the injection path, has a 60-second RTO against the control plane's 30 minutes, and is verified on a schedule of its own rather than as a side effect of testing the platform.

How it is realised on Google Cloud. Lease issuer and kill switch are separate Cloud Run services in both the control region and the warm-standby region, each with a dedicated service account and IAM binding, reading and writing only the regional Spanner guardrail instance. They share no library with the run controller beyond the signed-claim format. A scheduled self-chaos job kills the issuer in staging and in one production cluster on a fixed cadence and asserts that faults clear within the TTL.

What it buys

• A control-plane rollout, or a control-plane outage, cannot take the abort path with it.
• The abort path's dependency list is short enough to be read in one sitting, which is the only way to keep the claim true over time.
• Its RTO can be set an order of magnitude tighter than the rest of the platform because there is much less to recover.

What it costs

• Two deployments, two release pipelines and two on-call surfaces for what one team thinks of as one system.
• The signed-claim format becomes a versioned contract between the two, with all the compatibility discipline that implies.
• Duplication is required by design: the safety plane may not import the control plane's convenience libraries, so some code exists twice.

Choose differently when. Merge them when the platform is small enough that one team ships it weekly and the abort path has three dependencies anyway — separating them then buys ceremony rather than safety. Re-separate the moment the control plane grows a dependency the abort path does not need.

Why it holds up over time. The principle outlasts the products: whatever replaces Cloud Run, the brake should have fewer dependencies than the accelerator. The risk to this decision is gradual rather than sudden — a convenience import here, a shared cache there — which is why the dependency list of the abort path is an explicit review item rather than a convention.

Lesson. Put the asymmetry in the topology. If one component must keep working when the others do not, it should not be able to be broken by their release.

ADR-03 · Telemetry blindness aborts the run, and the verdict is INCONCLUSIVE rather than a pass

Status: Accepted · Shown on views: 13, 14, 18, 09

When the platform can no longer see whether the target is healthy, does the experiment continue?

Context. The tempting answer is to keep going: the fault is bounded, the run is short, and the metrics pipeline will probably come back. The problem is that the hypothesis is the only thing distinguishing an experiment from an outage, and evaluating it requires signals. A run that continues while blind has stopped being an experiment and become an unsupervised fault. There is a second, subtler failure: if a blind run that produces no alerts is recorded as a pass, the platform converts missing telemetry into false confidence, and does it silently, exactly when the observability estate is already degraded.

Decision. Inability to evaluate the hypothesis is an abort condition of the same standing as a breach. If telemetry coverage during the injection window falls below a configured completeness threshold, the run aborts and the verdict is INCONCLUSIVE. A run the platform could not observe is never recorded as passing.

How it is realised on Google Cloud. The evaluator tracks per-signal sample completeness over each 10-second window against expected cadence. Falling below the threshold stops the steady-state confirmation, which stops lease renewal, which expires the fault. Completeness is recorded in the evidence bundle so the verdict can be re-examined. The INCONCLUSIVE share is a reported metric on view 18 precisely because it is the number that tells you the platform is spending fear and returning nothing.

What it buys

• The platform never reports health it did not observe, which is what makes a HYPOTHESIS_HELD verdict worth anything.
• Observability degradation becomes visible through the INCONCLUSIVE rate rather than hiding inside optimistic verdicts.
• Abort has one mechanism for breach and for blindness, so there is less code that must be correct.

What it costs

• Runs are wasted: a 6% INCONCLUSIVE rate is an assumed 6% of the platform's cost and of its users' patience returned as nothing.
• The completeness threshold is a tuning parameter with no obviously right value, and setting it too tight makes the platform look flaky.
• Owners of services with thin SLI coverage are effectively blocked, which turns a platform decision into a conversation about somebody else's instrumentation.

Choose differently when. Allow a bounded continuation on last-known-good when the fault class is provably self-limiting — a single pod terminated, where the worst case is already covered by the orchestrator — and the remaining window is shorter than the abort budget. That is a narrow exception and it should be declared per fault class, never as a global default.

Why it holds up over time. This decision gets harder to hold as the platform succeeds, because the pressure comes from users who have paid the fear and want an answer. The defence is that the INCONCLUSIVE rate is published: a number everyone can see is easier to fix at the source than a rule everyone wants waived.

Lesson. Absence of evidence is not evidence of health. A system that cannot distinguish "nothing went wrong" from "we could not tell" will eventually report the second as the first.

ADR-04 · The global kill switch is separately authorised and independently reachable

Status: Accepted · Shown on views: 13, 20, 07, 18

When somebody needs to stop everything, does the thing they press depend on the platform that is misbehaving?

Context. A kill switch inside the platform's own API is worth very little in the scenario it exists for. If the API is the thing failing, or an authorisation lookup against a degraded identity dependency is on its path, the switch is decorative. And the audience matters: the person pressing it may be an incident commander who has never used the platform, in the first minutes of an unrelated outage, who needs one certainty — that pressing it stops the injection.

Decision. The kill switch is a separate endpoint with its own IAM binding, reachable independently of the platform API, that halts all injection fleet-wide within 30 seconds. It is exercised on a schedule, and every use is treated as an event worth a conversation rather than as routine operation.

How it is realised on Google Cloud. A dedicated Cloud Run service in both regions, fronted by its own IAM policy granting a small named group, writing a single fleet-wide halt flag to the regional Spanner guardrail instance. Agents' watchdogs read the flag on their renewal path, so a halt takes effect within one renewal interval and no later than the TTL. A scheduled job presses it in staging weekly and in production quarterly, and the alert on kill-switch use is unconditional.

What it buys

• The one control that must work in the worst case does not depend on the components most likely to be failing.
• Because agents read the halt flag on the renewal path, the switch inherits the lease guarantee: worst case is the TTL, not a delivery.
• A small named group with its own binding makes the audit trail unambiguous.

What it costs

• Another service, another IAM policy, another thing that has to be kept working and rehearsed or it decays into decoration.
• A fleet-wide halt is blunt: it stops legitimate runs alongside the problem one, and there is no partial version of it by design.
• Quarterly production rehearsal is itself a small planned disruption that has to be scheduled and explained.

Choose differently when. Fold it into the API when the platform is one service and one cluster, where a separate endpoint has the same failure domain anyway. Split it out the moment there is more than one cluster or more than one team who might need to press it.

Why it holds up over time. Kill switches decay quietly: they stay in the architecture and stop working. The scheduled rehearsal is the only part of this record that keeps it true, which is why it is stated as a requirement rather than an operational nicety.

Lesson. A control for the worst case must be rehearsed in the ordinary case, or it is a diagram of a control.

Blast radius and guardrails

How far a run is allowed to reach, who is allowed to say no, and what the run spends to get its answer.

ADR-05 · Blast radius is a computed downstream closure, not the selector the author declared

Status: Accepted · Shown on views: 14, 08, 11, 09

Is the blast radius what the definition says it is, or what the dependency graph says the definition reaches?

Context. "Five per cent of replicas" sounds bounded and often is not. If all five per cent of the affected pods talk to the same cache tier, degrading them can saturate that tier and affect everything behind it. If the selector resolves to a zone that also hosts a shared broker, the run is a zone experiment wearing a service costume. Enforcing only the literal selector is cheap, predictable and systematically optimistic in exactly the cases that produce incidents. Computing the transitive downstream set is honest and depends on a graph that is always somewhat wrong.

Decision. Blast radius is computed before injection as three independent measures — share of the target's healthy replicas, share of production request volume, and the set of downstream services reachable from the affected instances — derived from the dependency graph. A run whose reachable set includes a service marked a shared single point of failure is refused. The declared selector is an input to the computation, never the answer.

How it is realised on Google Cloud. The radius calculator reads the service catalogue and the mesh-derived dependency graph, both no older than 24 hours or the run is blocked, and writes the three measures plus the reachable set into the run's guardrail evaluations in Spanner. Request-volume share comes from the mesh's own traffic statistics rather than from a declared estimate. The closure is re-checked during injection so a scale-up event cannot turn a compliant 5% into a non-compliant one silently.

What it buys

• The guardrail measures the property that matters — how far the failure can actually reach — rather than a proxy for it.
• The shared-SPOF veto turns the most dangerous class of experiment into an explicit refusal instead of a surprise.
• Request-volume share read from the mesh means the second measure cannot be talked down by an optimistic estimate.

What it costs

• The design now depends on graph freshness, and a stale graph blocks runs rather than degrading, which feels like the platform being broken.
• Over-conservative closures refuse legitimate runs, and the failure mode is quiet: teams route around the platform rather than complaining.
• Re-checking during injection adds a hot read on a path where latency matters.

Choose differently when. Enforce the declared selector when the estate has no shared tiers worth protecting, or when the dependency graph is unreliable enough that closures are mostly wrong — an inaccurate closure that refuses good runs is worse than an honest selector plus a careful author. Say which it is in the requirement rather than letting it be an implementation detail.

Why it holds up over time. The decision survives changes in how the graph is produced — mesh telemetry today, eBPF or distributed tracing later — because the calculator consumes a graph rather than a source. The risk is drift in what "shared single point of failure" means; that marking belongs in the service catalogue, owned by the people who know, not inferred here.

Lesson. When a limit is expressed as a percentage of something, check what that something is connected to. A bounded share of a fleet is not a bounded share of the blast.

ADR-06 · Intersecting blast radii are resolved by reservation, and the later run queues rather than fails

Status: Accepted · Shown on views: 14, 05, 11, 16

What happens when two experiments would affect overlapping parts of the fleet at the same time?

Context. Two individually compliant runs can be jointly catastrophic: 5% of one service plus 5% of a shared downstream is not 5% of anything. The problem is not detecting the overlap — the closures are already computed — but deciding what to do. Rejecting the second run trains people to retry in a loop, and at game-day concurrency the retry storm is on the platform itself. Running both and hoping is how a game day becomes an incident.

Decision. A permitted run holds a reservation over its computed radius for its declared duration. A run whose radius intersects a live reservation queues until the reservation is released, rather than being rejected. Conflicts are surfaced at plan time for game days, so a sequence is corrected before the session rather than during it.

How it is realised on Google Cloud. Reservations are rows in the regional Spanner guardrail instance, written in the same transaction that permits the run and released on terminal state or lease expiry. The radius calculator takes the reservation check as part of its evaluation, so intersecting runs never reach the lease issuer. The game-day sequencer runs the same check over a whole plan at authoring time and reports collisions as a plan error.

What it buys

• Concurrency is safe by construction rather than by convention, and the safety does not depend on users coordinating.
• Queueing keeps the contract simple: a permitted run will happen, just possibly later.
• Plan-time detection removes the worst game-day failure, which is rewriting the sequence live under time pressure.

What it costs

• The reservation store becomes the contended resource at high concurrency — which is the intended place for contention, and still a bottleneck to design for.
• A queued run can miss its window and be skipped, which users experience as the platform silently not running their experiment.
• A long-held reservation from a stuck run blocks others until its lease expires, so reservation lifetime is tied to the TTL rather than to the run's declared duration.

Choose differently when. Reject instead of queueing when experiments are cheap to re-issue and the caller is a pipeline that handles backoff properly. Queueing earns its complexity when a human is waiting and would otherwise sit in a retry loop.

Why it holds up over time. Reservation is a durable idea; the implementation will move as the store does. The pressure on it comes from scale: if concurrency grows an order of magnitude, radius intersection testing becomes the hot path and will need an index rather than a scan.

Lesson. Individually safe limits do not compose. If two compliant actions can be jointly non-compliant, something has to hold a reservation.

ADR-07 · Chaos spends a carved-out share of the service's own error budget

Status: Accepted · Shown on views: 14, 18, 11, 06

Does deliberate failure cost anything, and if so, whose budget does it come out of?

Context. If experiments are free, the only limit on them is politeness, and the argument about how much chaos is too much has no common currency. If they are charged to a separate allowance invented for the platform, the number is arbitrary and nobody believes it. The service's own SLO error budget is the one quantity the owners already argue about in terms they accept, which makes it the honest place to bill — and immediately raises the question of who owns the decision to spend it.

Decision. Each service reserves a share of its monthly SLO error budget for chaos — 10% as a stated assumption — debited when a lease is issued and settled to actual spend at abort or completion. A service that has exhausted its chaos share is refused further runs until the budget resets or its owner raises the share. The owner controls the share; the platform controls the accounting.

How it is realised on Google Cloud. The budget ledger is a Spanner table keyed by service and month, read from the SLO store's budget at the start of each period. Debits are estimated from the declared radius and duration at lease issue and reconciled against observed SLI impact when the run ends, so an aborted run pays only for what it spent. Consumption is reported per service and per team on view 18.

What it buys

• The cost of learning is expressed in the same unit as the cost of failing, so the trade-off can be argued rather than asserted.
• A service under SLO pressure automatically gets less chaos, which is the right behaviour and needs no human to enforce it.
• Owner-set shares mean a team can opt into more testing, which is a much better incentive than a platform-imposed cap.

What it costs

• Estimating a run's budget cost before it happens is genuinely hard, and over-estimating blocks legitimate runs.
• A service with no SLO cannot be charged, so it either cannot be tested or needs an exception path — and exception paths erode.
• An owner can set the share to zero and never test anything, and the platform has no answer to that beyond the coverage report.

Choose differently when. Skip budget accounting entirely while the platform is small and every run is attended: the human watching is the budget. Introduce it when unattended and continuous runs make the aggregate spend invisible, which is the point at which "it is only 1%" stops being checkable.

Why it holds up over time. This is the most organisational decision in the package and the one most likely to be renegotiated. The mechanism — debit, settle, refuse at exhaustion — survives; the 10% will not, and it is stated as an assumption so that changing it is a configuration change rather than a redesign.

Lesson. Bill deliberate risk in the same currency as accidental risk. A separate budget for planned failure is a number nobody will defend under pressure.

ADR-08 · The incident feed can veto injection without a human in the loop

Status: Accepted · Shown on views: 14, 08, 05, 13

During a real production incident, who decides whether experiments keep running?

Context. The decision is easy in the abstract and terrible in practice, because the person best placed to make it is the one with the least attention to spare. An incident commander in the first five minutes of an outage should not have to work out which of 200 in-flight experiments might be contributing, nor should a service owner watching their own run be asked to judge whether somebody else's incident is related. Leaving it to humans means the answer arrives late, and late is the only time it matters.

Decision. An open incident of configured severity suspends new injection automatically, and aborts any in-flight run whose target or downstream reachable set intersects the incident's affected services. No human confirmation is required to suspend; human judgement is required only to resume.

How it is realised on Google Cloud. The platform subscribes to the incident platform's event stream through Pub/Sub and maintains a suspension flag and an affected-service set in the Spanner guardrail instance. The incident gate is evaluated at lease issue and on every renewal, so an intersecting in-flight run stops being renewed and expires within the TTL. Resumption is an explicit action, recorded in the audit log with the principal who took it.

What it buys

• The decision is made in milliseconds by a rule rather than in minutes by a person who is busy.
• The asymmetry is honest: an unnecessary suspension costs a re-run, a missed suspension costs an outage lengthened by a self-inflicted fault.
• Because the gate is evaluated on the renewal path, suspension inherits the lease guarantee rather than needing its own delivery.

What it costs

• Two automatic suspensions a day is assumed, which makes interrupted runs ordinary and forces scheduling to treat skipping as normal.
• The platform now depends on the incident platform's severity taxonomy being meaningful, and on its affected-service field being filled in.
• A mis-scoped incident record can suspend a large amount of unrelated work, and the platform has no way to tell.

Choose differently when. Require human confirmation when incidents are declared loosely enough that most of them are noise, because a gate that fires constantly gets disabled. Fix the taxonomy first if you can; a veto based on an unreliable signal is worse than no veto, since it teaches people to work around it.

Why it holds up over time. The mechanism is stable. What will move is the severity threshold and the intersection rule, both of which are configuration. The real risk is the platform being blamed for interrupted runs and the threshold being quietly raised until the gate stops firing — which is why suspension counts are reported.

Lesson. Automate the decision that has to be made fastest by the person with the least time. Reserve human judgement for the direction where a mistake is cheap.

Injection and reversibility

How a fault is created, at which layer, and how the platform proves it is gone.

ADR-09 · Every fault is a capability-declaring adapter, and no adapter ships without a proven revert

Status: Accepted · Shown on views: 03, 07, 14, 17

How does a new kind of failure get added, and what stops one being added that cannot be undone?

Context. A chaos platform accretes fault types, and the pressure to add them comes with a deadline attached. If each is a script, reversibility is a property of whoever wrote it that afternoon, and the platform's central guarantee degrades one commit at a time. Worse, the most requested faults are often the least reversible — truncate this table, corrupt this message, expire this certificate — and without an explicit contract those requests get accommodated quietly rather than refused.

Decision. Every fault class is an adapter implementing inject, verify, revert and report_state, declaring the capabilities it needs. The control plane refuses a fault the target environment cannot reverse, and an adapter whose revert has not been exercised successfully against that service in a non-production environment within 30 days is ineligible for production.

How it is realised on Google Cloud. Adapters are Go plugins compiled into the agent binary with a declared capability manifest, so the set of adapters in a cluster is a property of a signed image rather than of runtime configuration. The revert-proof suite in Cloud Build exercises every adapter's inject/revert cycle a thousand times per release and fails the build on any unconfirmed revert. Non-production proving windows are tracked per service and per class in Spanner and enforced at validation.

What it buys

• Adding a fault class never touches the control plane, so the blast radius of a new adapter is one image.
• report_state makes drift detectable: the platform can tell the difference between a fault it applied and a fault it merely asked for.
• The 30-day proving window turns reversibility from a claim into a recent observation.

What it costs

• Four methods and a capability manifest is a real barrier to a five-line fault, and some legitimate experiments will not be worth the effort.
• Compiling adapters into the agent means a new fault class requires an agent release, which is the highest-risk release the platform has.
• Genuinely useful irreversible faults are refused, and the platform has to say no to them repeatedly.

Choose differently when. Allow scripted faults in non-production only, where reversibility is a convenience rather than a guarantee and the fast path is worth more than the contract. The line between environments should be enforced by the platform, not by intent.

Why it holds up over time. The contract is the durable part and it is technology-independent: inject, verify, revert, report_state would be the same four verbs on any platform. What will churn is the mechanism inside each adapter — eBPF displacing tc, mesh features arriving and leaving — which is exactly what the contract was drawn to contain.

Lesson. Make the extension point carry the guarantee. If reversibility is enforced by the interface, it cannot be eroded one urgent request at a time.

ADR-10 · Faults are injected at the mesh and node layers; in-application libraries are deferred

Status: Accepted · Shown on views: 03, 07, 14, 20

At which layer is a fault created — the proxy, the kernel, or the application's own code?

Context. The three layers differ in fidelity, privilege and adoption cost, and they are not substitutes. A mesh fault filter is portable, application-transparent and trivially reversible, but it only sees what the proxy sees: it cannot express a slow disk, a corrupted deserialisation or a connection pool exhausted inside the process. Kernel-level manipulation through traffic control or eBPF has much higher fidelity and needs privilege that is uncomfortable to hold in production. In-application libraries express dependency semantics best of all and require every one of 900 services to adopt them, which means the coverage report would measure adoption rather than resilience.

Decision. Injection happens at the mesh layer where the proxy can express the fault, and at the node layer through traffic control and cgroup limits where it cannot. In-application fault libraries are deferred: they are the right answer for a handful of specific semantics and the wrong basis for a platform that must cover a fleet it does not control.

How it is realised on Google Cloud. The agent configures mesh fault filters through the mesh's own control API for HTTP-level latency, error status and body faults, and applies tc/netem qdiscs and cgroup limits directly for packet-level, resource and node-level faults. Privileged classes — netem, node actions — require a distinct authorisation in the lease claim, so the ordinary case does not carry the dangerous capability.

What it buys

• No service has to adopt anything, so coverage measures resilience rather than library uptake.
• Mesh-level faults are reversible by removing a configuration, which is the cheapest possible revert.
• Splitting privileged classes into a separate authorisation keeps the common case low-privilege.

What it costs

• The agent holds node-level privilege in production, which is the largest standing capability in the estate and needs its own review cadence.
• Some genuinely important failures — in-process pool exhaustion, deserialisation faults — cannot be expressed at all, and the coverage report has to admit that.
• Two mechanisms means two reversal paths, and tc rules are the ones that get stuck.

Choose differently when. Go application-library-first when the estate is small, homogeneous and owned by one team — there the adoption cost is a day and the fidelity is worth it. Go mesh-only when policy will not permit node-level privilege in production, and accept that three fault classes are then out of scope rather than pretending otherwise.

Why it holds up over time. The layering argument is stable; the specific tools are not. eBPF is the likely successor to tc within the decade and the adapter contract exists so that is a swap rather than a redesign. The decision that will age is deferring application libraries — as service frameworks gain fault-injection hooks by default, the adoption argument weakens.

Lesson. Pick the injection layer by what you are allowed to require of other teams, not by what gives the best fidelity in a demo.

ADR-11 · Reversion is verified by re-reading target state, and an unconfirmed revert is an incident

Status: Accepted · Shown on views: 12, 13, 18, 17

Is a fault gone because the revert call returned, or because the platform looked?

Context. The gap between "revert returned successfully" and "the fault is gone" is where the worst outcomes live. A tc qdisc can survive the process that created it. A mesh filter can be removed from the control plane and remain in a proxy that has lost its connection. A cgroup limit can persist on a node whose agent has been rescheduled. In every one of these cases the platform's own records say the experiment ended cleanly, and production is still degraded, and nobody is looking for a fault because the run is closed.

Decision. After revert the agent re-reads target state and confirms absence. If the fault is still present after a bounded retry sequence, the out-of-band reverter is invoked, on-call is paged, and that fault class is suspended fleet-wide pending review. A post-abort settling check confirms the steady-state signals returned to baseline; a run whose signals did not recover is recorded as an unresolved finding rather than a completed experiment.

How it is realised on Google Cloud. report_state on each adapter is the read path, so verification uses the same interface as injection rather than a parallel implementation. Retries are bounded at three attempts over ten seconds before escalation. The out-of-band reverter runs in the same namespace under a break-glass service account, so its use is both possible and abnormal. Verification outcomes are recorded per applied fault, and the fleet-wide suspension is a flag on the same guardrail store the lease issuer reads.

What it buys

• The platform's record and production's state cannot diverge silently, which is the property every safety claim in this package rests on.
• Suspending the fault class fleet-wide turns one stuck revert into a bounded problem instead of a repeated one.
• The settling check catches the case where the fault is gone and the service has not actually recovered — which is a finding, not a clean run.

What it costs

• Every adapter needs a meaningful read path, which roughly doubles the work of adding a fault class.
• An assumed one unconfirmed revert per 900 injected faults means real pages for an on-call rotation, some of which will be verification bugs rather than stuck faults.
• Fleet-wide suspension of a class is disruptive: one bad node can stop that fault class everywhere until somebody reviews it.

Choose differently when. Trust the return value in non-production, where a stuck fault costs a cluster rebuild rather than customer impact. In production there is no version of this that is safe to skip, because the whole design assumes the fault is gone.

Why it holds up over time. This is a discipline decision and disciplines erode under alert fatigue. The pressure will be to downgrade the page to a ticket after the third false positive. The defence is that verification failures are counted rather than rated — a single occurrence is visible — and that the suspension makes ignoring it expensive.

Lesson. The difference between a system that reverted and a system that says it reverted is a read. Do the read.

Verification and verdicts

What the run is measured against, what traffic it affects, and what counts as a pass.

ADR-12 · Production experiments affect real customer traffic on a bounded slice

Status: Accepted · Shown on views: 14, 04, 09, 18

Does a production experiment degrade requests that real customers made, or a copy of them?

Context. There are three honest answers and the safest one is the least useful. Shadow traffic — mirroring real requests to a parallel pool — protects customers completely and systematically misses the failures worth finding, because retry storms, cache stampedes and queue backpressure are properties of correlated load meeting a shared resource, and a shadow pool has neither the correlation nor the sharing. Synthetic load against a canary pool is better still for safety and worse still for fidelity. Real traffic on a bounded slice finds the emergent behaviour and spends a small amount of customer experience to do it.

Decision. Production experiments affect real customer traffic on a slice bounded by the computed blast radius and paid for from the chaos error budget. Shadow and synthetic modes remain available for adapter proving and for fault classes whose fidelity does not depend on correlated load.

How it is realised on Google Cloud. The slice is the resolved target set: the mesh routes a bounded share of live requests through faulted proxies, and the share is one of the three computed radius measures. Request-volume share comes from the mesh's traffic statistics rather than from an estimate, so the 1% cap is measured. Budget debits at lease issue make the customer cost explicit in the currency the owner already argues about.

What it buys

• Findings are about how the system behaves under real correlated load, which is the only behaviour that predicts an outage.
• Capping on measured request-volume share rather than replica share puts the limit where the customer actually is.
• The cost is explicit, budgeted and refusable, which makes the trade-off arguable instead of implicit.

What it costs

• Real customers are affected, by design. An assumed 1% cap on a 140-million-subscriber service is still a large number of people on a busy evening.
• Regulatory or contractual constraints may make some services ineligible, and the platform needs an honest exclusion list rather than a workaround.
• The blast radius computation becomes safety-critical: an error there is an error in how many customers are affected.

Choose differently when. Use shadow traffic when the service's failure is customer-visible and unrecoverable within one request — a payment authorisation, a safety interlock — where no share of real traffic is an acceptable price. Say so per service in the catalogue rather than deciding it per run.

Why it holds up over time. The reasoning is independent of any platform and will not change: correlated load is what produces emergent failure. What may change is the cap, as the service's own resilience improves and the same 1% costs less. That is a configuration change and it is stated as an assumption for exactly that reason.

Lesson. The fidelity of a resilience test is mostly the fidelity of its load. A safe test of unrealistic traffic answers a question nobody asked.

ADR-13 · Steady state is evaluated on a dedicated high-resolution path over the owners' own SLIs

Status: Accepted · Shown on views: 02, 12, 09, 18

Does the platform reuse the production SLO pipeline to judge the hypothesis, or build a faster path of its own?

Context. This is the sharpest trade-off in the design. Reusing the SLO pipeline gives one definition of truth, shared with the people who are paged, and an aggregation lag measured in minutes — against an abort budget of ten seconds. Building a dedicated high-resolution path gives a breach visible in seconds and creates a second, potentially divergent computation of the same SLI, which means the platform can say "healthy" while the SLO says "breached" and neither is obviously wrong.

Decision. A dedicated high-resolution evaluation path reads the signals underlying the target's own SLIs and computes them on 10-second rolling windows, windowed to the baseline and injection periods only. The SLI definitions come from the SLO store; the platform does not invent its own. Divergence between the two computations is treated as a bug in the platform, not as an acceptable difference.

How it is realised on Google Cloud. The signal collector reads Cloud Monitoring time series at the highest resolution the underlying metrics support, only for signals the hypothesis names and only inside the windows, which is what keeps the cost bounded. The evaluator recomputes the SLI using the definition pulled from the SLO store rather than a local copy. Baseline and injection windows are stored in BigQuery so a verdict can be re-derived and compared against the SLO pipeline after the fact.

What it buys

• Breach to fault-cleared fits inside ten seconds, which is what makes every safety claim in the package achievable.
• Because SLI definitions are pulled rather than copied, a change by the owners reaches the platform without a platform change.
• Windowing collection to baseline and injection is what keeps the second path affordable — it is the dominant cost lever in the design.

What it costs

• Two computations of the same SLI exist, and reconciling them is ongoing work rather than a one-off.
• Services whose SLIs are computed from logs rather than metrics cannot be evaluated at this resolution, and are effectively blocked.
• 500,000 samples a second during a large game day is a real ingestion cost on top of the existing monitoring bill.

Choose differently when. Reuse the SLO pipeline when the abort budget can be minutes — a background batch service, a nightly pipeline — where the fault's cost accrues slowly and one truth is worth more than speed. That is a per-service property and could be declared as such rather than decided globally.

Why it holds up over time. The trade-off is structural and will outlive Cloud Monitoring: someone will always want the fast path and the shared definition, and cannot have both. The pressure over time is drift, as the platform's recomputation acquires convenience shortcuts. Publishing both numbers for a sample of runs is the cheapest defence.

Lesson. When speed and shared truth conflict, take the speed and borrow the definition. Inventing your own metric is how a green verdict stops meaning anything.

ADR-14 · A pass requires a positive recovery observation, not merely the absence of a breach

Status: Accepted · Shown on views: 12, 19, 18, 11

Does "no SLO breach during the injection window" mean the mitigation worked?

Context. It very often does not. A dependency timeout can pass without breaching anything because the traffic that would have exercised the fallback did not arrive, or because a cache was warm, or because the retry budget was never approached. Recording that as a pass teaches the owner that their fallback works when what was actually established is that nothing bad happened this time. The stronger form — assert that the circuit breaker opened, the fallback served, the retry budget held — is harder to express, requires the mitigation to be observable at all, and is the only version that answers the question the owner asked.

Decision. Where a mitigation can be observed, the hypothesis must assert it positively and the verdict depends on that assertion as well as on the absence of a breach. Where it genuinely cannot be observed, the definition must say so and the resulting pass is recorded as weaker evidence in the coverage report.

How it is realised on Google Cloud. A hypothesis carries two kinds of clause: steady-state thresholds on the owners' SLIs, and positive assertions over named signals such as a breaker-state metric, a fallback-served counter or a retry-budget gauge. The verdict engine requires every positive assertion to fire before recording HYPOTHESIS_HELD. Coverage reporting distinguishes a pass with positive confirmation from a pass without one, so the untested set is honest about what kind of evidence it holds.

What it buys

• A pass means a mitigation was seen working, which is the claim the owner wanted and the one worth recording for two years.
• Grading evidence makes the coverage report honest: it can distinguish proven from merely undisturbed.
• It creates a concrete, useful instrumentation ask — expose breaker state, fallback counts, retry budget — instead of a vague one.

What it costs

• Services without mitigation instrumentation can only earn the weaker pass, which turns a platform decision into a conversation about someone else's metrics.
• Definitions are harder to write, which shows up as the assumed 22% validation rejection rate.
• A positive assertion can be wrong in the other direction: a breaker that opened for an unrelated reason satisfies the clause without proving anything.

Choose differently when. Accept absence-of-breach as a pass for fault classes where the mitigation is the orchestrator's rather than the service's — a single pod terminated, where the assertion is that the replica set recovered and that is already visible. The distinction is per fault class, not per service.

Why it holds up over time. This is the decision most likely to be watered down, because it makes the platform harder to use and the weaker version still produces green results. The defence is the graded coverage report: as long as the two kinds of pass are counted separately, the erosion is visible.

Lesson. Nothing going wrong is not the same as something working. If a mitigation can be observed, require the observation.

Operating it

Who the agent proves it is, and where the money goes.

ADR-15 · Agents hold per-cluster workload identity and verify their own lease

Status: Accepted · Shown on views: 21, 20, 07, 16

Where is a lease checked — at the control plane that issued it, or at the agent that will act on it?

Context. If the agent simply does what the control plane tells it, then compromising the control plane compromises the fleet: an attacker who can call the injection path can widen a blast radius arbitrarily, and the agent has no basis to refuse. If the agent verifies, the control plane's compromise is bounded by what a valid signed claim can express. The related question is what the agent's own identity is worth: a credential present in 40,000 pods is a credential that will eventually leave them.

Decision. Each agent holds a short-lived, cluster-scoped workload identity, and verifies the lease's signature and scope itself before acting. A lease naming a target outside the agent's cluster, a fault class it is not authorised for, or an expiry in the past is refused in the data plane. The agent can verify but cannot mint.

How it is realised on Google Cloud. GKE Workload Identity gives each cluster's agent a distinct, non-exfiltratable identity. The lease issuer signs claims with a Cloud KMS asymmetric key; agents hold only the public key, so a compromised agent cannot issue a lease for anything. Privileged fault classes require a distinct authorisation inside the claim, so an ordinary lease does not carry the dangerous capability even if it is replayed.

What it buys

• A compromised control plane cannot widen a blast radius beyond what a valid, signed, scoped claim expresses.
• Agents cannot self-authorise, so the dangerous capability lives only where it is issued.
• Cluster-scoped identity means a compromised cluster cannot request faults in another one.

What it costs

• Key rotation becomes a fleet-wide coordination problem with a verification path that must tolerate two valid keys.
• Clock skew between issuer and agent becomes a correctness concern for expiry checks, not just a nuisance.
• Verification is easy to hollow out over time — a skipped scope check during an incident is a plausible and invisible regression.

Choose differently when. Let the agent trust the control plane when both are inside one strongly isolated trust domain and the threat model does not include control-plane compromise — a single-team platform in a single project. At fleet scale the second opinion is worth its cost.

Why it holds up over time. Signed, scoped, short-lived claims verified by the actor is a pattern that long predates this platform and will outlast it. What will change is the key custody mechanism. The risk to watch is verification decay rather than cryptographic obsolescence, which is why scope refusals are counted on view 18.

Lesson. Let the component that will act be the one that checks. Authority verified only by its issuer is authority with no second opinion.

ADR-16 · High-resolution collection is windowed, and cost is attributed per team

Status: Accepted · Shown on views: 18, 09, 10, 17

What stops a resilience platform's telemetry bill from exceeding the value of the findings it produces?

Context. The dedicated evaluation path in ADR-13 is the platform's largest variable cost, and the naive implementation — collect everything at high resolution, all the time, across the fleet — would be more expensive than the incidents it prevents. The second cost is subtler: if experiments appear free to the team running them, a team will schedule a thousand low-value runs and nobody will notice until the bill arrives centrally, where it looks like a platform problem rather than a usage one.

Decision. High-resolution signals are collected only for the signals a hypothesis names, and only inside the baseline and injection windows. Runs terminate early once the hypothesis is decided with sufficient confidence. Series are downsampled after 90 days. All cost is attributed per service and per team, and the platform's total is held to 0.4% of fleet production compute spend.

How it is realised on Google Cloud. The collector subscribes per run rather than per service, so the ingestion rate is a function of concurrency and not of fleet size. BigQuery partition expiry implements the 90-day boundary as configuration, with downsampled summaries retained for the life of the run record. Cost attribution joins run records to signal volume and storage in BigQuery and is reported alongside coverage on view 18, next to the incidents the findings prevented.

What it buys

• Ingestion scales with concurrency, so adding services to the fleet does not raise the telemetry bill.
• Per-team attribution makes the cost of a thousand low-value scheduled runs visible to the team that scheduled them.
• Reporting platform cost against prevented-incident cost makes the funding argument checkable rather than rhetorical.

What it costs

• Windowed collection means a signal not named in the hypothesis is not available afterwards, so a run cannot be re-analysed for something nobody thought to ask.
• Early termination trades a little statistical confidence for cost, and the confidence threshold is another parameter with no obviously right value.
• Downsampling after 90 days means old verdicts cannot be re-derived at the resolution that produced them, only checked against their stored summaries.

Choose differently when. Collect continuously at high resolution when the fleet is small enough that the bill is noise, because the ability to re-analyse any run for anything is genuinely valuable. Windowing earns its constraint at the point where ingestion would scale with fleet size rather than with usage.

Why it holds up over time. The structural insight — make cost a function of usage, not of estate size — is durable and product-independent. The 0.4% figure is not, and is stated as an assumption. The pressure over time will be requests to widen collection so runs can be re-analysed, which is a real need better met by naming more signals in the hypothesis.

Lesson. Make the cost of a platform scale with how much it is used, not with how much there is to use it on. Then show each team their share.

Every package used, in one table

Eleven terms that mean something specific in this package. Where a term is used loosely elsewhere in the industry, the definition here is the one the views and records assume.