Architecture Views

When a fault must stop, does something have to successfully tell the target to stop — or does the permission to be broken simply run out?

Context

Every published chaos tool starts from a command model: a controller injects, and later the same controller reverts. It is the obvious design and it works on the happy path. Its defect only shows up in the cases the platform exists for. The moments when a fault most needs to stop are exactly the moments when a stop command is least likely to arrive: the controller has crashed, the region is degraded, the network between the control plane and the cluster is the thing being tested. A command model makes the platform's worst failure mode — losing contact with the target — coincide with production's worst outcome, the fault staying. Softer variants do not fix it. A revert queued for retry still depends on a component that may be gone; a finaliser still depends on a controller reconciling; a timeout inside the injection process still depends on that process surviving.

Decision

Injection is held under a signed, target-bound, time-bound lease. The control plane decides that an experiment may run, computes the radius and issues the lease; the in-cluster agent is the only component that touches a target, and it holds the fault only while that lease is valid and renewed. Nothing in the system tells a target to stop being broken. Aborting is the absence of a renewal.

How it is realised on Google Cloud

The lease issuer runs on Cloud Run with its own service account and signs each lease with a Cloud KMS key; claims name the run, the resolved target set, the fault class and an expiry 15 seconds out. The agent — a DaemonSet in the chaos-system namespace — verifies the signature and scope itself and renews every 5 seconds. Guardrail and lease state lives in a regional Spanner instance, so no renewal or expiry decision crosses a region. Lease records are written to the run store as evidence of exactly how long permission was held.

Options weighed

• ChosenLease held in the data plane, renewed from the control plane: Every failure mode — partition, crash, region loss, telemetry blindness — converges on expiry without being handled as a special case. Costs a hot renewal path and an experiment that cannot survive a control-plane restart.
• RejectedController commands inject, then commands revert: Simpler to build and easier to reason about on the happy path, and wrong here: the revert depends on the delivery that the failures under test disrupt. Needs a watchdog anyway, at which point the lease is the watchdog done honestly.
• RejectedAgent-local timeout with no renewal: Removes the renewal traffic, and removes the ability to abort early on a breach — the fault runs its full declared duration whatever the service is doing. Acceptable only for experiments short enough that the duration is the abort budget.
• RejectedKubernetes finaliser or operator reconciliation: Reconciliation naturally re-establishes declared state, which means a controller recovering after a crash re-injects the fault. That is precisely the behaviour fail-closed forbids.

Consequences

What it buys

• There is no failure mode in which uncertainty keeps a fault alive, and no code path that has to be correct for that to be true.
• The abort budget stops depending on network delivery: the worst case for an unattended fault is the TTL, 15 seconds, by construction.
• Safety separates cleanly from orchestration — one component decides permission, another decides sequence, and permission wins.

What it costs

• Renewal becomes a load-bearing hot path: ~2,000 renewals a second expected and 40,000 at full fleet saturation.
• A long experiment is a chain of renewals, so a control-plane restart ends it rather than pausing it. There is no way to hold a fault steady across a deploy.
• Clock skew between the issuer and the agent becomes a safety concern rather than a nuisance, which is why the clock adapter is bounded to a declared offset.
• Every fault class must be expressible as something an agent can hold and drop — which rules out inherently irreversible faults, deliberately.

Choose differently when

Use a command model when the target environment cannot host an agent at all — a managed service you can only degrade through its own API, or a third-party dependency you can only fail by changing a DNS record. There the revert genuinely is a call, and the honest answer is to accept a longer abort budget and say so in the requirement rather than pretend the TTL applies.

Why it holds up over time

This is the decision least tied to Google Cloud and most likely to be eroded by a feature request rather than by a technology change. Nothing here depends on Cloud Run, Spanner or GKE. What would break it is a request for a fault that cannot expire — a destructive data operation, an irreversible migration — which is why the adapter contract in ADR-09 forces that request to be refused explicitly rather than accommodated quietly.

Do the components that start experiments and the components that stop them ship and fail together?

Context

It is natural to build one service that owns the whole experiment lifecycle. It is also the one coupling this platform cannot afford, because a bad rollout then removes the brake and the accelerator at the same time. The abort path is the only part of the design with no graceful degradation: every other failure class has a safe fallback, and abort-path failure has none. That asymmetry has to show up in the deployment topology, not only in the code.

Decision

Lease issuance, renewal and the kill switch are a separate deployment with their own service account, their own datastore and their own release cadence. The abort path holds strictly fewer dependencies than the injection path, has a 60-second RTO against the control plane's 30 minutes, and is verified on a schedule of its own rather than as a side effect of testing the platform.

How it is realised on Google Cloud

Lease issuer and kill switch are separate Cloud Run services in both the control region and the warm-standby region, each with a dedicated service account and IAM binding, reading and writing only the regional Spanner guardrail instance. They share no library with the run controller beyond the signed-claim format. A scheduled self-chaos job kills the issuer in staging and in one production cluster on a fixed cadence and asserts that faults clear within the TTL.

Options weighed

• ChosenSeparate deployment, separate identity, fewer dependencies, own verification schedule: Makes the asymmetry between starting and stopping visible in the topology, and lets the brake keep working through a control-plane rollout.
• RejectedOne lifecycle service owning inject and revert: Fewer moving parts and a single release, at the price of the one coupling the design cannot absorb: a rollout that breaks the service breaks the brake.
• Right elsewhereAbort logic embedded in the agent only, with no central issuer: Right for a small estate where a per-node timeout is the whole safety story. At 900 services the central issuer is what makes the kill switch and the breach-driven abort possible.

Consequences

What it buys

• A control-plane rollout, or a control-plane outage, cannot take the abort path with it.
• The abort path's dependency list is short enough to be read in one sitting, which is the only way to keep the claim true over time.
• Its RTO can be set an order of magnitude tighter than the rest of the platform because there is much less to recover.

What it costs

• Two deployments, two release pipelines and two on-call surfaces for what one team thinks of as one system.
• The signed-claim format becomes a versioned contract between the two, with all the compatibility discipline that implies.
• Duplication is required by design: the safety plane may not import the control plane's convenience libraries, so some code exists twice.

Choose differently when

Merge them when the platform is small enough that one team ships it weekly and the abort path has three dependencies anyway — separating them then buys ceremony rather than safety. Re-separate the moment the control plane grows a dependency the abort path does not need.

Why it holds up over time

The principle outlasts the products: whatever replaces Cloud Run, the brake should have fewer dependencies than the accelerator. The risk to this decision is gradual rather than sudden — a convenience import here, a shared cache there — which is why the dependency list of the abort path is an explicit review item rather than a convention.

When the platform can no longer see whether the target is healthy, does the experiment continue?

Context

The tempting answer is to keep going: the fault is bounded, the run is short, and the metrics pipeline will probably come back. The problem is that the hypothesis is the only thing distinguishing an experiment from an outage, and evaluating it requires signals. A run that continues while blind has stopped being an experiment and become an unsupervised fault. There is a second, subtler failure: if a blind run that produces no alerts is recorded as a pass, the platform converts missing telemetry into false confidence, and does it silently, exactly when the observability estate is already degraded.

Decision

Inability to evaluate the hypothesis is an abort condition of the same standing as a breach. If telemetry coverage during the injection window falls below a configured completeness threshold, the run aborts and the verdict is INCONCLUSIVE. A run the platform could not observe is never recorded as passing.

How it is realised on Google Cloud

The evaluator tracks per-signal sample completeness over each 10-second window against expected cadence. Falling below the threshold stops the steady-state confirmation, which stops lease renewal, which expires the fault. Completeness is recorded in the evidence bundle so the verdict can be re-examined. The INCONCLUSIVE share is a reported metric on view 18 precisely because it is the number that tells you the platform is spending fear and returning nothing.

Options weighed

• ChosenBlindness aborts; verdict INCONCLUSIVE: Keeps the guarantee that the platform never claims health it did not observe, at the cost of runs that have to be paid for twice.
• RejectedContinue on last-known-good signals: Cheap and superficially reasonable for short gaps, and it is the precise mechanism by which a degraded metrics pipeline turns into an unsupervised fault during an incident.
• RejectedContinue, but cap the remaining duration: A compromise that keeps the worst property — an unobserved fault in production — while making the rule harder to explain. If the cap is short enough to be safe it is short enough to abort.
• DeferredFall back to a synthetic probe against the target: Genuinely useful as a second opinion and a possible way to keep a run alive through a metrics gap. Deferred because a probe measures something different from the owners' SLIs, and two sources of truth need their own decision.

Consequences

What it buys

• The platform never reports health it did not observe, which is what makes a HYPOTHESIS_HELD verdict worth anything.
• Observability degradation becomes visible through the INCONCLUSIVE rate rather than hiding inside optimistic verdicts.
• Abort has one mechanism for breach and for blindness, so there is less code that must be correct.

What it costs

• Runs are wasted: a 6% INCONCLUSIVE rate is an assumed 6% of the platform's cost and of its users' patience returned as nothing.
• The completeness threshold is a tuning parameter with no obviously right value, and setting it too tight makes the platform look flaky.
• Owners of services with thin SLI coverage are effectively blocked, which turns a platform decision into a conversation about somebody else's instrumentation.

Choose differently when

Allow a bounded continuation on last-known-good when the fault class is provably self-limiting — a single pod terminated, where the worst case is already covered by the orchestrator — and the remaining window is shorter than the abort budget. That is a narrow exception and it should be declared per fault class, never as a global default.

Why it holds up over time

This decision gets harder to hold as the platform succeeds, because the pressure comes from users who have paid the fear and want an answer. The defence is that the INCONCLUSIVE rate is published: a number everyone can see is easier to fix at the source than a rule everyone wants waived.

When somebody needs to stop everything, does the thing they press depend on the platform that is misbehaving?

Context

A kill switch inside the platform's own API is worth very little in the scenario it exists for. If the API is the thing failing, or an authorisation lookup against a degraded identity dependency is on its path, the switch is decorative. And the audience matters: the person pressing it may be an incident commander who has never used the platform, in the first minutes of an unrelated outage, who needs one certainty — that pressing it stops the injection.

Decision

The kill switch is a separate endpoint with its own IAM binding, reachable independently of the platform API, that halts all injection fleet-wide within 30 seconds. It is exercised on a schedule, and every use is treated as an event worth a conversation rather than as routine operation.

How it is realised on Google Cloud

A dedicated Cloud Run service in both regions, fronted by its own IAM policy granting a small named group, writing a single fleet-wide halt flag to the regional Spanner guardrail instance. Agents' watchdogs read the flag on their renewal path, so a halt takes effect within one renewal interval and no later than the TTL. A scheduled job presses it in staging weekly and in production quarterly, and the alert on kill-switch use is unconditional.

Options weighed

• ChosenSeparate endpoint, separate authorisation, own verification schedule: Works when the platform does not, and is simple enough that its own failure modes are enumerable.
• RejectedA flag on the platform API: One less service to run, and it shares a failure domain with the thing it is supposed to stop.
• RejectedPer-cluster kill switches only: Better than nothing and worse than useless in a game day: an incident commander has to find and press six of them while the clock runs.
• Right elsewhereDeleting the agent DaemonSet as the break-glass: A legitimate last resort that should stay documented, and unusable as the primary control because it needs cluster-admin rights the incident commander may not have.

Consequences

What it buys

• The one control that must work in the worst case does not depend on the components most likely to be failing.
• Because agents read the halt flag on the renewal path, the switch inherits the lease guarantee: worst case is the TTL, not a delivery.
• A small named group with its own binding makes the audit trail unambiguous.

What it costs

• Another service, another IAM policy, another thing that has to be kept working and rehearsed or it decays into decoration.
• A fleet-wide halt is blunt: it stops legitimate runs alongside the problem one, and there is no partial version of it by design.
• Quarterly production rehearsal is itself a small planned disruption that has to be scheduled and explained.

Choose differently when

Fold it into the API when the platform is one service and one cluster, where a separate endpoint has the same failure domain anyway. Split it out the moment there is more than one cluster or more than one team who might need to press it.

Why it holds up over time

Kill switches decay quietly: they stay in the architecture and stop working. The scheduled rehearsal is the only part of this record that keeps it true, which is why it is stated as a requirement rather than an operational nicety.

Is the blast radius what the definition says it is, or what the dependency graph says the definition reaches?

Context

"Five per cent of replicas" sounds bounded and often is not. If all five per cent of the affected pods talk to the same cache tier, degrading them can saturate that tier and affect everything behind it. If the selector resolves to a zone that also hosts a shared broker, the run is a zone experiment wearing a service costume. Enforcing only the literal selector is cheap, predictable and systematically optimistic in exactly the cases that produce incidents. Computing the transitive downstream set is honest and depends on a graph that is always somewhat wrong.

Decision

Blast radius is computed before injection as three independent measures — share of the target's healthy replicas, share of production request volume, and the set of downstream services reachable from the affected instances — derived from the dependency graph. A run whose reachable set includes a service marked a shared single point of failure is refused. The declared selector is an input to the computation, never the answer.

How it is realised on Google Cloud

The radius calculator reads the service catalogue and the mesh-derived dependency graph, both no older than 24 hours or the run is blocked, and writes the three measures plus the reachable set into the run's guardrail evaluations in Spanner. Request-volume share comes from the mesh's own traffic statistics rather than from a declared estimate. The closure is re-checked during injection so a scale-up event cannot turn a compliant 5% into a non-compliant one silently.

Options weighed

• ChosenComputed closure from the dependency graph, with a shared-SPOF veto: Catches the case that produces real incidents. Costs a dependency on graph freshness and a class of over-conservative refusals.
• RejectedEnforce the declared selector only: Predictable, cheap and wrong in the shared-tier case, which is the case worth guarding. It optimises the metric — percentage of replicas — instead of the property.
• RejectedClosure as an advisory warning, selector as the enforced limit: Tempting as a migration step and it makes the guardrail optional in practice: a warning on a page nobody re-reads is not a limit.
• Right elsewhereManually declared downstream impact per definition: Right for a small system where the author genuinely knows the graph. At 900 services the author's mental model of the dependency graph is the thing being tested.

Consequences

What it buys

• The guardrail measures the property that matters — how far the failure can actually reach — rather than a proxy for it.
• The shared-SPOF veto turns the most dangerous class of experiment into an explicit refusal instead of a surprise.
• Request-volume share read from the mesh means the second measure cannot be talked down by an optimistic estimate.

What it costs

• The design now depends on graph freshness, and a stale graph blocks runs rather than degrading, which feels like the platform being broken.
• Over-conservative closures refuse legitimate runs, and the failure mode is quiet: teams route around the platform rather than complaining.
• Re-checking during injection adds a hot read on a path where latency matters.

Choose differently when

Enforce the declared selector when the estate has no shared tiers worth protecting, or when the dependency graph is unreliable enough that closures are mostly wrong — an inaccurate closure that refuses good runs is worse than an honest selector plus a careful author. Say which it is in the requirement rather than letting it be an implementation detail.

Why it holds up over time

The decision survives changes in how the graph is produced — mesh telemetry today, eBPF or distributed tracing later — because the calculator consumes a graph rather than a source. The risk is drift in what "shared single point of failure" means; that marking belongs in the service catalogue, owned by the people who know, not inferred here.

What happens when two experiments would affect overlapping parts of the fleet at the same time?

Context

Two individually compliant runs can be jointly catastrophic: 5% of one service plus 5% of a shared downstream is not 5% of anything. The problem is not detecting the overlap — the closures are already computed — but deciding what to do. Rejecting the second run trains people to retry in a loop, and at game-day concurrency the retry storm is on the platform itself. Running both and hoping is how a game day becomes an incident.

Decision

A permitted run holds a reservation over its computed radius for its declared duration. A run whose radius intersects a live reservation queues until the reservation is released, rather than being rejected. Conflicts are surfaced at plan time for game days, so a sequence is corrected before the session rather than during it.

How it is realised on Google Cloud

Reservations are rows in the regional Spanner guardrail instance, written in the same transaction that permits the run and released on terminal state or lease expiry. The radius calculator takes the reservation check as part of its evaluation, so intersecting runs never reach the lease issuer. The game-day sequencer runs the same check over a whole plan at authoring time and reports collisions as a plan error.

Options weighed

• ChosenReservation with queueing, plan-time conflict detection: Makes concurrency safe without teaching users to retry, and moves game-day collisions from the session to the plan.
• RejectedReject the intersecting run: Simplest to implement and it exports the problem to the caller, who implements a retry loop badly. At 200 concurrent runs that loop is a load problem on the guardrail store.
• RejectedRun both and rely on each run's own abort conditions: Each run's guardrails are sound individually and say nothing about the sum. This is the specific mechanism by which two safe experiments produce one outage.
• Right elsewhereA global concurrency limit instead of per-radius exclusion: A reasonable first version for a small platform, and far too blunt here: it would serialise unrelated experiments across a 900-service estate to guard a rare intersection.

Consequences

What it buys

• Concurrency is safe by construction rather than by convention, and the safety does not depend on users coordinating.
• Queueing keeps the contract simple: a permitted run will happen, just possibly later.
• Plan-time detection removes the worst game-day failure, which is rewriting the sequence live under time pressure.

What it costs

• The reservation store becomes the contended resource at high concurrency — which is the intended place for contention, and still a bottleneck to design for.
• A queued run can miss its window and be skipped, which users experience as the platform silently not running their experiment.
• A long-held reservation from a stuck run blocks others until its lease expires, so reservation lifetime is tied to the TTL rather than to the run's declared duration.

Choose differently when

Reject instead of queueing when experiments are cheap to re-issue and the caller is a pipeline that handles backoff properly. Queueing earns its complexity when a human is waiting and would otherwise sit in a retry loop.

Why it holds up over time

Reservation is a durable idea; the implementation will move as the store does. The pressure on it comes from scale: if concurrency grows an order of magnitude, radius intersection testing becomes the hot path and will need an index rather than a scan.

Does deliberate failure cost anything, and if so, whose budget does it come out of?

Context

If experiments are free, the only limit on them is politeness, and the argument about how much chaos is too much has no common currency. If they are charged to a separate allowance invented for the platform, the number is arbitrary and nobody believes it. The service's own SLO error budget is the one quantity the owners already argue about in terms they accept, which makes it the honest place to bill — and immediately raises the question of who owns the decision to spend it.

Decision

Each service reserves a share of its monthly SLO error budget for chaos — 10% as a stated assumption — debited when a lease is issued and settled to actual spend at abort or completion. A service that has exhausted its chaos share is refused further runs until the budget resets or its owner raises the share. The owner controls the share; the platform controls the accounting.

How it is realised on Google Cloud

The budget ledger is a Spanner table keyed by service and month, read from the SLO store's budget at the start of each period. Debits are estimated from the declared radius and duration at lease issue and reconciled against observed SLI impact when the run ends, so an aborted run pays only for what it spent. Consumption is reported per service and per team on view 18.

Options weighed

• ChosenCarve-out from the service's own SLO error budget, owner sets the share: Bills in a currency the owners already accept, and puts the ceiling in the hands of the people who carry the consequences.
• RejectedUnlimited experiments with rate limits only: Rate limits bound frequency, not harm. Twenty small runs can spend more budget than one large one and no rate limit notices.
• RejectedA separate chaos allowance unconnected to the SLO: Easy to implement and impossible to defend: the number has no meaning, so every conversation about it is a negotiation from first principles.
• RejectedCentral reliability team owns the chaos budget: Removes the owner's veto, which is the fastest way to make chaos something done to teams rather than by them. Findings nobody feels responsible for do not get fixed.

Consequences

What it buys

• The cost of learning is expressed in the same unit as the cost of failing, so the trade-off can be argued rather than asserted.
• A service under SLO pressure automatically gets less chaos, which is the right behaviour and needs no human to enforce it.
• Owner-set shares mean a team can opt into more testing, which is a much better incentive than a platform-imposed cap.

What it costs

• Estimating a run's budget cost before it happens is genuinely hard, and over-estimating blocks legitimate runs.
• A service with no SLO cannot be charged, so it either cannot be tested or needs an exception path — and exception paths erode.
• An owner can set the share to zero and never test anything, and the platform has no answer to that beyond the coverage report.

Choose differently when

Skip budget accounting entirely while the platform is small and every run is attended: the human watching is the budget. Introduce it when unattended and continuous runs make the aggregate spend invisible, which is the point at which "it is only 1%" stops being checkable.

Why it holds up over time

This is the most organisational decision in the package and the one most likely to be renegotiated. The mechanism — debit, settle, refuse at exhaustion — survives; the 10% will not, and it is stated as an assumption so that changing it is a configuration change rather than a redesign.

During a real production incident, who decides whether experiments keep running?

Context

The decision is easy in the abstract and terrible in practice, because the person best placed to make it is the one with the least attention to spare. An incident commander in the first five minutes of an outage should not have to work out which of 200 in-flight experiments might be contributing, nor should a service owner watching their own run be asked to judge whether somebody else's incident is related. Leaving it to humans means the answer arrives late, and late is the only time it matters.

Decision

An open incident of configured severity suspends new injection automatically, and aborts any in-flight run whose target or downstream reachable set intersects the incident's affected services. No human confirmation is required to suspend; human judgement is required only to resume.

How it is realised on Google Cloud

The platform subscribes to the incident platform's event stream through Pub/Sub and maintains a suspension flag and an affected-service set in the Spanner guardrail instance. The incident gate is evaluated at lease issue and on every renewal, so an intersecting in-flight run stops being renewed and expires within the TTL. Resumption is an explicit action, recorded in the audit log with the principal who took it.

Options weighed

• ChosenAutomatic suspension on incident open; explicit human resume: Asymmetric by design: stopping is free and automatic, restarting requires a person. The expensive mistake is only in one direction.
• RejectedNotify the facilitator and let them decide: Puts the decision with the person who has the least attention available, at the moment their attention is most contested.
• RejectedSuspend everything fleet-wide on any incident: Safer on paper and it makes the platform unusable: at 900 services some incident is always open somewhere, and a platform that never runs teaches nothing.
• DeferredIntersect on the incident's declared services only, not the closure: Cheaper and less disruptive, and it misses the case where the experiment's downstream set is what is contributing. Deferred pending evidence on how often closures are right.

Consequences

What it buys

• The decision is made in milliseconds by a rule rather than in minutes by a person who is busy.
• The asymmetry is honest: an unnecessary suspension costs a re-run, a missed suspension costs an outage lengthened by a self-inflicted fault.
• Because the gate is evaluated on the renewal path, suspension inherits the lease guarantee rather than needing its own delivery.

What it costs

• Two automatic suspensions a day is assumed, which makes interrupted runs ordinary and forces scheduling to treat skipping as normal.
• The platform now depends on the incident platform's severity taxonomy being meaningful, and on its affected-service field being filled in.
• A mis-scoped incident record can suspend a large amount of unrelated work, and the platform has no way to tell.

Choose differently when

Require human confirmation when incidents are declared loosely enough that most of them are noise, because a gate that fires constantly gets disabled. Fix the taxonomy first if you can; a veto based on an unreliable signal is worse than no veto, since it teaches people to work around it.

Why it holds up over time

The mechanism is stable. What will move is the severity threshold and the intersection rule, both of which are configuration. The real risk is the platform being blamed for interrupted runs and the threshold being quietly raised until the gate stops firing — which is why suspension counts are reported.

How does a new kind of failure get added, and what stops one being added that cannot be undone?

Context

A chaos platform accretes fault types, and the pressure to add them comes with a deadline attached. If each is a script, reversibility is a property of whoever wrote it that afternoon, and the platform's central guarantee degrades one commit at a time. Worse, the most requested faults are often the least reversible — truncate this table, corrupt this message, expire this certificate — and without an explicit contract those requests get accommodated quietly rather than refused.

Decision

Every fault class is an adapter implementing inject, verify, revert and report_state, declaring the capabilities it needs. The control plane refuses a fault the target environment cannot reverse, and an adapter whose revert has not been exercised successfully against that service in a non-production environment within 30 days is ineligible for production.

How it is realised on Google Cloud

Adapters are Go plugins compiled into the agent binary with a declared capability manifest, so the set of adapters in a cluster is a property of a signed image rather than of runtime configuration. The revert-proof suite in Cloud Build exercises every adapter's inject/revert cycle a thousand times per release and fails the build on any unconfirmed revert. Non-production proving windows are tracked per service and per class in Spanner and enforced at validation.

Options weighed

• ChosenCapability-declaring adapter with a mandatory, tested revert: Contains adapter churn away from the control plane and makes an irreversible fault an explicit refusal. Costs real effort per new fault class.
• RejectedScripts invoked by the agent: Fastest way to add a fault and the fastest way to lose the fail-closed guarantee, because nothing forces the script to have a revert worth the name.
• Right elsewhereAdopt a third-party chaos operator's fault library wholesale: Right for a team getting started, and it imports someone else's reversibility assumptions along with their coverage. Worth revisiting once the contract is stable enough to wrap them.
• RejectedAllow declared-irreversible faults with extra approval: This is the request that will keep arriving. Accepting it puts the first hole in the guarantee every other record depends on, and one hole is enough.

Consequences

What it buys

• Adding a fault class never touches the control plane, so the blast radius of a new adapter is one image.
• report_state makes drift detectable: the platform can tell the difference between a fault it applied and a fault it merely asked for.
• The 30-day proving window turns reversibility from a claim into a recent observation.

What it costs

• Four methods and a capability manifest is a real barrier to a five-line fault, and some legitimate experiments will not be worth the effort.
• Compiling adapters into the agent means a new fault class requires an agent release, which is the highest-risk release the platform has.
• Genuinely useful irreversible faults are refused, and the platform has to say no to them repeatedly.

Choose differently when

Allow scripted faults in non-production only, where reversibility is a convenience rather than a guarantee and the fast path is worth more than the contract. The line between environments should be enforced by the platform, not by intent.

Why it holds up over time

The contract is the durable part and it is technology-independent: inject, verify, revert, report_state would be the same four verbs on any platform. What will churn is the mechanism inside each adapter — eBPF displacing tc, mesh features arriving and leaving — which is exactly what the contract was drawn to contain.

At which layer is a fault created — the proxy, the kernel, or the application's own code?

Context

The three layers differ in fidelity, privilege and adoption cost, and they are not substitutes. A mesh fault filter is portable, application-transparent and trivially reversible, but it only sees what the proxy sees: it cannot express a slow disk, a corrupted deserialisation or a connection pool exhausted inside the process. Kernel-level manipulation through traffic control or eBPF has much higher fidelity and needs privilege that is uncomfortable to hold in production. In-application libraries express dependency semantics best of all and require every one of 900 services to adopt them, which means the coverage report would measure adoption rather than resilience.

Decision

Injection happens at the mesh layer where the proxy can express the fault, and at the node layer through traffic control and cgroup limits where it cannot. In-application fault libraries are deferred: they are the right answer for a handful of specific semantics and the wrong basis for a platform that must cover a fleet it does not control.

How it is realised on Google Cloud

The agent configures mesh fault filters through the mesh's own control API for HTTP-level latency, error status and body faults, and applies tc/netem qdiscs and cgroup limits directly for packet-level, resource and node-level faults. Privileged classes — netem, node actions — require a distinct authorisation in the lease claim, so the ordinary case does not carry the dangerous capability.

Options weighed

• ChosenMesh filters plus node-level tc and cgroups: Covers six fault classes with no application change, at the price of holding node-level privilege and of some faults being inexpressible.
• RejectedMesh only: Lowest privilege and genuinely attractive, and it cannot express resource pressure, node loss or clock skew — three of the six classes the requirement names.
• DeferredIn-application fault libraries: Highest fidelity for dependency semantics and the only way to test a connection pool honestly. Deferred because platform coverage would become a function of library adoption across 900 services.
• DeferredeBPF for everything: Better fidelity than tc with a cleaner reversal story, and a kernel-version dependency across six clusters that the adapter contract should absorb rather than the platform committing to now.

Consequences

What it buys

• No service has to adopt anything, so coverage measures resilience rather than library uptake.
• Mesh-level faults are reversible by removing a configuration, which is the cheapest possible revert.
• Splitting privileged classes into a separate authorisation keeps the common case low-privilege.

What it costs

• The agent holds node-level privilege in production, which is the largest standing capability in the estate and needs its own review cadence.
• Some genuinely important failures — in-process pool exhaustion, deserialisation faults — cannot be expressed at all, and the coverage report has to admit that.
• Two mechanisms means two reversal paths, and tc rules are the ones that get stuck.

Choose differently when

Go application-library-first when the estate is small, homogeneous and owned by one team — there the adoption cost is a day and the fidelity is worth it. Go mesh-only when policy will not permit node-level privilege in production, and accept that three fault classes are then out of scope rather than pretending otherwise.

Why it holds up over time

The layering argument is stable; the specific tools are not. eBPF is the likely successor to tc within the decade and the adapter contract exists so that is a swap rather than a redesign. The decision that will age is deferring application libraries — as service frameworks gain fault-injection hooks by default, the adoption argument weakens.

Is a fault gone because the revert call returned, or because the platform looked?

Context

The gap between "revert returned successfully" and "the fault is gone" is where the worst outcomes live. A tc qdisc can survive the process that created it. A mesh filter can be removed from the control plane and remain in a proxy that has lost its connection. A cgroup limit can persist on a node whose agent has been rescheduled. In every one of these cases the platform's own records say the experiment ended cleanly, and production is still degraded, and nobody is looking for a fault because the run is closed.

Decision

After revert the agent re-reads target state and confirms absence. If the fault is still present after a bounded retry sequence, the out-of-band reverter is invoked, on-call is paged, and that fault class is suspended fleet-wide pending review. A post-abort settling check confirms the steady-state signals returned to baseline; a run whose signals did not recover is recorded as an unresolved finding rather than a completed experiment.

How it is realised on Google Cloud

report_state on each adapter is the read path, so verification uses the same interface as injection rather than a parallel implementation. Retries are bounded at three attempts over ten seconds before escalation. The out-of-band reverter runs in the same namespace under a break-glass service account, so its use is both possible and abnormal. Verification outcomes are recorded per applied fault, and the fleet-wide suspension is a flag on the same guardrail store the lease issuer reads.

Options weighed

• ChosenVerify by re-reading state; unconfirmed revert escalates and suspends the class: Closes the gap between the record and reality. Costs a read path per adapter and an escalation that will sometimes be noise.
• RejectedTrust the revert's return value: Cheapest and it is the exact mechanism by which a stuck fault becomes an unexplained production degradation with a closed run record behind it.
• RejectedVerify by watching the SLIs recover: Necessary as the settling check and insufficient as verification: signals can recover while the mechanism remains, waiting for the next burst of traffic to matter.
• DeferredPeriodic fleet-wide sweep for orphaned faults: A good complement that catches what per-run verification misses — faults from runs whose records were lost. Deferred because it needs an inventory of what a clean node looks like.

Consequences

What it buys

• The platform's record and production's state cannot diverge silently, which is the property every safety claim in this package rests on.
• Suspending the fault class fleet-wide turns one stuck revert into a bounded problem instead of a repeated one.
• The settling check catches the case where the fault is gone and the service has not actually recovered — which is a finding, not a clean run.

What it costs

• Every adapter needs a meaningful read path, which roughly doubles the work of adding a fault class.
• An assumed one unconfirmed revert per 900 injected faults means real pages for an on-call rotation, some of which will be verification bugs rather than stuck faults.
• Fleet-wide suspension of a class is disruptive: one bad node can stop that fault class everywhere until somebody reviews it.

Choose differently when

Trust the return value in non-production, where a stuck fault costs a cluster rebuild rather than customer impact. In production there is no version of this that is safe to skip, because the whole design assumes the fault is gone.

Why it holds up over time

This is a discipline decision and disciplines erode under alert fatigue. The pressure will be to downgrade the page to a ticket after the third false positive. The defence is that verification failures are counted rather than rated — a single occurrence is visible — and that the suspension makes ignoring it expensive.

Does a production experiment degrade requests that real customers made, or a copy of them?

Context

There are three honest answers and the safest one is the least useful. Shadow traffic — mirroring real requests to a parallel pool — protects customers completely and systematically misses the failures worth finding, because retry storms, cache stampedes and queue backpressure are properties of correlated load meeting a shared resource, and a shadow pool has neither the correlation nor the sharing. Synthetic load against a canary pool is better still for safety and worse still for fidelity. Real traffic on a bounded slice finds the emergent behaviour and spends a small amount of customer experience to do it.

Decision

Production experiments affect real customer traffic on a slice bounded by the computed blast radius and paid for from the chaos error budget. Shadow and synthetic modes remain available for adapter proving and for fault classes whose fidelity does not depend on correlated load.

How it is realised on Google Cloud

The slice is the resolved target set: the mesh routes a bounded share of live requests through faulted proxies, and the share is one of the three computed radius measures. Request-volume share comes from the mesh's traffic statistics rather than from an estimate, so the 1% cap is measured. Budget debits at lease issue make the customer cost explicit in the currency the owner already argues about.

Options weighed

• ChosenReal traffic, bounded slice, budget-debited: The only mode that reproduces emergent load behaviour. Costs real customer experience, made explicit and capped rather than hidden.
• RejectedShadow traffic to a parallel pool: Zero customer cost and it misses retry storms, stampedes and backpressure — precisely the class of failure that turns a degradation into an outage.
• Right elsewhereSynthetic load against a dedicated canary pool: Right for proving a new adapter and for pre-promotion gating, where the question is "does the fault work" rather than "does the service survive it".
• RejectedReal traffic with no volume cap, radius by replicas only: Replica share and request share diverge badly under uneven routing: 5% of replicas can be 30% of traffic, and the cap that matters to a customer is the second one.

Consequences

What it buys

• Findings are about how the system behaves under real correlated load, which is the only behaviour that predicts an outage.
• Capping on measured request-volume share rather than replica share puts the limit where the customer actually is.
• The cost is explicit, budgeted and refusable, which makes the trade-off arguable instead of implicit.

What it costs

• Real customers are affected, by design. An assumed 1% cap on a 140-million-subscriber service is still a large number of people on a busy evening.
• Regulatory or contractual constraints may make some services ineligible, and the platform needs an honest exclusion list rather than a workaround.
• The blast radius computation becomes safety-critical: an error there is an error in how many customers are affected.

Choose differently when

Use shadow traffic when the service's failure is customer-visible and unrecoverable within one request — a payment authorisation, a safety interlock — where no share of real traffic is an acceptable price. Say so per service in the catalogue rather than deciding it per run.

Why it holds up over time

The reasoning is independent of any platform and will not change: correlated load is what produces emergent failure. What may change is the cap, as the service's own resilience improves and the same 1% costs less. That is a configuration change and it is stated as an assumption for exactly that reason.

Does the platform reuse the production SLO pipeline to judge the hypothesis, or build a faster path of its own?

Context

This is the sharpest trade-off in the design. Reusing the SLO pipeline gives one definition of truth, shared with the people who are paged, and an aggregation lag measured in minutes — against an abort budget of ten seconds. Building a dedicated high-resolution path gives a breach visible in seconds and creates a second, potentially divergent computation of the same SLI, which means the platform can say "healthy" while the SLO says "breached" and neither is obviously wrong.

Decision

A dedicated high-resolution evaluation path reads the signals underlying the target's own SLIs and computes them on 10-second rolling windows, windowed to the baseline and injection periods only. The SLI definitions come from the SLO store; the platform does not invent its own. Divergence between the two computations is treated as a bug in the platform, not as an acceptable difference.

How it is realised on Google Cloud

The signal collector reads Cloud Monitoring time series at the highest resolution the underlying metrics support, only for signals the hypothesis names and only inside the windows, which is what keeps the cost bounded. The evaluator recomputes the SLI using the definition pulled from the SLO store rather than a local copy. Baseline and injection windows are stored in BigQuery so a verdict can be re-derived and compared against the SLO pipeline after the fact.

Options weighed

• ChosenDedicated high-resolution path, SLI definitions from the SLO store: Makes the 10-second abort budget achievable while keeping one source for what the SLI means. Costs a second computation that has to be kept honest.
• RejectedReuse the production SLO pipeline as-is: One truth and no divergence risk, and an aggregation lag that makes the abort budget unachievable. The abort budget is the safety property; it wins.
• RejectedPlatform-specific health metrics: Fastest to build and it decouples the verdict from what customers feel, so a passing experiment stops meaning anything to the owners.
• DeferredSynthetic probes as the steady-state signal: Useful where SLI coverage is thin and it measures a different thing than production traffic. Deferred with ADR-03's fallback question.

Consequences

What it buys

• Breach to fault-cleared fits inside ten seconds, which is what makes every safety claim in the package achievable.
• Because SLI definitions are pulled rather than copied, a change by the owners reaches the platform without a platform change.
• Windowing collection to baseline and injection is what keeps the second path affordable — it is the dominant cost lever in the design.

What it costs

• Two computations of the same SLI exist, and reconciling them is ongoing work rather than a one-off.
• Services whose SLIs are computed from logs rather than metrics cannot be evaluated at this resolution, and are effectively blocked.
• 500,000 samples a second during a large game day is a real ingestion cost on top of the existing monitoring bill.

Choose differently when

Reuse the SLO pipeline when the abort budget can be minutes — a background batch service, a nightly pipeline — where the fault's cost accrues slowly and one truth is worth more than speed. That is a per-service property and could be declared as such rather than decided globally.

Why it holds up over time

The trade-off is structural and will outlive Cloud Monitoring: someone will always want the fast path and the shared definition, and cannot have both. The pressure over time is drift, as the platform's recomputation acquires convenience shortcuts. Publishing both numbers for a sample of runs is the cheapest defence.

Does "no SLO breach during the injection window" mean the mitigation worked?

Context

It very often does not. A dependency timeout can pass without breaching anything because the traffic that would have exercised the fallback did not arrive, or because a cache was warm, or because the retry budget was never approached. Recording that as a pass teaches the owner that their fallback works when what was actually established is that nothing bad happened this time. The stronger form — assert that the circuit breaker opened, the fallback served, the retry budget held — is harder to express, requires the mitigation to be observable at all, and is the only version that answers the question the owner asked.

Decision

Where a mitigation can be observed, the hypothesis must assert it positively and the verdict depends on that assertion as well as on the absence of a breach. Where it genuinely cannot be observed, the definition must say so and the resulting pass is recorded as weaker evidence in the coverage report.

How it is realised on Google Cloud

A hypothesis carries two kinds of clause: steady-state thresholds on the owners' SLIs, and positive assertions over named signals such as a breaker-state metric, a fallback-served counter or a retry-budget gauge. The verdict engine requires every positive assertion to fire before recording HYPOTHESIS_HELD. Coverage reporting distinguishes a pass with positive confirmation from a pass without one, so the untested set is honest about what kind of evidence it holds.

Options weighed

• ChosenPositive assertion required where observable; graded evidence where not: Answers the question the owner actually asked. Costs instrumentation the service may not have, and a harder definition to write.
• RejectedNo SLO breach is a pass: Trivially easy to evaluate and it systematically over-reports confidence, because the common reason nothing broke is that the mitigation was never reached.
• RejectedRequire a positive assertion always, no exceptions: The honest ideal, and it would block most services on day one. Graded evidence is the compromise that keeps the platform usable while keeping the report truthful.
• RejectedManual review of each run's signals: The most accurate option and it does not scale past a few runs a week, which is far below the assumed 2,000.

Consequences

What it buys

• A pass means a mitigation was seen working, which is the claim the owner wanted and the one worth recording for two years.
• Grading evidence makes the coverage report honest: it can distinguish proven from merely undisturbed.
• It creates a concrete, useful instrumentation ask — expose breaker state, fallback counts, retry budget — instead of a vague one.

What it costs

• Services without mitigation instrumentation can only earn the weaker pass, which turns a platform decision into a conversation about someone else's metrics.
• Definitions are harder to write, which shows up as the assumed 22% validation rejection rate.
• A positive assertion can be wrong in the other direction: a breaker that opened for an unrelated reason satisfies the clause without proving anything.

Choose differently when

Accept absence-of-breach as a pass for fault classes where the mitigation is the orchestrator's rather than the service's — a single pod terminated, where the assertion is that the replica set recovered and that is already visible. The distinction is per fault class, not per service.

Why it holds up over time

This is the decision most likely to be watered down, because it makes the platform harder to use and the weaker version still produces green results. The defence is the graded coverage report: as long as the two kinds of pass are counted separately, the erosion is visible.

Where is a lease checked — at the control plane that issued it, or at the agent that will act on it?

Context

If the agent simply does what the control plane tells it, then compromising the control plane compromises the fleet: an attacker who can call the injection path can widen a blast radius arbitrarily, and the agent has no basis to refuse. If the agent verifies, the control plane's compromise is bounded by what a valid signed claim can express. The related question is what the agent's own identity is worth: a credential present in 40,000 pods is a credential that will eventually leave them.

Decision

Each agent holds a short-lived, cluster-scoped workload identity, and verifies the lease's signature and scope itself before acting. A lease naming a target outside the agent's cluster, a fault class it is not authorised for, or an expiry in the past is refused in the data plane. The agent can verify but cannot mint.

How it is realised on Google Cloud

GKE Workload Identity gives each cluster's agent a distinct, non-exfiltratable identity. The lease issuer signs claims with a Cloud KMS asymmetric key; agents hold only the public key, so a compromised agent cannot issue a lease for anything. Privileged fault classes require a distinct authorisation inside the claim, so an ordinary lease does not carry the dangerous capability even if it is replayed.

Options weighed

• ChosenSigned claim, verified in the agent, cluster-scoped workload identity: Bounds a control-plane compromise to what a valid claim can say. Costs key management and a verification path that must not become a rubber stamp.
• RejectedAgent trusts an authenticated call from the control plane: Simpler and it makes the control plane a single point of total fleet compromise, with no second opinion anywhere.
• RejectedShared secret distributed to agents: Removes the KMS dependency and turns every agent into a minting authority. A secret in 40,000 pods is not a secret.
• DeferredPer-node rather than per-cluster identity: Tighter scoping and a real operational cost across 40,000 pods. Deferred until there is evidence that cluster scope is too coarse in practice.

Consequences

What it buys

• A compromised control plane cannot widen a blast radius beyond what a valid, signed, scoped claim expresses.
• Agents cannot self-authorise, so the dangerous capability lives only where it is issued.
• Cluster-scoped identity means a compromised cluster cannot request faults in another one.

What it costs

• Key rotation becomes a fleet-wide coordination problem with a verification path that must tolerate two valid keys.
• Clock skew between issuer and agent becomes a correctness concern for expiry checks, not just a nuisance.
• Verification is easy to hollow out over time — a skipped scope check during an incident is a plausible and invisible regression.

Choose differently when

Let the agent trust the control plane when both are inside one strongly isolated trust domain and the threat model does not include control-plane compromise — a single-team platform in a single project. At fleet scale the second opinion is worth its cost.

Why it holds up over time

Signed, scoped, short-lived claims verified by the actor is a pattern that long predates this platform and will outlast it. What will change is the key custody mechanism. The risk to watch is verification decay rather than cryptographic obsolescence, which is why scope refusals are counted on view 18.

What stops a resilience platform's telemetry bill from exceeding the value of the findings it produces?

Context

The dedicated evaluation path in ADR-13 is the platform's largest variable cost, and the naive implementation — collect everything at high resolution, all the time, across the fleet — would be more expensive than the incidents it prevents. The second cost is subtler: if experiments appear free to the team running them, a team will schedule a thousand low-value runs and nobody will notice until the bill arrives centrally, where it looks like a platform problem rather than a usage one.

Decision

High-resolution signals are collected only for the signals a hypothesis names, and only inside the baseline and injection windows. Runs terminate early once the hypothesis is decided with sufficient confidence. Series are downsampled after 90 days. All cost is attributed per service and per team, and the platform's total is held to 0.4% of fleet production compute spend.

How it is realised on Google Cloud

The collector subscribes per run rather than per service, so the ingestion rate is a function of concurrency and not of fleet size. BigQuery partition expiry implements the 90-day boundary as configuration, with downsampled summaries retained for the life of the run record. Cost attribution joins run records to signal volume and storage in BigQuery and is reported alongside coverage on view 18, next to the incidents the findings prevented.

Options weighed

• ChosenWindowed collection, early termination, downsampling, per-team attribution: Ties cost to concurrency rather than fleet size and puts the bill where the usage decision is made.
• RejectedContinuous fleet-wide high-resolution collection: Much simpler and it makes ingestion a function of fleet size, which is the one variable the platform cannot influence.
• RejectedReuse existing monitoring resolution only: Nearly free and it gives up the ten-second abort budget, which is the safety property the whole design is built on.
• RejectedCentral budget with no attribution: Administratively easy and it hides the usage decision from the team making it, so the only available control is a cap that punishes everyone.

Consequences

What it buys

• Ingestion scales with concurrency, so adding services to the fleet does not raise the telemetry bill.
• Per-team attribution makes the cost of a thousand low-value scheduled runs visible to the team that scheduled them.
• Reporting platform cost against prevented-incident cost makes the funding argument checkable rather than rhetorical.

What it costs

• Windowed collection means a signal not named in the hypothesis is not available afterwards, so a run cannot be re-analysed for something nobody thought to ask.
• Early termination trades a little statistical confidence for cost, and the confidence threshold is another parameter with no obviously right value.
• Downsampling after 90 days means old verdicts cannot be re-derived at the resolution that produced them, only checked against their stored summaries.

Choose differently when

Collect continuously at high resolution when the fleet is small enough that the bill is noise, because the ability to re-analyse any run for anything is genuinely valuable. Windowing earns its constraint at the point where ingestion would scale with fleet size rather than with usage.

Why it holds up over time

The structural insight — make cost a function of usage, not of estate size — is durable and product-independent. The 0.4% figure is not, and is stated as an assumption. The pressure over time will be requests to widen collection so runs can be re-analysed, which is a real need better met by naming more signals in the hypothesis.