Burn-Rate Abort

A team runs its first production chaos experiment in one cell, with an abort set to trigger at 50 errors a minute. The cell is small, so 50 errors a minute is a rate it reaches on a quiet Tuesday without any experiment. Either the abort trips constantly and the programme is abandoned, or the threshold is raised until it would not trip on a genuine regression.

A fixed error count is not comparable across traffic volumes, across cells or across times of day. Burn rate is: it expresses errors as a fraction of the error budget being consumed per unit of time, so the same threshold means the same thing in a cell serving 50 requests a second and one serving 5,000.

Why it matters

The abort is the mechanism that makes a production experiment approvable. Without a credible stop condition, the only honest answer to "what if this goes wrong" is "we will notice and react", which is not a control.

The mechanism matters for a second reason: burn rate has a time dimension, so it distinguishes a brief spike from sustained damage. A one-second blip consumes a negligible fraction of a monthly budget; the same error rate sustained for 10 minutes consumes a large one. A count-based threshold cannot tell those apart and will either over-trigger or under-trigger.

And it ties the experiment to the same currency as the rest of the reliability programme. The budget is the thing the organisation already agreed to spend, so an experiment that stops at a defined burn rate is spending an agreed amount rather than an unbounded one.

Implementation patterns

• Compute burn rate over a short window — a few minutes — against the journey SLO's budget. A multiple of the normal consumption rate (for instance, 10 times) is the usual threshold form.
• Use two windows. A short, fast one to catch a sharp regression, and a longer one to catch a slow leak the short window would miss. This is the standard multi-window burn-rate alerting shape, applied to an abort rather than to a page.
• The abort must not depend on the component being degraded. A switch served by the service under test is no switch. Keep it on an independent path, with its own credentials and its own route.
• Verify the abort before relying on it, by deliberately triggering it. An untriggered abort is a configuration, not a capability.
• Abort means restore, not just stop injecting. Define the return-to-known-state action — route the cell's traffic away, roll the configuration back, re-enable the fallback.
• Record every abort as a finding. An abort is the experiment working, not the experiment failing, and the reason it fired is the result.

Industry example

Multi-window, multi-burn-rate alerting is the approach documented in Google's SRE workbook and widely adopted since, precisely because fixed thresholds do not survive varying traffic. The same arithmetic transfers directly to automated canary analysis: Netflix's published work on Spinnaker's automated canary analysis compares a canary cohort against a control statistically rather than against absolute thresholds, for the same reason — an absolute number is not meaningful without the denominator.

The pattern's origin in experiment control rather than alerting is what makes it useful here: the budget supplies a pre-agreed amount of harm the organisation has already accepted, so the abort threshold is a translation rather than a negotiation.

Failure scenarios

• An abort that depends on the failing component, so the experiment cannot be stopped — the same circular dependency that makes real incidents long.
• A single long window, which lets a sharp regression run for minutes before the average moves.
• A single short window, which trips on normal noise in a low-traffic cell.
• An abort that stops injection without restoring state, leaving the system in a degraded configuration nobody is watching.
• A threshold tuned until it never fires, which is how a programme keeps its approval and loses its control.
• No budget to burn. If the journey SLO is not measured, there is no denominator and the pattern cannot be applied at all.

Trade-offs

Burn-rate aborts need a measured journey SLO, a short-window SLI computation, and an independent control path. That is real platform work, and teams without journey SLOs discover that this is the prerequisite rather than the feature.

The abort also costs experiment yield: an experiment that stops early produces a partial result, and an organisation that aborts often will learn more slowly than one that lets experiments run. That is the correct trade for a production experiment and the wrong one for staging, where there is no budget to protect.

When not to use it

In a lower environment. There is no user-visible budget to burn, so the abort should be a simple error or health threshold and the machinery is unnecessary.

And where no journey SLO exists. The pattern depends on a measured budget; without one, use an absolute threshold calibrated from the cell's own baseline, and treat building the SLO as the next piece of work. Prefer a crude baseline-relative threshold to a borrowed absolute number, which is the version that fails in both directions.

Interview question

Q: You are asked to approve the first production fault-injection experiment on a payments path. Describe the stop condition you would require, and how you would convince yourself it works.

What a strong answer covers: requiring a measured journey SLO first, since the budget is the denominator · a multi-window burn-rate threshold, short window for sharp regressions and longer for slow ones, expressed as a multiple of normal consumption · the independence requirement, that the abort path must not traverse the component under test · verifying it by deliberately triggering it before the experiment · defining abort as restore-to-known-state rather than stop-injecting · limiting the experiment to one cell so the affected population is nameable in advance · and the honest caveat that frequent aborts slow learning, which is the correct price in production.

Quick check

Quiz: Why is a fixed error-count abort threshold unusable across cells? — Because the same count is normal noise in a small cell and a serious regression in a large one; burn rate normalises by the error budget and adds a time dimension.

Flashcard: What two properties must a chaos experiment's abort have? — It must not depend on the component being degraded, and it must have been deliberately triggered at least once. An untriggered abort is a configuration rather than a capability.