Shortly before 10:00 UTC on 8 June 2021 a Fastly customer pushed a valid configuration change and about 85% of the network began returning errors. The bug had shipped on 12 May and lay dormant. Detection took about a minute and 95% of the network was recovered within 49 minutes. What class of failure is this?

The trigger

A software deployment in May introduced a defect that could only be reached by a particular configuration shape. No customer had that shape, so the deployment was correct under every configuration that existed when it shipped. Weeks later a customer made a valid change that contained it, and the dormant path executed globally.

Fastly's own summary is explicit that the customer was blameless: the change was within what the platform documented as permitted.

Why this is its own class

The code was deployed by you; the arming event was performed by a customer, on their schedule. That breaks the assumption underneath most release safety: that risk is concentrated at deploy time and decays afterwards. Here the risk was created at deploy time and realised weeks later by an external actor.

It follows that canarying the deployment proves nothing about this class, because the triggering input did not exist during the canary. The configuration space of a programmable platform is combinatorial, so it cannot be enumerated in testing either. "Test more" is not the lesson.

What the lesson actually is

1. Treat customer configuration as a program, and its rollout as a deployment. Compile and activate configuration progressively — a few points of presence first, then a region, then globally — with automatic abort on error ratio. The customer's change becomes a staged rollout even though the customer did not ask for one.
2. Property-test and fuzz the configuration surface, since you cannot enumerate it. Generate valid-but-unusual configurations against the evaluator continuously.
3. Make detection behavioural, not deploy-linked. Nothing had been deployed that morning, so an alert keyed to releases would have been silent. Fastly saw it in about a minute because the signal was the network's own error ratio.
4. Invest in the reverse. 95% recovery in 49 minutes is a strong number, and it is what turned a catastrophic failure into a bad hour. The same property that made the blast global — config propagates everywhere in seconds — made the mitigation global.

The general principle

On any platform that runs customer input, your blast radius is the union of every customer's next action. The controls that work are limiting how much of the fleet any one activation can touch, and being able to undo it faster than it spreads. The controls that do not work are the ones that assume you know what the input will be.

Common weak answers

• "More testing before release." The defect passed testing because the input did not exist yet.
• "Reject unusual configurations." Rejecting what you cannot handle is reasonable and it moves the boundary rather than removing it, since the failure is by definition in the configuration you believed you could handle.
• "Per-customer isolation." Real and expensive at edge scale, where the economics depend on shared infrastructure. Argue it with the numbers or argue for blast-radius limits instead.