In July 2019 Cloudflare pushed a WAF rule containing a regular expression that backtracked catastrophically, pegging CPU across its global network for roughly 27 minutes. The rule reached the whole fleet in seconds, by design, because security rules must propagate fast. Treating that as a delivery-system question rather than a regex question - what does this teach, and where would over-correcting be a mistake?

The situation they were in

Two requirements in direct conflict. A new attack signature must reach every edge location within seconds, because the attack is happening now and a slow rollout means hours of exposure. And any change that reaches every edge location within seconds has a global blast radius by construction.

The published account describes exactly that shape: the change was data, not code; it propagated through a distribution path built for speed; and the failure was CPU exhaustion across the network at once.

What the delivery system got wrong

Not the speed. The error was treating one change class as one delivery problem. A WAF rule set contains changes with very different urgency and very different risk, and they were all delivered on the same path:

• A signature for an attack in progress: seconds matter, risk is usually low because the rule is narrow.
• A broad rule-set update or an engine change: seconds do not matter, and the risk is the whole fleet.

One path for both means the risk profile of the second is delivered at the cadence of the first.

The second error is that the path had none of the controls a code deployment would have had. No staged rollout, no canary fleet, no automatic abort on a health signal, no simulation of the rule against recorded traffic before distribution. Configuration delivered as data had escaped the delivery system's safety machinery because nobody classified it as a deployment.

What they chose afterwards, and what generalises

The generalisable answers, in order of value:

1. Classify changes by urgency and risk, and give each class its own path. The emergency path stays fast and is restricted to narrow, verifiable changes. Everything else goes through a staged path. This is the whole lesson.
2. Make the risky property mechanically impossible rather than reviewed. For regular expressions, that means an engine with guaranteed linear-time matching, or a complexity limit checked before distribution. A review cannot reliably spot catastrophic backtracking; a checker can.
3. Resource guards at the point of execution. A CPU or time budget per rule evaluation, so a pathological input degrades one request rather than the machine.
4. Staged distribution with an automatic abort, even when staging is measured in seconds. A canary fleet that takes 10 seconds of the 30 is still a canary, and it is the difference between 27 minutes of global impact and 10 seconds of partial impact.
5. A global kill switch for the whole subsystem, tested, because the recovery from a bad global push is to stop executing the thing that was pushed.

When not to copy this, and where over-correcting is the mistake

Do not conclude that all configuration needs staged rollout. Most configuration is not executed code and has a small blast radius: a feature flag for one tenant, a timeout value, a copy change. Putting a 20-minute staged rollout in front of every configuration change makes operations slower and pushes people towards out-of-band edits, which is worse than what you started with.

The trigger for this machinery is specific: configuration that is interpreted or executed at runtime, distributed globally, and capable of consuming unbounded resources. Rule engines, routing rules, query templates, policy expressions and anything embedding a pattern language. A timeout value is none of those.

Also do not conclude that slow is safe. The emergency path exists for a reason, and removing it to prevent this class of incident trades a 27-minute outage for hours of exposure to attacks. The answer was two paths, not one slow one.

The honest summary

"Configuration is not deployment" is the bug. Anything distributed to production that changes behaviour is a deployment and deserves the same questions: what is the blast radius, what is the rollback, what aborts it automatically, and what proves it is safe before it ships.