Cloudflare 2019: One Regex, Global Outage
also called Cloudflare WAF Outage
A firewall rule containing a regular expression with catastrophic backtracking consumed CPU across Cloudflare's entire global network within seconds of deployment.
What happened
On 2 July 2019 Cloudflare deployed a new managed WAF rule intended to block a class of JavaScript attack. The rule contained a regular expression with catastrophic backtracking — a pattern whose matching cost grows exponentially with input length on certain inputs.
The rule was pushed to the entire global network essentially at once, because WAF rules were distributed through the same rapid configuration propagation mechanism that made Cloudflare's network responsive. CPU across every machine in every location was consumed by the regex engine. Cloudflare's network stopped serving traffic, returning 502s globally, for around 30 minutes.
Why it propagated
Cloudflare's configuration distribution is a genuine architectural strength: a customer changing a firewall rule sees it applied worldwide in seconds. That same property meant a defective rule reached every machine before anyone could react.
Speed of propagation and blast radius are the same property viewed from two directions. A system that can update globally in seconds can also fail globally in seconds.
The changes that followed
A global kill switch for WAF rules that could be activated without a full deployment. Staged rollout for rule changes rather than simultaneous global distribution. And protective measures against excessive CPU consumption by individual rules, including analysis of regex complexity before deployment.
The transferable lesson
Two lessons, and the second is the more general one.
Any configuration path that reaches production faster than your ability to react is an unguarded deployment path. Feature flags, WAF rules, routing configuration, pricing tables, DNS records and policy updates are all production changes, and they routinely bypass the staged rollout, canary analysis and rollback that code changes go through. Ask which of your configuration changes can reach 100% of traffic in under a minute.
Untrusted input meeting an unbounded algorithm is a denial-of-service vector, and regular expressions are the most common instance. Any regex applied to attacker-controlled input needs either complexity analysis or a matching engine with linear-time guarantees.