A configuration push to thousands of edge locations must complete in under a minute, yet a bad config could take down the whole network. How do you reconcile speed with safety?

Recognise the real tension

The speed requirement is genuine. A security block-list, a DDoS mitigation rule, or a rule blocking an active exploit must reach every location in seconds; a wave-based rollout taking an hour makes the system useless for its most important purpose.

And the danger is equally genuine: the same mechanism that delivers a good change everywhere in a minute delivers a bad one everywhere in a minute. Most large edge-network outages have this shape.

The resolution is not to pick a side. It is to recognise that not all configuration is the same, and to stop carrying it all on one path.

Separate the classes

• Class A — narrow, high-urgency, low-expressiveness. Block-lists, rate-limit values, feature kill switches. These have a restricted schema that cannot express arbitrary behaviour — a list of IPs cannot crash a proxy the way a routing rule can. They get the fast global path, with strict schema validation and bounded value ranges as the safety mechanism.
• Class B — expressive, lower-urgency. Routing rules, TLS configuration, origin definitions, transformation logic. These can express behaviour that breaks the data plane, so they go through waves with health gates, regardless of how urgently someone wants them.

Almost every serious incident in this category comes from putting a class B change on the class A path, and the discipline is refusing to do so even under pressure.

Safety mechanisms for the expressive path

• Linting and simulation: parse and validate, then apply the config to a shadow instance replaying real traffic and compare responses. A change that alters the response to production traffic in an unexpected way is caught before any user sees it.
• Wave-based rollout with health gates: one location, then a small set spanning different traffic profiles, then a percentage, then everything — each wave gated on error rate, latency, and connection success, with automatic hold.
• Bake time per wave long enough for the failure mode to appear, which for a configuration whose effect depends on traffic patterns means spanning a meaningful traffic period.
• Per-location health gating: a location that fails after applying the config reverts itself without waiting for a central decision, because the central decision may not be reachable.

The rollback property that decides everything

Rollback must not depend on the thing that just broke. If a bad configuration prevents an edge node from reaching the configuration service, it can never receive the fix, and recovery becomes a physical or out-of-band operation across thousands of locations.

Therefore:

• Every node retains the previous known-good configuration locally and reverts on its own if health checks fail after applying a new one. This is the single most important mechanism, because it works when nothing else does.
• A separate, minimal, out-of-band control channel for emergency rollback, deliberately simple enough that it is unlikely to be affected by a data-plane configuration error.
• Rollback is rehearsed, with a measured time-to-revert that is a known number rather than a hope.

The uncomfortable requirement

A "commit" of configuration should be as reviewable, testable and revertible as a code deploy — and it is almost never treated that way, because it feels like an operation rather than a change. The organisations that avoid this class of outage are the ones that made that distinction disappear.