Most incidents in an organisation are caused by configuration changes rather than code deploys. How should configuration be versioned, canaried, validated and rolled back with the same rigour as code?

Why configuration is the higher risk

Organisations apply substantial rigour to the code path — review, tests, build reproducibility, canary, staged rollout, automated rollback — and frequently apply none of it to configuration, which:

• Changes far more often than code, so there are many more opportunities to be wrong.
• Reaches production faster, often in seconds, because speed is the point.
• Has a wider blast radius, since one change typically applies fleet-wide rather than rolling gradually.
• Is interpreted by code with the same privilege as the code. A file parsed by a kernel driver can crash the kernel exactly as effectively as a bug in the driver — which is the CrowdStrike 2024 lesson stated generally.

Anything that changes production behaviour is a change, regardless of whether it compiles. The rigour of a rollout should be set by its blast radius and its interpreter's privilege, not by its file extension.

The controls, in order of value

1. Version control and review. A configuration change with no diff and no reviewer is an unreviewed production change. This is close to free and is frequently absent.

2. Validation before publication, against the real consumer. Schema validation, semantic linting, and bounded value ranges — so a malformed configuration is impossible to publish rather than merely unlikely. And critically, validate with the exact parser that will consume it: a validator that has drifted from the consumer is a subtle and common failure.

3. Simulation. Apply the change to a shadow instance replaying real traffic and compare responses. A change that alters behaviour unexpectedly is caught before any user sees it, which no amount of schema validation achieves.

4. Progressive rollout with health gates. One instance, one cell, one zone, one region, the rest — with an automatic hold when any wave's health signals degrade. There is no legitimate reason for "published" and "on every machine" to be the same moment.

5. Bake time per wave exceeding the time for the failure mode to appear. A configuration whose effect is only visible under peak load needs a window spanning a peak.

6. Rollback that does not depend on what just broke. The decisive control. If a bad configuration prevents a node from reaching the configuration service, it can never receive the fix. Therefore every node retains the previous known-good configuration locally and reverts itself on failing health checks, without waiting for a central decision that may be unreachable.

Separating urgency classes

The genuine tension is that some configuration must propagate globally in seconds — a security block-list, an exploit mitigation, a kill switch. A wave-based rollout taking an hour makes the mechanism useless for its most important purpose.

The resolution is not uniform slowness but two paths:

• Class A — narrow, urgent, low expressiveness: block-lists, numeric limits, boolean switches, with a restricted schema that cannot express arbitrary behaviour. A list of addresses cannot crash a proxy the way a routing rule can. Fast global path; safety from schema and range constraints.
• Class B — expressive, capable of changing execution behaviour: routing rules, TLS configuration, transformation logic, detection content. Waves and health gates, regardless of how urgently someone wants it.

Almost every serious incident in this category is a class B change placed on the class A path, and the discipline is refusing that under pressure — with the classification decided by what the content can do, not by how badly someone wants it deployed.