Unmanaged Change Surface
also called Out-of-Band Change Path, Ungated Change
Every mechanism that can alter production without passing through the deployment pipeline - so the organisation's rollout controls apply to a fraction of its changes and the incidents come from the rest.
An organisation has canaries, bake times, automated health gates and per-region sequencing. Its deployment discipline is genuinely good. Then half the fleet loses network connectivity because an operating-system security update ran on a schedule, globally, at once.
Nothing in the pipeline was wrong. The change did not go through the pipeline.
Why it matters
The pipeline is usually the most carefully engineered control an organisation has, and it covers the change type engineers think about most: application code. Meanwhile the estate accumulates other paths, each introduced for a good reason, each with no waves, no gates and no rollback story.
Most organisations can name ten such paths from memory and have closer to twenty. Package and OS updates · base-image refreshes · agent and sidecar auto-updates · certificate rotation · DNS changes · feature-flag services · content and rule updates · vendor-pushed configuration · hand-run database changes · cloud-provider maintenance events · IAM policy edits · CDN configuration.
The second reason it matters is correlation. Regions are only independent with respect to failures they do not share, and a global change schedule is a shared failure - so a mechanism that updates every region at 06:00 UTC defeats the isolation that having regions was supposed to buy.
Implementation patterns
- Inventory the paths explicitly, as a written list with an owner per path. The exercise takes an afternoon and is the highest-value part of this whole discipline.
- For each path, either route it through staged rollout with a health gate, or record an accepted risk with a named owner and a review date. Both are legitimate; silence is not.
- Sequence regions for everything, including patching. Nothing should reach all regions on one schedule.
- Give fast paths waves rather than delay. A global push in 15 minutes can be four waves of under four minutes with an automated gate. Speed and staging are not in conflict.
- Test on running machines, not fresh ones. A large class of infrastructure defect requires an already-running node and is invisible to any test that starts from a clean image.
- Alert on change events from each path, so an unattended mechanism at least appears on the timeline during an incident.
Industry example
Datadog's published analysis of its 8 March 2023 outage describes a systemd security update applied
automatically to a large number of virtual machines. On a running node, restarting systemd-networkd deleted
the routes installed by the container network plugin, taking the node offline; the interaction did not
reproduce on a fresh or rebooted node, because during a normal boot the network daemon starts before the
plugin installs its routes. Roughly 50-60% of production Kubernetes nodes lost connectivity, across five
regions, and the legacy patch mechanism had run fleet-wide with no staged rollout, health gate or approval.
The same shape recurs whenever a mechanism outside the pipeline touches every host: the blast radius exceeds anything a code deployment would have been permitted, precisely because nobody thought of it as a deployment.
Failure scenarios
- Fleet-wide simultaneous patching, removing the independence of regions and availability zones.
- Agent auto-update, where a vendor's release cadence becomes your deployment cadence.
- Certificate rotation with no canary, so an expired or mis-issued certificate affects everything at once.
- Feature-flag services as an ungated change path: a flag flip reaches 100% of traffic instantly and is usually not counted as a change at all.
- Hand-applied infrastructure fixes that leave the declared state and the real state divergent, so the next automated reconciliation is itself a surprise change.
- Recovery requiring per-host action, because the change was applied N times independently rather than rolled out once and therefore cannot be reverted once.
Trade-offs
Bringing every path under rollout control is real work and some of it is unwelcome. Security patching gains latency, which is a genuine security cost and must be argued honestly rather than waved away. Vendor-pushed updates may not offer staging at all, in which case the control is a contractual or architectural one - pinning versions, or isolating the component.
| Choose | Gains | Pays |
|---|---|---|
| Gate every path | One rollout discipline; bounded blast radius | Latency on patching; work per path; some paths need vendor cooperation |
| Gate only the pipeline | Cheap; covers the changes engineers make most | Incidents arrive from paths nobody models |
| Accept a path explicitly | Honest; reviewable; cheap | Only works if the review date is kept |
When not to use it
For a small estate with one region and one environment, the blast radius of any change is already everything, so staging buys little and the effort belongs in backups and recovery time instead.
It also does not apply to changes that are genuinely outside your control and cannot be staged - a cloud provider's own maintenance, a regulator-mandated block. Those are risks to design around rather than paths to gate, and the right response is architectural isolation rather than a gate you cannot enforce.
Interview question
Q: Your team's deployment pipeline is excellent and your incident rate is not improving. Where do you look, and how would you build the case for whatever you find?
What a strong answer covers: classifying a quarter of incidents by originating change type rather than by affected service, which is the evidence that makes the argument · enumerating change paths as a written inventory with owners · recognising that unattended patching and vendor agent updates typically outnumber code deploys · proposing waves rather than delay for the fast paths, so the security argument is not lost · the explicit accepted-risk register for paths that cannot be gated · and the honest caveat that gating patching adds exposure time, so the trade should be decided rather than assumed.
Quick check
Quiz: Why did region isolation not help during a fleet-wide unattended OS update? Because independence only holds for failures regions do not share, and a global update schedule is a shared failure - every region received the same change at the same moment.
Flashcard: What is the review question, instead of "how do we deploy safely"? — "What else can change production, and what governs that?" Most estates can name ten paths and have twenty.