An endpoint-security agent runs in the kernel on 40,000 of your Windows hosts. Your change-control standard covers every change your own engineers make, and you hold the agent at the previous sensor version through the vendor's update policy. On 19 July 2024 CrowdStrike pushed a Rapid Response Content update - configuration data delivered as template instances rather than a sensor release - at 04:09 UTC and reverted it at 05:27 UTC; Windows hosts on sensor 7.11 or later that were online in that window and received it crashed, and Microsoft's crash-report-derived estimate was about 8.5 million devices. Your change register for that day is empty. Which control did you not have, and what do you put in place?

The gap the register hides

Your change-control standard counts changes your organisation initiates, and the population of changes that can alter production behaviour is larger than that. A supplier that ships into your estate is an unregistered change source. The register was empty because nothing in it was designed to see a change you did not make.

Holding the agent one version back is a real control over one artefact class and says nothing about the others. CrowdStrike's own preliminary post-incident review separates the sensor release path from Rapid Response Content, and its remediation list includes a staggered deployment strategy and giving customers greater control over the timing and targeting of Rapid Response Content updates - which tells you what customer-side control over that path existed in July 2024.

What control you actually held

Three, and only three, in descending strength:

1. Exposure shaping, where the supplier offers rings or deferral for that artefact class. For the sensor it existed. For this path it did not.
2. Blast-radius limits of your own design: which hosts run the agent, and whether the systems you would recover with are inside the same population.
3. A written risk acceptance naming who owns the residual risk, because a supplier change path you cannot stage is not a gap to close - it is an exposure someone has to accept.

The recovery cost is where the second one is decided. The published remediation was a manual per-host procedure, and for encrypted disks it needed recovery keys. At 40,000 hosts and fifteen minutes of hands-on work each, that is 10,000 person-hours, which is why the key escrow, the jump hosts and the identity path used to perform the fix must not be in the same agent population.

The controls worth building

• A supplier change inventory. Per supplier, each artefact class that can change behaviour without your approval: agent code, detection content, cloud control-plane features, SaaS releases, model versions behind an API. For each one: can it be staged, what is the fastest rollback, who owns the residual risk. The output is deliberately a list with "no customer control" entries.
• A canary host population that takes supplier-staged changes first, sized so a total loss of it is survivable - a few hundred hosts across functions, never a single team, and never the recovery tooling.
• A reconciliation control: agent version and content revision per host against expected, daily. Threshold: any host more than two revisions from the fleet target for 48 hours raises a ticket. This is also the evidence that the staging policy is operating rather than merely configured.
• Procurement as the control of last resort. Notice of change, staged delivery and a documented rollback become contract terms for suppliers running in your kernel, because no architecture recovers that privilege once it is granted.

When this is the wrong answer

Deferring security content has its own loss function. Choosing to run yesterday's detections to avoid a bad push is a trade, not a free improvement, and for an agent whose job is blocking active intrusions the deferral can be the larger risk. Stage content only where the agent sits in a crash-critical path, and keep the window in hours. For a low-criticality fleet, take the supplier's fastest path and spend the effort on recovery time instead.

Common weak answers

• "Put vendor updates through the change advisory board." The supplier does not attend, and the change arrives in minutes. A board cannot gate a push it learns about afterwards.
• "Test it in staging first." You cannot. Content on that path reached every subscribed host; only the supplier's own rings create a first population.
• "Run two endpoint vendors." It doubles the kernel-level surface and the operating cost to halve the exposure to one class of event. At this size it usually loses.