Supplier Change Surface

A change register answers one question: which changes did we make. An estate running other people's software has a second population nobody counts - agent updates, detection content, provider-enabled control-plane features, SaaS releases, model versions behind an API - and each can alter production behaviour on a schedule the supplier sets.

The gap shows up as an empty register on the day everything broke. On 19 July 2024 CrowdStrike pushed Rapid Response Content - configuration template instances, not a sensor release - at 04:09 UTC and reverted it at 05:27 UTC; Windows hosts on sensor 7.11 or later that received it crashed, and Microsoft's crash-report-derived estimate was about 8.5 million devices.

The practice is to write the population down before an incident does it for you, one row per supplier artefact class, treating every row that reads "no customer control" as an accepted risk with a named owner.

Why it matters

The controls an organisation is proud of are aimed at its own changes: review, canary, staged rollout, rollback. A supplier push bypasses all of them, and the software most likely to push often holds the deepest privilege - kernel agents, identity integrations, build plugins.

Writing the surface down changes three decisions otherwise made by default:

• Where the recovery path lives. If the hosts you would recover with run the same agent in the same exposure group, you have one failure, not two systems.
• What procurement must ask for. Staged delivery and documented rollback are cheap to negotiate before signature and close to impossible afterwards.
• Who signs the residual. A push path you cannot stage is an exposure somebody has to accept at a level matching its blast radius.

Implementation patterns

• One row per artefact class, not per supplier: class, delivery mechanism, can it be staged, who controls the staging, fastest rollback, residual owner.
• A canary host population that takes supplier-staged changes first - a few hundred hosts across several functions, never a single team, never the tooling used to recover.
• Reconciliation as the operating evidence. Agent version and content revision per host against the fleet target, daily. Threshold: any host more than two revisions off target for 48 hours raises a ticket. Without it, a staging policy is configured rather than operating.
• Recovery arithmetic next to the row. The July 2024 remediation was manual and per-host, needing disk recovery keys: at 40,000 hosts and fifteen minutes each, 10,000 person-hours. That number justifies keeping key escrow and jump hosts out of the exposed population.
• Contract terms as the last resort: notice of change, staged delivery, rollback commitment, a right to defer.

Industry example

The July 2024 CrowdStrike event is the clearest public case, because the vendor documented the paths itself: its preliminary post-incident review separates Rapid Response Content from the sensor release path, and lists among the remediations a staggered deployment strategy and greater customer control over the timing and targeting of that content. Customers who had carefully held the sensor one version back found that the control they operated covered a different artefact class from the one that reached them.

Failure scenarios

• The empty register. Review finds no change recorded, and the outage is called unforeseeable rather than uncounted.
• Recovery inside the blast radius. Jump hosts, key escrow and the management console all run the agent that is crashing.
• Staging that exists on paper. A deferral policy set once has drifted on 30% of hosts, because it was treated as configuration rather than as a control.
• Over-deferral. A fleet three weeks behind on detection content runs known-stale defences: a different incident, same root cause, nobody priced the trade.

Trade-offs

When not to use it

For suppliers with no privileged execution and no shared failure mode, the row is the whole control. A design tool needs no canary population: write it down, note that there is no customer control, and spend the effort on the kernel agent and the identity provider. Where recovery is fast, exposure shaping also loses to restore time - if hosts can be reimaged in minutes at scale, that is where the money should go.

Interview question

Q: Your change register is empty for the day of an outage caused by a supplier's automatic update. What do you build, and what do you tell the risk committee you cannot build?

What a strong answer covers: the register counts changes we initiate, so the supplier population is structurally invisible · an artefact-class inventory with staging capability and residual owner per row · a canary population excluding the recovery path · reconciliation as the evidence staging is operating · and the limit, that unstageable paths leave only blast radius, recovery time and a signed acceptance.

Quick check

Quiz: Why does holding an agent at the previous version not protect you from that vendor's content updates? Because version staging and content delivery are separate artefact classes with separate controls, and the content path may offer no staging at all.

Flashcard: What do you do with the inventory rows that say "no customer control"? - They become signed risk acceptances with blast-radius and recovery-time limits, not open remediation items.