Recovery Cost Per Unit

Risk analysis usually stops at blast radius: how many things are affected. That is only half the quantity that matters. The total cost of a failure is the number affected multiplied by the cost of restoring each one, and the second term varies over many orders of magnitude.

A bad deploy affecting every server in a fleet is recovered by a rollback: the recovery cost per unit is approximately zero, because one action fixes all of them. A bad update affecting every laptop in a company, in a way that prevents them booting, has a recovery cost per unit measured in a technician's visit — and no amount of automation reduces it, because the automation cannot reach the device.

Why it matters

The two failures look identical in a blast-radius assessment and differ by weeks of impact. Organisations that reason only about how many systems are affected consistently under-invest in the failures where remediation is expensive, and over-invest in the ones where a rollback would have sufficed.

The property that drives the cost is specific and identifiable: whether the failure destroys the channel used to fix it. A failure that leaves the management path intact is cheap to recover regardless of scale. A failure that takes the management path with it converts a software problem into a logistics problem, and logistics does not scale the way software does.

Implementation patterns

• Identify the components whose failure removes remote access: kernel drivers, bootloaders, firmware, agent software, network device configuration, identity providers, VPN concentrators. These are a distinct risk class and deserve stricter controls than their blast radius alone would suggest.
• Preserve an independent recovery channel — an out-of-band management network, a lights-out controller, a secondary boot path, a device that can be recovered without the primary path.
• Local automatic rollback: the component detects repeated failure after a change and reverts to the last known-good state on the device, without contacting a server. This is the single mechanism that collapses recovery cost to near zero for this class of failure.
• Staged rollout sized by recovery cost, not by blast radius. A change to a component with expensive recovery should roll out an order of magnitude more cautiously than one that can be rolled back centrally.
• Rehearse recovery at scale, and measure it — the time to restore one unit, and the parallelism available. A recovery plan without a measured per-unit time is an estimate that will be wrong by a large multiple.
• Design for partial-fleet recovery, so restoration can be prioritised: critical devices and sites first, with a defined order rather than an ad hoc one.

Industry example

The July 2024 CrowdStrike incident is the clearest illustration. The defect — a malformed content file causing kernel-level crashes — had an enormous blast radius, and the reason it became a multi-day global event was recovery cost. Machines would not boot, so no remote tooling could reach them; each affected endpoint required manual intervention, across airlines, hospitals, banks and broadcasters, including devices in locked facilities and in the hands of non-technical users.

The contrast is instructive: a comparable defect in a server-side service would have been rolled back in minutes. Same class of bug, same scale of exposure, and a difference of several orders of magnitude in impact — entirely attributable to where the failure sat relative to the recovery channel.

Failure scenarios

• A change to a boot-critical component rolled out at the same pace as an application deploy.
• Automatic update mechanisms with no local fallback, so a bad update disables the mechanism that would deliver the fix.
• Recovery requiring credentials or tooling hosted on the system that has failed.
• Firmware or BIOS updates to a fleet without a staged approach — the most expensive recovery in computing.
• Network configuration pushed to devices reachable only over the network being configured, which is the same failure in a different domain.
• A recovery procedure that has never been timed, so the incident plan assumes hours and reality delivers days.
• No prioritisation order, so recovery effort is spread evenly across critical and trivial devices.

Trade-offs

Treating expensive-recovery components with proportionally greater caution means slower rollout of changes that are sometimes genuinely urgent — a security patch to an endpoint agent is exactly the case where speed has real value, and where slowness has a measurable cost in exposure.

Maintaining independent recovery channels — out-of-band management, secondary boot paths, local rollback logic — costs money and adds its own complexity and attack surface. An out-of-band management network is another network to secure.

The trade is deployment velocity and infrastructure cost against the severity of the worst case in a specific, identifiable class of components. The resolution is not uniform caution but classification: most changes roll out normally, and the small set whose failure would destroy the recovery path is treated as a different category entirely.

Interview question

"We push agent updates to 200,000 laptops. Tell me what your rollout looks like, then tell me what happens if the update prevents the machine from booting — and how much of your answer changes if the same defect were in a server-side service."