Local Revert

Server-side rollback assumes the affected system can still be reached. For a device, that assumption fails in exactly the case that matters: an update that prevents the device from booting, from joining the network, or from reaching its management plane cannot be fixed by any remote action.

Recovery then becomes physical — a person attending each device — which at fleet scale is a logistics problem rather than an engineering one, measured in days and dominated by travel rather than by effort.

Local revert removes that dependency. The device retains the previous known-good version, evaluates its own health after an update, and reverts autonomously when the new version fails.

Why it matters

The cost of a failure is its blast radius multiplied by its recovery cost per unit. A bad server deployment affecting an entire fleet is recovered by one rollback: the per-unit cost is approximately zero. A bad device update with the same blast radius, where the device cannot be reached, has a per-unit cost of a site visit — and the same defect becomes a different category of event entirely.

Local revert is the single mechanism that collapses the per-unit recovery cost back toward zero, which is why it belongs in any component whose failure could sever its own management path: firmware, bootloaders, kernel drivers, network device configuration, agent software and edge configuration.

Implementation patterns

• Dual-bank or A/B partitions: the update is written to the inactive slot and activated only after verification, with the previous slot left intact and bootable.
• A watchdog and health check evaluated after activation, with automatic reversion on repeated failure — and the health check must exercise the thing that matters, not merely confirm the process started.
• A boot counter, so a device that fails to complete startup a defined number of times falls back automatically.
• Verified boot, so a corrupted or tampered image is rejected rather than producing an unrecoverable device.
• The same mechanism for configuration and content, not only for firmware. A malformed configuration interpreted by a privileged component can be as fatal as bad code — which is the CrowdStrike lesson stated generally — and configuration is pushed far more often with far less validation.
• Revert reported when connectivity returns, so the fleet's reversion rate is a monitored rollout signal.
• A defined ceiling on reverts, so a device does not oscillate between versions indefinitely.
• Rehearsed and measured: a revert path that has never been exercised is an untested code path that will run under the worst conditions.

Industry example

The July 2024 CrowdStrike incident is the clearest demonstration of its absence. A content update caused kernel-level crashes on millions of endpoints; because the machines would not boot, no remote tooling could reach them, and recovery required physical intervention per device across airlines, hospitals, banks and broadcasters — extending a software defect into a multi-day global logistics event.

The contrasting practice is standard in embedded and automotive systems, where A/B partition updates with automatic fallback are long-established precisely because the recovery cost of a bricked vehicle or an inaccessible industrial controller is prohibitive. The technique is mature; its absence in software agents running on general-purpose machines is the gap that incident exposed.

Failure scenarios

• No retained previous version, so there is nothing to revert to.
• Revert requiring server contact, which fails in the case it exists for.
• A health check that only confirms the process started, missing a component that runs and is broken.
• Configuration and content exempted, when they change more often and are validated less.
• Verification skipped, so a corrupted image is activated.
• Unbounded revert loops, with the device oscillating between versions.
• Reverts not reported, so a rollout appears healthy while devices are silently falling back.
• The path never tested, discovered to be broken at the moment it is required.

Trade-offs

Dual-bank updates cost storage — two full copies of the software on every device — which is material on constrained hardware and is the usual reason the pattern is skipped in cost-sensitive fleets.

Automatic reversion also creates version fragmentation: a fleet in which some devices reverted is running mixed versions, and the backend must support both. A revert is not a clean state; it is a divergence that someone must reconcile.

And the health check is a real design problem. Too shallow and a broken device stays on the bad version; too aggressive and transient conditions trigger unnecessary reverts, producing fragmentation for no reason.

The trade is device storage, version fragmentation and health-check design effort in exchange for recovery that does not require reaching the device. For anything where a site visit is the alternative, it is not an optimisation but the difference between an incident and a catastrophe — and it should be sized by recovery cost rather than by blast radius alone.

Interview question

"We push agent updates to two hundred thousand machines. Tell me what happens if an update prevents them booting, what you would build so that answer is different, and what your health check has to actually verify for the mechanism to work."