Remediation Path Test

Every release process has a rollback story, and almost all of them assume the thing being rolled back is still running. Push a corrected artefact, the agent fetches it, the service restarts. That assumption holds for most defects and fails completely for the class of defect that removes the machine's ability to ask for the fix.

The test exercises recovery rather than describing it: take a host into the broken state deliberately - the malformed config, the crash-looping driver - and measure the cost of getting it back using only capabilities that survive the failure. The output is minutes of human time per host, and a procedure somebody can follow at 3am.

Why it matters

The cost of an incident is rarely the defect. It is blast radius multiplied by per-unit recovery cost, and per-unit recovery cost is the term nobody measures. A content defect corrected in 90 minutes that needs 20 minutes of hands-on work on each of 200,000 hosts is not a 90-minute incident; it is on the order of 50,000 to 100,000 person-hours, paid by customers your engineering capacity cannot help. Signed recovery media and a written procedure take one engineer about a week done calmly - and days during an incident, while everyone affected is down.

Implementation patterns

• Enumerate the artefacts that load before the network - kernel drivers, boot-time agents, bootloader configuration, early init units. Each needs a recovery path that does not use the network, because by the time it misbehaves the network is gone.
• Rehearse against the worst host configuration you support, not the newest: full-disk encryption, no local administrator, a till in a shop with no IT staff. A procedure that works on a developer laptop proves little.
• Pre-stage the capability: signed bootable recovery images tested per release, a scripted variant for fleets, and a key-escrow query so recovery keys come in bulk rather than one ticket at a time.
• Measure and publish the number - hands-on minutes per host, and the fleet total at your largest customer's size.

Industry example

On 19 July 2024 CrowdStrike's technical alert for the Channel File 291 defect told customers that hosts "still crashing and unable to stay online to receive the Channel File Changes" had to be booted into Safe Mode or the Windows recovery environment so the matching channel file could be deleted by hand. Over the following days it published guides for building bootable recovery images and an automated recovery USB tool.

That sequence is the point: the remediation path was constructed after it was needed. The defect was a content file with a parameter-count mismatch and a correction existed within about 90 minutes. Recovering individual machines, especially encrypted ones, took far longer, and no pipeline improvement would have shortened it.

Failure scenarios

• The rollback that assumes a booting host, which is the whole class described above.
• Recovery media that is never rebuilt, carrying a driver incompatible with current hardware.
• An unrehearsed procedure that turns out to need a credential only two people hold.
• Encrypted disks with no bulk key retrieval, so recovery is bounded by a help-desk queue.
• A restore tested only in staging, proving the data is restorable and not that a host is.

Trade-offs

This is effort spent on a path you hope never to use, competing with staged rollout and canarying, which reduce the probability of the event rather than its cost. Staging is the better first investment for most teams: cheaper, and it covers more defect classes.

The trade reverses when the per-unit recovery cost is large and paid by someone else, because then the expected cost of skipping the rehearsal is a multiple of the engineering it would take and the people paying it cannot reduce it. The planning question is not "should we test recovery" but "what is our recovery cost per host, and who pays it".

When not to use it

When the component cannot prevent the host from reaching the network. A user-space service that crash-loops leaves the machine patchable, so the self-update channel genuinely is the recovery path and recovery media is waste. The same applies to stateless containers behind an orchestrator, where remediation is "replace the instance" and ordinary deployment exercises it daily.

One question settles which world you are in: if this artefact is malformed, does the machine still reach the network? If yes, spend on staged rollout and fast rollback instead.

Interview question

Q: "You ship a kernel-mode agent to large enterprises. Your release process has canaries and a 15 minute rollback. What is still missing, and how would you put a number on it?"

What a strong answer covers: canaries reduce probability and not per-host cost; a boot-path defect removes the update channel, making rollback speed irrelevant; the per-host hands-on estimate and what worsens it (encryption keys, physical access); and the rule that any artefact loading before the network needs an out-of-band rollback.

Quick check

Quiz: Your rollback reaches every host in 15 minutes. Which defect makes that number meaningless? - One that stops the host booting or reaching the network, because the rollback channel needs a running host.

Flashcard: What does a remediation path test measure that a rollback drill does not? - Minutes of human time per host to recover without the shipped software working at all.