An OTA firmware update has bricked 3% of a 50,000-device fleet. What do you do, and what should have prevented it?

Halt immediately, then determine the correlation

Stop the rollout before anything else. Then find what the 1,500 have in common: hardware revision, previous firmware version, storage state, connectivity type, region. There is almost always a correlation, and it tells you whether the remaining fleet is at risk.

Establish whether "bricked" means unrecoverable

Three states, and they need different responses. Devices that boot but cannot reach the network may be recoverable if there is any other channel. Devices that fail to boot but have a working bootloader may recover to the previous partition. Devices with no bootloader are physical recoveries.

The proportion in each category determines the cost, and it is worth establishing before communicating a number.

What should have prevented it

Staged rollout with health gating. Test group, then 1%, then progressively larger, with an automated check between cohorts. At 1% this would have affected fifteen devices, not fifteen hundred.

Gating on check-in rate, not error rate. A bricked device does not report an error; it goes silent. A gate watching for errors proceeds happily while the fleet disappears. This is the specific control that would have caught it.

Cohorts ordered by the dimension where failures correlate — usually hardware revision. A change that works on the current revision and fails on the previous one is a common failure, and revision-ordered cohorts catch it first.

A/B partition update with a watchdog-confirmed boot, where the health check includes connectivity rather than merely successful startup. That single detail is the difference between an automatic revert and a brick.

The recovery plan

For recoverable devices, a targeted fix with the same staged discipline — the temptation to push a correction to everyone at once must be resisted.

For unrecoverable ones, a field replacement plan and a cost estimate. And a communication to whoever owns the customer relationship, before they hear it from users.