A firmware update bricked 4% of a deployed fleet. Each recovery requires a site visit. What went wrong and what prevents recurrence?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know the mechanisms that make firmware updates recoverable, and the fleet discipline that bounds the damage.
What went wrong
At least one of:
No A/B partitioning. The update overwrote the running image, so an interrupted or corrupt write left no working image to fall back to.
No automatic rollback. A watchdog should detect that the device failed to boot or failed to check in, and revert to the previous partition unattended.
Insufficient staged rollout. 4% of the fleet suggests the update reached far more devices than a first wave should have. A cohort of a few hundred with health gating would have caught it.
An untested hardware variant. Fleets accumulate revisions — different flash chips, different memory timings, different sensor versions — and the update was validated on one.
Power interruption during write, which in the physical world is routine rather than exceptional.
What prevents recurrence
A/B partitions, always. Signature verification by a bootloader that cannot itself be updated by an unsigned image. Watchdog-driven automatic rollback. Resumable transfer for intermittent links.
Staged rollout with health gating: an internal cohort, then a small production cohort, then expanding waves — each gated on devices checking in healthy, with automatic halt on any regression.
Hardware variant coverage in the test matrix, which requires an accurate registry of what is actually deployed.
The fleet capability underneath
This depends on fleet management: knowing what devices exist, what version each runs, which are reachable and which have gone silent. The last is the hardest, because absence is invisible to any monitoring that only processes messages that arrive — and it is exactly the signal that detects a bricked device.
What a strong answer adds
Recognising that the recovery cost dominates the design. Truck rolls dominate the operating cost of any deployed fleet, so the investment justification for A/B partitions and staged rollout is straightforward arithmetic: one avoided fleet-wide incident pays for all of it many times over.
Common weak answers
More testing before release. Blaming the hardware variant without addressing rollout staging.