advanced 3 min answer

A staged over-the-air rollout reaches 100% over two weeks. What happens to the 12% of devices that were offline for the whole window and come back three weeks later on the old firmware?

otaversion-skewdesired-statefleetdelta-updates
Show the full answer Hide the answer

What happens, in order

  1. They come back and nothing updates them, because the campaign is complete. A rollout modelled as a campaign has an end; a fleet does not.
  2. They may not be able to come back at all. If the old firmware's protocol version, cipher suite or certificate authority was retired during the window, the device cannot authenticate, and an update it cannot reach is not an update. A device offline for weeks may also hold an expired token and a wrong clock, so the bootstrap path must not depend on either.
  3. The upgrade path may not exist. If updates are built as deltas from N-1, a device at N-3 has no route. Each skipped version doubles the problem for the next campaign.
  4. They arrive together. A seasonal or fleet-wide reconnection means thousands of devices request a full image at once. At 40 MB per image, 24000 devices is close to a terabyte of egress in a short window, against a CDN and a network that were sized for a trickle.

Where it amplifies

The tail compounds across releases. Devices that miss one window are disproportionately likely to miss the next — they are the ones in poor coverage, in storage, or seasonally idle — so the fleet stratifies into a maintained majority and a growing stratum of ancient versions, which is eventually where the security finding lands.

What stops it

  • Model the rollout as desired state, not a campaign. The platform holds a target version per cohort; a device that checks in below target is updated whenever it appears, for as long as it takes.
  • Measure the version distribution, not the rollout percentage. The alert is on the tail — the count of devices more than two versions behind, and the age of the oldest active version — because the mean will look fine.
  • Support N-2 or better on the server, with the window written down and enforced by a test that runs the N-3 upgrade path in CI. Retiring a protocol version is then a scheduled event with a fleet query behind it rather than a surprise.
  • Keep the recovery path independent: A/B partitions with a watchdog that falls back on failure, and a bootstrap that works with a wrong clock and an expired token, because those are the conditions a long-offline device returns in.
  • Rate-limit the reconnect surge with staggered check-in windows and jitter, and prefer deltas with a full-image fallback.

What would have to be true for it to self-heal

That the update path is available for every version still in the field, with no expiry, and that the device can always reach it. That is a property you choose to maintain, and it costs a compatibility window you must not quietly close.

When this is over-engineering

For a consumer mobile app, the store does this for you and the equivalent question is only about your API's compatibility window. The machinery above earns its cost when a failed or missed update means a site visit, which is the defining economic fact of physical fleets and the reason their release engineering looks nothing like a web deployment.