Fleet Reconnection Storm

A large device fleet holds long-lived connections whose establishment cost is amortised over days. Any disruption destroys that amortisation instantly: a platform restart, a network event, a load balancer change, or a brief ingest failure disconnects the fleet, and every device attempts to reconnect at approximately the same moment.

Reconnection is far more expensive than steady-state messaging — TLS handshake, authentication, session establishment, subscription setup, and the buffered backlog each device has accumulated while offline.

The platform therefore receives a synchronised burst of its most expensive operation, saturates, rejects connections, and the rejected devices retry — sustaining the overload independently of the original fault, which was resolved minutes ago.

Why it matters

Steady-state capacity planning does not cover this at all. A platform sized for a million connected devices sending small messages has no capacity for a million TLS handshakes plus a million backlog drains in one minute, and the gap is frequently two orders of magnitude.

The characteristic incident shape follows: a short, minor trigger and a long, severe outage whose cause is the recovery rather than the fault. Analysis focused on the trigger learns nothing durable, because the trigger differs next time and the amplification is identical.

And the fix cannot be shipped during the incident. Devices already in the field run the firmware they run — client-side backoff logic cannot be updated in the moment, which is what makes server-side controls essential rather than complementary.

Implementation patterns

• Exponential backoff with full jitter on every device, from the first firmware release. The jitter is the mechanism — backoff without randomisation moves the synchronised herd rather than dispersing it.
• Server-directed reconnect timing, telling each device when to return. This is the control that matters, because it works on devices already deployed and puts admission rate in the hands of the side that knows the capacity.
• Admission control at the connection tier: accept at a sustainable rate and reject the rest with a retry-after, rather than accepting everything and collapsing.
• Cheap reconnection: session resumption, cached authentication where the security model permits, and deferred subscription setup, so a reconnect is not a full establishment.
• Backlog drain throttled and separated from the connection burst, since a device that reconnects and immediately uploads an hour of buffered telemetry doubles the problem.
• Cohort staggering derived from a hash of the device identifier, dispersing returns deterministically.
• Graceful draining for planned changes, moving devices gradually rather than dropping them — a drain that disconnects everything at once is a self-inflicted storm.
• Capacity planned for the recovery scenario, load-tested explicitly, since it is a different and much larger number than steady state.

Industry example

The pattern is universal in large device and persistent-connection fleets, and it is the same phenomenon as the websocket reconnect storms that extended chat-platform outages: the published analyses converge on server-controlled pacing and cheap resumption as the mechanisms that matter, because they address the amplification rather than the trigger.

The device case is harder in one specific respect. A web client can be updated by shipping new JavaScript; a deployed device cannot, so any mitigation that depends on client behaviour must have been correct in the firmware that shipped — which is why server-side admission control carries disproportionate weight here.

Failure scenarios

• Backoff without jitter, producing synchronised retry waves at increasing intervals.
• Client-side-only backoff, leaving the platform unable to influence deployed firmware.
• Expensive reconnection with full authentication and subscription rebuild every time.
• Buffered backlog uploaded immediately on reconnect, compounding the connection burst.
• Planned maintenance disconnecting the fleet at once.
• Admission control absent, so the platform accepts more than it can serve and collapses.
• Capacity modelled only for steady state, with the recovery scenario never tested.
• A firmware bug producing a reconnect loop, which is a distributed denial of service against your own platform from devices you cannot update.

Trade-offs

Server-directed pacing means deliberately keeping some devices disconnected longer so that the fleet recovers sooner. For telemetry that is straightforwardly correct; for devices whose function depends on connectivity — control, safety, alerting — it is a real product decision about which devices are admitted first, and it should be made deliberately with priority classes rather than by hash order.

Cheap reconnection requires holding session state server-side after disconnection, which costs memory and raises the question of where that state lives when the node holding it is what failed.

Provisioning for full-fleet reconnection is the alternative and is usually uneconomic, which is precisely why admission control plus cheap resumption is the standard answer rather than capacity. The trade is a slower recovery for some devices in exchange for a recovery that succeeds at all — and the mitigation must be in the firmware before it is needed, because it cannot be added afterwards.

Interview question

"A thirty-second network event disconnects our million-device fleet. Walk me through the next ten minutes, tell me which of your mitigations work on devices whose firmware we cannot change today, and tell me what should have been in the first firmware release."