After a five-minute network outage at your ingest tier, 200,000 devices reconnect simultaneously and the authentication service collapses. How do you fix this?

The mechanism

Every device detected the disconnection at roughly the same moment and retried immediately. Reconnection is the most expensive operation in the fleet's lifecycle — a TLS handshake plus authentication per device, paid per connection rather than amortised — so the entire fleet's most expensive operation arrived at once.

The authentication service then became slow, devices timed out and retried, and the retries added load to a service already failing. This is congestion collapse, and without backpressure it does not resolve on its own.

The fix on the device, which is where it must primarily live

Jittered exponential backoff. Not exponential alone — with identical backoff curves the fleet retries in synchronised waves, which is the same problem at longer intervals. Full jitter, where each device waits a random duration up to the current backoff bound, spreads the load properly.

This must be in firmware, which is the hard part: it means an update to 200,000 devices, and any device that cannot be updated retains the behaviour indefinitely. That is the argument for getting reconnection behaviour right before shipping hardware.

The fix at the ingest tier

Load shedding at the edge, rejecting connection attempts above a rate the authentication path can serve. A device that is rejected quickly and backs off is far better than one that occupies a connection slot waiting on a failing service.

Separate the authentication path's capacity from steady-state sizing. It is sized for reconnection storms, not for normal operation, and those differ by orders of magnitude.

Session resumption, so a reconnecting device that already has valid credentials skips the expensive path.

The design principle to carry forward

Fleet-wide synchronised behaviour is the characteristic failure mode of large device estates, and it is not limited to reconnection. Scheduled reporting on the hour, firmware checks at midnight, certificate renewal on a fixed date — each produces the same shape.

Jitter everything the fleet does on a schedule, and make that a review item for any firmware change that adds a periodic behaviour.