Two million devices reconnect simultaneously after a network partition. What happens and what prevents it?
Show the full answer Hide the answer
What is being tested
Whether you anticipate the reconnection storm, which is a self-inflicted second outage and the characteristic IoT ingest failure.
What happens
All two million reconnect within seconds, producing simultaneously:
- Two million TLS handshakes, which are CPU-expensive.
- Two million authentication requests.
- Two million session establishments.
- Buffered backlogs delivered at once — a device offline for an hour delivers an hour of readings.
The gateway falls over. Devices retry immediately. The storm repeats, now including the failed retries, and it becomes self-sustaining — outlasting the network event that caused it by a long way.
What prevents it
1. Jittered exponential backoff in the device firmware. Non-negotiable and the single highest-value control, because it alone spreads reconnection over minutes rather than seconds.
Critically: this must be in the firmware, which means it must be right before devices ship. A fleet without it cannot be fixed remotely, because you cannot reach the devices to update them.
2. Connection-rate limiting at the gateway, rejecting excess with a back-off instruction rather than dropping silently — a device that gets no answer retries immediately.
3. Capacity measured in concurrent connections, not messages per second. Load testing must simulate connection counts, or it measures the wrong thing entirely.
4. Accept, persist, acknowledge — then process asynchronously. Writing to a durable log immediately decouples ingest availability from processing capacity, so the pipeline can fall behind without dropping device data.
5. Bounded buffering with a defined shedding policy. Unbounded buffering converts a throughput problem into an outage.
6. Per-device rate limits, so one malfunctioning device cannot flood the pipeline.
The backlog problem
After reconnection, devices deliver buffered readings with old timestamps. Processing must handle event time differing substantially from arrival time, with an explicit lateness policy — and must not treat the backlog as a real-time spike.
The related traps
Device clocks are unreliable. Record both device time and arrival time, and never order on device time alone.
Duplicates are guaranteed, from at-least-once delivery plus device retries. Deduplication is required.