Reconnect Storm

Persistent-connection systems — chat, collaboration, gaming, trading, live streaming, IoT — hold millions of long-lived connections whose establishment cost is amortised over hours. A disruption destroys that amortisation instantly.

Every disconnected client reconnects at approximately the same moment, and each reconnection triggers authentication, authorisation, session state loading, subscription setup and a backfill of missed messages. The system receives a synchronised burst of its most expensive operation, saturates, rejects connections, and the rejected clients retry — sustaining the overload independently of the original fault, which was usually fixed minutes ago.

Why it matters

Steady-state capacity planning does not cover this at all. A system sized for a million concurrent connections with a low connection-establishment rate has no capacity for a million establishments in one minute, and the gap is often two orders of magnitude.

The consequence is a characteristic incident shape: a short, minor trigger followed by a long, severe outage whose cause is the recovery rather than the fault. Post-incident analysis that focuses on the trigger learns nothing useful, because the trigger will be different next time and the amplification will be identical.

Implementation patterns

• Exponential backoff with full jitter, always. The jitter is the mechanism — backoff without randomisation moves the synchronised herd to a later moment rather than dispersing it.
• Server-directed reconnect timing. The server tells each client when to return, so admission rate is controlled by the side that knows the capacity — and this works even when old client versions are in the field, which pure client-side backoff does not.
• Session resumption. A session token plus a last-received sequence number lets the server restore subscriptions from stored state and stream only the gap. A resuming reconnect is orders of magnitude cheaper than a re-establishing one, and this matters more than any rate limit.
• Server-side session grace periods longer than a typical network blip, so a brief disconnection costs nothing at all.
• Gateway admission control: accept at a sustainable rate and reject the rest with a retry-after, rather than accepting everything and collapsing.
• Cohort staggering derived from a hash of the session, spreading returns deterministically.
• Deferred state loading — establish the connection cheaply and load non-essential state after the user acts.
• Controlled draining for planned changes: refuse new connections while keeping existing ones and migrate clients gradually. A drain that disconnects everyone at once is a self-inflicted storm.

Industry example

Large chat and collaboration platforms have repeatedly experienced this pattern, including Slack's incidents in 2021 and 2022 where cloud-provider networking and configuration problems disconnected large numbers of clients and the recovery load substantially extended the impact. The published analyses converge on the same conclusions: server-controlled reconnection pacing and cheap session resumption are the mechanisms that matter, because they address the amplification rather than the trigger.

Failure scenarios

• Backoff without jitter, producing synchronised retry waves at increasing intervals.
• Client-side-only backoff, leaving the server unable to influence a fleet of old clients.
• Expensive reconnection — full authentication, full state rebuild, full history fetch — with no resumption path.
• Sessions destroyed immediately on disconnect, so a two-second blip costs a full re-establishment.
• Draining a region by dropping all connections at once, creating the storm deliberately.
• Draining into a destination without verified headroom, converting one failed region into two.
• No gap-detection protocol, so clients silently miss messages and the correctness failure outlives the availability one.
• Capacity planned only for steady state, with the recovery scenario never modelled or tested.

Trade-offs

Session resumption requires holding server-side session state after disconnection, which costs memory and introduces the question of where that state lives when the gateway that held it is the thing that died — often answered with a shared session store, which is itself a dependency and a potential bottleneck.

Server-directed pacing means deliberately keeping some users disconnected longer in order to bring everyone back sooner. That is correct and it is uncomfortable, and it must be a conscious product decision because the user experience of "please wait, reconnecting in 40 seconds" is worse for an individual and better for the population.

Provisioning for full-population reconnection is the alternative, and it is usually uneconomic — which is precisely why admission control plus cheap resumption is the standard answer rather than capacity.

Interview question

"A 30-second network partition disconnects two million websocket clients. Walk me through the next ten minutes in a system with no protections, then tell me the three changes you would make in priority order — and tell me which of them works even if we cannot update the client."