Reconnect Storm

Any system holding long-lived connections — chat, presence, live streaming, trading feeds, IoT fleets — faces the same asymmetry: connections are established gradually over hours as users arrive, and are lost all at once when something fails. The system was sized for the gradual case and must survive the instantaneous one.

A reconnect is also far more expensive than a steady-state message: it typically involves a TLS handshake, authentication, session lookup or creation, subscription re-establishment, and an initial state fetch. So the load spike is not proportional to the connection count — it is several times worse.

Why it matters

The disruption is usually survivable. Losing one gateway process out of many should be a non-event. The storm is what turns it into an outage: displaced clients reconnect immediately onto surviving capacity that is already serving its own load, pushing it over, displacing more clients, and so on.

Implementation patterns

• Jittered exponential backoff in the client, with a cap — and, crucially, enforced server-side. Client-side politeness is a request; old app versions, third-party clients and buggy releases will ignore it. The server must reject early reconnects.
• Session resumption instead of full refetch. A sequence number plus a bounded replay buffer turns reconnection into "send me what I missed", which is cheap. This is the single highest-leverage mitigation.
• Admission control on the connect path, prioritising resumes over cold connects and shedding the rest with a Retry-After.
• Staggered reconnection tokens — the server tells each client when to come back, spreading the return deterministically rather than hoping jitter is enough.
• Capacity headroom sized for losing a failure domain. If losing one of five zones is survivable only in theory, it is not survivable.
• Spread the return across regions, rather than sending every displaced client to the nearest surviving one.
• Connection-count limits per process, so a single node cannot accumulate an unrecoverable share.

Industry example

Real-time platforms with tens of millions of concurrent connections — chat and community platforms, live-streaming services, conferencing systems during a regional failure — all converge on the same design conclusions, because the arithmetic is unforgiving. A gateway process holding a hundred thousand sessions that dies produces a hundred thousand simultaneous TLS handshakes, authentications and state fetches against the remaining fleet.

The instructive detail is that the mitigations are almost entirely protocol decisions rather than capacity decisions: resumable sessions, sequence numbers, bounded replay buffers, and server-controlled reconnect scheduling. You cannot buy your way out of a reconnect storm with instances, because the storm scales with your user base and arrives faster than any autoscaler.

Failure scenarios

• Backoff implemented only client-side, so a legacy client population ignores it.
• Reconnection requiring a full state fetch, making each reconnect expensive by design.
• Deploys causing storms. A rolling restart that disconnects clients in large batches produces the same event routinely, which is why connection draining and slow rollouts matter for stateful tiers.
• Autoscaling as the answer, arriving minutes after the storm has already decided the outcome.
• Health checks that fail during the storm, causing the load balancer to remove nodes that are busy but recovering — accelerating the collapse.

Trade-offs

Resumable sessions cost server-side buffering and protocol complexity. Server-enforced backoff costs some legitimate clients a slower reconnection. Headroom for a failed domain costs real money — typically 20–30% of the fleet sitting idle for an event that may not happen this year.

The alternative is a system whose failure domains are theoretical. If you cannot afford the headroom, the honest response is to reduce the size of the failure domain — smaller cells, more of them — rather than to assume the storm will be gentle.

Interview question

"A gateway node holding 100,000 WebSocket sessions crashes. Walk me through the next 60 seconds across the fleet, and tell me which mitigations must already exist versus which you could apply during the incident."