A chat platform's websocket edge gateways lose a region to a cloud networking failure. How should reconnect storms, session resumption, message-gap recovery and gateway draining be designed so a 30-second outage does not become a 3-hour one?

Why a 30-second outage becomes a 3-hour one

Every disconnected client reconnects at once. Millions of clients that were spread across time now arrive in the same second, and each reconnection is far more expensive than a steady-state connection: authentication, authorisation, session state loading, subscription setup, and a backfill of everything missed while disconnected.

So the surviving capacity receives a simultaneous burst of the most expensive operation the system performs. It saturates, connections fail, those clients retry, and the system never reaches a state where it can serve. The outage is now self-sustaining and has nothing to do with the original network fault, which was fixed minutes ago.

Reconnect storm control

• Exponential backoff with full jitter, mandatory. Backoff without jitter merely moves the synchronised storm to a later moment; the randomisation is the entire mechanism, not a refinement.
• A server-directed reconnect delay. The gateway tells the client when to come back, so the server controls the admission rate instead of hoping clients behave. This is the single most valuable control, because it works even when old client versions are in the field.
• Admission control at the gateway: accept connections at a rate the backend can support and reject the rest with a retry-after, rather than accepting all and collapsing.
• Staggered reconnection by client cohort, derived from a hash of the session, so clients naturally spread.
• Cap the cost of a reconnect — defer non-essential state loading until after the connection is established and the user acts.

Session resumption

The expensive part of reconnection is rebuilding state. A resumable session makes reconnection cheap:

• The client holds a session token and the last message sequence number it received.
• On reconnect it presents both; the gateway restores the subscription set from server-side session state rather than re-computing it, and streams only messages after that sequence.
• Sessions survive on the server for a grace period longer than a typical network blip, so a brief disconnection costs nothing.

The distinction that matters: a reconnect that resumes is orders of magnitude cheaper than one that re-establishes. Designing for resumption is what makes the storm survivable, more than any rate limit.

Message-gap recovery

Clients must be able to detect and close gaps deterministically:

• Monotonic per-channel sequence numbers, so a client knows exactly what it missed.
• A bounded replay buffer on the server, and an explicit "gap too large, resynchronise" response when the client has fallen outside it. That path must exist and be tested; without it, clients silently miss messages.
• Resynchronisation must be cheap and incremental — fetching a channel's recent history, not the whole workspace, or the recovery path becomes its own load event.

Gateway draining

For planned changes, and for shifting load away from a degrading region:

• Refuse new connections while keeping existing ones, so load moves gradually.
• Ask clients to migrate at a controlled rate with a server-directed reconnect target, rather than dropping them all at once. A drain that disconnects everyone simultaneously is a self-inflicted reconnect storm.
• Ensure the destination has capacity first, because draining into a region that cannot absorb the load produces two failed regions.

The design principle

Recovery load is qualitatively different from steady-state load, and must be capacity-planned separately. The system needs enough headroom, or enough admission control, to absorb the entire population reconnecting — and since provisioning for that is usually uneconomic, admission control plus cheap resumption is the real answer.