A deployment drops 200000 WebSocket connections at once. Describe what happens next and how you would make it safe.

What is being tested

Whether you anticipate the reconnection storm, which is a self-inflicted second outage and a recurring cause of real incidents.

What happens

All 200,000 clients detect the disconnection within a second or two and reconnect immediately. That produces, simultaneously:

• 200,000 TLS handshakes, which are CPU-expensive.
• 200,000 authentication requests to the identity service.
• 200,000 session and state restorations, each likely hitting the database.
• 200,000 catch-up queries — "what did I miss since my last cursor" — which may be the most expensive part.

The newly deployed instances, which are cold, receive all of it at once. They fall over. Their clients reconnect. The storm repeats and now includes the clients from the failed retry. This is a self-sustaining loop that can outlast the deployment that started it by a long way.

Making it safe

1. Jittered exponential backoff on the client. Non-negotiable, and the single highest-value fix. Reconnect after a random delay in a growing window rather than immediately. This alone spreads 200,000 reconnections over minutes instead of seconds. Without jitter, backoff still produces synchronised waves.

2. Rolling deployment with a long pause between batches. Replace a small percentage at a time and wait for those connections to re-establish before continuing. The deployment takes longer; nothing falls over.

3. Connection draining with advance notice. Before shutting down, send clients a message telling them to reconnect, with a jittered delay. Clients migrate gracefully while the old instance is still serving, rather than being cut off. This turns a hard failure into a scheduled migration.

4. Cheap catch-up. The reconnection is expensive mostly because of what happens after it. A cursor-based protocol that returns a bounded set of missed messages, served from a cache or a purpose-built store, keeps recovery cheap. If catch-up means an unbounded query, recovery is the outage.

5. Admission control on reconnection. The server may reject with a "try again in N seconds" instruction, shedding load deliberately rather than collapsing.

6. Separate connection handling from application deployment. If a thin connection tier is deployed rarely and the application behind it deploys frequently, most deployments do not disturb connections at all. This is the structural fix, and it is what large real-time systems converge on.

The architectural context

WebSockets make an otherwise stateless tier stateful in one specific way: a connection is bound to one process for its lifetime. That has consequences teams consistently underestimate — a scale-out event does not relieve load on existing instances, because they only receive traffic when clients reconnect; and routing a message to a user requires knowing which server holds their connection, which needs either a shared registry or a bus every server subscribes to.

Real-time messaging products handle this by sharding connections so members of a channel are reachable from a bounded set of servers, treating presence as a deliberately approximate subsystem (high write volume, low value per write, nobody is harmed by a three-second-stale green dot), and treating very large channels as a special case rather than designing everything for them.

What a strong answer adds

Capacity is measured in concurrent connections and memory per connection, not requests per second — so load testing must simulate connection counts, not request rates, or the test measures the wrong thing entirely.