Design the connection layer for a chat platform holding tens of millions of concurrent WebSocket connections, where users belong to communities ranging from three people to a million. What are the main architectural decisions, and which one causes the most incidents?

The decisions that matter

1. Connections are stateful, so the gateway is a partitioned tier of its own. A connection is pinned to a process for its lifetime. That makes the gateway fleet fundamentally different from a stateless HTTP tier: you cannot load-balance an existing connection, and losing a process loses every session on it. Gateways therefore hold only session state, and every durable concern lives behind them.

2. Fan-out topology is chosen per community size, not globally. A three-person channel and a million-member channel are different problems wearing the same UI. Small channels can push directly to every connected member. Large ones cannot — a single message would produce a million writes. Large communities need a different path: subscribe-on-read, per-channel fan-out services, or aggregation nodes between the publisher and the connections.

3. Presence is the expensive part, not messages. Messages are user-generated and therefore bounded by human typing speed. Presence changes with every connect, disconnect, status change and activity update, and naive presence is O(members) per change. In a million-member community, one user going idle must not produce a million notifications. The answer is that presence is sampled and aggregated for large communities and precise only for small ones — visible counts become approximate above a threshold, which almost nobody notices.

4. Session resumption is a first-class protocol feature. Clients on mobile networks disconnect constantly. If reconnection means "fetch everything again", the system's steady-state load is dominated by reconnects. Sessions carry a sequence number; on resume the server replays only what was missed from a bounded buffer, and falls back to a full state fetch only if the gap is too large.

The decision that causes the most incidents

Reconnect storms. If a gateway process or an availability zone drops, every affected client reconnects — and they do it immediately, simultaneously, and to the remaining healthy capacity, which is now serving its own load plus the displaced load plus a full state fetch for each displaced session.

This is a self-amplifying failure: the surge causes more failures, which cause more reconnects. The defences are structural and must exist before the event:

• Jittered, capped exponential backoff in the client, enforced server-side by rejecting early reconnects.
• Resume rather than refetch, so a reconnect is cheap.
• Admission control on the connect path, prioritising resumes over cold connects.
• Capacity headroom sized for losing a failure domain, not for steady state.

The general lesson: in any system with millions of persistent connections, the dominant failure mode is not the disconnection — it is the coordinated reconnection.