A chat platform's Layer 7 proxy tier holds millions of WebSocket connections. Every proxy deployment disconnects everyone. What would you change?
Show the full answer Hide the answer
The diagnosis
Two independent problems have been combined into one.
First, the layer is wrong for the traffic. Layer 7 proxying earns its cost through per-request decisions — routing, retries, header inspection. A WebSocket connection is one logical request lasting hours, so those capabilities are used once and then the proxy spends hours holding state for a connection it is not making decisions about. The proxy tier becomes memory-bound on connection count rather than CPU-bound on request rate.
Second, and more damaging: the deployment model treats a stateful tier as stateless. Rolling a proxy disconnects every session it holds, which produces a reconnect storm caused by a routine deployment. The platform has made its worst failure mode a scheduled event.
What to change
1. Move connection termination to Layer 4 for the WebSocket path. Connection-level forwarding with minimal per-connection state handles far higher connection counts per node, and TLS terminates at the application tier where the session logic already lives.
2. Separate the connection tier from the routing tier. Keep Layer 7 for the HTTP API, where per-request routing is genuinely valuable, and give persistent connections their own path. One tier serving both means every deployment decision is a compromise.
3. Make draining a protocol feature, not an infrastructure one. Before a node is removed, it sends clients a "reconnect, and here is when" message. Clients reconnect on a server-scheduled stagger to another node. This turns a simultaneous mass disconnection into a controlled, spread migration — and it is the single highest-value change available.
4. Session resumption, so a reconnect replays a bounded buffer from a sequence number rather than refetching full state. Combined with staggered draining, a rolling deployment becomes almost invisible.
5. Long, jittered connection lifetimes. Connections are recycled continuously on a randomised schedule rather than all at deployment time. This also solves the standing problem with long-lived connections — that load imbalance persists indefinitely once connections are established — because there is a steady, gentle rebalancing.
6. Server-enforced reconnect backoff. Client politeness is a request; old clients and third-party clients ignore it. The server must reject early reconnects.
What not to change
Do not remove the proxy layer entirely. TLS termination, DDoS mitigation, IP-level rate limiting and observability at the edge are all worth having. The problem is not the existence of a proxy; it is proxying the wrong protocol at the wrong layer with the wrong deployment semantics.
The principle
Balancing granularity should match the unit of work, and deployment semantics must match the connection lifetime. A tier holding hours-long connections cannot be deployed like a tier holding millisecond requests, and no amount of capacity compensates for getting that wrong.