A platform must deploy a service holding millions of long-lived connections. Which deployment strategy applies, and what capability makes it work?
Show the full answer Hide the answer
The capability that makes it work
Draining as a protocol feature, plus session resumption.
Before an instance is removed, it tells connected clients to reconnect on a server-scheduled stagger. Clients reconnect elsewhere with a sequence number, and the server replays only what was missed from a bounded buffer.
With those two properties, a rolling deployment becomes nearly invisible. Without them, every deployment is a reconnect storm — the platform has made its worst failure mode a scheduled event.
How the strategies compare here
Rolling — replace instances gradually. Fine with draining and resumption; a series of mass disconnections without them.
Blue-green — worse for this workload. Switching disconnects every session simultaneously, which is a larger version of the problem, and it doubles capacity cost during the release.
Canary — the right default for bounding blast radius, with the caveat that for a connection tier "traffic" means new connections, so the canary population grows slowly and the signal takes longer to accumulate.
The supporting properties
- Long, jittered connection lifetimes, so connections recycle continuously rather than only at deployment. This also fixes the standing problem that load imbalance persists once connections are established.
- Server-enforced reconnect backoff, since client-side politeness is ignored by old and third-party clients.
- Admission control on the connect path, prioritising resumes over cold connects during any surge.
- Capacity headroom sized for losing a failure domain, not for steady state.
The database constraint that does not change
Schema changes must be backward compatible across the release, because both versions run simultaneously and clients may take hours to reconnect. Expand-and-contract regardless of deployment strategy — the strategy does not change the schema requirement.
The measure
Reconnection rate during a deployment, compared with steady state. A deployment that produces a large spike has not solved the problem; one that produces a gentle elevation has.