A platform must handle millions of concurrent connections per process cluster where most connections are idle most of the time. Which concurrency model fits, and what does the wrong choice cost?

The arithmetic

An operating system thread costs, at minimum, a stack allocation — commonly measured in hundreds of kilobytes to a megabyte — plus kernel bookkeeping and a share of scheduler overhead. A hundred thousand threads is tens of gigabytes of stack for connections that are, in aggregate, doing almost nothing.

Worse, context switching cost grows with thread count, so a process spends an increasing fraction of its time scheduling rather than working. The failure is not a cliff; it is a gradual degradation that looks like general slowness.

Why this workload specifically

Chat and presence connections are idle most of the time. A user's connection exists for hours and carries a message every few minutes. Allocating a thread per connection allocates resources proportional to connections, when the resource requirement is proportional to concurrent activity — which is orders of magnitude smaller.

Event-driven or lightweight-task models allocate per unit of work rather than per connection, so memory and scheduling track actual load.

What the wrong choice costs

• Memory as the binding limit, at a connection count far below what the CPU could serve.
• Latency variance from scheduler pressure, appearing as unexplained p99 spikes.
• Reduced blast-radius efficiency — fewer connections per process means more processes, and each process loss is a reconnect event.

What the right choice costs

It is not free, and the costs are real:

• Blocking anything blocks everything. A single synchronous file read, DNS lookup or CPU-heavy operation on the event loop stalls every connection it serves. This is the characteristic failure and it is genuinely hard to prevent as a codebase grows.
• Harder debugging. Stack traces do not show logical flow; causality must be reconstructed.
• Backpressure must be explicit. With threads, a slow consumer blocks naturally. In an event-driven model, work accumulates in queues until memory is exhausted unless bounds are imposed deliberately.

The complementary decision

Capacity is measured in connections, not requests per second, which changes everything downstream: CPU is the wrong autoscaling signal, since a process holding many idle connections has low CPU while approaching memory and file-descriptor limits. Scaling must key on connection count and memory.

And the same reasoning drives the deployment model: a tier whose processes hold hours-long connections cannot be deployed like a stateless one, which is why draining must be a protocol feature rather than an infrastructure setting.