Efficient Concurrency Point

Every shared resource has a concurrency level at which it is most productive. Below it, capacity is idle. Above it, added concurrency increases the time each unit of work takes by more than it increases parallelism, and total throughput falls.

For a database this shows up as the counter-intuitive result that a smaller connection pool produces higher throughput and lower latency than a larger one.

Why it matters

It produces one of the most reliable self-inflicted performance spirals: latency rises, the team concludes the pool is too small, increases it, contention worsens, latency rises further. The intervention that would help is the opposite of the intuitive one.

Implementation patterns

• A smaller pool with a queue in front of it. Requests wait in the application rather than overwhelming the database, throughput is higher, latency is lower and more predictable, and the queue makes the constraint visible rather than hiding it inside the database's response time.
• Size from the resource's characteristics, typically a small multiple of the database's core count, rather than one connection per concurrent request. The pool exists to keep the resource busy, not to avoid ever waiting.
• Separate queue time from service time in application metrics. High latency with low service time means waiting for a connection, and that distinction determines whether the pool should grow or shrink.
• Test the direction empirically: if throughput is flat while concurrency rises, the resource is past its efficient point and the pool should shrink.
• Reason at fleet level. Pool size is per instance and the resource sees the sum — twenty instances at fifty each is a thousand connections. Autoscaling the fleet silently scales the connection count, which is how a scaling event becomes a database incident.
• Use a pooler in front of ephemeral compute, since serverless and autoscaled workloads have no natural ceiling on client count.

Industry example

Consumer platforms such as ShareChat and Pratilipi with heavy read traffic against relational stores encounter this at every scaling step, and it is the reason a scaling event can degrade rather than improve performance. The same shape appears with thread pools, worker pools and any bounded downstream — the resource is not the only thing being shared; the contention for it is too.

Failure scenarios

• Increasing the pool in response to latency, which worsens contention.
• Per-instance pools summed across an autoscaling fleet, exceeding the resource's limit invisibly.
• No queue-time metric, so the constraint is invisible and the diagnosis is guesswork.
• Unbounded concurrency at a shared resource, converting a capacity limit into a collapse.
• Sizing by intuition, which reliably overshoots.

Trade-offs

A smaller pool means requests wait, and waiting is visible as latency even though total throughput is better. That is uncomfortable to explain and it is the correct trade: the alternative is worse latency for everyone plus a risk of tipping the resource into collapse.

The genuine tension is with burst absorption — a pool sized for the efficient point has no slack for a short spike, which is what the queue provides. Bounded queue plus modest pool is the shape, and the queue's bound is what prevents a latency problem becoming a memory problem.

Interview question

"Latency is up, CPU is low, and someone has proposed doubling the connection pool. Explain what you expect to happen, what you would measure first, and what you would do instead."