A service scales its application instances from 20 to 200 and the database begins refusing connections. What happened, and what is the correct architecture?

What happened

Connection count scaled with instance count, not with actual concurrency. Each instance holds a pool — say 20 connections — so 20 instances held 400 and 200 instances hold 4,000, which exceeds the database's limit.

The arithmetic is the problem: pools are sized per instance and the instance count is elastic, so total connections grow linearly with a number that autoscaling changes freely.

Why it is worse than a limit breach: database connections are expensive. Each consumes memory for buffers and session state, and in process-per-connection engines each is an operating system process. Thousands of mostly-idle connections consume memory the database needs for caching, so performance degrades well before the hard limit is reached.

The correct architecture

1. A connection proxy between applications and the database. Applications connect to the proxy freely; the proxy maintains a small pool of real database connections and multiplexes. Total database connections then track concurrency, not instance count. This is the structural fix.

2. Size the real pool using Little's Law. For 2,000 queries per second at 20 ms each, 40 connections suffice. Provisioning hundreds wastes database memory for no throughput benefit. The relation gives the number directly rather than by guesswork.

3. Transaction-level rather than session-level pooling where the application permits it, so a connection is held only for the duration of a transaction rather than for the session. This dramatically increases the number of clients one connection can serve — with the constraint that session-scoped features (temporary tables, prepared statements, session variables) stop working, which must be verified.

4. Per-tenant or per-service connection limits at the proxy, so one noisy service cannot consume the whole pool.

5. Bounded queueing at the proxy with fast rejection. When no connection is available, wait briefly and then fail fast. Unbounded waiting converts a database capacity problem into an application-wide stall.

The counter-intuitive part

A smaller pool often yields higher throughput. Beyond the point where the database's cores are busy, additional concurrent queries add contention — lock waits, buffer pressure, context switching — and total throughput falls. Teams respond to slowness by increasing pool size, which makes it worse.

The correct pool size is derived from the database's capacity to do concurrent work, not from the application's desire to issue it.