A service handles 100 requests per second with a pool of 10 worker threads. The team raises the pool to 100 threads. Throughput stays at roughly 100 rps and p99 latency gets much worse. What is the primary reason?

The mechanism

Little's Law: L = λW. Concurrency L is what the thread pool sets. Throughput λ equals L divided by W, the time each request spends in the system — but only if W stays constant.

It does not. Once the actual constraint is saturated, extra threads do not get served faster; they wait. L rises tenfold, W rises tenfold with it, and λ does not move. The extra concurrency has been converted into queueing, which is another name for latency.

The arithmetic

At 100 rps with 10 threads, W is 100 ms. Suppose the real constraint is a database connection pool of 10. With 100 threads, 90 of them are blocked waiting for a connection at any moment. L is now 100, λ is still 100 rps, so W is 1,000 ms. The service is exactly as fast and every user waits ten times longer.

Past the knee it gets worse rather than flat: each request now also pays scheduling and cache-pollution costs, so λ declines.

Why the other options fail

• Context switching is real and it is second-order here. A switch costs on the order of a microsecond; at these rates it is noise. It becomes the dominant term at thousands of runnable threads, which is a different regime, and it is not why the first doubling failed to help.
• An operating system cap exists in the form of thread limits and stack address space, but nothing caps a process at 100 threads. This option mistakes a hard limit for the soft ceiling that is actually binding.
• A copy of memory per thread is simply wrong: threads share the address space, which is what distinguishes them from processes. The real per-thread cost is the stack — commonly 1 MB of reserved virtual address space with far less resident — and it matters at tens of thousands of threads, not at 100.

What to do instead

1. Find the constraint. Walk the resources a request touches and look for the one that is saturated: connection pool, disk, a downstream service's concurrency, a lock.
2. Raise it, or accept it. A bigger connection pool is sometimes right and sometimes moves the constraint into the database, which is worse.
3. Limit concurrency just past the knee. A queue in front of a bounded pool gives you a place to shed and a visible signal, and it keeps latency for admitted requests bounded. Adding threads past the knee preserves neither throughput nor latency; limiting concurrency preserves both for the requests you admit.

When more threads is the right answer, and when not to add any

When the work is genuinely blocked on independent things. One hundred concurrent calls to a hundred different hosts share nothing, so concurrency converts directly into throughput and the only limits are memory and sockets. The failure in the question comes from shared contention; without it, the intuition holds.