A service autoscales on CPU. During incidents it never scales out, even as latency triples. Why, and what would you scale on instead?

Why CPU never triggers

The service is latency-bound, not compute-bound. Its threads are waiting — on a database, a downstream API, a lock, or a connection pool — and waiting consumes no CPU.

So as the dependency slows, requests accumulate, latency rises, the queue grows, and CPU stays flat or falls. The autoscaler is behaving exactly as configured and the configuration measures the wrong thing.

There is a second-order effect worth naming: as latency rises, each request occupies a thread for longer, so fewer requests are processed per second and CPU may actually decrease while the service is failing. The metric moves in the wrong direction.

What to scale on

Concurrency or in-flight requests per instance — the best general answer. It maps directly to Little's Law and to saturation, and it rises precisely when a service is struggling regardless of whether the cause is compute or waiting. Cloud load balancers expose per-target request counts for exactly this.

Queue depth, or oldest-message age, for worker pools consuming from a queue. Age is the better target because it maps directly to user-visible staleness.

Requests per second per instance, acceptable only if per-request cost is stable — it breaks when the request mix changes.

A latency target, which is intuitive and needs care: scaling out does not help if the bottleneck is downstream, and it can make things worse by adding concurrency to an already-saturated dependency.

The caveat that matters here

Scaling out may be the wrong response entirely. If the bottleneck is a shared database, adding instances increases concurrency against it and deepens the problem. Autoscaling protects against insufficient local capacity; it cannot fix a downstream constraint.

Which is why the design should pair autoscaling with a maximum, a circuit breaker on the slow dependency, and load shedding — so that when the constraint is elsewhere, the service degrades deliberately rather than multiplying pressure.

What a strong answer adds

Noting the two settings that determine whether autoscaling behaves well once the metric is right: a warm-up period longer than time-to-useful, or the scaler adds more instances while the previous ones are still starting and then over-corrects; and scale-in slower than scale-out, because being briefly over-provisioned costs far less than being under-provisioned when a spike returns.