A platform's rightsizing programme reduces instance sizes based on average utilisation, and incidents follow. What was wrong with the method?

What was wrong

Averages hide the peaks that determine required capacity. A service averaging 20% CPU may reach 85% during its daily peak, during a batch window, or when a peer instance fails and its load redistributes. Sizing to the average removes the headroom that absorbs all of those.

Three specific omissions:

1. Percentiles, not means. Size to the p95 or p99 of utilisation over a representative period, including peak days. Anything else is sizing to a condition that does not occur.

2. Failure headroom. If losing one instance in three means the survivors take 50% more load, capacity must accommodate that. Rightsizing to steady state removes the ability to survive a normal failure — turning a routine event into an incident.

3. The resource that actually binds. CPU is measured because it is easy. The binding constraint is frequently memory, connections, file descriptors or I/O, and a "rightsized" instance can be CPU-appropriate and memory-starved.

What else the method usually misses

• Burst behaviour. Some instance types provide burst credits that accumulate and exhaust. An instance that looks adequately sized may be running on credits that deplete under sustained load.
• Startup and warm-up. A smaller instance takes longer to warm caches and connection pools, which matters during scaling and recovery.
• Latency effects below saturation. Queueing delay rises non-linearly as utilisation increases, so an instance at 70% has materially worse latency than at 40% even though neither is saturated. Rightsizing purely on utilisation ignores the latency consequence entirely.
• Correlated demand. Reducing every instance simultaneously removes the aggregate buffer that individual instances were implicitly providing each other.

The method that works

1. Measure percentile utilisation across a full cycle, including peaks and batch windows.
2. Identify the binding resource, not the convenient one.
3. Add explicit failure headroom sized for the failure domain you intend to survive.
4. Change incrementally and observe, one step at a time rather than in a fleet-wide sweep.
5. Validate against the latency SLO, not against utilisation — the target is set by latency requirements, and the resulting headroom is a design parameter rather than waste.
6. Automate the reversal, so a rightsizing that degrades service is undone quickly.

The framing worth defending

Headroom is not waste; it is the buffer that keeps latency bounded and failures survivable. Presenting it that way — with the queueing arithmetic behind it — is necessary, because utilisation is the number leadership watches and "we run at 60% on purpose" needs an argument.