How should a platform with extreme scheduled peaks decide how much capacity headroom to hold, and what evidence should drive it?

What headroom is actually for

Three separate things that are often conflated:

• Forecast error. The peak will be larger than predicted some fraction of the time, and the headroom covers that distribution rather than a single expected value.
• Failure tolerance. Losing an instance or a zone at peak must not cause failure, so capacity must be sufficient after the loss — not before it.
• Degradation margin. Systems become less efficient under load: cache hit rates fall, garbage collection increases, connection contention rises. Throughput per instance at peak is lower than throughput per instance in a load test, and headroom absorbs the difference.

Sizing for only the first of these under-provisions for the other two, which is the most common capacity mistake.

The evidence that should drive it

• The distribution of past peaks against forecasts, not the average error. If the forecast has been 20% low once in ten events, the headroom decision is about that tail.
• The measured relationship between load and efficiency, from load tests that go beyond the target rather than stopping at it. The interesting number is where throughput stops increasing, and most load tests never find it.
• The cost of failing versus the cost of the capacity. For an event carrying a large share of revenue with a hard deadline, headroom is cheap. For a routine peak, it is not.
• What the shedding policy actually protects. Good shedding reduces the required headroom, because the consequence of under-provisioning becomes degradation rather than failure. These two decisions are linked and are usually made by different people.

The dependency check

Capacity planning that covers only the application tier is incomplete. The database connection limit, the downstream provider's rate limit, the notification vendor's quota, the identity provider's throughput and the NAT gateway's port capacity all have ceilings, and the peak finds whichever is lowest.

The planning exercise must enumerate every ceiling on the critical path and confirm each has margin — which is tedious and is the difference between a plan and an aspiration.

The discipline that improves it

Compare forecast against actual after every event, and record it. Capacity planning without a feedback loop does not improve, and the same shortfall repeats. The forecast error distribution is the single most valuable artefact a team with recurring peaks can build.