A retailer expects 20× normal traffic for a two-hour sale launch. Plan the readiness.
Show the full answer Hide the answer
Forecast, then convert to resources properly
Take the multiple from previous events and business projections, and plan for the forecast plus a margin, since being short during the event is unrecoverable.
Convert demand to resources using measured load-versus-resource curves, not a linear ratio. Cache hit rates fall as the working set grows; database performance degrades non-linearly with volume; and queueing delay grows as 1/(1−utilisation), so the last increment of load costs disproportionately.
Pre-scale, because autoscaling cannot help here
The spike arrives faster than instances start. The first minutes — the ones that matter most — fail while the system catches up. Provision to the forecast peak in advance and treat autoscaling as headroom above it.
Scale the things that are not instances, which is where these plans usually fail:
Database connections and pooler capacity. Third-party quotas — payment processors and identity providers have their own rate limits and require advance notice. Cloud service quotas, which often need a support request days ahead. Cache capacity so the working set still fits. Queue consumer counts.
Warm everything
Cold caches at the moment of a 20× spike produce a thundering herd against the origin. Warm the caches, pre-open connection pools, and pre-scale the CDN with the provider where that is an option.
Decide the degradation modes in advance
Rank features with the business before the event. Checkout and payment must work. Recommendations, reviews, live stock counts at nearby stores and personalisation are droppable — each with a defined fallback and a feature flag already in place and tested.
Add load shedding by priority at the edge, so that if capacity is exceeded the system serves checkout and rejects the rest, rather than degrading everything equally. And a queue-based waiting room for the most extreme case, which converts an overload into an orderly wait.
Test it
Load test at forecast peak plus margin against the actual production configuration, not a scaled-down approximation. Then stress test past it to learn the failure mode, and verify recovery.
Rehearse the degradation flags — a fallback that has never executed is a hypothesis.
Operate it
Change freeze for the surrounding period. A war room with the runbook, the flags and the decision authority present rather than on call. Business metrics on the primary dashboard — orders per minute is the number that matters, and it detects failures that leave every technical signal green.
Agreed abort criteria decided beforehand: at what point do you disable a feature, enable the waiting room, or shed traffic. Deciding that during the event is how the two hours are lost.
Afterwards
Capture the actual curves and compare them with the forecast. That comparison is the most valuable input to the next event, and it is routinely skipped.