advanced
3 min answer
An annual sale will bring 10× normal traffic in a 72-hour window. What preparation should begin months in advance, and what is prepared but hopefully never used?
Show the full answer Hide the answer
Months in advance
- Determine the actual multiple from last year's data plus growth, and plan for meaningfully above it. Forecasts for promotional events are unreliable in a predictable direction.
- Load test in production, at full scale, using real traffic patterns rather than synthetic uniform load. Testing in a scaled-down environment tells you almost nothing, because the failures at 10× are emergent — connection limits, lock contention, cache eviction behaviour, cardinality effects — and none of them appear at 1×.
- Test the whole journey, not individual services. The failure will be an interaction: a cache miss storm triggering a database saturation triggering a retry cascade.
- Identify the bottleneck, fix it, repeat. Every load test finds one bottleneck; removing it reveals the next, and the useful number is how many iterations were completed, not the peak achieved once.
- Verify capacity availability with providers, including instance types, database sizes, and third-party rate limits. A payment provider's rate limit is a hard ceiling on order throughput regardless of your own capacity, and it is negotiated months ahead.
- Review every third-party dependency for its behaviour at 10× and its failure mode.
Weeks in advance
- Warm standby capacity, provisioned and running, since scaling during the event is unreliable and slow.
- Pre-warm caches with catalogue, pricing and promotional data.
- Freeze changes, with a defined exception process. Changes shortly before a peak are the leading cause of failure at peaks.
- Rehearse the degradation ladder in production, verifying each switch works and measuring the capacity it releases.
- Game day the incident response: who commands, who can pull switches, how decisions are made, what is communicated.
Inventory reservation, which is where correctness fails
At 10× traffic with limited stock, the contended row is the failure point. The options:
- Reservation at cart-add with a TTL, so stock is held briefly and released if not purchased.
- An in-memory reservation service for high-demand items, making decisions at memory speed with asynchronous persistence — this is what makes flash sales work at all.
- Pre-split counters into buckets to remove single-row contention.
- An explicit oversell policy. Perfect accuracy under extreme contention costs latency; the business must decide whether a small oversell rate with fast checkout is better than strict accuracy with a slow one, and that decision belongs to them.
Traffic shaping
Admission control with a queue at the front door, so excess arrivals wait with an honest message rather than receiving errors. Prioritise by journey stage — a user in checkout is far more valuable than one browsing — and protect the checkout path with reserved capacity nothing else can consume.
Prepared and hopefully never used
- The kill-switch list, ordered, owned, rehearsed.
- A static fallback site: catalogue browsing from a fully cached, read-only version, so that a catastrophic backend failure degrades to a browsable store rather than an error page.
- A queue-everything mode where orders are accepted into a durable queue and confirmed asynchronously, decoupling order intake from fulfilment capacity. This preserves revenue when the fulfilment path is saturated, and it is the single most valuable emergency capability.
- A tested rollback for every change made in the preceding month.
- A communication plan for customers and internal stakeholders, written in advance.