A live platform has two weeks before a major event expected to bring ten times normal traffic. What should the readiness work consist of, in priority order?
Show the full answer Hide the answer
Priority order
1. Establish the actual constraint. Load test to failure — not to target — and produce the ordered list of what saturates first, second and third. Everything else depends on this list, and without it the two weeks will be spent optimising the wrong things.
2. Pre-scale, do not rely on autoscaling. The event is scheduled, so capacity should be in place beforehand. Autoscaling responds in minutes; the audience arrives in seconds. Include the stateful tiers, which usually cannot scale quickly at all — and check third-party rate limits, which no internal scaling raises.
3. Verify the degradation ladder rung by rung, under load. Each capability that will be shed must actually be shed and the resulting experience observed. The characteristic finding is that shedding one feature breaks another because the fallback was never run.
4. Freeze non-essential changes. A change freeze starting several days before is cheap and removes the largest single source of incidents. Keep an expedited path for genuine fixes.
5. Check every fixed limit against projected load. Connection pools, queue depths, partition counts, file-descriptor limits, third-party quotas, certificate expiry dates. These are invisible in normal operation and decisive at 10x, and each is a configuration change that takes minutes to make and hours to diagnose during the event.
6. Rehearse the human response. On-call staffing for the window, runbooks verified against reality, escalation paths tested, communication templates prepared, and a decision-maker identified for the degradation calls that automation does not cover.
7. Prepare the recovery. Post-event traffic decline, cache re-warming, and the return of any component taken out of rotation. Second outages happen during recovery, with cold caches and empty pools.
The specifics for live streaming
- Connection-based capacity planning, since CPU is the wrong signal for a connection-heavy tier.
- Chat fan-out separated from chat analytics, so counting messages cannot affect delivering them.
- Adaptive bitrate as a first-class degradation control — reducing quality is a far better failure mode than dropping viewers, and it must be tested as a deliberate control rather than assumed.
- CDN capacity confirmed with the provider, since a 10x event is their capacity question too.
The output
Not a green dashboard. A written list of what will be shed, in what order, triggered by what signal, with the expected user experience at each rung — reviewed with the business, and rehearsed. That document is the deliverable, and if it does not exist two days before the event, the readiness work is not finished.