Review this readiness plan for a live cricket final. Peak concurrency is forecast at 30 million viewers. The plan - pre-scale every tier to 2x forecast four hours before kickoff; an autoscaler built in-house that reacts to a custom concurrency metric; a login queue in front of authentication; cache warming for the home feed; and a war room of 40 people. What would you remove, what would you change, and what would you keep even though it looks wasteful?
Show the full answer Hide the answer
What the event actually requires
The forecast is the wrong place to start. The property that matters is the shape of the arrival curve, not its peak: a cricket final gains millions of viewers in the minutes around the toss and in the final over, and loses them in seconds at the end. Disney+ Hotstar has publicly reported peak concurrency of 25.3 million viewers for the 2019 World Cup semi-final and around 53 million for the 2023 semi-final, figures the platform itself announced. The engineering content of those numbers is the gradient: tens of thousands of new sessions per second, arriving faster than any reactive system can provision capacity.
That single fact decides most of this review.
What I would remove
- The in-house autoscaler. At this gradient a reactive loop is not merely late, it is harmful: it adds instances during the ramp, each of which must warm caches and open connections, so the fleet is at its coldest exactly when demand peaks. Pre-scaling already covers the event. Keep the autoscaler for the post-match decay and nothing else.
- The 40-person war room. Beyond roughly a dozen people the room's cost is coordination: parallel hypotheses, duplicated commands, and a bridge too noisy to hear the one person who knows. Name an incident commander, four tier owners, and a scribe. Everyone else is reachable, not resident.
The one change that matters
Add a written degradation ladder, ordered and pre-authorised, with the trigger for each rung. Without it, the war room invents policy under load. The ladder for a live stream is roughly: drop the personalised home feed to a static editorial one, then stop writing non-essential engagement events, then reduce the top rendition offered to new joiners, then disable chat and reactions, then serve a cached scoreboard page. Each rung names its signal and threshold (for example: concurrency above 80% of provisioned manifest capacity, or origin offload below 95%) and the person who may pull it without further approval.
The second change is smaller and still structural: cache warming must cover the manifest and segment paths, not the home feed. The home feed is the first thing the ladder sacrifices.
What I would keep, though it looks wasteful
The 2x pre-scale and its idle cost. For a four-hour window, double capacity on a fleet that normally costs a few thousand an hour is a few thousand pounds of insurance against the event the company bought broadcast rights for. The asymmetry is the argument: unused capacity costs money, missing capacity costs the audience and the rights fee.
Keep the login queue too, and make it the admission-control point for everything, not just authentication. A queue that holds users for 40 seconds with an honest position indicator is a far better experience than a stream that buffers for everyone, and it is the only mechanism that converts an unbounded arrival rate into a bounded one.
How I would argue this in the review
Lead with the gradient, not the peak, and ask one question of every item on the plan: does this act before the ramp or during it? Anything that only acts during the ramp is a liability at this gradient. Then rehearse: a dress rehearsal on a lesser match, with the ladder pulled deliberately, is what turns the plan from a document into a drill.
When not to pre-scale at all
For an event whose arrival curve is measured in hours rather than seconds - a sale that opens at nine and builds through the morning, a product launch with an email send spread over a day - reactive scaling is the cheaper and better answer, and pre-scaling 2x for twelve hours is waste. The rule is the ratio of ramp time to provisioning time: pre-scale when new capacity cannot be hot before the peak arrives, and let the autoscaler work when it can. Below roughly a few hundred thousand concurrent users most of this plan is over-built; one pre-scaled tier, a bounded queue and a written degradation ladder cover it.