advanced 2 min answer

Disney+ Hotstar reported a peak of about 59 million concurrent viewers during the India vs Australia Cricket World Cup final in November 2023, ahead of about 53 million at the semi-final days earlier. A platform carrying an event of that shape has a continuity plan with a 30-minute recovery time objective and a documented escalation to a named executive. The ad-decision service starts returning in 4 seconds instead of 40 ms at the twelfth over. What happens, and what does it say about the plan?

live-eventsdegradationrtopeak-loadcontinuity-plan
Show the full answer Hide the answer

Minute by minute

Second 0 to 30. Ad decisioning slows. The player's ad-break call now blocks for 4 seconds. Nothing has failed, so no alert fires on error rate.

Minute 1 to 3. The first ad break arrives. Tens of millions of clients call simultaneously; each holds a connection 100 times longer than the capacity model assumed. Connection pools and worker threads on the stitching tier fill, because the tier was sized on throughput at 40 ms and concurrency is the product of rate and duration. Little's law does the damage here, not the ad service.

Minute 3 to 6. Clients time out and retry. Offered load rises while effective capacity falls. The playback path, which shares the same edge fleet, starts shedding. Viewers see buffering, which is the only symptom anyone outside engineering recognises.

Minute 6 onwards. Someone proposes failing over the ad service. An executive is paged. The match continues.

Where it amplifies

The amplifier is the shared fleet and the client retry, not the slow dependency. A dependency that is slow rather than down is the failure mode that defeats most continuity plans, because the plan's triggers are written in terms of availability and the dependency is available.

Why the 30-minute objective is the wrong unit

A recovery time objective assumes the work can be done later. A live event has no later. The 12 minutes it takes to reach the executive, decide and fail over is roughly 20 overs of a match that will not be replayed. Measured against the business event, a 30-minute RTO on a three-hour one-shot is not a target; it is an acceptance of losing a sixth of the revenue.

The right unit is the fraction of the event the plan permits you to lose, which converts straight into design budget: if the business will accept 2%, the decision has to be automatic because no human chain fits in 3 minutes.

What stops it

  • A degradation ladder agreed before the event, with the commercial owner's signature on each rung: ad decisioning falls back to a pre-cached house creative, then to no break at all. Both cost revenue and both cost less than buffering.
  • A trigger on the ladder, not on an alert. Rung one fires automatically when ad-call p99 crosses 500 ms for 20 seconds. No human in the loop.
  • Deadline budgets on the client call, so a slow ad decision is a fast failure rather than a held connection, and bulkheads so the ad path cannot consume the playback fleet's capacity.
  • A rehearsal at the previous fixture. The semi-final was the dress rehearsal for the final, and a platform that does not inject latency during it is choosing to discover this live.

When this is the wrong answer

For a system where the work genuinely is deferrable, a batch settlement, an internal report, an order that can be placed an hour later, a 30-minute RTO with a human decision is sensible and the automation would be over-built. The distinction is whether the clock belongs to you or to the world.