A streaming service in the mould of Hotstar or JioCinema expects about 20 million concurrent viewers for a cricket final, with roughly 3 million joining in the first minute. Technical dashboards for CDN, API and player services are all green in rehearsal even when a carrier-specific problem breaks playback. Design the stream-start metric and the alerts that would catch it within 2 minutes.
Show the full answer Hide the answer
Requirements that drive the structure
The outcome the business sells is "press play, see video". The signal is stream-start success: of the playback attempts in a window, the share that reach first frame within a time limit. The targets are detection within 2 minutes, a view per carrier and region, and no per-viewer cardinality.
The design
- Client beacons. The player emits an
attemptat press and afirst_frameorfailwith a reason code. At 3 million joins in 60 s that is about 50,000 attempts per second, around 100,000 events per second with outcomes: easily handled by a stream processor. - Aggregate in 10-second windows by a bounded key: carrier (top 20 plus "other"), region (20), client platform (6), CDN (3). That is about 7,000 series, not millions. Anything with a user id or device id stays in logs.
- Two statistics per window: start success rate, and time-to-first-frame p95 (a histogram, since percentiles cannot be averaged).
- Synthetic players from a handful of vantage points, including two carrier networks, start a stream every 30 s. They are the only signal when real attempts are scarce, and the fastest check on a new CDN.
The hard part: ramps
During the join surge the absolute counts rise 50-fold in minutes, so "attempts below threshold" and "failures above N" are both meaningless. Alert on the rate, with a comparison to expectation: success rate below 97% for 3 consecutive windows in any segment with at least 2,000 attempts, or more than 3 points below the same segment's trailing 15-minute value. A minimum-volume rule stops tiny segments paging.
What it costs
Beacon traffic from 20 million players adds up if each sends a heartbeat every 10 s: 2 million events per second. Sample heartbeats; never sample attempt outcomes.
How it fails
Client beacons fail with the problem they report: a blocked network loses the beacon, so a total outage reads as a quiet segment. Page on a sudden drop in attempts per segment as well as on the failure rate. Beacons also arrive late; use event time with a 30 s lateness allowance.
What I would build first
The attempt and first-frame beacon with the five-key aggregation and one alert, then synthetic players, then the per-CDN breakdown. Rehearse by injecting failures for one carrier in a load test and measure the minutes to page.
When not to do this
For a service with a few thousand concurrent viewers, a server-side count of playback sessions started and ended, with a single success ratio, gives most of the value without client instrumentation.