advanced 2 min answer

A streaming service in the mould of Hotstar or JioCinema expects about 20 million concurrent viewers for a cricket final, with roughly 3 million joining in the first minute. Technical dashboards for CDN, API and player services are all green in rehearsal even when a carrier-specific problem breaks playback. Design the stream-start metric and the alerts that would catch it within 2 minutes.

business metricsstream start successlive eventssynthetic monitoringrum
Show the full answer Hide the answer

Requirements that drive the structure

The outcome the business sells is "press play, see video". The signal is stream-start success: of the playback attempts in a window, the share that reach first frame within a time limit. The targets are detection within 2 minutes, a view per carrier and region, and no per-viewer cardinality.

The design

  1. Client beacons. The player emits an attempt at press and a first_frame or fail with a reason code. At 3 million joins in 60 s that is about 50,000 attempts per second, around 100,000 events per second with outcomes: easily handled by a stream processor.
  2. Aggregate in 10-second windows by a bounded key: carrier (top 20 plus "other"), region (20), client platform (6), CDN (3). That is about 7,000 series, not millions. Anything with a user id or device id stays in logs.
  3. Two statistics per window: start success rate, and time-to-first-frame p95 (a histogram, since percentiles cannot be averaged).
  4. Synthetic players from a handful of vantage points, including two carrier networks, start a stream every 30 s. They are the only signal when real attempts are scarce, and the fastest check on a new CDN.

The hard part: ramps

During the join surge the absolute counts rise 50-fold in minutes, so "attempts below threshold" and "failures above N" are both meaningless. Alert on the rate, with a comparison to expectation: success rate below 97% for 3 consecutive windows in any segment with at least 2,000 attempts, or more than 3 points below the same segment's trailing 15-minute value. A minimum-volume rule stops tiny segments paging.

What it costs

Beacon traffic from 20 million players adds up if each sends a heartbeat every 10 s: 2 million events per second. Sample heartbeats; never sample attempt outcomes.

How it fails

Client beacons fail with the problem they report: a blocked network loses the beacon, so a total outage reads as a quiet segment. Page on a sudden drop in attempts per segment as well as on the failure rate. Beacons also arrive late; use event time with a 30 s lateness allowance.

What I would build first

The attempt and first-frame beacon with the five-key aggregation and one alert, then synthetic players, then the per-CDN breakdown. Rehearse by injecting failures for one carrier in a load test and measure the minutes to page.

When not to do this

For a service with a few thousand concurrent viewers, a server-side count of playback sessions started and ended, with a single success ratio, gives most of the value without client instrumentation.