A live platform's dashboards show mean latency, which stays flat during an incident where many users experience severe delays. Why do averages hide this, and what should be measured?
Show the full answer Hide the answer
Why averages hide it
An average is dominated by the bulk of the distribution. If 95% of requests are fast and 5% take twenty times longer, the mean moves modestly — and the mean was never the user experience anyway, because no user experiences the mean.
Worse, in a system where a user session involves many requests, the probability that a session encounters at least one slow request is far higher than the per-request slow rate. A 1% slow-request rate across fifty requests means about 40% of sessions hit one.
What to measure instead
Percentiles: p50, p90, p99, p99.9. The tail is where the pain is, and the shape of the distribution is the diagnostic. A p50 that is flat while p99 doubles tells you something specific — usually contention, garbage collection, cache misses on a subset, or one bad instance.
Histograms rather than pre-computed percentiles per instance. This is a technical detail with real consequences: percentiles cannot be averaged. Taking the mean of each instance's p99 gives a number with no meaning. Histograms can be summed and the percentile computed from the aggregate, which is correct.
Segmented percentiles. By region, device class, network type and — for a streaming platform — by stream size, because load concentrates on the largest streams and their users are the ones who notice.
A max, or a very high percentile, for anything user-facing. p99.9 catches the failures that affect a small number of users completely, which are invisible at p99 and matter enormously to those users.
The counting mistake that accompanies it
Averaging rates across instances. A fleet where one instance is failing every request and forty are healthy shows a low aggregate error rate. Error rate should be computed from summed counters, and per-instance outliers should be alerted on separately — because one bad instance in a fleet is a common failure that aggregates are designed to hide.
The practical rule
Alert on percentiles and error budgets; investigate with histograms and traces; never make a decision from a mean. The average exists to make a graph look calm, and during an incident that is precisely the wrong property.