YouTube ran an opt-in experiment called Feather from 2009 that cut the video watch page from as much as 1.2 MB and dozens of requests down to 98 KB and 14 requests. The engineer who built it later published the outcome - after a week of data, average page latency had gone up. What happened, and what should it change about how a team reads its own field performance data?
Show the full answer Hide the answer
What the data said
A page roughly twelve times smaller, with a fraction of the requests, measured slower on average across the experiment population. Every instinct says the measurement is broken.
The explanation
It was not broken. Plotting by region, the published account describes a disproportionate rise in traffic from Southeast Asia, South America, Africa and remote parts of Siberia, where the average page load on Feather was over two minutes. For those users the old page had never finished loading at all.
The lighter page did not slow any existing session. It added sessions that had never existed in the data, and those sessions were the slowest in the world.
The mechanism, named
A field metric is computed over the population that completes the measurement. Change the experience and you change who completes it - so the metric is a ratio whose denominator you just edited. The aggregate moved because the composition moved, and no individual user got worse.
This is the mirror image of the artefact most teams know. Beacon survivorship makes a worsening experience look better, because the users who abandon never report. This makes an improving experience look worse, because users who previously could not report now can. Both are the same bug: treating an aggregate over a shifting population as a measurement of the experience.
What it means for your dashboards
- Put a volume counter next to every latency metric. If sessions in a region rose 10% this week, this week's percentile is not comparable to last week's, and nothing else you conclude is safe.
- Segment before concluding - country, connection class, device class. The aggregate is the one number guaranteed to hide a population change.
- Compare cohorts, not periods. A holdout on a fixed population, or the same users before and after, answers "did the experience get better". Period-over-period p75 answers a question nobody asked.
- Watch the tail and the business metric together. Core Web Vitals are assessed at the 75th percentile, so an influx of 10% very slow sessions can move the assessment on its own while plays, sign-ups or conversions rise.
What the instrumentation costs
Segmented percentiles are not free: country x connection class x device class multiplies the series you store and the queries you run, so teams default to the aggregate to save money. That saving is what breaks the analysis - an unsegmented latency metric fails silently during exactly the changes you most want to evaluate, and the gap is only discoverable after the experiment, when the data you needed was never collected.
When not to copy the conclusion
The lesson is not "ship a 98 KB page", and it is certainly not "a rising average is fine". A product whose audience is already on fast connections has no hidden population to unlock, and the same byte cut there will show the improvement you expected - if it does not, you have a regression, not a composition effect.
The claim "our latency got worse because we reached more people" is only available to a team that can prove it, which requires per-segment and per-volume data to have been collected before the experiment. Without that, it is indistinguishable from an excuse, and a reviewer should treat it as one.