A team upgrades to HTTP/3 and median page load time is unchanged. Was the upgrade worthless?

What is being tested

Whether you choose the right metric for the change being evaluated, which is a general skill that happens to be examined here through a protocol.

The reasoning

HTTP/3's main contribution is eliminating TCP-level head-of-line blocking. In HTTP/2, all streams share one TCP connection, so a single lost packet stalls every multiplexed stream until it is retransmitted — including streams whose data arrived perfectly well. QUIC makes streams independent, so loss affects only the stream it belongs to.

That benefit is proportional to packet loss. On a clean fibre connection with near-zero loss, there is almost nothing to gain, and the median user is usually on a good connection. On a mobile network with 2% loss, the difference is substantial.

So measuring at the median measures the population that had no problem to fix.

Two further benefits that also do not show at the median:

• Connection migration. A connection survives an IP address change, so moving from Wi-Fi to cellular does not drop it. This affects mobile users specifically and shows as fewer failed and restarted requests, not as faster median load.
• Faster handshake including encryption setup, which matters most where round trips are expensive — that is, for distant users.

What to measure instead

• p95 and p99, not median.
• Segmented by network type and geography. The improvement is concentrated in mobile users and in regions with poorer connectivity, and aggregating hides it completely.
• Connection failure and request retry rates, which capture the migration benefit.
• Time to first byte on constrained connections specifically.

Measurements published by organisations sitting at the edge of a large share of internet traffic consistently show this shape: modest aggregate improvement, substantial improvement for the worst-connected users.

The general lesson

Protocol and infrastructure upgrades of this kind are tail-latency and equity improvements. If your success metric is the median, you will systematically conclude that work benefiting your worst-off users did nothing — and you will stop doing it. Choosing the metric before the experiment is what prevents that.

A related caution worth adding

Check the fallback rate. Corporate networks and some middleboxes block or throttle UDP, so a share of clients silently fall back to HTTP/2. That is fine, but it means measured adoption is not the same as deployed adoption, and it should be monitored rather than assumed.