You migrate a mobile API to HTTP/2 and p99 latency gets worse for users on cellular networks, while p50 improves. Explain and decide what to do.
Show the full answer Hide the answer
The mechanism
TCP head-of-line blocking.
HTTP/2 multiplexes all streams onto one TCP connection. TCP guarantees ordered delivery, so a single lost packet stalls every multiplexed stream until it is retransmitted — not only the stream whose data was lost.
HTTP/1.1 with six parallel connections confines the damage to one connection; the other five keep moving.
That explains both halves of the observation. On a clean network HTTP/2 wins — fewer handshakes, header compression, no connection-count limit — hence better p50. On a lossy cellular network the shared connection concentrates the impact of loss, hence worse p99.
What to do
Adopt HTTP/3 (QUIC). It runs on UDP with per-stream ordering, so loss affects only its own stream. It also folds TLS into the transport handshake — one round trip, zero on resumption — and supports connection migration: the connection is identified by a connection ID rather than the IP/port four-tuple, so a phone moving from Wi-Fi to cellular keeps its connection instead of re-handshaking.
For a mobile API on variable networks, connection migration alone often justifies it.
Keep HTTP/2 as fallback. Some networks block or deprioritise UDP. Clients should negotiate down cleanly.
Measure by network type, not in aggregate. This regression is invisible in a global p99 that mixes fixed-line and cellular traffic.
Migration hygiene
Remove HTTP/1.1 workarounds that now hurt. Domain sharding is the important one — it forces multiple connections again and defeats multiplexing entirely. Concatenation, sprites and inlining become unnecessary.
Do not adopt server push. It proved unhelpful in practice and has largely been removed.
What a strong answer adds
Naming the general pattern: one shared ordered channel converts an isolated failure into a shared one. The same shape appears as a poison message blocking a queue partition and a slow task blocking a thread pool. Recognising it transfers across layers.