Network Performance

Definition

Two independent properties, constantly confused. Latency is how long one round trip takes and is bounded below by the speed of light in fibre — roughly 150 ms round trip between Europe and the west coast of the United States, and no amount of money changes it. Bandwidth is how much data flows per unit time and can generally be bought.

The architectural implication: you cannot optimise your way out of a latency problem by adding capacity. You can only make fewer round trips, or make them shorter.

The levers, in order of effectiveness

1. Fewer round trips. Batch, denormalise, parallelise. A screen requiring six sequential calls across an ocean cannot be fast; the same six in parallel can.
2. Shorter distance. Serve from a location near the user — CDN, edge, regional replica. This is the only lever that reduces the per-round-trip cost.
3. Connection reuse. Keep-alive and pooling avoid paying handshake and slow start repeatedly.
4. Smaller payloads. Compression and efficient encoding, which help most on constrained networks.
5. Then, bandwidth and server-side optimisation, which are usually where teams start.

Industry example

Netflix's approach to delivery is the strongest illustration of lever two taken to its conclusion: rather than serving video from regions, appliances are placed inside internet service providers' networks, so the content is a very short network path from the viewer. Popular titles are pushed to those appliances before they are requested, during off-peak hours.

Three architectural ideas are worth extracting. Predictive placement — knowing what will be popular and pre-positioning it — converts a latency and bandwidth problem into a scheduling problem. Moving the data rather than the request is available whenever content is large, popular and not personalised. And peering and physical topology are architecture, not an operational detail: at sufficient volume, where your bytes enter the network is a first-order design decision.

Failure scenarios

• Chatty protocols across a high-latency link, where each call is fast and the sequence is not.
• A latency budget that is never stated, so no one notices when a new dependency consumes a third of it.
• Cross-region calls introduced accidentally by a service placed in a different region from its database.
• Cross-availability-zone traffic on every request, adding both latency and transfer charges.
• Measuring at the server, so DNS, TLS and the client's network — most of what the user experiences — are invisible.

Interview question

"A page loads in 200 ms in your region and 2 seconds in Asia. The backend is equally fast for both. What is happening and what do you change?"