A nightly replication job copies 400 GB over a 10 Gbps path with an 80 ms round-trip time and about 0.1% packet loss. A single TCP stream sustains roughly 5 Mbps and the window never grows. Which change recovers most of the link?
Show the full answer Hide the answer
The deciding property
Loss-based congestion control treats every lost packet as a congestion signal and halves its window. On a long path with steady low-level loss, the window spends its life being halved and slowly rebuilt, and the achievable rate is set by the loss rate and the round-trip time - not by the link capacity.
The classical approximation (Mathis and colleagues, 1997) is:
throughput ≈ MSS / (RTT × sqrt(p))
With a 1460-byte segment (11,680 bits), an 80 ms round trip and p = 0.001:
11,680 / (0.08 × 0.0316) ≈ 11,680 / 0.00253 ≈ 4.6 Mbps
That is the observed 5 Mbps. The measurement is not a symptom of a misconfiguration; it is the formula. It also explains why the same job is fast between hosts in one data centre: at 0.5 ms round trip the same loss rate yields roughly 740 Mbps.
Why the correct answer works
Two independent routes to the same place:
- Parallel streams multiply the result, because each stream suffers loss independently. 400 GB in a four-hour window needs about 230 Mbps, so roughly 50 streams gets there and 150 saturates a good fraction of the link. This is why bulk-transfer tools default to multiple connections.
- BBR estimates bottleneck bandwidth and round-trip propagation time and paces to that model, rather than inferring congestion from loss. On a path with non-congestive loss it keeps the window open, which is exactly this case. The honest caveat: BBR can take more than its share when competing with loss-based flows, so it is a sender-side choice to make deliberately on paths you own.
Why the other options fail
- A 40 Gbps link changes the term that is not binding. The single stream will still sustake about 5 Mbps, because nothing in the formula references capacity. This is the most expensive wrong answer and the one most often bought.
- Larger socket buffers are necessary but not sufficient. The bandwidth-delay product here is 10 Gbps × 0.08 s = 100 MB, and a loss-based window will never approach it: it is being halved roughly every few round trips. Buffers matter when the window is capped by configuration; here it is capped by loss.
- TCP_NODELAY disables Nagle's algorithm, which affects latency for small interactive writes. A bulk transfer sends full segments, so it changes nothing.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| The loss turns out to be congestive, from a saturated uplink | Fix the uplink; do not add streams | Parallel streams on a congested link make the congestion worse |
| The path is a shared internet route with other tenants | Parallel streams with a rate cap, not BBR | Unilateral BBR on shared paths displaces other traffic |
| Loss is caused by a faulty optic or one bad ECMP member | Find and remove it | 0.1% loss is often one path in a bundle, and fixing it beats working around it |
What it costs
Parallel streams complicate restart and ordering, and a 150-connection transfer is 150 entries in every firewall and NAT state table on the path. BBR changes fairness characteristics for everyone sharing the bottleneck. Neither is free, and both are far cheaper than a link upgrade that does nothing.
When this is the wrong diagnosis
Inside one data centre, where round trips are sub-millisecond, the bound is in the gigabits and the constraint is almost always the application, the disk or the receiver. It is also the wrong lens for small transfers: a 50 KB response completes in a couple of round trips and never reaches a steady-state window, so its latency is set by handshakes and retransmission timers instead. And if the measured throughput is far below the bound, something else is capping it - a receive buffer, a rate limit, or a single-threaded reader - so compute the bound first and only then decide what you are fixing.