concept

Bandwidth-Delay Product

Bandwidth multiplied by round-trip time — the amount of data that must be in flight to keep a link fully utilised.

tcpthroughputtuning

On a 1 Gbps link with 100 ms RTT, the product is about 12.5 MB. Unless that much data is unacknowledged in flight, the link cannot be saturated: the sender transmits a window's worth and then waits.

This is why a large file transfer between continents is slow despite ample bandwidth on both ends — the constraint is the TCP window, not capacity. Default window sizes were set for much smaller products and are frequently the limiting factor on long fat networks.

The remedies: window scaling enabled and tuned, parallel connections to multiply the effective window, protocols designed for it, or — most practically — not transferring data across long distances, by processing it where it lives or replicating in the background rather than on demand.

The same reasoning explains why replication lag across regions is more sensitive to tuning than replication within one.