Path MTU Discovery Failure

Every network path has a maximum packet size. When a sender emits a packet too large for some link, the router at that link is supposed to discard it and return an ICMP message stating the maximum. The sender learns the limit and reduces its packet size. This is path MTU discovery, and it works invisibly almost all the time.

It fails when the ICMP message is blocked — by a firewall configured to drop all ICMP, by a middlebox, or by an intermediate network with an over-zealous policy. The sender never learns the limit. It keeps sending packets that keep being discarded, with no error reaching either end. The connection establishes perfectly and then hangs the moment any substantial data flows.

The signature symptom

Small requests work; large transfers hang. A health check succeeds, an API call returns, a page loads — and a file download, a large response, or a repository clone stalls indefinitely. From one network and not another. Intermittently, depending on the path.

This matches "slow sometimes, from some places" almost exactly, which is why it belongs in every network troubleshooting checklist and why it is so often missed: it is invisible from every server-side dashboard, because from the server's perspective it sent the data promptly.

Why it is hard to diagnose

• Server metrics show nothing wrong. The server wrote the bytes; it has no visibility into their fate.
• It is path-specific, so it cannot be reproduced from the office or from a cloud test client.
• It looks like an application problem, so investigation starts in the wrong place.
• The connection is established, ruling out the obvious network suspects.

Implementation patterns

• Do not block all ICMP. The reflexive firewall rule that drops ICMP entirely breaks path MTU discovery for everything behind it. ICMP type 3 code 4 (fragmentation needed) must be permitted.
• MSS clamping at tunnel endpoints and edge devices, which forces the negotiated segment size down at connection setup so oversized packets are never generated. This is the standard mitigation where tunnels are involved.
• Conservative MTU on tunnelled paths. VPNs, overlay networks and encapsulated container networking all reduce the effective MTU, and a default MTU inside a tunnel is a common cause.
• Packetisation-layer MTU discovery, where the transport probes for a working size rather than relying on ICMP. This is what modern transports do, and it is a strong argument for them on hostile paths.
• Active measurement with large payloads from diverse vantage points, so the problem is detected before a customer reports it.

Industry example

Any platform serving large transfers to arbitrary networks encounters this — source-code hosting where clones stall, artefact registries where large image layers hang, backup services, and media upload paths.

The instructive part is organisational rather than technical. The reports arrive as "clones are slow sometimes, from some places", server metrics are healthy, error rates are normal, and the team cannot reproduce it. Without client-side telemetry there is no data at all — the platform has excellent observability of the half of the system it operates and none of the half between it and its users.

The durable fix is not the MTU setting. It is building client-side timing telemetry segmented by network and region, plus synthetic probes performing representative large transfers from diverse networks, so this entire class of problem becomes a dashboard rather than a mystery.

Failure scenarios

• Blanket ICMP blocking applied as a security hardening step, breaking discovery for everyone behind it.
• Container or overlay networking with a default MTU inside an encapsulating tunnel.
• Cloud VPN or interconnect paths with a lower MTU than either endpoint assumes.
• A change in routing moving traffic onto a path with a smaller MTU, so a system that worked yesterday breaks with no local change.

Trade-offs

The mitigations are cheap and safe — permitting one ICMP type costs essentially nothing in security terms, and MSS clamping is a standard configuration. The real cost is the diagnostic infrastructure: client-side telemetry and synthetic probes require investment in measuring what you do not operate.

That investment is justified beyond this one problem, since it is the only way to see any network-path issue between you and your users.

Interview question

"Customers report that large downloads hang from certain networks while everything else works. Your server metrics are clean. Walk me through your investigation, and tell me what you would build so the next occurrence is not a mystery."