A developer platform receives reports that clones are slow "sometimes, from some places". Server metrics are healthy, error rates are normal, and the team cannot reproduce it. How do you investigate a problem that is invisible from the server side?

Why server metrics cannot see it

Server-side metrics measure the server's view: time from first byte received to last byte sent. They are blind to everything before and after — DNS resolution, connection establishment, packet loss on the return path, receive-window limits at the client, and any middlebox between the two.

A transfer that takes 40 seconds because of 2% packet loss on the return path shows as a perfectly healthy server-side response. The server did its job promptly; the bytes took a long time to arrive. The metric and the complaint are measuring different things.

The investigation, in order

1. Establish what "slow" and "sometimes" mean. Get specifics: which networks, which regions, which repository sizes, what time of day, and — critically — whether it is throughput that is low or time to first byte that is high. Those have entirely different causes, and conflating them wastes the whole investigation.

2. Instrument the client. This is the decisive step. Without client-side timing broken into DNS, connect, TLS, time to first byte and transfer, you are guessing. For a platform with a command-line client, adding opt-in timing telemetry is the highest-value diagnostic investment available.

3. Separate throughput from latency problems. - Low throughput on a long-lived transfer points at packet loss, congestion control, receive-window limits, or a bandwidth constraint somewhere on the path. The classic signature is throughput that is inversely proportional to round-trip time — a hallmark of loss-based congestion control on a long path. - High time to first byte with normal throughput points at DNS, connection establishment, or server-side queueing before the response starts.

4. Look for path-specific effects. "Some places" is the strongest clue in the report. Candidates: a particular transit provider with congestion at peak hours; an MTU mismatch causing fragmentation or black- holed packets on one path; a corporate middlebox inspecting and buffering traffic; or an anycast routing change sending a region to a distant, overloaded location.

5. Check MTU specifically for large transfers. Path MTU discovery failures are a classic "large transfers hang, small requests are fine" symptom, produced by an intermediary dropping the ICMP messages that discovery depends on. It matches "sometimes, from some places" exactly, and it is invisible from every server-side dashboard.

6. Deploy active measurement. Probes from many networks and regions performing representative operations, continuously. This turns "we cannot reproduce it" into a dashboard, and it is the only way to detect path-specific degradation before customers report it.

What to build so this is not hard next time

• Client-side timing telemetry, segmented by network, region and operation.
• Synthetic probes from diverse vantage points, alerting on regional deviation rather than on global averages.
• Percentile metrics segmented by geography and network, because a global p99 hides a region where every user is affected.
• Correlation identifiers spanning client and server, so a customer report maps to a specific set of server-side traces.

The generalisable lesson

A problem the server cannot see requires measurement where the user is. Most organisations have excellent server-side observability and nearly none from the client, so an entire class of user-visible problem — the network between you and them — is invisible until enough people complain. Investing in client-side measurement is not a diagnostic nicety; it is the only way to have data about the half of the system you do not operate.