p99 latency on checkout tripled overnight. Dashboards look normal, no deployment went out, and every service reports healthy. How do you find it?

What the interviewer is testing

Systematic diagnosis in a distributed system when the obvious signals are clean — which is the realistic version of this problem.

Why the dashboards look normal

Three usual reasons, and eliminating them is the first move:

• Averages hide it. A p99 tripling barely moves a mean. If the dashboard shows averages, it is not capable of showing this problem.
• Aggregation hides it. Service-level metrics average across endpoints, tenants and regions. A single endpoint, tenant or availability zone can triple while the aggregate moves a few percent.
• The health checks do not test the slow path. "Healthy" usually means the process is alive.

The method

1. Find which slice is slow. Break p99 down by endpoint, region, availability zone, instance, customer segment, API version and client type. This is what high-cardinality event data is for, and it is where most investigations end — the answer is usually "one zone", "one instance", "one tenant" or "one endpoint".

2. Compare a slow trace with a fast one. Not a slow trace alone — the comparison is what localises it. Take a p99 trace and a p50 trace for the same endpoint and diff the span durations. The hop that grew is the hop to investigate. This step is why tail-based sampling matters: with head-based sampling at 1%, the slow traces you need have mostly been discarded.

3. Distinguish "slow work" from "waiting". A span that is long because the service was computing is a different problem from a span that is long because it was queued. Look at connection pool saturation, thread pool queue depth and time-to-first-byte versus total. Pool exhaustion is the most common cause of this exact symptom and it presents as "the database is slow" while the database is idle.

4. Ask what changed that was not a deployment. This is the crux, given "no deployment": - Data growth — a table crossed a threshold and the planner changed plan. - A configuration or feature flag change — config is deployed too, usually with less rigour. - A dependency's deployment, not yours. - A managed service maintenance event or a noisy neighbour. - Certificate or credential rotation adding a handshake. - Traffic mix shift — same total volume, different composition. A marketing campaign concentrating traffic on one product creates a hot key with no change in request rate. - A cache hit rate drop — the single most common invisible cause. Aggregate latency degrades with no error and no deployment. - Cron collision — a nightly job now overlapping peak.

5. Correlate against infrastructure events. Cloud provider health history, node replacements, autoscaling activity, and any change log for shared platform components.

What to instrument afterwards

The investigation reveals what was missing. Almost always: latency split by dependency, connection pool utilisation, cache hit ratio as a first-class metric, and a deploy-and-config-change marker overlaid on every latency graph.

What a strong answer adds

Naming tail amplification as a candidate explanation: if checkout fans out to ten services, a modest p99 increase in one of them produces a much larger increase at the top. The service that looks fine individually can be the cause, which is why per-dependency latency from the caller's perspective matters more than each service's self-reported health.