p99 latency on the booking endpoint jumped from 300 ms to 4 s at 09:00 today. Error rate is normal. Walk through your diagnosis.

What the interviewer is testing

Whether you have a systematic method rather than a list of guesses.

Establish the shape first

Is it all requests or a subset? Compare p50, p95 and p99. If p50 is unchanged and only p99 moved, a subset of requests or a subset of instances is affected — which points to skew, a hot partition, one bad instance or a specific parameter. If the whole distribution shifted, it is systemic.

Is it all instances or some? Per-instance latency compared against the fleet. One divergent instance is gray failure — a failing disk, a degraded network path, an exhausted pool — and the fix is to eject it, not to investigate it live.

Did anything change? Deployments, configuration, feature flags, schema migrations, and — the ones people forget — changes in dependencies owned by other teams, and provider-side changes.

Then follow the trace

Take a slow trace and find where the time goes. This single step eliminates most of the hypothesis space in minutes, and it is why tracing at the boundaries is worth instrumenting before you need it.

The candidates it will point at:

A downstream dependency got slower, which moves the question to them.

Database query plan changed. A table crossed a size threshold, statistics were refreshed, or an index is no longer selected. Latency jumps discontinuously at a specific time for this reason more often than people expect.

Lock or row contention, which rises non-linearly with concurrency — this fits a 09:00 onset precisely, when traffic ramps.

Connection pool saturation, showing as time spent waiting to acquire rather than executing.

Cache hit rate collapse — a deployment flushed it, keys changed, or a TTL cohort expired together.

The 09:00 detail

The time is informative: it is the traffic ramp. That favours contention, queueing or pool exhaustion — things that appear at concurrency — over a code change, which would have appeared at deployment time.

Queueing delay rises hyperbolically near capacity, so a system at 85% utilisation degrades sharply with a small load increase.

What a strong answer adds

Checking saturation of the actual constraint rather than CPU. Connection pool in-use count, queue depth and in-flight request count predict this; CPU utilisation lags it and looks fine.

Common weak answers

Restarting services and declaring it fixed. Scaling out, which can worsen a database contention problem by adding connections.