0.5% of requests time out. It is not correlated with any instance, endpoint, customer or time of day. Where do you look?

What the interviewer is testing

Whether you have a method for the hardest class of problem — low-rate, uncorrelated failures.

Establish what "uncorrelated" means

First verify the claim, because it is usually incomplete. Check correlation against: source instance, destination instance, availability zone, connection age, request size, response size, downstream dependency, and time since the last deployment.

Connection age and zone pairing are the two most commonly overlooked, and they frequently reveal the pattern.

The candidates for genuinely uncorrelated low-rate timeouts

Garbage collection pauses. A stop-the-world pause of a few hundred milliseconds on either side affects whichever request happens to be in flight — which is genuinely random. Check GC pause distribution against the timeout threshold.

Connection pool acquisition. A request that arrives when the pool is momentarily exhausted waits. This shows as time before the request is sent, which is invisible unless you instrument acquisition separately from execution.

Cross-zone network variance. Packet loss or elevated latency on some paths, affecting a fraction of connections. Correlate by source-destination zone pair.

DNS resolution. An occasional slow or failed lookup, particularly if resolution happens per connection and a resolver is degraded.

TCP retransmission. A dropped packet costs a retransmission timeout, which can be substantial and is invisible at the application layer.

Noisy neighbours on shared infrastructure, affecting whichever instance is co-located with a busy workload at that moment.

A downstream doing the same thing to you, one layer removed.

The instrumentation that resolves it

Break the request into phases: DNS, connection acquisition, TLS handshake, time to first byte, time to last byte. The phase where the time goes narrows the candidate list to one or two.

Most clients do not expose this by default, and adding it is usually the fastest path to an answer.

What a strong answer adds

Deciding whether it is worth chasing. 0.5% at the timeout threshold may be an acceptable cost, and the cheaper mitigations — hedged requests against another replica after the p95, or a longer timeout with a bounded retry — may resolve the user impact for far less effort than root-causing it.

That is a legitimate engineering judgement, and stating it explicitly is better than an open-ended investigation.

Common weak answers

Increasing the timeout without measuring where the time goes. Assuming the network without evidence.