A marketplace's search page is slow. CPU is moderate, memory is fine, and the database reports healthy query times. How do you find the bottleneck?

The first move

Decompose the latency, do not survey resources. Resource dashboards answer "is anything saturated", which is a different question from "where is the time going". A request that spends 800 ms of a 1-second response waiting for something will show no resource saturated anywhere.

Get a latency breakdown per request: time in each service, time in each dependency call, time queued before processing, and time serialising the response. Distributed tracing provides this directly; without it, timing instrumentation at each boundary is the fallback.

Where the time usually is when nothing is saturated

1. Waiting for a connection or a worker slot. Queue time before processing begins. Invisible in every resource metric and in the database's own query timings, because the query has not started.

2. Sequential dependency calls. Six calls of 100 ms each, made in sequence, is 600 ms with every component reporting good latency. The fix is parallelism, not speed.

3. The N+1 pattern. One query for results and then one per result. The database reports every query as fast because each is; the aggregate is hundreds of round trips.

4. Serialisation and rendering. Assembling a large response can dominate, and it typically appears nowhere in any dependency metric.

5. Client-side time. For a search results page with many images and scripts, the server may be a small fraction of what the user experiences.

6. Tail effects in fan-out. If the page requires ten parallel calls, its latency is the maximum of ten, not the average. With a p99 of 500 ms on each, roughly 10% of pages hit at least one slow call.

The systematic method

• Measure at percentiles, segmented. A slow p99 with a fine p50 points at contention, cache misses on a subset, or one bad instance — a different investigation from uniform slowness.
• Compare against a known-fast case. The same page with a different query, or for a small seller versus a large one. Differences localise the cause.
• Follow the largest term first, and re-measure after each change. Optimising a component that contributes 5% cannot yield more than 5%, however satisfying the work.
• Check for a hard concurrency limit using Little's Law: if throughput times latency equals a pool size exactly, that pool is the bottleneck and no resource metric will show it.

The principle

Systems are limited by their slowest serialised path, not by their most utilised resource. A saturated resource is one kind of bottleneck; a queue, a sequential chain and a fixed concurrency limit are others, and they are invisible to utilisation monitoring. Start from the latency budget and work inward.