Critical Path Analysis

A trace shows every operation and its duration. Summing those durations is meaningless when work runs in parallel: a request with ten concurrent 100 ms calls has one second of span time and 100 ms of latency.

Critical path analysis extracts the chain of spans that determines the response time — the sequence where each element must complete before the next can start, ending at the response. Only time on that path is latency; everything else is capacity.

Why it matters

Optimisation targeted by span duration is frequently wasted. The slowest span in a trace is often a parallel call that finishes well inside the critical path, and making it twice as fast changes the user's latency by zero — while the small, unglamorous sequential span that everything waits for goes unexamined.

Per-service latency metrics are equally misleading in this respect: a service can have high latency and contribute nothing to the user's experience, and a service with low latency can be the bottleneck because it is called sequentially forty times.

The critical path is the only view that answers "what should I fix?"

Implementation patterns

• Compute the path from parent-child relationships and timing, walking backward from the response through the spans that were blocking at each moment.
• Aggregate across many traces, not one. A single trace is an anecdote; the useful artefact is the distribution of critical-path composition across thousands of requests, showing which service contributes the most blocking time at p99.
• Analyse slow traces separately from fast ones. The critical path at p99 is frequently a different shape from the one at p50 — a cache miss, a retry, an extra round trip — and the p50 path is not the one to optimise.
• Look for accidental serialisation: calls in a loop, awaited sequentially, that could be issued concurrently. This is the most common and most easily fixed finding.
• Count repeated calls to the same dependency within one trace, which surfaces N+1 patterns that no latency metric reveals.
• Include queue and scheduling time, not just execution time, since waiting for a worker is blocking time that belongs on the path.
• Re-run after each change, because removing the top contributor promotes a different one and the second iteration's target is rarely predictable.

Industry example

Search and feed-serving systems with strict tail-latency budgets — Airbnb's search ranking being a documented example — rely on this analysis, because those systems fan out to many components in parallel and the naive reading of per-component latency gives the wrong optimisation target almost every time.

It is also the standard method behind fan-out latency work generally: when a response requires results from dozens of parallel calls, the latency is set by the slowest of them, which makes tail latency of individual components — not their averages — the quantity that matters, and makes hedged requests and adaptive timeouts the effective interventions.

Failure scenarios

• Optimising the longest span rather than the longest blocking span.
• Analysing one trace and generalising from it.
• Using p50 traces to plan work aimed at p99.
• Incomplete context propagation, so part of the path is missing — and the gap looks like an absence of work rather than an absence of instrumentation, which actively misleads.
• Ignoring queue time, so the analysis blames the executing service when the delay was waiting for capacity.
• Missing async boundaries, where a message queue breaks the trace and the path appears to end at the producer.
• Treating the critical path as static, when it changes under load, after a cache is cold, and after each optimisation.

Trade-offs

The analysis requires complete, well-formed tracing with accurate parent-child relationships and clock consistency, which is a substantial instrumentation investment — and it degrades badly with partial coverage, because a missing segment silently removes itself from the path.

It is also more work than reading a latency dashboard, and for a simple system with two or three sequential calls it tells you nothing you did not already know. Its value scales with fan-out and depth, which is precisely where intuition fails.

The trade is instrumentation investment and analytical effort in exchange for knowing which optimisation will actually move user-visible latency — and in a system with meaningful parallelism, that knowledge is the difference between weeks of work with no effect and days of work with a measurable one.

Interview question

"p99 latency is 2 seconds and the slowest span in our traces is a 1.4-second call to the recommendation service. Tell me why making that call twice as fast might change nothing, and tell me exactly what you would compute to find the work that matters."