A team has profiled a service and found the top three functions by CPU time. Why is optimising them often the wrong next step?

Why it is often wrong

CPU time may not be where the latency is. A service spending 80% of its wall-clock time waiting on network calls and 20% computing will show a CPU profile dominated by that 20%. Optimising the top CPU function perfectly might improve total latency by a few percent, while the actual opportunity is entirely in the waiting.

The profile shows aggregate, not the path that matters. The top function may be dominated by a background or batch code path, while the user-facing request path is elsewhere in the profile.

Amdahl's law bounds the return. A function accounting for 10% of time yields at most 10% improvement even if it becomes free. Teams routinely spend weeks on the top function for a bounded and disappointing result.

The correct sequence

1. Start from the latency budget, not the profile. What is the target, what is the current p99, and how is that time distributed across waiting, queueing and computing? Distributed tracing answers this; profiling does not.

2. Identify the largest term and its nature. If it is waiting, the answers are parallelism, caching, batching or removing the dependency — none of which appear in a CPU profile. If it is queueing, the answer is concurrency limits or capacity. Only if it is computation does the CPU profile become the right tool.

3. Profile the specific path, not the aggregate. Filter to the request type that matters.

4. Check allocation, not just CPU. Allocation rate drives garbage collection, which drives tail latency. An allocation profile frequently reveals a larger opportunity than a CPU profile, and this is one of the most commonly missed findings.

5. Measure the improvement in the metric you care about, end to end, and stop when the target is met.

The higher-leverage questions

Before optimising code, ask whether the work is necessary at all:

• Can it be cached? Not doing the work beats doing it faster.
• Can it be precomputed? Moving it off the request path removes it from the budget entirely.
• Can it be done in parallel? Six sequential 100 ms calls become 100 ms.
• Can it be done less often? Batching, debouncing, coalescing.
• Is it needed for this request? A large amount of computed data is frequently never used by the caller.

Algorithmic and architectural changes routinely yield an order of magnitude; micro-optimisation yields a few percent. The profile is a tool for the last stage, not the first.