Profiling

Definition

A profiler samples execution to build a statistical picture of where time or memory is spent, usually visualised as a flame graph. It answers the question at a granularity below APM: not "which query" but "which function, called from where".

The kinds and what each finds

• CPU profiling. Which code paths consume processor time. Finds inefficient algorithms, unexpected serialisation costs, regular expression backtracking, and — very commonly — logging.
• Allocation profiling. What is creating garbage. Often more valuable than CPU profiling in managed runtimes, because allocation rate drives collection pauses which drive tail latency.
• Heap profiling. What is retained. The tool for memory leaks.
• Blocking and lock profiling. Where threads wait. Finds contention that CPU profiling shows as idle.
• Wall-clock profiling. Includes waiting, which is what you want when the service is slow but the CPU is idle — usually the actual situation.

Choosing the wrong kind is the most common mistake: a service that is slow while using 5% CPU will show nothing useful in a CPU profile.

Continuous profiling

Profiling in production, continuously, at low sampling rates. This changes the practice materially: you can compare a profile from during the incident with one from the previous week, and you can attribute cost — which code paths consume the most CPU across the fleet, and therefore the most money.

The cost-attribution use is under-appreciated. At scale, "which function is responsible for 8% of our compute bill" is a question with a direct financial answer.

Practical guidance

• Profile in production, or at least under production-like load and data. Development profiles find development problems.
• Profile before optimising. Intuition about where time goes is reliably wrong, and the classic outcome is a week spent optimising something that was 2% of the total.
• Compare profiles — before and after a change, or between a fast and a slow instance. The difference is more informative than either profile alone.
• Look for the surprising width, not the deepest stack. A flame graph's value is in the unexpectedly wide frame.

Failure scenarios

• Profiling the wrong resource, most often CPU when the service is blocked on I/O.
• Optimising without measuring first, then again without measuring after.
• Profiling only in development, where data volumes and cache states differ fundamentally.
• A profiler with high overhead left enabled in production.

Interview question

"A service is slow and CPU utilisation is 4%. What do you profile and what do you expect to find?"