A developer-tools company needs to find a performance regression that only appears under real production workloads. What are the options for profiling in production, and what are their costs?

Why the alternatives fall short

Reproducing in a test environment. The premise of the question is that the regression only appears under real workloads — real data distributions, real concurrency, real cache states, real garbage-collection pressure. Synthesising that faithfully is often harder than the original problem, and a reproduction that differs in one dimension may not exhibit the behaviour at all.

Targeted instrumentation. Requires knowing where to look, which is the thing you do not know. Each guess costs a deploy cycle, and the instrumentation itself becomes permanent clutter.

Tracing alone. Tracing shows time spent between services and in instrumented spans. It cannot show that a regression is inside one function in one service — it will show the service got slower and stop there. Tracing and profiling answer different questions and neither substitutes.

How continuous profiling works and what it costs

Sampling profilers interrupt at intervals and record the stack. Overhead is proportional to sampling frequency and is typically small at a low rate — a few percent or less, which is acceptable for the capability.

What you get:

• CPU flame graphs aggregated across the fleet, showing where time is actually spent.
• Comparison across versions. The highest-value capability: diff the profile before and after a deploy and the regression is visible directly rather than inferred.
• Memory allocation profiles, which frequently matter more than CPU — allocation drives garbage-collection pauses, which drive tail latency.
• Attribution by tenant, endpoint or version, when profiles carry labels.

The costs and constraints

• Overhead, small but non-zero, on every production instance.
• Storage and processing for continuous profile data across a fleet.
• Symbolisation, which requires debug symbols matching each deployed build — an operational requirement teams routinely get wrong, producing unreadable profiles.
• Sensitive data risk, since stack traces and allocation sites can reveal internal structure; profiles need the same access controls as logs.

The synthesis worth stating

The mature approach uses all three in their proper roles: metrics say something is slow, tracing says which service, profiling says which code. Each answers a different question, and an organisation that has two of the three will find the third's absence during exactly the investigation that needs it.

For a company whose product is developer tooling, there is an additional argument: profiling data across a large deployed fleet is also product feedback about real-world usage that no test suite generates.