Profile Symbolization

A fleet-wide profiler has been running for a month. The flame graphs are full of hex addresses, nobody opens them, and the team is debating whether to raise the sampling frequency. Raising it produces more unresolved addresses. The agent is working; the stage after it never ran.

Profiling is a pipeline — capture, unwind, symbolize, label, store, compare — and symbolization is the stage that depends on something outside the observability stack entirely: the build.

Why it matters

Production binaries are stripped, so what a profiler captures is a list of addresses. Turning those into names requires the debug information from the exact build that is running, located by the binary's build ID. If that information was never kept, nothing recovers the names afterwards, and the profiles for that release are permanently unreadable. The fix therefore lands in continuous integration, which is why it is so often nobody's job.

Distinguishing it from unwinding matters diagnostically. A missing unwinder shows up as a single frame named after the interpreter, not as hex; hex means addresses arrived and were never resolved.

Implementation patterns

• Upload debug information on every release, keyed by build ID, to a debuginfo store the backend can query. Parca's server, for example, exposes a symbolizer with address-to-line resolution, a demangling mode and configurable upstream debuginfod servers.
• Keep build IDs stable and present. A reproducible build that strips symbols but preserves the build ID is the configuration you want; a build that omits it makes the symbols unlinkable even when you have them.
• Handle JIT runtimes separately. A JVM or V8 frame has no static symbol at all, so the runtime must emit a mapping while it runs, and that mapping has to be collected alongside the profile.
• Alert on unsymbolized profiles. A counter of profiles whose build ID has no matching symbols, alerting above a small threshold, is what keeps this fixed after the first week.
• Symbolize on the server, not the host. Shipping debug information to every node to resolve locally multiplies storage and still fails for binaries built elsewhere.

Industry example

The split is visible in the OpenTelemetry eBPF profiler and its backends, which is the shape most continuous-profiling deployments in production now take: the agent captures and unwinds, using .eh_frame data so that neither frame pointers nor host debug symbols are required, and symbolization lives in the backend, which is where the debuginfo store and the debuginfod lookups sit. Open-source backends document symbolization as a first-class component precisely because it is a separate system with its own storage and failure modes.

Failure scenarios

• Symbols never uploaded. Hex frames for the whole release, discovered during the investigation that needed them.
• Build ID mismatch. A rebuild with a different compiler flag produces a new build ID, so the symbols on file belong to a binary that was never deployed, and resolution fails or, worse, succeeds against the wrong build and names the wrong functions.
• Symbols garbage-collected before the profiles. Debug information retained for 7 days and profiles for 30 leaves three weeks of unreadable history.
• Inlining confusion. Aggressive optimisation attributes time to the inlining caller, so a hot function appears to have vanished until inline frames are resolved too.

Trade-offs

When not to use it

If the only question asked of profiling is how much CPU each service uses, aggregate metrics answer it and no symbol pipeline is needed. For a short-lived experiment, running a profiler on an unstripped build in one environment is faster than wiring the CI step. Build the pipeline when profiles are expected to explain a regression across releases, because that is the use that requires names, and names require the build to have cooperated.

Interview question

Q: Your continuous profiler produces flame graphs full of hex addresses. How do you work out which stage is broken, and what do you change?

What a strong answer covers: separating unwinding from symbolization by the signature each failure leaves; build IDs and a debuginfo store populated in CI; JIT runtimes needing in-process mappings; alerting on profiles whose build ID has no symbols; retention alignment between symbols and profiles; and that nothing resolves addresses retroactively once the artefacts are gone.

Quick check

Quiz: Hex frames or a single frame named after the interpreter — which is a symbolization failure? — Hex frames. The interpreter frame means an unwinder for that runtime is missing.

Flashcard: Why is unreadable profiling history unfixable after the fact? — Symbols must be kept at build time and keyed by build ID; once the artefacts are gone there is nothing left to map the captured addresses onto.