advanced 3 min answer

"We rolled out a fleet-wide profiler. Half the frames in the flame graphs are hex addresses and nobody opens them. Make continuous profiling something engineers actually use." That is the whole prompt. How do you take it apart?

profilingsymbolizationebpfbuild-idcontinuous-profiling
Show the full answer Hide the answer

What the interviewer is testing

Whether you know that profiling is a pipeline — capture, unwind, symbolize, label, store, compare — and can say which stage is broken. The agent is working, so "turn on the profiler" is a failed answer. The prompt is also a product question: a flame graph nobody opens is not a telemetry gap, it is a tool with no job.

The clarifying questions that change the answer

Which runtimes are in the fleet? Are production binaries stripped, and does the build preserve a build ID? Is the agent in-process or kernel-level? Who operates the backend and what is the retention? Are profiles labelled with the same service and version attributes as traces? The answers decide whether this is a CI problem or a backend problem.

Where hex addresses come from

Two distinct failures with different fixes, and conflating them is the usual mistake.

Unwinding produces the stack. A kernel-level agent can walk native stacks from .eh_frame data, needing neither frame pointers nor DWARF debug information on the host, as the OpenTelemetry eBPF profiler documents, with separate unwinders for the JVM, Python, Ruby, PHP, V8 and .NET. A missing unwinder shows up as a single frame named after the interpreter, not as hex.

Symbolization turns addresses into names, and hex means this stage never ran. Stripped binaries carry a build ID; symbols must reach a debuginfo store at build time, or be fetched from a debuginfod server, and the backend then resolves addresses and demangles the result. Parca's server exposes exactly these parts, including configurable upstream debuginfod servers and a demangling mode. Nothing recovers names retroactively once the build artefacts are gone, which is why the first fix lands in CI rather than in the observability stack.

The arc of a strong answer

  1. Upload debuginfo keyed by build ID on every release, and alert when a profile arrives whose build ID has no symbols. That alert is the one that keeps this fixed.
  2. Label every profile with service, version and deployment, so the default view is a comparison rather than an absolute graph.
  3. Normalise before comparing: CPU seconds per thousand requests, not per container, or a traffic change reads as a regression and the tool loses credibility the first time it is used.
  4. Give it a job. The question it should answer by itself is "what got slower in 2.4", posted in the release channel with the two profiles already diffed.
  5. Then talk about cost: roughly 1% CPU and a few hundred megabytes per host is the published budget for a kernel-level agent, and 2 to 4 weeks of retention covers release comparison.

What a strong answer refuses

Fleet-wide always-on allocation and lock profiling, whose overhead is a different order from CPU sampling and which almost nobody queries. Separate in-process profilers in six languages, each with its own operational surface and its own failure mode. A year of raw profiles, which answers no question that a release comparison does not.

Common weak answers

  • "Raise the sampling frequency." More unresolved addresses, more cost, no names.
  • "Use flame graphs." The visualisation is not the problem; the symbol pipeline is.
  • "Profile in staging instead." Staging lacks the workload mix that produced the regression, which is the reason production profiling exists at all.