A data platform has good tracing and still cannot explain why a specific job is slow. What does tracing not tell you, and what does?

What tracing gives and does not give

Tracing shows where time is spent between services — this call took 200ms, that one took 3 seconds. It is excellent at finding the slow component in a distributed request.

It does not show what is happening inside a single unit of work. A span that takes 3 seconds could be CPU in one function, garbage collection, lock contention, memory allocation pressure, I/O wait, or serialisation. The span says three seconds and nothing more.

For a data-processing job, most of the time is inside one process, so tracing has almost nothing to say.

What profiling gives

• CPU profiles showing which functions consume time. For a compute-heavy job this is usually the whole answer, and it is frequently something unexpected — serialisation, a regular expression compiled in a loop, a per-row allocation.
• Allocation profiles, which for JVM-based data processing are often more revealing than CPU, because allocation pressure drives garbage collection which drives pauses.
• Lock contention profiles, which explain the case where CPU is low and throughput is poor.
• Continuous profiling in production, which is the important variant. Profiling on demand requires reproducing the problem; continuous profiling means the data already exists for the slow run that happened at 2am.

The data-specific dimensions neither one covers

For a distributed data job the dominant costs are frequently structural rather than computational:

• Data skew, where one partition holds far more than the others and the job's duration is set by the slowest task. Visible in task-duration distribution, not in a profile.
• Shuffle volume, which is usually the largest cost in a distributed join and is a query-plan property.
• Small file counts, where the job spends its time in file listing and opening rather than in reading.
• Spill to disk, when a stage exceeds available memory.

These are read from the execution plan and the task metrics, which is a third category of tooling beyond tracing and profiling.

The practical sequence

Task duration distribution → execution plan → profile. Skew and plan problems dominate and are cheap to check; a profile is the right tool once you know the work is genuinely computational rather than structural. Reaching for the profiler first is the most common wasted afternoon in data engineering.