Review this instrumentation. A checkout service emits about 400 spans per request, every span carries roughly 30 attributes including the full request body, tracing runs at 100% with no sampling, and the team reports that they cannot find anything in the traces. What would you remove, what would you change, and what would you keep even though it looks excessive?

What is actually required

A trace answers one question: where did the time go, and what was the request doing when it went wrong. Every span that does not help answer that is cost with no return — in money, in ingestion, and most importantly in the reader's attention. A 400-span waterfall is unreadable, and unreadability is the reported symptom.

What I would remove

• Spans around pure computation. A span wrapping a function that does no I/O, takes no lock and touches no queue tells you a duration you could have got from a profiler at a thousandth of the cost. These are usually the majority of the 400.
• The request body on every span. It is the same bytes repeated 400 times, it is the largest single line item on the bill, and in a checkout service it is customer and payment data being copied into a system with a different retention and access model than the one it was collected under.
• Attributes duplicated on every child. Tenant, build, region and flags belong on the root span once. Backends propagate or join them; repeating them multiplies attribute volume by the span count for no additional information.

The change that matters

Introduce tail-based sampling and stop paying for the uninteresting 99%. Keep every trace that contains an error, every trace over the latency objective, and roughly 1% of the rest, weighted so counts can still be reconstructed. At 100% with no sampling the team is paying the maximum price for the minimum utility: the traces they need are buried among the ones they do not.

Second: fix span naming. Span names must be low-cardinality operations (POST /checkout, db.query.orders) with the varying parts in attributes. A name containing an order id makes the backend's per-operation aggregates meaningless, which is another reason nothing is findable.

What I would keep, even though it looks odd

• Every network boundary, every lock wait and every queue wait, even if that is forty spans. These are the only places where time is spent waiting on something outside the process, and they are the entire answer to "where did the time go".
• A span on the cache lookup even when it takes 1 ms. The 1 ms hit is noise; the miss path is the story, and you cannot see the miss rate in a trace you did not record.
• One rich root span. The place to put the thirty attributes is here, once: tenant, plan, build, region, feature flags, client version. This is what makes traces filterable, and filtering is what turns a trace store into a diagnostic tool.

What it costs, in numbers

400 spans at roughly 30 attributes is about 12,000 attribute writes per request, and OpenTelemetry SDKs cap attributes per span at 128 by default, so some of this is being truncated silently already. Vendors bill by span volume and by ingested bytes; at 1,000 rps this design ingests 400,000 spans a second. Tail sampling at the shape above typically removes 95% or more of the volume while keeping essentially all of the diagnostic value.

How I would argue this in the review, and when not to

Not as "you over-instrumented". The test is empirical and unarguable: name one incident that was diagnosed using a span that wraps pure computation. Instrument the boundaries, sample at the tail, and put the cardinality on the root, then ask again in a month whether anything is missing.

When not to cut: a service in its first weeks in production, or one being actively migrated, genuinely benefits from head-heavy instrumentation, because nobody yet knows which boundaries matter. The decision flips once the service is stable and the trace bill is a visible line item — at that point keep the spans that have appeared in a postmortem and delete the rest. The same reasoning ran through the industry's move from per-function tracing to the OpenTelemetry semantic conventions after 2021: fewer, standard, boundary-shaped spans are findable, because the backend can aggregate them.