Span Attribute Budget

Tracing has no natural stopping point. Every function can be a span and every span can carry every field in scope, and each addition is individually defensible. The result, reached by many teams within a year of adopting tracing, is a few hundred spans per request carrying tens of attributes each — a waterfall nobody can read and a bill nobody can justify.

A span attribute budget is the decision, taken once and enforced in review, about where the spans go and what rides on them. The useful form is not a number in a document but a rule about shape: spans at boundaries, attributes at the root.

The reason this is a discipline rather than a preference is that both quantities multiply. Four hundred spans at thirty attributes is twelve thousand attribute writes per request, and at a thousand requests a second that is four hundred thousand spans a second arriving at a backend that bills by span and by ingested byte.

Why it matters

Readability fails before cost does. The point of a trace is to answer "where did the time go", and a human answers that by scanning a waterfall. Past a few dozen spans the scan stops working, and the team's reported symptom becomes "we have tracing and we do not use it" — which is the expensive outcome, because the bill continues either way.

Span naming carries the same risk. A span named with an order id in it produces one aggregate per order, so the backend's per-operation statistics, which is how you find the slow operation in the first place, become meaningless.

Implementation patterns

• Span at every boundary, nowhere else. Network calls, lock acquisition, queue waits, and anything that blocks on something outside the process. These are where time is actually spent waiting, and they are the entire answer to the question traces exist for.
• Attributes on the root, once. Tenant, plan, build, region, client version, feature flags. High-cardinality identity belongs here, where it makes traces filterable without being repeated on every child.
• Low-cardinality span names, high-cardinality attributes. GET /orders/{id}, never GET /orders/8831.
• No payload bodies. They are the single largest cost line and they move personal data into a store with different retention and access rules than the one it was collected under.
• Sample at the tail: all errors, everything over the latency objective, roughly 1% of the rest, recording the rate so counts can be reweighted.
• Know the hard limits. OpenTelemetry SDKs cap attributes per span at 128 by default and backends truncate beyond their own limits, so an over-instrumented span is already losing data silently.

Industry example

OpenTelemetry's semantic conventions exist because of this pressure: the ecosystem converged after 2021 on a standard, deliberately small set of attributes per span kind — HTTP method, route, status, peer service — so that backends can aggregate across services written by different teams in different languages. The convention is as much a budget as a vocabulary: it names what is worth carrying, which is a short list.

Failure scenarios

• The unreadable waterfall: 400 spans, no obvious critical path, and a team that has stopped opening traces.
• Silent truncation: attributes past the SDK limit are dropped without an error, so instrumentation appears to work and the field you need is missing.
• Aggregates destroyed by naming: high-cardinality span names make per-operation latency statistics useless, which removes the fastest way into an investigation.
• Sensitive data in the trace store, discovered during a privacy review rather than a design review.
• Cost growth tracking span count rather than traffic, so a refactor that adds instrumentation raises the bill with no traffic change and no owner.

Trade-offs

Fewer spans means less detail, and occasionally an incident where the span you cut would have helped. That is a real cost and it is smaller than it feels, because the alternative is a trace store nobody opens. Tail sampling buys most of the saving at the price of buffering every in-flight trace in the collector, which is memory proportional to span rate multiplied by the decision window, plus the loss of spans arriving after the decision.

When not to use it

A service in its first weeks in production, or one under active migration, deserves head-heavy instrumentation, because nobody yet knows which boundaries matter and the cheapest way to find out is to record too much for a while. The budget should be applied once the service is stable and the trace bill is a visible line item, and the right rule at that point is empirical: keep the spans that have appeared in a postmortem, and delete the rest.

Interview question

Q: A team emits 400 spans per request at 100% sampling, every span carrying the request body, and says they cannot find anything in their traces. What do you cut, what do you change, and what do you keep even though it looks excessive?

What a strong answer covers: removing spans around pure computation (that is a profiler's job) and the repeated payload; introducing tail sampling with errors and slow requests kept in full; fixing span names to low cardinality with identity moved to root attributes; keeping every boundary span even if there are forty of them, plus the cache span whose miss path is the story; and arguing it empirically — name one incident diagnosed by a span that wraps pure computation.

Quick check

Quiz: Why does a high-cardinality span name hurt more than a high-cardinality attribute? Because backends aggregate per operation name, so an id in the name produces one aggregate per id and destroys the per-operation statistics used to find a slow operation.

Flashcard: Where do spans go and where do attributes go? Spans at boundaries — network calls, locks, queue waits — and the high-cardinality attributes once on the root span, which keeps traces filterable without multiplying attribute volume by the span count.