advanced 2 min answer

A payments consumer fails on a message, retries five times over two hours, and parks it in a dead-letter queue. Nine days later an operator redrives it. The producer injected a W3C traceparent header, head sampling is 10% at the root, and the tracing backend keeps data for 7 days. What does the trace view show for the redriven message?

distributed tracingtrace contextdead-letter queuespan linksretentionsampling
Show the full answer Hide the answer

Second by second

The producer put traceparent into the message attributes, carrying the trace id, the parent span id and a sampled flag decided at the root nine days earlier. Nine days later the consumer extracts that header and starts a span as its child.

Two things happen depending on the flag:

  • In the 90% of cases where the root was not sampled, the flag says "do not record". A parent-based sampler obeys it. The redrive produces no spans at all. The failure that was parked for nine days has no trace, and the one moment someone is looking at it is the moment tracing says nothing.
  • In the 10% where it was sampled, the consumer's new span arrives carrying a trace id whose original spans were deleted after 7 days. The backend shows a trace with a missing root: an orphan fragment, a child whose parent is a dangling id, which many UIs hide or render as a lone bar with a zero-length parent.

Where it amplifies

Every retry in the original two hours reused the same context, so those five attempts are children of the same long-dead parent. The apparent "trace" mixes a request from nine days ago with a redrive today, and the gap between spans is nine days of queue time that the timeline squashes or scales into uselessness.

What the user (the investigator) sees

An empty search for the trace id from the failed order, or a stub. The conclusion "tracing is unreliable for async" gets repeated in the retro.

What stops it

A mechanism, not vigilance:

  1. On redrive, start a new trace and attach the old context as a span link (a pointer from the new trace to the old one) instead of a parent. The new trace is complete, recent and sampled by a rule you choose; the link records provenance.
  2. Store the original trace id and the business id (order or payment id) as plain message attributes and as log fields, so retrieval does not depend on the trace store.
  3. Force sampling for redrives and dead-lettered messages by setting the sampled flag in the redrive tool. These are rare, always interesting, and cheap to keep.
  4. Use tail-based sampling in the collector that keeps all traces with errors, so the failed first attempts were kept at the time.

What would have to be true for it to self-heal

Trace retention longer than the longest message lifetime, which is rarely affordable: 30 days of traces for every service costs roughly four times a 7-day window. So the design should assume the old trace is gone.

When this is the wrong answer

For queues where messages live seconds, plain parent-child context is correct and links add nothing. The link design pays when message age can exceed trace retention, which means any dead-letter, scheduled or batch path.