Async Trace Context Carrier

A request enters a service, which publishes a message and returns. A consumer picks it up, calls two other services and fails. In the trace view the request ends at the publish and the consumer's work appears as a separate, unconnected trace, or not at all. Tracing libraries propagate context along function calls and HTTP headers; a message queue is neither, so the context must be carried by hand on the message. An async trace context carrier is that deliberate step, plus the rules for using it when the message is consumed later.

Why it matters

Synchronous paths are well covered by automatic instrumentation. The business-critical failures often are not synchronous: a payment settles in a worker, an email is sent from a queue, a webhook is retried for hours. When context is lost at the queue boundary, the part of the system that does the most deferred work is untraceable and incident reviews end with "we cannot see what happened after the publish". It also fixes a sampling problem: if the producer's decision is not carried, the consumer samples independently and you get half-traces.

Implementation patterns

• Inject at publish. Add the W3C traceparent value (version, trace id, parent span id, flags) and optionally tracestate to the message's attributes or headers. It is about 55 characters, small enough for every message. W3C Trace Context became a W3C Recommendation in 2021.
• Extract at consume. The consumer reads the attributes and creates its span with that context. OpenTelemetry's messaging instrumentation does this for common clients; check each client library for header forwarding.
• Parent or link, chosen by message age. For messages consumed within seconds, make the consumer span a child of the producer span. For delayed, batched or redriven messages, start a new trace and attach a span link to the carried context, so the new trace is complete and recent while the link records where the message came from.
• Persist for jobs. For scheduled jobs and outbox rows, store the trace context in a column when the row is written, and read it when the job runs.
• Carry business identifiers too: order or payment id in message attributes and log fields, so retrieval does not depend on the trace store.
• Force-sample the interesting paths: set the sampled flag for redrives and dead-lettered messages, since they are rare and always worth keeping.

Industry example

An archetypal payments platform processes messages through a queue with five retries over two hours and a dead-letter queue that operators redrive days later. With plain parent-child propagation, a message redriven nine days later reuses a trace id whose original spans are gone from a 7-day backend, and if the original root was not sampled the redrive produces no spans at all. The platform changed the redrive tool to start a new sampled trace with a link to the carried context. This describes a design archetype, not a documented incident at any named company.

Failure scenarios

• Messaging client drops headers, so the consumer starts a fresh root and the trace splits in two.
• Stale parent: the carried parent span is older than trace retention, and the backend shows an orphan with a missing root.
• Sampled flag honoured blindly: 90% of consumer work produces no spans because the producer's head sampling said no.
• Fan-in batch with one parent: a consumer processing 500 messages cannot have 500 parents; use one batch span with 500 links.
• Header size and trust: accepting tracestate from untrusted publishers without a size bound can bloat messages or carry attacker-chosen values.

Trade-offs

When not to use it

Queues where messages live for seconds and every consumer is instrumented gain little from links; plain parent-child context is right and simpler. If the system has no tracing backend and a good business correlation id, carrying a trace context without a place to view it adds bytes without value.

Interview question

Q: Traces stop at the queue, and a failed message redriven last week has no trace. How do you fix the design?

What a strong answer covers: injecting and extracting context by hand where libraries do not; parent versus link depending on delay and retention; carrying the business id; forcing sampling for redrives and dead letters; tail sampling for errors; batch consumers with links; and a test asserting a trace id across each hop.

Quick check

Quiz: When should a consumer link to rather than parent from the carried context? When the message may be consumed after a long delay, as a batch, or after trace retention.

Flashcard: What does a traceparent header carry? — Version, trace id, parent span id and the sampled flag.