Correlation IDs
also called Request ID, Trace ID
A single identifier attached at the edge and carried through every hop, synchronous and asynchronous, that stitches an investigation together.
Definition
An identifier generated at the system boundary, included in every log line, span and message produced while handling that request, and propagated to every downstream call. The cheapest observability mechanism that exists and the one most often incomplete.
Why it is the highest return per unit of effort
Without it, an incident investigation means correlating nine services' logs by timestamp, which is unreliable at any volume and impossible under load — many requests share a millisecond. With it, one query returns everything that happened for one request across the entire estate.
It costs an afternoon to implement and pays back on the first incident.
The details that make or break it
Generate at the true edge — the load balancer or gateway — and accept a client-supplied one only from trusted callers, so a mobile client's session can be traced end to end without allowing an untrusted caller to poison your identifier space.
Return it in the response, and show it in user-facing error messages. "Something went wrong. Reference: 7f3a-91c2." Support can then find the exact request in seconds instead of asking the customer what time it was.
Propagate across asynchronous boundaries. This is where implementations fail. The ID must travel in the message envelope onto the queue, into the batch job, into the retry an hour later. A trace that stops at the queue omits the part you most need.
Carry more than one identifier. A trace ID for the whole request, and a causation or session ID that links a chain of related operations — an order, a checkout session, a user's whole visit. Different investigations need different scopes.
Include it in third-party calls where the API permits, so provider support can find your request in their systems.
Failure scenarios
- Lost at the queue boundary, so asynchronous work is untraceable.
- Regenerated at an internal hop, splitting one request into two unrelated traces.
- Not returned to the user, so support investigations start from a timestamp and a guess.
- Only in some services, so the trail ends exactly where the problem is.
- Personal data used as the identifier — an email address as a correlation ID puts personal data into every log line in the estate.
Interview question
"A customer reports an error at 14:32 yesterday. What would you want to have built so that investigating this takes two minutes?"