Cross-Service Debugging
Investigating a failure that spans multiple services by moving between traces, logs, metrics and profiles along a single correlated request.
Debugging a monolith means a stack trace. Debugging a distributed system means reconstructing what happened across services that each hold part of the story, and the difficulty is proportional to how poorly they are correlated.
The workable sequence: start from a symptom metric to establish scope and timing — one endpoint or all, one region or all, since a deployment or since a dependency changed. Move to traces for affected requests to find which hop consumed the time or produced the error. Move to logs filtered by trace ID for that hop, which is where the specific reason lives. Move to profiles if the hop is slow with no obvious cause.
Each transition depends on correlation: metrics with exemplars linking to representative traces, traces linked to logs by trace ID, and consistent labels across all three. Without those links the sequence becomes four separate searches joined by guesswork and timestamps.
The failure modes that resist this method: partial failure, where the system is healthy in aggregate and broken for one tenant or one code path, which requires dimensional breakdown rather than averages; asynchronous causation, where the trigger occurred minutes earlier in a queue consumer; and emergent behaviour such as retry storms and cascading timeouts, where no single component is at fault and the answer is visible only in the interaction.