Debugging Distributed Systems
Localising a regression when every service reports healthy.
4 to work through
-
intermediate
A deployment completes successfully. Twenty minutes later latency has tripled, but only on new instances. Diagnose.
2 min answer -
advanced
A collaborative editing platform receives a report that one document occasionally shows different content to different users, but it cannot be reproduced. How do you investigate a rare, non-deterministic, distributed bug?
2 min answer -
advanced
A platform of 40 services has logs only, and incidents take hours to diagnose. Design the observability strategy and its rollout order.
2 min answer -
advanced
A streaming consumer's lag is growing steadily while workers show low CPU and no errors. What are the candidate causes and how do you distinguish them?
2 min answer
2 terms in this topic
Cross-Service Debugging
Investigating a failure that spans multiple services by moving between traces, logs, metrics and profiles along a single correlated request.
practiceDebugging Distributed Systems
A method for diagnosing failures whose cause is in a different service from the symptom.
Neighbouring topics
Observability
General material on understanding a system from its outputs.
Logging
What to log, at what level, and what must never appear in a log.
Structured Logging
Machine-parseable events with stable names and consistent fields.
Metrics
Counters, gauges and histograms, and percentiles rather than means.
Cardinality
The label that multiplies series count and the bill with it.
Distributed Tracing
Reconstructing one request's path across every service it touched.
Correlation IDs
One identifier propagated through every hop and every log line.
Sampling
Head-based versus tail-based, and keeping the traces that matter.
Health Checks
Liveness versus readiness, and the check that causes the outage.
Alerting
Symptom-based, actionable, user-impacting — and linked to a runbook.
Alert Fatigue
How noise makes the real page invisible, and the structural fix.
Dashboards
Answering 'is it us' in under a minute, for someone who was asleep.
Application Performance Monitoring
Attributing latency to code paths, queries and dependencies.
Profiling
Continuous CPU and memory attribution in production.
Business Metrics
Orders per minute alongside error rate, because healthy is not enough.
SLO Monitoring
Burn-rate alerting that fires on user impact rather than on thresholds.
Log Management
Aggregation, retention tiering, search and the cost of keeping everything.
Telemetry Cost
Observability bills that rival compute, and where to cut without going blind.
OpenTelemetry
Instrumenting once against an open standard rather than a vendor agent.