Debugging Distributed Systems

Definition

In a distributed system the symptom and the cause are usually in different places, and the failure is often an interaction rather than a component. Debugging is therefore a search problem, and having a method matters more than having intuition.

The method

1. Establish the blast radius. All users or some? All regions or one? All endpoints or one? Started when? This narrows the search more than anything else and takes two minutes.

2. Ask what changed. Deployments, configuration changes, feature flags, infrastructure changes, third-party incidents, traffic pattern shifts. The overwhelming majority of incidents follow a change, and annotated deployment markers on dashboards frequently answer the question outright.

3. Follow one failing request end to end. A single trace often reveals in seconds what aggregate metrics take an hour to suggest. This is why correlation IDs and tracing pay for themselves.

4. Work down the dependency graph, checking saturation at each level. Not error rates — saturation: thread pools, connection pools, queue depths, disk, memory. Cascading failures are resource exhaustion propagating, and the exhausted resource is usually further down than the symptom.

5. Distinguish cause from amplification. The initiating fault is often small — one slow dependency — and everything visible is the system attacking itself through retries and pool exhaustion. Fixing the amplification stops the bleeding; fixing the cause prevents recurrence. Both are needed and they are different work.

The failure patterns worth recognising immediately

• Everything is slow, nothing is erroring → saturation somewhere; look for pool exhaustion.
• One endpoint slow, others fine → that endpoint's specific dependency or query.
• Errors started exactly at a deployment → the deployment, until proven otherwise.
• Errors ramping gradually → a leak, a growing dataset, or a cache slowly becoming ineffective.
• Failure at a periodic interval → a scheduled job, a token expiry, a certificate rotation.
• Only some users → a shard, a region, a client version, or a feature flag cohort.
• Recovered on its own → almost always something with a timeout or a retry that eventually gave up; it will return.

What makes this possible

The tooling has to exist before the incident: correlation IDs propagated everywhere including asynchronously, traces with tail-based sampling so the slow request was captured, deployment annotations on dashboards, saturation metrics for every pool, and a dependency map that is accurate.

Without those, every investigation starts from zero and takes hours. With them, most take minutes, which is why the observability investment is a reliability investment.

Interview question

"Requests are timing out but every service reports normal error rates and CPU. What is your first hypothesis and how do you test it?"