How long to wait: timeouts and deadlines in distributed systems
How production systems decide how long to wait on a remote call, why the framework default is almost always wrong, and why one number cannot govern a whole call chain.
A field guide to timeouts and deadlines: the difference between a local per-call timeout and a propagated deadline, the stack of independent timers every mature system exposes, and the three failure classes that produce nearly every timeout incident in the public record. Reconstructed from the repository record of gRPC, Envoy, Go, Node, Kubernetes, MongoDB, Finagle, requests/httpx and PostgreSQL, plus GitLab's public incident tracker. After reading it an architect can decide a default consciously, choose between a per-call timeout and a budget for a given call graph, and test that a configured timeout actually fires.
The maintainers of the most-used HTTP libraries in the world still cannot agree on whether a default timeout should exist: requests parked the question in a milestone named 'Bankruptcy' and Go's own 'zero means infinity is a bad default' issue has been open for eight years, so an architect cannot inherit a safe default and must decide it themselves.
What you get out of it
- A timeout is a local resource-protection decision; a deadline is a distributed correctness decision. Moving from 'every service has a timeout' to 'the request carries a propagated budget' is the highest-leverage change here, and home-grown stacks almost always skip it.
- There is no 'the timeout': Envoy alone ships a route timeout (15s), stream idle (5min), connection idle (1h) and request-headers timeout (off), and the unset ones keep vendor defaults you never chose.
- Three of the four incidents fired the timeout correctly and still hurt, because operators read the timeout as the problem rather than as an alarm about the connection, lock or pool behind it.
- The same behaviour flips on the trust boundary: honouring a client's propagated deadline is correct between internal services and a denial-of-service vector at the untrusted edge (Envoy PR #5294).
- A timeout the platform accepts but never enforces is worse than none: Kubernetes ignored documented exec-probe timeouts across four minor versions, so the enforcement itself is what you must test.
Scope
Why this, now. Service meshes and multi-hop RPC have made the deadline-versus-timeout distinction load-bearing for ordinary teams, and the biggest libraries are still openly arguing about what the default should be.
What it does not cover. Circuit breakers, load shedding, idempotency and the wider metastable-overload dynamics (covered by the companion dig 'The retry is the outage'); retries and hedging are treated only where they interact with the deadline budget.
Other field guides
When two hold the lock: what the repositories admit about distributed mutual exclusion
Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions whe…
22 sources · 13 organisations · 3 postmortemsWhen the answer is wrong, not late: ten years of CockroachDB read from its own record
A decade of one distributed database company read from artefacts nobody edits for effect: technical advisories with affected version ranges, design r…
26 sources · 3 organisations · 8 postmortemsTime cannot go backwards, and other lies production believed
Two postmortems (Cloudflare 2017, Azure leap day 2012), three Linux kernel commits, the Go and Rust standard-library design records, Kubernetes leade…
23 sources · 14 organisations · 2 postmortems