A caller waits on a reply that may never come. Deciding how long to wait, and what to do when the wait runs out, looks like a one-line config value and is one of the most consequential choices in a distributed system. This guide reconstructs, from the public repository record, what the frameworks default to, why a single number cannot serve a whole call chain, and what production teams ship in its place.
The problem is bounding an unbounded wait. A process asks another process for something and cannot know whether the answer is late, lost, or never coming. The timeout is the only tool it has to reclaim the thread, the connection and the attention it committed to that request, and the deadline is that same decision carried across a whole chain of calls.
Strip away the technology and the problem is old: you make a request to something you do not control, and you must decide, alone and in advance, how long to hold resources open for a reply that may never arrive. Hold too long and one slow dependency drains your connection pool, your threads, your memory, until you are down for a reason that has nothing to do with your own code. Give up too soon and you fail requests that would have succeeded, and if you retry them you can turn a slow dependency into an overloaded one. There is no safe default, because the right answer depends on latency you have not measured yet and on a call graph the framework author never saw.
What makes this a distributed-systems problem rather than a coding tip is that the decision does not stay local. A user-facing request fans out into a tree of downstream calls, and every node in that tree is separately deciding how long to wait. If the leaf waits longer than the root, the root gives up while the leaf keeps working, holding resources for an answer nobody will read. The unit that actually matters is not the per-call timeout but the deadline: one budget, set once at the edge, that every downstream call is measured against. gRPC, Finagle and MongoDB's drivers each converged on that idea independently, which is the first sign it is load-bearing rather than fashionable.
Two vocabulary distinctions run through everything below, and getting them straight is half the battle. A timeout is a duration measured from now (“wait 5 seconds”). A deadline is a point in time (“be done by 12:00:03”) that can be handed to a downstream service, which subtracts elapsed time and turns it back into a shorter timeout for its own calls. And an idle timeout (fires when no bytes move) is a different instrument from an overall timeout (fires when the whole operation takes too long); confusing them is how people accidentally cut off long streaming responses that were perfectly healthy.
Scope. This guide is about the timeout and deadline decision itself: defaults, layering, propagation and the failures each produces. It touches retries and hedging only where they interact with deadlines. It deliberately does not cover circuit breakers, load shedding, idempotency, or the wider metastable-overload story, which the companion dig “The retry is the outage” already treats. And because the research ran against a network that reached the code-hosting and package sites but not most engineering blogs, its evidence is the repository record: source, issues, pull requests, specs, KEPs and GitLab's public incident tracker. Where the blog layer would normally add colour, this guide is thinner; where the repository record is unusually rich, it is thicker than most.
Across HTTP clients, RPC frameworks, proxies, databases and orchestrators, the same three-layer shape recurs: a stack of independent timers, a mechanism to carry a shared budget across hops, and a policy for what happens when the budget runs out. The differences between systems are which layers they default on, and whether the budget is carried at all.
There is no single “the timeout.” Every mature system exposes a stack of them, each measuring a different segment of a request's life, and the reference architecture is that stack plus the wiring that keeps the segments consistent with one another. Envoy's own FAQ is the clearest catalogue: it documents a connection-level idle timeout defaulting to 1 hour, a stream idle timeout defaulting to 5 minutes, a request headers timeout that is disabled by default, and a route timeout that “defaults to 15 seconds” and is the one that bites [13]. A reader who sets one of these and assumes the others do not exist is the reader who files an incident.
The layer that separates a resilient system from a fragile one is the middle box: whether the
budget travels. gRPC defines it on the wire as a header, grpc-timeout, carrying a
value and a unit, and specifies that “if Timeout is omitted a server should
assume an infinite timeout” [16]. The deadlines guide explains the mechanism that
makes it safe across machines with unsynchronised clocks: rather than shipping an absolute
timestamp, “gRPC converts the deadline to a timeout from which the already elapsed time is
already deducted” [17]. Each hop receives less budget than the last, which is
exactly the property you want: the leaf of the call tree can never legitimately outlive its root.
Finagle reaches the same design from a different tradition. Its Deadline is a
“broadcast Context that represents when the request should be completed
by” [23], attached to the request and propagated automatically to every downstream
call. The word “broadcast” is the tell: the deadline is not a parameter you remember to
pass, it is ambient context that rides along whether or not the programmer thinks about it. That is
the difference between a mechanism that works and one that works only when everyone remembers it,
and it is the same reason gRPC makes propagation the default in Java and Go while leaving it opt-in
in C++ [17].
Connect, TLS, read/idle and overall timers, each bounding a different segment. Setting one and ignoring the rest is the most common mistake, because the unset ones keep their vendor defaults, which range from 15 seconds to infinity.
A deadline attached to the request and propagated hop by hop, decremented by elapsed time. This is the box most home-grown stacks are missing, and its absence is invisible until a deep call tree starts doing orphaned work.
What happens when the clock runs out: fail fast, retry within remaining budget, hedge a parallel copy, or cancel downstream work. The gRPC server “automatically cancelling a call” on deadline is a policy, not an accident.
Specified at: gRFC A6, MongoDB CSOT
The database sits at the bottom of every one of these stacks and plays by
its own rules, which is why it deserves separate mention. PostgreSQL's statement_timeout
defaults to zero, meaning disabled, and the manual actively warns against setting it globally:
“Setting statement_timeout in postgresql.conf is not recommended
because it would affect all sessions” [25]. The database's position is that it cannot know
your application's intent, so it defaults to waiting forever and hands you the decision. That is
defensible for a database and catastrophic as an application-wide philosophy, and the gap between
those two readings is where GitLab spent three separate incidents, below.
Four forks recur across every system that got this right: what the default should be, whether one number or a budget governs a call chain, whether a proxy may trust a client's requested deadline, and whether to retry or hedge when the clock runs low. Each has a condition that flips the answer.
The default fight is the most revealing, because it is being fought right now, in public, by the
maintainers of the most-used libraries in the world, and they cannot agree. The requests
library has debated a default timeout since 2016; issue #3070, “Consider making Timeout
option required or have a default,” was closed and parked in a milestone literally named
“Bankruptcy” [5]. Eight years later a maintainer opened PR #6709 to add 10-second connect
and 30-second read defaults; as of this writing it is still open, its milestone bumped from 2.33.0
to 2.34.0 and then removed entirely [6]. The successor library, httpx, simply shipped
DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0) and moved on [7]. Go's own Russ Cox
filed “make default configs have better timeouts” in February 2018, noting that timeout
fields treat “zero = infinity” and should instead have “a reasonable default”;
it remains open in the Backlog milestone in 2026 [8]. The lesson for an architect is not
which side is right. It is that if the library authors cannot settle this for their users, you
cannot inherit a safe default from them, and the decision is yours whether you make it consciously
or not.
MongoDB's Client Side Operations Timeout spec is the cleanest statement of why the budget beats
the knobs. It observes that users had “serverSelectionTimeoutMS,
socketTimeoutMS, connectTimeoutMS, maxTimeMS, and
wTimeoutMS” and that “because some of these timeouts are additive, it is
difficult to set a combination which ensures control will be returned to the user after a specified
amount of time” [22]. Their answer, a single timeoutMS governing the whole
operation including retries, comes with a deliberate one-way door: “If timeoutMS
is specified at any level, it cannot be later changed to unset at a lower level” [22]. Once
a budget exists, no inner layer is allowed to opt back into waiting forever, which is precisely the
loophole that makes ad-hoc timeout stacks leak.
grpc-timeout header is presentPR #5294 is the rejected decision worth studying, because it shows the trust boundary changing the
answer to an otherwise identical question. The author proposed that Envoy disable its stream idle
timeout for gRPC requests, reasoning that such requests are already bounded by their
grpc-timeout header, so the proxy's timer is redundant and harmful. Maintainer
mattklein123 closed it: at the edge, letting the request set the timeout “is not safe,”
and any such behaviour must be “opt-in based on config” [14]. The reasoning is sound
and the consequence is a footgun: because Envoy will not trust the header, its 15-second route
default silently truncates long streaming RPCs, which is exactly the bug users kept filing (see
failures, below). Both the rejection and the resulting pain are correct at once, which is what makes
it a real trade-off rather than a mistake.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Default value | Finite (5–60s) | Infinite / unset | A hang is worse than a spurious timeout | You ship a general library, not an app | httpx / Go #24138 |
| Chain governance | Propagated deadline | Per-call timeout | Leaf must not outlive root; bounds retries too | Call depth is 1 | gRPC / MongoDB CSOT |
| Edge trust | Proxy timers win | Honour client header | Untrusted client can't set your budget | Hop is internal, client trusted | Envoy #5294 |
| Enforce a documented timeout that never worked | Enforce it, gate + warn | Keep the bug on by default | “kubelet should do what is specified in the API” | Never; correctness wins, opt-out provided | k8s #97057 |
| Out of budget, response missing | Retry within remaining deadline | Hedge a parallel copy | Hedging multiplies load; only safe if idempotent | Read-only op and tail latency dominates | gRFC A6 |
A timeout is a local resource-protection decision; a deadline is a distributed correctness decision. Systems that only have timeouts protect each node while letting the call tree do orphaned work. The move from “every service has a timeout” to “the request carries a budget” is the single highest-leverage change in this whole area, and it is the one home-grown stacks almost always skip.
Three failure classes account for nearly every timeout incident in the public record: the timeout that fired but was the symptom not the cause; the default nobody chose that cut off healthy work; and the timeout that was documented but never enforced. GitLab's tracker and the Kubernetes and Envoy issue threads are unusually candid on all three.
IN (..) returning >200 rows slow from milliseconds to over two minutes; those queries exhausted the connection pool and the whole site went dark. The statement timeout fired everywhere, which looked like the problem and was actually the alarm.ANALYZE namespaces;.IN (..) pattern, and add alerting on abnormally long query execution, not just on the timeout firing.UNAVAILABLE: upstream request timeout. The request carried x-envoy-expected-rq-timeout-ms: 15000, a value the developer never set. Envoy will not trust the client's grpc-timeout at the edge (see PR #5294), so its own default won.0s for streaming routes, or bound them with max_stream_duration or an idle timeout instead of the overall one.timeoutSeconds I set on an exec probe.ExecProbeTimeout gate with warning events so operators could find who had been relying on the bug.The fourth incident opens onto a decision worth its own paragraph, because it shows how dangerous fixing a timeout bug can be. When KEP-1972 finally made the kubelet enforce the documented 1-second default, a large population of clusters had unknowingly depended on the bug, running probes that took longer. Microsoft's AKS team proposed PR #97057 to ship the fix disabled by default, arguing that enforcing it suddenly “puts those previously operational Kubernetes environments at risk” and “will be breaking a very large number of folks” [20]. SIG Node refused: andrewsykim answered that “the kubelet should really do what is specified in the API” and derekwaynecarr chose to “leave the feature gate on and enable opt out” [20]. The compromise, correctness by default with an escape hatch and loud warning events, is the template for changing any long-wrong timeout: enforce the right behaviour, but give operators a season and a signal to discover their dependence on the old one.
A timeout is only ever a proxy for a resource you are protecting: a connection, a thread, a lock, a pod's liveness. Three of these incidents fired the timeout correctly and still hurt, because the operators read the timeout as the problem rather than as an alarm about the resource behind it. The fourth hurt because the timeout was documented but never wired to anything. Debug a timeout by naming the resource it guards, not by editing the number.
The defaults shipped by the frameworks an architect actually uses, and the incident figures that show what those defaults cost when left unexamined. All are read directly from source, docs or incident reviews, dated to the point of check.
| Setting / metric | Value | System | Context | As of | Source |
|---|---|---|---|---|---|
| Default request timeout | none | requests (Python) | Waits forever unless caller sets one; unresolved since 2016 | 2026-09 | #3070 |
| Default request timeout | 5.0 s | httpx (Python) | Shipped in code, the answer requests declined | 2026-09 | source |
| Default connect/read timeout | 60 s | botocore (AWS SDK) | Applied when caller passes none | 2026-09 | source |
http.DefaultClient timeout | none | Go net/http | &Client{}; zero means infinity | 2026-09 | #22982 |
| Route (overall) timeout | 15 s | Envoy | Bites streaming responses that never end | 2026-09 | FAQ |
| Stream idle timeout | 5 min | Envoy | Fires when no bytes move on the stream | 2026-09 | FAQ |
| Connection idle timeout | 1 h | Envoy | Upstream connection with no active streams | 2026-09 | FAQ |
| gRPC deadline (unset) | infinite | gRPC | “wait effectively forever” if omitted | 2026-09 | guide |
| Server default request timeout | none→300 s | Node.js http | Removed in v13, reinstated at 5 min in v18 for DoS safety | 2026-09 | http.md |
| Server headers timeout | min(reqTimeout, 60 s) | Node.js http | Slowloris protection when no reverse proxy fronts it | 2026-09 | http.md |
statement_timeout | 0 (off) | PostgreSQL | Global setting explicitly discouraged | 2026-09 | config.sgml |
Hedge maxAttempts ceiling | 5 | gRPC (gRFC A6) | Values >5 clamped to 5 without error | 2026-09 | A6 |
| Outage from ms→min query slowdown | 154 min | GitLab.com | Pool exhaustion; traffic −65% | 2021-03 | incident |
| Degradation from one hot-table txn | 30 min | GitLab.com | Webhooks table, wide blast radius | 2021-09 | incident |
Every figure here is measured from a primary artefact: a constant in source, a value in official docs, or a number in an incident review. None are vendor performance claims. Two caveats. First, defaults move: Node's server request timeout went from none to 300s between v13 and v18, so pin the version before trusting a number. Second, GitLab's statement_timeout was set to 10s in a 2016 decision record [4]; the later incidents are not evidence that timeout was wrong, but that a timeout cannot compensate for a query-planner regression or a hot-table lock.
Every source behind this page, graded. The mix is deliberately weighted to the repository record, code, issues, pull requests, specs and incident trackers, because the research network reached those and not the engineering-blog layer. Filter by kind.
A PostgreSQL 11 optimizer bug slowed IN (..) queries from ms to >2 min, exhausting the pool and taking the whole site down for 154 minutes.
A long transaction on the webhooks table caused database timeouts with a “large blast radius” because so many transactions touch webhooks; 30 minutes of site-wide degradation.
The request-path statement timeout also fires on maintenance: a schema migration hit it and needed a purpose-built index to complete.
A streaming RPC through Envoy died at 15s carrying x-envoy-expected-rq-timeout-ms: 15000, a value the developer never set.
Rejected because honouring a client's grpc-timeout at the edge “is not safe”; must be opt-in per config.
Reproduced across four minor versions; a documented timeoutSeconds the kubelet silently never enforced.
A vendor asked to keep the bug on by default to avoid mass breakage; SIG Node chose correctness with an opt-out and warning events.
Issue closed to the “Bankruptcy” milestone in 2016; a 2024 PR to add 10s/30s defaults still open, milestone slipped then dropped.
Zero-means-infinity called a bad default by rsc in 2018 (still open); DefaultClient has none; and a set timeout that wasn't visible downstream let goroutines pile up.
Deleted the 2-minute default as semver-major because it was “surprising and problematic” and “specific to Node.js.” Shipped v13.
DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0) in httpx; DEFAULT_TIMEOUT = 60 in botocore. The answer the older cores wouldn't commit to.
“A broadcast Context that represents when the request should be completed by,” propagated automatically across service boundaries.
Collapses five-plus additive timeout knobs into one whole-operation budget, and forbids un-setting it once specified at any level.
Hedging sends the same RPC again after hedgingDelay without waiting for failure, bounded by the one call deadline and a failure-ratio throttle; maxAttempts capped at 5.
Makes the kubelet honour the documented probe timeout, GA-on-by-default with a gate and warning events for those who relied on the bug.
GitLab.com ran with statement_timeout = 0 (infinite) until this record set it to 10s, with backup tasks overriding back to 0.
Defines the deadline on the wire, and that an omitted timeout means the server should assume infinite.
The canonical catalogue: route 15s, stream idle 5min, connection idle 1h, request headers disabled by default; explicit warnings about streaming.
No default deadline; propagation opt-in in C++, default in Java/Go; deadline shipped as remaining-time to survive clock skew.
Server request timeout none→300s in v18; headers timeout defaults to min(reqTimeout, 60s); must be non-zero for Slowloris protection without a reverse proxy.
Defaults to 0 (disabled) and warns against a global setting because “it would affect all sessions.”
The AWS SDK for Python applies a 60-second connect and read default when the caller passes none.
Six rungs from a hang you can watch to a deadline you can prove. The line from toy to real is at rung 3, where the timeout stops being local and starts being a budget the whole chain shares.
Write a client that calls a server which sleeps forever, using your language's default HTTP or RPC client with no timeout set. Watch the caller block.
Done when: the client is stuck and you can point to the exact line that owns the frozen thread. Teaches: “no timeout” is a decision, and it is the default in more places than you expect.
Add a read/idle timeout and, separately, an overall timeout. Make the server dribble one byte every few seconds so the idle timer never fires but the overall one does. Then make it go fully silent so only the idle timer fires.
Done when: you can trigger each timeout independently and name which is which. Teaches: why Node removed then re-added its server timer, and why Envoy needs both a stream idle and a route timeout.
Insert a middle service: client → A → B. Pass a deadline from the client, have A subtract the time it already spent before calling B, and have B refuse work if the remaining budget is negative.
Done when: a slow A leaves B with less budget, and a client-side deadline of zero means B is never called. Teaches: the deadline, not the timeout, is the unit that composes; this is the rung that crosses from toy to real.
When the deadline expires at the client, propagate cancellation so B actually stops. Instrument B to log whether it kept running after the client gave up.
Done when: B's log shows it stopped, not that it finished work nobody read. Teaches: the gRPC server “automatically cancelling a call” is the difference between a deadline and a lie; an expired deadline with no cancellation still burns the resource.
Add retries with jitter, but bound the total by the remaining deadline rather than a fixed attempt count. Then hedge a second read-only request after a delay and cancel the loser.
Done when: a saturated dependency does not receive more load under retry than under no-retry, and hedging only fires on the slow tail. Teaches: gRFC A6's rule that the call deadline bounds the whole retry/hedge fan-out; hedging trades load for tail latency and only on idempotent calls.
Add a test that asserts the timeout actually fires: inject a slow dependency and assert the call fails within the budget, not that it eventually returns. Assert cancellation reached the leaf.
Done when: the test fails if someone removes the timeout or breaks propagation. Teaches: the Kubernetes exec-probe lesson, a timeout the platform accepts but never enforces is worse than none, so the enforcement itself is what you test.
The queries that surfaced this material. The repository-record ones are the highest yield here, because the network reached code hosts when it could not reach blogs; adapt them to your own stack's frameworks.
<library> default timeout issue "no default" OR "zero means"repo:<org>/<repo> is:pr is:closed is:unmerged timeout defaultDEFAULT_TIMEOUT path:**/*.py OR path:**/*.go"surprising and problematic" OR "not common in other" timeoutgrpc-timeout OR "deadline propagation" path:doc"broadcast context" OR "request deadline" <framework>"client side operations timeout" OR "operation timeout" specgl-infra/production statement_timeout OR "database timeouts" incident"upstream request timeout" OR "expected-rq-timeout" issue<proxy> 15s OR "route timeout" streaming broke<platform> timeout "not respected" OR "ignored" issuefeature gate timeout "on by default" OR "opt out" KEP OR RFCrepo:<org>/<repo> is:issue timeout "does not" OR "silently"