Evidence ledger
22 sources
Checked 22 Sep 2026
Evidence ledger
One row per claim in How long to wait: timeouts and deadlines in distributed systems: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Every URL was fetched during the research session on 2026-09-22. The research environment's outbound network policy allowed github.com, raw.githubusercontent.com, gitlab.com and the public package registries; engineering blogs, paper archives and video hosts were unreachable, so this guide is deliberately built from the repository record (code, issues, pull requests, specs, KEPs and GitLab's public incident tracker). That limit is stated in the guide.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitLab | 2021-03-09 High number of database statement timeouts (incident review) | postmortem | 2021-03 | 2026-09-22 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3875 | A full site outage of ~2.5 hours was produced not by a missing timeout but by queries slowing from milliseconds to minutes and exhausting the connection pool. | "For approximately 2.5 hours ... GitLab.com experienced a full site outage"; queries "taking a much longer time to execute than normal (from ms to over 2 minutes)"; 154 minutes downtime; traffic dropped 65%; PostgreSQL 11 optimizer "overestimates the cardinality ... when an IN (..) clause is evaluated to result in more than 200 results". |
| 2 | GitLab | 2021-09-07 Postgres transactions showing high rate of statement timeouts (incident review) | postmortem | 2021-09 | 2026-09-22 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5493 | A single long-running transaction on a hot table caused ~30 minutes of site-wide degradation; the statement timeout fired everywhere else, which was the symptom, not the cause. | "A long running transaction on the webhooks table that resulted in database timeouts, because so many transactions involve webhooks this problem had a large blast radius"; impact "09:48 - 10:16 (30 minutes)" with delays until 11:25 UTC. |
| 3 | GitLab | 2022-03-31 Post-deploy migration failing with statement timeout | postmortem | 2022-03 | 2026-09-22 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6737 | The request-path statement timeout fires on the maintenance path: a schema migration hit it and needed a purpose-built index to complete. | Post-deploy migration failed with statement timeout on gstg; MR gitlab-org/gitlab!84072 (specialized index) merged 2022-03-31 15:10 UTC, after which "the migration completes successfully". |
| 4 | GitLab | Set PostgreSQL statement_timeout to a non infinite time | adr | 2016-06 | 2026-09-22 | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/40 | GitLab.com ran with an infinite statement timeout until a deliberate decision record changed it, with a per-workload carve-out for backups. | "Currently this setting is set to 0 for GitLab.com meaning there is no timeout"; set to 10 seconds because "No query should ever take longer than this and if it does the query is the problem"; backup Rake tasks override back to 0. Created 2016-06-02 (GitLab API created_at). |
| 5 | Python Software Foundation (requests) | Consider making Timeout option required or have a default (#3070) | source | 2016-03 | 2026-09-22 | https://github.com/psf/requests/issues/3070 | requests has debated a default timeout since 2016 and closed the issue without one, filing it under the "Bankruptcy" milestone. | Opened by mlissner 2016-03-29; closed; assigned to the "Bankruptcy" milestone; opener "didn't realize the importance of providing a timeout" until a production hang. |
| 6 | Python Software Foundation (requests) | Add default timeout (PR #6709) | source | 2024-05 | 2026-09-22 | https://github.com/psf/requests/pull/6709 | A maintainer-authored PR adding 10 s connect / 30 s read defaults has been open since May 2024 and its milestone has slipped twice and then been removed. | Authored by sigmavirus24, 2024-05-21; proposes defaults of 10 s connect and 30 s read; milestone moved 2.33.0 to 2.34.0 (March 2026), then removed (May 2026); still open. |
| 7 | Encode (httpx) | httpx/_config.py | source | checked at master | 2026-09-22 | https://github.com/encode/httpx/blob/master/httpx/_config.py | The successor library to requests answered the same question the other way: a 5-second default timeout in code. | DEFAULT_TIMEOUT_CONFIG = Timeout(timeout=5.0) (line 246 at check time). |
| 8 | Go team | net/http: make default configs have better timeouts (#24138) | source | 2018-02 | 2026-09-22 | https://github.com/golang/go/issues/24138 | Go's own leadership proposed fixing zero-means-infinity defaults in 2018; the issue is still open in the Backlog milestone eight years later. | Opened by rsc 2018-02-26; Client, Server and Transport "timeout fields in which zero = infinity" should instead be "a reasonable default"; state Open, milestone Backlog. |
| 9 | Go team | net/http: add InactivityTimeout to http.DefaultClient (#22982) | source | 2017-12 | 2026-09-22 | https://github.com/golang/go/issues/22982 | http.DefaultClient ships with no timeout at all and the request to bound it sits unresolved for compatibility reasons. | var DefaultClient = &Client{}; demonstration of a request hanging indefinitely; labels NeedsFix and FrozenDueToAge; Backlog. |
| 10 | Go team | net/http: Client.Timeout is not propagated to Request's Context Deadline (#31657) | source | 2019-04 | 2026-09-22 | https://github.com/golang/go/issues/31657 | Even where a timeout exists, work continues past it if the deadline is not visible downstream: the caller got its error while goroutines piled up. | "Goroutines take off while the request returns the expected error. The goroutines increase and increase until they eventually drop off." Closed with milestone Go 1.14. |
| 11 | Node.js | http, http2: remove default server timeout (PR #27558) | source | 2019-05 | 2026-09-22 | https://github.com/nodejs/node/pull/27558 | Node deleted its 2-minute default server timeout as a semver-major change because it surprised users of long-lived connections. | "Timing out and closing the socket after two minutes have elapsed is surprising and problematic for users. This behavior was specific to Node.js, and doesn't seem to be common in other language runtimes." Shipped in v13.0.0 (2019-10-22). |
| 12 | Node.js | doc/api/http.md (server.requestTimeout, server.headersTimeout) | vendor | checked at main | 2026-09-22 | https://github.com/nodejs/node/blob/main/doc/api/http.md | The deleted default came back at different layers for security: headersTimeout (added v10.14.0/v11.3.0) and requestTimeout, whose default went from none to 300 s in v18.0.0. | requestTimeout "added: v14.11.0", changed in v18.0.0: "The default request timeout changed from no timeout to 300s (5 minutes)"; headersTimeout default "the minimum between server.requestTimeout or 60000"; "It must be set to a non-zero value ... to protect against potential Denial-of-Service attacks in case the server is deployed without a reverse proxy in front." |
| 13 | Envoy | FAQ: How do I configure timeouts? (docs/root/faq/configuration/timeouts.rst) | vendor | checked at main | 2026-09-22 | https://github.com/envoyproxy/envoy/blob/main/docs/root/faq/configuration/timeouts.rst | Envoy ships at least half a dozen distinct timers with distinct defaults: route timeout 15 s, stream idle 5 min, connection idle 1 h, request headers timeout disabled. | Route timeout: "This timeout defaults to 15 seconds. This is typically a problem for streaming responses"; "The default stream idle timeout is 5 minutes"; "The default idle timeout if not otherwise specified is 1 hour"; request timeout "not enforced by default as it is not compatible with streaming requests". |
| 14 | Envoy | conn_manager: Disable stream idle timeout for gRPC requests (PR #5294, closed unmerged) | source | 2018-12 | 2026-09-22 | https://github.com/envoyproxy/envoy/pull/5294 | The proposal to honor the client's gRPC deadline in place of the proxy's idle timer was rejected because at the edge the client is untrusted. | mattklein123: "This change allows a potentially untrusted request to override a timeout, which at the edge is not safe"; must be "opt-in based on config". Closed without merging 2019-01. |
| 15 | Envoy (user report) | Envoy proxy with GRPC server streaming getting UNAVAILABLE: upstream request timeout (#17697) | source | 2021-08 | 2026-09-22 | https://github.com/envoyproxy/envoy/issues/17697 | Three years after the FAQ documented it, the 15-second default was still cutting off streaming RPCs for users who never chose it. | Client received "UNAVAILABLE: upstream request timeout"; request carried 'x-envoy-expected-rq-timeout-ms', '15000'. |
| 16 | gRPC | PROTOCOL-HTTP2.md (grpc-timeout wire format) | adr | checked at master | 2026-09-22 | https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md | The deadline travels on the wire as a header, and its absence is defined as infinite. | "Timeout → \"grpc-timeout\" TimeoutValue TimeoutUnit"; "If Timeout is omitted a server should assume an infinite timeout." |
| 17 | gRPC | Deadlines guide (grpc.io source, content/en/docs/guides/deadlines.md) | vendor | checked at main | 2026-09-22 | https://github.com/grpc/grpc.io/blob/main/content/en/docs/guides/deadlines.md | gRPC sets no deadline by default; propagation converts the deadline to a remaining-time timeout to shield against clock skew; propagation is opt-in in C++ and default in Java and Go. | "By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever"; "gRPC converts the deadline to a timeout from which the already elapsed time is already deducted"; "In some languages this behavior needs to be explicitly enabled (e.g. C++) and in others it is enabled by default (e.g. Java and Go)". |
| 18 | gRPC | gRFC A6: gRPC Retry Design (hedging) | adr | last updated 2024-08 | 2026-09-22 | https://github.com/grpc/proposal/blob/master/A6-client-retries.md | Hedging is specified as sending the same RPC again after a fixed delay without waiting for failure, bounded by the one call deadline and a failure-ratio throttle. | "After hedgingDelay has elapsed without a successful response, the second RPC will be issued"; "gRPC's call deadline applies across all attempts"; maxAttempts values "greater than 5 are treated as 5"; status "Implemented in: Java, .NET, Node, Go except hedging, and C-Core except hedging". |
| 19 | Kubernetes | KEP-1972: Kubelet Exec Probe Timeouts | adr | 2020-09 | 2026-09-22 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/1972-kubelet-exec-probe-timeouts/README.md | The documented probe timeout was never enforced for exec probes; the fix shipped behind a GA-on-by-default gate plus warning events so operators could see who relied on the bug. | "Kubelet today does not respect exec probe timeouts. This is considered a bug"; "existing workloads on Kubernetes that relied on this bug may unexpectedly see their probes timeout"; "If the feature gate ExecProbeTimeout is disabled ... add warning event". |
| 20 | Kubernetes / Microsoft AKS | kubelet: ship new ExecProbeTimeout featuregate as false (PR #97057, closed unmerged) | source | 2020-12 | 2026-09-22 | https://github.com/kubernetes/kubernetes/pull/97057 | A cloud vendor formally proposed keeping the bug on by default to avoid mass breakage; SIG Node refused, choosing correctness with an opt-out. | jackfrancis: enforcing suddenly "puts those previously operational Kubernetes environments at risk"; palma21: "it will be breaking a very large number of folks"; andrewsykim: "The kubelet should really do what is specified in the API"; derekwaynecarr: "leave the feature gate on and enable opt out". Closed unmerged. |
| 21 | Kubernetes (user report) | exec-type liveness or readiness probes ignore timeout (#94080) | source | 2020-08 | 2026-09-22 | https://github.com/kubernetes/kubernetes/issues/94080 | Users reproduced the unenforced timeout across four minor versions; the same container's Docker healthcheck honored it. | Reproduced on 1.15.11-eks, 1.16.14, 1.17.11, 1.18.8; expectation "the command is aborted after the timeout is reached and that the probe is considered failed". |
| 22 | MongoDB | Client Side Operations Timeout specification | adr | Accepted; checked at master | 2026-09-22 | https://github.com/mongodb/specifications/blob/master/source/client-side-operations-timeout/client-side-operations-timeout.md | MongoDB's cross-driver spec collapses five-plus additive timeout knobs into one whole-operation budget, and forbids turning the budget off once set. | "Users have many options to set timeouts ... serverSelectionTimeoutMS, socketTimeoutMS, connectTimeoutMS, maxTimeMS, and wTimeoutMS ... users are often unsure which timeout to use. Because some of these timeouts are additive, it is difficult to set a combination which ensures control will be returned to the user after a specified amount of time"; "If timeoutMS is specified at any level, it cannot be later changed to unset at a lower level." |
| 23 | Twitter (Finagle) | Contexts.rst (Deadline broadcast context) | source | checked at develop | 2026-09-22 | https://github.com/twitter/finagle/blob/develop/doc/src/sphinx/Contexts.rst | Finagle carries the deadline as request-scoped broadcast context across service boundaries. | "com.twitter.finagle.context.Deadline.current — A broadcast Context that represents when the request should be completed by." |
| 24 | AWS (botocore) | botocore/httpsession.py | source | checked at develop | 2026-09-22 | https://github.com/boto/botocore/blob/develop/botocore/httpsession.py | The AWS SDK for Python ships a 60-second default for both connect and read. | DEFAULT_TIMEOUT = 60 (line 82 at check time); applied when timeout is None. |
| 25 | PostgreSQL | doc/src/sgml/config.sgml (statement_timeout, transaction_timeout) | vendor | checked at master | 2026-09-22 | https://github.com/postgres/postgres/blob/master/doc/src/sgml/config.sgml | Postgres defaults to no statement timeout and explicitly advises against setting one server-wide. | "A value of zero (the default) disables the timeout"; "Setting statement_timeout in postgresql.conf is not recommended because it would affect all sessions." |