Evidence ledger
33 sources
Checked 06 Sep 2026
Evidence ledger
One row per claim in A million clients, four hundred connections: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems share a small budget of database connections among a very large number of clients; what the pool costs, where it sits, and how it fails.
Checked date for every row: 2026-09-06.
Retrieval method note. This guide was built in a sandboxed environment whose egress policy
allows direct fetching only of github.com and gitlab.com. Rows marked fetched were
retrieved in full. Rows marked search-verified are sources on blocked hosts whose exact
quoted sentences and figures were retrieved verbatim in this session through live web search
result content, not from memory; the URLs come from those live results. No quote in this ledger
is reconstructed from memory. The page's link checker will report the blocked hosts as
unreachable from this environment; that is a property of the build sandbox, not of the links.
| # | Org | Title | Tier | Published | Method | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | HikariCP (Brett Wooldridge) | About Pool Sizing (project wiki) | source | n.d. (wiki, circa 2014, still live) | fetched | https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing | The doctrine: pools should be small and saturated | "You want a small pool, saturated with threads waiting for connections." |
| 2 | HikariCP | About Pool Sizing | source | n.d. | fetched | https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing | Sizing formula | "connections = ((core_count * 2) + effective_spindle_count)" |
| 3 | HikariCP | About Pool Sizing | source | n.d. | fetched | https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing | Shrinking the pool alone cut response times ~50x in Oracle's demo | "reducing the connection pool size alone, in the absence of any other change, decreased the response times of the application from ~100ms to ~2ms -- over 50x improvement" |
| 4 | HikariCP | About Pool Sizing | source | n.d. | fetched | https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing | 10,000 users does not mean 10,000 connections | "a connection pool of 10,000 would be shear insanity"; recommends "a small pool of a few dozen connections at most" |
| 5 | PostgreSQL | Commit dc7420c2c92 "snapshot scalability" (Andres Freund) | source | 2020-08 | fetched | https://github.com/postgres/postgres/commit/dc7420c2c92 | Why many connections slowed everyone: snapshot construction touched every proc | xmin accesses caused "many cache misses inside GetSnapshotData()", "the most significant source of GetSnapshotData() scaling poorly on larger systems" |
| 6 | Microsoft / Citus (Andres Freund) | Analyzing the Limits of Connection Scalability in Postgres | blog | 2020-10-08 | search-verified | https://www.citusdata.com/blog/2020/10/08/analyzing-connection-scalability/ | One active connection slows >2x when many idle connections exist; snapshot scalability is the binding constraint | "A single active connection slows down by more than 2x due to concurrent idle connections"; "the most pressing issue for connection scalability is snapshot scalability" |
| 7 | Microsoft / Citus (Andres Freund) | Analyzing the Limits of Connection Scalability in Postgres | blog | 2020-10-08 | search-verified | https://www.citusdata.com/blog/2020/10/08/analyzing-connection-scalability/ | The per-connection memory myth: ~1.3 MiB (huge_pages) to ~7.6 MiB true overhead, not tens of MB | idle connection ~16 MiB RSS but "true Proportional Set Size overhead is closer to 7.6 MiB"; "With huge_pages=on, the true overhead drops to about 1.3 MiB" |
| 8 | PgBouncer | PR #845 "Support of prepared statements" (knizhnik) | source | opened 2023-05-14, merged 2023-10-05 | fetched | https://github.com/pgbouncer/pgbouncer/pull/845 | The merged implementation of prepared statements in transaction mode | JelteF merged 2023-10-05: "I released v1.21.0 today, which includes this feature!" |
| 9 | PgBouncer | PR #757 "Server-side prepared statements cache" (dashorst) | source (closed unmerged) | opened 2022-08-18, closed 2023-08-29 | fetched | https://github.com/pgbouncer/pgbouncer/pull/757 | The rejected sibling attempt; a year of review found hash collisions and protocol bugs | "To make prepared statements work in this mode would need PgBouncer to keep track of them internally, which it does not do"; JelteF: "To avoid confusion, I'm going to close this in favor of #845." |
| 10 | PgBouncer | Release 1.21.0 "The one with prepared statements" | source | 2023-10-16 | fetched | https://github.com/pgbouncer/pgbouncer/releases/tag/pgbouncer_1_21_0 | The feature was the most requested in the project's history and is worth 15-250% throughput | "Add support for protocol-level named prepared statements! This is probably one of the most requested features"; "increase query throughput anywhere from 15% to 250%, depending on the workload" |
| 11 | PgBouncer | Issue #1021 "Feature: Multi-threading in PgBouncer" (JelteF) | adr | 2024-02-08, open as of 2026-09 | fetched | https://github.com/pgbouncer/pgbouncer/issues/1021 | The recorded design argument for/against multi-threading; the so_reuseport workaround's stated costs | so_reuseport setup "not super easy"; "Pool limits aren't shared across multiple processes"; "Statistics queries only report metrics from a single process" |
| 12 | PgBouncer | doc/config.md (pool modes, defaults) | vendor | current master | fetched | https://github.com/pgbouncer/pgbouncer/blob/master/doc/config.md | Pool mode semantics and defaults; transaction pooling forbids session features | transaction: "Server is released back to pool after transaction finishes"; "clients must not use any session-based features, since each transaction ends up in a different connection"; default_pool_size "Default: 20"; max_client_conn "Default: 100" |
| 13 | pgjdbc | Issue #869 "PrepareThreshold=0 but prepared statement is stored" | source | 2017-07-20 | fetched | https://github.com/pgjdbc/pgjdbc/issues/869 | The long tail of client casualties: driver still prepares under fetchSize+autoCommit=false, breaking transaction pooling | prepareThreshold=0 + autoCommit=false + fetchSize>0 yields ERROR: prepared statement "S_1" does not exist |
| 14 | GitLab | Runbook docs/pgbouncer/README.md | source | current master | fetched | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/pgbouncer/README.md | How GitLab actually deploys the pooler: dedicated hosts for the primary, co-located processes for replicas, transaction mode | "Primary (read-write) has 3 dedicated hosts in front of the database host"; "Replica (read-only) had 3 PgBouncer processes running on the same host that is running the PostgreSQL process"; clients "use long-lived connections to execute transactions from different requests spread over time" |
| 15 | GitLab | Issue #6981 "More scalable database connection pooling (currently pgbouncer)" | adr | n.d. (production-engineering tracker) | fetched | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/6981 | The design problem statement: single-threaded pooler pegging one core degrades query latency | "significant degradation of database query latencies when we scaled up the web worker fleet and/or bumped pgbouncer's max_client_conn"; processes "reaching their limits in terms of CPU (pegging one core)" because "pgbouncer is single-threaded" |
| 16 | GitLab | Production incident #7565 "pgbouncer_client_conn_primary saturation" | postmortem | 2022-08-08 | fetched | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7565 | The client-connection budget of the pooler tier itself saturates in production | Incident declared 2022-08-08 12:16 UTC for pgbouncer client connection saturation on the primary pool (review fields left incomplete; the incident record itself is the evidence) |
| 17 | Red Hat | About the Quay.io Outage: Post Mortem | postmortem | 2020 (outage 2020-05) | search-verified | https://www.redhat.com/en/blog/about-the-quay.io-outage-post-mortem | An unbounded per-worker connection default turned load into a lock-up | "a storm of tens of thousands of database connections, effectively locking the MySQL instance"; workers used "default database connection settings"; load tests found "10,000 connections as the maximum before Quay began returning 502 errors"; fix made per-gevent-worker connection limits configurable |
| 18 | GitHub | February service disruptions post-incident analysis | postmortem | 2020-03-26 | search-verified | https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis/ | Crossing a connection-count threshold changed the pooling tier's behavior | "In this third incident involving ProxySQL, active database connections crossed a critical threshold that changed the behavior of this new infrastructure." |
| 19 | Honeycomb | Incident Review: What Comes Up Must First Go Down | postmortem | 2023-08 (incident 2023-07-25) | search-verified | https://www.honeycomb.io/blog/incident-review-what-comes-up-must-first-go-down | Recovery is the second incident: reconnection and cold caches re-saturate the connection budget | outage 13:40-14:48 UTC; "Bringing ingest back without a cache would make it go down again, either through overload or database connection saturation"; they "denied all Shepherd traffic with a 5xx error to protect the database" and failed over to a replica |
| 20 | Honeycomb | Postmortem: RDS Clogs & Cache-Refresh Crash Loops | postmortem | 2019 (incidents Oct 4 and Oct 11) | search-verified | https://www.honeycomb.io/blog/postmortem-rds-clogs-cache-refresh-crash-loops | A slightly slow database multiplies concurrent DB work; crash-restart loops clear caches and re-hit the database | incidents 21:02-21:56 UTC and 15:00-16:02 UTC; "RDS was stalled at ~90% CPU"; more goroutines refreshing the cache created "a feedback loop"; fix: only "a single goroutine at a time refreshes the cache" |
| 21 | OpenAI | Scaling PostgreSQL to power 800 million ChatGPT users | casestudy | 2026 (covered by InfoQ 2026-02) | search-verified | https://openai.com/index/scaling-postgresql/ | The largest published single-primary deployment rations connections with a pooler on every replica | one primary, "approximately 50 read replicas", over "a million queries per second"; "PgBouncer (a connection pooling proxy) in front of every replica"; "Average connection time dropped from 50 milliseconds to 5 ms" |
| 22 | Figma | The growing pains of database architecture | blog | 2023 (describes 2020) | search-verified | https://www.figma.com/blog/how-figma-scaled-to-multiple-databases/ | The standard first move: insert a pooler when app connections reach the thousands | "added PgBouncer as a connection pooler to limit the impact of a growing number of connections (which were in the thousands)"; database traffic growing "approximately 3x annually" |
| 23 | Figma | PGKeeper: Building the Bouncer We Needed for Postgres | blog | 2026 (reports Q4 2025 results) | search-verified | https://www.figma.com/blog/pgkeeper-building-the-bouncer-we-needed-for-postgres/ | The pool's endgame is admission control, not connection reuse | outgrew PgBouncer due to "single-threaded architecture, lack of load management, and limited extensibility"; PGKeeper is a Go-based gRPC service with "pool warming, token-bucket rate limiting", "two-stage admission control using a priority-based semaphore with CoDel and adaptive LIFO"; "In Q4 2025 alone, PGKeeper prevented more than 20 incidents" |
| 24 | Notion | The Great Re-shard: adding Postgres capacity (again) with zero downtime | blog | 2023-07-17 | search-verified | https://www.notion.com/blog/the-great-re-shard | The pooler is the control plane for zero-downtime failover; explicit connection budgets per instance | expanded 32 to 96 instances; "created four new PgBouncer clusters, each managing 24 databases"; "increased connections per PgBouncer per shard to 8"; "limited total connections per Postgres instance to 200"; failover paused traffic in the PgBouncer cluster and resumed it on the new databases |
| 25 | Cloudflare | Pools across the sea: how Hyperdrive speeds up access to databases | blog | 2025-04-08 | search-verified | https://blog.cloudflare.com/how-hyperdrive-speeds-up-database-access/ | Connection setup costs seven round trips; edge poolers amortise it; transaction-mode pooling with per-client prepared statement tracking | "seven round-trips to your database before you can even send a query: the TCP handshake (1x), TLS negotiation (3x), and database authentication (3x)"; Hyperdrive keeps "track of what statements have been prepared by a given client, as well as what statements have been prepared on each origin connection in the pool" |
| 26 | Supabase | Supavisor: Scaling Postgres to 1 Million Connections | casestudy | 2023 | search-verified | https://supabase.com/blog/supavisor-1-million | The multiplexing ratio at the extreme: 1,000,000 clients onto 400 server connections | "connect to a million clients simultaneously" with "the pooler triaging the clients' queries to 400 database connections", achieving "20,000 queries per second"; 500,000 connections per Supavisor node |
| 27 | Supabase | supabase/supavisor README | source | current master | fetched | https://github.com/supabase/supavisor | Idle client connections are cheap at the pooler tier when the pooler is built for them | "250,000 idle connections on a single 16 core node with 64GB of ram"; motivation: PgBouncer "is single-threaded, making it difficult to scale"; goal of "zero-downtime scaling" |
| 28 | ClickHouse | How we scale PgBouncer in ClickHouse Managed Postgres | blog | n.d. (recent; checked 2026-09-06) | search-verified | https://clickhouse.com/blog/pgbouncer-clickhouse-managed-postgres | Measured single-process ceiling and the SO_REUSEPORT fleet remedy | "single process peaks around 87k transactions/sec and then gets worse under more load, sliding to 77k at 256 clients"; the fleet "keeps climbing to roughly 336k transactions/sec, about 4x"; "on a 16-vCPU box ... one core does all the connection pooling while the other fifteen sit idle" |
| 29 | Crunchy Data | Postgres at Scale: Running Multiple PgBouncers | blog | 2022 | search-verified | https://www.crunchydata.com/blog/postgres-at-scale-running-multiple-pgbouncers | Planning ceiling for one pooler process | "a single PgBouncer can process up to 10,000 connections. More specifically, 1,000 or so can be active at one time"; "you will never be able to devote more than 1 of your processors to PgBouncer" |
| 30 | AWS | Performance impact of idle PostgreSQL connections | vendor | 2020 | search-verified | https://aws.amazon.com/blogs/database/performance-impact-of-idle-postgresql-connections/ | Measured cost of idle connections on a small instance | db.m5.large with 1,000 idle connections: TPS "dropped from 1,249 to 1,140" (8.7%); select-only "dropped from 378 to 206 transactions per second (a 46% drop)"; idle connections "consumed the free OS memory that resulted in smaller OS cache" |
| 31 | AWS | RDS Proxy docs: Avoiding pinning | vendor | current | search-verified | https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-pinning.html | Managed multiplexing silently degrades to per-session when session state appears | "RDS Proxy automatically pins a client connection to a specific DB connection when it detects a session state change that isn't appropriate for other sessions"; "When a connection is pinned, each later transaction uses the same underlying database connection until the session ends." |
| 32 | Brandur Leach | How to Manage Connections Efficiently in Postgres, or Any Database | blog | 2018-10-15 | search-verified | https://brandur.org/postgres-connections | The hosted ceiling is low: the largest providers cap around 500 | "The largest cloud Postgres providers like Heroku top out at 500 connections", smaller plans at "20 or 25" |
| 33 | JP Camara | PgBouncer is useful, important, and fraught with peril | blog | 2023-04-12 | search-verified | https://jpcamara.com/2023/04/12/pgbouncer-is-useful.html | Transaction pooling breaks session semantics in ways that surface as production incidents | "the road to downtime is paved with session level statements"; SET statement_timeout can leak because "pgBouncer may switch Postgres backends between transactions" |
| 34 | Marc Brooker (AWS) | Open and Closed, Omission and Collapse | blog | 2023-05-10 | search-verified | https://brooker.co.za/blog/2023/05/10/open-closed.html | Why a bounded pool changes system behavior: closed loops queue, open loops collapse | "mean response times are significantly lower in closed systems than in open systems"; congestive collapse is "the most important thing to understand about open loop systems"; retries add arrival traffic |
| 35 | Schroeder, Wierman, Harchol-Balter | Open Versus Closed: A Cautionary Tale (NSDI '06) | paper | 2006-05 | search-verified | https://www.usenix.org/legacy/event/nsdi06/tech/full_papers/schroeder/schroeder.pdf | The formal basis: closed and open workload models behave vastly differently; most designers ignore the distinction | "system designers pay little attention to whether a workload generator is closed or open"; the paper shows "a vast difference in behavior between open and closed models" and derives eight principles |
| 36 | AWS (Barnhart, Brooker, et al.) | Resource Management in Aurora Serverless (VLDB Vol 17 No 12) | paper | 2024 | search-verified | https://assets.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf | The managed-database view: capacity for connections and buffers is oversubscribed and reactively managed | Aurora Serverless "adapts and fine tunes well-established ideas related to resource over-subscription; reactive control informed by recent measurements"; a core policy issue is "how to define 'heat' (resource usage features on which to base decision making)" |
| 37 | Jelte Fennema-Nio (Microsoft) | Comparing Postgres connection pooler support for prepared statements (POSETTE 2024) | talk | 2024-06 | search-verified | https://www.youtube.com/watch?v=O3gLgN517JA | Prepared statement support now differs by pooler; the protocol work is the hard part | "For a long time, connection poolers meant that you couldn't use prepared statements ... recently that has changed because prepared statements are now supported by PgBouncer, Odyssey, pgcat, and Supavisor" (talk abstract; timestamped citation unavailable from this build environment, claim taken from the published abstract) |
| 38 | Bohan Zhang (OpenAI) | Scaling Postgres to the next level at OpenAI (POSETTE 2025) | talk | 2025-06 | search-verified | https://posetteconf.com/2025/talks/scaling-postgres-to-the-next-level-at-openai/ | The connection-management chapter of the largest published Postgres deployment | session on "how they effectively manage connection pooling with PgBouncer" and scaling read replicas against millions of QPS (talk page; claim taken from the published session description) |
Tier tally
Counting distinct documents (a document with several claims has several rows above):
- postmortem: 5 (GitLab #7565, Quay.io, GitHub Feb 2020, Honeycomb 2023, Honeycomb 2019)
- source: 8 (HikariCP wiki, Postgres commit dc7420c2c92, pgbouncer PR #845, PR #757, release 1.21.0, pgjdbc #869, GitLab pgbouncer runbook, supavisor README)
- adr: 2 (pgbouncer #1021, GitLab #6981)
- casestudy: 2 (OpenAI, Supabase 1M benchmark)
- blog: 10 (Freund, Figma growing pains, Figma PGKeeper, Notion, Cloudflare, ClickHouse, Crunchy Data, Brandur, JP Camara, Brooker)
- paper: 2 (Schroeder et al. NSDI '06, Barnhart et al. VLDB '24)
- vendor: 3 (pgbouncer config.md, AWS idle connections, AWS RDS Proxy pinning docs)
- talk: 2 (Fennema-Nio POSETTE 2024, Zhang POSETTE 2025)
Distinct documents: 34. Distinct hosts: 22 (github.com, gitlab.com, citusdata.com, redhat.com, github.blog, honeycomb.io, openai.com, figma.com, notion.com, blog.cloudflare.com, supabase.com, clickhouse.com, crunchydata.com, aws.amazon.com, docs.aws.amazon.com, brandur.org, jpcamara.com, brooker.co.za, usenix.org, assets.amazon.science, youtube.com, posetteconf.com). Vendor share: 3/34, well under a third.