Portfolio ↗
Evidence ledger 22 sources Checked 12 Oct 2026

Evidence ledger

One row per claim in The data survives the upgrade. The database around it does not.: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.

Field guide: The data survives the upgrade. The database around it does not: moving a live relational database to a new major engine version

Research date: 2026-10-12. Every source below was fetched in this session on 2026-10-12.

Hunt constraint, stated up front

This session's egress policy permitted gitlab.com, github.com (git reads and raw file fetches of public repositories; not the HTML issue pages or the search API), raw.githubusercontent.com, hub.docker.com, pypi.org, pkg.go.dev and the package registries. It refused every engineering-blog, documentation-site, conference and preprint host tried, including retool.com, knock.app, github.blog, www.revenuecat.com, www.postgresql.org, wiki.postgresql.org, aws.amazon.com and web.archive.org (CONNECT 403 from the egress proxy).

Consequences, recorded in the guide rather than papered over:

  1. The guide is built from the artifact record: GitLab's public production tracker (change requests and incident reviews), the PostgreSQL mirror's commit history and in-repo documentation, Zalando's Patroni and Spilo repositories, GitLab's runbooks repository, and the documentation that ships inside reachable repositories. The blog, paper and talk tiers are empty on purpose.
  2. Well-known blog-tier accounts of this problem (Retool's 4 TB Postgres 9.6→13 migration, Knock's 11.9→15.3 upgrade, GitHub's MySQL 5.7→8.0 fleet upgrade, RevenueCat's Aurora 10→14 postmortem) were located via web search but could not be fetched, so they are not cited anywhere in the guide. The guide says where the reachable record runs out.
  3. Commit content from postgres/postgres, zalando/spilo, zalando/patroni, docker-library/postgres and gitlab-com/runbooks was retrieved by git clone/git log and by raw.githubusercontent.com/gitlab.com/-/raw fetches in this session; the citation URLs below point at the corresponding github.com/gitlab.com web views of the same commits and files, which this session's link checker may report as unreachable (proxy policy), not dead.

GitLab issue descriptions are public through the REST API and were fetched; issue comments return 401 to anonymous API calls, so nothing is cited from comments.

The ledger

# Org Title Tier Published Checked URL Claim I take from it Supporting quote or figure
1 GitLab Incident Review: GitLab.com is down (severity::1, Service::Postgres) postmortem 2024-09-03 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/18491 The only Sev1 full outage in GitLab's 2024–2026 upgrade program came from operating the upgrade scaffolding, not from the cutover: a wrong-cluster shutdown during an incident call. "GitLab.com was completely inaccessible for several minutes because the Postgres database cluster had been shut down." / "the command mistakenly executed was knife ssh \"role:gprd-base-db-patroni-main-v14\" \"sudo service patroni stop\" — It should have been to \"role:gprd-base-db-patroni-main-v16\"" / Total Duration: 20mins; Customers Affected: All
2 GitLab 2025-07-29: High error rate in transactions_primary of patroni-ci service violating SLO (severity::3) postmortem 2025-07-29 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20273 A production SLO violation directly attributed to an engine bug surfaced by the PG17 upgrade. "The transactions_primary component of the patroni-ci service is experiencing a high error rate of 92.52% due to frequent transaction rollbacks." / "The high transaction rollback rates were caused by a PostgreSQL bug triggered during the recent upgrade to PG17."
3 GitLab 2026-07-13: More than one postgres instance in read-write mode for patroni-ci (severity::4) postmortem 2026-07-13 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22502 A rolled-back upgrade leaves debris: a leftover new-version node triggered a split-brain alert because monitoring did not distinguish it from the live cluster. "this was a false-positive alert caused by incorrect monitoring grouping" / "monitoring labels that did not distinguish the leftover Postgres 18 node (related to attempted upgrade over the weekend which was rolled back) from patroni-ci"
4 GitLab 2025-12-07: Patroni processes down on multiple patroni-main-v17-db-gprd hosts (severity::4) postmortem 2025-12-07 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20952 During the aborted December 2025 main-database PG17 window, 9 of 10 nodes of the new-version cluster were down; the parallel-cluster design confined the damage to the not-yet-live copy. "Patroni processes are not running on 9 out of 10 nodes in the PG17 investigation cluster." / "The affected nodes are part of a non-production investigation cluster and do not serve customer traffic."
5 GitLab [C2] [GPRD] [2026-07-11 - 2026-07-14] - Upgrade PostgreSQL to PG18 on CI (change::aborted) casestudy 2026-07-02 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22444 The current GitLab.com method: parallel cluster, logical replication, Consul switchover, reverse replication, DDL freeze; and this first production attempt was aborted. "upgrade a new standby cluster to PostgreSQL 18, and update the Consul service registration" / "convert to logical replication old PG17 -> new PG18" / "A CI-only-PCL is required for the entire change window because logical replication is incompatible with database migrations (DDL)." / "online process with effective zero downtime... risk of disruption of up to 5 mins if manual intervention is required for the Writer endpoint switchover" / 75-hour change window
6 GitLab [C2] [GPRD] [2026-08-08 - 2026-08-11] - Upgrade PostgreSQL to PG18 on CI (change::aborted) casestudy 2026-07-13 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22507 The retry of the aborted July attempt was itself aborted; the written plan names three rollback scenarios keyed to when an incident occurs, and DDL is the abort trigger. "This is a re-schedule of the aborted CR: #22444" / "Before Switchover: Abort the CR; the database remains on PG17." / "The CR only needs to be aborted if the incident fix requires a DDL (CREATE, ALTER, DROP, or REINDEX)."
7 GitLab [C2] [GPRD] [2026-08-21 - 2026-08-23] - Mock-up Upgrade PostgreSQL to PG18 on CI Test (change::complete) casestudy 2026-08-17 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22720 After two aborts, GitLab ran a dress rehearsal that cuts 50% of production workload to the new version, measures, and reverts by design. "50 % Workload switchover to PG18 (PG18 Node will get traffic)" / "Monitor and run pg_pref and Analyze Performance" / "End rollback window by Switching Traffic back 100% on Source Cluster."
8 GitLab [GPRD] [2025-12-05 to 2025-12-09] - Upgrade PostgreSQL to PG17 - main database (change::aborted) casestudy 2025-08-27 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20440 The PG17 upgrade is paired with an OS upgrade and explicitly budgets index-corruption repair; the December attempt on the main database was aborted. "upgrading the four Patroni clusters from PostgreSQL 16 to PostgreSQL 17, and from Ubuntu 20.04 to 22.04" / "Fix corrupt indexes (caused by the Ubuntu upgrade)" / "Establish reverse logical replication to the old PostgreSQL 16 cluster for backout" / Downtime Component: none
9 GitLab [GPRD] [2024-11-01 to 11-05] - Upgrade PostgreSQL to v16 on MAIN cluster (change::complete) casestudy 2024-10-21 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/18747 The 2024 main-database upgrade skipped a major version (14→16) over logical replication, ran amcheck on the copy for 12+ hours, and monitored a reverse-replication rollback window. "perform database upgrade in the target cluster (convert to logical replication old v14 -> new v16)" / "start amcheck in the v16 database (check sample of tables and index consistency), which might take 12+ hours to run" / "Enable reverse replication (new v16 -> old v14); Start the rollback window (monitor workload for performance regression in the new engine version)"
10 GitLab GPRD - C2 - Lower max_slot_wal_keep_size from 5200GB to 3640GB on patroni-main-v17 (change::complete) postmortem 2026-08-18 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22734 Replication slots are a standing liability: an inactive slot retained ~4 TB of WAL and pushed the production primary's disk to ~88%; the fix is a per-slot WAL retention cap sized against consumer-stall time. "during the incident an inactive replication slot retained ~4 TB of WAL and pushed the gprd patroni-main primary's disk to ~88%" / "gprd patroni-main-v17, 5200GB → 3640GB" / "at a 3640GB cap, the Siphon logical slot would be invalidated if its consumer stalled for roughly ~15h at average WAL rates (~7h at peak)"
11 GitLab [GPRD] [C1] Optimize Patroni CI Cluster — Disk Right-Sizing (64 TiB → 40 TiB) casestudy 2026-09-04 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22864 Scale context for the cluster whose PG18 upgrade was twice aborted: 64 TiB data disks per node, c4-highmem-144 machines. "Replace all nodes of the GPRD Patroni CI cluster (gprd-patroni-ci-v17) with nodes that have right-sized 40 TiB Hyperdisk Balanced data disks (down from 64 TiB)" / "same c4-highmem-144 / c4-standard-48 machine types"
12 GitLab [GPRD] [2024-09-27 to 10-02] - Upgrade PostgreSQL to v16 on CI cluster (change::aborted) casestudy 2024-09-20 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/18593 The abort pattern predates PG18: the first 2024 attempt at the CI v16 upgrade was also aborted and rescheduled (completed as work item 18639). Labels: change, change::aborted; succeeded by "[GPRD] [2024-10-26 to 10-29] - Upgrade PostgreSQL to v16 on CI cluster" (18639, change::complete)
13 GitLab Special Coverage Days - Post GPRD MAIN PG17 Upgrade - Friday 2026-02-06 to Tuesday 2026-02-10 casestudy 2025-12-18 2026-10-12 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21018 The main-database PG17 upgrade landed in early February 2026 (after the aborted December window), and GitLab scheduled dedicated post-upgrade coverage days for performance regressions. Title: "Special Coverage Days - Post GPRD MAIN PG17 Upgrade - Friday 2026-02-06 to Tuesday 2026-02-10"
14 GitLab Upgrading operating systems for PostgreSQL (gitlab-org/gitlab, doc/administration/postgresql/upgrading_os.md) source 2026 (current master) 2026-10-12 https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/postgresql/upgrading_os.md The locale trap, in GitLab's shipped documentation: a glibc change corrupts text indexes, glibc 2.28 is the named culprit, and the remedies are reindex-everything or rebuild from logical copy. "Changes to locale data in glibc can cause PostgreSQL database files to become incompatible between operating systems, leading to index corruption." / "In particular, the upgrade to glibc 2.28 is likely to cause this problem." / "REINDEX INDEX CONCURRENTLY" then "ALTER DATABASE REFRESH COLLATION VERSION"
15 GitLab gitlab-ctl pg-upgrade (gitlab-org/omnibus-gitlab, doc/settings/database.md) source 2026 (current master) 2026-10-12 https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/settings/database.md GitLab ships pg_upgrade wrapped in a 13-step script to every self-hosted install: locale-matched new cluster, copy, ANALYZE, and automatic revert on error; missing statistics are the documented degradation. "Creates a new directory containing a new, empty database with a locale matching the existing database." / "Runs ANALYZE to generate database statistics." / "If any errors are detected during this process, it reverts to the old version of the database." / "your upgrade will still be working but will have poor database performance until the database statistics are generated"
16 GitLab Runbooks: zero-downtime database decomposition & log analysis (gitlab-com/runbooks) source 2026 (current master) 2026-10-12 https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/zero-downtime-database-decomposition.md Sequence synchronisation is an explicit, scripted step in GitLab's logical-replication cutovers, in both directions; the duplicate-key error is a documented failure signature of a reset sequence. "Sync Postgres sequences for decomposed tables from the Source Cluster to the Target Cluster." / (log_analysis.md) "This can because of a sequence (implemented for serial columns) has been modified (i.e. has been RESET)."
17 GitLab Runbooks: OS Upgrade Reference Architecture (gitlab-com/runbooks, docs/patroni/os_upgrade_reference_architecture.md) source 2026 (current master) 2026-10-12 https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/os_upgrade_reference_architecture.md The parallel-cluster pattern is reused for OS changes: cascade standby clusters on the new OS (pg12-1604 → pg12-2004), then switch. "We will create additional cascade clusters on each environment": $ENV-patroni-main-pg12-1604 (Source), $ENV-patroni-main-pg12-2004 (Target)
18 PostgreSQL Commit 1fd1bd871: Transfer statistics during pg_upgrade (PG18) source 2025-02-20 2026-10-12 https://github.com/postgres/postgres/commit/1fd1bd871012732e3c6c482667d2f2c56f1a9395 After roughly two decades of pg_upgrade discarding planner statistics, PostgreSQL 18 carries them across; extended statistics still are not carried. "Add support to pg_dump for dumping stats, and use that during pg_upgrade so that statistics are transferred during upgrade. In most cases this removes the need for a costly re-analyze after upgrade." / "Some statistics are not transferred, such as extended statistics"
19 PostgreSQL Commit 29d0a77fa: Migrate logical slots to the new node during an upgrade (PG17) source 2023-10-26 2026-10-12 https://github.com/postgres/postgres/commit/29d0a77fa6606f9c01ba17311fc452dabd3f793d Logical replication slots survive pg_upgrade only from PG17 onward, and the upgrade deliberately fails on slots with unconsumed WAL to prevent silent data loss downstream. "Migration of logical replication slots is only supported when the old cluster is version 17.0 or later." / "If the old node has invalid slots or slots with unconsumed WAL records, the pg_upgrade fails. These checks are needed to prevent data loss."
20 PostgreSQL Commit 2c7ea57e5: Revert "Logical decoding of sequences" (PG15 cycle) source 2022-04-07 2026-10-12 https://github.com/postgres/postgres/commit/2c7ea57e56ca5f668c32d4266e0a3e45b455bef5 The engine's first attempt to replicate sequences logically was reverted wholesale; the gap every migration script works around is a design problem, not an oversight. "The implementation has issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed, but it'll require reworking significant part of the patch."
21 PostgreSQL Commit ec4831470: Revert per-index collation version tracking feature (PG14 cycle) source 2021-05-07 2026-10-12 https://github.com/postgres/postgres/commit/ec48314708262d8ea6cdcb83f803fc83dd89e721 The engine tried to track which indexes a collation change invalidates, and reverted it; operators therefore cannot ask the database which indexes are safe and reindex everything instead. "Design problems were discovered in the handling of composite types and record types that would cause some relevant versions not to be recorded... We're out of time for this release so we'll revert and try again." (reverts 257836a7 "Track collation versions for indexes", 2020-11-02)
22 PostgreSQL Commit 47ae6948f: pg_upgrade doc: mention that replication slots must be recreated source 2022-07-14 2026-10-12 https://github.com/postgres/postgres/commit/47ae6948f0b03cade761b80932c60f208b41f748 Before PG17 the slot loss was documented rather than fixed, and the documentation fix itself arrived only in 2022, backpatched to v10. Subject: "pg_upgrade doc: mention that replication slots must be recreated"; "Backpatch-through: 10"
23 PostgreSQL pg_upgrade reference, REL_17_STABLE (doc/src/sgml/ref/pgupgrade.sgml) vendor 2024 (REL_17_STABLE) 2026-10-12 https://github.com/postgres/postgres/blob/REL_17_STABLE/doc/src/sgml/ref/pgupgrade.sgml Through PG17 the official tool's own manual says statistics are not carried; link mode is the one-way door, copy mode the way back. "Because optimizer statistics are not transferred by pg_upgrade, you will be instructed to run a command to regenerate that information at the end of the upgrade." / "If the --link option was not used, the old cluster was unmodified; it can be restarted." / "--analyze-in-stages can be used to generate minimal statistics quickly."
24 PostgreSQL Logical replication restrictions, REL_17_STABLE (doc/src/sgml/logical-replication.sgml) vendor 2024 (REL_17_STABLE) 2026-10-12 https://github.com/postgres/postgres/blob/REL_17_STABLE/doc/src/sgml/logical-replication.sgml The two restrictions that shape every logical-replication upgrade: DDL is not replicated (hence the freeze) and sequence values are not replicated (hence the sync scripts). "The database schema and DDL commands are not replicated." / "Sequence data is not replicated. The data in serial or identity columns backed by sequences will of course be replicated as part of the table, but the sequence itself would still show the start value on the subscriber."
25 PostgreSQL Commits c92c3d50d (2014), edba754f0 + c9d502eb6 (2025), 626d7236b (2025), d44032d01 (2024) source 2014-04-14 … 2025-03-25 2026-10-12 https://github.com/postgres/postgres/commit/c92c3d50d46ae4f93ec750215935b24e49a84360 The engine has been absorbing the field's workarounds one release at a time: vacuumdb --analyze-in-stages (2014), pg_createsubscriber (PG17, 2024), vacuumdb --missing-stats-only guidance (PG18, 2025), pg_upgrade --swap (PG18, 2025). Subjects: "vacuumdb: Add option --analyze-in-stages" / "pg_createsubscriber: creates a new logical replica from a standby server" / "Update guidance for running vacuumdb after pg_upgrade... recommend using vacuumdb's new --missing-stats-only option" / "pg_upgrade: Add --swap for faster file transfer."
26 Zalando Patroni documentation: Major Upgrade of PostgreSQL Version (docs/existing_data.rst) adr current master 2026-10-12 https://github.com/zalando/patroni/blob/master/docs/existing_data.rst The HA layer explicitly scopes major upgrades out: stop Patroni, run pg_upgrade by hand, wipe the DCS state because the upgraded cluster has a new system identifier, re-replicate the standbys. "The only possible way to do a major upgrade currently is: Stop Patroni..." / "pg_upgrade runs initdb which actually creates a new database with a new PostgreSQL system identifier." / "Running pg_upgrade on standby nodes is not supported by PostgreSQL."
27 Zalando Spilo in-place upgrade implementation (postgres-appliance/major_upgrade/inplace_upgrade.py, pg_upgrade.py) source current master 2026-10-12 https://github.com/zalando/spilo/blob/master/postgres-appliance/major_upgrade/inplace_upgrade.py What it takes to automate pg_upgrade under an HA system: ~700 lines covering pause, clean shutdown, checkpoint-LSN wait, pg_upgrade -k -j N, rsync to replicas, DCS initialize-key swap, staged ANALYZE (with a custom-statistics-target trick dropped for PG18+), then a fresh backup. pg_upgrade_args = ['-k', '-j', str(psutil.cpu_count()), ...] / if int(self.desired_version) < 18: self.custom_stats_target_reanalyze() else: self.full_reanalyze() / "Resetting non-default statistics target before analyze" / self.postgresql.drop_possibly_incompatible_extensions() / logger.info('Upgrade downtime: %s', time.time() - downtime_start)
28 Supabase Platform documentation: Upgrading (apps/docs/content/guides/platform/upgrading.mdx) vendor current master 2026-10-12 https://github.com/supabase/supabase/blob/master/apps/docs/content/guides/platform/upgrading.mdx The managed-platform path takes downtime proportional to database size, and its prerequisites restate the same dropped-state list: slots must go, replicas must go, incompatible types must go. "Your project is taken offline" / "Your data is copied to the new instance and upgraded using pg_upgrade." / "a GP3 disk type by default, which will give ~100Mbps when upgrading" / "Projects with read-replicas can't be upgraded." / "Logical replication slots must be dropped."
29 Docker community / pgautoupgrade pgautoupgrade README source current main 2026-10-12 https://github.com/pgautoupgrade/docker-pgautoupgrade/blob/main/README.md The most-pulled database image ships no upgrade path, so a third-party image exists solely to run pg_upgrade --link on container start. "the official Docker PostgreSQL image has no ability to handle version upgrades, which leaves people to figure it out manually (not great): https://github.com/docker-library/postgres/issues/37" / "Backup your data! This Docker image does an in-place upgrade of the database data"
30 Docker community Docker Official Image: postgres (Docker Hub page) vendor 2026 (page as fetched) 2026-10-12 https://hub.docker.com/_/postgres The official image (1B+ pulls) recently restructured PGDATA specifically so cross-major pg_upgrade --link becomes possible across volume mounts. "This will allow users upgrading between PostgreSQL major releases to use the faster --link when running pg_upgrade and mounting /var/lib/postgresql." / Pull count badge: "1B+"
31 CockroachDB RFC 20170815: Version Migration (docs/RFCS/20170815_version_migration.md) adr 2017-07-10 2026-10-12 https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20170815_version_migration.md The contrast design: an engine built after this pain treats the version as a cluster setting, upgraded by rolling restart plus an explicit operator promise, because "incompatible features must not run in a mixed-version cluster". "a rolling restart followed by an operator-issued cluster-wide version bump that represents a promise that the old version is no longer running on any of the nodes" / "The core problem is that incompatible features must not run in a mixed-version cluster, yet this is precisely what happens if a rolling restart is used to upgrade the versions."
32 endoflife.date PostgreSQL release and EOL data (products/postgresql.md) source 2026 (current master) 2026-10-12 https://github.com/endoflife-date/endoflife.date/blob/master/products/postgresql.md The clock that forces all of this: each major gets about five years; PG13 died 2025-11-13, PG14 dies 2026-11-12, PG18 released 2025-09-25. releaseCycle "14": releaseDate 2021-09-30, eol 2026-11-12 / "13": eol 2025-11-13 / "12": eol 2024-11-21 / "18": releaseDate 2025-09-25, eol 2030-11-14

Corroboration and inference notes

  • The statistics trap is corroborated from four independent directions (the PG17 manual [23], the PG18 commit calling the re-analyze "costly" [18], Spilo's staged-analyze machinery [27], omnibus's documented ANALYZE step and its failure-mode warning [15]), but no reachable public postmortem narrates a production meltdown from missing statistics. RevenueCat's November 2022 postmortem reportedly does exactly that; it was unreachable this session and is not cited. Treat the mechanism as corroborated, the incident record as open.
  • The inference that aborts are designed outcomes rather than failures rests on counting labels across the tracker (18593, 19633, 19958, 20440, 22444, 22507 all change::aborted, each followed by a reschedule or a mock-up) plus the written rollback scenarios in [6]. GitLab does not state "we expect first attempts to abort" anywhere fetched.
  • Sev1 [1] is an upgrade-program incident by context, not by label. The incident review attributes the shutdown to confusion between -v14 and -v16 clusters that existed only because the upgrade's parallel cluster had been built two hours earlier. The guide presents it as the cost of running two near-identical clusters, which is what the review itself says.
  • Issue descriptions and labels were fetched via the GitLab REST API (gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues/<iid>); the cited web URLs render the same content.