The data survives the upgrade. The database around it does not

The territory

A storage system that must keep accepting writes has to be moved onto a new major version of its own engine, and the calendar, not the roadmap, decides when.

State the problem without naming the technology: a system of record holds tens of terabytes, serves every request path in the business, and runs an engine whose maintainers support each major version for about five years. When that clock runs out, staying put means running unpatched. Moving means changing the one component that holds all the state, under live writes, using official tooling that was designed around a stop-the-world step. Every team that operates a relational database hits this; the only choice is which shape of risk to take.

The deadline is not abstract. The PostgreSQL community's own release data, maintained in endoflife.date's source repository, gives every major five years: version 12 died in November 2024, 13 in November 2025, and 14 dies on 2026-11-12, one month after this guide's research date. GitLab.com's main database was still on 14 in late 2024 (its upgrade change request calls 14 "an outdated Postgres engine"). Whoever is on 14 today is in the last month of support.

Aborted upgrade attempts across GitLab.com's 2024 to 2026 campaigns, each rescheduled and re-run

GitLab production tracker, 2026

~4 TB

WAL retained by one inactive replication slot, pushing the production primary's disk to ~88%

GitLab change request, 2026

5 yr

Support life of a PostgreSQL major version; v14 reaches end of life 2026-11-12

endoflife.date, 2026

20 min

The only Sev1 in GitLab's upgrade programme: a wrong-cluster shutdown, not a cutover

GitLab incident review, 2024

The surprise in this record, and the organising idea of this guide: the upgrade moves the table bytes perfectly and silently drops almost everything derived from them. Four kinds of state do not come along: the planner's statistics, the current values of sequences (on the logical path), the validity of text indexes whose sort order the C library defines, and replication slots. Each omission has its own decade-long paper trail in the engine's commit history, its own revert, and its own workaround independently reinvented by every operator in this corpus. The engine has been absorbing those workarounds one release at a time, and that cadence, slots in 17 (2024), statistics in 18 (2025), sequences attempted in 2022, reverted, and now being redone, is why this topic reads differently in 2026 than it did three years ago.

Figure 1 · The engine absorbs the field's workarounds, one release at a time

Every entry is a commit in the PostgreSQL repository. The two reverts in the middle are the point: the engine tried to carry collation versions and sequences, and backed out both. Sources: 2014, 2021, 2022, 2023, 2025.

Diagram source

Who the record covers. GitLab.com, which runs its database change management and incident reviews in a public tracker and is mid-flight on its third annual major upgrade (its CI cluster alone keeps 64 TiB data disks per node, per its right-sizing change request). Zalando, whose Spilo appliance automates in-place upgrades for Patroni-managed clusters on Kubernetes. Supabase, whose managed platform documents a copy-and-upgrade path with planned downtime. The Docker official image, pulled over a billion times, which ships no upgrade path at all. And the PostgreSQL project itself, whose commits and reverts are the most honest design record in this corpus.

Scope. This guide covers major-version upgrades of PostgreSQL under live write load: the mechanics, the dropped state, the cutover, and the campaign around them. It does not cover minor-version patching, MySQL fleet upgrades, heterogeneous migrations between engines, or managed-cloud extended-support billing. One constraint shaped the hunt: this session's network could reach repository hosts and GitLab's API but no engineering-blog, conference or paywalled host, so the well-known blog-tier accounts (Retool's 4 TB migration, Knock's zero-downtime write-up, GitHub's MySQL 8.0 fleet upgrade, RevenueCat's statistics postmortem) were located but could not be fetched, and are deliberately not cited. What remains is arguably better: what organisations wrote down for themselves, not for an audience.

How it is actually built

Every implementation in the record is the same two moves: build a second database on the new version, then move the identity. Everything else is repair work for what the copy left behind.

Strip the four implementations to their shape and they agree. First, a complete second database comes into existence on the new major version: a parallel cluster fed by logical replication at GitLab, a new data directory beside the old one in Zalando's Spilo and in GitLab's omnibus package, a fresh instance that the platform copies data into at Supabase. Second, the identity of "the database" moves: GitLab updates a Consul service registration, Spilo renames data directories and swaps the cluster's initialize key in its distributed configuration store, omnibus repoints symlinks, Supabase moves the project endpoint. The write outage, where there is one, is exactly the width of the identity move.

Figure 2 · The reference shape: build the second database, then move the name

The GitLab.com variant. The dashed reverse path is established after cutover and is what makes the change abortable for days afterwards. Reconstructed from the PG18 change request and the 2024 v16 change request.

Diagram source

The reason the window is long is not the data copy. It is that logical replication, the only change stream that crosses major versions, refuses to carry two things. The engine's own manual is plain about both: The database schema and DDL commands are not replicated, and Sequence data is not replicated (PostgreSQL 17 documentation). The first restriction is why GitLab's change requests impose a production change lock for the entire multi-day window: A CI-only-PCL is required for the entire change window because logical replication is incompatible with database migrations (DDL) (PG18 CI change request, July 2026). The second is why sequence synchronisation appears as an explicit scripted step, in both directions, in GitLab's cutover runbooks. Miss it and every insert into a serial-keyed table fails at cutover, because the subscriber's sequences would still show the start value, as the manual puts it.

Figure 3 · What the upgrade carries, and what it drops

The left column survives any upgrade path. Everything on the right has been dropped by at least one official path, and each entry names the release that finally carried it, if any has. Sources: pg_upgrade manual, slot migration commit, statistics commit, GitLab OS-upgrade documentation.

Diagram source

The identity switch

Three mechanisms in the record: a Consul service registration flip (GitLab.com), a data-directory rename plus a forced swap of the cluster's initialize key (Spilo, because, as Patroni's documentation notes, pg_upgrade runs initdb and the upgraded cluster has a new PostgreSQL system identifier), and symlink repointing inside the package (omnibus). The switch is the only moment the application notices.

Runs this way at: GitLab, Zalando, omnibus installs

The repair crew

Every implementation ships the same post-copy fixes. Spilo resets per-column statistics targets, runs a staged ANALYZE, then restores the targets, about eighty lines just for statistics; omnibus documents an ANALYZE step and warns the install will have poor database performance until the database statistics are generated; GitLab runs amcheck against the new cluster for 12+ hours and schedules index rebuilds. None of this is optional decoration. It is the other half of the upgrade.

Sources: inplace_upgrade.py, omnibus docs, v16 change request

The guard rails

pg_upgrade --check runs before anything is touched (Spilo refuses to proceed past it). GitLab's change requests carry three written rollback scenarios keyed to when an incident occurs, and the written rule that only a fix requiring DDL forces an abort. The HA layer itself stands aside: Patroni's documentation states plainly that stopping Patroni is step one of any major upgrade.

Sources: Spilo, PG18 change request, Patroni docs

One divergence deserves its name. The in-place family (Spilo, omnibus, Supabase, pgautoupgrade) accepts an outage as wide as pg_upgrade plus the repair crew, and in exchange needs no second fleet, no DDL freeze and no sequence handling, because pg_upgrade's physical path carries sequences. The parallel-cluster family (GitLab.com) buys a near-zero-downtime cutover and a real rollback window, and pays with weeks of doubled infrastructure, a frozen schema, and the full dropped-state repair list. Section 03 turns that trade into conditions.

A second divergence is what "rollback" means. Spilo passes -k (hard links) to pg_upgrade for speed, which forfeits the old cluster the moment the new one writes; its safety is a fresh backup taken immediately after. Omnibus uses the copy mode and documents that if any errors are detected during this process, it reverts to the old version of the database. The engine's manual states the line both sides stand on: If the --link option was not used, the old cluster was unmodified; it can be restarted (PG17 pg_upgrade reference). Link mode is a one-way door, and both projects knowingly picked a different side of it.

The decisions that matter

Five forks in the road, each with the condition that flips it. The most instructive rejections here were made by the engine's own committers.

Stop-the-world pg_upgrade, or a parallel cluster over logical replication?

Chosen

GitLab.com: parallel cluster, logical replication, Consul cutover, reverse replication. Downtime Component: none on a database measured in tens of terabytes.
Zalando, omnibus, Supabase: in-place pg_upgrade, because they operate fleets of clusters where per-cluster ceremony must be near zero.

Rejected

GitLab.com rejected in-place for the SaaS platform: no rollback window once link-mode writes begin, and an outage proportional to repair time.
The fleet operators rejected logical replication: a DDL freeze and sequence choreography per tenant database does not automate across thousands of installs.

Flips when

Minutes of downtime are affordable: in-place wins on simplicity, and carries sequences for free.
Downtime is unaffordable or a post-cutover rollback is demanded: the parallel cluster is the only published path, and the DDL freeze comes with it.

Link mode speed, or copy mode retreat?

Chosen

Spilo: pg_upgrade -k -j <all cores>, minutes of downtime on large clusters, old directory deleted after a post-upgrade backup succeeds.
Omnibus: copy mode, with an automatic revert path it exercises on any error.

Rejected

Spilo rejected copy mode: doubling tens of terabytes of disk per cluster, fleet-wide, for a window nobody watches.
Omnibus rejected link mode: its users are thousands of unattended self-hosted installs where an automatic revert is worth hours of copying.

Flips when

You have a tested restore and a tight window: link.
Nobody will be watching when it fails: copy, and let the tool retreat. PG18's --swap (2025) narrows the gap from the link side.

Reindex everything, or trust the database to know which indexes a locale change broke?

Chosen

GitLab pairs each engine upgrade with an OS upgrade, assumes text indexes are invalid, and budgets Fix corrupt indexes (caused by the Ubuntu upgrade) as a plan step, validated by dedicated production rehearsal change requests.

Rejected

The selective path. The engine itself tried to track which indexes a collation change invalidates and reverted the feature before release: Design problems were discovered... we'll revert and try again. The database cannot tell you which indexes are safe.

Flips when

The OS, container image and glibc are byte-identical across the move: no reindex needed.
Anything crosses glibc 2.28, named by GitLab's documentation as the likely breaker: reindex every text index, then refresh collation versions.

Figure 4 · Choosing the mechanism: the conditions, as a tree

Terminal nodes are actions. The left branch is where most fleets live; the right branch is where most write-ups come from. Conditions derived from GitLab's change requests, Spilo and Supabase's platform docs.

Diagram source

Decision	Chosen	Rejected	Because	Flips when	Evidence
Upgrade cadence	Annual, one major per year (GitLab: 16 in 2024, 17 in 2025/26, 18 in 2026)	Multi-version jumps on a long cycle	The 2024 jump was itself two majors (14 to 16); the campaign cost amortises once the playbook is rehearsed	Fleet too small to keep a standing playbook: batch versions and jump	v16 CR, v17 CR
Slots across the upgrade	PG17+: let pg_upgrade migrate them; it fails on slots with unconsumed WAL rather than lose data	Silently recreating slots and losing the consumer's position	These checks are needed to prevent data loss	Old cluster is pre-17, or the platform cannot pause consumers: drop and resync, as Supabase requires	commit 29d0a77fa, Supabase docs
Sequences on the logical path	Scripted sync at cutover, in both directions	Engine-level replication of sequences	The engine committed it and reverted it: not clear how this could be fixed (2022)	A future release carries it; until then the script is mandatory	revert commit, runbooks
Statistics after upgrade	Pre-18: staged ANALYZE immediately, before resuming traffic	Serving traffic on a statistics-free planner	Omnibus: performance stays poor until the database statistics are generated	PG18+: statistics transfer in the upgrade; run `vacuumdb --missing-stats-only` for the remainder	omnibus docs, PG18 commit
Who owns the upgrade	A dedicated script above the HA layer (Spilo's 700-line inplace_upgrade.py; GitLab's playbooks)	Teaching the HA system to upgrade itself	Patroni scopes it out: The only possible way to do a major upgrade currently is: Stop Patroni...	An engine designed for it: CockroachDB upgrades by rolling restart plus an explicit version bump, by design	Patroni docs, CockroachDB RFC

The contrast worth carrying

CockroachDB's 2017 version-migration RFC is what the other side of this trade looks like: the cluster version is a setting, upgrades are rolling restarts, and the operator issues a promise that the old version is no longer running on any of the nodes before incompatible features switch on. The RFC names the core problem PostgreSQL's tooling externalises onto operators: incompatible features must not run in a mixed-version cluster, yet this is precisely what happens if a rolling restart is used to upgrade the versions. Engines designed after this pain treat the upgrade as a protocol, not an event.

What broke in production

Five incidents from GitLab's public tracker, 2024 to 2026. Not one of them is the cutover. They group into three classes: two databases that look identical, a new engine that behaves differently, and scaffolding with carrying costs.

Class 1 · You are now operating two databases that look identical

Postmortem

The wrong cluster got the shutdown command

AssumptionDuring an incident call, an operator can safely target one of two clusters whose role names differ by a version suffix.

What happenedHours after the new v16 standby cluster was built, responders investigating an unrelated Sidekiq degradation decided to stop the new cluster to shrink the search space. The command ran against role:gprd-base-db-patroni-main-v14, the live production cluster, instead of -v16. GitLab.com was completely inaccessible for several minutes because the Postgres database cluster had been shut down.

Blast radiusSeverity 1. All customers, all requests, 20 minutes, 503s at the edge.

FixRe-issue the start command; the review proposes replacing raw knife ssh with tooling that understands cluster roles.

Design ruleThe moment a parallel cluster exists, every destructive command has two near-identical targets. Name campaign clusters unmistakably, and put a guard between humans and fleet-wide service stops before building the second cluster, not after.

SourceGitLab incident review, 2024-09-03

Postmortem

The rolled-back upgrade left a node behind, and monitoring saw split-brain

AssumptionOnce an upgrade attempt is rolled back, the system is back to its pre-change state.

What happenedAfter the aborted PG18 weekend, a leftover PG18 node still reported as part of the CI cluster. Alerting grouped it with the live cluster and fired more than one postgres instance in read-write mode, the split-brain alarm, a false positive caused by incorrect monitoring grouping.

Blast radiusSeverity 4, no customer impact, but responder time spent ruling out the scariest alert a database can raise.

FixA monitoring label change so leftover campaign nodes no longer group with the live cluster.

Design ruleA rollback is not complete until the debris is unobservable. Decommissioning the abandoned cluster, and partitioning monitoring by campaign, is part of the abort procedure, not cleanup for later.

SourceGitLab incident, 2026-07-13

Figure 5 · The Sev1, step by step: the outage lives in the scaffolding

The new cluster was innocent; its existence was not. Sequence reconstructed from GitLab's incident review.

Diagram source

Class 2 · The new engine behaves differently under your workload

Postmortem

An engine bug surfaced by the upgrade pushed the SLO error rate to 92.52%

AssumptionA version that passed staging, rehearsal clusters and a production registry upgrade behaves the same on the CI cluster's workload.

What happenedDays after the CI cluster moved to PG17, the primary's transaction component showed a high error rate of 92.52% due to frequent transaction rollbacks, which the incident attributes to a PostgreSQL bug triggered during the recent upgrade to PG17.

Blast radiusSeverity 3, SLO-violating error rate on the component, no stated customer impact.

FixCompleted the remaining upgrades, silenced the alert class for future upgrade windows rather than paging the on-call.

Design ruleBudget a post-cutover observation window with named owners. GitLab formalised this as multi-day Special Coverage Days after the main-database PG17 cutover; the rollback window exists precisely because version-specific behaviour only shows under the real workload.

SourceGitLab incident, 2025-07-29; coverage days

Postmortem

Nine of ten nodes of the new-version cluster went down, inside the upgrade window

AssumptionThe new cluster, built and synced, is as operable as the old one.

What happenedPatroni processes are not running on 9 out of 10 nodes in the PG17 investigation cluster, during the December 2025 main-database upgrade window. Cause recorded as unknown.

Blast radiusNone for customers: the parallel-cluster design meant the broken cluster do[es] not serve customer traffic. The December change request was aborted; the upgrade landed in February.

FixSilence, investigate, re-run the campaign two months later.

Design ruleThe parallel cluster is both the risk (two identical targets) and the containment (failures before cutover cost a reschedule, not an outage). Preserve that asymmetry: never route traffic to the target early for convenience.

SourceGitLab incident, 2025-12-07; aborted CR

Class 3 · The scaffolding has carrying costs

Postmortem

An inactive replication slot quietly retained ~4 TB of WAL

AssumptionReplication slots, the mechanism every logical upgrade and CDC pipeline depends on, are safe to leave standing.

What happenedDuring the incident an inactive replication slot retained ~4 TB of WAL and pushed the gprd patroni-main primary's disk to ~88%. A slot is a promise to keep WAL until its consumer reads it; a stalled consumer converts that promise into unbounded disk growth on the primary.

Blast radiusProduction primary within reach of disk exhaustion, which is a full outage; caught before the cliff.

FixCap per-slot retention: max_slot_wal_keep_size lowered from 5200 GB to 3640 GB, sized so the capped slot survives roughly 15 hours of consumer stall at average WAL rates (7 at peak) before being invalidated.

Design ruleSize slot retention as a fraction of disk against measured WAL rate and a stall budget, and alert on slot lag long before the cap. An uncapped slot is an outage with a variable fuse.

SourceGitLab change request citing INC-12794, 2026-08

The dog that did not bark

Across three annual campaigns, roughly a dozen production clusters and six aborted attempts, GitLab's public record contains no incident in which the cutover itself, the seconds where writes move to the new primary, harmed customers. The incidents live before it (wrong-cluster shutdown, broken target cluster) and after it (engine bug under load, leftover-node alerts, slot retention). This matches the pattern in this practice's earlier dig on live resharding: the rehearsed moment is safe, the months around it are not.

One famous failure class is absent from the reachable record: a post-upgrade statistics meltdown, where the planner serves traffic with no row estimates. The mitigation machinery is everywhere (Spilo's staged ANALYZE, omnibus's warning, the PG18 fix calling re-analyze costly), which is strong indirect evidence, but no fetchable postmortem narrates one. Treat the mechanism as corroborated and the incident record as open.

Figure 6 · One cluster's road to PG18: the attempt, not the upgrade, is the unit of planning

GitLab's CI cluster, 2026. Two aborted production attempts, then a deliberate 50% dress rehearsal that reverts by design; the cluster was still on PG17 at research time. From 22444, 22507 and 22720.

Diagram source

Read the aborts correctly

Six aborted change requests in this record (CI v16 in 2024; staging and main PG17 in 2025; CI PG18 twice in 2026) might look like a struggling programme. The written rollback scenarios say otherwise: every abort happened at a stage designed to be abortable, and every abort cost a reschedule rather than an outage. The inference this guide draws, visible in the labels rather than stated by GitLab: a mature upgrade programme treats the attempt as the unit of planning, and prices aborts as normal outcomes.

Numbers you can plan against

Everything quantitative in the reachable record, dated and sourced. Measured and documented figures only; the one derived figure shows its arithmetic.

Metric	Value	At	Context	As of	Source
Support life of a major version	~5 years	PostgreSQL	v14: released 2021-09-30, EOL 2026-11-12; v18 released 2025-09-25	2026	endoflife.date
Change window per cluster upgrade	75 h	GitLab.com	DDL freeze Friday 22:00Z to Tuesday 14:00Z, CI cluster, logical path	2026-07	CR 22444
Write interruption at cutover	seconds	GitLab.com	enqueued during the switchover (a few seconds); up to 5 min if the writer switch needs manual intervention	2026-07	CR 22444
Consistency check on the target	12+ h	GitLab.com	amcheck over a sample of tables and indexes on the v16 cluster before cutover	2024-11	CR 18747
Data disk per CI-cluster node	64 TiB	GitLab.com	Right-sized to 40 TiB in September 2026; c4-highmem-144 machines	2026-09	CR 22864
WAL retained by one inactive slot	~4 TB	GitLab.com	Primary disk at ~88% before intervention	2026-08	CR 22734
Per-slot WAL retention cap	5200 → 3640 GB	GitLab.com	Cap sized to ~7% of data disk; capped slot survives ~15 h consumer stall at average WAL rate, ~7 h at peak	2026-08	CR 22734
Sev1 outage from the campaign	20 min	GitLab.com	All customers; wrong-cluster shutdown during preparation, not cutover	2024-09	incident review
Post-upgrade SLO error rate	92.52%	GitLab.com	Transaction rollbacks from a PostgreSQL bug surfaced by the PG17 upgrade	2025-07	incident
Managed-platform upgrade throughput	~100 Mbps	Supabase	Vendor figure for the default GP3 disk during its offline copy-and-upgrade; downtime scales with database size	2026	platform docs
Derived: offline window for 500 GB at that rate	~11 h	derived	500e9 bytes at 100 Mbps = 12.5 MB/s gives 40,000 s, about 11 h; estimate, assuming the vendor means megabits and the copy dominates	2026	estimate from the row above
Official image pulls, no upgrade path shipped	1B+	Docker Hub	The postgres official image; upgrades delegated to third-party pgautoupgrade or manual pg_upgrade	2026-10	Docker Hub
Aborted first attempts	6	GitLab.com	change::aborted labels across the v16, v17 and v18 campaigns, 2024 to 2026, each rescheduled	2026-10	tracker labels

Read these carefully

GitLab's figures are operational numbers from its own change records, the strongest kind in this table. The Supabase throughput is a vendor's own figure for its own platform and the unit is stated ambiguously; the derived row shows the arithmetic so you can rerun it with your size and their current figure. Nothing here covers what staying put costs: managed-cloud extended-support pricing was unreachable this session and is deliberately absent rather than quoted from memory.

The evidence wall

Every source behind this page, graded. The blog, paper and talk tiers are empty because this session's network could not reach those hosts; the ledger shipped beside this file names what was found but left uncited.

Postmortem GitLab2024-09

Incident Review: GitLab.com is down

Severity 1, all customers, 20 minutes. A shutdown aimed at the idle new v16 cluster hit the production v14 cluster: one suffix wrong in a knife ssh role. The only full outage in three years of upgrade campaigns, and the cutover was weeks away.

Carry forwardBuilding the parallel cluster doubles the targets of every destructive command. Guard the blast radius before the second cluster exists.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/18491

Postmortem GitLab2025-07

High error rate in transactions_primary of patroni-ci violating SLO

92.52% error rate from transaction rollbacks, attributed to a PostgreSQL bug triggered by the PG17 upgrade. The version cleared staging and rehearsal; the bug waited for the production workload.

Carry forwardVersion-specific behaviour appears only under real load. The rollback window and a staffed observation period are the mitigation, not more staging.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/20273

Postmortem GitLab2026-07

Patroni processes down on multiple patroni-main-v17 hosts

Nine of ten nodes of the new-version cluster down during the December upgrade window, cause recorded as unknown; zero customer impact because the cluster was not yet serving. The December change request was aborted, the upgrade landed in February.

Carry forwardPre-cutover failures cost a reschedule, not an outage. That asymmetry is the parallel cluster's whole value; protect it.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/20952

Postmortem GitLab2026-08

Inactive replication slot retained ~4 TB of WAL (INC-12794 follow-up)

The change request lowering max_slot_wal_keep_size documents the incident: a stalled slot consumer held ~4 TB of WAL and drove the production primary's disk to ~88%. The new cap is sized against measured WAL rates and a stall budget of hours.

Carry forwardEvery logical-replication upgrade and CDC pipeline leaves slots behind. Cap them as a fraction of disk and alert on lag well before the cap.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/22734

Case study GitLab2026-07

Upgrade PostgreSQL to PG18 on CI (both production attempts, aborted)

The full written method: standby cluster on PG18, logical replication, Consul switchover, reverse replication, 75-hour DDL freeze with the stated reason, three rollback scenarios keyed to incident timing, and the DDL-only abort rule.

Carry forwardWrite the rollback scenarios into the change plan before the window opens; GitLab's aborts are cheap because the abort path was already written.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/22444

Case study GitLab2026-08

Mock-up Upgrade PostgreSQL to PG18 on CI Test

After two aborts: a dress rehearsal that moves 50% of production workload to PG18, measures performance on both versions, and ends by switching 100% back and dropping replication, by design.

Carry forwardRehearse the rollback as the planned ending of a real traffic test, not as a contingency you hope never to run.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/22720

Case study GitLab2024-11

Upgrade PostgreSQL to v16 on MAIN cluster

The 2024 main-database upgrade: two majors in one hop (14 to 16) over logical replication, amcheck on the target for 12+ hours, reverse replication and a monitored rollback window. The template the later campaigns refined.

Carry forwardThe logical path decouples you from version adjacency: skipping a major costs nothing extra once the campaign machinery exists.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/18747

Case study GitLab2025-08

Upgrade PostgreSQL to PG17, main database (December attempt, aborted)

Pairs the engine upgrade with Ubuntu 20.04 to 22.04 and budgets Fix corrupt indexes (caused by the Ubuntu upgrade) as a plan step; validated beforehand by dedicated production change requests that rebuild indexes under logically replicated production workload.

Carry forwardIf the OS moves with the engine, index corruption is a planned work item, not a risk. Budget the reindex.

gitlab.com/gitlab-com/gl-infra/production/-/work_items/20440

Source PostgreSQL2025-02

Commit 1fd1bd871: Transfer statistics during pg_upgrade

PostgreSQL 18 finally carries planner statistics across the upgrade, removing, in the commit's words, the need for a costly re-analyze after upgrade in most cases. Extended statistics still do not transfer.

Carry forwardPre-18, the ANALYZE is part of the outage. From 18, verify the remainder with vacuumdb --missing-stats-only instead of re-analyzing everything.

github.com/postgres/postgres/commit/1fd1bd871

Source PostgreSQL2023-10

Commit 29d0a77fa: Migrate logical slots to the new node during an upgrade

From PG17, pg_upgrade carries logical replication slots, and deliberately fails on slots with unconsumed WAL: These checks are needed to prevent data loss. Before this, every CDC consumer lost its position at every major upgrade.

Carry forwardUpgrading from 17 or later, pause consumers and let the tool carry slots; from anything older, plan the drop-and-resync explicitly.

github.com/postgres/postgres/commit/29d0a77fa

Source PostgreSQL2022-04

Commit 2c7ea57e5: Revert "Logical decoding of sequences"

The engine's attempt to replicate sequences was reverted whole: issues, mostly due to combining transactional and non-transactional behavior of sequences. It's not clear how this could be fixed. The gap every cutover script works around is a hard design problem, not neglect.

Carry forwardTreat sequence sync as mandatory cutover choreography until a release ships it; check release notes before deleting your script.

github.com/postgres/postgres/commit/2c7ea57e5

Source PostgreSQL2021-05

Commit ec4831470: Revert per-index collation version tracking

The feature that would let the database name which indexes a glibc change broke was committed in 2020 and reverted in 2021 over design problems with composite and record types. It has not returned.

Carry forwardYou cannot ask the database which text indexes are safe after a locale change. Reindex them all, then REFRESH COLLATION VERSION.

github.com/postgres/postgres/commit/ec4831470

Source Zalando2026

Spilo: major_upgrade/inplace_upgrade.py

About 700 lines to automate pg_upgrade under Patroni: sanity checks, maintenance pause, clean shutdown, checkpoint-LSN wait, pg_upgrade -k -j N, rsync to replicas, DCS identity swap, staged ANALYZE with a statistics-target trick (dropped for PG18+), then a fresh backup. The script logs its own downtime.

Carry forwardThis file is the honest size of "just run pg_upgrade" in an HA cluster. Read it before estimating your own automation.

github.com/zalando/spilo/.../inplace_upgrade.py

Source GitLab2026

Runbooks: cutover sequence sync, upgrade procedures, failure signatures

The operational layer in public: explicit sequence-sync steps in both directions in the decomposition runbook, an OS-upgrade reference architecture built from cascade clusters, and a log-analysis entry naming duplicate-key errors from a reset sequence as a known signature.

Carry forwardThe duplicate-key storm right after cutover has one boring cause. Put sequence sync in the runbook and the signature in the triage guide.

gitlab.com/gitlab-com/runbooks/.../zero-downtime-database-decomposition.md

Source GitLab2026

omnibus-gitlab: gitlab-ctl pg-upgrade

The upgrade shipped to every self-hosted install: a 13-step wrapper that builds a locale-matched new cluster, copies with pg_upgrade, runs ANALYZE, and if any errors are detected during this process, it reverts to the old version. The docs name degraded performance from missing statistics as the expected failure mode.

Carry forwardCopy mode plus automatic revert is the right default for unattended upgrades; link mode is for watched ones.

gitlab.com/gitlab-org/omnibus-gitlab/.../database.md

Source pgautoupgrade2026

docker-pgautoupgrade README

A third-party image whose whole purpose in life is running pg_upgrade --link on container start, existing because the official Docker PostgreSQL image has no ability to handle version upgrades, with the open issue linked. Its first warning: back up, this is in-place.

Carry forwardIf your platform ships databases in containers, the upgrade path is your product's problem; the base image will not solve it.

github.com/pgautoupgrade/docker-pgautoupgrade

Decision record Zalando / Patroni2026

Patroni documentation: Major Upgrade of PostgreSQL Version

The HA layer's recorded position: The only possible way to do a major upgrade currently is: Stop Patroni, run pg_upgrade by hand, then wipe the DCS state because the upgraded cluster carries a new system identifier, and re-replicate every standby.

Carry forwardYour HA system will not own this change. Someone has to write the layer above it, which is what Spilo's script is.

github.com/zalando/patroni/blob/master/docs/existing_data.rst

Decision record CockroachDB2017-08

RFC: Version Migration

The counterfactual design: cluster version as a setting, rolling-restart upgrades, and an operator-issued bump that is a promise that the old version is no longer running on any of the nodes. Written by an engine that watched this problem from outside.

Carry forwardWhen evaluating a new engine, ask how it upgrades itself. The answer predicts a decade of your weekends.

github.com/cockroachdb/cockroach/.../20170815_version_migration.md

Vendor PostgreSQL2024

pg_upgrade and logical replication manuals, REL_17_STABLE

The authoritative statements of the gaps: statistics are not transferred (through 17), the old cluster is only restartable if --link was not used, DDL and sequence data are not replicated logically, and --analyze-in-stages exists to make the statistics gap survivable.

Carry forwardThe manual's Restrictions sections are the checklist of what your cutover script must do. Read them per target version; they shrink each release.

github.com/postgres/postgres/blob/REL_17_STABLE/doc/src/sgml/ref/pgupgrade.sgml

Vendor Supabase2026

Platform documentation: Upgrading

The managed path, stated plainly: the project goes offline, data is copied and upgraded with pg_upgrade, downtime scales with size at the disk's throughput, and the prerequisites are the dropped-state list again: slots dropped, read replicas deleted, incompatible types removed.

Carry forwardA managed platform does not delete this problem; it fixes the method and hands you the downtime. Read the prerequisites as your compatibility debt.

github.com/supabase/supabase/.../upgrading.mdx

Vendor Docker / GitLab docs2026

Official image PGDATA restructuring; GitLab OS-upgrade documentation

The official postgres image recently moved PGDATA so that cross-major pg_upgrade with --link works across volume mounts, a decade after the upgrade issue was opened. GitLab's documentation carries the collation warning: locale changes in glibc can cause PostgreSQL database files to become incompatible... leading to index corruption, with 2.28 named.

Carry forwardThe container image's directory layout and the host's glibc version are part of your database's on-disk format. Pin both; change them deliberately.

gitlab.com/gitlab-org/gitlab/.../upgrading_os.md

Build a miniature, then productionise it

Seven rungs from a container experiment to a rehearsed campaign. The toy crosses into the real at rung four, where the cutover gets a clock on it.

Hit the format boundary

Load a few gigabytes into the official postgres image on a named volume, then point the next major's image at the same volume and watch it refuse to start. Then run pgautoupgrade:latest against a copy of the volume and read its logs end to end.

Done when: you can explain why the refusal exists and what the auto-upgrade did instead. Teaches: the on-disk format boundary that makes all of this necessary.

Run pg_upgrade by hand, all three modes

On a 10 GB copy: --check first, then time --copy, --link and (on PG18) --swap. Delete a file from the old cluster after a --link run and observe what you have forfeited.

Done when: a three-row table of wall-clock times and a one-line answer to "what is restartable after each mode". Teaches: the speed-versus-retreat trade in section 03, physically.

Inventory the dropped state

Before and after an upgrade to a pre-18 target, diff: pg_stats row counts per table, sequence last values, replication slots, and EXPLAIN output for one selective query. Run the query before ANALYZE and watch the plan.

Done when: you have a written list of every object the upgrade did not carry, produced by queries, not by this guide. Teaches: the derived-state inventory as a habit rather than a fact.

Cut over on the logical path, against a write generator

Publication on the old major, subscription on the new, a writer inserting into a serial-keyed table throughout. Cut over by repointing the connection string. Do it once without syncing sequences and collect the duplicate-key errors; then script the sync and measure the write gap.

Done when: writes resume on the new primary with correct keys and the measured gap is under ten seconds. Teaches: the cutover choreography, and the sequence trap as lived experience.

Add the reverse path, then use it

Before cutover, prepare reverse replication from new to old. After cutover, run for ten minutes, then roll back. Verify no writes were lost in either direction with row counts and checksums.

Done when: you have gone forward and back with zero lost writes, twice. Teaches: rollback as a rehearsed ending, GitLab's mock-up in miniature.

Corrupt an index with glibc, then catch it

Build a text-indexed table in a container with an old glibc (anything pre-2.28), mount the data directory into an image with a newer glibc, run amcheck, and read the damage. REINDEX, refresh collation versions, re-run amcheck clean.

Done when: amcheck reports corruption before and silence after, and you can say which of your real indexes are exposed. Teaches: the environment is part of the on-disk format.

Turn it into a campaign

Write the change plan for your real database in GitLab's shape: schedule, the three rollback scenarios keyed to timing, the abort rule, slot WAL caps with a stall budget, monitoring that distinguishes campaign clusters from live ones, and a wrong-cluster drill where a colleague tries to stop production and a guard stops them.

Done when: a dry run aborts cleanly at each of the three stages and the drill is caught by tooling, not attention. Teaches: that the attempt, with its abort paths, is the deliverable.

Keep hunting

The queries that found this material, copyable. The tracker and commit-log searches are the ones ordinary search never runs, and they are where the operational truth lives.

Public change and incident trackers

https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?search=upgrade postgresql&scope=all
site:gitlab.com gl-infra production "Upgrade PostgreSQL" aborted
site:gitlab.com gl-infra "Incident Review" postgres
"change::aborted" postgres site:gitlab.com

The engine's own record

git log --grep="pg_upgrade" --grep="statistics" --all-match --oneline
git log --grep="Revert" --grep="sequence" --all-match --oneline
git log --grep="collation version" --oneline
git log --grep="replication slots" --grep="pg_upgrade" --all-match

Operational code and runbooks

github.com/zalando/spilo path:postgres-appliance/major_upgrade
site:gitlab.com gitlab-com/runbooks pg_upgrade OR "analyze-in-stages"
"max_slot_wal_keep_size" incident OR postmortem
"gitlab-ctl pg-upgrade" revert

The blog tier this guide could not reach

postgres major version upgrade postmortem "went wrong"
"zero downtime" postgres upgrade logical replication sequences
glibc collation postgres index corruption incident
"we upgraded" postgres 4TB OR "our main database"

References

GitLab, Incident Review: GitLab.com is down GitLab production tracker, 2024-09-03. Checked 2026-10-12.
GitLab, High error rate in transactions_primary of patroni-ci service violating SLO GitLab production tracker, 2025-07-29. Checked 2026-10-12.
GitLab, More than one postgres instance in read-write mode for patroni-ci GitLab production tracker, 2026-07-13. Checked 2026-10-12.
GitLab, Patroni processes down on multiple patroni-main-v17-db-gprd hosts GitLab production tracker, 2025-12-07. Checked 2026-10-12.
GitLab, Lower max_slot_wal_keep_size from 5200GB to 3640GB on patroni-main-v17 GitLab production tracker, 2026-08-18. Checked 2026-10-12.
GitLab, Upgrade PostgreSQL to PG18 on CI (first attempt, aborted) GitLab production tracker, 2026-07-02. Checked 2026-10-12.
GitLab, Upgrade PostgreSQL to PG18 on CI (second attempt, aborted) GitLab production tracker, 2026-07-13. Checked 2026-10-12.
GitLab, Mock-up Upgrade PostgreSQL to PG18 on CI Test GitLab production tracker, 2026-08-17. Checked 2026-10-12.
GitLab, Upgrade PostgreSQL to PG17, main database (aborted December window) GitLab production tracker, 2025-08-27. Checked 2026-10-12.
GitLab, Upgrade PostgreSQL to v16 on MAIN cluster GitLab production tracker, 2024-10-21. Checked 2026-10-12.
GitLab, Upgrade PostgreSQL to v16 on CI cluster (first attempt, aborted) GitLab production tracker, 2024-09-20. Checked 2026-10-12.
GitLab, Special Coverage Days, Post GPRD MAIN PG17 Upgrade GitLab production tracker, 2025-12-18. Checked 2026-10-12.
GitLab, Optimize Patroni CI Cluster, Disk Right-Sizing (64 TiB to 40 TiB) GitLab production tracker, 2026-09-04. Checked 2026-10-12.
GitLab, Upgrading operating systems for PostgreSQL gitlab-org/gitlab documentation, current master. Checked 2026-10-12.
GitLab, omnibus-gitlab database settings: gitlab-ctl pg-upgrade gitlab-org/omnibus-gitlab documentation, current master. Checked 2026-10-12.
GitLab, Runbooks: zero-downtime database decomposition gitlab-com/runbooks, current master. Checked 2026-10-12.
GitLab, Runbooks: OS Upgrade Reference Architecture gitlab-com/runbooks, current master. Checked 2026-10-12.
PostgreSQL, Transfer statistics during pg_upgrade (commit 1fd1bd871) postgres/postgres, 2025-02-20. Checked 2026-10-12.
PostgreSQL, Migrate logical slots to the new node during an upgrade (commit 29d0a77fa) postgres/postgres, 2023-10-26. Checked 2026-10-12.
PostgreSQL, Revert "Logical decoding of sequences" (commit 2c7ea57e5) postgres/postgres, 2022-04-07. Checked 2026-10-12.
PostgreSQL, Revert per-index collation version tracking feature (commit ec4831470) postgres/postgres, 2021-05-07. Checked 2026-10-12.
PostgreSQL, pg_upgrade doc: mention that replication slots must be recreated (commit 47ae6948f) postgres/postgres, 2022-07-14. Checked 2026-10-12.
PostgreSQL, vacuumdb: Add option --analyze-in-stages (commit c92c3d50d) postgres/postgres, 2014-04-14. Checked 2026-10-12.
PostgreSQL, pg_createsubscriber: creates a new logical replica from a standby (commit d44032d01) postgres/postgres, 2024-03-25. Checked 2026-10-12.
PostgreSQL, pg_upgrade: Add --swap for faster file transfer (commit 626d7236b) postgres/postgres, 2025-03-25. Checked 2026-10-12.
PostgreSQL, pg_upgrade reference (REL_17_STABLE source) postgres/postgres documentation source, 2024 branch. Checked 2026-10-12.
PostgreSQL, Logical replication, Restrictions (REL_17_STABLE source) postgres/postgres documentation source, 2024 branch. Checked 2026-10-12.
Zalando, Patroni documentation: Major Upgrade of PostgreSQL Version zalando/patroni, current master. Checked 2026-10-12.
Zalando, Spilo: inplace_upgrade.py zalando/spilo, current master. Checked 2026-10-12.
Supabase, Platform documentation: Upgrading supabase/supabase, current master. Checked 2026-10-12.
pgautoupgrade, docker-pgautoupgrade README pgautoupgrade/docker-pgautoupgrade, current main. Checked 2026-10-12.
Docker, postgres Official Image Docker Hub, page as of 2026-10. Checked 2026-10-12.
CockroachDB, RFC: Version Migration cockroachdb/cockroach, 2017-08-15. Checked 2026-10-12.
endoflife.date, PostgreSQL release and EOL data endoflife-date/endoflife.date, current master. Checked 2026-10-12.

The territory

Figure 1 · The engine absorbs the field's workarounds, one release at a time

How it is actually built

Figure 2 · The reference shape: build the second database, then move the name

Figure 3 · What the upgrade carries, and what it drops

The identity switch

The repair crew

The guard rails

The decisions that matter

Figure 4 · Choosing the mechanism: the conditions, as a tree

What broke in production

Class 1 · You are now operating two databases that look identical

Figure 5 · The Sev1, step by step: the outage lives in the scaffolding

Class 2 · The new engine behaves differently under your workload

Class 3 · The scaffolding has carrying costs

The dog that did not bark

Figure 6 · One cluster's road to PG18: the attempt, not the upgrade, is the unit of planning

Numbers you can plan against

The evidence wall

Incident Review: GitLab.com is down

High error rate in transactions_primary of patroni-ci violating SLO

More than one postgres instance in read-write mode for patroni-ci

Patroni processes down on multiple patroni-main-v17 hosts

Inactive replication slot retained ~4 TB of WAL (INC-12794 follow-up)

Upgrade PostgreSQL to PG18 on CI (both production attempts, aborted)

Mock-up Upgrade PostgreSQL to PG18 on CI Test

Upgrade PostgreSQL to v16 on MAIN cluster

Upgrade PostgreSQL to PG17, main database (December attempt, aborted)

Commit 1fd1bd871: Transfer statistics during pg_upgrade

Commit 29d0a77fa: Migrate logical slots to the new node during an upgrade

Commit 2c7ea57e5: Revert "Logical decoding of sequences"

Commit ec4831470: Revert per-index collation version tracking

Spilo: major_upgrade/inplace_upgrade.py

Runbooks: cutover sequence sync, upgrade procedures, failure signatures

omnibus-gitlab: gitlab-ctl pg-upgrade

docker-pgautoupgrade README

Patroni documentation: Major Upgrade of PostgreSQL Version

RFC: Version Migration

pg_upgrade and logical replication manuals, REL_17_STABLE

Platform documentation: Upgrading

Official image PGDATA restructuring; GitLab OS-upgrade documentation

Build a miniature, then productionise it

Hit the format boundary

Run pg_upgrade by hand, all three modes

Inventory the dropped state

Cut over on the logical path, against a write generator

Add the reverse path, then use it

Corrupt an index with glibc, then catch it

Turn it into a campaign

Keep hunting

References