XID wraparound  / field guide
Practitioner field guide · Data platforms · 2026-10-16

When maintenance loses the race: the vacuum PostgreSQL will not let you skip

PostgreSQL stamps every write with a 32-bit transaction ID and keeps the database writable only while a background job freezes old rows faster than new IDs are spent. This guide reconstructs how that race is run and lost, from the engine's own source and commits, five years of GitLab's public incident tracker, and the issue threads of the tools that collide with it. You leave able to size the job, keep its path clear, and read the gauge before the engine reads it for you.

47 ledger entries 9 organisations 9 incidents read in full Evidence through October 2026 Read: 35 min
01

A counter with a cliff, and a janitor who must keep ahead of it

The problem, stated without the database's vocabulary: a system hands out a finite, reusable sequence number on every write, and can only reuse a number once a background job has certified every row that still carries it. The job is the capacity limit. Nobody provisions it as one.

Why this topic, and why now: in the eight months to October 2026 the PostgreSQL project changed the counter's warning margin, the order in which autovacuum picks tables, the width of two of the engine's other counters, and the text of every warning on the path to the cliff; and on 5 August 2026 an anti-wraparound vacuum blocked GitLab.com's deploys for five hours. The mechanism is twenty years old. The engineering around it is moving fast, and most operators' runbooks have not moved with it.

The arithmetic is short. varsup.c sets the point of real danger at "halfway around from the oldest potentially-existing XID": the oldest unfrozen transaction ID in the database plus 231, about 2.147 billion transactions. A row is safe to keep past that only if it has been frozen, marked as visible to everyone so its original ID no longer matters. Freezing is done by VACUUM. When the oldest unfrozen ID in a table passes autovacuum_freeze_max_age (default 200 million), autovacuum is forced to run an anti-wraparound vacuum on it whether or not it is otherwise needed. If that still does not keep up, the engine warns on every ID it hands out from 100 million before the limit, and from 3 million before the limit it refuses to assign any more: in the documentation's words, "only read-only transactions can be started" [1][6].

231
Transactions between the oldest unfrozen row and the hard limit, about 2.147 billion
<3 days
For GitLab.com to burn the 200M XIDs that force an anti-wraparound vacuum
5 h
GitLab.com deploys blocked behind one anti-wraparound vacuum, August 2026
5+ days
Anti-wraparound vacuums seen on self-managed GitLab databases left on default settings
The surprise

The failure that recurs in the record is not the database running out of IDs. It is the safety mechanism arriving. PostgreSQL's lock manager cancels an autovacuum that is in someone's way, unless that autovacuum is "working to protect against Xid wraparound" [4]. So the one vacuum you cannot hurry is also the one vacuum that will not step aside, and it arrives on the engine's schedule, not yours. GitLab's deploy pipeline, pg_repack users on Amazon RDS and, by the index of its postmortem, Joyent in 2015 all met it at the lock queue rather than at the cliff. In March 2025 the engine's own developers started moving freezing out of these vacuums for exactly that reason [13].

Figure 1 · The ages a row passes through, and what the engine does at each

50M vacuum_freeze_min_age

200M autovacuum_freeze_max_age

1.6B vacuum_failsafe_age

limit minus 100M

limit minus 3M

Age 0
row written, XID stamped

Eligible to freeze
if a vacuum visits the page

Anti-wraparound autovacuum
mandatory, does not yield to lock waiters

Failsafe
cost delay off, index vacuuming skipped

WARNING on XID assignment
percentage of space left

Stop
no new XIDs, read-only transactions only

50M vacuum_freeze_min_age

200M autovacuum_freeze_max_age

1.6B vacuum_failsafe_age

limit minus 100M

limit minus 3M

Age 0
row written, XID stamped

Eligible to freeze
if a vacuum visits the page

Anti-wraparound autovacuum
mandatory, does not yield to lock waiters

Failsafe
cost delay off, index vacuuming skipped

WARNING on XID assignment
percentage of space left

Stop
no new XIDs, read-only transactions only

Every threshold after the second is the engine taking control away from the operator: from 200 million the vacuum is mandatory and will not yield, from 1.6 billion it drops its throttle, and 3 million before the limit writes stop. Defaults from guc_parameters.dat and varsup.c at HEAD, October 2026.
Diagram source

Who has written this down in public, at what depth: the PostgreSQL project, in source comments and commit messages that argue with each other across a decade; GitLab, whose production and product trackers are public and hold the wraparound incidents, change requests and proposals cited here, from 2019 to August 2026; and the maintainers of pg_repack, pganalyze's collector, Joyent's pgstatsmon and Postgres.ai's how-to library. The three famous wraparound outages (Sentry in 2015, Joyent in 2015, Mailchimp's Mandrill in 2019) sit behind hosts this session could not reach; they appear here only as one-line entries in a public index [42], and nothing is claimed from them beyond those lines.

Scope. Covered: transaction-ID and multixact-ID wraparound in PostgreSQL, the vacuum machinery that prevents it, what blocks that machinery, and how operators measure and schedule it. Not covered: table and index bloat as a performance problem except where it slows freezing, sequence and primary-key exhaustion (see the earlier guide on choosing the next ID), and managed-service internals that are not public.

02

One counter, one job, and everything that can stand in the job's way

The common shape across the engine's design and every operator in the corpus: a consumer that spends the counter, a horizon that limits what may be frozen, a throttled job that freezes, and, at the operators who have been bitten, a second layer that measures and schedules the job instead of trusting it.

Figure 2 · The wraparound control loop and its four blockers

Operator layer

Engine maintenance

age grows

advance relfrozenxid

pins xmin, nothing newer freezes

Horizon holders

Long or idle
transactions

Prepared
transactions

Replication slots
standby feedback

Index builds
ANALYZE snapshots

Write transactions
spend one XID each

Limits
vac, warn, stop

Autovacuum
launcher

Workers
cost-throttled

Freeze pages
set all-frozen bit

Age as share of limit
vacuum time vs horizon

Scheduled freeze
in low-load windows

Operator layer

Engine maintenance

age grows

advance relfrozenxid

pins xmin, nothing newer freezes

Horizon holders

Long or idle
transactions

Prepared
transactions

Replication slots
standby feedback

Index builds
ANALYZE snapshots

Write transactions
spend one XID each

Limits
vac, warn, stop

Autovacuum
launcher

Workers
cost-throttled

Freeze pages
set all-frozen bit

Age as share of limit
vacuum time vs horizon

Scheduled freeze
in low-load windows

Write traffic drains the budget continuously; only the freeze path refills it, and that path can be stalled by anything holding the xmin horizon. The operator layer exists at GitLab and is absent from the engine. Reconstructed from maintenance.sgml, GitLab runbooks 9f15781 and incident #4633.
Diagram source

The consumer. Every transaction that writes takes an XID; read-only transactions do not. That sounds like a property of the workload, and it is, but the record shows maintenance spending the budget too. In February 2021 GitLab installed cron jobs that ran ANALYZE by hand to work around stale planner statistics after a monthly bulk update [31]; two days later it opened a change to remove them because "Ongres recognized we are near emergency autovacuum to prevent wraparound". The fix for one symptom was eating the margin of another. The engine's own docs make the same point about recovery: do not reach for VACUUM FULL at the stop limit, "because it requires an XID" [6].

The horizon. Freezing can only certify rows older than the oldest snapshot anyone still holds. The documentation lists the holders in the order to remove them: old prepared transactions, long-running open transactions, and replication slots, including slots "created for replication to servers that no longer exist" [6]. GitLab's incidents add two the docs do not dwell on. A migration's CREATE INDEX ran for five and a half hours before anyone looked [21], and in June 2022 autovacuum's own ANALYZE, throttled by its cost budget and "not constrained by statement_timeout", held a snapshot for 39 minutes [34]. GitLab's repack runbook now states the consequence plainly: "A pinned xmin blocks autovacuum database-wide, not just on the target table" [27].

The job. Autovacuum is a launcher plus a fixed pool of workers, throttled by a cost model so it does not compete with production I/O. The pool size, autovacuum_max_workers, needs a restart to change, as does autovacuum_freeze_max_age itself, which is PGC_POSTMASTER [5]; the source comment explains that updating shared state from a configuration hook "doesn't look practical". Until March 2026 workers took tables in catalogue order; Nathan Bossart's commit adding a score says the list was processed "in the same order as the list", and that under the new scheme "a table nearing transaction ID wraparound should be vacuumed first" [15]. That change ships in version 19. Every production system in this corpus ran without it.

The operator layer. This is the component that appears only after an incident. GitLab's arrived in a burst. On 18 May 2021 an on-call engineer estimated the database was 12 hours from shutdown; it was later found to be 27 days out [21]. The corrective action was a metric. The merge request that delivered it, merged 24 May, reveals that the metric had existed all along: it was renamed because "the old metric was not consumed in any dashboards, alerts or recording rules", and its multixact half was switched to mxid_age "to correct the age calculation" [22]. Three days later the runbooks gained a severity-1 saturation point measuring age against 231 minus 3 million [23], and on 2 June a second one that is the most useful idea in this guide: total autovacuum time per worker over a day, divided by the time left until the wraparound horizon at the current XID rate [24]. It is a capacity ratio. It says whether the job could finish a full pass before the counter runs out, which an age chart cannot.

Freeze map

Since 9.6 the visibility map carries an all-frozen bit per page, so an anti-wraparound vacuum skips pages already frozen instead of rescanning the table. Before it, the 2016 commit says, wraparound meant "full-table scans" every time.

Source: a892234, 2016

Failsafe

From version 14, a vacuum that finds its table past 1.6 billion stops throttling and skips index vacuuming to freeze as fast as it can. Its authors expected it to fire "long after the autovacuum began".

Source: 1e55e7d, 2021

Capacity gauge

GitLab's ratio of vacuum time to time-until-horizon, with a soft SLO at 20% and a hard one at 33%. The only artefact in the corpus that treats freezing as throughput rather than as a state.

Source: GitLab runbooks, 2021

03

Six forks, and the condition that flips each

What the engine and its operators chose, what they rejected, and why. Where the reason lives on a host this session could not reach, the guide says so rather than inventing it.

The deepest decision is the width of the counter, and the repository records it as a long series of partial answers rather than one. A 32-bit XID per row header was the original choice. InnoDB, by contrast, stores a 6-byte transaction ID per row, DATA_TRX_ID_LEN = 6 [44], which at a sustained million transactions a second lasts about 8.9 years (248 / 106 seconds; an estimate at an illustrative rate). OrioleDB, a storage engine for PostgreSQL, uses 64-bit IDs and claims to eliminate "the well-known and painful wraparound problem", while saying it "is not recommended for production usage" [43]. PostgreSQL itself has widened around the problem: SLRU page numbers in November 2023, after "repeated bugs in the area of handling SLRU wraparound in the past, some of which have caused data loss" [17]; multixact member offsets in December 2025, while "Multixids are still limited to 2^31" [18]; and the OID counter in September 2026 [19]. At HEAD the row XID is still 32 bits. The argument for why the full 64-bit patch keeps returning from review is on the commitfest and mailing-list hosts, which were unreachable; this guide records the outcome and not the reasons.

Decision: raise the freeze threshold, or freeze more often?

Argued for (GitLab, 2022)
  • Keep autovacuum_freeze_max_age at 200M and move freezing into low-load windows first.
  • "If cleaning your house hurts and takes forever, do it more often, not less." The scheduler had to exist before any raise.
Deferred
  • A five-step ladder from 200M to 1B over eight weeks, proposed because GitLab.com hit the threshold in under three days.
  • A bigger threshold makes each forced vacuum rarer and larger, and the forced one does not yield.
Flips when
  • Most pages are already all-frozen, so a forced pass is cheap. Then the docs' advice wins: if commit-log storage is trivial, setting the threshold to its two-billion maximum "is recommended".

That fork is a real disagreement between two credible sources, and the difference in context explains it. The PostgreSQL documentation prices the threshold in storage: 200 million transactions "translates to about 50MB of pg_xact storage and about 2GB of pg_commit_ts storage" [6], so on a large database raising it costs almost nothing. GitLab's database engineer priced it in the size of the eventual mandatory vacuum on multi-terabyte, high-churn tables, and the incident record backs that pricing: the expensive event is not the threshold being crossed but the non-yielding vacuum that follows, at an hour nobody picked [26]. The rule: raise the threshold only on tables whose freeze debt you are already paying down some other way.

Decision: when the protective vacuum is in your way, cancel it or wait?

Chosen
  • Wait, and keep heavy DDL and repacks out of its path. GitLab's repack runbook: it "blocks pg_repack and doesn't yield. You can just wait".
  • Pre-freeze before planned events: GitLab ran VACUUM FREEZE 24 hours ahead of its PostgreSQL 12 upgrade.
Rejected
  • Terminating it. "Even if we terminate it it will restart in seconds, since it is a mechanism to keep the database integrity."
  • Skipping it in the engine: tried in 2019, reverted in 2020 (see below).
Flips when
  • The vacuum is the victim, not the blocker: a pg_repack drop step or an aborting index scan is what stalls it. Then remove the other party, which is what the RDS user in pg_repack #371 did with pg_cancel_backend.

The engine tried the third option once. In March 2019 a commit made autovacuum skip anti-wraparound jobs that had been marked non-aggressive, since they were "redundant" [10]. A year later it was reverted and back-patched to 12, because on clusters with many relations it "could cause autovacuum to lock down, with autovacuum workers attempting repeatedly those jobs on the same relations for the same database, that just kept being skipped" [11]. A skipped wraparound job does not go away; it comes back at the front of the queue. That is the same lesson GitLab's on-call learned by hand in 2021, written into the engine's history as a revert.

Decision: at the stop limit, single-user mode or an ordinary VACUUM?

Chosen (upstream, Oct 2023)
  • Stay in multi-user mode, remove the horizon holders, then VACUUM the oldest tables.
  • Single-user mode "disables transaction ID wraparound safeguards that are designed to prevent data loss".
Rejected
  • The old hint, which Robert Haas called "terrible advice", and which took the whole site down to fix a problem that only blocks writes.
Flips when
  • You must TRUNCATE or DROP a huge unneeded table rather than vacuum it; the docs keep the 3M margin for exactly that.

That change was made on master only. "No back-patch," the commit says, partly because people "might even have monitoring scripts that look for" the old message [9]. The cost of that caution is visible three years later. GitLab's wraparound alert runbook, last edited in July 2026, still tells its on-call that at wraparound "the database will automatically shutdown" and that "recovery would require entering single-user mode" [25]. Its first step is the right one (find and terminate very old transactions), and the misdescribed end state may never be reached; but an engineer who believes the database is about to shut down makes different choices at 03:00 than one who knows it will refuse writes and keep serving reads. Operational knowledge about this failure lags the engine by years, because the failure is rare enough that nobody rereads the runbook until it happens.

Figure 3 · What to do, by where the oldest unfrozen XID sits

no

yes

yes

no

no

yes

yes

no

age of datfrozenxid
above freeze_max_age?

Track vacuum time
against the horizon

Anything pinning xmin?
tx, prepared, slot, feedback

Remove the pin first
commit, terminate, drop slot

Is the vacuum advancing?
pg_stat_progress_vacuum

Find what aborts it
corrupt index, lock conflict

Done before
100M remain?

Let it run
hold DDL and repacks

Manual VACUUM, no cost delay
oldest tables first

no

yes

yes

no

no

yes

yes

no

age of datfrozenxid
above freeze_max_age?

Track vacuum time
against the horizon

Anything pinning xmin?
tx, prepared, slot, feedback

Remove the pin first
commit, terminate, drop slot

Is the vacuum advancing?
pg_stat_progress_vacuum

Find what aborts it
corrupt index, lock conflict

Done before
100M remain?

Let it run
hold DDL and repacks

Manual VACUUM, no cost delay
oldest tables first

Every leaf is an action. The order follows the documentation's recovery steps and GitLab's runbooks: clear the horizon before adding vacuum effort, because effort spent behind a pinned horizon freezes nothing. Sources: maintenance.sgml, GitLab alert runbook, GitLab #381443.
Diagram source
DecisionChosenRejectedBecauseFlips whenEvidence
Row XID width32 bits plus freezing; widen neighbouring counters piecemeal64-bit row XIDs (OrioleDB, out-of-tree patch); 48-bit as InnoDBReasoning on unreachable hosts; repository shows only outcomesBurn rate makes the freeze pass longer than the horizon even when nothing pins itbd8d9c9, OrioleDB
Freeze threshold200M, freeze on a schedule (argued, not yet decided)Ladder to 1B, deferredRarer forced vacuums are larger, and they do not yieldTables mostly all-frozen alreadyGitLab, 2022
Blocked by the protective vacuumWait; pre-freeze before planned workTerminate itIt restarts within secondsThe vacuum is the blocked partyGitLab #4215
Skip redundant wraparound jobsRevertedSkip (2019)Workers locked down retrying skipped relationsNo condition found in the recorddd9ac7d
Recovery at the stop limitMulti-user VACUUMSingle-user modeDisables the safeguards, takes the system downA table must be dropped or truncated2406c4e
What to alert onAge as share of limit, and vacuum time against the horizonAge alone, unwatchedAn age chart cannot say whether a pass can finishSmall databases where a full pass takes minutesgitlab-exporters !224
04

What broke: four ways to lose the race, and one way to not see it

Grouped by failure class. Only one incident in the corpus reached the stop limit. Every other one is the race going badly long before the cliff, which is where an architect can still do something about it.

The classes, as I am naming them (the sources use a different word each time): the pin, something holds the horizon so freezing cannot advance; the treadmill, a vacuum runs but never completes a pass; the guard that will not step aside, the mandatory vacuum colliding with a lock someone else needs; the pile-up, many tables reaching the threshold together; and the misread gauge, the number existing but nobody reading it correctly.

Figure 4 · The guard that will not step aside: a deploy migration meets an anti-wraparound vacuum

Deploy pipelineLock managerPost-deploymigrationAnti-wraparoundautovacuumDeploy pipelineLock managerPost-deploymigrationAnti-wraparoundautovacuumautovacuumflagged forwraparound, nocancel sentholds SHARE UPDATE EXCLUSIVE on deploymentsrequests a conflicting tablelockwaitdeadlock check afterdeadlock_timeoutlock_timeout, migration failsretry fails, deploysblockedvacuum ends, lock releasedmigration markedcomplete by hand,retry passes
Deploy pipelineLock managerPost-deploymigrationAnti-wraparoundautovacuumDeploy pipelineLock managerPost-deploymigrationAnti-wraparoundautovacuumautovacuumflagged forwraparound, nocancel sentholds SHARE UPDATE EXCLUSIVE on deploymentsrequests a conflicting tablelockwaitdeadlock check afterdeadlock_timeoutlock_timeout, migration failsretry fails, deploysblockedvacuum ends, lock releasedmigration markedcomplete by hand,retry passes
The migration needs a lock that conflicts with VACUUM's SHARE UPDATE EXCLUSIVE. The deadlock checker would cancel an ordinary autovacuum here and does not cancel this one. Reconstructed from GitLab #22642, proc.c and mvcc.sgml; the lock modes the migration requested are not stated in the incident and are inferred.
Diagram source
Postmortem

GitLab.com 2026: the guard blocks deploys for five hours

AssumptionA post-deploy migration on deployments can always get its lock within the timeout, or retry and succeed.
What happenedA wraparound-prevention vacuum on the table made the migration job fail from 03:17 UTC on 5 August; after the vacuum ended, two retries still failed on lock timeouts.
Blast radiusSeverity 3, labelled Deploys-blocked-gprd::5hr and the same for staging. No user-facing outage.
Fix"After that we marked the migration as complete." A retry then passed. No structural change is recorded in the description.
Design ruleAny pipeline that takes table locks on a schedule must check freeze age first, or it will meet the one vacuum that never yields.
Postmortem

pg_repack on RDS 2023: reads and writes blocked for half an hour

AssumptionA repack's temporary table is invisible to maintenance.
What happened"autovacuum: VACUUM ANALYZE repack.table_36175838 (to prevent wraparound)" ran on the temp table while repack's drop step needed a lock; "all reads and writes to the table were failing".
Blast radiusOver half an hour on PostgreSQL 14, Amazon RDS. Ended by pg_cancel_backend on the vacuum.
FixA maintainer pointed to a related 1.5.0 change that "might help"; closed in June 2025 as possibly fixed. The 2020 report of the same interaction (#247, "made the database inaccessible") is still open and unanswered.
Design ruleA tool that copies a table creates a new table with an old freeze age. Freeze or check age before the swap.
Postmortem

GitLab.com 2022: a weekend reindex starves the guard for 18 hours

AssumptionAutomated reindexing is background housekeeping that cannot hurt other housekeeping.
What happenedReindexing one ci_builds index "blocked wraparound-prevention autovacuum on gitlab.com ... for over 18 hours".
Blast radius"Significant dead tuple accumulation and degraded query performance"; a CI runner SLO breach (#7852).
FixAn issue to make reindexing avoid or recover; auto-closed in June 2025 with the label SLO::Missed.
Design ruleTwo maintenance jobs that need the same table need one scheduler. Rank freezing above every other kind.
Postmortem

Self-managed GitLab 2022: the treadmill, a vacuum that aborts before it freezes

AssumptionIf autovacuum is running on a table, that table's age is going down.
What happenedA corrupt index on ci_builds made each vacuum fail during index cleanup ("failed to re-find parent key in index"), before the heap was frozen. "We suspect this had been running in a loop since the corruption occurred, and that nothing is logged."
Blast radiusThe customer's database exhausted the ID space and went read-only: "the root cause of a customer emergency".
FixProposed: the application should watch the counter and warn admins at 80%, warn all users at 90% and switch itself read-only at 95%. Still open in October 2026.
Design ruleAlert on age not falling, not on vacuum not running. A vacuum that restarts forever looks healthy on every activity chart.
Postmortem

GitLab.com 2021: the pin, an 11-hour migration and a wrong estimate

AssumptionA migration that passed its test pipeline will finish in a normal deploy window.
What happenedA CREATE began at 13:30 and was found at 18:33 after dead-tuple alerts; it was halted at 00:36 the next day. On-call estimated 12 hours to shutdown and "a maximum of 15 minutes warning".
Blast radiusSeverity 4 near miss. The estimate was later struck through: wraparound "was still 27 days out".
Fix"Wraparound transaction monitoring and alerting", delivered six days later as gitlab-exporters !224.
Design ruleDistance to the limit has to be a computed number on a dashboard before the incident, because nobody computes it correctly during one.
Postmortem

GitLab.com 2021: the pile-up, several guards at once under 90% load

AssumptionMandatory vacuums arrive one at a time.
What happened"The database was under a load over 90%. Several auto vacuums to prevent wraparound were in place. Reindex activity in place." The primary shut down and Patroni did not fail over.
Blast radiusRecorded only as a corrective-action investigation; impact figures are in comments this session could not read.
FixThe investigation issue; in May 2021, scheduled VACUUM FREEZE of candidate tables a day before the PG12 upgrade (#4215).
Design ruleTables created or bulk-loaded together share a freeze age and will demand their mandatory vacuum together. Stagger them on purpose.

The pile-up has an engine-side confirmation. The 2020 commit that added insert-triggered autovacuum says that before it, insert-only tables could receive their first vacuum ever as an anti-wraparound vacuum, and that "new clusters, or recently pg_restored clusters" could have many large tables with "the same relfrozenxid, which could result in large numbers of tables requiring an anti-wraparound vacuum all at once" [12]. A restore or a migration resets every table's age to the same value. At GitLab's burn rate, three days of quiet later they all come due together.

The treadmill shows up again in 2026, at scale and without corruption. GitLab's database team, gathering signals from self-managed customers, reports "multi-day autovacuum runs, including the dreaded autovacuum: VACUUM <table> (to prevent wraparound) holding a lock for 5+ days", and one case where a customer's ANALYZE "was blocked for 2 days behind a wraparound autovacuum that had already been running for 5" [35]. The cause it names is default throttling: cost delays of 10 to 20 ms, three workers, small maintenance_work_mem. "Customers only discover the issue after it has caused an incident." Compare GitLab.com's own settings: six workers, a 5 ms delay and a cost limit of 6,000 [26]. The same application, with a cost budget of 1,200 units per millisecond of delay against 10 to 20 on the customers' settings (6,000 / 5 against 200 / 10 or 200 / 20; an estimate of relative throughput, ignoring worker count): roughly two orders of magnitude apart in how fast its janitor is allowed to work.

Postmortem

GitLab.com 2021: a workaround spends the margin

AssumptionExtra ANALYZE runs are harmless.
What happenedCron jobs added during a February 2021 incident ran analyzes on a schedule; the team removed them because the database was "near emergency autovacuum to prevent wraparound".
Blast radiusNear miss; a C2 change to delete the crons.
FixCrons removed; the root cause, a monthly bulk update of nearly every row, was redesigned to reset lazily.
Design ruleEvery scheduled job that writes or snapshots spends from the same budget as production. Inventory them.
Postmortem

GitLab.com 2022: autovacuum's own ANALYZE pins the horizon

AssumptionAutovacuum cannot get in its own way.
What happenedA throttled autovacuum ANALYZE held its snapshot for 39 minutes; no other vacuum could remove anything newer meanwhile.
Blast radiusNo customer impact this time; the incident notes it "has in the past and can again" caused CPU starvation on replicas.
FixThrottle autovacuum less so its analyzes finish sooner.
Design ruleThrottling the janitor lengthens every snapshot it holds. A gentle vacuum is not a free one.
The famous three, not read

The public index of postmortems summarises Sentry's 2015 outage as down "for most of a working day", Mandrill's 2019 one as "a partial outage lasting a day and a half", and Joyent's 2015 Manta outage as "PostgreSQL's transaction wraparound maintenance taking a lock on something, and a Joyent query that unnecessarily tried to take a global lock" [42]. The last of those is the guard-at-the-lock-queue class again, nine years before GitLab's. Their full write-ups were unreachable from this session; read them before relying on any detail beyond these lines.

The misread gauge is the class that connects the others. GitLab's metric existed but was "not consumed in any dashboards, alerts or recording rules", and computed the multixact age with the wrong function [22]. pganalyze's collector, a commercial monitoring agent, matched autovacuum log lines in a way that missed "anti-wraparound autovacuum, without aggressive" until a fix merged on 23 September 2026 [40]. Joyent's monitoring agent has an open issue asking that these vacuums be labelled "wraparound" rather than "aggressive" [39], and Postgres.ai's guidance notes multixact wraparound is "usually missing in observability tools" [41]. Even the engine conceded the point in March 2026, adding a percentage to its warnings because "this more clearly expresses the urgency of the situation" [16]. Four independent parties, one lesson: the number is easy to collect and hard to read.

05

Numbers you can plan against

The engine's limits are measured from source at a stated branch. Operator figures are reported by the operator. Derived rows show their arithmetic and are labelled estimates.

Figure 5 · Ten years of the engine moving the furniture around the cliff

2016All-frozen bit letswraparoundvacuums skip frozenpages2019Skip redundantanti-wraparoundvacuums2020Skip reverted afterautovacuumlock-downInsert-triggeredautovacuumWarn at 40M, stop at3M2021Failsafe at 1.6 billion2023Single-user advicewithdrawnSLRUs indexed by64-bit integers2025Freeze more ininsert-triggeredvacuumsMultiXactOffsetwidened to 64 bits2026Warn at 100M with apercentageWraparound-firsttable priorityOID counter widenedto 8 bytesPostgreSQL commits on the wraparound path
2016All-frozen bit letswraparoundvacuums skip frozenpages2019Skip redundantanti-wraparoundvacuums2020Skip reverted afterautovacuumlock-downInsert-triggeredautovacuumWarn at 40M, stop at3M2021Failsafe at 1.6 billion2023Single-user advicewithdrawnSLRUs indexed by64-bit integers2025Freeze more ininsert-triggeredvacuumsMultiXactOffsetwidened to 64 bits2026Warn at 100M with apercentageWraparound-firsttable priorityOID counter widenedto 8 bytesPostgreSQL commits on the wraparound path
None of these changes widened the row XID. Each one either made the pass cheaper, made the warning louder, or moved freezing earlier, which is the operator's playbook applied by the engine itself. Commits in the PostgreSQL repository; dates are commit dates.
Diagram source
MetricValueAtContextAs ofSource
Usable XID distance before the hard limit231 ≈ 2.147BPostgreSQLFrom the oldest unfrozen XID in the clusterHEAD 2026-10varsup.c
Stop margin1M → 3MPostgreSQL12 and 13 stop at 1M; 14 onward at 3M, because of "edge-case bugs" near the limit2020-08cd5e822
Warning margin11M → 40M → 100MPostgreSQL12: 10M before the stop; 14: 40M; development branch: 100M2026-0348f11bf
Forced-vacuum threshold200MPostgreSQLDefault; restart needed to change globally; per-table reloption allowedHEAD 2026-10guc_parameters.dat
Failsafe threshold1.6BPostgreSQLVersion 14 onward; drops cost delay and index vacuuming2021-041e55e7d
Storage cost of the 200M horizon~50 MB / ~2 GBPostgreSQLpg_xact and pg_commit_ts respectivelyHEAD 2026-10maintenance.sgml
Time to burn 200M XIDs< 3 daysGitLab.comMain database, on average2022-01GitLab runbook
Implied burn rate and full headroom> 67M/day; ~32 daysGitLab.comEstimate: 200M / 3 days; (231 − 3M) / 67M per day, if freezing stopped entirely2022-01derived
Database age when monitoring was fixed199,895,030GitLab.comage(datfrozenxid) on the primary, against a 200M threshold2021-05-21!224
Wraparound saturation SLOs60% / 70%GitLab.comSoft and hard, as share of 231 − 3M; severity 12021-054ef250e
Vacuum-capacity SLOs20% / 33%GitLab.comDaily vacuum seconds per worker over seconds to horizon2021-069f15781
Autovacuum settings6 workers, 5 ms, 6000GitLab.comWorkers, cost delay, cost limit2022GitLab runbook
Anti-wraparound vacuum duration on defaults5+ daysSelf-managed GitLabCustomer signals; one ANALYZE blocked 2 days2026-05#600630
Reindex blocking the guard> 18 hGitLab.comci_builds, weekend automatic reindex2022-10#377720
Deploys blocked by the guard5 hGitLab.comProduction and staging deploy label2026-08#22642
Reads and writes blocked> 30 minRDS userpg_repack temp table under anti-wraparound vacuum2023-11pg_repack #371
Practitioner danger line> 1BPostgres.ai"Values above 1B have to be considered dangerous"undatedhowto 0044
InnoDB per-row transaction ID48 bits; ~8.9 years at 1M tx/sMySQL 8.4Estimate for the duration: 248 / 106 s, illustrative rate8.4 branchdata0type.h

What to plan against. Two quantities decide whether you are safe, and the engine reports neither directly. The first is your burn rate: XIDs per day, read from txid_current() deltas on the primary. Divide the 2.14 billion usable IDs by it and you have your total runway from a fully frozen state; GitLab.com's was about a month. The second is your full-pass time: how long a complete freeze of your largest, least-frozen tables takes at your actual cost settings. GitLab's ratio of the two, with a hard SLO at one third, is the closest thing in the public record to a sizing rule, and it is a sensible one. A pass that takes a third of your runway leaves two thirds for everything in the failure section to go wrong. Nobody has published a measured full-pass time for a named multi-terabyte table, which is the number you most need and will have to measure yourself.

Read these carefully

The version-specific limits matter: a runbook written against 12 or 13 has a 1M stop margin and a warning only 10M before it, and on those versions the single-user advice was still in the server's own hint text. The 100M warning and the table prioritisation are on the development branch for version 19; production clusters in October 2026 run without them. GitLab's three-day figure is from January 2022 and its workload has grown since; treat it as an order of magnitude for a large SaaS, not a benchmark.

06

The evidence wall

Graded by what each artefact can prove. The corpus is three hosts deep (GitHub, GitHub raw content, GitLab) because those were the only hosts this session could reach; there are no papers or talks, and the famous blog postmortems are cited only through an index.

PostmortemGitLab2026-08

Wraparound prevention vacuum in Postgres blocking PDMs

Post-deploy migrations failed repeatedly from 03:17 UTC while an anti-wraparound vacuum ran on deployments; deploys blocked about five hours.

Carry forwardGate scheduled DDL on freeze age, not just on load.
gitlab.com/gitlab-com/gl-infra/production/-/issues/22642
PostmortemGitLab (customer)2022-11

Detect and react to imminent PostgreSQL transaction ID exhaustion

A self-managed database went read-only after a corrupt index made every vacuum of ci_builds abort before freezing, silently.

Carry forwardAlert on age failing to fall, not on vacuum activity.
gitlab.com/gitlab-org/gitlab/-/issues/381443
PostmortemGitLab2021-05

Long-running CREATE INDEX blocking vacuum

An 11-hour migration held the horizon; on-call estimated 12 hours to shutdown, later corrected to 27 days. Monitoring followed within a week.

Carry forwardCompute distance to the limit before you need it.
gitlab.com/gitlab-com/gl-infra/production/-/issues/4633
PostmortemGitLab2022-10

Automatic reindexing can block vacuum progress

A weekend reindex held a lock that blocked the wraparound vacuum on ci_builds for over 18 hours.

Carry forwardOne scheduler for all maintenance on a table, freezing first.
gitlab.com/gitlab-org/gitlab/-/issues/377720
PostmortemGitLab2021-03

Investigate the PG shutdown and no failover

The primary shut down at over 90% load with several anti-wraparound vacuums and a reindex in flight.

Carry forwardStagger freeze ages; mandatory vacuums cluster.
gitlab.com/gitlab-com/gl-infra/production/-/issues/3967
PostmortemGitLab2021-02

Remove cron jobs created during incident

Manual ANALYZE crons installed as an incident workaround were removed because the database was "near emergency autovacuum to prevent wraparound".

Carry forwardWorkarounds spend the same budget as the workload.
gitlab.com/gitlab-com/gl-infra/production/-/issues/3498
PostmortemGitLab2022-06

Long-lived transaction in primary db

Autovacuum's own throttled ANALYZE held a snapshot for 39 minutes, preventing cleanup everywhere.

Carry forwardThrottling the janitor lengthens its snapshots.
gitlab.com/gitlab-com/gl-infra/production/-/issues/7297
Postmortemreorg / pg_repack2023-11

All target table reads & writes blocked because of autovacuum on repack temporary table

On RDS PostgreSQL 14, an anti-wraparound vacuum on a repack temp table and repack's drop step together blocked a table for over half an hour.

Carry forwardCopied tables arrive with old freeze ages.
github.com/reorg/pg_repack/issues/371
Postmortemreorg / pg_repack2020-09

Wraparound vacuum prevention sanity check

The same interaction three years earlier: "The combination of the two made the database inaccessible." Unanswered and open.

Carry forwardA tool's silence on a known hazard is not a fix.
github.com/reorg/pg_repack/issues/247
SourcePostgreSQL2026-10

varsup.c, SetTransactionIdLimit

Where the hard, warning and stop limits are computed, with the comment refusing to make the warning configurable.

Carry forwardThe margins are fixed; your only lever is the freeze rate.
raw.githubusercontent.com/postgres/postgres/.../varsup.c
SourcePostgreSQL2026-10

proc.c, autovacuum cancellation on lock wait

A blocked backend cancels an autovacuum only if it is not flagged PROC_VACUUM_FOR_WRAPAROUND.

Carry forwardThe mandatory vacuum is the one that never yields.
raw.githubusercontent.com/postgres/postgres/.../proc.c
SourcePostgreSQL2021-04

Add wraparound failsafe to VACUUM

Past 1.6 billion, vacuum drops its cost delay and skips index work; designed to fire late inside an already-running wraparound vacuum.

Carry forwardThe engine's emergency brake is speed, not cancellation.
github.com/postgres/postgres/commit/1e55e7d
SourcePostgreSQL2019-03 / 2020-03

Skip redundant anti-wraparound vacuums, and its revert

A skip optimisation locked autovacuum into retrying the same relations; reverted and back-patched to 12.

Carry forwardDeferred wraparound work returns at the head of the queue.
github.com/postgres/postgres/commit/dd9ac7d
SourcePostgreSQL2025-03

Trigger more frequent autovacuums with relallfrozen

Moves freezing into insert-triggered vacuums because "anti-wraparound vacuums are not automatically canceled by conflicting lock requests".

Carry forwardFreeze early in cancellable work.
github.com/postgres/postgres/commit/06eae9e
SourcePostgreSQL2020-03

Trigger autovacuum based on number of INSERTs

Insert-only tables used to get their first vacuum as a wraparound vacuum, many at once after a restore.

Carry forwardRestores and bulk loads synchronise freeze ages.
github.com/postgres/postgres/commit/b07642d
SourcePostgreSQL2026-03

Add rudimentary table prioritization to autovacuum

Before version 19, tables near wraparound waited their turn in catalogue order.

Carry forwardOn current releases, prioritise by hand.
github.com/postgres/postgres/commit/d7965d6
SourcePostgreSQL2023-11 / 2025-12

Index SLRUs by 64-bit integers; Widen MultiXactOffset to 64 bits

Wraparound bugs "have caused data loss"; the widening proceeds one counter at a time, and row XIDs stay 32-bit.

Carry forwardPlan for 32-bit XIDs for the life of your cluster.
github.com/postgres/postgres/commit/bd8d9c9
SourcePostgreSQL2020-08 / 2026-03

Warning and stop margins: 40M and 3M, then 100M

The stop margin grew because of "edge-case bugs" near the limit; the warning grew for workloads that burn IDs fast.

Carry forwardKnow which margins your version has.
github.com/postgres/postgres/commit/cd5e822
SourceGitLab2021-06

runbooks: monitor total vacuum capacity

Vacuum seconds per worker per day divided by seconds until the wraparound horizon; soft 20%, hard 33%.

Carry forwardSize the janitor as throughput against a deadline.
gitlab.com/gitlab-com/runbooks/-/commit/9f15781
SourceGitLab2021-05

runbooks: pg txid wraparound saturation monitoring

Wraparound as a severity-1 saturation point against 231 minus 3M, SLOs at 60% and 70%.

Carry forwardExpress age as a share of the real limit.
gitlab.com/gitlab-com/runbooks/-/commit/4ef250e
Sourcepganalyze2026-09

Match with non-aggressive anti-wraparound vacuum with log (#879)

A monitoring agent's log parser missed one class of anti-wraparound vacuum until September 2026.

Carry forwardTest that your tooling sees every vacuum variant.
github.com/pganalyze/collector/pull/879
SourceJoyent / Triton2019-07

pgstatsmon #28: "aggressive" vacuum mode should be something like "wraparound"

Manta's operator asks its monitor to name wraparound vacuums distinctly; the change log records splitting them out.

Carry forwardLabel the disruptive vacuum by its cause.
github.com/TritonDataCenter/pgstatsmon/issues/28
Decision recordPostgreSQL2023-10

Reword messages about impending (M)XID exhaustion

Withdraws single-user-mode advice as "terrible", corrects "shut down" to "refuses commands that assign" IDs; master only.

Carry forwardRecover in multi-user mode, safeguards on.
github.com/postgres/postgres/commit/2406c4e
Decision recordGitLab2021-05 / 2026-07

runbooks: pg_xid_wraparound saturation alert

Right first step (terminate old transactions); still predicts shutdown and single-user recovery three years after upstream withdrew both.

Carry forwardRe-read the wraparound runbook against your version's docs.
gitlab.com/gitlab-com/runbooks/-/blob/.../pg_xid_wraparound_alert.md
Decision recordGitLab2026-10

runbooks: pg_repack checklist

Repacks are gated on freeze age because the forced vacuum "cannot be deferred or skipped" and a pinned xmin would block it.

Carry forwardMake freeze age a precondition of heavy maintenance.
gitlab.com/gitlab-com/runbooks/-/blob/.../pg_repack.md
Decision recordGitLab2026-06

Database State: Transaction-ID wraparound & freeze-age monitoring

Proposal to ship freeze-age monitoring in the product's admin pages, MultiXact included, with escalation above about 1.5 billion.

Carry forwardIf customers run your database, ship the gauge with the app.
gitlab.com/gitlab-org/gitlab/-/issues/602137
Case studyGitLab2021-04

Execute VACUUM FREEZE for multiple tables

A six-hour scheduled freeze 24 hours before the PG12 upgrade, because a terminated wraparound vacuum "will restart in seconds".

Carry forwardPay freeze debt before planned events.
gitlab.com/gitlab-com/gl-infra/production/-/issues/4215
Case studyGitLab2020-05

Per-table autovacuum_freeze_max_age before a PG upgrade

Tables over 95% of the threshold given a 500M per-table limit to keep forced vacuums out of the upgrade, then reset.

Carry forwardPer-table reloptions are the no-restart lever.
gitlab.com/gitlab-com/gl-infra/production/-/issues/2095
Case studyGitLab2026-05

Unoptimized auto-vacuum configuration on self-managed PostgreSQL

Default throttling yields wraparound vacuums lasting 5+ days; the team's tuning playbook is not in customer docs.

Carry forwardDefaults size the janitor for a small database.
gitlab.com/gitlab-org/gitlab/-/issues/600630
BlogPostgres.aiundated

How to monitor transaction ID wraparound risks

Thresholds from practice: alert from roughly 200M to 1B, treat above 1B as dangerous; multixact often unmonitored.

Carry forwardMonitor mxid_age alongside age.
github.com/postgres-ai/postgres-howtos/.../0044...md
IndexDan Luuliving

post-mortems (Sentry 2015, Joyent 2015, Mandrill 2019 entries)

One-line summaries of the three best-known wraparound outages; their source pages were unreachable here.

Carry forwardJoyent's was the lock-queue class, in 2015.
github.com/danluu/post-mortems
VendorOrioleDB2026

OrioleDB README

64-bit transaction IDs and no dedicated VACUUM, claimed; "not recommended for production usage".

Carry forwardThe structural fix exists, not yet for production.
github.com/orioledb/orioledb
07

Build a miniature, then productionise it

Six rungs on a disposable PostgreSQL container. The first three make the mechanism visible; the last three are the operator layer GitLab built after its incidents.

Watch the counter move

Start PostgreSQL with autovacuum_freeze_max_age at its minimum, run pgbench, and chart age(datfrozenxid) and per-table age(relfrozenxid) every ten seconds.

Done when: pg_stat_activity shows a worker whose query ends in "(to prevent wraparound)" and the age drops after it.  Teaches: the sawtooth, and that read-only queries do not move it.

Pin the horizon

In a second session open a REPEATABLE READ transaction and leave it idle. Keep pgbench running.

Done when: the wraparound vacuum runs to completion and the age does not fall.  Teaches: effort behind a pinned xmin freezes nothing; remove the pin before adding workers.

Meet the guard at the lock queue

Load a table large enough that its vacuum takes minutes. Run ALTER TABLE ... ADD COLUMN with lock_timeout against an ordinary autovacuum, then against an anti-wraparound one.

Done when: the log shows "canceling autovacuum task" in the first case and your ALTER timing out in the second.  Teaches: the proc.c rule, and why deploy pipelines need a freeze-age gate.

Reach the cliff and come back

Build the xid_wraparound test module from the PostgreSQL tree (its 2023 commit calls the helpers "handy for manually testing XID wraparound") and use it to consume IDs to the warning and stop limits on a throwaway cluster.

Done when: you recover from the stop limit in multi-user mode, in the docs' order, without single-user mode.  Teaches: what the error actually blocks, and that reads keep working.

Build the two gauges

Export age as a share of 231 − 3M (XID and multixact), and GitLab's ratio of daily vacuum seconds per worker to seconds until the horizon at the measured burn rate. Alert on the ratio, and on age that has not fallen for a day.

Done when: an injected pin from rung 2 fires the "not falling" alert well before the age alert.  Teaches: capacity, not state, is the signal.

Take the schedule back from the engine

Write a job that freezes the largest tables over about 80% of the threshold in a low-load window, oldest first, and a pre-check that refuses DDL, repack and reindex jobs on tables within range.

Done when: a week of production-like load produces no anti-wraparound vacuum on any table above your size cut-off.  Teaches: the forced vacuum is a missed appointment; schedule it and it never comes.

08

Keep hunting

The queries that produced this corpus. GitLab's API serves public issue descriptions without authentication, and the PostgreSQL history is searchable from a blobless clone.

Operator incident trackers

  • curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?search=wraparound&per_page=60"
  • curl "https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/issues?search=freeze_max_age"
  • "to prevent wraparound" site:github.com issues

The engine's own argument

  • git log --date=short -i --grep=wraparound --format="%h %ad %an %s"
  • git log --date=short -i --grep=failsafe --format="%h %ad %s"
  • git log -S"xidWrapLimit - 3000000" -- src/backend/access/transam/varsup.c

Runbooks and monitoring

  • git clone --depth 1 https://gitlab.com/gitlab-com/runbooks.git && grep -ril wraparound runbooks/docs
  • "mxid_age" "datfrozenxid" exporter

Collisions with tools

  • "wraparound" pg_repack OR pg_partman OR reindex is:issue
  • "PROC_VACUUM_FOR_WRAPAROUND" lock cancel
09

References

Numbering follows the evidence ledger in sources.md; ledger rows that support no claim on this page are omitted. All checked 2026-10-16.

  1. PostgreSQL, src/backend/access/transam/varsup.c HEAD af09050, October 2026.
  2. PostgreSQL, varsup.c on REL_14_STABLE Compared with REL_12_STABLE.
  3. PostgreSQL, src/backend/storage/lmgr/proc.c HEAD af09050.
  4. PostgreSQL, guc_parameters.dat HEAD af09050.
  5. PostgreSQL documentation source, maintenance.sgml HEAD af09050.
  6. PostgreSQL documentation source, mvcc.sgml Lock modes; with maintenance.sgml for storage cost.
  7. Sawada and Geoghegan, Add wraparound failsafe to VACUUM 2021-04-07.
  8. Haas et al., Reword messages about impending (M)XID exhaustion 2023-10-17.
  9. Horiguchi and Paquier, Skip redundant anti-wraparound vacuums 2019-03-31.
  10. Paquier, Revert "Skip redundant anti-wraparound vacuums" 2020-03-31.
  11. Albe and Praliaskouski, Trigger autovacuum based on number of INSERTs 2020-03-28.
  12. Plageman, Trigger more frequent autovacuums with relallfrozen 2025-03-03.
  13. Sawada and Haas, Change the format of the VM fork to add a second bit per page 2016-03-01.
  14. Bossart, Add rudimentary table prioritization to autovacuum 2026-03-27.
  15. Bossart, Add percentage of available IDs to wraparound warnings 2026-03-20.
  16. Orlov et al., Index SLRUs by 64-bit integers rather than by 32-bit integers 2023-11-29.
  17. Orlov and Linnakangas, Widen MultiXactOffset to 64 bits 2025-12-09.
  18. Paquier, Widen Object ID counter to 8 bytes 2026-09-10.
  19. GitLab, 2026-08-05: Wraparound prevention vacuum in Postgres blocking PDMs Production tracker.
  20. GitLab, 2021-05-18: Long-running CREATE INDEX blocking vacuum Production tracker.
  21. GitLab, feat(postgres): add TXID wraparound monitoring Merged 2021-05-24.
  22. GitLab runbooks, pg txid wraparound resource saturation monitoring 2021-05-27.
  23. GitLab runbooks, monitor total vacuum capacity 2021-06-02.
  24. GitLab runbooks, pg_xid_wraparound Saturation Alert At HEAD 844a2b8, October 2026.
  25. GitLab runbooks, PostgreSQL VACUUM Started 2022-01-25.
  26. GitLab runbooks, pg_repack Last edited 2026-10-12.
  27. GitLab, Detect and react to imminent PostgreSQL transaction ID exhaustion 2022-11-03, open.
  28. GitLab, Automatic reindexing can block vacuum progress 2022-10-12.
  29. GitLab, Investigate the PG shutdown and no failover 2021-03-15.
  30. GitLab, Remove cron jobs created during incident 2021-02-03.
  31. GitLab, Set tables with increased autovacuum_freeze_max_age temporarily 2020-05-08.
  32. GitLab, Execute VACUUM FREEZE for multiple tables 2021-04-13.
  33. GitLab, 2022-06-21: Long-lived transaction in primary db Production tracker.
  34. GitLab, Unoptimized auto-vacuum configuration on self-managed PostgreSQL 2026-05-20.
  35. GitLab, Database State: Transaction-ID wraparound & freeze-age monitoring 2026-06-04.
  36. pg_repack #371, All target table reads & writes blocked 2023-11-09.
  37. pg_repack #247, Wraparound vacuum prevention sanity check 2020-09-24.
  38. TritonDataCenter pgstatsmon #28 2019-07-31.
  39. pganalyze collector #879 Merged 2026-09-23.
  40. Postgres.ai, How to monitor transaction ID wraparound risks Undated.
  41. Dan Luu, post-mortems Index entries for Sentry, Joyent, Mandrill.
  42. OrioleDB, README 2026.
  43. MySQL, storage/innobase/include/data0type.h Branch 8.4.
  44. Misch, Change XID and mxact limits to warn at 40M and stop at 3M 2020-08-01.
  45. Bossart, Bump transaction/multixact ID warning limits to 100M 2026-03-20.