Ten years of GitLab.com read out of its own public incident tracker, design documents and rejected merge requests. One company, two components that resisted horizontal scaling, and every architectural move since 2016 explainable as a response to one of them.
A product grows at a fixed multiple every year. Underneath it sit two components that cannot be split without rewriting the application. What do you do for the next ten years?
That is the problem this guide is about, and GitLab is an unusually good place to watch it
play out, because GitLab runs its own product in public. The incident tracker for GitLab.com
is a public issue board whose first entries date from April 2016. The architecture design
documents, including the ones marked rejected, are markdown files in a public
repository. The service topology is a directory of configuration files anyone can list. A
solution architect can therefore do something here that is normally impossible: follow a
single company's stateful tier across a decade and check every claim against the artefact
the engineers wrote at the time, rather than the retrospective they wrote afterwards.
The two components are a PostgreSQL cluster and a filesystem holding Git repositories.
Neither could be sharded without changing the application above it, and GitLab's own capacity
process says so in as many words: resources are classified by whether they scale
horizontally, and non-horizontally scalable resources (such as CPU on the primary
PostgreSQL instance, for example) require much longer-term strategies for remediation and
are therefore considered higher priorities in the capacity planning
process
.32 Everything in the decade that follows is a
long-term strategy for one of those two resources.
The most consequential number in GitLab's decade is not a latency or a request rate. It
is 100 GB, a target for the on-disk size of a single physical table, enforced by static
analysis that refuses new columns on tables above it. GitLab arrived at that rule after
discovering that the size of one table had become the limit on how often the company could
deploy: building a single index on ci_builds took between 1.5 and 6 hours,
and this process blocks deployments as migrations are being run
synchronously
.13 The database stopped being a latency
problem and became a release-engineering problem, and that is the transition most capacity
models never predict.
ci_builds table, 22.7% of the entire databaseWhat this guide covers. GitLab.com's server-side state and topology between 2016 and September 2026: the PostgreSQL estate, Git storage, the key-value tier, background jobs, search, the routing plane and the incidents that shaped them. What it does not cover. The product itself, the Duo model stack beyond where it appears in incidents, security and compliance architecture, the self-managed and Dedicated packaging paths, and the money. No infrastructure spend figure appears anywhere in this guide because none was found in a source that could be opened. GitLab's cost-management handbook page describes a process and links to dashboards that are not public.
Reconstructed from the service catalogue, the controlled production architecture document and the Cells design, all of which are public files rather than diagrams drawn for a conference.
GitLab.com runs in Google Cloud's us-east1 region on four Kubernetes clusters,
and the interesting sentence in the controlled architecture document is the exception clause:
Most of GitLab.com is deployed on Kubernetes using GitLab cloud native helm chart. There
are a few exceptions for this which are mainly the datastore services like
.33 Ten years of platform
modernisation moved the stateless tier into containers and left the stateful tier exactly
where it was. That split is the single most reliable pattern in this corpus, and it is worth
noticing before anything else: the components that took a decade of architectural attention
are precisely the ones that never became cattle.PostgresSQL, Gitaly, Redis,
Elasticsearch
The topology can be counted rather than estimated. The metrics catalogue that drives
GitLab.com's monitoring holds 117 service definitions.27
Fourteen of them have names beginning redis, covering cache, sessions, shared
state, rate limiting, feature flags, the Action Cable pub/sub path, the container registry,
a repository cache and the Sidekiq queues; five more are Memorystore instances including a
quarantine instance for misbehaving job classes. Four are Patroni-managed PostgreSQL
clusters (main, ci, sec, registry), each
with its own pgbouncer fleet. The single Redis and single PostgreSQL of 2016 have become
nineteen key-value services and four database clusters, and each split was paid for
separately.
Requests are classified and routed to a cell by a Cloudflare Worker before they reach any GitLab code, and the authoritative map of which organisation lives on which cell is a Topology Service backed by Cloud Spanner. Both are new surfaces, in a different cloud from the application, introduced because routing had to exist somewhere that no single cell owns.
Source: Cells ADR 001, service catalogue
Sidekiq jobs are routed to a specific Redis by application-level rules rather than by
Redis Cluster, because the key distribution defeats the cluster's hashing:
Redis Cluster is not suitable for Sidekiq since there are only a small subset of hot
keys
. The recommended shape is one queue per shard, with a Kubernetes deployment
sized per shard.
Source: Sidekiq sharding runbook
Siphon reads the PostgreSQL logical replication stream, serialises change events onto a
queue and lets consumers load them into ClickHouse or Snowflake. Its stated goal is
zero or minimal impact on the availability of the PostgreSQL cluster
, which is the
same instinct as the table size target expressed in a different layer: take work off the
component that cannot be split.
Where the reconstruction is thinner: node counts, instance sizes and per-service capacity are not public. The metrics catalogue names every service and its service-level indicators but not its fleet size, and the forecasting outputs that would show saturation headroom are classified confidential under GitLab's own data standard.32 An architect can therefore learn the shape of this system in detail and cannot learn what it costs to run, which is an unusual combination and worth keeping in mind when this corpus is used as a benchmark.
Chosen, rejected, the stated reason, and the condition under which the rejected option becomes the right one.
GitLab keeps its rejected designs. The Cells directory contains a folder named
rejected holding two complete router proposals, and one of them explains the
practice in its own header: This documentation will be kept even if we decide not to
implement this so that we can document the reasons for not choosing this
approach
.21 That habit is why this section can be written
at all, and it is the cheapest thing in this guide for another organisation to copy.
main, ci and
sec run as separate clusters on GitLab.comgitlab_schema
classification applied to every tablewe cannot use PostgreSQL schema due to complex migration procedures
up to 1000 lines of code
more vendor locking with Cloudflare, but we are already heavily dependent on them
apart from the self-managed, which is a low priority requirement. If that requirement were high priority, this decision inverts.
The fourth path on that tree is where GitLab's record is most instructive, because GitLab
took the replication branch twice and abandoned it twice. Gitaly exists because Git on
network storage was unusable: the original design document records a P99 of over 30 seconds
to open a repository object against 15 milliseconds of CPU, and names the filesystem as the
culprit.23 Moving to bare metal was considered and refused on
the grounds that customers run in the cloud. Praefect then added replication and read
distribution on top of Gitaly. A transaction manager, write-ahead log and Raft service came
after that. In 2026 both are gone. The Scaling Git proposal states the verdict on the first:
while we tried to address this issue with read distribution via Praefect, that effort has
basically failed to yield a horizontally scalable cluster due to various
issues
.26 The transaction management design document now
carries the status rejected and an unusually candid note: the design was
largely built and is now being unwound, rather than turned down at
review
.24 The removal epic sizes the unwinding at a merge
request of 365 files, too large to review as one MR
.25
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Queue pending CI builds | A separate, purpose-built table | Redis-backed queuing | Queue building scanned the largest table in the estate; the team evaluated both and shipped the table in October 2021 | CI/CD Scaling, 2021 |
| Shard the job queues | Application-level routing to one Redis per queue | Redis Cluster | There are only a small subset of hot keys, so cluster hashing does not spread the load | Sidekiq sharding runbook |
| Code search | Zoekt alongside Elasticsearch | Extending Elasticsearch to cover code | Code search needs exact and regex matching; Elasticsearch stays for issues, merge requests and comments | Zoekt design document, 2022 |
| Analytics load | Change data capture off the replication stream into ClickHouse | Querying the OLTP replicas | Goal stated as zero or minimal impact on PostgreSQL availability | Siphon, 2024 |
| Cell granularity | Organizations as the logical boundary, Cells as the physical one | Subdomains per tenant | Cookie leakage across subdomains, integration breakage, and naming collisions with higher impact | Cells design document |
| Disaster recovery for cells | Backup and restore | Geo replication (ADR 006, superseded by ADR 024) | The decision log records both, in order, with the later one replacing the earlier | Cells decision log |
One more decision deserves separate mention because it is about process rather than
technology. The Cells programme is on its fourth design. The current document opens by
saying so: Protocells is the current iteration of the Cells architecture, replacing the
earlier Cells 1.0, Cells 1.5, and Cells 2.0 iterations with a new focus on permanently
reducing load on the legacy cell's database
.18 The
narrowing is the interesting part. Cells 2.0 aimed at a public, open-source contribution
model across cells; Protocells aims at one measurable outcome, taking load off one database,
and explicitly defers cross-organisation contribution, users spanning cells and migration of
public organisations to future scope. Four years of design iteration produced a smaller goal,
not a bigger one, and the programme became deliverable at the point where it stopped trying
to be complete.
Eleven incident records spanning 2016 to 2026, grouped by failure class rather than by year. Three of the four classes recur across the whole decade.
Read in bulk, GitLab's incident corpus sorts into four classes. The state layer failing quietly and being noticed by a human hours later. Maintenance work on a large table becoming an incident in its own right. Two control planes disagreeing about the same object. And the mitigation for one incident causing the next. The first and the last are the ones that span the entire decade unchanged.
have to be executed manually, leaving space to interpretation.
I only noticed this because I looked into the automated restore host by chance.The automated restore had stopped because the backup was too old, and that condition did not alert.
CREATE statement that had been running for five and a half hours, with transaction ID wraparound approaching.we came within 12 hours of a full database shutdown due to a long-running migration. A related incident three weeks later saw index creation saturate the network on the PostgreSQL leader through WAL volume.
the PDB correctly blocked the pod eviction, but GKE's load balancer controller removed the NEG because NEGs are removed independently of PDB. The pod stayed running (PDB worked), but became unreachable via the ILB.
the mis-configuration revert wasn't able to be merged because pipelines couldn't be processed (due to the incident itself).
blocks deployments and blocks feature-flags.What is missing from this corpus is as informative as what is in it. There is no public
incident in which a completed architectural split failed in the way its designers feared. No
postmortem describes a decomposed database losing consistency across the
main and ci boundary, and none describes a cell failing and being
contained. That absence has two readings: either the splits work, or the failure modes they
introduce have not yet been exercised at a scale that produces an incident. The corpus
cannot distinguish them, and an architect copying this design should assume the second until
their own drain test says otherwise.
Every figure below is GitLab measuring its own system, with the date it was measured. There is no independent measurement of GitLab.com in this corpus.
| Metric | Value | Context | As of | Source |
|---|---|---|---|---|
| Largest table, total on-disk size | 2,975 GB | ci_builds, 1,551 GB data plus 941 GB across 30 indexes; 22.7% of the whole database | 2021-06 | Design doc |
| Growth rate of that table | 300 GB / month | Projected to approach 5 TB by end of 2021 without intervention | 2021-06 | Design doc |
| Single index build time | 1.5–6 h | On the largest tables during busy periods; runs synchronously and blocks deployment | 2021-06 | Design doc |
| Physical table size target | < 100 GB | Including indexes; action expected from 10 GB | 2021-06 | Design doc |
| CI builds created per day | > 5 M | Forecast at the time to reach 20 M/day in the first half of 2024 | 2021-01 | CI/CD Scaling |
| Cumulative CI builds stored | 2 bn | 1 bn passed February 2021, 2 bn February 2022 | 2022-02 | CI/CD Scaling |
| Serialised data in two columns | 600 GB + 300 GB | ci_builds.options and ci_builds.yaml_variables, user-provided content | 2021-02 | CI/CD Scaling |
| Memory held per in-flight clone | 4–6 GB | git-pack-objects for a gitlab-org/gitlab clone; caps concurrent clones per node regardless of CPU | 2026-06 | Scaling Git |
| Repository open latency before Gitaly | P99 > 30 s | Against roughly 15 ms of CPU time, which is what identified the filesystem as the cause | 2016 | Gitaly design |
| Production services monitored | 117 | Definition files; 14 Redis, 5 Memorystore, 4 PostgreSQL clusters, 4 pgbouncer fleets, 6 examples or aggregates | 2026-09 | Metrics catalogue |
| Kubernetes clusters in production | 4 | With similarly configured staging clusters; datastores excluded from Kubernetes | 2026-09 | Production architecture |
| Rails codebase | 2.2 M lines | Ruby only; the frontend is a separate single Webpack bundle | 2026 | Monolith decomposition |
| Worst recorded Sidekiq backlog | > 12 M jobs | Caused by a deferral mitigation from a previous incident | 2026-05 | Incident |
| Redis memory growth before intervention | 35 GiB | On the catchall_b Sidekiq shard during the May 2026 near-miss | 2026-05 | Incident review |
| Largest single incident blast radius in corpus | 24.5 M requests | Over 4 h 53 m, at least 53,726 namespaces on the CI path | 2026-02 | Incident review |
| Cells headroom requirement | 10x (high), 100x (medium) | Stated as prioritised requirements, not as achieved figures | 2026-09 | Cells goals |
All of the above are measured by GitLab and reported by GitLab; none is independently verified. The forecast figures in the CI scaling document (20 million builds a day by 2024) are projections made with Prophet in 2021, and the corpus contains no document saying whether they were met. Three quantities are unknown and worth naming: fleet sizes per service, saturation headroom (GitLab's forecasting output is classified confidential under its own data standard, so the method is public and the values are not), and cost of any kind. If you need a cost comparison, this corpus cannot give you one.
The planning number an architect should actually carry away is not on the table, because it is a ratio rather than a measurement. GitLab's index build time crossed the length of its deployment window somewhere around a table size of one terabyte, and the response was a target four times smaller than that with intervention starting at one hundredth of it. If you want one heuristic from this decade, it is that the size at which a table becomes a release-engineering problem is roughly an order of magnitude below the size at which it becomes a query-performance problem, and you will hit the first one first.
Every source behind this page, graded and filterable. All of them are GitLab's own primary record; the reason for that, and what it costs the guide, is stated below the wall.
The earliest detailed postmortem in the tracker. Two databases both took writes for six hours after an unalerted Pacemaker failover, and the losing node's writes were discarded deliberately to restore a single primary.
One of a cluster of issues filed on 1 February 2017, the day after a database incident.
It states that the existing 24-hour snapshot has proven to be inadequate as a recovery
technique
, which dates the moment GitLab's backup posture changed.
Filed alongside the snapshot issues. The corrective action is a shell prompt: show the environment in colour and show more of the hostname, so an engineer on a console knows which machine they are on.
The 2018 move off Azure was rehearsed with announced maintenance windows. The test plan reads as a list of things that went wrong in the first dry run, including closing and reopening the front door correctly and verifying that it closed.
Three days after the migration, automated base backups were absent in the new environment and nothing alerted. Found by an engineer who happened to look at the restore host.
A near miss recorded in plain language: twelve hours from a full database shutdown because an index build had been running for five and a half hours and vacuum could not proceed.
231 minutes of 503s, root-caused to a configuration change. Its labels are as
informative as its body: blocks deployments and
blocks feature-flags, the two levers an incident response needs most.
The clearest single document in the corpus. It contains the PDB and NEG disagreement, the measured blast radius, and the sentence about the revert that could not be merged because the pipelines were down.
A severity 1 in which the stated cause is an earlier incident's mitigation: deferred audit event jobs accumulated until Redis memory was exhausted and the cluster lost quorum, with a backlog above 12 million jobs.
A near-repeat one week after the severity 1, with the same worker and the same shard, because the corrective actions had not yet shipped. It also identifies the feedback loop: deferring the job made Sidekiq itself the dominant source of new enqueues.
The pivot document of the decade. It sets a numeric target, lists the top 30 tables with sizes, ties specific incidents to specific tables, and spawns two ADRs that turn the target into a static-analysis rule.
Records the integer primary-key ceiling on ci_builds as an existential
deadline for December 2021, the two options considered for build queuing, and the
decision to move serialised columns to a metadata table to slow growth.
The fourth iteration of the cell architecture, narrowed to one goal: reducing load on the legacy cell's database. Carries a decision log of 28 ADRs, one marked obsolete, and a future-scope section listing what the earlier iterations promised and this one defers.
Requirements written as a prioritised table, including 10x headroom at high priority and
100x at medium, and cost per user similar or lower to GitLab.com
as an explicit
constraint on the design.
A complete ADR: two proofs of concept, one chosen, the alternatives named, and the consequences accepted in writing including increased vendor lock-in and a requirement knowingly unmet.
A full design kept after rejection, with the reason preserved: cells bounce requests they cannot serve, which requires buffering and allows a request to be processed more than once.
A design that was implemented and is now being deleted. The header explains that
rejected is the closest available status and that the work was largely
built and is now being unwound
, along with the features that depended on it.
The third attempt at horizontal Git scaling, and the document that declares the second one failed. Proposes object storage as the source of truth with stateless Gitaly nodes and a content-addressed MVCC format, and names agentic workloads as a rising load source.
Moves analytical consumers off the transactional database by streaming logical replication events into a pub/sub layer, with an explicit non-goal of letting the application emit custom events.
Keeps Elasticsearch for issues and comments and adds a purpose-built engine for code, on the grounds that code search needs exactness and regex support that the general engine does not provide.
Proposed in 2023 and still proposed in 2026, now with a section headed "the agentic imperative" arguing that modular boundaries are a prerequisite for agents, which cannot carry the implicit context a long-tenured engineer carries.
The sharding design of 2021, opened as a draft merge request and closed unmerged. The approach that eventually shipped instead is domain decomposition followed by the Cells programme.
The removal programme for the Gitaly transactional cluster, with the order of work and
the reason: the code is dead weight: a large surface to maintain
. One constituent
merge request touches 365 files.
The original argument for putting an RPC layer in front of Git, with the measurement that motivated it and the alternative that was refused. Still in the repository ten years later, now describing a system whose storage model is being replaced.
Documents the live decomposition into main, ci and
sec, and the application-level gitlab_schema classification
that enforces it, including why PostgreSQL schemas were not used.
117 service definitions in jsonnet, which is the most precise public statement of GitLab.com's topology: every Redis family, every PostgreSQL cluster, the topology service, Zoekt, Siphon and ClickHouse all appear as first-class services.
The operational procedure for adding a Redis shard and moving workers onto it, including why Redis Cluster was not used and the recommendation of one queue per shard.
Describes classifying every monitored resource by whether it scales horizontally, forecasting saturation daily with Prophet through a tool called Tamland, and filing the warnings as issues on a public board.
The audited description of what runs GitLab.com: Google Cloud us-east1,
four production Kubernetes clusters, and datastores deliberately outside them.
Establishes the cadence that the database size target was protecting: changes deploy to GitLab.com multiple times per day from packages built off the default branch, and self-managed packages follow monthly.
Thirty documents on the wall, drawn from a ledger of thirty-three, one host, one organisation. There are no engineering blog posts,
conference talks, papers or vendor case studies in this guide. The network policy applied
to the session that produced it resolved gitlab.com and refused every other
host, including GitLab's own blog, documentation site and status page. Rather than cite
pages that could not be opened, the hunt was run entirely against what answered. The result
is a corpus weighted almost entirely toward the three highest-value tiers, incidents,
decision records and source, and completely missing the outside view. Every performance
number here is GitLab measuring GitLab. Treat the mechanisms as well evidenced and the
outcomes as self-reported.
Six rungs. The line between toy and production-shaped is at rung four, where you stop adding capacity and start removing work.
Load a table to 50 GB in PostgreSQL with five indexes, then build a sixth index without
CONCURRENTLY while a write workload runs. Time it. Then do it with
CONCURRENTLY and time the vacuum behind it.
Done when: you can state your own index build time as a function of table size, in minutes. Teaches: why a size target, rather than a query budget, is the enforceable control.
Write a check that fails CI when a migration adds a column to a table above your threshold, or an index to a table above half of it. Wire it into the same job that runs your linters, with an explicit override that requires a named approver.
Done when: a deliberately oversized migration is rejected by CI and the override path is documented. Teaches: the difference between a policy and a control.
Take a service with one PostgreSQL database and move one bounded context to a second database. Classify every table. Find every cross-domain join and foreign key you have to break, and write down what replaces each one.
Done when: both databases run with no cross-database transactions and the classification is enforced in code. Teaches: that decomposition costs are paid in the application layer, not the database layer.
Run two complete copies of the application, each with its own database, and a small stateless router that maps tenant to instance through a lookup service. Keep the router under a few hundred lines. Measure the added latency at the edge.
Done when: a tenant can be moved between instances and the router picks up the change without a deploy. Teaches: that the hard part of cells is the authoritative map, not the routing.
Build a job queue on Redis, add a deferral mechanism for a misbehaving job class, then make the deferral re-enqueue through the same queue. Watch memory. Now replace deferral with a drop and a counter.
Done when: you have a graph showing the deferral path growing without bound and the drop path flat. Teaches: the May 2026 failure class, which is the one most likely to be in your own system already.
List every action your incident response needs: revert a config change, disable a feature flag, drain a zone, roll back a deploy. For each, write down which systems must be healthy. Then take the most coupled one and build a path that bypasses it.
Done when: one emergency action can be completed with your CI system deliberately switched off. Teaches: the lesson GitLab recorded in February 2026 and had not yet converted into a corrective action.
This guide was assembled almost entirely through one API rather than a search engine. The queries below are the ones that produced it, and they work against any public GitLab instance.
curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?labels=incident,severity::1&state=closed&per_page=60&order_by=created_at&sort=desc"curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?labels=incident-review&state=all&per_page=20&sort=desc"curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?state=all&per_page=10&order_by=created_at&sort=asc"curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fcontent-sites%2Fhandbook/repository/tree?path=content/handbook/engineering/architecture/design-documents&per_page=100"curl "https://gitlab.com/gitlab-com/content-sites/handbook/-/raw/main/content/handbook/engineering/architecture/design-documents/cells/_index.md"grep -l "status: rejected" *.mdcurl "https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/merge_requests?state=closed&search=blueprint&per_page=50"curl "https://gitlab.com/api/v4/groups/gitlab-org/epics/23151"curl "https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/merge_requests/64115"curl "https://gitlab.com/api/v4/projects/gitlab-com%2Frunbooks/repository/tree?path=metrics-catalog/services&per_page=100"curl "https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction"curl "https://gitlab.com/api/v4/groups/gitlab-com%2Fgl-infra/projects?per_page=100&simple=true"Two notes for anyone repeating this. Discussion notes on merge requests and issues require authentication, so the argument inside a thread is not reachable anonymously even when the issue body is; plan to work from descriptions, labels and linked epics. And the tracker's own format changed over the decade, from hand-written timelines in 2016 to structured incident.io exports from 2025 onward, which means a text search tuned to one era silently misses the other.
Numbered in the order they first appear. The evidence ledger in
sources.md uses its own numbering, one row per claim rather than one per
document, so a document supporting three claims appears three times there and once here.