GitLab is the rare large engineering organisation that argues about its architecture in public:
design documents with a status field, twenty-eight numbered decision records for one programme,
rejected proposals kept in a rejected/ directory, and incident reviews with the
blast radius filled in. This guide reconstructs the decade from those artefacts and extracts the
rule that decided almost every fork in the road, which turns out not to be traffic.
One codebase has to serve the largest instance of itself and tens of thousands of installations that customers run on their own hardware. Every scaling move has to survive both. That constraint, not request volume, explains most of what follows.
ci_builds, at 22.7% of the entire production databasegit-pack-objects clone of the main repository holds, capping clones per nodeState the problem without naming the company: you run a stateful application whose data model is one connected graph, you host the busiest instance of it yourself, and you ship the identical software to customers who run it on a single virtual machine. You cannot adopt an architecture your customers cannot operate, and you cannot refuse to scale the instance you host. Most standard answers, managed sharded databases, a Kafka bus, edge compute, a service mesh, fail the first test or fork the product into two products.
I am going to call that constraint the parity tax, and I am naming it because the
sources do not. They call it "self-managed support", "feature parity across deployment models",
or a non-goal. It shows up in the routing decision, where GitLab's own architecture decision
record notes that Cloudflare Workers meet every requirement
"apart from the self-managed, which is a low priority requirement".
It shows up in change data capture, where Debezium was rejected because it
"requires Kafka (this can be a problem for self-managed instances)".
It shows up in analytics, where ClickHouse shipped for the hosted product in FY24 Q3 while
self-managed integration was deferred
"due to the uncertain costs and management requirements".
Three teams, three technologies, one tax.
The cellular architecture, GitLab's flagship answer to horizontal scale, was put on hold in its entirety. The design document lists Cells 1.0, 1.5 and 2.0 each as "(On hold)" and says "Protocells replaces Cells 1.0, Cells 1.5, and Cells 2.0 with a new focus of permanently reducing load on the database". That is four years of work, twenty-eight numbered decision records and a live routing service re-aimed at a narrower target, with regional compliance, multi-cloud support, load re-balancing and global administration moved to the non-goals list. In the same window, the Gitaly team wrote of Praefect, the clustering layer they shipped for replication and read distribution, that it "has basically failed to yield a horizontally scalable cluster" and is "treated as a failed design".
Read those two together and the decade has a shape. Every attempt to scale a shared stateful tier while leaving the ownership boundary alone bought between one and three years and left a new consistency obligation behind. The only changes that removed load permanently changed who owns the data: move blobs out of Postgres, move analytics into a different engine, move a customer's whole organisation onto its own instance. Those are product decisions wearing infrastructure clothes, which is why they take four years.
creation-date in 2026.What this guide covers: the architecture of one company's hosted platform and the
decisions behind it, traced through primary artefacts published between 2016 and September
2026. What it deliberately does not cover: comparison with competitors, cost, and
anything the public record cannot support. This dig ran under a network policy that reached
gitlab.com and github.com and nothing else, so there are no
conference talks, no papers, and no independent measurement here. Every figure below is GitLab
measuring its own systems for its own engineers, which is credible and is not the same as
corroborated.
The 2026 shape, reconstructed from the runbooks repository, the cells decision records and the service documentation. Note where the diagram is wide: that width is the decade's work.
The application is still one Rails codebase. What changed underneath it is the shared tier.
The runbooks repository, generated from the SRE team's service catalogue, holds directories for
patroni, patroni-ci, patroni-registry and
patroni-sec, each with a matching pgbouncer, and roughly twenty Redis
estates: redis-sidekiq, redis-sessions,
redis-cluster-cache, redis-cluster-shared-state,
redis-ratelimiting, redis-tracechunks and more, plus
memorystore entries for the managed instances they are moving to. One database and
one Redis became four and twenty by pulling workloads apart one at a time, each split justified
by its own incident history.
Repositories moved from a shared network filesystem to Gitaly, an RPC service that keeps Git on local disk, then to Praefect for replication, and now toward object storage with an MVCC layer. Each generation was a response to the previous one's ceiling, and the current proposal states the goal plainly: "Gitaly nodes are stateless."
Sources: Gitaly design, Scaling Git
Sidekiq runs against many Redis instances, routed at the application layer, because "Redis Cluster is not suitable for Sidekiq since there are only a small subset of hot keys". The SRE recommendation is one queue per shard, which makes a shard a blast-radius boundary as much as a capacity one.
Source: Sidekiq sharding runbook
Analytical queries left the OLTP database for ClickHouse, fed by Siphon, a purpose-built logical-replication consumer. The motivation section is unusually direct: analytical workloads in the OLTP database "led to numerous production incidents over the years".
Sources: ClickHouse usage, Siphon
The divergence point worth marking is the routing layer at the top of the diagram. It exists because of the cells programme, it runs on Cloudflare Workers rather than in GitLab's own compute, and the HTTP router project has been under active development since February 2024. For a self-managed customer that row does not exist. For GitLab.com it is now the first thing every request touches, which is worth holding in mind when the failure catalogue shows a Cloudflare outage taking the whole site down for twenty-five minutes in December 2025.
Two components deserve attention because they show the cost of the splits rather than the
benefit. First, ci_namespace_mirrors and ci_project_mirrors: when the
CI tables moved to their own database, the namespaces and projects
rows they referenced could no longer be joined, so those rows are copied into the CI database
and reconciled by consistency-check workers, with a runbook for the alerts that fire when the
copies drift. The runbook notes that the mirroring happens
"even ... on single database mode",
which is the parity tax again: the hosted platform's sharding scheme became part of the
software everyone runs. Second, the Topology Service, a new cluster-wide component on Cloud
Spanner that exists only to answer "which cell owns this organisation". Partitioning a system
always adds a component whose job is to know where things are, and that component is
necessarily global.
Five forks, each with the rejected option and the stated reason, and the condition under which the rejected option becomes the right one for you.
git-pack-objects holds is what makes a node's ceiling a
memory ceiling; below that, replication is adequate and much simpler.main and
ci databasescells/rejected/| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Repository storage, 2016 | Gitaly RPC on local disk | Network filesystem | P99 of 30 wall seconds to open a repository against 15 ms of CPU | Gitaly design doc |
| Repository availability, 2020 | Praefect replication and voting | Doing nothing | Gitaly had explicitly deferred replication in 2016 | Gitaly documentation |
| Repository scale, 2026 | Object storage plus MVCC, stateless nodes | Praefect, networked filesystems, Git in a database | Praefect "treated as a failed design"; writes must land on all nodes at once | Scaling Git |
| Build queue, 2021 | Dedicated Postgres queuing tables | Redis queuing | Chosen after evaluating both; the queue was being built from a 2 billion row table | CI/CD scaling |
| Job fan-out, ongoing | Application-level Sidekiq sharding | Redis Cluster | "only a small subset of hot keys" | Sharding runbook |
| Registry metadata, 2020 | Metadata in Postgres, online garbage collection | Metadata in the blob store | Offline mark and sweep needed read-only mode for "hours/days" | Registry metadata DB |
| Cell disaster recovery | Backup and restore (ADR-024) | Geo replication (ADR-006) | A reversal recorded in the same decision log rather than a rewritten document | Cells decision log |
| CI autoscaling, 2022 | Provider plugins and a new autoscaler | Continuing the Docker Machine fork | Upstream stopped non-bugfix work in 2018 and the repository was archived | docker/machine issue 4537 (now docker-archive-public/docker.machine) |
Seven incident reviews, published by GitLab with impact numbers filled in. They fall into three classes, and the classes map onto the architecture above more neatly than anyone would want.
First, saturation of a shared queue or pool, where one workload consumes what many workloads share. Second, the coordination plane, the components that decide who is primary and who may connect, whose failure is total rather than partial. Third, a single stateful node, which after a decade still decides availability for the repositories on it. Provider outages are a fourth category, architectural in one respect only: the routing layer the cells programme adds sits inside one of them.
catchall_b Sidekiq Redis shard. Deferral was toggled on, and, in the review's words, "deferring ... jobs caused sidekiq itself to re-enqueue more of them, amplifying the problem". Redis memory grew to 35 GiB before an engineer intervened.sec cluster paused. Pause is a legitimate intermediate state, and the review notes the playbook produced no error output, so the stall was not obvious.gitlab-org namespace", including every pipeline and integration touching it.All of these are GitLab measuring GitLab, taken from design documents and incident reviews. None is independently verified, and the older ones describe a system that has since changed.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Largest single table, with indexes | 2,975 GB | GitLab.com | ci_builds, 30 indexes, 22.7% of the database | Jun 2021 | design doc |
| Growth of that table | 300 GB/month | GitLab.com | Forecast to approach 5 TB within the year | Jun 2021 | design doc |
| Target physical table size | < 100 GB | GitLab.com | Act at 10 GB; the number is called pragmatic, not derived | Jun 2021 | design doc |
| Single B-tree index build | 1.5–6 h | GitLab.com | Blocks deployments because migrations run synchronously | 2021 | design doc |
GIN trigram reindex on notes | aborted at 12 h | GitLab.com | Blocked other vacuum activity | 2021 | design doc |
| CI builds created per day | > 5 million | GitLab.com | Target for the programme was 20 million per day | 2021 | design doc |
| Cumulative builds | 1B → 2B | GitLab.com | 1 February 2021 to February 2022 | 2022 | design doc |
| Builds per day at peak | ~4 million | GitLab.com | Stated in the runner autoscaling document | Jan 2022 | design doc |
| Serialized data in two columns | 600 GB + 300 GB | GitLab.com | ci_builds.options and ci_builds.yaml_variables | Feb 2021 | design doc |
| Memory per in-flight clone | 4–6 GB | GitLab.com | git-pack-objects for the gitlab-org/gitlab repository | Jun 2026 | design doc |
| Repository open latency before Gitaly | P99 ~30 s | GitLab.com | Against 15 ms of CPU time, which is what implicated the filesystem | 2016 | design doc |
| Redis memory before intervention | 35 GiB | GitLab.com | Sidekiq shard catchall_b during the deferral loop | May 2026 | incident review |
| Failed requests, site-wide outage | ~13 million | GitLab.com | 503s reported by the CDN, ~100% of traffic | Nov 2022 | incident review |
| Requests during edge outage | ~155 million | GitLab.com | 25 minutes, all public APIs | Dec 2025 | incident review |
| Downtime to resize a tenant's stateful services | 20–40 min | GitLab Dedicated on AWS | Quoted as the comparison when sizing a cell | 2024–2026 | ADR-009 |
Everything above is measured by GitLab on its own systems, and nothing is corroborated by an outside party. Three specific cautions. The 2021 database figures describe the state that motivated partitioning and decomposition, so they are a picture of the problem rather than of today. The build-rate figures disagree with each other, "more than 5 million ... each day" in the 2021 document against "around 4 million builds per day at peak" in the 2022 one, which most likely reflects daily total against peak-day rate rather than a decline; the documents do not reconcile them, so treat both as order-of-magnitude. The one number that has aged well is the 100 GB table target, which reads as an operational constant rather than a scale-dependent one, and which the authors themselves call pragmatic. Unknown: nothing in this corpus states a cost, a request rate, or a total database size, so any cost model you build from this page would be invention.
Fifty claims from thirty-two artefacts, all first-party and all fetched on
6 September 2026. There are no blog posts, papers or talks here, and that is a limitation of
this dig rather than of the record: the session's network policy reached only
gitlab.com and github.com. The upside is that every card below is
the artefact itself rather than a description of it.
The founding argument for the RPC layer, with the measurement that triggered it and an explicit scope exclusion for replication and high availability.
Proposes stateless Gitaly nodes over object storage and, in its alternatives section, declares Praefect a failed design with four named mechanisms.
Build volumes, the 32-bit primary key that would have taken the site down before December 2021, and the rejection of Redis for build queuing.
The top thirty tables with sizes and index counts, plus the operational consequences: six-hour index builds, vacuum that cannot keep up, an abandoned reindex.
Partitioning driven by a product rule that already existed: queued builds older than twenty-four hours are discarded.
What decomposition left behind: namespaces and projects copied
into the CI database, consistency-check workers, and alerts for drift.
Why Redis Cluster was rejected for job queues, and the recommendation of one queue per shard so that a shard is an isolation boundary.
The service inventory, generated from the SRE service catalogue: four Patroni clusters, matching PgBouncer sets, roughly twenty Redis estates, and managed-Redis entries.
Metadata in the blob store forced offline garbage collection measured in hours or days, and feature work was frozen until the storage model changed.
Debezium, PeerDB and Airbyte each assessed and rejected, with the Kafka dependency named as the disqualifier for self-managed installations.
The second database engine, adopted for analytics on the hosted product first, with self-managed integration deferred over cost and management uncertainty.
Documents four years of depending on an abandoned upstream, the decision to fork it, and the plugin architecture meant to end that dependency.
The upstream announcement that non-bugfix work had stopped, on the repository GitLab's CI autoscaling depended on. The repository is now archived and read-only.
Twenty-eight numbered decision records, three iterations marked on hold, and a stated replacement by a narrower programme.
The re-scoped programme. Names GitLab.com the "legacy cell", targets database load relief, and moves regional compliance, multi-cloud, rebalancing and global admin to non-goals.
Istio rejected after a proof of concept, two alternative routing designs rejected, and the self-managed gap accepted in writing along with the vendor concentration.
Two years after the custom rule engine shipped, its linear rule evaluation is being replaced progressively, with unmigrated rules falling through to the old engine.
Sizing a cell from published reference architectures, with the acknowledgement that resizing stateful services later may need twenty to forty minutes of downtime.
A complete alternative routing design, kept in a rejected/ directory and
cited by the ADR that rejected it.
The routing service as a live repository, created February 2024 and active through September 2026, which is what separates this programme from a design exercise.
A migration path for replacing Praefect's coordination with Raft, opened as a draft and closed without merging, before the 2026 proposal took a different route entirely.
2.2 million lines of Ruby, a status field that still reads proposed three
years on, and a fresh justification built around autonomous coding agents.
The tenancy boundary as a product concept: GitLab.com splits into a shared public experience and isolated enterprise organisations.
Consolidates a per-feature upload story into one path, and is the counterpart to moving artifacts, LFS and uploads out of the database entirely.
The end state of the 2016 decision, written as a support boundary: only local storage is supported for repository data, and network filesystems are not.
Roughly 100% of traffic, about 13 million 503s, caused by a local tool acting on production and removing the coordination layer under Patroni.
Three hours of dropped audit events, 35 GiB of Redis memory, and a mitigation that re-enqueued the jobs it was meant to shed. A repeat of an S1 a week earlier.
Eight hours of queueing violations, with deployments and feature flags blocked, from two background worker classes holding transactions open.
17,383 customers and 445,621 Git requests affected in 24 minutes, because a switchover step waiting on a hardware key timed out silently.
Memory spike, then a reboot that failed to remount the data disk: three and a half hours for everything in one large namespace.
Nearly three hours, severity 1, with Pages, registry, runners, LFS, artifacts and traces failing together and 17 or more pages in minutes.
About 155 million requests affected, with follow-up actions recorded to reduce the dependency. The cells routing layer runs at that same provider.
Six rungs against a toy application with one hot table, one queue and one shared store. The line between toy and real is rung four.
Write the query from the GitLab design document that lists the top thirty tables by total size, index size and index count, and run it weekly. Add a forecast from the last twelve weeks and project when each table crosses 100 GB and when each integer sequence exhausts.
Done when: you can name the date each of your top three tables crosses the threshold. Teaches: capacity work starts with a forecast, and sequence exhaustion has no graceful degradation.
Find the retention promise you already keep, then partition the hot table on it and route reads through a helper that knows about the partitions. Do not invent a new retention rule for the migration.
Done when: dropping the oldest partition is a metadata operation and no application code changed. Teaches: partitioning is a product decision expressed in DDL.
Move one domain's tables to a second database. Where a foreign key crossed the line, create a mirror table, a synchronisation path and a consistency checker that emits a mismatch count. Alert on the count.
Done when: you can break the mirror deliberately and your checker finds it before a user does. Teaches: what decomposition actually costs, which is a reconciliation loop per broken join.
Route one job class to its own queue on its own Redis, then run a load generator that floods it. Watch the other queues. Then enable a deferral or retry mitigation and flood it again, looking specifically for the mitigation increasing enqueue rate.
Done when: the flood degrades only the isolated class, and you have a graph of the mitigation's own load. Teaches: shard boundaries are blast-radius boundaries, and shed-load paths need their own load test.
Introduce a tenant identifier in every request path, a lookup service that maps tenant to instance, and a router that reads the map. Run two instances behind one hostname. Make every identifier globally unique now, as GitLab did, so a later move is possible.
Done when: a request for tenant B lands on instance B with no application change. Teaches: partitioning adds a global component whose availability now bounds everything.
Migrate one tenant's data to the second instance, flip its mapping, and then go looking for the dangling references: objects on the source that point at the moved tenant, and pages on the target that reference users left behind. Fix them by rendering gracefully rather than by copying more data.
Done when: both instances render correctly with references missing, and you can state the write-unavailable window in seconds. Teaches: the hard part of cellular architecture is the graph that crosses the boundary, not the routing.
This page was assembled almost entirely through one public REST API. These are the calls that produced it, and they work for any organisation that keeps its engineering record on GitLab.
GET /api/v4/projects/42817607/repository/tree?path=content/handbook/engineering/architecture/design-documents&ref=main&per_page=100GET /api/v4/projects/42817607/repository/files/<url-encoded path>/raw?ref=mainGET /api/v4/projects/42817607/repository/tree?path=<...>/design-documents/cells/rejected&ref=mainGET /api/v4/projects/7444821/issues?labels=incident-review&order_by=created_at&sort=desc&per_page=40GET /api/v4/projects/7444821/issues?labels=incident-review&created_before=2023-01-01T00:00:00ZGET /api/v4/projects/7444821/issues/<iid> # the Key Information table carries duration and impactGET /api/v4/projects/2009901/merge_requests?state=closed&search=praefectjq '.[] | select(.state=="closed" and .merged_at==null) | .title'grep -A2 '^status: proposed' <design-doc>/_index.md # then read creation-dateGET /api/v4/projects/1148549/repository/tree?path=docs&ref=master&per_page=100jq -r '.[].name' tree.json | grep -cE '^(patroni|redis)' # count the estatesGET /api/v4/projects/<group%2Fproject> # created_at and last_activity_atThe transferable trick is not the API. It is that four artefact types
answer four different questions, and only one of them is the one everybody reads. A design
document tells you what a team intended. A rejected/ directory tells you what they
considered and refused. A closed merge request tells you when they gave up on something. A
runbook directory listing tells you what they actually run. Where those four disagree, the
disagreement is the finding.