GitLab publishes 207 architecture design documents and 172 labelled incident reviews in repositories anyone can read. Reading a decade of them together shows an architecture that kept decomposing the parts that were large while every serious outage arrived through the parts that were shared. This guide reconstructs the decisions, the rejected ones included, and turns them into rules you can apply to a platform of your own.
A product designed to be installed once per company, operated instead as one shared instance for everybody, for ten years.
State the problem without the company's name and it is one most platform teams will recognise. You have an application whose data model assumes a single installation: one identifier space, one queue, one repository filesystem, one relational database holding every tenant's rows in the same tables. Then you sell it as a service. Every tenant you add makes the shared parts bigger, and none of them makes the shared parts more divisible. At some point growth stops being a capacity question and becomes a shape question.
GitLab has been living that problem in public since before 2016, and it is unusually
useful to study because the working-out is published rather than narrated. The company's
architecture design documents sit as Markdown in a Git repository, 207 of them, each with a
status field and a list of alternatives considered. Rejected designs are kept on purpose:
the Cells set has a rejected/ directory whose documents say, in the file
itself, that they are retained “so that we can document the reasons for not choosing
this approach”. Production incidents get a review issue in a public project, 172 of
them carrying the incident-review label, with customer impact, request counts
and root cause filled into a template.
The horizontal-scalability programme changed its own goal. Cells opened in September 2022 as “a new architecture for our software as a service platform… horizontally scalable, resilient, and provides a more consistent user experience”. Four iterations later the current one, Protocells, is described as “replacing the earlier Cells 1.0, Cells 1.5, and Cells 2.0 iterations with a new focus on permanently reducing load on the legacy cell's database”. A platform re-architecture narrowed, over four years and in its own words, into a database relief programme. That is not a criticism; it is the most honest thing in the corpus, and it is what a multi-year decomposition actually looks like from the inside.
What this guide covers: the request path, the storage layers underneath it, and the
tenancy boundary, between roughly 2016 and September 2026, as evidenced by GitLab's design
documents, decision records, repository contents and public incident reviews.
What it deliberately does not cover: GitLab's self-managed and Dedicated products
except where they constrain the SaaS design, its AI features, its commercial history, and
any comparison with how other companies solved the same problem. It also does not cover
GitLab's own blog posts. This session's network policy blocked every host except
gitlab.com and github.com, so the guide is built entirely from
material inside repositories. That is a real limitation and it cuts both ways: no
narrative gloss, but also no independent account and no dissenting voice.
Four layers moved over the decade, at four different speeds, and the order they moved in is the interesting part.
Git storage moved first and is still moving. Repositories used to be read straight off a shared filesystem by the Rails application. Gitaly put an RPC boundary in front of that, and the migration is complete in the sense the README states flatly: “All application code accesses Git repositories via Gitaly.” Its goal line is “Fault-tolerant horizontal scaling of Git storage in GitLab, and particularly, on GitLab.com”. The fault-tolerance half went through Praefect, described in the repository's HA design document as “a transparent front end to all Gitaly shards” that routes gRPC calls and runs quorum votes across replicas. The horizontal-scaling half is not finished. In June 2026 a new proposal, Scaling Git, opens by conceding the original constraint is still in force: “Gitaly stores the authoritative copy of a repository on the serving nodes' filesystem. Compute and storage are thus tightly coupled with one another, which makes it hard to scale either of these dimensions.” Its answer is object storage as the single source of truth with stateless Gitaly nodes over a local cache, which depends on Git's pluggable object database work being upstreamed first.
The relational layer moved on a number. In June 2021 a design document set a target that reads like an operational rule rather than an architecture principle: physical tables on GitLab.com under 100 GB including indexes, with action starting at 10 GB. It is worth noticing what that replaces. “Split the monolith” is unfalsifiable and never finishes; “this table is 2.5 TB and grows 300 GB a month” produces a queue of work with an order. The decade's database output, decomposition into separate main and CI databases, partitioning, retention, and now change-data-capture into ClickHouse through a component called Siphon, is what that queue looks like when it is worked.
The analytical load left by replication rather than by service extraction. Siphon, designed in November 2024, “delivers serialized CDC (change data capture) data from the PostgreSQL logical replication stream to a queueing system” for consumers to load into ClickHouse or Snowflake. The notable choice is the non-goal: it is PostgreSQL only, and it is explicitly “mostly transparent to application developers making changes to the Rails application”. Analytics was moved off the transactional database without asking the monolith's authors to change how they write.
The tenancy boundary moved last, and it is the one that forced everything else. The Cells goals document does not hedge: “GitLab.com operates as a single monolithic instance with a shared database, which creates a fundamental scalability bottleneck”, and the estimate behind it names the two resources that cannot be fixed by more of the previous decade's work: PostgreSQL and Redis are “non-horizontally scalable resources, even when database partitioning and decomposition are taken into account”. That sentence is the hinge of the whole ten years. Partitioning buys time on the tables; it does not divide the instance.
The logical boundary had to be invented before anything could be split, because the data model had no unit of ownership big enough to move. Organizations became that unit, and the rule that makes it work is stated as a constraint on code, not on infrastructure: cross-organization operations must go through public APIs.
Evidence: Cells design document
Moving a tenant between cells is impossible while two cells can mint the same primary key. ADR 008 hands each cell a range of sequence values from the Topology Service at provisioning time, after recording five alternatives including composite keys and logical replication. Identity is the first thing a split breaks.
Evidence: Cells ADR 008
This is no longer only a design document. The main Rails repository now carries a
cells-mailroom directory at its root, beside app and
workhorse. The architecture is landing as code in the thing it is meant to
divide, which is how these migrations always go and why they take years.
Evidence: repository tree, gitlab-org/gitlab
Three forks where the rejected option is written down, and the condition that would flip each one.
The line worth pausing on is not the choice but the consequence GitLab wrote next to it: “More vendor locking with Cloudflare, but are already heavily dependent on them.” That is an honest ADR and a common one. Twenty-two months later, on 5 December 2025, a Cloudflare outage took GitLab.com down for 25 minutes, affecting “All SaaS users” and roughly 155 million requests, and the incident review's follow-up actions include reducing the Cloudflare dependency. Both documents are correct. The gap between them is the thing to learn from: an accepted dependency is a decision to accept an outage you have not scheduled, and the ADR is the right place to say how big that outage may be, not merely that the dependency exists.
status: proposed in September 2026Read those two rows together and a rule falls out that applies well beyond this company.
Code decomposition and runtime decomposition solve different problems and are routinely
confused because both are called “breaking up the monolith”. GitLab's record
shows the runtime split advancing through four designs and two clouds while the code split
sat at proposed for three years. If your pain is a saturated shared resource,
module boundaries will not touch it.
That closed merge request is the most interesting artefact in the corpus, and it needs stating carefully. Nothing in it says the transactional layer is being abandoned; it is an exploratory branch, and it was closed. What it demonstrates is that removing several years of consensus machinery was concrete enough for someone to produce the diff, in the same month that a proposal appeared to solve the same problem a different way. Consensus-over-disks and object-storage-as-truth are competing answers to one question, and this repository is one of the few places you can watch the competition rather than read its result.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Tenant routing | Cloudflare Worker over Topology Service | Istio; request buffering; routes learning | PoC results; buffering meant duplicate execution | Cells ADR 001 |
| Cluster-wide identifiers | Per-cell bigint sequence ranges issued at provisioning | Global claim service; composite keys; logical replication | Tenant mobility requires non-overlapping keys with no runtime coordination | Cells ADR 008 |
| Routing control-plane store | Cloud Spanner, multi-region configuration under test | Reusing the existing PostgreSQL estate | Needs high availability with strong consistency, globally | Cells ADR 015 |
| Where cells run | AWS us-east-1, control plane left in GCP | Keeping everything in one cloud | Selected on measured cross-cloud latency to the Topology Service | Cells ADR 019 |
| Analytics off the primary | Logical-replication CDC into ClickHouse or Snowflake | Bespoke per-feature sync tooling | One standard extraction path, transparent to application developers | Siphon, 2024-11 |
| Deploying to many cells | A change-coordinator engine with priorities and rings | CI pipelines plus resource_group ordering, inherited from Dedicated | Adequate at one tenant per merge request; not at cell scale | Coordinating changes in Cells, 2024-07 |
Five reviews from the public corpus, chosen because between them they cover every failure class in it. Not one of them is a bug in the 2.2 million lines of Ruby.
Group them and three classes account for all five. A shared runtime resource saturated by one workload (the Redis pair). A stateful node that owns everyone on it (the Gitaly incident). A change path reaching production where the request path could not (Consul, and arguably Cloudflare, since GitLab was downstream of somebody else's change). The classes are worth naming because they predict where your next incident comes from better than any component inventory does.
gitlab-org namespace: 500s on merge requests, comments and reviews for 3 hours 28 minutes, 2026-03-24, with integrity checks continuing afterwards.All measured and published by GitLab, with the date each was measured, because several are now years old and the growth rates are the point.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Target ceiling, physical table size | 100 GB | GitLab.com | Including indexes; action starts at 10 GB | 2021-06 | Design doc |
| Largest table, on disk | 2.5 TB | GitLab.com | ci_builds: 1.5 TB data, 1 TB across 31 indexes | 2021-06 | Design doc |
| Growth of that table | 300 GB/mo | GitLab.com | Forecast to approach 5 TB within the year if untouched | 2021-06 | Design doc |
| CI builds created | >5 M/day | GitLab.com | Named as the reason database limits were slowing development | 2021-01 | CI/CD Scaling |
| Cumulative builds stored | 1 B → 2 B | GitLab.com | 1 billion by 2021-02-01, 2 billion by 2022-02 | 2022-02 | CI/CD Scaling |
| Stated growth target | 20 M/day | GitLab.com | “Enable future growth by making processing 20M builds in a day possible” | 2021-01 | CI/CD Scaling |
| Redis memory at intervention | 35 GiB | redis-sidekiq | Single-threaded, so CPU did not scale with the resize | 2026-05 | Incident review |
| Peak Sidekiq backlog | 12 M jobs | GitLab.com | Pipeline creation went near zero; >10,000 pipelines affected | 2026-05 | Incident review |
| Requests lost, worst full outage | ~13 M | GitLab.com | 503s reported by the CDN; about 100% of traffic | 2022-11 | Incident review |
| Requests affected, edge outage | ~155 M | GitLab.com | 25 minutes, all SaaS users | 2025-12 | Incident review |
| Refs in a large repository | 866 k | Android, upstream Git | Why ref storage needed replacing; packed-refs scans linearly | 2026-09 | git reftable doc |
| Public record size | 207 / 172 | GitLab | Design documents / issues labelled incident-review | 2026-09-25 | Issue list |
Every figure here is measured and published by the organisation that runs the system, which makes them credible about their own systems and silent about anyone else's. None is a vendor claim and none is independently verified. The database figures are from June 2021 and are certainly stale by now; treat them as a growth rate and a ratio, not as a current state. The number to carry is not 2.5 TB, it is 300 GB a month against a 100 GB ceiling, which tells you the ceiling was being crossed roughly every four months by one table alone. That is the arithmetic that justifies a decade of partitioning work, and it is the arithmetic to do on your own largest table this week.
Everything this page rests on, graded. Sixteen decision records, five incident reviews, six code and repository artefacts, no vendor material and, for the reason given in section 1, no engineering blog posts.
Severity 1. Deferred audit-event jobs exhausted Redis memory, broke cluster quorum and left a backlog of more than 12 million Sidekiq jobs across web, CI and Git.
The near-repeat one week later, with the clearest statement in the corpus of why vertical scaling stopped working: Redis is single threaded, so the resize added memory and not concurrency.
One storage node degraded, then failed to mount its data disk after a mitigation reboot, removing an entire namespace for three and a half hours.
An upstream edge outage removed the product and the responders' own runbooks and dashboards at the same time. Quantified: all SaaS users, about 155 million requests, 25 minutes.
A routine task run from a development environment changed production and removed Consul, on which Patroni depends. About 13 million 503s.
The programme document itself, revised through 2026. Records that Protocells replaces Cells 1.0, 1.5 and 2.0, and restates the goal as relieving the legacy cell's database.
Names the constraint outright: PostgreSQL and Redis are non-horizontally-scalable resources even after partitioning and decomposition, and shared infrastructure produces noisy-neighbour effects between top-level groups.
Chooses an edge runtime over Istio and two internal designs, and records the consequences in plain language, including further vendor lock-in and no self-managed support.
The obvious design, kept in a rejected/ directory. Its recorded cost is
that a request may be sent and processed more than once.
The second rejected router, requiring routable information encoded in the URI and a pre-flight request per route.
2.2 million lines of Ruby, created 2023-05-22, still status: proposed,
with the 2026 revision re-arguing the case from agentic development rather than scale.
The measurable target that organised a decade of database work, with the table inventory that justified it.
Traces a 2012 data model to a 2021 constraint, with build volumes, cumulative counts and a stated target of 20 million builds a day.
Proposes object storage as the source of truth with stateless Gitaly compute and an MVCC manifest, conceding that compute and storage are still coupled today.
Change data capture from PostgreSQL logical replication into a queue, for ClickHouse and Snowflake consumers, deliberately transparent to application developers.
Deployment engine inherited from the Dedicated product, based on CI pipelines, judged insufficient once changes must be sequenced and prioritised across many cells.
First cells provisioned in AWS us-east-1 while the Topology Service stays in GCP, chosen on measured cross-cloud latency.
The Topology Service needs high availability with strong consistency, so the routing control plane runs on managed Spanner rather than the existing PostgreSQL estate.
Sequence ranges issued per cell at provisioning, with five alternatives recorded and linked to the discussion threads that produced them.
Terraform modules to standardise a dependency that several teams had already adopted independently for DNS, WAF and Workers.
Saturation forecasting as an owned artefact that predicts SLO violations and their dates, extended from GitLab.com to Dedicated tenants.
A 365-file change deleting Raft, the write-ahead log and the transaction manager from Gitaly. Opened 2026-06-10, closed without merging, six days before the object-storage proposal appeared.
Transactions run on GitLab.com behind a flag and log, on startup, that the feature is not production ready and “might lead to various issues including data loss”.
Praefect as a transparent front end over Gitaly shards, with the vocabulary of accessors, mutators, voters and quorum that the transactional design is built on.
States the completed half of the migration, “All application code accesses Git repositories via Gitaly”, against the goal that is still open, fault-tolerant horizontal scaling of Git storage.
cells-mailroom/ in the main Rails repositoryCell-specific code sitting at the root of the monolith, beside app,
workhorse and sidekiq_cluster.
The upstream work GitLab's storage plans depend on. Quantifies why ref storage is a scaling problem: 866,000 refs in Android, a 62 MB packed-refs file scanned linearly and rewritten for a two-ref transaction.
incident-review issue list172 labelled reviews in a public project, each on a template with customer impact, request counts, timeline and root cause. This is the corpus the failure section is drawn from, and it is queryable through the REST API.
Six rungs. The line between a toy and something production-shaped is crossed at rung four, where you stop building the split and start testing it.
List every runtime resource in your system that all tenants share and that cannot be divided by work already in flight. Next to each, the saturation metric and the shape of its ceiling: memory, concurrency, disk, connections.
Done when: the list is shorter than ten items and someone disagrees with one of them. Teaches: that "shared" and "large" are different problems.
Take your largest table or busiest shared store, measure its growth rate, and compute the month it crosses a threshold you are willing to name. GitLab's ratio is the model: 300 GB a month against a 100 GB target.
Done when: a calendar date exists and is wrong by a defensible margin. Teaches: forecasting schedules work; utilisation does not.
Find the entity in your data model that could own a tenant's data completely. If none exists, that is the project: everything downstream needs it. Write the rule that cross-unit access must go through a public interface.
Done when: you can name, for any row in your largest table, the unit that owns it. Teaches: why tenancy work starts in the data model and not in the infrastructure.
Stand up two copies of the application with separate databases, a lookup service mapping unit to instance, and a thin router in front. Give each instance a non-overlapping identifier range at provisioning time.
Done when: a request reaches the right instance and a row created on either can be inserted into the other without collision. Teaches: that the router is easy and the identifier space is not.
Migrate one unit between instances with no data loss, then deliberately leave a dangling reference: a comment by a user who did not move. Render both sides.
Done when: pages on source and target still render with references missing. Teaches: that graceful degradation of cross-boundary references is the real cost of splitting, not the copying.
Flood one tenant's jobs into your shared queue and watch every other tenant. Then take the router or its lookup service away and measure what remains available. Do both with your runbooks hosted somewhere else.
Done when: you can name the first mechanism that degrades, and your availability ceiling is written as a product of cells, router and third parties. Teaches: that a cellular architecture inherits a new global singleton, and that recovery tooling belongs in a different failure domain.
Ten years of this company's record point one way. You do not choose where your architecture fails; your singletons choose, and they keep choosing right through a decomposition programme because decomposition removes the large things long before it removes the shared ones. So order the work by divisibility rather than by size: find the resources every tenant touches and no partition scheme reaches, name them in writing, and treat each one as an outage you have already agreed to have. The ones you cannot remove, such as a routing control plane or an edge provider, are not exceptions to that rule. They are the next chapter of it, which is why the incident review that mattered most here is the one where the responders could not reach their own runbooks.
The queries that produced this page. They are API calls rather than search strings, which is the point: for a company that publishes its record as repository data, the API is the search engine.
https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?labels=incident-review&per_page=50https://gitlab.com/api/v4/projects/gitlab-com%2Fgl-infra%2Fproduction/issues?search=Incident%20Review&in=title&state=closed&sort=asccurl -sSI '.../issues?labels=incident-review&per_page=1' | grep x-total.../projects/gitlab-com%2Fcontent-sites%2Fhandbook/repository/tree?path=content/handbook/engineering/architecture/design-documents&per_page=100tree?path=...design-documents/cells/rejectedtree?path=...design-documents/cells/decisions.../projects/gitlab-org%2Fgitaly/merge_requests?state=closed&order_by=updated_at.../merge_requests/8895/notes?sort=asc.../projects/<ns%2Fproject>/repository/tree?path=&per_page=100.../repository/files/<url-encoded-path>/raw?ref=mainTwo habits transfer to any organisation, not only this one. First,
look for a rejected/ or superseded/ directory before you read the
accepted design; the reasons stored there are worth more than the conclusion. Second, when a
vendor's incident reviews are issues rather than status-page entries, you can count them,
group them by service label and read the recurrence, which is a far better input to a
procurement decision than an availability percentage.