Evidence ledger
30 sources
Checked 20 Sep 2026
Evidence ledger
One row per claim in When the database cannot be split: ten years of GitLab.com, read from its own incidents and design documents: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Guide: When the database cannot be split: ten years of GitLab.com, read from its own incidents and design documents Research date: 2026-09-20. Every URL below was fetched during that session.
A note on the shape of this corpus
Every source is GitLab's own public primary record on gitlab.com: the production incident
tracker, the architecture design documents in the handbook repository, the operations runbooks,
merge requests and epics. There are no engineering-blog posts, conference talks, papers or vendor
case studies in this ledger, and their absence is not a judgement about them. The network policy
applied to this session resolved gitlab.com and github.com and refused every other host,
including about.gitlab.com, docs.gitlab.com, handbook.gitlab.com and status.gitlab.com;
GitHub repositories outside one allow-listed project were also refused. Rather than cite material
that could not be opened, the hunt was run entirely against the hosts that answered.
The consequence for the reader is specific: this corpus is strong on what GitLab decided, what it rejected, what broke and when, and weak on anything only an outsider would say. Where a claim rests on GitLab describing its own system, the prose says so.
Tiers follow the skill's grading: postmortem, source, adr, casestudy, blog, paper,
talk, vendor.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitLab | Postgres DB Split brain incident led to data loss - postmortem | postmortem | 2016-07-12 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/48 | The decade opens with a data-loss incident caused by two databases both accepting writes, detected by a human five hours later | "at 1:46:18 UTC pacemaker lost track of the master"; "We had a period of data loss between 1:46 and 7:48, whatever was stored in DB5 got lost because we elected it as the victim." |
| 2 | GitLab | Postgres DB Split brain incident (why section) | postmortem | 2016-07-12 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/48 | Detection, not failover, was the missing piece | "We had a failover but no alerting letting us know that something happened, this led to confusion evaluating the actual state" |
| 3 | GitLab | Update PS1 across all hosts to more clearly differentiate between hosts and environments | postmortem | 2017-02-01 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/111 | A corrective action filed the day after the 2017-01-31 database incident targets the shell prompt, not the database | "This setup is confusing because it doesn't show the environment one is in (staging vs production)" |
| 4 | GitLab | Hourly LVM snapshots of the production databases | postmortem | 2017-02-01 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/113 | The 24-hour snapshot interval was declared inadequate as a recovery technique the day after the incident | "Right now we only generate an LVM snapshot once every 24 hours ... This has proven to be inadequate as a recovery technique." |
| 5 | GitLab | GCP Migration: Short Maintenance Window to Test Fixes from July 21 Findings | casestudy | 2018-07-26 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/366 | The 2018 cloud migration was rehearsed with announced maintenance windows and dry runs rather than a live cutover | "The purpose of this window is to test fixes to some of the issues we found during the July 21 dry-run"; steps include "closing the GitLab.com front-door completely and correctly, plus verification of said closure" |
| 6 | GitLab | 2018-08-14 No automated wal-e database backups in gprd | postmortem | 2018-08-14 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/414 | Eighteen months after the 2017 data-loss incident, automated base backups were silently absent in the new environment and found by chance | "It seems like cronjobs are simply missing from gprd. The latest backup is the one I was taking manually right after the GCP migration on August 11th."; "I only noticed this because I looked into the automated restore host by chance." |
| 7 | GitLab | 2021-05-18: Long-running CREATE INDEX blocking vacuum | postmortem | 2021-05-18 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4633 | Routine index maintenance on a very large table came within hours of stopping the database | "we came within 12 hours of a full database shutdown due to a long-running migration ... a CREATE statement that had been running for 5h30m" |
| 8 | GitLab | 2021-06-06: CREATE INDEX operation on Postgres causing high network saturation | postmortem | 2021-06-06 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4823 | Index creation on a multi-terabyte table saturates the replication path, not just the table | "CREATE INDEX Query created some WALs that temporarily hammered the network on our PSQL leader, but did not cause any SLO impact." |
| 9 | GitLab | 2023-07-07: Site-wide outage triggered by restart of stale terraform pipeline | postmortem | 2023-07-07 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15997 | A configuration change path, not the request path, produced a 231-minute site-wide outage | "Impact Duration: ~16:15 UTC - 19:36 (231 minutes)"; "Root cause: ~\"RootCause::Config-Change\"" |
| 10 | GitLab | 2026-02-03: Redis deployment error causes delayed CI processing and GitLab Duo outage | postmortem | 2026-02-03 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21192 | A Redis configuration change took out CI processing and every AI feature at once | "A configuration change to Redis increased CPU reservations on Sentinel nodes, causing a Redis outage that prevented workhorse from connecting to Redis." |
| 11 | GitLab | Incident Review: Redis deployment error causes delayed CI processing and GitLab Duo outage | postmortem | 2026-02-04 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21208 | Two Kubernetes controllers disagreed about whether a pod was serving: the disruption budget held the pod, the load-balancer controller withdrew it | "The PDB (pod disruption budget) correctly blocked the pod eviction, but GKE's load balancer controller removed the NEG (network endpoint group) because NEGs are removed independently of PDB. The pod stayed running (PDB worked), but became unreachable via the ILB." |
| 12 | GitLab | Incident Review (same), "What was difficult" | postmortem | 2026-02-04 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21208 | The fix could not be shipped because shipping runs on the thing that was broken | "The mis-configuration revert wasn't able to be merged because pipelines couldn't be processed (due to the incident itself)." |
| 13 | GitLab | Incident Review (same), Key Information | postmortem | 2026-02-04 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/21208 | Blast radius of the February 2026 incident | "Customers Affected: At least 53,726 unique namespaces ... 1266 unique Duo workflows"; "Requests Affected: 24,543,552"; "Total Duration: 4 hours, 53 minutes" |
| 14 | GitLab | 2026-05-12: Error rates violating SLO | postmortem | 2026-05-12 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22103 | The mitigation applied to an earlier incident became the cause of the next one | "Earlier mitigation for a previous incident led to audit event jobs being deferred, which then accumulated and consumed all available memory on the Redis Sidekiq cluster ... breaking the cluster quorum."; "the Sidekiq queue backlog reached over 12 million jobs" |
| 15 | GitLab | Incident Review: Redis primary CPU saturation on redis-sidekiq nodes approaching capacity limit | postmortem | 2026-05-20 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22170 | The same worker, shard and failure mode recurred one week later because the corrective actions had not shipped yet | "This is a near-miss repeat of the S1 incident from one week earlier — the same worker, the same Redis shard, the same failure mode. The corrective actions from the FCL (circuit breaker, runbook, Redis streaming buffer) had not yet been fully deployed when this recurrence happened."; "Redis memory grew from normal levels to 35 GiB before intervention." |
| 16 | GitLab | Incident Review (same), deferral feedback loop | postmortem | 2026-05-20 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/22170 | The load-shedding mechanism amplified the load it was shedding | "Deferring AuditEvents::AuditEventStreamingWorker jobs led to sidekiq itself becoming the dominant source of new enqueues for that same job class, causing Redis memory to grow continuously rather than stabilize." |
| 17 | GitLab | Database Scalability: Limit on-disk table size to < 100 GB for GitLab.com | adr | 2021-06-23 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/database_size_limits/_index.md | GitLab set a numeric physical-table size target and enforced it with static analysis | "we target a table size less than 100 GB for a physical table on GitLab.com (including its indexes)"; decision log: "ADR-002: Limiting new columns for tables larger than 100 GB", "ADR-003: Limiting new indexes for tables larger than 50 GB" |
| 18 | GitLab | Database Scalability (same), table census | adr | 2021-06-23 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/database_size_limits/_index.md | One table held 22.7% of the entire database | Table: ci_builds 2975 GB total, 1551 GB table, 941 GB index, 30 indexes, 22.7% of total database size; merge_request_diff_commits 1890 GB, 14.4% |
| 19 | GitLab | Database Scalability (same), operational consequences | adr | 2021-06-23 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/database_size_limits/_index.md | Table size set the deployment cadence | "in 2021, we see B-Tree creation take up to 6 hours for a single B-Tree index. This impacts our ability to deploy frequently"; "index creation time varies between 1.5 to 6 hours during busy times. This process blocks deployments as migrations are being run synchronously" |
| 20 | GitLab | CI/CD Scaling | adr | 2021-01-21 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/ci_scale/_index.md | A 2012 Rails default nearly took GitLab.com down in 2021 | "we had estimated that would run out of the capacity of the integer type to store primary keys in ci_builds table before December 2021. If it had happened without a viable workaround or an emergency plan, GitLab.com would go down." |
| 21 | GitLab | CI/CD Scaling (same), scale and rejected option | adr | 2021-01-21 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/ci_scale/_index.md | Queue-building against the large table was replaced by a purpose-built table, not by Redis | "we are creating more than 5 million builds each day on GitLab.com"; "We evaluated using Redis queuing, or a separate table that would accelerate SQL queries used to build queues. We decided to proceed with the latter." |
| 22 | GitLab | Address the Primary Key Overflow risk for the ci_builds table (epic 5657) |
source | 2021-03-19 | 2026-09-20 | https://gitlab.com/groups/gitlab-org/-/epics/5657 | The primary-key migration was tracked as a closed epic, confirming it shipped | Epic title and state: "Address the Primary Key Overflow risk for the ci_builds table", closed |
| 23 | GitLab | Draft: Add Database Sharding blueprint (MR !64115) | source | 2021-06-15 | 2026-09-20 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64115 | The sharding blueprint of 2021 was never merged; the design that superseded it is Cells | State: closed, merged_at: null. "This describes an multistep process how a GitLab Database Sharding should be approached to achieve the best and iterative outcome." |
| 24 | GitLab | Multiple Databases (development documentation) | source | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/database/multiple_databases.md | GitLab.com runs three decomposed databases, and the isolation is enforced in the application rather than by PostgreSQL | "The main databases are main, ci, and sec... On GitLab.com we are using separate main, ci, and sec databases."; "we cannot use PostgreSQL schema due to complex migration procedures. Instead we implemented the concept of application-level classification." |
| 25 | GitLab | Decompose the GitLab application database into multiple databases (work item 6168) | source | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/groups/gitlab-org/-/work_items/6168 | The decomposition programme has its own tracking item, referenced from the development docs | Referenced from Multiple Databases as "decomposed the GitLab application database into multiple databases" |
| 26 | GitLab | Cells design document | adr | created 2022-09-07, current 2026 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/_index.md | The current cell design is the fourth iteration, and its purpose is narrower than the first three: relieve one database | "Protocells is the current iteration of the Cells architecture, replacing the earlier Cells 1.0, Cells 1.5, and Cells 2.0 iterations with a new focus on permanently reducing load on the legacy cell's database." |
| 27 | GitLab | Cells design document (same), decision log | adr | 2026 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/_index.md | The cell programme carries 28 numbered ADRs, one of them marked obsolete | Decision log lists ADR-001 through ADR 028, including "ADR-007: Cells 1.0 for internal customers only (obsolete)" |
| 28 | GitLab | Cells: Goals, Glossary and Requirements | adr | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/goals.md | The programme's headroom target is stated as a requirement with a priority, not as a slogan | Requirements table: "Provides 10x headroom |
| 29 | GitLab | Cells ADR 001: Routing Technology using Cloudflare Workers | adr | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/decisions/001_routing_technology.md | Routing was put in a vendor edge runtime knowingly, with the self-managed requirement dropped | "Cloudflare Workers meets all our requirements apart from the self-managed, which is a low priority requirement."; "More vendor locking with Cloudflare, but are already heavily dependent on them." |
| 30 | GitLab | Proposal: Stateless Router using Requests Buffering (rejected) | adr | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/rejected/proposal-stateless-router-with-buffering-requests.md | Rejected designs are kept deliberately so the reasoning survives | "This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach."; "this proposal always sends requests to any of the Cells. If the requests cannot be processed, the requests will be bounced back ... This requires that request to be buffered." |
| 31 | GitLab | Proposal: Stateless Router with Routes Learning (rejected) | adr | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/cells/rejected/proposal-stateless-router-with-routes-learning.md | The competing router design required routable information in the URI and a pre-flight request | "The with learning routes proposal requires that routable information is always encoded in URI, and the router sends a pre-flight request." |
| 32 | GitLab | Gitaly design | source | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md | Gitaly exists because Git on network storage was measurably unusable, and moving to metal was considered and refused | "our P99 spikes up to 30 wall seconds, while the CPU time keeps in the realm of the 15 milliseconds. Pointing at filesystem access as the culprit."; "We considered to move to metal to fix our problems with higher performance hardware. But our users are using GitLab in the cloud so it should work great there." |
| 33 | GitLab | Transaction management in Gitaly (status: rejected) | adr | created 2023-05-30, marked rejected 2026 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/gitaly_transaction_management/_index.md | A write-ahead log, transaction manager and Raft service were built and are now being removed | "The write-ahead log, transaction manager and Raft service described here were built for the Gitaly-Cluster-without-Praefect direction. That direction has been dropped ... the design was largely built and is now being unwound, rather than turned down at review." |
| 34 | GitLab | Remove Raft, WAL and transaction code (epic 23151) | source | 2026-08-07 | 2026-09-20 | https://gitlab.com/groups/gitlab-org/-/epics/23151 | The removal is a live engineering programme with a measured size | "gitlab-org/gitaly!8895 - Remove raft, wal, transactions. Open, 365 files changed. Too large to review as one MR"; "the code is now dead weight: a large surface to maintain" |
| 35 | GitLab | Scaling Git (proposal) | adr | 2026-06-16 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/scaling-git/_index.md | The second attempt at horizontally scaling Git storage is also declared a failure, and object storage is proposed instead | "While we tried to address this issue with read distribution via Praefect, that effort has basically failed to yield a horizontally scalable cluster due to various issues."; "object storage ... is the single source of truth for repository data" |
| 36 | GitLab | Scaling Git (same), per-repository limits | adr | 2026-06-16 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/scaling-git/_index.md | A single repository's throughput ceiling is set by memory per clone, and agents are raising the load | "git-pack-objects for a gitlab-org/gitlab clone holds ~4–6 GB of anonymous memory, capping in-flight clones per node regardless of CPU."; "we see a significant rise in RPC calls that are caused by agentic workloads" |
| 37 | GitLab | GitLab.com service metrics catalogue | source | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog/services | The production topology is enumerable: 117 service definitions, of which 14 are Redis services and 4 are PostgreSQL clusters | File list contains 117 blobs including redis-cluster-cache.jsonnet, redis-sidekiq.jsonnet, patroni.jsonnet, patroni-ci.jsonnet, patroni-sec.jsonnet, patroni-registry.jsonnet, topology-service-grpc.jsonnet, zoekt.jsonnet, siphon.jsonnet, clickhouse-cloud.jsonnet |
| 38 | GitLab | Sidekiq Sharding runbook | source | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/sidekiq/sharding.md | Redis Cluster was rejected for the job queues because the key distribution is wrong for it | "Redis is not horizontally scalable and Redis Cluster is not suitable for Sidekiq since there are only a small subset of hot keys."; "The recommended setup is to have 1 queue per shard" |
| 39 | GitLab | Siphon design document | adr | 2024-11-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/siphon/_index.md | Analytical load is being moved off PostgreSQL by change-data-capture rather than by replicas | "delivers serialized CDC (change data capture) data from the PostgreSQL logical replication stream to a queueing system. From the queueing system, consumers can process the data and ingest it into other database systems (e.g. ClickHouse, Snowflake)"; goal: "Zero or minimal impact on the availability of the PostgreSQL cluster" |
| 40 | GitLab | Use Zoekt for code search | adr | 2022-12-28 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/code_search_with_zoekt/_index.md | A general-purpose search engine was kept for text and replaced for code | "While Elasticsearch has proven useful for other types of search (issues, merge requests, comments, etc.), it is not ideally suited for code search where users expect matches to be precise (no false positives) and flexible" |
| 41 | GitLab | Rails Monolith Decomposition | adr | created 2023-05-22, status proposed | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/architecture/design-documents/modular_monolith/_index.md | The monolith is 2.2 million lines of Ruby, and by 2026 the stated reason to modularise it is agents, not humans | "It has over 2.2 million lines of Ruby code"; "The monolith was built for human developers who carry years of context in their heads. Agents have no such context"; "it is the structural prerequisite for agentic development at scale" |
| 42 | GitLab | Capacity Planning for GitLab Infrastructure | casestudy | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/infrastructure-platforms/capacity-planning.md | Capacity planning is organised around which resources are horizontally scalable and which are not | "non-horizontally scalable resources (such as CPU on the primary PostgreSQL instance, for example) require much longer-term strategies for remediation and are therefore considered higher priorities in the capacity planning process" |
| 43 | GitLab | Capacity Planning (same), data classification | casestudy | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/infrastructure-platforms/capacity-planning.md | The forecasting method is public but the forecast values are not | "Tamland's output data is considered ORANGE data and must remain confidential. This includes forecast plots and information about SLO violations." |
| 44 | GitLab | Tamland (capacity forecaster) | source | repo created 2020-08-24 | 2026-09-20 | https://gitlab.com/gitlab-com/gl-infra/tamland | The forecasting tool itself is open, and runs daily as a CI job | Project description: "Tamland is a forecaster." Handbook: "It relies on Facebook's Prophet library for forecasting time series data and generates forecasts on a daily basis." |
| 45 | GitLab | Production Architecture (controlled document) | casestudy | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/infrastructure-platforms/production/architecture/_index.md | The stateful tier is deliberately outside Kubernetes | "Most of GitLab.com is deployed on Kubernetes using GitLab cloud native helm chart. There are a few exceptions for this which are mainly the datastore services like PostgresSQL, Gitaly, Redis, Elasticsearch."; "GitLab.com uses 4 Kubernetes clusters for production" |
| 46 | GitLab | Deployments and Releases | casestudy | checked 2026-09-20 | 2026-09-20 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/deployments-and-releases/_index.md | GitLab.com is deployed from the default branch several times a day, and that cadence is what long index builds threatened | "changes deploying to GitLab.com multiple times per day, and packages being released for self-managed users several times a month" |
Tier mix
| Tier | Count |
|---|---|
| postmortem | 16 |
| adr | 17 |
| source | 8 |
| casestudy | 5 |
| blog | 0 |
| paper | 0 |
| talk | 0 |
| vendor | 0 |
Distinct hosts: 1 (gitlab.com). Distinct organisations: 1 (GitLab Inc.), which is the point of a
single-company archaeology and also its main limitation. Rows 1 to 46 are 46 claims drawn from 33
distinct documents; the evidence wall in the guide carries 30 of those documents as cards, and the
three it leaves out are ones whose claim is already made by a stronger source on the wall.
What is deliberately not here
- Money. GitLab's cost-management handbook page describes a process and links to dashboards that are not public. No infrastructure spend figure appears in this guide because none was found in a reachable source.
- Forecast values. Tamland's outputs are classified confidential by GitLab's own data standard, so the capacity method is citable and the numbers are not.
- Outside voices. No independent measurement of GitLab.com appears here. Every performance and scale number is GitLab measuring itself.