GitHub kept one Rails monolith and one MySQL estate for a decade and built, in public, the tooling to operate them: gh-ost, freno, orchestrator, Trilogy, Spokes. This guide rebuilds that decade from GitHub's own postmortems, design posts and repositories, and shows why the shared database it began splitting in 2019 was still being emptied, one table cohort at a time, when agent traffic arrived in 2026.
Why GitHub, and why now: in August 2026 GitHub moved a 24-table
authentication cohort off mysql1, the cluster whose partitioning began in 2019,
and served its first production MySQL primary from Azure. The decade can now be read end to end.
mysql1 by moving one 24-table auth cohort, seven years after partitioning beganThe problem this dig follows, stated without the company's name: how do you keep a product available when most of its features share one relational store, the store lives in buildings you own, and demand grows faster than you can split either? GitHub is the cleanest public case because it chose, again and again, not to rewrite. It kept the Rails monolith (upgraded in place, then bumped to Rails main every week), kept MySQL, and spent its engineering on the layer around them.
The partitioning of mysql1 was announced as a success in 2021, with load on the
receiving hosts down 50% (GitHub, 2021).
Five years later the cluster still held an authentication cohort worth about a million
queries per second of replica reads (GitHub, Aug 2026),
and a February 2026 outage had overloaded the "core database cluster that handles
authentication and user management" (GitHub, Mar 2026).
Partitioning removed the easy tables first. The tables left behind were the oldest, the
most joined and the hottest, which is to say the ones every request touches.
Scope. This guide covers the github.com runtime: the monolith, its relational tier, Git storage, deploys and the hosting move. It does not cover GitHub Actions runners, Copilot's model serving, Enterprise Server packaging, or organisational changes beyond the 2025 move into Microsoft's CoreAI group as it bears on the Azure decision.
The shape that held from about 2017 to 2025, reconstructed from GitHub's posts. Each box is attributable to a named source. In 2026 two of them began to change: the primaries started moving to Azure and Spokes is being replaced.
The request path stayed boring on purpose. In 2017 GitHub moved the web and API tier from Unicorn processes supervised by God, deployed by Capistrano over SSH, onto Kubernetes clusters in its "metal cloud" (GitHub, 2017). The application did not change shape. Rails was upgraded in place by dual-booting the app against two lockfiles, which took about eighteen months for 3.2 to 5.2 and avoided a long-lived branch (GitHub, 2018); by 2023 a bot opened a pull request every Monday to move the monolith to the latest Rails main (GitHub, 2023). Even the database driver was owned: Trilogy, GitHub's own implementation of the MySQL wire protocol, served the monolith from 2015 and was open-sourced in 2022 (GitHub, 2022).
The change path is where the engineering went. gh-ost (2016) changes schemas by tailing the binary log instead of installing triggers. freno (2017) is a throttler that batch writers consult before writing, so replication lag is rationed rather than discovered. In 2018 GitHub replaced VIP and DNS failover with orchestrator running raft across data centres, publishing the current primary to Consul's key-value store, from which consul-template rewrites identical HAProxy writer pools on every GLB node (GitHub, 2018). The design goal was explicit: data-centre independence and no "unreliable cooperative workflows". Four months later it produced the decade's longest outage.
Held "a large portion of the data used by GitHub.com's core features": users, repositories, issues, pull requests. Every postmortem in 2020 and 2022 names it.
Three copies per repository in 2016, commit on two acknowledgements; five full copies by default by 2026, with a three-phase quorum commit on ref updates. Being replaced.
One inference worth flagging. GitHub has not published a design document for the
August 2026 primary moves, but its own gh-ost repository shows the tool being taught a new
job: a --move-tables mode, first committed on 15 May 2026, whose flag text reads
"a special mode that allows you to move tables between database clusters"
(gh-ost, feature-move-tables).
As of 18 October 2026 the branch is not in master. Neither the report nor the
branch says the two are connected, but a binlog-tailing copier with a postponed cut-over is
exactly the tool you would want for moving a 24-table cohort off a hot cluster.
Six forks GitHub took in public. Three of them it later reversed or narrowed, which is what makes the "flips when" column checkable rather than speculative.
| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Online schema change | Binlog-tailing copy, gh-ost (2016) | Trigger-based tools | Triggers are interpreted, share the query's transaction and contend for locks; concurrent migrations were unsafe | MySQL 8 instant DDL covers the change, or the table is small and cold | GitHub, 2016 |
| Replication lag control | Cooperative throttler, freno (2017) | Discovering lag from stale reads | Batch jobs and migrations are the lag source and can wait | Writers you do not control (third-party clients, agents) generate the load | GitHub, 2017 |
| Primary failover | orchestrator/raft + Consul + GLB (2018) | VIP and DNS | Split brain and DNS caching | Cross-region promotion: narrowed after Oct 2018 | InfoQ on GitHub, 2018 |
| Deploy exposure | 2% then 20% canary, 5-minute automated gates (2021) | Single 2% canary | Problems surfaced only after full rollout | The fault is in a dependency's config, not the deployed code | GitHub, 2021 |
| Monolith runtime | Keep Rails, track main weekly | Rewrite into services | Upgrade cost amortised; product kept shipping | Hot paths outgrow Ruby: Ruby-to-Go moves are reported in 2026 (secondary sources only) | GitHub, 2023 |
| Hosting | Own metal (2016 to 2025), Azure from 2025, AWS capacity added 2026 | Public cloud, until 2025 | Control and cost; then "limited opportunities to bring more capacity online in the North Virginia region" | Physical capacity cannot be added where the primaries live | The New Stack, 2025 |
The hosting row is the one an architect should read twice. GitHub's CTO told staff in October 2025 that the move to Azure was "existential", that the plan was eighteen months with a six-month buffer, and that earlier Azure migrations had been, in The New Stack's words, "halting and sometimes failed" (The New Stack, 2025). The reason given was not price or features. It was that the region holding the primaries could not take more servers. A decade of tooling had made a single-region relational core operable; it had not made it portable, and portability became the requirement the moment the building ran out of power and floor space.
Eight published incidents fall into three classes. The first class names
mysql1 outright; the second is the change path to the database; the third is
growth outrunning capacity, made worse by retries.
mysql1 primary; ProxySQL, its descriptor limit silently downgraded to 65,536, could not keep up; later an application change to query patterns "rapidly increased load on the master".mysql1; configuration audit of the proxy tier.mysql1 as a single point of contention.Read across the classes and one pattern stands out. Every Class A incident is a local change (a query, a proxy limit, a cache TTL) that became global because the store underneath was shared. Every Class B incident is a control-plane action that did exactly what it was configured to do. Neither class is solved by more database hardware, and neither was solved by the tooling ring of Figure 2, because the ring governs how the store changes, not who shares it.
Each row is dated. "Measured" means GitHub reported it about its own production; "internal" means a benchmark; "secondary" means only press or summaries carry it.
| Metric | Value | Kind | Context | As of | Source |
|---|---|---|---|---|---|
| Degraded service, Oct 2018 partition | 24 h 11 m | Measured | Triggered by 43 s of lost connectivity | 2018-10 | GitHub |
| West Coast writes before diagnosis | ~40 min | Secondary | Divergent writes on both coasts | 2018-10 | InfoQ |
| mysql1 incidents, February 2020 | 8 h 14 m | Measured | Four events | 2020-02 | GitHub |
| Load drop on hosts that took mysql1 data | 50% | Measured | After 2019 to 2021 partitioning | 2021-09 | GitHub |
| MySQL fleet | 1,200+ hosts, 300+ TB, 50+ clusters | Measured | Azure VMs plus bare metal | 2023 | GitHub |
| MySQL query rate | 5.5M QPS | Measured | Fleet-wide, at MySQL 8 upgrade | 2023 | GitHub |
| Canary stages | 2%, then 20% | Measured | 5-minute automated gates | 2021-01 | GitHub |
| Monolith dev environment start | 45 min to 10 s | Measured | Codespaces with prebuilds | 2021-08 | GitHub |
| Full unavailability, Aug 2024 | 36 min | Measured | 23:02 to 23:38 UTC, database routing | 2024-08 | GitHub |
| Git replicas per repository | 3 (2016), 5 default (2026) | Measured | Quorum commit on writes | 2016, 2026 | GitHub |
| Git activity | 473.3B events / month | Measured | More than doubled since Sep 2025 | 2026-08 | GitHub |
| Write throughput, new Git storage | up to 35× | Internal | Benchmark, not production | 2026-10 | GitHub |
| Monthly commits | 1.4B to 2.9B | Measured | April to August 2026 | 2026-08 | GitHub |
| Platform load on Azure | 12% to 58% | Measured | May to August 2026 | 2026-08 | GitHub |
| Replica reads removed from mysql1 | ~1M QPS | Measured | One 24-table auth cohort moved | 2026-08 | GitHub |
| Query-hygiene savings | 120k QPS; 59,000 DB-seconds / hour | Measured | Same month | 2026-08 | GitHub |
| Capacity target | 10× (Oct 2025) to 30× (Feb 2026) | Secondary | Summaries of the April 2026 CTO post | 2026-04 | GitHub (via summaries) |
Two derived figures are mine. First, the 1M QPS removed from mysql1 replicas
in 2026 is, on its own, about 18% of the entire 2023 fleet's 5.5M QPS (1.0 / 5.5); the
two numbers are three years apart and one is replica-only, so treat it as an order of
magnitude, not a ratio. Second, the "seven years" is 2019 (partitioning starts) to 2026
(auth cohort moves). GitHub has never published what fraction of mysql1
remains, its current QPS, or its table count; those are the numbers an architect would
most want and they are open.
GitHub's decade is not a story of bad choices. Keeping the monolith was right by any measure GitHub published: weekly Rails main, a 10-second dev environment, no rewrite freeze. Keeping MySQL and building gh-ost, freno and orchestrator around it was right too; those tools are still the reference implementations other companies adopt. What the record shows is narrower and more transferable.
"GitHub is constrained on data server capacity with limited opportunities to bring more capacity online in the North Virginia region." GitHub CTO memo, October 2025, as reported by The New Stack
First, decomposition of a shared store has an exit metric or it has no end. Partitioning by schema domain moves cold, self-contained domains first because they are cheap to move. That is rational per step and wrong in aggregate: the residue is the identity, permission and user tables that everything joins, which carry the peak load and the widest blast radius. GitHub's 2021 post measured success as load removed (50%). The 2026 incidents measured what remained. Write the target as "no feature's availability depends on the shared cluster", track the residue, and move the hot cohort while you still have capacity headroom to run a dual-write cut-over, not after.
Second, the tools that operate a store are part of the store's architecture, and they age like people. orchestrator's maintainer stopped reviewing pull requests in February 2023 and the repository was archived in February 2025 with a pointer to Percona's fork (git history). gh-ost fell from 667 commits in 2016 to 20 in 2023, then revived to 87 in 2026 as GitHub needed resumable migrations (October 2025) and a table-mover (May 2026). If your failover or migration path depends on a tool with a bus factor of one, that dependency belongs on the risk register next to the database.
Third, owning the building is a capacity bet with a lead time measured in years. GitHub ran its relational primaries in owned data centres while public-cloud VMs were already in its fleet by 2023. The move only became urgent when the region could not grow, and then it had to happen during the steepest demand curve in the company's history, with feature work deprioritised and AWS capacity added as well. A system whose primaries cannot move without an eighteen-month programme has a portability debt; pay it while demand is flat.
List the tables every request touches (sessions, users, permissions, feature flags, settings). For each, write down which cluster holds it, what else shares that cluster, and how long it would take to move it to its own primary in another region. If the answer for the identity tables is "the shared cluster" and "we have not planned it", that is the outage GitHub had in February 2026, waiting for your version of a cache TTL change.
Every source behind this page, graded. GitHub publishes no ADRs; three
posts that name a rejected design are graded as decision records. Page retrieval was by
search excerpt; repositories were read first-hand with git. The full ledger with quotes is
in sources.md.
A 43-second partition let orchestrator promote West Coast primaries; both coasts held writes the other lacked, so GitHub failed forward and ran degraded for 24 hours 11 minutes.
Independent account: about 40 minutes of West Coast writes before diagnosis; remediation includes no cross-region primary promotion and an active/active/active design.
Four events, 8 h 14 m, all from contention in mysql1: a misrouted heavy query, a ProxySQL descriptor limit of 65,536, an application query-pattern change.
Six months after the partitioning post, peak-load contention in mysql1 exhausted the database proxy's connections and failed writes across Git, PRs and Actions.
A database configuration change failed routing-layer health checks fleet-wide; the read-only endpoint vanished and github.com was down for all users for 36 minutes.
A user-settings cache TTL cut from 12 h to 2 h, plus Monday peak and a model release, overloaded the core authentication and user-management cluster on 9 February.
Agentic workflows accelerated sharply from late December 2025; a merge-queue regression hit 230 repositories and Elasticsearch overloaded in the same week.
A Central US component failed to scale at a new peak; a client retry loop slowed recovery. Monthly commits doubled from 1.4B to 2.9B since April; Azure carried 58% of load.
First production MySQL primary served from Azure on 11 August; a 24-table authentication cohort moved off mysql1, removing about 1M QPS of replica reads.
Rejects trigger-based migration: triggers are interpreted, share the query's transaction and contend for a second table's locks; GitHub would not run them concurrently.
Replaces VIP and DNS failover with orchestrator/raft across DCs, Consul KV and GLB/HAProxy writer pools, to remove split brain and DNS caching.
mysql1 split by schema domain from 2019; receiving hosts' load down 50%; cross-domain transactions and polymorphic tables named as the hard cases.
Spokes keeps five full replicas with a quorum commit, so read replicas slow writes; the replacement stores repositories in Azure Blob Storage with cache-like workers. 35x write throughput is an internal benchmark.
1,200+ hosts, 300+ TB, 50+ clusters, 5.5M QPS, a mix of Azure VMs and bare metal, upgraded over a year without SLO impact.
Git replicated at the application layer: three copies on three servers, writes committed on two acknowledgements, automatic re-replication on server loss.
Web and API moved from Unicorn, God, Puppet and Capistrano onto Kubernetes on GitHub's own metal, without changing the application's shape.
A cooperative throttler that batch writers consult before writing; it observes lag and never proxies traffic.
Dual-booting against two lockfiles let the monolith move framework versions over about eighteen months without a long-lived branch.
A single 2% canary missed problems; GitHub added a 20% stage and automated 5-minute gates.
The monolith's development environment went from a 45-minute bootstrap to 10 seconds with shallow clones, nightly images and prebuild pools.
GitHub's own MySQL wire-protocol client, serving the monolith since 2015 and open-sourced in 2022, with no libmysqlclient dependency.
Every Monday a bot bumps the monolith to the latest Rails main; it ships the next day.
CTO memo: capacity-constrained in North Virginia, 18 months plus a 6-month buffer to leave owned DCs, earlier Azure moves halting.
New Copilot Pro, Pro+ and Student sign-ups paused because agentic sessions used far more resources than plans assumed.
Independent report on the Git storage rebuild: push latency bound by the slowest replica; no completion date given.
The author of both tools stepped back from reviews and releases for an undefined period.
From 15 May 2026, a --move-tables mode "to move tables between database clusters", with a mandatory postponed cut-over. Not merged to master as of 18 October 2026.
Merged 16 October 2025: long migrations can resume from a checkpoint instead of restarting.
Commits per year fell from 667 (2016) to 20 (2023) and recovered to 87 in 2026, tracking GitHub's own need for the tool.
Maintainer notice in February 2023 ("I will not be reviewing pull requests"); archived on 1 February 2025 with a pointer to Percona's fork.
GitHub replaced git-receive-pack on the push path: "git-receive-pack does more than we need it to"; no outside contributions accepted.
Survey of public incidents: retries are the most common sustaining effect, behind over half of the sustained overloads catalogued.
Safe online schema change decomposes into intermediate states such that no server is more than one schema version behind.
GitHub presented its Kubernetes move as a conference keynote. Schedule listing only; the recording was not retrievable, so no timestamped claim is cited.
Orchestrator (50-minute hands-on session) and gh-ost (Jonah Berquist) presented as GitHub production tools. Session listing only.
Six rungs that reproduce GitHub's shape and its two dominant failures on a laptop. The line from toy to real is between rungs 3 and 4.
Run a small web app against a MySQL primary with two replicas (Docker Compose). Give it
a users table every request reads and three feature tables that join it.
Done when: a load generator drives all four features and you can see per-table QPS. Teaches: which tables are the shared core before anyone tells you.
Add a column to the busiest feature table with gh-ost while the load runs, then repeat with a trigger-based tool and compare primary latency.
Done when: you have p99 graphs for both runs. Teaches: why GitHub rejected triggers in 2016.
Cut a cache TTL in front of users from hours to minutes and double the load.
Watch every feature degrade, including ones that never touch the cache.
Done when: an unrelated feature's error rate rises. Teaches: the February 2026 mechanism.
Move two feature domains to their own primaries. Report load removed from the shared cluster and, separately, the share of peak QPS that remains on it.
Done when: both numbers are on one dashboard. Teaches: why "50% removed" and "still the blast radius" are both true.
Copy users to a new primary by tailing the binlog, hold the cut-over behind a
flag file, then flip it under load and record write unavailability.
Done when: the cut-over pause is measured in milliseconds to seconds and no write is lost. Teaches: what the gh-ost move-tables branch is for.
Put a replica behind 70 ms of injected latency, run orchestrator (or Percona's fork), partition the primary for 45 seconds and let it promote. Then forbid cross-boundary promotion and repeat.
Done when: you have divergent GTID sets in run one and none in run two. Teaches: October 2018, and why geography belongs in failover policy.
The queries and git commands that found the material above.
"GitHub availability report" mysql1github.blog post-incident analysis orchestrator partition"Addressing GitHub's recent availability issues""The August 17 outage, and the work ahead""Partitioning GitHub's relational databases" schema domains"MySQL High Availability at GitHub" VIP DNS"Building Git infrastructure for agent-scale development" Spokesgh-ost triggerless "not prepared to run multiple concurrent"git ls-remote https://github.com/github/gh-ost | grep movegit log --format='%ad' --date=format:%Y | sort | uniq -cgit merge-base --is-ancestor <branch> origin/masterGitHub Azure migration memo North Virginia capacityGitHub AWS capacity 2026 Copilot agents"ran a production MySQL primary from Azure"