Reliability & Operations 18 Oct 2026 29 min read

The cluster nobody finished emptying: ten years of GitHub

How GitHub's monolith, MySQL estate, Git storage and hosting changed between 2016 and 2026, reconstructed from its postmortems, design posts and repositories: the tooling ring built around one shared database (gh-ost, freno, orchestrator, Trilogy), the 2019 partitioning of mysql1 that stopped short of the hottest tables, and the 2025-2026 exit from owned data centres under agent-driven load.

A single-company dig built from eight GitHub postmortems and availability reports, four design posts that name a rejected option, and first-hand git reads of gh-ost, orchestrator and spokes-receive-pack. It gives an architect the decade in four phases, six decisions with the condition that flips each, eight incidents in three failure classes with transferable rules, a dated numbers table, and a six-rung ladder that reproduces both dominant failures on a laptop.

The finding that surprised me

The mysql1 partitioning GitHub declared a success in 2021 (load down 50%) still left an authentication cohort worth about a million queries per second of replica reads on the shared cluster in 2026, and the February 2026 outage overloaded exactly that kind of table: the cheap domains moved first and the tables every request joins stayed behind.

What you get out of it

  • Every Class A incident (February 2020, March 2022, February 2026) is a local change (a query, a proxy limit, a cache TTL) made global by the shared mysql1 cluster.
  • The October 2018 outage was automation doing what it was configured to do: orchestrator promoted a primary the application could not use at cross-country latency, turning 43 seconds into 24 hours 11 minutes.
  • gh-ost, the schema-change tool, grew a --move-tables mode in May 2026 'to move tables between database clusters'; it is still an unmerged branch in October 2026.
  • orchestrator's maintainer stopped reviewing PRs in February 2023 and the repository was archived in February 2025; the tools that operate a store age with the people who wrote them.
  • The Azure move was forced by physical capacity in North Virginia, not price; Spokes is being replaced by blob storage because added replicas slowed writes as agent traffic doubled Git activity.

Scope

Why this, now. In August 2026 GitHub moved a 24-table authentication cohort off mysql1 and served its first production MySQL primary from Azure, and in October 2026 it published the replacement for Spokes, so the decade that began with gh-ost and DGit can be read end to end.

What it does not cover. GitHub Actions runners, Copilot model serving, GitHub Enterprise Server packaging and organisational history; mysql1's remaining size and QPS, the Azure cut-over mechanics and any cross-cloud latency figures are marked open because GitHub has not published them.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.