Evidence ledger
35 sources
Checked 18 Oct 2026
Evidence ledger
One row per claim in The cluster nobody finished emptying: ten years of GitHub: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Dig: The cluster nobody finished emptying: GitHub's architecture, 2016 to 2026. Research date: 2026-10-18. All rows checked 2026-10-18.
Retrieval limitation, stated up front. This session's egress policy refused direct HTTP
fetches: curl to github.blog was answered with a 403 on CONNECT by the proxy, and WebFetch
could not resolve DNS. Two channels worked. The first was search-engine retrieval, which returns
excerpts and quoted fragments of a page rather than the whole document. The second was anonymous
git reads of public repositories (github/gh-ost, openark/orchestrator via the
github/orchestrator redirect, github/freno, github/glb-director, github/trilogy,
github/spokes-receive-pack) through the session's git proxy. Rows graded source were read
first-hand from git: commit hashes, authors, dates, messages, branch and pull-request refs, and
ancestry checks against the default branch. Every other row rests on retrieved excerpts of the
primary page; quotes are reproduced only where the retrieval returned them as quotation, otherwise
the row says "(retrieved summary)". The HTTP link check in verify.mjs could not reach these
hosts from this session, so links are recorded as indexed, not re-fetched.
Grading notes. GitHub publishes no formal ADRs. Three posts are graded adr because each
names the design it rejected and why: the 2016 gh-ost post (trigger-based migration), the 2018
MySQL HA post (VIP and DNS failover) and the 2026 Git infrastructure post (Spokes' replicated
local disks). Monthly availability reports and CTO incident posts are graded postmortem. The
2023 MySQL 8 post is graded casestudy because it is the only fleet-wide figure set. News
reporting (The New Stack, The Register, InfoQ) is graded blog and is used only for what
GitHub's own pages do not carry, flagged where it is the sole source.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-10-18 | https://github.blog/2018-10-30-oct21-post-incident-analysis/ | Degraded service for 24h 11m after an optical-equipment maintenance cut the East Coast hub from the primary DC | "degraded service for 24 hours and 11 minutes"; "loss of connectivity between our US East Coast network hub and our primary US East Coast data center" |
| 2 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-10-18 | https://github.blog/2018-10-30-oct21-post-incident-analysis/ | Orchestrator promoted West Coast primaries; both sites held writes the other lacked, so failback was unsafe | "The database clusters in both data centers now contained writes that were not present in the other data center" (retrieved excerpt) |
| 3 | InfoQ | GitHub incident analysis | blog | 2018-11 | 2026-10-18 | https://www.infoq.com/news/2018/11/github-incident-analysis | 43 seconds of lost connectivity; West Coast took writes for nearly 40 minutes before the problem was understood | "during the 43 seconds of lost connectivity, the US East Coast data center accepted writes that were not replicated"; "nearly 40 minutes" (retrieved summary) |
| 4 | InfoQ | GitHub incident analysis | blog | 2018-11 | 2026-10-18 | https://www.infoq.com/news/2018/11/github-incident-analysis | Remediations included preventing cross-region primary promotion and an active/active/active data-centre design | "preventing Orchestrator from promoting database primaries across region boundaries"; "active/active/active design" (retrieved excerpt) |
| 5 | GitHub | February service disruptions post-incident analysis | postmortem | 2020-03-26 | 2026-10-18 | https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis | Four events, 8h 14m total, from resource contention in mysql1 | "a total of eight hours and 14 minutes over four distinct events"; "resource contention in our mysql1 database cluster" |
| 6 | GitHub | February service disruptions post-incident analysis | postmortem | 2020-03-26 | 2026-10-18 | https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis | A query meant for replicas hit the mysql1 primary; ProxySQL's descriptor limit had been downgraded to 65536; a query-pattern change slowed the cluster for all dependents | "application logic changes to database query patterns rapidly increased load on the master of our mysql1 database cluster" (retrieved via DevClass/InfoQ quotation) |
| 7 | GitHub | An update on recent service disruptions | postmortem | 2022-03-23 | 2026-10-18 | https://github.blog/2022-03-23-an-update-on-recent-service-disruptions | March 2022 incidents again from mysql1 contention; the database proxy hit its connection limit and writes failed across services | "resource contention in our mysql1 cluster, which impacted the performance of a large number of our services and features during periods of peak load" |
| 8 | GitHub | Partitioning GitHub's relational databases to handle scale | adr | 2021-09-27 | 2026-10-18 | https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale | Partitioning of mysql1 began in 2019, by schema domain | mysql1 "housed a large portion of the data used by GitHub.com's core features"; work began in 2019 (retrieved summary) |
| 9 | GitHub | Partitioning GitHub's relational databases to handle scale | adr | 2021-09-27 | 2026-10-18 | https://github.blog/2021-09-27-partitioning-githubs-relational-databases-scale | Load on the hosts that took over mysql1 data fell 50%; cross-domain transactions and polymorphic tables were the hard cases | "reduced load by 50%"; transactions "can no longer guarantee consistency" once tables move to separate databases (retrieved summary) |
| 10 | GitHub | gh-ost: GitHub's online schema migration tool for MySQL | adr | 2016-08-01 | 2026-10-18 | https://github.blog/2016-08-01-gh-ost-github-s-online-migration-tool-for-mysql | Trigger-based migration rejected: interpreted triggers add overhead and lock contention, so concurrent migrations were unsafe | "Given the above trigger overhead, we are not prepared to run multiple concurrent trigger-based migrations." |
| 11 | GitHub | MySQL High Availability at GitHub | adr | 2018-06-20 | 2026-10-18 | https://github.blog/2018-06-20-mysql-high-availability-at-github/ | VIP and DNS failover replaced by orchestrator/raft across DCs, Consul KV, GLB/HAProxy writer pools, anycast | Stated goals: DC independence, tolerate DC failure, remove "unreliable cooperative workflows", "lossless failovers" (retrieved summary) |
| 12 | GitHub | Upgrading GitHub.com to MySQL 8.0 | casestudy | 2023-12-07 | 2026-10-18 | https://github.blog/2023-12-07-upgrading-github-com-to-mysql-8-0/ | 1,200+ hosts, 300+ TB, 50+ clusters, 5.5M QPS; fleet already mixed Azure VMs and bare metal | "1200+ MySQL hosts"; "300+ TB"; "50+ database clusters"; "5.5 million queries per second"; "Azure Virtual Machines and bare metal hosts in our data center" |
| 13 | GitHub | Kubernetes at GitHub | blog | 2017-08-16 | 2026-10-18 | https://github.blog/2017-08-16-kubernetes-at-github/ | Web and API moved from Unicorn/God/Puppet/Capistrano to Kubernetes on GitHub's own metal | "all web and API requests are served by containers running in Kubernetes clusters deployed on our metal cloud" |
| 14 | GitHub | Introducing DGit | blog | 2016-04-05 | 2026-10-18 | https://github.blog/2016-04-05-introducing-dgit | Git replicated at the application layer, three copies, commit on two acknowledgements | "three copies of every repository, on three different servers"; write committed after at least two replicas confirm (retrieved summary) |
| 15 | GitHub | Building Git infrastructure for agent-scale development | adr | 2026-10-06 | 2026-10-18 | https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/ | Spokes (five full replicas by default, 3PC quorum on ref updates) is being replaced by Azure Blob Storage plus cache-like compute workers | "Adding read replicas slows writes, and losing quorum halts writes entirely"; worker failure "comparable to a cache miss" |
| 16 | GitHub | Building Git infrastructure for agent-scale development | adr | 2026-10-06 | 2026-10-18 | https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/ | Git activity more than doubled Sep 2025 to Aug 2026 to 473.3B events a month; up to 35x write throughput in internal benchmarks | "473.3B events per month"; "up to 35x higher write throughput" (internal benchmark) |
| 17 | The Register | So long, Spokes | blog | 2026-10-09 | 2026-10-18 | https://www.theregister.com/devops/2026/10/09/so-long-spokes-github-rewrites-storage-to-restore-reliability-just-in-time-for-agentic-hordes/5302148 | Independent report of the Spokes rebuild; push latency bound by slowest replica; no completion date | "a push takes as long as its slowest replica" (retrieved summary) |
| 18 | GitHub | Improving how we deploy GitHub | blog | 2021-01-25 | 2026-10-18 | https://github.blog/2021-01-25-improving-how-we-deploy-github/ | Canary widened from one 2% stage to 2% then 20%, separated by automated 5-minute timers | "The first is at 2%..."; "A second canary stage was introduced at 20%" |
| 19 | GitHub | Upgrading GitHub from Rails 3.2 to 5.2 | blog | 2018-09-28 | 2026-10-18 | https://github.blog/2018-09-28-upgrading-github-from-rails-3-2-to-5-2 | Monolith upgraded by dual-booting (Gemfile_next.lock), about 18 months, no long-running branch | Rails 5.2.1 on 2018-08-15 after "a year and a half" (retrieved summary) |
| 20 | GitHub | Building GitHub with Ruby and Rails | blog | 2023-04-06 | 2026-10-18 | https://github.blog/2023-04-06-building-github-with-ruby-and-rails/ | Weekly automated bump of the monolith to Rails main | Monday PR bumps Rails to latest main; shipped next day (retrieved summary) |
| 21 | GitHub | Introducing Trilogy | blog | 2022-08-25 | 2026-10-18 | https://github.blog/2022-08-25-introducing-trilogy-a-new-database-adapter-for-ruby-on-rails | Own MySQL protocol client, serving the monolith since 2015, open-sourced 2022 | Trilogy has served "GitHub's Rails monolith" query traffic "since 2015" (retrieved summary) |
| 22 | GitHub | Mitigating replication lag and reducing read load with freno | blog | 2017-10-13 | 2026-10-18 | https://github.blog/2017-10-13-mitigating-replication-lag-and-reducing-read-load-with-freno/ | Cooperative throttler: writers ask before writing; it never proxies | Throttler that clients consult; triggers are batch jobs and schema changes (retrieved summary) |
| 23 | GitHub | GitHub's Engineering Team has moved to Codespaces | blog | 2021-08-11 | 2026-10-18 | https://github.blog/2021-08-11-githubs-engineering-team-moved-codespaces/ | Monolith dev environment from 45-minute bootstrap to 10 seconds | "a 45-minute bootstrap"; "in 10 seconds" |
| 24 | GitHub | GitHub Availability Report: August 2024 | postmortem | 2024-09 | 2026-10-18 | https://github.blog/enterprise-software/availability-reports/github-availability-report-august-2024/ | A database config change made hosts fail routing-layer health checks; read-only endpoint unreachable; all users down 23:02-23:38 UTC | "faulty configuration change"; routing service "could not find healthy databases" (status page wording) |
| 25 | GitHub | Addressing GitHub's recent availability issues | postmortem | 2026-03-11 | 2026-10-18 | https://github.blog/news-insights/company-news/addressing-githubs-recent-availability-issues-2/ | Feb 9 2026: a user-settings cache TTL cut from 12h to 2h on Feb 7 plus Monday peak overloaded the core auth and user-management database cluster | TTL "from 12 hours to 2 hours"; "took much longer to understand why the read load kept increasing" (retrieved excerpt) |
| 26 | GitHub | An update on GitHub availability | postmortem | 2026-04-28 | 2026-10-18 | https://github.blog/news-insights/company-news/an-update-on-github-availability/ | Agentic workflows "accelerated sharply" from late Dec 2025; April 23 merge-queue regression (230 repos) and April 27 Elasticsearch overload | "Since the second half of December 2025, agentic development workflows have accelerated sharply" (quoted in coverage); 10x to 30x capacity target is reported by secondary summaries only |
| 27 | GitHub | The August 17 outage, and the work ahead | postmortem | 2026-08-20 | 2026-10-18 | https://github.blog/news-insights/company-news/the-august-17-outage-and-the-work-ahead/ | A Central US component failed to scale at a new traffic peak; auth failures spread; client retry loop added load during recovery; no code or config change involved | Monthly commits "1.4 billion to 2.9 billion since April"; Azure "roughly 58%" of platform load, up from 12% in May |
| 28 | GitHub | GitHub availability report: August 2026 | postmortem | 2026-09-10 | 2026-10-18 | https://github.blog/news-insights/company-news/github-availability-report-august-2026/ | First production MySQL primary served from Azure on Aug 11, two more on Aug 27 | "ran a production MySQL primary from Azure for the first time on August 11" |
| 29 | GitHub | GitHub availability report: August 2026 | postmortem | 2026-09-10 | 2026-10-18 | https://github.blog/news-insights/company-news/github-availability-report-august-2026/ | A 24-table authentication cohort moved off mysql1, removing about 1M QPS from its replicas; query hygiene removed 120k QPS | "24-table authentication cohort"; "about one million queries per second"; "120,000 queries per second"; "59,000 seconds of wasted database work per hour" (retrieved summary) |
| 30 | The New Stack | GitHub will prioritize migrating to Azure over feature development | blog | 2025-10 | 2026-10-18 | https://thenewstack.io/github-will-prioritize-migrating-to-azure-over-feature-development/ | CTO memo: capacity-constrained in North Virginia; 18 months plus 6 buffer to leave own DCs; earlier Azure moves "halting" | "constrained on data server capacity with limited opportunities to bring more capacity online in the North Virginia region"; "existential" |
| 31 | The Register | Microsoft's GitHub suspends Copilot account sign-ups | blog | 2026-04-20 | 2026-10-18 | https://www.theregister.com/2026/04/20/microsofts_github_grounds_copilot_account/ | Demand was rationed at the product layer: new Copilot Pro, Pro+ and Student sign-ups paused | "paused signups for GitHub Copilot Pro, Pro+, and Student plans" |
| 32 | openark / GitHub | gh-ost repository, feature-move-tables branch | source | 2026-05-15 to 2026-06-26 | 2026-10-18 | https://github.com/github/gh-ost/tree/feature-move-tables | gh-ost gained a mode to move tables between database clusters; not in master as of 2026-10-18 | Flag text: "This is a special mode that allows you to move tables between database clusters"; first commit 8c45ee1 (2026-05-15); branch head a6d682a (2026-06-26) not an ancestor of master |
| 33 | GitHub | gh-ost PR #1595, Resume from checkpoint | source | 2025-10-16 | 2026-10-18 | https://github.com/github/gh-ost/pull/1595 | Migrations became resumable in October 2025 | master commit 1557a95 "Resume from checkpoint (#1595)", 2025-10-16 |
| 34 | GitHub | gh-ost commit history | source | 2016-2026 | 2026-10-18 | https://github.com/github/gh-ost | Activity fell from 667 commits (2016) to 20 (2023) and recovered to 87 in 2026 to 2026-10-18 | git log counts per year: 667, 325, 214, 99, 105, 78, 148, 20, 63, 36, 87 |
| 35 | openark | orchestrator repository | source | 2023-02-28, 2025-02-01 | 2026-10-18 | https://github.com/openark/orchestrator | Maintainer stopped reviewing PRs Feb 2023; repository archived Feb 2025 pointing to Percona's fork | 2f032d8a: "I am taking time off from maintaining this repo. I will not be reviewing pull requests."; 730db91f: "This repository has been archived." |
| 36 | GitHub | spokes-receive-pack | source | 2022-09-21 | 2026-10-18 | https://github.com/github/spokes-receive-pack | GitHub replaced git-receive-pack on the push path with its own reduced implementation | "git-receive-pack does more than we need it to, spokes-receive-pack implements the bits of it that we still need." |
| 37 | Shlomi Noach | Reducing my OSS involvement, and how it affects orchestrator & gh-ost | blog | 2023-02 | 2026-10-18 | https://code.openark.org/blog/mysql/reducing-my-oss-involvement-and-how-it-affects-orchestrator-gh-ost | Both tools depended on one author's attention | Will not respond to issues, review pull requests or produce releases for an undefined period (retrieved summary); URL taken from commit 2f032d8a |
| 38 | Penn State, UNH, Twitter | Metastable Failures in the Wild (OSDI '22) | paper | 2022-07 | 2026-10-18 | https://www.usenix.org/conference/osdi22/presentation/huang-lexiang | Retries are the dominant sustaining effect in metastable failures | Load increase due to retries "over 50%" of sustaining effects (slides) |
| 39 | Online, Asynchronous Schema Change in F1 (PVLDB 6(11)) | paper | 2013 | 2026-10-18 | https://research.google/pubs/online-asynchronous-schema-change-in-f1/ | Safe online schema change decomposes into states no server is more than one version behind | "avoid corrupting the database so long as all servers are no more than one schema version behind at any time" | |
| 40 | CNCF / GitHub | Kubernetes at GitHub (KubeCon NA 2017 keynote, Jesse Newland) | talk | 2017-12-08 | 2026-10-18 | https://kccncna17.sched.com/jessenewland | GitHub presented its Kubernetes move as a keynote; schedule listing only, recording not retrieved | Listing: "Kubernetes at GitHub", Principal Site Reliability Engineer, GitHub |
| 41 | openark / GitHub | Practical orchestrator, BoF, GitHub and other talks at Percona Live 2017 | talk | 2017-04-19 | 2026-10-18 | https://code.openark.org/blog/mysql/practical-orchestrator-bof-github-and-other-talks-at-percona-live-2017 | Orchestrator and gh-ost were presented as GitHub's production tools at Percona Live 2017; session listing only | 50-minute hands-on orchestrator session; gh-ost session by Jonah Berquist on 25 April |
Evidence gaps, recorded as findings. No GitHub source describes mysql1's remaining table count or QPS before August 2026; the "seven years" framing is derived (2019 start in row 8, 2026 move in row 29). The link between the Feb 9 2026 cluster (row 25: "authentication and user management") and mysql1 (row 29: "authentication cohort") is inferred from wording, not stated. No talk timestamps could be cited because no recording was retrievable; the two talk rows are schedule-level evidence. No closed-unmerged pull request could be confirmed without the GitHub API; the move-tables work is an unmerged feature branch, which is a weaker signal.