Evidence ledger
20 sources
Checked 04 Oct 2026
Evidence ledger
One row per claim in Keeping the main branch green: thirteen years of merge queues, read from the repositories that ran them: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how large projects stopped trusting "tested at review time" and built merge queues: speculative batching, human-curated rollups, and what the platform's built-in queue still cannot do.
Network note for this session: outbound HTTPS reached github.com (git operations),
api.github.com and raw.githubusercontent.com only. Every source below was fetched in full in
this session, either as a file over raw.githubusercontent.com or as repository content via
git clone. Engineering-blog hosts, arxiv.org, dl.acm.org, docs.github.com, github.blog,
opendev.org and YouTube were unreachable (proxy policy denial), which is why the ledger
contains no paper or talk tier rows; the known unreachable material is listed at the end.
GitHub issue/PR discussion threads were also unreachable (the API is session-scoped and HTML
is blocked), so issue references are limited to what repository files and commit history
record, plus one issue cited by title only, flagged as such.
Measured rows marked "derived" were computed in this session from cloned git history; the method is stated in the row.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | OpenStack / Zuul | Project Gating (Zuul v3 docs) | adr | ≤2019-04-16 (last GitHub mirror commit) | 2026-10-04 | https://raw.githubusercontent.com/openstack-infra/zuul/dc9347c1223e3c7eb0399889d03c5de9e854a836/doc/source/user/gating.rst | Gating means testing the change as it will be merged, and informal pre-merge testing "does not scale very well" | "A gating system should always test each change applied to the tip of the branch exactly as it is going to be merged." |
| 2 | OpenStack / Zuul | Project Gating (same document) | adr | ≤2019-04-16 | 2026-10-04 | same as #1 | Speculative parallel testing degrades to serial under failure | "In the best case, as many changes as execution contexts are available may be tested in parallel and merged at once. In the worst case, changes are tested one at a time (as each subsequent change fails, changes behind it start again)." |
| 3 | Mozilla / Graydon Hoare | bors (original) README and history | source | first commit 2013-02-01; maintenance note added 2021 | 2026-10-04 | https://raw.githubusercontent.com/graydon/bors/master/README.md | The original 2013 integrator was a stateless cron loop for mozilla/rust on Buildbot, later superseded by Homu and Bors-NG | "We assume bors is run in a loop, perhaps once per minute from cron … it reloads its entire state from github and buildbot"; "superseded by multiple enhanced rewrites: Homu … Bors-NG" (2021 note). First commit dated from git log --reverse. |
| 4 | Rust project | Homu README | source | repo 2014-12-18 to 2025-11-04 (git log) | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/homu/master/README.md | Pre-merge PR CI cannot protect the default branch; the test must run just before the merge; Homu added state and webhooks because of GitHub rate limits | "after several other pull requests are merged into the default branch, your pull request can still break things after being merged"; "the test procedure should be executed just before the merge"; "This is essential because of GitHub's rate limiting." |
| 5 | Rust project | Homu commit history: try builds disabled during migration | source | 2025-07-22 | 2026-10-04 | https://github.com/rust-lang/homu/commit/4d48344dce7e911d924c9ef840a9da8d1cbac5ac | Homu was dismantled incrementally while the replacement took over; try builds moved first | Merge commit "Merge pull request #236 from Kobzol/disable-try-builds" containing "Disable try builds", dated 2025-07-22 in git history. |
| 6 | bors-ng / Michael Howell | Bors-NG README | source | checked-out state 2026-10-04; deprecation notice added 2023 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng/master/README.md | The canonical semantic-conflict example (rename + new call site both pass alone, fail together); batching with bisection; complexity claim | "Once they both get merged, the main branch will go red (Method bifurcate() not found)"; "it splits the batch into two batches"; "The one-at-a-time strategy is O(N) … The batching strategy is O(E log N)". |
| 7 | bors-ng | TMIB 76: "This April, bors-ng is deprecated" | adr | 2023-05-01 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng.github.io/master/_posts/2023-05-01-tmib-76.md | Bors-NG deprecated itself when GitHub's built-in queue arrived; the stated reasons are platform-integration bugs a third party structurally cannot fix | "there's a bunch of bugs in bors-ng that we can't fix, but they can. For example … there's no way to make mark the PR as 'merged'"; "GitHub merge queues get a button in the interface, while Bors-NG relies on a command"; "I pretty much expected this to happen after GitHub Actions and GitLab merge trains came out." |
| 8 | bors-ng | TMIB 76 (same document) | blog | 2023-05-01 | 2026-10-04 | same as #7 | The maintainer counts popularising the rule, not the tool, as the success | "it helped popularize the Not Rocket Science rule that the mainline branch should be tested before being pushed to mainline. There is now turnkey support for that built into the platform." |
| 9 | bors-ng | "About semantic conflicts" (pitch essay) | blog | 2017-02-02 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng.github.io/master/_posts/2017-02-02-pitch.md | Per-PR CI checks each change in isolation; the merged build can still fail | "But it only checks each one in isolation. The merged build can still fail. You need to check the combination, before it goes to master." |
| 10 | bors-ng | "Whirlwind" lineage guide | blog | 2017-04-06 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng.github.io/master/_posts/2017-04-06-whirlwind.md | Bors-NG was built because Homu tests one PR at a time and is hard to self-administer | "Homu doesn't scale very well. Homu tests pull requests one at a time"; "Bors-NG was created after Homu, to solve two major problems it had". |
| 11 | bors-ng | "Downtime from 5:47 PM to 2:00 AM UTC the next day" | postmortem | 2017-08-24 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng.github.io/master/_posts/2017-08-23-we-were-down.md | The merge bot itself shipped a production-breaking change that both its unit and integration tests missed; ~8h outage for every repo on the hosted instance | "we screwed up and pushed a broken pull request into production … every time you r+-ed a pull request, it would not actually start anything"; "The integration test … left the author field unset, so it was always nil." |
| 12 | bors-ng | TMIB 78 (final newsletter) | blog | 2023-07-01 | 2026-10-04 | https://raw.githubusercontent.com/bors-ng/bors-ng.github.io/master/_posts/2023-07-01-tmib-78.md | The newsletter itself ended in June 2023, two months after deprecation | "This June is the last TMIB instance"; "a third-party bot that predates GitHub Merge Queues and does mostly the same thing". |
| 13 | Rust project | Rollup Procedure (rust-forge) | adr | last modified 2026-09-25 (git log) | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/rust-forge/main/src/release/rollups.md | Rust CI takes ~3.5h per queue build; rollups exist because the serial queue scales poorly; PRs are risk-graded always/maybe/iffy/never by humans | "every pull request must be tested after merge … As PR volume increases this can scale poorly, especially given the long (~3.5hr) current CI duration"; "The rollup command accepts four values always, maybe, iffy, and never"; "the job of the PR queue is to test PRs, not to land them." |
| 14 | Rust project | Bors service page (rust-forge) | source | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/rust-forge/main/src/infra/docs/bors.md | The production queue for rust-lang/rust is now the rewritten rust-lang/bors, deployed on the project's own ECS cluster | "The infrastructure team manages a merge queue bot called 'Bors' … automatically deployed from the rust-lang/bors repository onto our ECS cluster." |
| 15 | Rust project | New bors design document | adr | checked-out state 2026-10-04; repo started 2022-11-14 | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/bors/main/docs/design.md | The 2026 rewrite still runs exactly one auto build at a time; rollups are unrolled after merge for blame attribution | "Only one auto build runs at a time to ensure that each PR is tested against the same branch state it will be merged into"; "When a rollup PR is merged … bors creates a separate unrolled build for each of its rollup member." |
| 16 | Rust project | New bors design document (same) | adr | checked-out state 2026-10-04 | 2026-10-04 | same as #15 | A first design (poll all check suites) was abandoned for webhook-ordering race conditions; completion webhooks can arrive out of order | "GitHub would send us a webhook that a check suite was completed, but when we then asked the GitHub API … it was still marked as pending"; "the 'Check suite was completed' webhook … can actually be received before a webhook that tells us that the last workflow of that check suite was completed." |
| 17 | Rust project | Infra team 2025 Q3 recap | blog | 2025-10-16 | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/blog.rust-lang.org/main/content/inside-rust/infrastructure-team-2025-q3-recap-and-q4-plan.md | Try builds moved to the new bors from July 2025 | "Starting in July, all try builds (@bors try) have run exclusively through the new bors." |
| 18 | Rust project | Infra team 2025 Q4 recap | blog | 2026-01-13 | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/blog.rust-lang.org/main/content/inside-rust/infrastructure-team-2025-q4-recap-and-q1-2026-plan/index.md | The migration off Homu completed in Q4 2025, eleven years after Homu's first commit | "We have now enabled the new bot to merge rust-lang/rust PRs, completing the migration off Homu." |
| 19 | Rust project | Infra team 2026 Q2 recap | blog | 2026-07-15 | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/blog.rust-lang.org/main/content/inside-rust/infrastructure-team-2026-q2-recap-and-q3-plan/index.md | Mergeability checks fell from ~30 min to ~1 min by moving from REST to GraphQL; satellite repos that adopted GitHub merge queue lost reviewer delegation and had to rebuild it in a bot | "cut Bors pull request mergeability check times from an average of 30 minutes to just 1 minute"; "When we switched our repositories to GitHub merge queues instead of bors … we lost the ability to delegate approval on behalf of the reviewer." |
| 20 | Rust project | rust-lang/rust CI outage postmortem 2026-07-02 | postmortem | 2026-07 (committed to infra-team repo) | 2026-10-04 | https://raw.githubusercontent.com/rust-lang/infra-team/main/service-catalog/rust-ci/post-mortems/20260702-rust-outage/README.md | A kernel.org mirror outage closed the rust-lang/rust tree for ~4 days; queue availability is the product of every external dependency's availability | "the tree had to be closed"; timeline: reported 2026-07-02 07:48, tree re-opened 2026-07-06 04:26; "If each external dependency has 99% uptime, that drops to merely about 5" (external dependencies before CI uptime falls below 95%). |
| 21 | Kubernetes | Tide documentation (history and features) | adr | checked-out state 2026-10-04; page last modified 2026-06-02 (git log) | 2026-10-04 | https://raw.githubusercontent.com/kubernetes-sigs/prow/main/site/content/en/docs/components/core/tide/_index.md | Tide (2017) replaced mungegithub's Submit Queue; its search-driven design was forced by GitHub API rate limits; it batches and serves many repos from one instance | "Tide was created in 2017 by @spxtr to replace mungegithub's Submit Queue. It was designed to manage a large number of repositories … without using many API rate limit tokens"; "Automatically runs batch tests and merges multiple PRs together whenever possible." |
| 22 | Kubernetes | Maintainer's Guide to Tide | source | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/kubernetes-sigs/prow/main/site/content/en/docs/components/core/tide/maintainers.md | A human merge invalidates the whole pool's running tests; batches take priority over single PRs; GitHub search-index corruption silently hides mergeable PRs | "Don't let humans (or other bots) merge … Every merge invalidates currently running tests for that pool"; "Any merge to a pool kicks all other PRs in the pool back into Queued for retest"; "you may have encountered a rare bug with GitHub's search indexing." |
| 23 | Kubernetes | Prow Tide config source | source | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/kubernetes-sigs/prow/main/pkg/config/tide.go | Batch size is bounded per org/repo by batch_size_limit; the limit is a config knob, not a constant |
"BatchSizeLimitMap is a key/value pair of an org or org/repo as the key and" the size limit as value; func (t *Tide) BatchSizeLimit(repo OrgRepo) int. |
| 24 | GitHub | Managing a merge queue (github/docs source) | vendor | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/github/docs/main/content/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue.md | The built-in queue speculates like Zuul (temporary main/pr-N branches including predecessors); concurrency 1–100; an explicit setting tolerates flaky checks; queue-jumping rebuilds everything behind |
"Pull requests that have failed required checks can be added to a group as long as the last pull request in the group has passed … useful if you have intermittent test failures"; "The maximum number of merge_group webhooks to dispatch (between 1 and 100)"; "jumping to the top of a merge queue will cause a full rebuild of all in-progress pull requests." |
| 25 | GitLab | Merge trains documentation (gitlabhq source) | vendor | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/gitlabhq/gitlabhq/master/doc/ci/pipelines/merge_trains.md | Trains run up to 20 parallel pipelines by default; a failure restarts every pipeline behind it; an admin can "merge immediately", which aborts the whole train | "each merge train can run a maximum of 20 pipelines in parallel"; "All pipelines for merge requests queued after the removed merge request restart"; "Merge immediately without restarting merge train pipelines" (flagged as risking new failures on the target branch). |
| 26 | Google / Chromium | Chromium commit queue docs | vendor | checked-out state 2026-10-04 | 2026-10-04 | https://raw.githubusercontent.com/chromium/chromium/main/docs/infra/cq.md | The CQ separates dry runs from submitting runs, retries failed test shards to absorb flake, and offers a larger "Mega-CQ" for risky changes | "The CQ will normally try to retry failed test shards (up to a point) to work around" flakiness; "The Mega CQ takes much longer, and should only be used for particularly risky CLs." |
| 27 | Smarkets | marge-bot README | blog | checked-out state 2026-10-04; project started 2017 | 2026-10-04 | https://raw.githubusercontent.com/smarkets/marge-bot/master/README.md | The Not Rocket Science Rule quote and attribution; rebase-and-retry by hand "just doesn't scale" at 5–10 min CI; batch mode falls back to merging the first MR on failure rather than bisecting | "automatically maintain a repository of code that always passes all the tests. — Graydon Hoare, main author of Rust"; "the number of times one need's to rebase-and-try-to-accept starts to become unbearable"; "If the batch job fails for any reason, we fall back to merging the first merge request, before attempting a new batch job." |
| 28 | Rust project | rust-lang/rust git history, September 2026 | source | measured 2026-10-04 | 2026-10-04 | https://github.com/rust-lang/rust | Derived: 130 queue builds landed 737 PRs in Sept 2026; 70 of 130 builds were rollups carrying 677 PRs (9.7 PRs per rollup; 91.9% of PRs rode a rollup) | Counted from a --filter=blob:none clone: first-parent commits 2026-09-01..2026-10-01 matching "Auto merge of #" (130), of which branch :rollup- (70); commits matching "Rollup merge of #" (677). |
| 29 | Kubernetes | kubernetes/kubernetes git history, September 2026 | source | measured 2026-10-04 | 2026-10-04 | https://github.com/kubernetes/kubernetes | Derived: 379 PRs merged to master by the queue in Sept 2026 (12.6/day), all by kubernetes-prow[bot] | Counted from a bare clone: first-parent commits 2026-09-01..2026-10-01 matching "Merge pull request" (379); committer kubernetes-prow[bot] on inspected commits. |
| 30 | Kubernetes | test-infra issue #13551, "tide: serial merges should occur when batches fail" | source | 2019 (issue number era); title located via search 2026-10-04 | 2026-10-04 (title only) | https://github.com/kubernetes/test-infra/issues/13551 | A failing batch can starve all merges while it retests; the project tracked falling back to serial merges as the fix. Cited by title only: the thread body was unreachable from this session. | Issue title: "tide: serial merges should occur when batches fail"; search-result description: Tide "spent many hours retesting a failing batch of PRs" with no PRs merging. Treat the description as unverified. |
Known but unreachable from this session
These exist and are worth the reader's time; the session's network policy (GitHub-only) meant they could not be fetched, so nothing in the guide relies on their content.
- Ananthanarayanan et al., "Keeping Master Green at Scale" (Uber SubmitQueue), EuroSys 2019. The one peer-reviewed treatment of speculative merge queues; dl.acm.org and the author mirrors were blocked.
- Graydon Hoare, "The Not Rocket Science Rule Of Software Engineering" (graydon2.dreamwidth.org, 2014). Quoted verbatim in #27; the original host was blocked.
- The GitHub changelog entries for merge queue public beta (2023-02-08) and GA (2023-07); github.blog was blocked. The beta date is attested by the link text in #6/#7's repo.
- matklad, "GitHub merge queue" critique (2023-06-18), referenced in #12; github.io pages were blocked.
- OpenStack gate operational lore (docs.opendev.org); the Zuul v3 docs survive on the retired GitHub mirror used in #1/#2.