Evidence ledger
34 sources
Checked 29 Aug 2026
Evidence ledger
One row per claim in The cutover is the easy part: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how teams redistribute a live, growing OLTP database across more machines without stopping writes, and what the public record says actually goes wrong.
All links fetched 2026-08-29. One row per claim. Quotes are copied, not paraphrased.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Notion | Herding elephants: lessons learned from sharding Postgres at Notion | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | The logical/physical shard split: many logical shards packed onto fewer machines so later moves are re-packing, not re-hashing | "480 logical shards evenly distributed across 32 physical databases", "15 per database" |
| 2 | Notion | Herding elephants (same post) | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | Shard key chosen from the product's containment model, not from data volume | "partition data by workspace ID"; each block "belongs to exactly one workspace" |
| 3 | Notion | Herding elephants (same post) | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | A managed sharding layer was rejected for loss of control over placement | Citus and Vitess rejected because "the actual clustering logic is opaque, and we wanted control over the distribution" |
| 4 | Notion | Herding elephants (same post) | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | Verification was staffed as an independent implementation, not a review | "migration and verification logic were implemented by different people"; dark reads compared old and new |
| 5 | Notion | Herding elephants (same post) | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | The switchover was bounded by double-write throughput, and cost five minutes of scheduled downtime | "five minutes of scheduled maintenance"; "Double-write throughput was the primary bottleneck in our final switch-over" |
| 6 | Notion | Herding elephants (same post) | blog | 2021-10-06 | 2026-08-29 | https://www.notion.com/blog/sharding-postgres-at-notion | Stated regrets: sharded too late, did not optimise for zero downtime, should have used a combined primary key | Team lists three regrets, including that they should have used a "combined primary key instead of a separate partition key" |
| 7 | Notion | The Great Re-shard: adding Postgres capacity (again) with zero downtime | blog | 2023-07-17 | 2026-08-29 | https://www.notion.com/blog/the-great-re-shard | Capacity trigger for the second reshard, and the 32 → 96 change | "Some of the shards were exceeding 90% CPU utilization at peak traffic"; 32 databases to 96 |
| 8 | Notion | The Great Re-shard (same post) | blog | 2023-07-17 | 2026-08-29 | https://www.notion.com/blog/the-great-re-shard | Copy time is dominated by index maintenance, not by data volume | Skipping index creation during copy and rebuilding after cut sync time "from 3 days to 12 hours" |
| 9 | Notion | The Great Re-shard (same post) | blog | 2023-07-17 | 2026-08-29 | https://www.notion.com/blog/the-great-re-shard | Second reshard achieved a sub-second user-visible stall using PgBouncer pause and a replication direction flip | "At worst, a user would have experienced about a second of a 'saving' loading spinner"; post-migration CPU/IOPS "around 20% during peak traffic" |
| 10 | Figma | How Figma's Databases Team Lived to Tell the Scale | blog | 2024-03-14 | 2026-08-29 | https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ | Separating logical from physical sharding lets the risky half be rolled back by config | "We separated the concept of 'logical sharding' at the application layer from 'physical sharding' at the Postgres layer" |
| 11 | Figma | Same post | blog | 2024-03-14 | 2026-08-29 | https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ | Views as the logical shard boundary, measured overhead | CREATE VIEW table_shard1 AS SELECT * FROM table WHERE hash(shard_key) >= min AND < max; overhead "minimal ... and less than 10%" worst case |
| 12 | Figma | Same post | blog | 2024-03-14 | 2026-08-29 | https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ | A single global tenant key was rejected; multiple per-table keys grouped into colos were chosen | Keys UserID, FileID, OrgID; "Tables within a colo support cross-table joins and full transactions when restricted to a single sharding key" |
| 13 | Figma | Same post | blog | 2024-03-14 | 2026-08-29 | https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ | Distributed SQL engines were rejected on migration cost and operational familiarity | CockroachDB, TiDB, Spanner and Vitess rejected: "switching to any of these alternative databases would have required a complex data migration" |
| 14 | Figma | Same post | blog | 2024-03-14 | 2026-08-29 | https://www.figma.com/blog/how-figmas-databases-team-lived-to-tell-the-scale/ | The measured cost of the first physical shard split | "only ten seconds of partial availability on database primaries and no availability impact on replicas"; "roughly nine months to shard our first table"; stack grew "almost 100x since 2020" |
| 15 | Slack | Scaling Datastores at Slack with Vitess | blog | 2020-12-01 | 2026-08-29 | https://slack.engineering/scaling-datastores-at-slack-with-vitess/ | Multi-year duration and end-state scale of a full migration onto a sharding platform | "99% of all Slack MySQL traffic having been migrated"; "2.3 million QPS at peak", "median query latency is 2 ms, and our p99 query latency is 11 ms" |
| 16 | Slack | Same post | blog | 2020-12-01 | 2026-08-29 | https://slack.engineering/scaling-datastores-at-slack-with-vitess/ | Shard key moved from workspace to channel to flatten load, and correctness was checked by parallel double reads | "shard the message data by the unique id of the channel"; "a parallel double-read diffing system so we were sure that the Vitess-powered tables had the same semantics" |
| 17 | Slack | Slack's Incident on 2-22-22 | postmortem | 2022-04-26 | 2026-08-29 | https://slack.engineering/slacks-incident-on-2-22-22/ | A shard key that suits writes can make every cache miss a scatter query across every shard | "Since the data is sharded by user ID, even one channel missing from cache meant the application had to successfully run a query on every shard"; keyspace held "channel membership which is sharded by user" |
| 18 | Slack | Slack's Incident on 2-22-22 | postmortem | 2022-04-26 | 2026-08-29 | https://slack.engineering/slacks-incident-on-2-22-22/ | The structural fix was a second copy of the data under a different shard key | "modified the problematic scatter query to read from a table that is sharded by channel"; mitigation "throttle client boot requests" |
| 19 | Slack | The Query Strikes Again | postmortem | 2023-11-15 | 2026-08-29 | https://slack.engineering/the-query-strikes-again/ | One tenant's bulk operation concentrates on one shard and takes out the primary and its replacements | One shard held "6% of the user's subscription data"; "the high volume of write load also led the Vitess tablets to run out of memory on the shard primary, which caused the kernel to OOM-kill the MySQL process" |
| 20 | Slack | The Query Strikes Again | postmortem | 2023-11-15 | 2026-08-29 | https://slack.engineering/the-query-strikes-again/ | Automated replacement of a lagging replica becomes a loop that prevents recovery | "an infinite-loop of the primary failing, a replica being promoted to primary, a replacement replica being provisioned, trying (and failing) to catch-up" |
| 21 | GitHub | Partitioning GitHub's relational databases to handle scale | blog | 2021-09-27 | 2026-08-29 | https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ | Scale before and after a multi-year vertical partitioning programme | 2019: "950,000 queries/s on average, 900,000 queries/s on replicas, and 50,000 queries/s on the primary"; 2021: 1.2M/s with "the average load on each host halved" |
| 22 | GitHub | Same post | blog | 2021-09-27 | 2026-08-29 | https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ | The write cutover is measured in tens of milliseconds when it is a replication-topology change rather than a data copy | Six-step cutover taking "only a few tens of milliseconds for our busiest database tables" |
| 23 | GitHub | Same post | blog | 2021-09-27 | 2026-08-29 | https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ | Even a Vitess adopter does not use it for every move | "because of factors like deployment topology and read-your-writes support, we didn't choose Vitess as the tool to move database tables in every case" |
| 24 | GitHub | GitHub availability this week (Sept 2012 incident) | postmortem | 2012-09-14 | 2026-08-29 | https://github.blog/2012-09-14-github-availability-this-week/ | The load of an online schema migration, not its logic, triggered a failover cascade and a data-integrity event | "Monday's migration caused higher load on the database than our operations team has previously seen ... So high, in fact, that they caused Percona Replication Manager's health checks to fail on the master" |
| 25 | GitHub | Same postmortem | postmortem | 2012-09-14 | 2026-08-29 | https://github.blog/2012-09-14-github-availability-this-week/ | Blast radius of a split-brain during migration-induced failover | "16 of these repositories were private, and for seven minutes from 8:19 AM to 8:26 AM PDT ... were accessible to people outside of the repository's list of collaborators"; "one hour and 46 minutes of downtime" |
| 26 | Stripe | Online migrations at scale | blog | 2017-02-02 | 2026-08-29 | https://stripe.com/blog/online-migrations | The canonical four-phase dual-write migration | Phases: "Dual writing to the existing and new tables to keep them in sync", backfill, "Changing all read paths", "Changing all write paths ... to only write to the new table" |
| 27 | Stripe | Same post | blog | 2017-02-02 | 2026-08-29 | https://stripe.com/blog/online-migrations | Verification runs continuously in production against live reads, not as a one-off diff | Used GitHub's Scientist: "If the results don't match, raise an error alerting our engineers to the inconsistency"; "hundreds of millions of Subscriptions objects" |
| 28 | Shopify | Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale | blog | 2021-09-24 | 2026-08-29 | https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale | Per-tenant movement as the balancing primitive, using batch copy plus binlog tailing | "Ghostferry performs batch copying by iterating over the set of tables on the source, selecting relevant rows based on the shop's ID"; cutover entered when the binlog queue is "effectively real-time—on the order of seconds" |
| 29 | Shopify | Same post | blog | 2021-09-24 | 2026-08-29 | https://shopify.engineering/mysql-database-shard-balancing-terabyte-scale | Verification is layered before, during and after, and the algorithm is formally specified | "a suite of verifiers that run before, during, and after the data migration"; "modelled and represented in a formal specification ... written in TLA+"; imbalance improved from ~4x to ~2x |
| 30 | Shopify | ghostferry README | source | 2018-04 (repo, main branch read 2026-08-29) | 2026-08-29 | https://github.com/Shopify/ghostferry/blob/main/README.md | The formal specification is explicitly not a proof | "the specification might not be entirely correct as proofs remain elusive"; "copy data from one mysql instance to another with minimal amount of downtime" |
| 31 | GitHub | gh-ost doc/cut-over.md | source | repo doc, read 2026-08-29 | 2026-08-29 | https://github.com/github/gh-ost/blob/master/doc/cut-over.md | The non-atomic swap leaves a window with no table at all; gh-ost's two-connection swap removes it | "In between the two renames there's a brief period of time where your table just does not exist, and queries will fail"; "an atomic, two-step blocking swap" |
| 32 | GitHub | gh-ost issue #82, "Describing safe, blocking, atomic, pure-mysql cut-over phase" | adr | 2016-06-26 | 2026-08-29 | https://github.com/github/gh-ost/issues/82 | Design record enumerating every failure point of the swap and showing each returns to a safe state | "If both C10 and C20 die, no problem: LOCK is cleared; RENAME lock is cleared"; "Replication only sees the RENAME" |
| 33 | Vitess | Issue #7059, exception thrown to app during resharding cutover | source | 2020-11-20 | 2026-08-29 | https://github.com/vitessio/vitess/issues/7059 | Buffering during cutover is best-effort; applications still see errors | Reporter saw SQLNonTransientException "operation not allowed in state NOT_SERVING", ~65 failed requests; "I expected SQLNonTransientException won't be thrown to App" |
| 34 | Vitess | Issue #8462, Request Buffering during Sharding | source | 2021-07-13 | 2026-08-29 | https://github.com/vitessio/vitess/issues/8462 | The named limits of cutover buffering: reactive, per shard, primary only, never inside a transaction | "Buffering does not happen for queries in an existing transaction because that's never safe to do"; buffering "fails critically during re-sharding operations" |
| 35 | Vitess | PR #15393, VStream automatic resume after reshard (closed unmerged) | source | opened 2024-03-01, closed 2024-03-02 | 2026-08-29 | https://github.com/vitessio/vitess/pull/15393 | Downstream consumers of a change stream break at a reshard; the fix was proposed and withdrawn in a day | "we will start a new copy resume phase to automatically resume when a Reshard has occurred since the last client stream"; closed unmerged by its author |
| 36 | Vitess | Issue #10134, RFC: VDiff2 | adr | 2022-04-24 | 2026-08-29 | https://github.com/vitessio/vitess/issues/10134 | The verification step is the part that does not scale, and unresumable diffs are the failure | "For large tables it can take a lot of time (hours and days) and require a large amount of memory"; "If there is a network issue then we cannot resume the operation. A fresh snapshot has to be taken" |
| 37 | Vitess | How Traffic Is Switched (VReplication internals) | vendor | docs for v22, read 2026-08-29 | 2026-08-29 | https://vitess.io/docs/22.0/reference/vreplication/internal/cutover/ | The documented cutover is a topology and denylist change; the docs do not state the write-stall duration | Tables "added to the 'denylist' on the source keyspace which vttablet uses to reject queries"; "cutovers are performed manually by the user executing the SwitchTraffic and ReverseTraffic actions" |
| 38 | CockroachDB | RFC: online schema change (docs/RFCS/20151014) | adr | 2015-10-14 | 2026-08-29 | https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20151014_online_schema_change.md | An independent reimplementation of the F1 state protocol, with the lock alternative explicitly rejected | DELETE_ONLY / WRITE_ONLY / PUBLIC states; "We could try to come up with some sort of global table lock mechanism ... This seems as difficult as the current proposal and much worse for the user experience" |
| 39 | Online, Asynchronous Schema Change in F1 | paper | 2013 (PVLDB 6(11)) | 2026-08-29 | https://research.google/pubs/online-asynchronous-schema-change-in-f1/ | The correctness rule underneath every staged migration: participants may be at most one version apart | "a protocol for schema evolution in a globally distributed database management system with shared data, stateless servers, and no global membership"; safe "so long as all servers are no more than one schema version behind" | |
| 40 | Same paper, PDF | paper | 2013 | 2026-08-29 | https://www.vldb.org/pvldb/vol6/p1045-rae.pdf | Primary PDF of the above, for the reader | Authors Rae, Rollins, Shute, Sodhi, Vingralek; implemented in F1, "the database management system that stores data for Google AdWords" | |
| 41 | Amazon | Amazon DynamoDB: A Scalable, Predictably Performant, and Fully Managed NoSQL Database Service | paper | 2022 (USENIX ATC) | 2026-08-29 | https://www.amazon.science/publications/amazon-dynamodb-a-scalable-predictably-performant-and-fully-managed-nosql-database-service | Scale reference for an automatically-resharded store | "89.2 million requests per second" during 2021 Prime Day, single-digit millisecond response |
| 42 | Amazon | Lessons learned from 10 years of DynamoDB | blog | 2022-10-21 | 2026-08-29 | https://www.amazon.science/blog/lessons-learned-from-10-years-of-dynamodb | Splitting a partition by size does not split its load; hot partitions and throughput dilution are the result | "Hot partitions happened because customer workloads were not uniformly distributed"; "Throughput dilution happened when partitions that had been split to handle increased load ended up with so few keys that they could quickly max out their meager allocated capacity" |
| 43 | AWS | Summary of the Amazon DynamoDB Service Disruption, Sept 2015 | postmortem | 2015-09 | 2026-08-29 | https://aws.amazon.com/message/5467D2 | Adding partitions grew the metadata each storage server must fetch until the fetch itself timed out | GSIs caused "partition data to the membership lists quickly double or triple"; responses "exceeded the retrieval and transmission time allowed by storage servers"; ~55% error rate by 2:37am, restored 7:10am PDT |
| 44 | Monzo | We had issues with Monzo on 29th July | postmortem | 2019-09-08 | 2026-08-29 | https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on-july-29th | Adding capacity to a live sharded store is itself the dangerous operation | New servers "had joined the cluster, assumed responsibility for some parts of the data (certain partition keys to balance the load), but hadn't yet streamed it over"; each node took "approximately 8-10 minutes to remove safely" |
| 45 | Monzo | Same postmortem | postmortem | 2019-09-08 | 2026-08-29 | https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on-july-29th | The rehearsal that passed did not exercise the failure, because one node cannot break quorum | "when we added six servers to production, the data ownership had two or three members reallocated to the new nodes, meaning we didn't have the same guarantee"; impact 13:10 to ~15:08, full recovery 23:00 |
| 46 | Discord | How Discord Stores Trillions of Messages | blog | 2023-03-06 | 2026-08-29 | https://discord.com/blog/how-discord-stores-trillions-of-messages | Migration throughput is an engineering choice, not a property of the data | Custom Rust migrator hit "speeds of up to 3.2 million per second", finishing in about nine days; dual-writing to both stores during the move |
| 47 | Discord | Same post | blog | 2023-03-06 | 2026-08-29 | https://discord.com/blog/how-discord-stores-trillions-of-messages | Partition-level hot spots degrade a whole quorum-replicated cluster | Messages partitioned "by the channel they're sent in, along with a bucket"; latency p99 moved from "40-125ms on Cassandra" to 15ms; 177 Cassandra nodes to 72 ScyllaDB nodes |
| 48 | Sharding Pinterest: How we scaled our MySQL fleet | blog | 2015 (states "in production ... for 3.5 years now" after an early-2012 launch) | 2026-08-29 | https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f | The opposite design: never move a row, move whole shards | "We made a design decision that once a piece of data lands in a shard, it never moves outside that shard"; "If we had to move data, it was better to move an entire virtual node to a different physical node" | |
| 49 | Same post | blog | 2015 | 2026-08-29 | https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f | Embedding the shard id in the object id removes the lookup table, and forbids rebalancing forever | "The shard ID is 16 bits, type ID is 10 bits and local ID is 36 bits"; capacity added by opening new ranges and by splitting a machine's range after replication | |
| 50 | Same post | blog | 2015 | 2026-08-29 | https://medium.com/pinterest-engineering/sharding-pinterest-how-we-scaled-our-mysql-fleet-3f341e96ca6f | The build was easy and the migration was not | "Implementing it was relatively straightforward, but turning it on and moving all the data over from the old machines was super tough" | |
| 51 | Adyen | Efficiently repartitioning large tables in PostgreSQL | blog | 2024-11-27 | 2026-08-29 | https://www.adyen.com/knowledge-hub/efficiently-repartitioning-large-tables-in-postgresql | Repartitioning within one machine has the same shape as resharding across machines, including the exclusive lock at the end | ~100TB from a single table across shards; batches of 100,000 rows; final step needs "locking the invoices table in exclusive mode"; historical rows changing during load need triggers or CDC |
| 52 | Mercari | Designing a Zero Downtime Migration Solution with Strong Data Consistency, Part IV | blog | 2024-11-13 | 2026-08-29 | https://engineering.mercari.com/en/blog/entry/20241113-designing-a-zero-downtime-migration-solution-with-strong-data-consistency-part-iv/ | Dual write has enumerable race conditions and needs an independent consistency checker | Three named race conditions A, B, C; consistency batch built on Spanner's ReadOnlyTransaction "which does not lock any rows or tables"; "up to hundreds of billions" of records |
| 53 | InfoQ (reporting Etsy) | Etsy Migrates 1000-Shard, 425 TB MySQL Sharding Architecture to Vitess | casestudy | 2026-04-11 | 2026-08-29 | https://www.infoq.com/news/2026/04/etsy-vitess-sharding-migration/ | A fifteen-year-old application-level sharding scheme can be moved onto a platform by porting the routing, not the data | ~1,000 shards, 425 TB, 1.7M requests/s; "more than 30 different IDs were used"; Ella Yarmo-Gray: "Five years, approximately 2,500 pull requests and 6,000 queries later" |
| 54 | Postgres FM | Episode 100, "To 100TB, and beyond!" (Sammy Steele of Figma, Arka Ganguli of Notion, Derk van Veen of Adyen) | talk | 2024-06-07 | 2026-08-29 | https://postgres.fm/episodes/to-100tb-and-beyond/transcript | Practitioners from three sharded shops converge on starting earlier, and disagree on the key | Steele: "data moves quite frequently between orgs, which makes that kind of sharding model quite hard ... for each table, picking a set of a few shard keys"; van Veen: "if you do it now, when you have all the time in the world" |
| 55 | Postgres FM | Same episode | talk | 2024-06-07 | 2026-08-29 | https://postgres.fm/episodes/to-100tb-and-beyond/transcript | The copy step is slow for a reason nobody predicts: index maintenance during logical replication | Steele: "it's really, really expensive to have indexes on your database table while you're doing logical replication"; Ganguli: "it's been like 4 months now and I'm still like, oh my God, this is a lot" |
| 56 | SE Radio | Episode 510, Deepthi Sigireddi on how Vitess scales MySQL | talk | 2022-05-04 | 2026-08-29 | https://se-radio.net/2022/05/episode-510-deepthi-sigireddi-on-how-vitess-scales-mysql/ | Reshard is deliberately manual, and reverse replication is what makes it reversible | "There is a lot of human intervention or orchestration in this process, but that is somewhat by design because re-sharding is somewhat of a scary thing to do"; "keep the source in sync with the new shards so that if something goes wrong ... we can quickly fall back" |
| 57 | Percona | Shard early, shard often | blog | 2009-11-16 | 2026-08-29 | https://percona.com/blog/2009/11/16/shard-early-shard-often | The dissenting position, from before the current generation of accounts | Morgan Tocker: "There's an order of magnitude better performance that can be gained by focusing on query/index/schema optimization. The gains from sharding are usually much lower" |
| 58 | Zeng et al. | SLSM: An Efficient Strategy for Lazy Schema Migration on Shared-Nothing Databases | paper | 2024-04-05 (arXiv) | 2026-08-29 | https://arxiv.org/abs/2404.03929 | The staged-state protocol's known cost: new schemas wait on a full backfill | The approach "leads to delays in the deployment of new schemas since it requires waiting for massive data backfill" |
| 59 | Vitess | Reshard reference (v25) | vendor | docs read 2026-08-29 | 2026-08-29 | https://vitess.io/docs/25.0/reference/vreplication/reshard/ | What a platform-level reshard exposes as an operation | Reshard workflow with SwitchTraffic / ReverseTraffic and --enable-reverse-replication |
Absences worth recording
- No public postmortem found that attributes data loss or corruption to the atomic cutover
itself. Searched with the failure vocabulary (
postmortem "data loss" during table swap,cutover incident report database,gh-ost cut-over incident) and via curated postmortem collections. The incidents that exist are about the load of the migration (row 24), the shard key chosen months earlier (rows 17, 19, 42), and the capacity addition afterwards (rows 43, 44). This is an absence of evidence, not evidence of absence, and it is stated as such in the guide. - No dollar figures. Nobody in this corpus publishes the cost of a resharding programme in currency. The published cost is in calendar time and headcount: nine months to the first sharded table at Figma (row 14), three years at Slack (row 15), five years and ~2,500 pull requests at Etsy (row 53).
- Etsy's own account is not directly citable.
etsy.com/codeascraftreturns 403 to automated fetches, so the InfoQ report (row 53) is cited instead and labelled as secondary.