The cutover is the easy part
How teams redistribute a live, growing transactional database across more machines without stopping writes, and what the public record says actually goes wrong.
A field guide to online resharding, reconstructed from thirteen production accounts (Notion, Figma, Slack, GitHub, Shopify, Discord, Etsy, Pinterest, Stripe, Adyen, Mercari, Monzo, AWS) and six published incidents. It gives you the six-stage reference architecture every account converges on, the six shard-key and tooling decisions with the condition that flips each one, a failure catalogue grouped into four classes, and planning numbers with dates. After reading it you can review a resharding plan by asking the two questions that predict its failures: which hot queries filter on a key that is not the shard key, and what happens the twelfth time someone adds capacity.
No published postmortem in this corpus blames the cutover, the step every team rehearses and fears. The incidents come from the shard key chosen nine months earlier and from adding capacity six months later, and two of the six happened to teams whose migration had already succeeded.
What you get out of it
- The cutover cost is the drain, not the repoint: GitHub reports tens of milliseconds, Notion about one second, Figma ten seconds, and the difference is how much replication lag each team was willing to carry into the swap.
- A shard key is chosen for the write path and fires on the read path. Slack sharded channel membership by user id, and when a cache restart emptied 25% of memcached, 'even one channel missing from cache meant the application had to successfully run a query on every shard'.
- Splitting by size does not split load. DynamoDB names the two consequences, hot partitions and throughput dilution, and calls the uniform-access assumption 'really easy to overlook when designing a system and potentially difficult to fix when in production'.
- Stripe's four-phase dual write and Google's F1 schema-change protocol are the same protocol: decompose one unsafe transition into states such that no two participants are ever more than one step apart. CockroachDB reimplemented it independently in 2015.
- Adding capacity to an already-sharded store is the operation that produced two of the six incidents, and it is the one nobody rehearses at production cardinality. Monzo's single-node test could not break quorum; six nodes could.
Scope
Why this, now. Etsy published in April 2026 how it moved a 2010-era, 1,000-shard, 425 TB sharding scheme onto Vitess over five years and roughly 2,500 pull requests, which closes a loop that Notion (2021, 2023), Figma (2024) and Slack (2020) opened, and makes the whole decision space visible in primary sources for the first time.
What it does not cover. Analytical stores and lakehouse partitioning, where nothing reads and writes the same row concurrently; multi-region placement and data residency; caching strategy except where a cache miss caused an incident; and schema change, covered only where its correctness protocol turns out to be the same as the data migration's.
Other field guides
Keeping one failure from reaching everyone
Slack, Shopify, Salesforce, DoorDash, Amazon EBS and AWS all cut their systems into independent replicas so that one fault hurts some customers rathe…
28 sources · 22 organisations · 5 postmortemsThe renewal failed a month before anyone noticed
A field guide to the machine credential lifecycle, built from ten published incidents at Bazel, DigiCert, Microsoft, Let's Encrypt and Ericsson, the …
22 sources · 17 organisations · 4 postmortemsThe retry is the outage
A field guide to metastable failure: the class of outage where the system stays down after the thing that broke it has been repaired, because the ret…
30 sources · 19 organisations · 7 postmortems