Thirteen production accounts of splitting a live database across more machines, from Notion, Figma, Slack, GitHub, Shopify, Discord, Etsy, Pinterest, Stripe, Adyen, Mercari, Monzo and AWS, plus six published incidents. The minute everybody rehearses, the one where writes stop, is measured in seconds and has never been blamed for an outage in this corpus. The damage comes from the shard key chosen nine months earlier and from the capacity added six months later.
The problem, stated without the word sharding: one machine holds the data and answers the queries, the business now needs several machines to do both, and the application is not permitted to stop while the data is spread out.
Every account in this guide was written by someone who had already done the work, and they agree on the shape of the operation to a degree that is unusual. Route the traffic through something you control. Copy the data. Tail the changes the copy is racing. Compare the two copies until you believe them. Stop writes for a moment and move the pointer. Keep replication running backwards so you can undo it. Then, months later, delete the old copy. Notion, Figma, GitHub, Shopify, Slack, Stripe and Vitess all describe that sequence, in that order, with different words for each stage.
What they do not agree on is anything about the shard key, which is the decision that determines whether the resulting system is operable. Notion partitions by workspace because "each block belongs to exactly one workspace". Figma looked at the same idea and rejected it: Sammy Steele, who led the work, said on Postgres FM that at Figma "data moves quite frequently between orgs, which makes that kind of sharding model quite hard", so they picked a set of keys per table instead. Slack started with workspace and moved to channel to flatten the distribution. Etsy ended up with "more than 30 different IDs" used as sharding keys across its tables. Pinterest embedded the shard number in the object id itself, permanently.
That divergence matters because of what the incident record contains. Of the six published incidents reconstructed here, none is caused by the atomic table swap that everyone rehearses and fears. Two are caused by the shard key interacting with a read pattern nobody modelled (Slack, twice). Two are caused by adding capacity to an already-sharded store, which is the operation teams treat as routine once the hard migration is behind them (Monzo, DynamoDB). One is caused by the load of a migration rather than its logic, and ended with sixteen private repositories readable by strangers for seven minutes (GitHub, 2012). One is a failure of the verification tool rather than the system it verifies (Vitess VDiff v1).
The cutover is short, reversible and well understood. The shard key is long-lived, irreversible in practice, and chosen before you have the operational data that would tell you whether it is right. Spend your design review on the key and your rehearsal budget on the capacity-addition procedure you will run twelve times afterwards, not on the swap.
Scope. This guide covers online redistribution of an operational transactional store across more machines: choosing the partition key, moving the data, verifying it, switching traffic, and adding capacity afterwards. It does not cover analytical stores and lakehouse partitioning, which have different constraints because nothing is reading and writing the same row concurrently; it does not cover multi-region placement or data residency; it does not cover caching strategy except where a cache miss caused an incident; and it treats schema change only where its correctness protocol turns out to be the same protocol as the data migration, which is section three.
Six stages appear in every published account. The stage teams skip is the fifth, and it is the one that decides whether a bad cutover is an incident or a shrug.
Before any of the six stages, there is a precondition that every account treats as obvious and that no tutorial states plainly: you cannot move a database the application addresses directly. Figma built DBProxy, a Go service that parses SQL into an abstract syntax tree and routes on the extracted shard id. GitHub routed through ProxySQL. Notion drove its cutover by editing PgBouncer configuration. Slack put VTGate in front. Shopify keeps a routing table that maps a shop to a pod. Pinterest keeps a shard-to-host configuration table in ZooKeeper, and writes that "this config only changes when we need to move shards around or replace a host". Five organisations, five different technologies, one architectural move: insert a layer whose only job is to answer "which machine holds this key", and make that answer changeable at runtime. Everything downstream is a variation on editing that answer safely.
--enable-reverse-replication, and in
Figma's config-flag rollback, and is absent from the accounts that took scheduled downtime.
Reconstructed from
Notion,
Stripe,
Shopify and
Vitess.A bulk reader that walks the source in batches and writes the target. The universal finding is that its throughput is an engineering choice rather than a property of the data. Discord replaced ScyllaDB's migrator with one written in Rust and reached "up to 3.2 million per second", turning three months into nine days. Notion found the cost was index maintenance, not volume: deferring index creation cut a copy "from 3 days to 12 hours". Adyen batches 100,000 rows and commits between batches.
The copy takes hours or days, so live writes must be captured from a position taken before the copy started and replayed after it. Shopify streams the binlog and filters to one shop. Notion's first migration used an audit-log table populated by triggers; its second used Postgres logical replication. Stripe made the application write to both stores. All three are the same mechanism with different durability and different failure modes.
The stage that separates the confident accounts from the nervous ones. Notion ran dark reads against a follower and, notably, had "migration and verification logic implemented by different people". Stripe ran GitHub's Scientist in production so that live reads compared both stores continuously. Slack built "a parallel double-read diffing system". Shopify runs verifiers before, during and after, and specifies the algorithm in TLA+.
Stop writes, wait for the tailer to reach zero lag, change the routing, resume. GitHub's six-step version takes "only a few tens of milliseconds for our busiest database tables". Notion pauses PgBouncer, confirms catch-up, flips the replication direction and resumes. Vitess stops writes on the source primary and refuses to switch if replication lag exceeds a configured bound. The three descriptions are interchangeable.
After the swap, the new primary replicates back to the old one, so rollback is a routing
change rather than a data recovery. Vitess ships this as a default
(--enable-reverse-replication); Deepthi Sigireddi describes the intent as "keep
the source in sync with the new shards so that if something goes wrong or we made a mistake,
we can quickly fall back". Notion flipped the replication direction as step three of its
per-database failover. Figma got the same property differently, by making the risky change a
percentage-based config flag.
Figma's contribution is to split the operation in two. First make the data look
sharded while it is still on one machine, using a view per shard
(WHERE hash(shard_key) >= min AND < max), measured at "less than 10%"
worst-case overhead. Roll that out by percentage, with a config rollback. Only then perform
the physical split, which is the irreversible part. Nobody else in this corpus separates the
two stages, and everybody else's riskiest step is bigger as a result.
Runs this way at: Figma
One caution about that sequence, from the source rather than the documentation. Vitess buffers
queries during the switch so that applications see a pause instead of an error, but the
buffering is best-effort in ways the docs do not emphasise. Issue 8462, opened by a Vitess
maintainer in July 2021, states plainly that "buffering does not happen for queries in an
existing transaction because that's never safe to do", that buffering engages only after a
query has already failed, and that the mechanism "fails critically during re-sharding
operations" because it is scoped per shard and cannot tell when a multi-shard reshard has
finished. Issue 7059 is the user-visible consequence: an application receiving
operation not allowed in state NOT_SERVING for roughly 65 requests during a
cutover, from an engineer who wrote "I expected SQLNonTransientException won't be thrown to
App". Plan for a small number of hard errors at the swap, and make sure the callers retry.
Six forks where the published accounts split, each with the condition that flips the answer. The first one is the only irreversible decision in the whole programme.
| Decision | Chosen by | Rejected by | Stated reason | Evidence |
|---|---|---|---|---|
| Single tenant shard key | Notion, early Slack | Figma | Entities move between tenants; tenant sizes are heavy-tailed | Postgres FM 100, 2024 |
| Rows never move shard | Shopify | Item-by-item movement is "prone to error"; but tenant balancing needs it | Pinterest, 2015 | |
| Application dual write | Stripe, Mercari, Notion 2021 | Notion 2023 | Double-write throughput bottlenecked the switchover | Notion, 2021 |
| Build the router | Figma, GitHub, Notion | Slack, Etsy | Opaque placement was unacceptable; migration cost of switching engines was not | Figma, 2024 |
| Logical before physical | Figma | All others | Makes the reversible half of the change carry the rollout risk | Figma, 2024 |
| Deferred index build | Notion, Figma | Default tooling | Indexes make logical replication "really, really expensive" | Notion, 2023 |
| Verification by a second author | Notion | Unstated elsewhere | A verifier written by the migration's author shares its assumptions | Notion, 2021 |
Stripe's 2017 post describes a four-phase dual write: write both, backfill, move reads, move writes. Google's F1 paper describes a schema change protocol with intermediate states called delete-only and write-only, and proves that "many common schema changes can cause anomalies and database corruption" unless they are decomposed, with the whole argument resting on servers being "no more than one schema version behind". CockroachDB's October 2015 RFC reimplements exactly those states and rejects the alternative in one sentence: a global table lock "seems as difficult as the current proposal and much worse for the user experience".
These are the same protocol. In both cases the system has two representations of the truth, participants adopt the new one at different times, and correctness is preserved by inserting intermediate states such that no two participants are ever more than one state apart. Stripe's phase two exists for the same reason as F1's write-only state: a reader that has not yet moved must still find data that a writer that has already moved has written. I have not found a source that names this rule, so I will: the one-step rule. It is the reason you cannot collapse a migration from four deploys into two, no matter how much the schedule wants you to, and it is the question to ask of any migration plan you are reviewing. Which two components are more than one step apart, and for how long?
Six published incidents, grouped into four classes. Note what is absent: none of them is the swap.
Both of Slack's published database incidents are in this class, and they are the most transferable material in this guide, because the fault was designed in years before it fired and it fired without anyone touching the database.
Both incidents in this class happened to teams who had already sharded successfully and were doing the routine thing afterwards. This is the class an architecture review never covers, because the review happens before the migration and this happens after it.
I did not find a public postmortem attributing data loss or corruption to the cutover itself: not to gh-ost's atomic swap, not to a Vitess SwitchTraffic, not to a PgBouncer repoint. I searched the failure vocabulary directly and through curated postmortem collections. Two readings are available. Either the swap is genuinely the well-solved part, which the design record supports: gh-ost's cut-over design enumerates each connection death and shows every one returns to the pre-cutover state, and notes that "replication only sees the RENAME". Or the failures exist and are not published, because a cutover that silently drops writes is discovered weeks later as a data-quality complaint, not as an outage. Both readings argue for the same control: an independent reconciliation job that runs after the swap and keeps running, rather than a diff that runs before it.
Everything quantitative in the corpus, with the organisation, the context and the date. The most useful column is the last one, because half of these will be wrong in three years and you should know which half.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Write stall at physical shard split | 10 s | Figma | Partial availability on primaries only; replicas unaffected | 2024 | Figma |
| Write stall at cutover | tens of ms | GitHub | Six-step replication-topology cutover, busiest tables | 2021 | GitHub |
| Worst user-visible stall | ~1 s | Notion | Per-database failover, 32 to 96 databases | 2023 | Notion |
| Scheduled downtime, first sharding | 5 min | Notion | Listed afterwards as a regret | 2021 | Notion |
| Logical shards / physical databases | 480 / 32 | Notion | 15 logical schemas per machine, so later moves re-pack rather than re-hash | 2021 | Notion |
| Databases after second reshard | 96 | Notion | Trigger was shards "exceeding 90% CPU utilization at peak" | 2023 | Notion |
| CPU and IOPS after reshard | ~20% | Notion | Down from over 90% at peak | 2023 | Notion |
| Backfill time, with vs without indexes | 3 d → 12 h | Notion | Deferring index creation during logical replication | 2023 | Notion |
| Backfill time, first sharding | 3 days | Notion | Production backfill via audit-log double write | 2021 | Notion |
| Query overhead of shard views | <10% | Figma | Worst case; "minimal" in most cases | 2024 | Figma |
| Time to shard the first table | ~9 months | Figma | Includes building DBProxy; first sharded tables live September 2023 | 2024 | Figma |
| Database stack growth | ~100× | Figma | Since 2020 | 2024 | Figma |
| Peak query rate after migration | 2.3M QPS | Slack | 2M reads, 300K writes; median 2 ms, p99 11 ms | 2020 | Slack |
| Migration duration to 99% of traffic | ~3 years | Slack | Started 2017; 0 QPS to 2.3M QPS on the new platform | 2020 | Slack |
| Query rate before partitioning | 950K QPS | GitHub | Single cluster: 900K on replicas, 50K on the primary | 2019 | GitHub |
| Query rate after, and per-host load | 1.2M QPS, load halved | GitHub | Same tables across several clusters | 2021 | GitHub |
| Sharded fleet scale | 1,000 shards / 425 TB / 1.7M rps | Etsy | Application-level sharding from ~2010, reported secondhand | 2026 | InfoQ on Etsy |
| Platform migration effort | 5 years, ~2,500 PRs | Etsy | Moving shard routing into Vitess vindexes; 6,000 queries touched | 2026 | InfoQ on Etsy |
| Bulk copy throughput | 3.2M records/s | Discord | Custom Rust migrator; three months of work reduced to nine days | 2023 | Discord |
| Node count after re-platforming | 177 → 72 | Discord | Cassandra to ScyllaDB; read p99 40-125 ms to 15 ms | 2023 | Discord |
| Shard imbalance, before and after balancing | ~4× → ~2× | Shopify | Ratio of most-used to least-used shard, in a simulation of their rebalancer | 2021 | Shopify |
| Single-table repartition scale | ~100 TB | Adyen | One table across shards; batches of 100,000 rows | 2024 | Adyen |
| Failed requests observed during a cutover | ~65 | Vitess user | NOT_SERVING errors despite buffering; single reported instance | 2020 | Vitess issue 7059 |
| Incident duration, capacity addition | ~10 h | Monzo | 13:10 to 23:00; majority restored by 15:08 | 2019 | Monzo |
| Incident duration, metadata overload | ~5 h | AWS DynamoDB | Peak error rate ~55% | 2015 | AWS |
| Peak request rate, auto-sharded store | 89.2M rps | DynamoDB | 2021 Prime Day; upper bound on what automatic partitioning achieves | 2022 | USENIX ATC |
Measured: every row above comes from the operator of the system, except Etsy's, which is InfoQ reporting Etsy's own account because Etsy's blog is not machine-fetchable. Derived: nothing. Unknown: currency. No account in this corpus publishes what a resharding programme cost in money. The honest planning figure is calendar time with a team attached: nine months to the first sharded table at Figma, three years at Slack, five years at Etsy. If you need a budget line, price those durations against a team of four to eight engineers and treat anything faster as a claim to verify.
Stale risk: the QPS and node-count rows age fastest and are useful only as ratios. The stall durations and the backfill-versus-index result are mechanism, and should still hold. The Etsy row is a 2026 snapshot of a system built in 2010, which is its own lesson about how long a shard key lives.
Every source behind this page, graded. Filter by kind. The full ledger, one
row per claim with the supporting quote, ships beside this file as sources.md.
A cache restart on 25% of the fleet turned every client boot into a query against every shard, because channel membership is sharded by user id and the query filters by channel.
One customer's bulk user removal concentrated writes on the shard holding 6% of that tenant's data, OOM-killed the primary, and put replica replacement into an infinite loop.
Six new nodes took ownership of partition ranges before the data had streamed to them, so reads succeeded and returned nothing. One flag governed two behaviours; the single-node rehearsal could not reproduce it.
Secondary indexes multiplied partition counts, membership lists doubled or tripled, and after a network blip the metadata fetch itself exceeded the time storage servers allowed.
A zero-downtime schema migration produced load beyond anything the team had seen, failed the primary's health checks, and started a failover cascade that ended in a split brain and sixteen private repositories briefly readable by strangers.
480 logical shards on 32 machines, keyed by workspace id, migrated with an audit-log double write, dark reads and five minutes of scheduled downtime. Names three regrets, including sharding too late.
32 to 96 databases with no scheduled downtime, using Postgres logical replication, dark reads, a PgBouncer pause and a replication-direction flip. Deferring index creation cut the copy from three days to twelve hours.
The only account that separates logical sharding (views, percentage rollout, config rollback) from physical sharding (the irreversible failover), and the only one that publishes the resulting stall: ten seconds on primaries.
Three years from 0 to 2.3M QPS on Vitess, a shard-key change from workspace to channel, and a parallel double-read diffing system to prove semantics matched.
Vertical partitioning of a 950K QPS cluster, with a six-step write cutover taking tens of milliseconds, and a candid note that Vitess was not the right tool for every move.
The canonical four-phase dual write, with backfill run offline over database snapshots and continuous verification using Scientist against live production reads.
Ghostferry batch-copies one tenant's rows while tailing the binlog, then cuts over when the queue is seconds deep. Verifiers run before, during and after; the algorithm is written in TLA+.
Hot partitions degraded a quorum-replicated cluster; the fix was a re-platform with dual writes and a custom Rust migrator at 3.2M records per second, nine days instead of three months.
The strict alternative: the shard id lives in the object id, and "once a piece of data lands in a shard, it never moves outside that shard". Capacity comes from opening ranges or splitting a machine's range after replication.
The in-machine analogue: batches of 100,000 rows, detach and attach at the end, and an explicit warning that rows changing during the load need triggers or change data capture.
Enumerates three named race conditions in a dual write and builds a consistency checker on a non-locking read-only transaction. Hundreds of billions of records in scope.
The clearest published statement that splitting by size does not split load, with the two named failure modes: hot partitions and throughput dilution.
The dissent, and it predates every other source here. Morgan Tocker argues the gains from query, index and schema work are an order of magnitude larger than the gains from sharding, and that sharding first makes tuning harder.
Why the naive rename leaves "a brief period of time where your table just does not exist", and how a two-connection blocking swap removes that window while keeping the operation atomic from replication's point of view.
The design argument itself: each connection death is enumerated and shown to return to the pre-cutover state. "If both C10 and C20 die, no problem." Replication sees only the RENAME.
A maintainer's own account of the limits: buffering is reactive, scoped per shard, primary only, and "does not happen for queries in an existing transaction because that's never safe to do". It "fails critically during re-sharding operations".
The user-visible version of the same fact: about 65 requests failed with
operation not allowed in state NOT_SERVING during a reshard cutover, from an
engineer who expected buffering to hide it.
A proposal to let change-stream consumers resume automatically after a reshard, opened and closed by its author within a day. The underlying fact stands: a reshard breaks downstream consumers of the change stream, and that consequence is rarely in the migration plan.
Verification at scale took "hours and days", needed large memory in a component meant to be lightweight, and could not resume: "A fresh snapshot has to be taken and we start VDiff from scratch."
An independent reimplementation of F1's protocol, with DELETE_ONLY, WRITE_ONLY and PUBLIC states, backfill in small transactions, and the global-lock alternative rejected in one sentence.
A live data mover with a formal specification, and an unusually honest caveat next to it: "the specification might not be entirely correct as proofs remain elusive".
Proves that common schema changes corrupt data when servers hold different versions, and that decomposing them into intermediate states is safe "so long as all servers are no more than one schema version behind".
The engineering account behind automatic partitioning at 89.2M requests per second, and the source of the partition-splitting lessons cited above.
States the cost of the staged-state protocol plainly: it "leads to delays in the deployment of new schemas since it requires waiting for massive data backfill", and proposes lazy migration instead.
Figma, Notion and Adyen in one round table. The clearest statement of the shard-key disagreement, plus the finding that indexes make logical replication "really, really expensive", and Notion's engineer saying four months in that "this is a lot".
On why reshard is deliberately manual: "There is a lot of human intervention or orchestration in this process, but that is somewhat by design because re-sharding is somewhat of a scary thing to do."
A 2010-era application-level scheme with "more than 30 different IDs" as sharding keys, ported onto Vitess by writing custom vindexes rather than moving data. Five years and about 2,500 pull requests.
The operation as a product: Create, VDiff, SwitchTraffic, ReverseTraffic, Complete, with reverse replication on by default and a lag bound that refuses the switch.
Documents the topology and denylist changes at cutover, and notably does not state the write-stall duration or the buffering limits, which is why the issue tracker above is the better source for both.
Seven rungs. The line between toy and real is at rung four, where you stop moving data and start proving you moved it correctly.
One table, two Postgres or MySQL instances, and a function in the application that maps a key to an instance. Write a load generator that inserts and reads by key.
Done when: you can point at any row and say which instance holds it without querying either. Teaches: routing is the first artefact, not the last.
Put the mapping in a config file or a key-value store that the application reads at runtime, the way Pinterest keeps its shard-to-host table in ZooKeeper. Change the map while the load generator is running.
Done when: you can move a shard's traffic to a different instance without a deploy. Teaches: the precondition in section two; everything later is an edit to this map.
Add a third instance. Capture a change-stream position, bulk copy one shard's rows, then replay changes from that position. Deliberately let the copy take minutes while writes continue.
Done when: a row written during the copy exists on the target with the same value. Teaches: the copy and the live writes race, and the position you captured is the whole correctness argument.
Write a comparator: sample rows from both sides and diff them, then run it continuously against live reads the way Stripe used Scientist. Now inject a deliberate mismatch, for example by dropping every hundredth replayed change.
Done when: the comparator finds your injected corruption before you tell it where to look. Teaches: an unexercised verifier is decoration. This is the rung where the exercise stops being a toy.
Stop writes, drain the tailer to zero lag, repoint the map, resume. Then start replication from the new instance back to the old one and roll the whole thing back with the load generator still running.
Done when: you have cut over and back twice with zero lost writes, and you can state the stall duration at p100. Teaches: rollback after a cutover is a data problem, and the reverse channel is what turns it into a routing problem.
Four experiments. Hold a long-running transaction open across the swap. Kill the copier mid-batch. Send 90% of your load to one key. Then take the cache away from a read path whose filter key is not the shard key, and watch the fan-out.
Done when: you can predict which of the four produces an error your client cannot retry. Teaches: the Slack incidents, at laptop scale.
Go from three instances to five while everything is running. Instrument the shard map's size, the per-shard load spread, and the time a new instance spends owning data it has not yet received.
Done when: your dashboard would have caught Monzo's failure, meaning it shows ownership and data presence as two separate signals. Teaches: the migration happens once; this happens every quarter for the life of the system.
The queries that actually produced the material above. The vocabulary matters more than the operators: engineers who have done this write "cutover", "backfill", "dark reads" and "shard key", never "sharding best practices".
"sharding" "lessons learned" postgres "we" -tutorialintitle:"how we" sharded OR resharded database zero downtime"logical shards" "physical" migration blog how many shards"dual write" backfill "read path" migration engineering blogpostmortem "adding nodes" OR "scaling up" cluster outage incident report"hot shard" OR "hot partition" incident "shard key" engineeringsite:slack.engineering incident datastore shard"schema migration" postmortem failover health check loadpath:docs/RFCS "schema change" OR reshardingrepo:vitessio/vitess is:issue reshard cutover bufferingrepo:vitessio/vitess is:pr is:closed is:unmerged reshard"alternatives considered" sharding "we chose" design docapi.github.com/search/issues?q=repo:ORG/REPO+is:pr+is:closed+is:unmerged+TERM"delete only" "write only" schema change intermediate statespostgres.fm OR se-radio transcript sharding shard key"we hated moving data" OR "never moves outside that shard"Two notes on method, because both cost time here. GitHub's web search will not filter closed unmerged pull requests reliably from a browser, but the REST search endpoint will, and that is how PR 15393 surfaced. And several of the best primary sources refuse automated fetches: Etsy's Code as Craft returns 403, and the USENIX and ACM PDFs return 403 to some clients while resolving fine in a browser. When a source is unreachable, say so and cite the accessible report rather than quietly dropping the claim.