A stateful service runs on 16 large nodes, each holding a large in-memory working set. You must move to 200 small nodes under live traffic, for finer failure granularity and better bin-packing. Sequence the migration, and say what changes that nobody plans for.

The sequence, each step reversible

1. Make the ownership rule explicit. Which node owns which keys, and how a client finds out. If that is currently an implicit hash over 16, it must become an explicit, configurable, versioned partition map before anything moves. Nothing else is safe until this exists.
2. Introduce the routing indirection while still pointing at 16 nodes. Identical behaviour, new mechanism, fully reversible, and it is where the bugs will be found cheaply.
3. Move to a partition count independent of node count — a large fixed number of logical partitions, say 4,096, mapped onto nodes. This is the step that turns every future resize into a data move rather than a re-keying, and skipping it is the decision that makes the next resize as painful as this one.
4. Grow in stages: 16 → 32 → 64 → 200, measuring at each. Not because the code differs, but because the second-order effects below appear gradually and you want the curve rather than the endpoints.
5. Decommission the large nodes only after a full failure drill on the new topology, including a node loss during a rebalance.

What nobody plans for

• Cache hit rate falls even though total memory is unchanged. The same working set split 200 ways means each node holds a two-hundredth of it, and any request whose key is not local now misses. With a long-tailed access pattern the aggregate hit rate can drop sharply, because the tail no longer fits anywhere it is asked for.
• Connection count multiplies by the node count. Clients that held 16 connections each now hold 200. A fleet of 500 clients goes from 8,000 to 100,000 connections, which is memory on both sides and a limit somewhere you have not looked.
• Anything all-to-all grows superlinearly. Gossip, membership, rebalancing and scatter-gather reads all scale with node count or its square. A scatter-gather that took the maximum of 16 tail latencies now takes the maximum of 200, and tail amplification means the effective p99 is far worse even though every node is individually faster. This is usually the biggest surprise in the whole migration.
• Per-node fixed costs multiply. An agent, a sidecar, a runtime heap floor, a log stream and a metric set, 200 times over. At 500 MB of overhead per node that is 100 GB you did not previously pay.
• Node failures become routine. That is the point — each one is smaller — but it means recovery must be automatic and rehearsed, because an event that used to happen twice a year now happens most weeks.

Where data can diverge, and how you would know

During a partition-map change, two nodes can believe they own a key. The map must carry a version, every response must be stamped with the version that served it, and clients must reject a response stamped older than the map they hold. Without that, a write can land on the previous owner after the map moved and be silently lost at the next rebalance.

The rollback at each stage

Steps 1 to 3 are configuration changes reversible in minutes. Step 4 is reversible per stage, which is the reason for doing it in four moves rather than one.

How long it really takes

The routing indirection and the partition map are the calendar time; the node-count change itself is an afternoon. Teams that estimate this by the last step are wrong by a factor of ten, and the estimate is where the migration gets approved.

When this is the wrong answer

When the working set genuinely needs to be co-resident. If the access pattern is broad and the whole set has to be hot, splitting it 200 ways trades a cache hit for a network hop on most requests, and the finer failure granularity is not worth a tenfold latency increase. The same applies when the dominant read is a scatter-gather: going wider makes tail amplification worse, so the reliability argument and the latency argument point in opposite directions and the latency one usually wins. Measure the hit-rate curve against node count on a copy before committing.