Slack has written that it began moving its sharded MySQL estate to Vitess in 2017 and that by December 2020 Vitess served 99% of its query load - around 2.3 million queries per second at peak. Before the move it sharded by workspace with a separate metadata cluster to find a workspace's shard. What forced the change and what did three years of it cost?

The situation they were in

Workspace sharding works beautifully until a workspace stops being the unit of everything. The scheme has two structural problems and both get worse with success. First, a shard's load is set by its largest workspace, so one enormous customer makes one shard hot and no amount of rebalancing helps while the unit of placement is the workspace. Second, the mapping lives in the application: every service needs the metadata lookup, the routing rules and the resharding logic, and every new service re-implements them slightly differently.

Published figures give the scale that makes this acute: roughly 2.3 million queries per second at peak, about 2 million of them reads.

What they chose

A proxy layer rather than a library. Vitess puts routing, shard awareness and query rewriting behind the wire protocol, so an application connects to what looks like one MySQL and the topology becomes an operational concern rather than an application one.

That choice buys three things worth naming:

• Resharding becomes an operation rather than a project, because the layer that knows the mapping can move data and update routing without an application release.
• One implementation of the hard parts - routing, scatter-gather, connection limits, query consolidation - instead of one per service.
• Incremental migration. Traffic moves table by table and keyspace by keyspace, which is the only way a three-year migration can be survivable.

What it cost them

Three years, and a second data layer to operate for most of that time. The 99% figure from December 2020 is the honest shape of such a programme: the last per cent is the hardest, because the remaining workloads are the ones with odd access patterns, raw SQL, or a dependency on behaviour the abstraction changed.

There is also a permanent cost: a proxy is now on the critical path of every query. It adds a hop, it has its own failure modes and its own capacity to plan, and the team has to become expert in it. Features that depend on database behaviour the proxy does not expose have to be rewritten rather than ported.

Where copying this is a mistake

The decision rule is the number of services, not the number of queries. A proxy layer pays for itself when many independent services must agree on a sharding scheme; a single application with one data-access module gets most of the benefit from a routing library and none of the operational burden.

Below roughly a terabyte of data and ten thousand writes per second, the honest answer is still one primary with replicas: Slack's problem was not that MySQL was too slow, it was that the sharding logic was spread across an organisation. Copying the destination without having that problem buys the costs and none of the benefit. And the timeline is the part most often under-quoted to a steering committee: a multi-year programme needs a sponsor who will still be there at the end.

Common weak answers

• "They outgrew MySQL." They kept MySQL. What changed was where the sharding knowledge lived.
• "Sharding by workspace was a mistake." It was the right first design and it was still serving millions of queries per second. The failure mode was organisational scaling, not a modelling error.
• "They should have gone to a distributed SQL engine instead." A defensible alternative, and it trades a three-year incremental migration for a rewrite of the data layer in one step - a larger single bet, with a smaller operational surface at the end.