Slack was built in 2013 on one assumption: a workspace is a closed world, so it can live on one database shard and one message server. Over ten years, three product decisions broke that assumption. This guide rebuilds what Slack changed in response from its own postmortems and talks, from the Vitess commit history and from the branches of its Vitess fork. It shows where the old shard key kept causing outages after the migration was declared finished.
Why Slack, why now, and the one finding to carry out of this page.
Most writing about sharding treats the shard key as a database decision. Slack's record says it was a product decision, and so was every revision of it. In 2016, according to Mike Demmer's QCon talk, Slack gave each new workspace one database shard and one message-server shard, and almost everything a workspace did stayed on that shard. That was correct for a product where a channel, a user, an emoji and a file belonged to exactly one team. Then Slack shipped Enterprise Grid in 2017, which split one customer across many workspaces. Next came shared channels, later Slack Connect, which put one channel inside two companies. In 2023 and 2024 came Unified Grid, which made the organisation rather than the workspace the thing a user signs in to. Slack's own shared-channels post says the feature "challenged Slack's fundamental assumption that the workspace is the atomic unit of partitioning customer data".
Why this company, why now. Several earlier digs used Slack as a supporting example, for a scatter-read incident and for cells, but none covered it as a company. Its record is now unusually complete. Laura Nolan wrote three long postmortems. Two QCon SF 2024 talks cover the cellular migration and the Unified Grid re-architecture. Engineering posts as recent as October 2025 (Chef) and July 2026 (Shipyard) show the change path being partitioned the same way the request path was.
The worst datastore outage Slack has documented came after the Vitess migration was reported as 99% complete, and the migration caused it. In February 2022 a group-DM membership lookup ran against a table sharded by user ID, so every cache miss became "a query on every shard". Database load rose "superlinearly in relation to the percentage of cache misses" (Slack, 2022). The git record shows Slack had treated the scatter as something to tolerate since 2018, when it upstreamed a directive that returns partial scatter results as warnings. A 2019 branch for a "limited scatter query planner" on Slack's fork never reached Vitess main. In this case the cache was what kept the database within capacity, and nobody had written down a capacity budget for it.
Scope. Covers the server-side architecture: partitioning, datastores, job queue, edge, real-time, deploy and host management, and the published incidents. Does not cover the desktop and mobile clients, search ranking, AI features, data warehouse or Salesforce integration, and Slack's costs, none of which the public record describes at architecture depth.
The 2026 shape, reconstructed from Slack's own posts and talks. Each layer is attributed, and each still shows a trace of the workspace model.
Slack moved from PHP 5 to HHVM in 2016 "mainly because it ran the existing PHP code
faster". It then stayed on HHVM to adopt Hack's gradual types, so a large codebase
could be typed one file at a time instead of being rewritten. The in-memory MySQL fake
Slack published in 2019, hack-sql-fake, shows how deeply that codebase
assumes MySQL semantics.
Sources: Adams 2016, Sandler 2020, hack-sql-fake
The migration began in 2017. By December 2020, Vitess served 99% of queries at a 2.3M QPS peak, with a 2 ms median and 11 ms p99. Messages are sharded by channel ID. Some membership data is sharded by user ID. Org-level data has its own shards. Slack kept MySQL and replaced only the routing and topology layer.
Sources: Slack 2020, Percona Live 2018
No Slack post describes the cache tier as a component in its own right. The 2-22-22 postmortem shows what it does: it absorbs reads that would otherwise scatter to every shard. On that reading, it is a capacity component with no published capacity budget.
Inferred from: Nolan 2022
After Redis filled up and could not even dequeue, Slack put Kafka in front of it as a durable backlog. A Go service, Kafkagate, enqueues to Kafka. Redis was kept as the working queue. This was the minimum change that let Slack keep the job workers it already had. Throughput as of 2017 was 1.4B jobs a day, peaking at 33k per second.
Source: Slack 2017
Channel servers hold channel state in memory and are placed by consistent hashing. Ring managers (CHARMs) replace an unhealthy host within 20 seconds, and a host carries around 16M channels at peak. Gateway servers hold each user's subscriptions and run in several regions. This tier was keyed by channel from the start, so the workspace model mattered least here.
Source: Slack 2023
The webapp ships about 12 times a day: dogfood, then a 2% canary, then 10, 25, 50, 75 and 100%. Chef's single production environment was split into prod-1 to prod-6, mapped by availability zone. Shipyard (2026) turns EC2 hosts into immutable artifacts with metric-based rollback. Most applications run on Bedrock (EKS plus Karpenter).
Sources: deploys, Chef 2025, Shipyard 2026
Each layer adopted a different key, and the order matters. The real-time tier used channel keys from the start, so shared channels cost it little. The datastore moved to entity keys between 2017 and 2020. Request routing still read a workspace ID from the session token until Unified Grid. In the third-party write-up of Ian Hoffman's QCon talk, the token after Unified Grid "no longer determines the workspace shard". It names the org, and the server checks the org shard and then the shard of each workspace the user belongs to. On that description, Slack's highest-volume request path now does a bounded fan-out by design. It was the right trade for the product. It also means per-request shard count grows with how many workspaces a user is in, and Slack has not published that distribution.
What Slack chose, what it passed over and why, and the condition under which the choice would go the other way.
SCATTER_ERRORS_AS_WARNINGS (PRs
#4228 and #4293), so a scatter returns partial results when some shards fail.no_scatter
in 2021 (#8439). Nothing published says whether Slack enables it.| Decision | Chosen | Passed over | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Language of the webapp | PHP to HHVM to Hack, typed incrementally (2016) | Rewrite in another language | Speed first, then types without a rewrite | The runtime's community collapses; Slack still depends on HHVM in 2026 | Sandler 2020 |
| Job queue durability | Kafka in front of Redis (2017) | Replace Redis with Kafka | Smallest change that removes the memory ceiling and leaves workers alone | The worker fleet is being rewritten anyway | Slack 2017 |
| Client boot | Edge cache that lazy-loads (Flannel, 2017) | Full workspace model at login | Large teams made the full model unworkable | Teams stay small; the full model is simpler | Wei 2017 |
| Vitess: fork or upstream | Fork with date-stamped sync branches (2017 to 2021), then upstream releases | Stay on a long-lived fork | Inferred: the fork's cost grew with every upstream release | Upstream rejects a change you cannot live without | tinyspeck/vitess branches |
| Websocket edge | Envoy (2021) | Dedicated HAProxy fleets | Dynamic config without reloads; the 2020 outage was a reload workaround that failed | Backend sets change rarely | Slack 2021 |
| Request routing key | Org in the token, then fan out to member workspaces (Unified Grid) | Workspace in the token | Users belong to many workspaces; switching and duplicated client data cost more | Most users belong to one workspace | Slack 2024 |
| Config management rollout | Chef split into six AZ-mapped environments, run on signal | One production environment on cron | A bad cookbook reached every AZ at once | The fleet is one AZ | Slack 2025 |
No architecture decision records were published. Slack has not released ADRs or RFCs for any of these choices. Its reasoning appears only in blog posts and conference talks, which give the option chosen far more often than the options turned down. The "passed over" column above is the guide's reconstruction wherever a cell says "inferred". This is the main limit of the record: you can see what Slack did, but rarely what it argued against.
Seven published incidents, grouped by failure class rather than by date. Three of them come from the partition key, and three from glue that sat in the request path without anyone treating it as part of it.
None of them was a database losing data or a consensus bug. All three Class A incidents happened where a read keyed by one noun hit a table keyed by another, with a cache holding the difference. All three Class B incidents happened in machinery that was supposed to add or route capacity, and that machinery failed under the same conditions that made capacity necessary. The Huang et al. study of 22 metastable failures across 11 organisations (OSDI '22) found retries to be the most common sustaining effect. 2-22-22 fits that pattern: once the trigger passed, the overload continued on its own.
Corroborated across: OSDI '22, Hochstein 2021
Every figure with its date. Rows marked "secondary" come from a summary, not Slack's own page.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Datastore peak | 2.3M QPS | Slack | Vitess, median 2 ms, p99 11 ms | 2020-12 | Slack |
| Share of query load on Vitess | 99% | Slack | Migration started 2017 | 2020-12 | Slack |
| Async jobs | 1.4B / day | Slack | Peak 33,000 per second | 2017-12 | Slack |
| Edge cache connections | 4M | Slack | Flannel, 600k client QPS (secondary) | 2017 | StackShare |
| Channels per channel server | ~16M | Slack | Peak; CHARM replaces a host in <20 s | 2023 | Slack |
| Webapp deploys | ~12 / day | Slack | 2% canary then 10/25/50/75/100 | 2020 | Slack |
| AZ drain time | 5 min | Slack | 1% granularity, 99.99% SLA (secondary) | 2023 | InfoQ |
| Apps on Bedrock (EKS) | >80% | Slack | Secondary | 2024-05 | InfoQ |
| Emergency scale-up attempted | 1,200 hosts | Slack | Jan 2021, mostly never served | 2021-01 | The Register |
| Instances unavailable, Feb 2025 | ~50% | Slack | Of those relying on the affected DB | 2025-02 | Slack status |
| Slack-domain commits to Vitess | 1,950 | vitessio/vitess | Peak 786 in 2021, 18 in 2022, 13 in 2026 to Oct 12 | 2026-10 | git log, measured |
| Slack fork branches | 88 | tinyspeck/vitess | slack-* branches; last versioned tip 2022-03-29 | 2026-10 | git ls-remote, measured |
Measured by this guide: the two git rows. They count commit author addresses
ending in slack-corp.com, so they miss Slack engineers who committed under
personal addresses (Demmer also appears as demmer@gmail.com), and they
undercount. Reported by Slack: the datastore, job, real-time and deploy rows.
Secondary: Flannel, drain time and Bedrock share. Unknown, and the figures
an architect most needs: the cache hit rate the membership path needs to stay within
capacity, the per-request shard fan-out after Unified Grid, and any cost figure. Slack
has published none of these. Every Slack figure here is at least a year old, apart from
the Chef and Shipyard design descriptions.
A derived figure, labelled as such. Suppose a membership read normally hits cache and a miss fans out to every shard. Database reads per request are then roughly m × S, where m is the miss rate and S the shard count. A keyspace sharded by channel would cost m × 1. On 100 shards, a drop in hit rate from 99% to 90% multiplies membership load on the database tenfold, from 1 to 10 shard-reads per request, against 0.01 to 0.1 for the channel-keyed copy. The arithmetic is linear in m. Slack observed superlinear growth, which only happens if per-query latency also rises with load and retries add queries. That is the metastable loop, and it is this guide's inference, not a figure Slack published.
Three claims, each tied to evidence, and the rule they add up to.
1. The shard key encodes the product's ownership model, and only product decisions change it. All three moves of Slack's partition (workspace, then entity, then org) followed a product launch. None of them followed a database limit on its own. The CNCF case study shows that the workspace key had already run out of room for the largest customers. Slack's own shared-channels post shows the key stopped matching the data once channels crossed companies. When product managers plan a feature that lets an object belong to two of anything, the database team should be in the room.
2. A migration is not done when the queries move. It is done when no read crosses the old key. The Vitess post reported 99% of queries migrated in December 2020. Fourteen months later, the worst datastore incident in the record came from a read that still asked by channel against data now keyed by user. Slack's 2018 scatter directive and its unmerged 2019 limited-scatter branch show the cost was known and treated as an availability problem, by returning partial results, rather than a capacity problem. The postmortem's own remedy points to a second copy keyed by channel. When you re-key a table, list every query that reads it by a different noun. Each one needs a second keyspace, a capacity budget for the cache in front of it, or a test that runs it with the cache cold.
3. Slack ended up with three partitions, and each answers a different question. Data is partitioned by entity, which decides where a row lives. Traffic is partitioned by availability zone, which decides where a failure can spread. Change is partitioned by availability zone as well: Chef prod-1 to prod-6, progressive deploys, Shipyard. That one decides how far a bad change can reach. Slack did not plan this. Each partition was added after an incident: the job queue in 2017, the AZ gray failure in 2021, and the cookbook blast radius described in the 2025 Chef post. Cells did not remove Class A risk, because Vitess keyspaces still span zones. A cell drain cannot fix a scatter storm.
Choose your partition keys for the ownership model the product will have, not the one it has today. Keep the data, failure and change partitions as separate design decisions. Then, for every read that crosses a partition, write down what absorbs the fan-out and what that component's capacity is. At Slack the unwritten answer was "the cache". Its budget was discovered on 22 February 2022.
Every source behind this page, graded. Non-git sources were read through search-engine excerpts because direct fetches were blocked in this session; see the ledger.
A membership table sharded by user turned each cache miss into a query on every shard, and load grew superlinearly with the miss rate. The postmortem names a channel-keyed keyspace as the efficient alternative.
Transit Gateways saturated. Provisioning, autoscaling and monitoring all depended on the same network.
HAProxy server-slot sync exited early when no slot was free. State went stale, and scale-down then removed live backends.
A long-running purge job overloaded the datastore on two consecutive days. The job was shimmed to stop it.
Database maintenance and a caching latency defect together left about half of dependent instances unavailable for most of a day.
The customer-facing window, 6:00 to 9:14 AM PST, and a trigger described as a configuration change.
1,950 commits from slack-corp.com addresses, peaking at 786 in 2021 (VTAdmin, workflow RPCs) and falling to 18 in 2022.
Slack's directive, renamed from SHARD_PARTIAL to SCATTER_ERRORS_AS_WARNINGS, returns partial results when some shards fail.
An operator flag that fails any plan containing a scatter. It was added upstream three years after Slack's partial-scatter directive.
A 2019 Slack branch for limiting scatter in the planner. A merge-base check shows it never reached Vitess main. The guide found no PR record for it.
88 slack-* branches. Date-stamped sync and release branches run from 2017 to 2020, then version-stamped ones (8, 9, v10, v11) in 2021, ending 2022-03-29.
An in-memory MySQL simulation for Hack unit tests, still maintained in 2026.
A log and trace search engine Slack built itself, started as KalDB in 2020 and renamed in April 2024. Its goals include "coordination free ingestion".
Migration from 2017, 99% of query load, 2.3M QPS peak, 2 ms median and 11 ms p99. Messages re-keyed by channel.
States outright that shared channels broke the assumption that the workspace is the atomic unit of partitioning.
The June 2021 gray failure, AZ-siloed cells, and a request path written in four languages.
Slack was re-oriented around the user and the organisation instead of the workspace, after Enterprise Grid made users members of many workspaces.
Redis filled up and could not dequeue. Kafka was put in front as a durable backlog. 1.4B jobs a day.
Production Chef split into six AZ-mapped environments, with runs triggered by signal and a 12-hour fallback.
Channel servers placed by consistent hash, CHARM replacement in under 20 seconds, around 16M channels per host.
HHVM in 2016 for speed, then Hack for incremental types.
Large teams made it unworkable to download the full model at boot, so data is loaded lazily at the edge.
About 12 deploys a day, with dogfood, a 2% canary and percentage steps.
The websocket edge moved from dedicated HAProxy fleets to Envoy.
Hosts become immutable layered AMIs with metric-based rollback. Data nodes are named as future work.
An independent reading of the January 2021 postmortem as saturation spreading across layers.
Differential observability: failure detectors see health while applications see failure. Slack's June 2021 incident fits this exactly.
22 failures across 11 organisations, with retries the most common sustaining effect. This matches how 2-22-22 behaved.
The workspace-sharded starting point, and the three changes: decomposing the message server, edge caching, and Vitess.
The token names the org, and the server checks the org shard plus member workspace shards. A daily-use prototype came first.
The talk version of the cells migration, from the engineer who led it.
Workspace sharding put all of a customer's load on one cluster, with nowhere to go when it outgrew a host.
Reproduce Slack's partition migration and its failure mode at a scale you can break on a laptop.
Four MySQL or Postgres instances. Route each workspace to one by hash of workspace ID, and store channels and memberships there.
Done when: every API call touches exactly one database. Teaches: why the 2013 design was right for its product.
Let one channel belong to two workspaces on different shards. Try putting it on one side, then copying it to both.
Done when: you can name the read that becomes cross-shard under each option. Teaches: the assumption shared channels broke.
Put Vitess (or a thin router) in front. Shard messages by channel and membership by user, and move the data with dual writes and read diffing.
Done when: the router reports zero tenant-keyed queries. Teaches: "migrated" counts queries, not access patterns.
Cache group-membership lookups. Under load, restart half the cache nodes and watch per-shard QPS. Add client retries with backoff.
Done when: you have a curve of database load against miss rate, including the knee. Teaches: the 2-22-22 mechanism and the miss-rate budget.
First add admission control on client boot. Then add a membership copy sharded by channel, kept in sync from the primary write, and rerun the cache restart.
Done when: the cold-cache curve stays flat with the second keyspace. Teaches: shedding load buys time; a second key removes the problem.
Run two copies of the stateless tier in two "zones" behind Envoy with weighted clusters. Inject packet loss only between zones, then drain one in 1% steps.
Done when: a drain completes with no client-visible errors and depends on nothing in the drained zone. Teaches: gray failure, and why the drain must be out-of-band.
Make every config or image change roll out one zone at a time, triggered by a signal, with automatic rollback on an error-rate metric.
Done when: a deliberately bad config never reaches the second zone. Teaches: the third partition, the one Slack added last.
The queries and commands that found the material above.
slack.engineering incident "sharded by user ID" cacheslack.engineering "workspace is the atomic unit"slack-status.com 2025-02-26qconsf.com Slack 2024 re-architected cellulargit log --format='%ad|%ae' --date=format:%Y | grep slack-corp | cut -d'|' -f1 | sort | uniq -cgit ls-remote https://github.com/tinyspeck/vitess | grep refs/heads/slackgit merge-base --is-ancestor ts/brirams_limited_scatter_query_planner HEADgit log --oneline | grep -iE 'scatter.*(warning|limit)|no_scatter'