Evidence ledger
32 sources
Checked 16 Oct 2026
Evidence ledger
One row per claim in The workspace stopped being the shard: ten years of Slack: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Dig: The workspace stopped being the shard — Slack's architecture, 2016 to 2026. Research date: 2026-10-16. All rows checked 2026-10-16.
Retrieval limitation, stated up front. This session's egress policy blocked direct HTTP
fetches to every host, including slack.engineering; WebFetch could not resolve DNS. Two
channels worked: search-engine retrieval, which returns excerpts and quoted fragments of a page
rather than the whole document, and anonymous git reads of public GitHub repositories through
the session's git proxy. Consequences: rows graded source were read first-hand from git
history (commit authors, dates, messages, branch names, merge-base checks); every other row rests
on retrieved excerpts of the primary page, and quotes below are reproduced only where the
retrieval returned them as quotation. The HTTP link check in verify.mjs could not run, so links
are recorded as indexed, not re-checked. Where a figure comes only from a secondary summary, the
row says so and the page labels it.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Slack (Mike Demmer) | Scaling Slack: The Good, the Unexpected, and the Road Ahead (QCon SF) | talk | 2018-11 | 2026-10-16 | https://www.infoq.com/presentations/slack-scalability-2018/ | The 2016 design put each workspace on one DB shard and one message-server shard | talk abstract: three changes, "decomposition of the real-time message service, client-side lazy loading via edge caching, and fine-grained horizontal sharding using Vitess"; InfoQ summary: "assigning each new workspace to one database shard and one message server shard" (retrieved summary) |
| 2 | Slack | Scaling Slack, QCon SF 2018 slides | talk | 2018-11 | 2026-10-16 | https://qconsf.com/sf2018/system/files/presentation-slides/scaling_slack_-_qcon_2018.pdf | 2016 shape: Java message server, PHP webapp, MySQL; client downloaded the full workspace model at login | slides define the workspace as the unit holding channels, messages and users; section "Workspace Sharding" (retrieved excerpt) |
| 3 | Slack | How Slack Built Shared Channels | blog | n.d. (excerpt undated) | 2026-10-16 | https://slack.engineering/how-slack-built-shared-channels/ | Shared channels broke the founding partition assumption | "challenged Slack's fundamental assumption that the workspace is the atomic unit of partitioning customer data"; Slack "had been built with the assumption that channels (and everything else ...) belonged to only one team" |
| 4 | Slack | Scaling Datastores at Slack with Vitess | casestudy | 2020-12 | 2026-10-16 | https://slack.engineering/scaling-datastores-at-slack-with-vitess/ | Migration began 2017; Vitess serves 99% of query load; 2.3M QPS peak; median 2 ms, p99 11 ms | "99% of our overall query load"; peak 2.3 million QPS; median 2 ms, p99 11 ms (retrieved excerpt) |
| 5 | Slack | Scaling Datastores at Slack with Vitess | casestudy | 2020-12 | 2026-10-16 | https://slack.engineering/scaling-datastores-at-slack-with-vitess/ | Messages table sharded by channel ID instead of workspace | messages "sharded by channel ID" (retrieved excerpt via later post) |
| 6 | Simon Willison | Scaling datastores at Slack with Vitess (link note) | blog | 2020-12-01 | 2026-10-16 | https://simonwillison.net/2020/Dec/1/scaling-datastores-slack-vitess | Dates the Vitess post | dated 2020-12-01 |
| 7 | Percona | Percona Live 2018: Migrating to Vitess at (Slack) Scale | blog | 2018 | 2026-10-16 | https://www.percona.com/blog/percona-live-2018-migrating-to-vitess-at-slack-scale/ | Slack moved from application-level shard routing and manual administration to Vitess topology management | "simple application-based shard routing and manual administration" (retrieved summary) |
| 8 | CNCF | Slack case study | vendor | n.d. | 2026-10-16 | https://www.cncf.io/case-studies/slack/ | Workspace sharding concentrated a customer on one cluster with no headroom | all load from a customer on a single database cluster; nowhere to go when it outgrew one host (retrieved summary) |
| 9 | Slack (Laura Nolan) | Slack's Incident on 2-22-22 | postmortem | 2022-04-26 | 2026-10-16 | https://slack.engineering/slacks-incident-on-2-22-22/ | User-sharded membership turned every cache miss into a scatter; load superlinear in miss rate | "Since the data is sharded by user ID, even one channel missing from cache meant the application had to successfully run a query on every shard"; "read load increased superlinearly in relation to the percentage of cache misses" |
| 10 | Slack (Laura Nolan) | Slack's Incident on 2-22-22 | postmortem | 2022-04-26 | 2026-10-16 | https://slack.engineering/slacks-incident-on-2-22-22/ | Slack names the channel-keyed copy as the efficient keyspace; fix also read from replicas | "a keyspace that has membership of a channel sharded by channel would have been more efficient for this query"; "the query was also updated to read from replicas as well as Vitess primaries" (excerpt via HN quote) |
| 11 | Slack (Laura Nolan) | Slack's Incident on 2-22-22 | postmortem | 2022-04-26 | 2026-10-16 | https://slack.engineering/slacks-incident-on-2-22-22/ | Recovery by throttling client boots; first step up too large | "Our initial attempt increased the limit by too much. Database load increased again beyond sustainable limits" |
| 12 | Slack | Status history, Tuesday February 22, 2022 | postmortem | 2022-02-22 | 2026-10-16 | https://slack-status.com/2022-02-22 | Window 6:00 to 9:14 AM PST; trigger a configuration change | outage 6:00 AM to 9:14 AM PST; "configuration change that caused a sudden spike in database activity" (retrieved summary) |
| 13 | yfractal (Medium) | Slack's Incident on 2-22-22 Reading Note | blog | 2022 | 2026-10-16 | https://medium.com/@yfractal/slacks-incident-on-2-22-22-reading-note-57c77d1c4e10 | Trigger chain: Consul agent upgrade restarted memcached nodes, cache hit rate fell (secondary) | "upgrade Consul agent => need restart memcached node => cache hit decreased" |
| 14 | Slack (Laura Nolan) | A Terrible, Horrible, No-Good, Very Bad Day at Slack | postmortem | 2020-07 | 2026-10-16 | https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ | May 12 2020: HAProxy slot-state sync failed when slots ran out; scale-down removed the old instances HAProxy still knew | sync program could not "find any empty slots", so running HAProxy instances "weren't getting their state updated"; instance count up 75% during incident (retrieved excerpts) |
| 15 | Slack | Status history, Tuesday May 12, 2020 | postmortem | 2020-05-12 | 2026-10-16 | https://slack-status.com/2020-05/147dad376c8946ff | Customer-visible window 4:45 to 5:33 PM PDT | 4:45 p.m. to 5:33 p.m. PDT (retrieved summary) |
| 16 | Slack (Laura Nolan) | Slack's Outage on January 4th 2021 | postmortem | 2021-02-01 | 2026-10-16 | https://slack.engineering/slacks-outage-on-january-4th-2021/ | TGWs did not scale; provision-service degraded by the problem it was fixing; monitoring lost | "Our TGWs did not scale fast enough"; open files limit "the most significant bottleneck", plus an AWS quota; dashboards unavailable (retrieved excerpts) |
| 17 | The Register | Slack fingers AWS auto-scaling failure in January outage postmortem | blog | 2021-02-02 | 2026-10-16 | https://www.theregister.com/2021/02/02/slack_outage_aws_autoscaling/ | Slack attempted to add 1,200 web servers; monitoring sat in its own VPC behind the TGWs | "We attempted to add 1,200 servers to our web tier" |
| 18 | Lorin Hochstein | Slack's Jan 2021 outage: a tale of saturation | blog | 2021-02-08 | 2026-10-16 | https://surfingcomplexity.blog/2021/02/08/slacks-jan-2021-outage-a-tale-of-saturation/ | Independent analysis framing the incident as saturation across layers | title and framing (retrieved) |
| 19 | Slack | Slack's Migration to a Cellular Architecture | blog | 2023-08 | 2026-10-16 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | June 30 2021 intermittent AZ link failure motivated AZ-siloed cells over 1.5 years; request path in Hack, Go, Java and C++ | each AZ "contains a completely siloed backend deployment"; services in the request path "written in Hack, Go, Java, and C++" (retrieved excerpts) |
| 20 | InfoQ | Slack Migrates to Cell-Based Architecture on AWS | blog | 2024-01 | 2026-10-16 | https://www.infoq.com/news/2024/01/slack-cell-based-architecture | Drain within 5 minutes at 1% granularity; 99.99% SLA | "incrementally draining all the traffic away from the affected availability zone within 5 minutes"; "1% granularity" (secondary summary) |
| 21 | Microsoft Research et al. (Huang et al.) | Gray Failure: The Achilles' Heel of Cloud-Scale Systems (HotOS '17) | paper | 2017-05 | 2026-10-16 | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | Differential observability: detectors see health while applications see failure, exactly Slack's June 2021 split view | the system's failure detectors "may not notice problems even when applications are afflicted" (retrieved summary) |
| 22 | USENIX (Huang et al.) | Metastable Failures in the Wild (OSDI '22) | paper | 2022-07 | 2026-10-16 | https://www.usenix.org/conference/osdi22/presentation/huang-lexiang | 22 metastable failures across 11 organisations; retries the most common sustaining effect; ~45% of triggers engineer errors | "22 metastable failures from 11 different organizations" (abstract excerpt); retry finding from talk slides |
| 23 | Slack | Scaling Slack's Job Queue | blog | 2017-12 | 2026-10-16 | https://slack.engineering/scaling-slacks-job-queue/ | Redis hit its memory limit during DB contention and could not dequeue; Kafka placed in front of Redis; 1.4B jobs/day, 33k/s peak | "over 1.4 billion jobs at a peak rate of 33,000 per second" (retrieved excerpt) |
| 24 | Slack (Bing Wei) | Flannel: An Application-Level Edge Cache to Make Slack Scale | blog | 2017-05-31 | 2026-10-16 | https://slack.engineering/flannel-an-application-level-edge-cache-to-make-slack-scale | Large teams made full-model boot unworkable; lazy load at the edge | "the number of channels, users and bots become unwieldy" (retrieved excerpt) |
| 25 | StackShare | Flannel summary | blog | 2017-05-31 | 2026-10-16 | https://stackshare.io/posts/102366439899855989 | Flannel at edge since January 2017; 4M simultaneous connections, 600k client QPS (secondary) | "4 million simultaneous connections at peak and 600k client queries per second" |
| 26 | Slack (Keith Adams) | Taking PHP Seriously | blog | 2016-10-13 | 2026-10-16 | https://slack.engineering/taking-php-seriously/ | Slack chose to evolve PHP (via Hack) rather than rewrite | essay arguing PHP's request model; Hack adds gradual typing |
| 27 | Slack (Scott Sandler) | Hacklang at Slack: A Better PHP | blog | 2020-04 | 2026-10-16 | https://slack.engineering/hacklang-at-slack-a-better-php/ | Moved to HHVM in 2016 for speed, stayed for Hack's incremental types | switched to HHVM in 2016 "mainly because it ran the existing PHP code faster" (retrieved summary) |
| 28 | Slack | Deploys at Slack | blog | 2020-04 | 2026-10-16 | https://slack.engineering/deploys-at-slack/ | ~12 scheduled deploys/day; dogfood, 2% canary, then 10/25/50/75/100 | "about 12 scheduled deploys a day"; canary "about 2% of production traffic" (retrieved excerpt) |
| 29 | Slack | Migrating Millions of Concurrent Websockets to Envoy | blog | 2021-03 | 2026-10-16 | https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/ | wss-primary/wss-backup moved from dedicated HAProxy fleets to Envoy | "wss-primary and wss-backup" behind HAProxy in multiple AWS regions (retrieved excerpt) |
| 30 | Slack | Real-time Messaging | blog | 2023-04 | 2026-10-16 | https://slack.engineering/real-time-messaging/ | Channel servers hash-assigned; CHARMs replace an unhealthy CS within 20s; ~16M channels per host at peak | "replacement of unhealthy CSs within 20 seconds"; "16 million channels per host" (retrieved excerpt) |
| 31 | Slack | Unified Grid: How We Re-Architected Slack for Our Largest Customers | blog | 2024 | 2026-10-16 | https://slack.engineering/unified-grid-how-we-re-architected-slack-for-our-largest-customers/ | Re-oriented around user and org; Enterprise Grid (2017) split orgs into workspaces | re-oriented "around the user and organization, instead of the workspace" (retrieved summary) |
| 32 | Slack (Ian Hoffman) | Changing the Model: Why and How We Re-Architected Slack (QCon SF 2024) | talk | 2024-11 (InfoQ 2025-04-18) | 2026-10-16 | https://www.infoq.com/presentations/slack-rearchitecture/ | Token now names the org, not the workspace shard; server checks org shard plus member workspace shards | "the API token no longer determines the workspace shard"; checks "only workspaces the user belongs to" (third-party write-up of the talk) |
| 33 | Slack (Cooper Bethea) | Slack's Migration to a Cellular Architecture (QCon SF 2024) | talk | 2024-11 | 2026-10-16 | https://www.infoq.com/presentations/slack-cellular-architecture/ | Talk version of the cells migration, led by Bethea | session listing |
| 34 | Slack | The Query Strikes Again | postmortem | 2022-10 incident | 2026-10-16 | https://slack.engineering/the-query-strikes-again/ | A long-running async purge job overloaded the datastore on two consecutive days; shimming the job | "a long-running job (async job)"; "an edge-case bug in Datastores automation which led to failure to handle a flood of requests" |
| 35 | Slack | Status history, Wednesday February 26, 2025 | postmortem | 2025-02-26 | 2026-10-16 | https://slack-status.com/2025-02-26 | Maintenance action plus a caching defect overloaded a database; ~50% of instances relying on it unavailable; 6:45 AM to 4:13 PM PST | roughly 50% of instances relying on this database became unavailable (retrieved summary) |
| 36 | Slack | Applying Product Thinking to Slack's Internal Compute Platform | blog | 2024 | 2026-10-16 | https://slack.engineering/applying-product-thinking-to-slacks-internal-compute-platform/ | Bedrock: build code, package a container, allocate compute | (retrieved summary) |
| 37 | InfoQ | Slack EKS and Karpenter | blog | 2024-05 | 2026-10-16 | https://www.infoq.com/news/2024/05/slack-eks-karpenter | Bedrock on EKS + Karpenter; over 80% of Slack applications on it | "over 80% of Slack applications" (secondary summary) |
| 38 | Slack | Advancing Our Chef Infrastructure: Safety Without Disruption | blog | 2025-10-23 | 2026-10-16 | https://slack.engineering/advancing-our-chef-infrastructure-safety-without-disruption/ | Single prod Chef environment split into prod-1..prod-6 mapped by AZ; signal-driven runs; 12-hour fallback | "mapped each instance to one of these more specific environments based on its Availability Zone" |
| 39 | Slack | Shipyard: How We Built Slack's Next-Generation EC2 Platform | blog | 2026-07 | 2026-10-16 | https://slack.engineering/shipyard-how-we-built-slacks-next-generation-ec2-platform/ | Instances become deployable artifacts (layered AMIs), progressive rollout with metric rollback; data nodes are future work | "Future work targets long-lived instances like data nodes and singleton services" (aggregator excerpt) |
| 40 | Vitess (git) | vitessio/vitess commit history, Slack-domain authors | source | 2017-2026 | 2026-10-16 | https://github.com/vitessio/vitess | 1,950 commits by slack-corp.com addresses: 337 (2017), 451 (2018), 144 (2019), 181 (2020), 786 (2021), 18 (2022), 5 (2023), 2 (2024), 3 (2025), 13 (2026 to Oct 12) | measured: git log --format=%ad\|%ae --date=format:%Y \| grep slack-corp, on 35,801 commits |
| 41 | Vitess (git) | PR #4228 "partial-scatter" and #4293 "partial-scatter-with-warnings" from tinyspeck | source | 2018-10-27, 2018-10-29 | 2026-10-16 | https://github.com/vitessio/vitess/pull/4228 | Slack upstreamed a directive that returns partial scatter results as warnings | commit 2018-10-16 "rename the SHARD_PARTIAL directive to SCATTER_ERRORS_AS_WARNINGS" (Michael Demmer) |
| 42 | Vitess (git) | PR #8439 "no-scatter" from planetscale | source | 2021-07-13 | 2026-10-16 | https://github.com/vitessio/vitess/pull/8439 | Upstream later added an operator flag that refuses scatter plans | "fail executor if the no_scatter flag is enabled and the plan contains a scatter" |
| 43 | Slack (git) | tinyspeck/vitess fork: branch brirams_limited_scatter_query_planner | source | 2019-04-11 | 2026-10-16 | https://github.com/tinyspeck/vitess/tree/brirams_limited_scatter_query_planner | A 2019 Slack branch for a limited scatter planner never reached upstream main | git merge-base --is-ancestor → not in vitessio/vitess main; tip commits "some examples", "add the test output to show" |
| 44 | Slack (git) | tinyspeck/vitess fork: branch inventory | source | 2017-2022 | 2026-10-16 | https://github.com/tinyspeck/vitess | 88 slack-* branches: date-stamped sync/release branches 2017-2020, version-stamped (8, 9, v10, v11) in 2021; last tip 2022-03-29 | git ls-remote; tip of slack-vitess-v11.0.4.pre.v3 dated 2022-03-29 |
| 45 | Slack (git) | slackhq/hack-sql-fake | source | first commit 2019-05-14 | 2026-10-16 | https://github.com/slackhq/hack-sql-fake | In-memory MySQL simulation for Hack unit tests | "an in-memory simulation of MySQL" (README) |
| 46 | Slack (git) | slackhq/astra | source | first commit 2020-07-07; renamed 2024-04-04 | 2026-10-16 | https://github.com/slackhq/astra | Log search engine built in-house (as KalDB), renamed Astra in 2024; still active Oct 2026 | "Rename code/comment/metrics/config occurrences to Astra (#840)" 2024-04-04 |