Discord's Message Store Migrations

Three migrations, three distinct causes

MongoDB → Cassandra (published 2017, ~100 million messages). The working set stopped fitting in memory, and read latency became unpredictable. They needed a store with predictable write throughput and linear horizontal scaling for a workload that is overwhelmingly append-then-read-recent.

Cassandra → ScyllaDB (published 2023, trillions of messages). Cassandra worked but its operational profile did not. The specific pain was garbage collection pauses on the JVM causing latency spikes, and compaction falling behind under load. ScyllaDB — a C++ reimplementation of the Cassandra data model with a shard-per-core architecture and no garbage collector — removed both without a data model change, because the model was compatible.

The data model decision that made it work

The partition key is (channel_id, bucket) where bucket is a fixed time window. This is the substance of the case study, and it is a transferable pattern:

• Partitioning by channel alone would put every message of a busy channel in one enormous partition — a classic hot partition, unbounded in size.
• Adding a time bucket bounds partition size and matches the query pattern exactly, because clients read recent messages in a channel. The most recent bucket is hot, older buckets are cold, and cold buckets can be aged out.

Model for the read pattern, then bound the partition. That single sentence covers most wide-column data modelling.

The hot-partition problem they wrote about

Even with bucketing, a handful of very large channels behaved differently from everything else, and concurrent requests for the same hot partition amplified load. Their answer was request coalescing in an intermediary data service written in Rust: many concurrent requests for the same data result in one database query, whose result is fanned back out.

That is the single-flight pattern applied at the data tier, and it is the reusable fix for any system where popularity is heavily skewed — which is nearly all of them.

The architectural lesson

"We outgrew our database" is usually imprecise. Each migration here had a specific, named constraint: working set versus memory, then GC pauses and compaction throughput. Naming the constraint is what makes the choice defensible — and it is what tells you whether a different database is even the right kind of fix, since two of the three problems here were solved by data modelling and a caching layer rather than by the store.