Discord: Hot Partitions at Trillions of Messages

The problem

Discord stores an extremely large volume of messages — publicly described as passing a trillion — and the natural partition key is the channel, since messages are almost always read by channel.

Partitioning by channel alone has a defect that only appears in production: channel activity is enormously skewed. Most channels are quiet. A small number are extremely active, and a single very large community's main channel can generate more traffic than thousands of ordinary ones combined.

A partition is served by specific nodes, so a hot channel concentrates read and write load on those nodes regardless of how much total capacity the cluster has. Adding nodes does not help, because the partition does not split.

What they did

The partition key combines channel identifier with a time bucket. A given partition holds one channel's messages for a bounded period, so even the busiest channel's data is spread across many partitions over time.

This preserves the access pattern — reading recent messages in a channel touches one or a small number of partitions — while bounding how large and how hot any single partition can become.

They also publicly documented migrating the underlying store, having encountered latency and operational characteristics they could not resolve within the original engine at their scale.

The trade-off

Compound keys make some queries harder. Reading a long history now spans multiple partitions and requires the client to iterate buckets. Bucket size is a tuning decision: too large and hot partitions return, too small and ordinary reads fan out across many partitions.

The transferable lesson

Model your key distribution with real data before choosing a partition key, and assume skew. Almost every real-world distribution is skewed — channels, tenants, products, users, geographic regions — and a key chosen from the logical model rather than from the distribution will produce a hot partition eventually.

The general remedy is a compound key that adds a dimension along which the hot entity spreads: time, a bounded random suffix, or a sub-entity. Each has a cost in query complexity or ordering, and choosing between them requires knowing which property you actually need to preserve.