Partitioning and Sharding

Definition

Two related things worth separating:

• Partitioning usually means splitting a table within one database — by range, list or hash — so that queries touch less data and maintenance operates on smaller units. The application does not change.
• Sharding means splitting across independent databases. The application must now know, or be able to derive, where data lives, and cross-shard operations become the application's problem.

Partitioning is a tuning decision. Sharding is an architectural one.

What partitioning buys cheaply

Time-ranged partitioning is the highest-value low-risk case: partition by month, and dropping old data becomes an instant metadata operation rather than a DELETE that generates enormous write load and bloat. Index maintenance, vacuuming and backups operate per partition. Queries that filter on the partition key skip the rest entirely.

What sharding buys, and what it costs

Bought: write throughput beyond one machine, dataset size beyond one machine, blast-radius isolation (one shard's incident affects a subset of users), and data residency placement.

Sold: joins across shards, cross-shard transactions, straightforward migrations (now N of them, in different states), cross-shard uniqueness, and cheap changes of mind.

Industry example

Slack partitions on the workspace, and it is a good illustration of choosing a boundary that already exists in the domain. Almost all activity — channels, messages, members, files — is contained within one workspace, so the overwhelming majority of queries route to a single shard and never fan out.

That produces three benefits simultaneously: performance (single-shard queries), isolation (a workspace's load and incidents are contained), and a natural unit for migration and residency. A large enterprise workspace can be moved to dedicated infrastructure without changing the model.

The general rule this illustrates: the best shard key is a boundary the business already has. If you have to invent one, you will fight it forever, because your queries will keep crossing it.

Failure scenarios

• Skew. One workspace, tenant or user is a large fraction of the data and load. This is the normal case, not an edge case, and the mitigation must be planned: dedicated shards for the largest, sub-sharding on a secondary dimension, or a directory that allows placement decisions.
• Cross-shard queries becoming common, which means the key is wrong and the system is now slower than the single database it replaced.
• Resharding under load with a plain hash, which remaps almost every key. Consistent hashing or a directory limits movement — build the mechanism before the crisis.
• Global uniqueness assumed where none exists, so two shards allocate the same user-visible identifier.

Interview question

"What is the difference between partitioning and sharding, and which one would you reach for when a table has grown to two billion rows but write throughput is fine?"