intermediate 2 min answer

A visual discovery platform serves enormous read volume over user-curated collections, with writes that are rare by comparison. A team proposes sharding the primary database immediately. What lower-complexity options should be evaluated first, and what evidence would justify sharding?

read-heavyshardingcachingreplicaspinterestwhat-would-you-change
Show the full answer Hide the answer

The ladder, cheapest first

1. Find out what is actually slow. A surprising share of "we need to shard" conversations end when someone looks at the top ten queries by total time. The usual finding is one or two access patterns dominating, often a missing index or an N+1 that generates thousands of small queries per page.

2. Indexes and query shape. A composite index matching the dominant filter and sort, covering indexes that avoid the table lookup entirely, and removing indexes nobody uses (each one taxes every write). This routinely buys an order of magnitude for the cost of a migration.

3. Read replicas. For a workload this read-skewed, replicas are the single highest-leverage move. They scale reads nearly linearly, are operationally well-understood, and are reversible. The cost is replication lag, which forces one decision: which reads must be served from the primary. Usually the answer is "immediately after the user's own write, and almost nothing else" — solvable by routing a user to the primary for a short window after they write.

4. Caching the hot set. Read-heavy discovery workloads have extreme popularity skew; a modest cache absorbs a large fraction of reads. Cache-aside with sensible TTLs first, before anything more elaborate.

5. Denormalisation of the read path. Precompute the shapes actually rendered — a materialised collection view rather than a five-way join at request time.

6. Only then, sharding. And even then, consider vertical partitioning by table group first: moving one high-volume table family onto its own database is far simpler than sharding a single table.

The evidence that actually justifies sharding

Sharding is justified when a single primary cannot absorb the write volume or hold the working set, and that is a specific, measurable claim:

  • Write throughput approaching the primary's sustained ceiling, with no remaining low-cost wins.
  • Working set materially exceeding memory, so the buffer cache hit rate has collapsed and I/O dominates.
  • A dataset whose size makes routine operations — backup, restore, schema migration, index rebuild — operationally infeasible within the maintenance window.
  • Vertical scaling exhausted, meaning the largest available instance is already in use.

Note what is not on the list: read volume. Read volume is answered by replicas and caches, and sharding for reads is a common, expensive category error.

What sharding costs, so the trade is honest

Cross-shard queries and joins become application code. Transactions no longer span the data. Unique constraints across shards require a separate mechanism. Rebalancing is a project. Every operational runbook multiplies. And the shard key becomes an effectively irreversible decision — changing it later means rewriting the dataset.

The judgement being tested

Whether you can resist the most common failure mode in data architecture: adopting the solution for a problem one order of magnitude larger than the one you actually have, and paying its complexity every day until you get there — if you ever do.