Sharding Patterns
Directory versus embedded keys, logical shards and rebalancing.
3 to work through
-
advanced
A multi-tenant database is being sharded. One tenant holds 40% of the data and generates 60% of the queries. What is your shard key and what do you do about that tenant?
3 min answer -
advanced Multiple choice
A product must shard its primary data store. Which sharding pattern fits, and which decision is irreversible?
2 min answer -
advanced
Notion sharded its Postgres estate into 480 logical shards in 2021 and in 2023 moved those shards onto a larger number of physical machines. A team that shards into 32 databases and later needs 96 faces a far harder job. What did the 2021 decision buy and what does it cost to maintain?
3 min answer
3 terms in this topic
Logical Shard
A unit of partitioned data that routing targets by name rather than by machine, so data can be moved between physical hosts without changing how anyt…
practiceShard Key Selection
Choosing the attribute that determines a record's partition, which fixes the system's distribution, query patterns and future flexibility.
patternSharding Patterns
Splitting data across independent stores by a key, and the choice of that key — which is the decision that will define the system for years.
Neighbouring topics
Architecture Patterns
General material on architectural patterns and their trade-offs.
Layered Architecture
The default shape, its clarity, and where a technical partition fails.
Event-Driven Architecture
Publishing facts, and trading comprehensibility for decoupling.
Pipes and Filters
Independent transformation steps composed into a pipeline.
Publish/Subscribe
Broadcast to every subscriber, as distinct from work distribution.
Competing Consumers
Scaling throughput with instances, at the cost of ordering.
Saga
Local transactions with compensating actions across services.
CQRS
Separate models for writing and reading, each optimised for its job.
Event Sourcing
The event log as the system of record, with state as a projection.
Outbox
Making the event atomic with the business write it describes.
Strangler Fig
Incremental replacement behind a routing facade.
Anti-Corruption Layer
Translating a foreign model at the boundary so it does not leak in.
Sidecar & Ambassador
Cross-cutting behaviour in a co-deployed process.
Service Mesh
Sidecars applied estate-wide, and the scale at which that pays.
API Gateway
A single entry point for policy, routing and protocol translation.
Backend for Frontend
A narrow backend per client experience, owned by that client's team.
Cell-Based Architecture
Complete isolated copies each serving a subset of customers.
Materialized Views
Precomputed query results, refreshed incrementally or in full.
Orchestration vs Choreography
A coordinator that knows the flow, or services that react to events.