DynamoDB has evolved through adaptive capacity, on-demand mode and global tables. Which failure mode did each address, and what does the sequence teach about designing partitioned systems?
Show the full answer Hide the answer
The original failure mode
Provisioned throughput was divided evenly across partitions. A table with 1,000 write units across 10 partitions gave each partition 100 — and a workload whose keys were not uniformly distributed throttled on the hot partition while the table as a whole was mostly idle.
This is the fundamental problem of any partitioned system: capacity is provisioned globally and consumed locally. The user sees "throttled" on a table they have paid to be fast.
What each change addressed
- Adaptive capacity — the service reallocates unused throughput toward hot partitions, and later splits partitions by observed load rather than only by size. This makes the common case of mild skew invisible. It does not solve a genuinely single hot key, because one key cannot be split.
- On-demand mode — removes provisioning entirely, scaling with observed traffic. This addresses a different failure: capacity planning being wrong for spiky or unpredictable workloads, where the choice was overprovisioning permanently or throttling at the peak. It costs more per request and buys away an entire category of operational error.
- Global tables — multi-region replication with last-writer-wins conflict resolution. This addresses regional availability and read latency, and quietly imports a consistency model that many applications cannot tolerate: concurrent writes to the same item in two regions silently lose one.
What the sequence teaches
Uniform provisioning over non-uniform demand is the recurring failure of partitioned systems, and the mitigations arrive in a predictable order: rebalance within the existing partitions → split by load rather than size → remove the provisioning decision entirely.
None of them solve a single hot key, which remains an application-level problem. The answer there is always the same shape: change the key. Add a bounded random suffix and scatter-gather on read, pre-split a counter into buckets summed at read time, or move the contended item into a purpose-built in-memory service with async persistence.
And the last item is the important one for architects: global tables' last-writer-wins is a data model decision presented as an infrastructure feature. Adopting it for a counter, a balance, or a set silently produces wrong answers under concurrent regional writes — and the wrongness is invisible until someone reconciles.