Hot Key

Partitioned systems distribute data by key and assume demand is roughly uniform across keys. Real workloads are not uniform: one product goes viral, one celebrity has fifty million followers, one voucher code is redeemed by everyone at the same second, one tenant is a thousand times larger than the median.

The result is that one partition is saturated while the cluster as a whole is mostly idle, and adding nodes does not help because the traffic is destined for a key that lives on exactly one of them.

Why it matters

Hot keys defeat the central promise of horizontal scaling. The dashboards show low average utilisation, the capacity model says there is headroom, and users see throttling and timeouts — a combination that consistently produces the wrong response, which is to add capacity that goes unused.

It is also a latent property of the data, not of the code. The system works for years and then a single event — a promotion, a launch, a public figure joining — makes an existing key hot, with no deployment involved.

Implementation patterns

• Change the key, which is the only structural fix. Append a bounded random or hashed suffix to spread writes across N sub-keys, and scatter-gather across all N on read. This trades read amplification for write distribution.
• Pre-split counters into buckets, each incremented independently, summed on read. Exact totals, no contention, at the cost of a fan-out read.
• Split by observed load rather than by size, so the storage layer reacts to traffic skew automatically — this handles mild skew invisibly and cannot help a single key.
• A dedicated in-memory reservation service for the extreme case: the contended item is held in memory, decisions are made at memory speed, and persistence is asynchronous. This is how limited inventory and voucher redemption are made to work at flash-sale scale.
• Read-path caching with request coalescing, so a hot read key is served from cache and simultaneous misses collapse into one origin fetch rather than a stampede.
• Hybrid fan-out for the celebrity case — write-time fan-out for ordinary accounts, read-time merge for the few accounts whose follower count makes write-time fan-out untenable.
• Detection before mitigation: per-key traffic sampling and top-K tracking, because a hot key that is not measured is diagnosed by guessing.

Industry example

DynamoDB's evolution is largely a sequence of responses to this problem. Provisioned throughput was originally divided evenly across partitions, so skewed access throttled on one partition while the table was idle overall. Adaptive capacity reallocates unused throughput toward hot partitions and later split partitions by observed load. On-demand mode removes the provisioning decision entirely.

None of them solve a genuinely single hot key, and the documentation is direct about this: the remedy is a different key design in the application. That boundary — what the platform can absorb versus what only the data model can fix — is the durable lesson.

Failure scenarios

• A monotonically increasing partition key (timestamp, sequential ID), which makes the current partition permanently hot for writes while all others are cold.
• A single global counter or balance row under high concurrency, serialising every writer on one lock.
• A voucher or inventory row at the moment a promotion opens.
• A celebrity account in a write-time fan-out design.
• A tenant identifier as the shard key with one enormous tenant, so multi-tenancy provides no isolation for the largest customer.
• Cache expiry synchronised across clients, producing a stampede onto one key at the same instant.

Trade-offs

Every mitigation costs something. Suffix-and-scatter multiplies read cost by the fan-out factor and complicates any query that needs the exact set. Bucketed counters make the total a computed value rather than a stored one, which is fine for a view count and awkward for a balance. In-memory reservation introduces a durability window in which acknowledged decisions are not yet persisted, requiring a recovery story.

The judgement is about how hot, how often, and how correct the answer must be. A mildly hot read key needs a cache. A hot write key on an approximate counter needs buckets. A hot write key on a value that must be exact and must never oversell needs a purpose-built reservation service, and that is a significant piece of engineering that should be justified by an actual measurement rather than by anticipation.

Interview question

"A promotion goes live and one voucher row takes ten thousand writes per second while the rest of the database is bored. Give me four mitigations in increasing order of engineering cost, and tell me which you would ship this week and which you would only build if the promotion becomes a permanent feature."