Access Pattern Skew

Caching pays when access is skewed: if a small fraction of keys accounts for a large fraction of requests, a small cache captures most of the benefit. If access is close to uniform, no cache size helps much, because the working set is the whole dataset.

Skew is therefore the first thing to measure before adding a cache, and it is measured rather than assumed — the distribution varies enormously between workloads and even between surfaces of the same product.

Why it matters beyond caching

Skew governs several apparently unrelated behaviours:

• Cache effectiveness, as above.
• Hot keys and hot partitions, where a well-balanced key space still produces one overloaded node because the imbalance is in the workload, not the mapping.
• Load test realism, where uniform random generation produces excellent cache behaviour and no contention — neither of which the real workload exhibits.
• Tail latency, since cache misses on the long tail are the requests that experience the slow path.

Implementation patterns

• Measure the distribution directly — the share of requests taken by the top 1%, 10% and 50% of keys. This single measurement determines cache sizing, hot-key strategy and load test design.
• Size the cache to the hot set, not to the dataset. Beyond the hot set, additional cache capacity buys very little.
• Selective caching. Caching everything means the long tail evicts the hot set; caching only what is likely to be reused often outperforms caching everything.
• Separate handling above a threshold — replication of hot keys across cache nodes, local in-process caches for the hottest, request coalescing so a hot key's expiry does not stampede.
• Generate load tests from the real distribution, not from uniform random selection.

Industry example

Two cases show opposite ends of the spectrum.

A search platform has a very long tail of unique or near-unique queries, so query-result caching has a bounded hit rate no matter the size — while index segments and document data have excellent reuse. Caching at the wrong level of the stack produces the classic symptom: a cache that works correctly and helps very little.

A commerce platform during a peak event has extreme skew, with a handful of products taking most traffic. Here caching is enormously effective and hot keys are the dominant risk — which is precisely what a load test using uniform random product selection will never reveal. The test distributes load evenly, gets excellent cache behaviour, passes at five times peak, and misses the contention that causes the real failure.

The same property that makes caching work makes hot keys dangerous, and a load test that does not reproduce the distribution reproduces neither.

Failure scenarios

• Adding a cache before measuring skew, and buying staleness, invalidation and a new failure mode for little benefit.
• Uniform load generation, which flatters the system in exactly the dimension that will fail.
• Sizing the cache to the dataset, paying for capacity that changes nothing.
• Assuming yesterday's distribution. Skew shifts with campaigns, seasons and viral events, and the shifts are precisely the risky moments.
• Monitoring aggregate hit rate only, which hides that the hot set is being evicted by the tail.

Trade-offs

Exploiting skew means treating some keys differently — replication, dedicated paths, local caches — which introduces a second code path exercised rarely and therefore prone to rot. Selective caching requires a policy about what to cache, which is a judgement that can be wrong.

The alternative — uniform treatment — is simpler and leaves the benefit unclaimed while leaving the hot-key risk fully present. For workloads with strong skew, which is most consumer-facing ones, uniform treatment is the more expensive choice.

Interview question

"Before adding a cache, what single measurement would you take, and how would the answer change your design? Then tell me how that same measurement changes your load test."