Caching Strategies in Practice

Definition

Storing a copy of data closer to its consumer to reduce latency, load or cost — accepting that the copy may be stale.

Why it matters

Caching is the highest-leverage performance technique available, and it introduces the hardest class of bug: two sources of truth that can disagree, where the disagreement is silent and the symptom appears somewhere unrelated.

Implementation patterns

Cache-aside. The application checks the cache, and on a miss reads the source and populates it. Simple, resilient to cache failure, and every miss pays the full latency. The default.

Read-through. The cache itself loads on miss. Cleaner application code, cache becomes a dependency.

Write-through. Writes go to cache and source synchronously. Cache never stale, writes slower.

Write-behind. Writes hit the cache and are flushed asynchronously. Fast writes, and durability risk if the cache fails before flushing.

Refresh-ahead / stale-while-revalidate. Serve the stale value immediately and refresh in the background. Excellent for read-heavy data with tolerable staleness, and it eliminates the miss latency entirely for the user.

Failure scenarios

Cache stampede. A popular entry expires and thousands of concurrent requests all miss and hit the origin simultaneously. The origin saturates, so nothing repopulates the entry, so the storm continues. Remedies: request coalescing so one miss fetches while others wait; probabilistic early expiry; stale-while-revalidate; and TTL jitter so entries populated together do not expire together.

Cold cache after restart. A system that cannot serve its traffic without a warm cache has a hidden dependency, discovered during a cache incident. Warm deliberately before taking traffic.

Invalidation missed on a derived entry. The product page updates and the category listing does not, because the code invalidated the obvious key and not the aggregate ones.

Caching the error. A 500 response cached with a long TTL turns a transient failure into a persistent one.

Unbounded key growth, where a cache key includes something high-cardinality and the hit rate collapses to near zero while memory fills.

Industry example

Map tile serving is the ideal case and shows why: tiles are immutable and deterministically addressed, so they cache perfectly at every layer, and popularity is heavily skewed so a modest cache achieves a very high hit rate. The design lesson is the layering — static geometry cached for long periods, volatile overlays like traffic served separately with short lifetimes and composited by the client. Mixing them would force the whole artefact down to the volatile component's lifetime.

Slack's Flannel is the opposite end: personalised, authenticated, query-shaped data that no CDN can cache, requiring an application-aware edge cache that understands the domain.

Trade-offs

Every cache is a staleness decision. The question is not whether to accept stale data but how stale, for which data, and what happens when the answer is wrong.

Versioned cache keys are frequently better than invalidation: include a version in the key so a change bumps the version, the old entry is never requested again and expires naturally. This avoids distributed purge entirely.

Interview question

A product catalogue is cached with a 10-minute TTL. Merchandisers complain price changes take 10 minutes to appear. What do you propose?

Look for a move from time-based to event-based invalidation with the TTL retained as a backstop — which reduces both staleness and origin load simultaneously. Strong answers identify key tracking as the real work (which aggregate entries contain this product) and offer versioned keys as the simpler alternative to distributed purge.