Uber's Docstore (a sharded MySQL-backed document store) serves reads dominated by recently active trips. CacheFront put Redis in front of it. Why build the cache into the storage layer rather than leaving it to each caller, and how must invalidation, read-your-writes and post-failover warming behave?

Why the cache belongs in the storage layer

When each calling team builds its own cache, the organisation acquires N independent implementations of the hardest problem in the stack, each with its own invalidation bug, its own TTL guess, and its own behaviour during a Redis outage. The failures are then diagnosed N times.

Putting the cache behind the storage client's interface means:

• One invalidation implementation, driven by the store's own change stream rather than by each caller remembering to invalidate.
• Consistency semantics that are stated once and can be reasoned about, rather than differing per team.
• Callers get the benefit without the complexity — they issue reads, and the tier decides.
• Cache behaviour during failure is centrally controlled, which matters enormously (below).

The workload justifies it: reads dominated by a small set of recently-active entities is the ideal cache shape — high hit rate, small working set, and the hot set naturally ages out.

Invalidation

Do not invalidate from the application. Drive it from the database's replication stream: a change to a document produces a change event, and the cache tier deletes or updates that key. This makes invalidation a consequence of the write actually being committed rather than an action someone might forget.

Delete rather than update unless the new value is cheap and complete — writing a partial value into cache creates a corruption that survives until TTL.

Keep a TTL anyway, as a bound on how long any missed invalidation can persist. The TTL is the blast-radius limit on invalidation bugs, not the primary mechanism.

Read-your-writes

The change stream is asynchronous, so a read immediately after a write can hit a stale entry. Two mechanisms:

• Write-through for the writer's own path: the write updates the cache synchronously in addition to the stream doing so eventually.
• A short per-key "recently written" marker that forces reads of that key to bypass cache for a window longer than the replication lag.

Choose per use case. A trip's status needs read-your-writes; an aggregate count does not. Applying the strict version everywhere destroys the hit rate, which is the point of the cache.

The failover case, which is where naive designs break

After a shard failover, the cache holds entries from the old primary. Two dangers:

• Stale entries served as current. The cache must be invalidated for the affected key range on failover, which requires the cache tier to know about shard topology changes — an argument for integration.
• A cold cache producing a thundering herd. Every miss goes to a database that has just failed over and is at its most fragile. Without protection the recovery load exceeds normal load and the new primary falls over too.

The controls: request coalescing, so simultaneous misses on the same key produce one origin read; admission throttling on cache fills; and deliberate warming by replaying recent access patterns before restoring full traffic.

The property that must be designed explicitly

What happens when the cache tier itself is unavailable. If every read falls through to the database, the database receives its full uncached load — which it has not been provisioned for, because the cache has been absorbing most of it for years. A cache that a system cannot survive losing is not a cache; it is an undeclared primary datastore. Either the origin is provisioned for the full load, or the tier must shed load when the cache is down.