A search platform adds a cache and sees only a small latency improvement. What are the likely reasons, and what should be measured?

What to measure first

Hit rate, and latency separated by hit versus miss. These two numbers explain most cache disappointments immediately:

• Low hit rate → the cache is not being used. Investigate why.
• High hit rate but small improvement → the cache was not on the expensive path, or the miss path dominates the average.

The second case is common and counter-intuitive: with a 90% hit rate, if a hit takes 1 ms and a miss takes 500 ms, the average is 51 ms — dominated entirely by the 10% of misses. Improving hit rate from 90% to 95% halves the average, whereas making hits faster achieves almost nothing.

Likely reasons for a low hit rate

1. The working set exceeds the cache. Search queries have a very long tail; a substantial fraction are unique or near-unique. No cache size makes those hit, and the achievable hit rate is bounded by the query distribution rather than by the cache.

2. Cache key too specific. Including a session id, a timestamp, or a parameter that varies unnecessarily fragments the cache so that logically identical requests miss. This is the most common implementation defect and it is invisible without inspecting keys.

3. TTL too short, so entries expire before being reused. If the mean time between requests for a key exceeds the TTL, the hit rate approaches zero regardless of size.

4. Eviction pressure from low-value entries. Caching everything means the hot set is evicted by the long tail. Selectively caching only what is likely to be reused often outperforms caching everything.

5. Cache placed at the wrong layer — caching the cheap step while the expensive one is uncached.

The structural questions

Is the workload cacheable at all? For a search platform, query results have poor reuse and index segments and document data have excellent reuse. Caching at the wrong level of the stack produces exactly this symptom — a cache that works correctly and helps little.

Is there skew to exploit? Caching pays when access is skewed. Measure the distribution: if the top 1% of keys account for a large share of requests, a small cache captures most of the benefit. If access is close to uniform, caching gives little at any size.

What is the cost of a miss? A cache in front of a cheap operation cannot help much. Put caches in front of the expensive path.

The trap to avoid

Adding a cache before understanding the access distribution. A cache introduces staleness, an invalidation problem, an additional failure mode, and a component that becomes load-bearing. If the measured benefit is small, all of that has been bought for nothing — and the cache will still be there, still needing to be operated, long after the disappointment is forgotten.