Caching Layers for LLM Applications
The four distinct caches in a mature LLM system, what each requires to be correct, and why semantic caching is the one that is dangerous.
Caching is the highest-return optimisation in most LLM applications, and "the cache" is usually four different mechanisms with different keys, different invalidation rules and very different risk profiles.
The four caches
Prompt prefix cache. The provider or serving stack retains the computed KV cache for a shared prompt prefix, so a repeated system prompt is not recomputed. It is exact, requires no correctness reasoning, and needs only that the prefix be byte-identical and appear first. The engineering is entirely in prompt construction: stable content first, variable content last, and no timestamp or per-user field ahead of the shared part.
Exact response cache. Key on a hash of the full request, including model, parameters and the complete prompt. Returns the identical response. Correct by construction, and its hit rate depends on how often identical requests recur, which is high for shared content and near zero for personalised interaction.
Embedding cache. Text-to-vector mappings are deterministic for a given model version, so they cache indefinitely, keyed by text hash plus model version. This is often the cheapest large saving in a retrieval system, since re-embedding unchanged documents is pure waste.
Semantic cache. Return a stored response for a request judged similar enough to a previous one. This is the one that is not correct by construction and is where the risk lives.
Why semantic caching is dangerous
Similarity is not equivalence. Two questions with high embedding similarity can have different correct answers, and the classic failures are negation, "is X safe" and "is X not safe" embed closely, quantities, "5 units" and "50 units", and entity substitution where the embedding is dominated by the shared structure rather than by the differing name.
A semantic cache hit returns a confident wrong answer with no indication that a cache was involved, and the error rate is a function of a threshold that has to be tuned against measured harm rather than against hit rate.
Where it is safe: high-volume, low-stakes, genuinely repetitive queries with a conservative threshold and an exclusion list for anything personalised, time-sensitive or safety-relevant. Where it is not: anything where a subtly wrong answer matters, which is most of what people want to cache.
When it breaks
Personalised content in a shared cache is a data leak. A response conditioned on one user's data, cached and returned to another, is a serious incident. Cache keys must include everything the response depended on, and user-conditioned responses generally should not be shared-cached at all.
Staleness is invisible. A cached answer about a changing fact is wrong without any error. Time-to-live has to reflect the volatility of the underlying content rather than a uniform default, and content-based invalidation is better where the source is observable.
Cache hit rate is not the metric. A semantic cache tuned for hit rate is tuned toward returning wrong answers. The metric is cost saved per unit of quality lost, which requires measuring the quality lost, which almost nobody does.
Prefix caching is quietly defeated. Adding a timestamp, reordering tool definitions, or inserting a per-user field before the shared prompt breaks byte-identity and the cache silently stops hitting. Monitoring the cached-token ratio is what surfaces it, and without that instrumentation the regression looks like a cost increase with no cause.
12 flashcards for this concept
Click a card to reveal the answer.