Semantic Caching and Response Reuse

"What is your refund policy?" and "how do refunds work?" want the same answer. An exact-match cache misses both times; a semantic cache embeds the incoming prompt, finds the nearest stored prompt, and if the similarity clears a threshold, returns the stored response without calling the model at all. GPTCache packaged this pattern as reusable infrastructure and made it a common production component (Bang, NLP-OSS at EMNLP 2023).

The saving is total for a hit: no prefill, no decode, no tokens billed, and latency drops from seconds to the cost of a vector search. That is a large enough prize to make the failure mode worth understanding precisely.

Two caches that are routinely confused

Prefix or KV caching reuses the computed key-value tensors for an exact token-prefix match. The system prompt is unchanged, so its KV cache is reused verbatim. This is exact, lossless, and carries no correctness risk; the only question is hit rate.

Semantic caching reuses a response for an approximately matching prompt. It is lossy by construction. The system is asserting that two different inputs deserve the same output, and that assertion can be wrong.

The two live at different layers, compose freely, and have entirely different risk profiles. Conflating them is how semantic caching gets deployed with the risk assessment appropriate to prefix caching.

The threshold is the entire design

A single cosine-similarity threshold governs the tradeoff. Lower it and hit rate rises along with false hits, where a stored answer is returned for a question that needed a different one. Raise it and correctness improves while savings evaporate.

The reason a static threshold cannot resolve this is that similarity distributions for correct and incorrect reuse overlap. Two prompts at cosine 0.93 may be paraphrases in one domain and materially different questions in another; embedding geometry alone does not separate them. vCache attacks exactly this by learning a per-cached-prompt threshold online, giving user-specified error-rate guarantees rather than a global knob, and reports up to 12.5x higher cache hit rate and 26x lower error rate than static-threshold and fine-tuned-embedding baselines (Schroeder et al., ICLR 2026, arXiv:2502.03771).

The framing that paper introduced is the useful one: a semantic cache should be specified by the error rate you are willing to accept, and the threshold derived from it, rather than the reverse.

When it breaks

Negation and small perturbations sit above threshold. "Is this covered under warranty?" and "is this not covered under warranty?" embed very close together. So do two prompts differing only in an account number, a date, or a units qualifier. These are precisely the differences that change the correct answer, and precisely the ones sentence embeddings are trained to discount.

Personalised and stateful responses must never be cached. If the answer depends on who is asking, on conversation history, or on retrieved documents that vary per user, a cross-user hit is a data leak rather than a cost saving. The cache key has to include the tenant and any authorisation-relevant context, which sharply reduces the hit rate that made it attractive.

Staleness has no natural expiry. A cached answer about pricing, availability, or policy stays confidently wrong until someone invalidates it. Unlike a KV cache, whose contents are derived and can be recomputed at will, a semantic cache holds business claims and needs an invalidation strategy tied to the underlying source of truth.

Vendor hit rates and production hit rates differ. Marketing figures often reflect benchmark workloads with high query repetition. A real hit rate depends entirely on how concentrated the traffic is, and a long-tail workload can see almost none. Measure before budgeting the saving.

The cache becomes an attack surface. Because entry is by similarity rather than equality, an adversary who can insert prompts may be able to plant an entry that later serves a crafted response to another user's benign query. Any cache shared across trust boundaries needs the same scrutiny as any other shared mutable store.