Semantic Cache
Caching model responses keyed by the meaning of the request rather than by its exact text, so near-duplicate questions are served without a model call.
Exact-match caching is nearly useless for natural language, because "what is your refund policy" and "how do refunds work" are different strings and the same question. A semantic cache embeds the query and returns a cached response when an existing entry is within a similarity threshold.
Where it pays: high-volume assistants with a long tail of repeated questions, where hit rates of 30–60% are achievable and each hit removes both the cost and the latency of a model call.
Two risks to design around. False hits: two questions can be semantically close and require different answers — "can I cancel my order" and "can I cancel my subscription" — so the threshold must be tuned conservatively and measured, not guessed. And personalisation: a cached response that embedded one user's data must never be served to another, so the cache key has to include the permission and personalisation scope.