Prompt Caching Infrastructure

A typical RAG chat request sends the same 4,000-token system prompt and the same 8,000 tokens of retrieved context to the model on every turn. The model recomputes the KV cache for all of it, every time. Prompt caching makes that absurdity go away: compute the prefix once, persist its KV cache, and on subsequent requests with the same prefix, the model resumes from where the cache left off. The cost reduction is 70-90% on the cached portion, and the latency win is the same shape (you skip the prefill for cached tokens).

What "the cache key" actually is

The shared design across providers: the cache key is the exact byte prefix of the prompt up to the caching boundary. One byte different, one extra space, one reordered system message - cache miss. The cache is content-addressed by hashing the token sequence, sometimes block by block, sometimes as the full prefix.

This has two practical consequences:

1. Order your prompt from most-stable to least-stable. System prompt -> tool definitions -> RAG context -> conversation history -> current user turn. Anything that changes invalidates everything after it.
2. Do not interpolate dynamic values into the cacheable prefix. A timestamp in the system prompt is the most common foot-gun. Once a day at midnight, your hit rate goes to zero and your bill triples.

The three providers

Anthropic prompt caching

Anthropic exposes caching via cache_control markers on content blocks. You explicitly mark up to four breakpoints in the prompt; everything before each marker is cached as a separate entry.

• Default TTL: 5 minutes, optional 1-hour TTL with cache_control: {type: "ephemeral", ttl: "1h"}.
• Cache writes cost 25% more than base input tokens (5-min TTL) or 2x (1-hour TTL). Cache reads cost 10% of base input tokens. So a cache hit is a 10x cost reduction on the cached portion.
• Minimum cacheable length: 1,024 tokens for Sonnet/Opus, 2,048 for Haiku.
• Workspace-level isolation so one tenant's cache cannot be read by another.
• Pre-warming is supported via max_tokens: 0 requests.

The mental model: pay 1.25x once to write a 5-min cache; pay 0.1x to read from it. Break-even after ~3 reads. If your prefix gets reused more than 3 times in 5 minutes, caching pays. In practice, multi-turn chat hits this trivially.

OpenAI prompt caching

OpenAI's model is automatic and opaque: any prompt over 1,024 tokens is eligible, the system hashes the prefix in chunks, and you get a cached_tokens field in the usage response telling you how many were hits. No explicit markup. Cached tokens are billed at 50% of standard input price (some models 25%). TTL is unspecified but typically 5-10 minutes of inactivity.

The trade-off vs Anthropic: less control, less savings (2x vs 10x reduction), but zero integration work. If you do nothing differently, you start getting cache hits.

vLLM prefix caching

For self-hosted serving, vLLM ships automatic prefix caching. Enable with enable_prefix_caching=True on engine init. The mechanism: the KV cache is split into fixed-size blocks (typically 16 tokens), each block is hashed by its content plus the hash of the prior block, and incoming requests reuse any matching prefix already resident in HBM.

• The hit-or-miss is per block, so partial-prefix hits work.
• Eviction is LRU at block granularity, with in-flight sequences pinned.
• The win is in prefill latency (you skip computing KV for cached tokens). Decode is unaffected.
• Reported hit rates of 50-90% on production chat workloads.

SGLang's RadixAttention is the same idea organised as a radix tree over prefixes, which makes shared-prefix branching (one system prompt, many user turns) more memory-efficient still.

What to put in the cached vs uncached portion

[CACHED PREFIX - changes rarely]
  System prompt (instructions, persona, guardrails)
  Tool / function definitions
  Few-shot examples
  Static RAG context (e.g. product catalogue extract)

[CACHE BOUNDARY - cache_control marker on the last block above]

[UNCACHED SUFFIX - changes per request]
  Per-query retrieved chunks (different each query)
  Conversation history (grows each turn, can use a second cache breakpoint)
  Current user message

A second cache breakpoint after the conversation history is the standard multi-turn pattern: the system + tools prefix is stable for the lifetime of the conversation, the conversation history is stable across the next turn, and only the new user message is uncached.

Cache-warming patterns

For predictable, high-value prefixes (your standard system prompt, your top 100 RAG documents), explicitly warm the cache on a schedule:

# Pre-warm Anthropic cache for a known prefix - one cheap request
client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1,        # cheap; we are not generating, only writing the cache
    system=[
        {"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}
    ],
    messages=[{"role": "user", "content": "ping"}],
)

For vLLM, the equivalent is a single prefill request through the engine. Running this every 4 minutes (just under the 5-minute TTL) keeps a hot cache permanently warm at trivial cost.

KPIs to track

A common failure mode: cache hit rate is 80% but cost only dropped 20%. Diagnosis: your cached portion is small relative to the per-request retrieved context. Move more into the cached prefix (e.g. cache top-K retrieved chunks if K is stable per session).

When it falls down

• Dynamic content in the prefix. Date, user_id, request_id, A/B test variant - any of these in the system prompt destroys the cache. Move them to the uncached suffix.
• Prompt-template "improvements" deployed at noon. A single character change in the system prompt invalidates every cached prefix globally. Coordinate prompt changes with cache TTLs or accept the cliff.
• High concurrency, single prefix. All providers handle this, but self-hosted vLLM with insufficient HBM will thrash the block cache. Monitor block eviction rate.
• Streaming with very short responses. TTFT win dominates the latency benefit. If your response is one sentence, the cache mostly saves cost, not latency.
• Multi-region deployments. OpenAI and Anthropic do not (publicly) share cache across regions. Pin your traffic to one region to keep hit rate high.

Further reading

• Anthropic prompt caching documentation - cache_control markup, TTL options, pricing, and minimum lengths.
• vLLM automatic prefix caching - the user-facing flag, plus the design doc behind it.
• Efficient Memory Management for Large Language Model Serving with PagedAttention - the vLLM paper that explains why block-hashed KV cache reuse is possible at all.