The cache now has a filename

In the shutdown path of vLLM's new disk backend there is a comment that does more data governance work than most data maps: "Slot contents can encode user prompts, so drop the name now rather than leaving them readable until the next run overwrites the file." The author was solving a small problem — what to do with a scratch file when the server stops — and to solve it properly they had to say plainly what is in the file. It is the most precise sentence about the location of user data that I read this fortnight, and it is a code comment.

The change it belongs to shipped on 26 August in vLLM v0.28.0: disk offloading for the CPU offload connector. Mechanically it is unglamorous and rather good. When GPU memory fills, key-value blocks are copied by DMA into a small pinned staging buffer and written into a preallocated file with pwritev, O_DIRECT by default so the page cache does not consume the host memory the feature exists to conserve. There is one file per rank, so tensor-parallel workers cannot corrupt each other. The file is opened O_EXCL and O_NOFOLLOW, mode 0600, unlinked first in case an earlier one survived, and unlinked again on shutdown. The default size, if you turn it on and do not think about it, is a hundred gibibytes per rank.

That default is the whole story. A hundred gigabytes of somebody's prompt state, per rank, on a node's local NVMe, arrives when an operator adds a key to kv_connector_extra_config in the pursuit of throughput. Nothing else in the organisation is notified. The cache has become a storage tier, and it is being introduced by performance settings rather than by storage design — which means that every question we ask of a store, from where it lives to who can read it to how long it keeps things, is now live for a component still filed under optimisation.

Start with what is actually in the blocks. vLLM's prefix cache is content-addressed by the user's own words: a block's hash is built from the tokens in that block, the hash of its parent block, and extras such as LoRA identifiers and multimodal input hashes. This is why the reuse works at all. It also means the index into the cache is a function of the prompt, and the value it points to is the model's internal state derived from that prompt. These are not the prompt, in the way a database row is a value and not the form someone typed it into. They are also not unrelated to it, which is exactly what the shutdown comment concedes.

Then consider that the cache is shared on purpose. Reuse across requests is the entire point of prefix caching; a private per-request cache would save nothing. vLLM offers an opt-out — a per-request cache_salt mixed into the first block's hash, which the design docs describe as preventing "timing-based attacks where an adversary could infer cached content by observing latency differences" — and the docs are candid that with it, "cache sharing is limited to users or requests that explicitly agree on a common salt." The mechanism is well made. It is also opt-in, which means the default posture of a serving instance is one shared namespace across everything it is asked to answer, and the platform team is usually not the party who knows whether that is acceptable. The same documentation is similarly frank about the hashing choices underneath: the default is SHA-256, but the faster non-cryptographic option carries a written warning that collisions "can cause undefined behavior or even leak private information in multi-tenant environments," to be weighed against the performance benefit. None of this is buried. It is all in the design docs, one flag away from a throughput win, and it is addressed to a reader who is tuning a server rather than one who is answering for a dataset.

Now the part that has no equivalent word in the serving vocabulary. The disk file is a fixed number of slots, and a block stays in it until another block needs the slot. Ask an inference team how long derived user content persists in that file and the honest answer is: until cache pressure removes it. That is an eviction policy. It is a perfectly good one. But a retention schedule cannot accept "it depends on how busy we were" as a duration, and the two concepts are being served by the same mechanism under only one of the two names. Nobody is hiding anything. The vocabulary simply has not caught up with the fact that a cache with a filename is also a copy with a lifetime.

The safeguards, read carefully, are all scoped to the process. Unlink on shutdown is a graceful-shutdown path; a SIGKILL, an out-of-memory kill or a node that loses power does not run it. The next start unlinks before opening, which genuinely closes the window in the ordinary case — but the length of that window is set by when the process is restarted, not by anything the operator has chosen, and a node image, a volume snapshot or a debugging capture taken in between sits entirely outside it. The engineering here is better than average precisely because it was written by people who understood the risk. The risk is simply not shaped like a process.

It is also not staying on one machine. The same release notes list a companion change allowing an out-of-tree Python module to register itself as a secondary cache tier through configuration alone, no fork required. Point that at LMCache, whose README describes "persistent, tiered KV cache offloading and reuse" across "CPU memory, local storage, and remote backends", enabling reuse "across requests, sessions, and engine instances", with backends including Redis and S3-compatible object storage, and the picture changes shape. At that point the KV cache is a distributed, durable, cross-instance store of user-derived state with a network boundary and a credential — and it is still configured in the same block of serving flags, by the same team, on the same change ticket. All of the above is taken from the two projects' own source and documentation rather than from anyone's account of them, which is the right evidence for a claim of this kind: what a cache does when its buffer fills is not a matter of interpretation.

The strongest objection is that none of this is new and the alarm is misplaced. Operating systems have paged process memory, prompts included, to swap for decades. Databases write user data to disk on every commit. CDNs cache personalised responses at the edge. No one demands a privacy assessment for the page cache, and the reason is sound: a cache derived from data you are already permitted to process, on infrastructure you already control, inherits the controls of the thing it is derived from. On that view vLLM's disk backend is a well-built temporary file, it is off by default, it requires a deliberate configuration to exist at all, and the people who wrote it thought harder about its file mode than most application teams think about their log retention. All of that is true, and it is why this is not a scandal.

But two things did change, and they are the reasons the analogy does not fully hold. The first is that the sentence organisations have been making about this data is an application-tier sentence. "We do not retain your prompts" is a statement about a request handler and a log pipeline; the KV cache sits below both, and until recently it did not need to be in scope because it evaporated with the process. It no longer does. The second is that the control surface has moved. Creating durable state used to require a schema, a migration and a review; it now requires a key in a values file owned by the group whose objective function is tokens per second per dollar. There is no governance process in any organisation I know of that triggers on a Helm chart edit, and there is no reason one should — which is precisely why the tier will keep being created without anyone noticing that it was.

This is the general form of the thing, and it will recur well beyond inference. Performance work moves data. Every tier added for latency is a copy of something, and copies acquire obligations that the original already discharged somewhere else. The discipline that architecture can actually offer here is not caution about the optimisation — the optimisation is good, the economics of serving models are brutal, and the alternative to caching is buying more accelerators. It is the much less glamorous work of noticing when a config key has quietly created a store, and then making that store answerable to the same questions as every other store the organisation owns: what is in it, who can read it, how long it lives, whether it appears on the map. That work starts with a question an architect can ask this week and most cannot currently answer: in each environment, what is the offload backend set to, what path does it write to, is the volume behind that path encrypted at rest, and does anything downstream of the request handler know that the path exists.

The comment in that shutdown handler is, in miniature, the state of the practice. The person closest to the bytes wrote down what the bytes are, and protected them as far as their scope allowed — one file, one process, one machine. Everyone downstream of that scope inherited a store and a word that says it is not one. The next time a serving stack gets faster, the useful question is not whether the change is safe. It is which copy of your users just came into existence, and who was told.