Self-Hosted AI Now Keeps Its KV Cache on Disk, and vLLM Never Expires It

The usual argument for self-hosted AI is that prompts never leave your perimeter. That claim is easy to check in two places: where the weights run, and where request logs go. On 10 September the vLLM team wrote up a third place. Tiered KV cache offloading moves the key-value cache out of GPU memory into host RAM, and from there onto a filesystem, an S3-compatible bucket, or a peer vLLM instance. The write-up is about throughput. The tier's source code raises a retention question, and has no answer for it.

The pattern: a cache that outlives the request

A performance cache gets a durable tier. Because it was approved as a throughput setting, it inherits nobody's retention policy. Blocks computed from prompts land on persistent storage, while a data map drawn before the tier existed still lists two places prompts live.

Today's instance: vLLM's disk and object tiers

The blog dates tiering to v0.22. The v0.22.0 tag, published 29 May, has the tiering package with a filesystem tier. The object-store and peer-to-peer tiers are in v0.29.0, tagged 9 September. The reason to turn it on: on two H100s serving Qwen3.6-35B-A3B, the blog reports that storage offloading more than doubled throughput beyond 128 concurrent conversations.

What the v0.29.0 code does:

• Naming. file_mapper.py writes each block under root_dir, in a folder named for the model plus a digest of its configuration, and names the file by the block's prefix-cache hash with a .bin suffix.
• Permissions. Both the Python writer in fs/io.py and the compiled one in csrc/fs_io.cpp create block files with mode 0644. The Python writer creates missing directories with os.makedirs at its default mode. I ran that store function under umask 022. It produced a 0644 block inside a 0755 directory. Any account on the host that can reach root_dir can read the blocks.
• Deletion. The tier deletes files in exactly two situations: a temp file after a failed write, and a block that reads back shorter than expected, which is treated as corruption. Its shutdown() stops the thread pools and returns. According to the abstract SecondaryTierManager docstring, evicting blocks when capacity runs out is the implementation's job. The filesystem tier's submit_store only queues writes. The object tier's manager has no delete call at all.
• Documentation. The v0.29.0 usage page does not contain the words encryption, security, privacy, retention, or salt. Its only advice about deleting anything is to remove directories orphaned by a config change, to reclaim disk.

The falsifiable claim is this. In vLLM v0.29.0, a block written to the filesystem or object tier stays until something outside vLLM deletes it, unless vLLM finds it truncated on read. What would show it wrong: a configuration key, or a code path in either tier, that removes blocks by age or by capacity.

The file name gives the prefix away too

According to the tier's own docstring, sharing one root_dir across instances works by default. For cryptographic hash algorithms, the chain seed comes from a fixed default in the source unless PYTHONHASHSEED is set, so identical tokens produce identical file names on every instance. The tier's lookup is a file-existence check. So without a salt, a directory listing tells anyone who can compute the hash whether a given prompt prefix has been processed.

vLLM already has the fix for the in-memory version of this problem. Its prefix-caching design document names a timing side channel and prescribes cache_salt. In kv_cache_utils.py the salt is folded into the first block's hash, and it chains through the rest of the prefix. Offload keys are built from those same hashes, so a salt also changes the file names. The chat and completion endpoints accept cache_salt as a string of 1 to 1,024 characters.

The paper at arXiv 2608.09225 shows why the salt should be set, not merely available. It reports existing timing attacks reaching up to 100% success against unprotected vLLM 0.26.0 and SGLang. It also reports that per-tenant salting kept about 93% of the prefix-cache benefit.

The second instance: llama.cpp slot files

llama.cpp has the same shape in an opt-in form. The server's --slot-save-path is off by default. When it is set, a save action writes a slot's prompt cache to a named file in that directory, and a restore action reads it back. The server source has an erase action, but it clears the slot's tokens in memory. Nothing in tools/server deletes a saved file. It lands in the same place: KV state becomes a file, and how long it lives depends on whoever remembers to clean up.

Capacity pressure is what pushes teams toward these tiers. As our self-hosted LLM capacity post found, a single KV pool is shared by every concurrent session, and spilling it to disk is the obvious next step.

Why self-hosted AI teams keep walking into it

The tier gets reviewed as a capacity setting. The benchmark repository linked from the vLLM post ships an external evictor for its NVMe volume. The evictor runs as root. Its cleanup and target thresholds are percentages, 80 and 55, the metric it exports is disk usage percent, and its one time-based setting is a three-minute access-time threshold. None of that is a retention period. If cleanup starts only when a volume passes a usage percentage, block lifetime is set by traffic, not by policy.

Our PoC to production checklist files caching under cost control. That is the column a tier like this gets approved in, so the data question does not come up. Which storage tiers may hold prompt-derived state belongs in the same document as the model choice for a private LLM deployment.

The cheapest way out

Before you enable a filesystem or object tier:

1. Put root_dir on a local, encrypted volume, owned by the vLLM service account, and run chmod 700 on it.
2. Audit for readable blocks. This should print 0: find /var/lib/vllm-kv -type f -name '*.bin' -perm -004 | wc -l
3. Expire by age, matching your prompt-log retention in minutes. Run this from cron: find /var/lib/vllm-kv -type f -name '*.bin' -mmin +1440 -delete The tier skips writing a block that already exists, so a file's modification time is its first write. A block deleted while the server runs costs a recompute: the tier marks a failed load as a miss.
4. For the object tier, add a bucket lifecycle expiration rule of the same length. vLLM will not delete those objects.
5. Have your gateway inject a per-tenant cache_salt on every request, rather than trusting clients to send one.