Self-Hosted LLM Serving, Split in Two: vLLM's Multi-Turn Cache Trusts Your Router, and Its Own Router Doesn't Check

A self-hosted LLM on vLLM usually starts as one vllm serve process doing everything, and the guide names what that costs: a long prompt arrives, and every stream already generating on that GPU stutters until it is processed. On 29 September the vLLM project published a long, practical guide to the alternative, built on v0.30.0: run prefill and decode as separate instances, move tokenization and parsing to CPU-only boxes, and pass the KV cache between them.

The guide is candid about when not to do this. Its one correctness warning is where the adoption work sits.

What shipped, and what v0.30.0 breaks

v0.30.0 was released on 22 September. The guide assumes it, and builds on three server shapes: an ordinary vllm serve with a KV connector role, a GPU-free render tier started with vllm launch render, and a token-in, token-out engine started with vllm serve --tokens-only. The render tier turns an OpenAI-style request into token IDs, and a matching derender step turns output IDs back into content, reasoning and tool calls.

The upgrade cost is one breaking change. The release notes put the scale-out endpoints (/render, /derender, /inference/v1/generate) behind a new --enable-scale-out flag on plain vllm serve, and remove the VLLM_ENABLE_SCALE_OUT_ENDPOINTS environment variable that used to switch them on. If you set that variable today, those endpoints disappear on upgrade.

Prefill decode disaggregation, by the guide's own numbers

The headline measurement is honest about its baseline. On one box with two L40S GPUs over PCIe, Qwen2.5-7B-Instruct and roughly 8k-token prompts, the collocated p99 inter-token latency jumped to 169 ms at 0.4 requests per second while the split setup held at 29 ms and never passed 52 ms.

The same box shows the price. The two L40S cards cannot copy peer to peer, and each prompt's roughly 470 MB of KV cache took about 1.3 s to reach decode. At 0.2 requests per second the split setup's median time to first token was 2.2 s against 0.7 s collocated. The guide's own table recommends staying collocated when time to first token is the binding constraint.

The independent figures point the same way. The llm-d project's P/D guide, run on gpt-oss-120b with a 20:1 input-to-output ratio at 45 queries per second, reports mean end-to-end latency down 59 percent and p95 down 67 percent. The same table shows mean time to first token rising from 532 ms to 1157 ms. A split deployment trades first-token latency for a flat tail.

The multi-turn mode, and the check nobody runs

Standard disaggregation moves the cache one way, so on turn two prefill recomputes the answer decode just generated. Setting bidirectional_kv_xfer on both instances lets prefill pull those blocks back from decode instead, and compute only the new tokens.

It works only if the new prompt is an exact continuation of what decode generated. The block alignment in _apply_prefix_caching in v0.30.0 trims by block count from the end of the sequence; that function does not compare token content. If a client removes anything from the middle of the history, prefill loads cache computed for different tokens and the model answers from it. The guide notes that Qwen3's own chat template drops thinking traces between turns, which is enough to trigger it. The result is wrong output, not an error.

The guide points readers to issue #43094 as the tracking issue. That issue was closed by the stale bot on 19 September, as not planned, ten days before the guide was published. In its thread, a contributor added that many coding harnesses rewrite tool calls between turns, so the exposure is wider than reasoning models: any agent framework that compacts or edits history qualifies.

The NixlConnector documentation states that vLLM currently assumes the router detects the mismatch. The router vLLM ships, disagg_proxy_multiturn.py, does not. It caches decode's transfer parameters keyed on a client-supplied conversation_id, and on the next turn with that ID it attaches them without inspecting the messages. Pull request #44838 adds exactly the missing guard: validate the continuation, and fall back to recompute when the history was rewritten or the response held reasoning or tool-call fields the proxy cannot reconstruct. It was opened on 8 June, has had no maintainer review, and was marked stale on 16 September.

The failure policy will not save you either. kv_load_failure_policy defaults to fail and can be set to recompute, but both handle a transfer that failed. A mismatched transfer succeeds.

The verdict

Adopt one-way prefill/decode now if your p99 inter-token latency misses its target under load and your GPUs can talk over NVLink, peer to peer, or RDMA. Skip it if time to first token is what you are measured on.

Do not turn on bidirectional transfer behind the reference proxy for any client that edits history. That is the central claim here, and it is falsifiable: it is wrong if #44838 or an equivalent guard lands in the proxy, or if your router already rejects non-prefix continuations. Either one makes the mode safe to evaluate.

This is the gap our PoC-to-production checklist puts first: evaluation you can defend. A latency benchmark cannot see this bug, because wrong tokens arrive just as fast as right ones. It is also a sibling of the cache-lifetime problem in our look at vLLM's on-disk KV tier: in both, the cache is correct only under an assumption the serving layer does not enforce. Surfacing that kind of assumption is what a private LLM deployment review is for.

Before you split your self-hosted LLM

1. Run nvidia-smi topo -p2p r between the prefill and decode GPUs. Anything other than OK is the setup behind the guide's 1.3 s transfers; benchmark before committing.
2. Send long prompts one at a time and read decode's KV Transfer metrics line. An average transfer time in the hundreds of milliseconds is the first thing to fix.
3. Benchmark against one collocated vllm serve --data-parallel-size 2 on the same GPUs using the guide's vllm bench serve command with --goodput ttft:2000 tpot:30, comparing p99 inter-token latency, time to first token and goodput.
4. Before enabling bidirectional_kv_xfer, make your router store each turn's prompt token IDs plus the IDs decode generated, and attach cached transfer parameters only when the next prompt's IDs start with exactly that sequence. A false miss costs a recompute; a false hit costs a wrong answer.
5. Then replay ten real multi-turn conversations from your agent harness at temperature 0 with the mode on and off. Any difference in output is the mismatch, and the mode stays off.
6. If you set VLLM_ENABLE_SCALE_OUT_ENDPOINTS, replace it with --enable-scale-out before upgrading to v0.30.0.