A Self-Hosted LLM Has One Context Pool and Four Users By Default

A self-hosted LLM is chosen on one number and served on another. The number in the decision memo is the model's context window. The number that governs what a user experiences at 10am on a Tuesday is how much of the KV pool one session gets once the pool is divided among everyone else talking at the same time. The two are joined by configuration flags nobody set, because they have defaults, and the defaults do not error.

llama.cpp tagged v0.4.0 on 4 September. The release lists initial Qwen3.8-Flash-Next and Nemotron-3-Puzzle support, on-demand tensor reading, video input options, a ggml update to 0.23.0 carrying sparse flash attention and RDMA work — and, in one clause, per-slot server context limits. That clause is the one worth reading the code for.

What the flag does, and what the defaults were doing already

The server README documents the new option as --kv-unified-per-slot N: a context limit per parallel slot, unset by default, with behavior unchanged when unset. When it is set without -c/--ctx-size, the shared KV pool is sized to n_parallel times N.

The surrounding defaults are what make this interesting. -c/--ctx-size defaults to 0, meaning the value is loaded from the model. -np/--parallel defaults to -1, meaning auto. And -kvu/--kv-unified — a single unified KV buffer shared across all sequences — is documented as enabled whenever the slot count is auto.

Auto is not left abstract in the code. In tools/server/server.cpp, when the parallel count is negative the server resolves it to 4 slots and turns the unified buffer on. So the default posture of a llama.cpp server is four sequences sharing one buffer sized from the model, with no documented per-slot floor. Nothing reserves capacity for the fourth conversation.

The cap is a minimum, and it can fail to bind without failing

Reading tools/server/server-context.cpp answers the question the release note leaves open: what a slot actually gets. The helper n_ctx_slot() returns the smallest of three numbers — the per-slot context the pool can supply, the value of --kv-unified-per-slot when it is set, and the context the model was trained with. It is a ceiling, not a reservation, and it only ever lowers.

That matters when you pin the pool yourself. If you pass -c and then set a per-slot cap larger than what the pool can give each slot, the cap can never bind. The server does not refuse to start. It logs a warning saying the cap has no effect, reports the value slots are actually limited to, and tells you to either raise the pool with -c or drop -c so the pool is sized from n_parallel times the cap. If nobody reads startup logs, that is a flag set, a ticket closed, and no change in behavior.

There is a sharper trap underneath. The auto-sizing in server.cpp only fires when the cap is set, the context size is still 0, and an internal minimum-context field has not been raised to its maximum. In common/arg.cpp, passing -c 0 explicitly is what raises that field; the comment beside it describes the case as the user asking for the full context size, which disables context reduction. common/common.h shows the field defaults to 4096, so it is only at its maximum if you asked for it. The consequence: omitting -c and writing -c 0 are described identically by the help text, and once --kv-unified-per-slot is in play they produce different pool sizes. The flag also reads from LLAMA_ARG_KV_UNIFIED_PER_SLOT and is registered for the server example only, so it can arrive from a container environment file rather than from the command line anyone is reading.

vLLM has the same shape with a different mechanism

This is not a llama.cpp defect, which is why it is worth naming as a pattern. vLLM's tuning documentation exposes the same underlying fact through max_model_len, max_num_seqs and max_num_batched_tokens. Its documented default when the KV cache cannot hold the working set is to preempt requests and recompute them, and the listed mitigations are to raise GPU memory utilization or to lower the sequence and batched-token limits. It also warns that pinning --kv-cache-memory to skip start-up profiling caps concurrency if you guess low and fails at allocation if you guess high.

Two runtimes, two mechanisms — cap the slot, or preempt and recompute — and one shared premise: the thing you bought is concurrency, and the context window is what is left after dividing.

Why teams keep walking into it

Because the failure is quiet and shaped like a product complaint. Nobody gets a 500. Somebody's long document review starts forgetting the top of the document, in the afternoon, when four other people are using the box. By the time it reaches an engineer it has been reworded as a model quality problem, and those get answered by trying a bigger model rather than by reading a start-up log. That substitution is what turns a working prototype into a stalled rollout, as in why enterprise AI doesn't ship and in POC to production AI: the operational variable is invisible, so the model gets blamed. Deciding the per-session ceiling before the hardware order is part of what a private LLM deployment has to specify, alongside the model choice.

The cheapest way out

Pick the ceiling you will honour per session, in tokens, and the number of concurrent sessions you will serve. Then let the server do the arithmetic rather than doing it yourself:

llama-server -m model.gguf -np 4 --kv-unified-per-slot 8192

With no -c at all, that sizes the pool to 4 times 8192, or 32768 tokens, and prints the multiplication in the start-up log. Three checks before you call it configured:

1. Grep the start-up output for the sizing line. If it is absent while the cap is set, you passed -c somewhere — including -c 0 — and the pool was not auto-sized.
2. Grep for the warning that the cap has no effect. If it is there, your cap is above per-slot pool capacity and is doing nothing.
3. Confirm your ceiling is at or below the model's trained context, because n_ctx_slot() takes the minimum regardless.

The falsifiable claim: on a llama.cpp server at defaults, no session holds a reserved share of the KV pool, and a per-slot cap set without checking pool capacity can be inoperative while appearing configured. What would show it wrong is a start-up log reporting a per-slot reservation with no cap set, or n_ctx_slot() returning anything other than the minimum of pool capacity, cap and trained context.