The CPU Serving Speedup You Won't Measure

On 29 July the Arm team published a performance write-up on the vLLM blog covering Arm CPU inference. Running Llama 3.1 8B on Neoverse V2 cores, they report up to 2.7x throughput at BF16, up to 4.8x throughput and 5.7x better time per output token at INT8 W8A8, and up to 6.2x throughput with 7.8x TPOT and a 2.6x gain on time to first token at W4A8. The engineering behind those figures is specific and unglamorous: switching the memory allocator to mimalloc, using LSE atomic instructions in libgomp, prepacking dense-layer weights through oneDNN and Compute Library, rebuilding paged attention around BFMMLA instructions, and routing INT8 through KleidiAI micro-kernels. Each carries its own number — the allocator change alone is credited with 2.3x offline throughput, which the post says it leaves out of the plots because those gains would dominate the scale.

Then read the baseline. The post names it plainly: October 2025, when the team first benchmarked vLLM on Arm and found performance well below what they expected. That date matters more than any of the multiples, because the first prebuilt Arm wheel is v0.11.2, published on 20 November 2025 according to the GitHub releases API. The baseline is a state of the tree from weeks before there was a released Arm wheel to install.

The assumption this invalidates

The assumption worth retiring is not that CPU inference is a demo. It is that a published multiple describes the improvement available to you.

If your team is running Arm CPU inference at all, you installed a wheel. The earliest one you could have installed already contained an unknown share of these gains, and vLLM has put out a release roughly every two weeks across the spring and summer — v0.26.0 landed on 27 July, two days before the post. Whatever you measure when you upgrade is the delta from your current wheel to that one, not from a pre-wheel tree you would have had to build from source to run at all. The 6.2x is real as reported and irrelevant as a planning input.

The second thing to retire is the idea that the fastest configuration is a flag. It is not. The vLLM repository's own Arm installation document states that the platform offers basic model inferencing and serving with NEON support and data types FP32, FP16 and BF16. INT8 does not appear in that line. The W4A8 documentation in the same repository does say that Arm CPUs accelerate the scheme through KleidiAI — so the capability is there, but the two pages describe different support surfaces, and the installation page you work from to stand the thing up describes the slower half.

Getting to W4A8 means running a separate offline step: install llm-compressor in a virtualenv the docs explicitly say to keep apart from vLLM's, run GPTQ calibration over 512 samples at a sequence length of 2048, and choose between a groupwise recipe the docs recommend for accuracy and a channelwise one they recommend for inference speed. That is a pipeline with an owner and a rerun schedule, not a configuration change. It is exactly the line item that the gap between a proof of concept and production is made of — the demo shows the throughput chart, and the budget discovers the calibration job.

What to do this quarter

The decision in front of most teams is not Arm versus GPU in the abstract. It is whether a specific low-concurrency workload — internal classification, summarization over a queue, batch enrichment — can move off accelerator capacity you are paying for by the hour. Four steps:

1. Measure absolute TPOT, not the multiple. Install the arm64 wheel or pull vllm/vllm-openai-cpu:latest-arm64, and set VLLM_CPU_KVCACHE_SPACE and VLLM_CPU_OMP_THREADS_BIND before you benchmark anything. The installation doc also warns that TCMalloc must be present and on LD_PRELOAD before running a CPU wheel; skip that and your first number is wrong.
2. Skip the calibration job on the first pass. The announcement names RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 as one of the checkpoints that performs well, and its model card reports an OpenLLM average of 62.71 against 63.03 for the unquantized model — 99.5% recovery, built with llm-compressor and GPTQ using a 1% damping factor over 256 sequences. Download it. Only build a W4A8 checkpoint yourself once W8A8 has proven the workload fits.
3. Size the box on memory, not vCPU count. The Arm doc says compatibility testing was done on AWS Graviton3, and that BF16 needs ARMv8.6-A or later, listing Graviton3, Graviton4 and AmpereOne as processors that provide it. AWS's C7g page puts Graviton3 in the C7g family, which runs from c7g.medium at 1 vCPU and 2 GiB up to c7g.16xlarge at 64 vCPU and 128 GiB. Weights and whatever you hand to VLLM_CPU_KVCACHE_SPACE both come out of that same instance memory, so pick the size from measured resident memory.
4. Judge it against a threshold, not a multiple. Fix the p95 TPOT your product actually tolerates before the first run, and treat it as pass/fail. If CPU serving misses it at your real concurrency, the throughput multiple is not an argument.

Do this and the answer feeds straight back into the buy, build, or fine-tune decision: a smaller model that clears your latency bar on commodity CPU moves the cost side of that comparison off per-token pricing and onto instance hours you already know how to forecast.

What would show this wrong

The central claim here is that the reported multiples will not reproduce for a team upgrading between released versions, because the baseline predates the first prebuilt Arm wheel. Two things would falsify it. A release-to-release comparison — v0.11.2 against v0.26.0 on the same Graviton3 instance, same model, same concurrency — landing anywhere near 6.2x would show the pre-wheel baseline is not the explanation. So would the Arm installation page adding INT8 to its supported data types alongside a documented path that does not require an offline calibration step, which would make the fastest tier a flag after all.

Neither has happened yet. Until one does, the honest planning number is the one you measure on your own instance, at your own concurrency, against the wheel you are already running.