vLLM vs llama.cpp on a Mac: vllm-metal Wins the First Token, Not the Finish

If you are choosing between vLLM and llama.cpp to serve a local model on a Mac, the question used to be settled by platform: vLLM was a CUDA server. vllm-metal changes that. It is a plugin that runs upstream vLLM's scheduler, paged KV cache and OpenAI-compatible server on Apple Silicon, with MLX doing the execution. The vLLM team announced it on 22 September with benchmarks against llama.cpp and oMLX, and shipped v0.30.0 the next day.

The launch post's headline result is that under concurrent agent traffic, vllm-metal gets the first token out soonest. On averages, that part holds up in the raw data. The broader reading, that it is simply the fastest server for several agents on one Mac, does not.

What shipped

The announcement covers v0.28.0, the first release numbered to match its vLLM core. Since then v0.29.0 landed on 11 September and v0.30.0 on 23 September, both tracking the matching vLLM core. Requirements are unchanged: Apple Silicon, macOS 15 or newer, and a native arm64 Python 3.12. v0.30.0 adds Homebrew as an install route alongside the installer script.

For anyone already running it, v0.30.0 is not a drop-in upgrade:

• VLLM_METAL_MEMORY_FRACTION is gone. The memory budget is set with --gpu-memory-utilization, as in upstream vLLM.
• The legacy non-paged serving path and its VLLM_METAL_USE_PAGED_ATTENTION switch are removed. Serving always uses paged attention.
• Quantized KV cache dtypes such as fp8 and int8_per_token_head are rejected at startup, because Metal stores the paged cache in the model dtype.
• MLX stays pinned to exactly 0.32.1.

The release notes also keep a known-boundaries list. Prefix caching for hybrid models with gated-delta-net layers is still experimental, because cached and uncached runs still diverge on some prompts. GGUF Q4_K and Q5_K blocks can be repacked, but the loader does not route them yet.

The benchmark the post shows

The launch post measured SiliconBench's agent split: 100 multi-turn prompts of roughly 4.6K input tokens each, on an M5 Pro with 64 GB. It says all three model comparisons use 4-bit weights, and that on Qwen3.8-27B vllm-metal had the lowest time to first token and end-to-end latency at two and four concurrent requests.

SiliconBench is public, and the post points to it for the scripts and results. The 4-bit Qwen3.8-27B runs are not there. The repository's 4-bit model profile says those results go to a separate results/Qwen3.8-27B-4bit/ directory, and no such directory exists on any branch. The same profile explains why it was kept apart. The GGUF file llama.cpp loads is about 8.8% heavier than the MLX checkpoint, because K-quants mix precisions. The profile says the 4-bit figure therefore reflects what users deploy, and supports weaker conclusions about the serving layer than the 8-bit run does. The 8-bit profile is matched at 26.6 GiB on both sides.

The figure captions confirm they are different runs. The 27B chart reports llama.cpp completing 99 of 100 requests. The published 8-bit run has all three engines at 100 of 100. For the MoE model, Qwen3.6-35B-A3B, the chart and the repository agree exactly, including mlx_lm finishing only 38, 40 and 38 requests.

What the published run says

The footprint-matched 8-bit Qwen3.8-27B results, recorded on 24 and 25 August:

• First token. vllm-metal leads at two concurrent requests, averaging 11.9 seconds against 13.6 for oMLX and 13.7 for llama.cpp. At four it averages 14.7 seconds against 26.9 for oMLX and 15.6 for llama.cpp, though llama.cpp's median first token, 11.1 seconds, beats vllm-metal's 11.7.
• Finishing the request. oMLX, with its SSD cache, has the lowest average end-to-end latency at two requests (27.1 seconds against 29.6) and at four (47.9 against 52.7).
• Throughput. oMLX has the highest output throughput at every concurrency level. At four requests it produces 5.9 tokens per second to vllm-metal's 5.2.

The Qwen3.6-35B-A3B run shows the same split. vllm-metal's average first token barely moves from one to four requests, 1.6 to 2.0 seconds, while oMLX with a RAM cache climbs from 1.6 to 4.9. But at one request, oMLX's average gap between output tokens is 8.8 milliseconds against vllm-metal's 14.9. vllm-metal admits new work fast. Once a response is streaming, it is not the fastest decoder in this data.

Two boundary conditions in the setup

llama.cpp was capped at four slots. The Gemma 4 E4B chart notes that llama.cpp ran with --parallel 4, and the appendix configures it that way. The llama.cpp server README gives the default for that flag as automatic. At eight and sixteen concurrent requests, llama.cpp's first-token numbers include time spent queued behind four slots, so they say little about the engine.

Empty answers are counted, not explained. In the published Gemma run, vllm-metal returned zero tokens with a success status on 7 of the 100 prompts, and oMLX on 6. llama.cpp completed all 100. Every failure sample recorded for those two engines was a Hermes agent trace. These are not the heavy prompts: the failures ran between about 2,400 and 4,400 input tokens, against a dataset median of about 3,500. Latency averages only cover completed requests. A server that replies instantly with nothing is the silent degradation that our production checklist warns about.

Verdict

Adopt vllm-metal if one Mac serves several agents or users at once and the number that hurts is time to first token. That is the win the published data supports.

Stay on what you have if the Mac serves one person, or if your workload is long generations where time between tokens matters more. The 8-bit data gives that case to oMLX, and the earlier Splash teardown shows engine-specific runtimes are chasing the same machine.

The central claim here: on a footprint-matched model, vllm-metal's lead is time to first token, not throughput or completion time. It would be wrong if a matched 4-bit run on v0.30.0, for example with the UD-Q4_K_S file the profile names as the matched alternative, shows vllm-metal also finishing requests first at two and four concurrent requests.

Before you switch, read the published numbers yourself:

curl -sL https://raw.githubusercontent.com/WindChimeRan/SiliconBench/HEAD/results/Qwen3.8-27B/m5pro/agent/comparison.json | jq -r '.results[] | .framework as $f | .concurrency_results[] | [$f, .concurrency, .successful, (.ttft_avg_ms/1000|floor), (.latency_avg_s|floor), (.itl_avg_ms|floor)] | @tsv'

The columns are engine, concurrency, completed requests, first-token seconds, end-to-end seconds and inter-token milliseconds. Then rerun the post's own vllm bench serve recipe on your prompts at your real concurrency, with three rules. Start llama.cpp with --parallel at least equal to that concurrency. Count any response with zero output tokens as a failure. Keep an engine only if it completes every request and wins on the metric your users actually wait on.