To Run an LLM Locally in Transformers, the Metal Kernel Has to Be There

If you run an LLM locally today, you probably run two stacks. One is llama.cpp, or something built on it, to serve the quantized weights. The other is transformers, for the work that wants a Python object rather than a server: a tokenizer experiment, a fine-tune, a logits probe. Hugging Face published a path on 22 September that collapses those into one. It works. It works on Apple silicon, for one model family, and only while a particular kernel is reachable.

What shipped

Marc Sun, Arthur Zucker and Lysandre published the announcement on 22 September. The change lets AutoModelForCausalLM.from_pretrained take a gguf_file argument and keep the weights packed, running matmuls directly on the packed blocks rather than unpacking them into a dense tensor at load. The documentation on the main branch is explicit about the condition: weights stay packed when the Hub kernel ggml-org/ggml-quantization is available, and the loader defaults to MPS when it is present.

It is not in a release. The announcement says to install transformers from main until the next one, and the newest tagged release on GitHub is v5.17.0, published on 9 September — thirteen days before the post describing the feature.

The benchmark was a MacBook Pro M2 Max with 32 GB of unified memory on macOS 26.6, across three checkpoints, and the post reports throughput close to llama.cpp. It also says, in the same breath, that the transformers figures include prefill while the llama.cpp figures are decode only. Those are not the same measurement, and the post says so rather than leaving a reader to find it.

Running an LLM locally without that kernel costs you the quantization

This is the part worth internalising before anyone on your team runs the install line. The documentation describes the fallback plainly: if the kernel is not available, the model is dequantized at load, which unpacks every weight and hands back a plain dense model.

The size of that fallback is readable off the model repository. For unsloth/Qwen3.5-4B-GGUF, the Q4_K_M file is 2.74 GB and the BF16 file is 8.42 GB. Those are the two ends of the trade. You asked for the small file and the loader gave you the big model, with no error, because the result is a working model either way. The only symptom is memory. That is the failure shape we argued is the default one for AI systems in PoC to production: not a stack trace, a number nobody is watching.

So the check is numeric rather than behavioural. Load a 4B Q4_K_M and watch resident memory. A figure in the region of the 2.74 GB file means the packed path engaged. A figure in the region of the 8.42 GB BF16 file means it did not, and you are paying dense weights for a quantized download.

The kernel is metal, and that is not a caveat about this week

The announcement says the packed inference path is MPS-only for now. The kernel repository says something more precise. Its Hub metadata lists supportedDriverFamilies as metal, and nothing else. The attention kernel, ggml-org/ggml-attn, reports the same single family. Both were last modified on 2 September, and the Hub reports 27 downloads for the quantization kernel and 21 for the attention one. Both are owned by ggml-org rather than by Hugging Face, and neither is gated.

One published driver family means there is nothing for a CUDA box to load. On a Linux GPU machine there is no packed path to take, so the weights are dequantized at load and you are back to a dense model. If your private deployment is a rack rather than a laptop, this release changes nothing about what you run — which is the usual shape of the gap between a local demo and a production deployment.

What it breaks

Four behaviour changes are worth reading before an upgrade. Three of them are on the documentation page rather than in the announcement.

Packed loads force float32. The documentation says the loader picks it because it is faster on MPS, and returns a warning if you set another dtype. A pipeline that pins bfloat16 is now a pipeline that gets a warning and a different dtype.

Attention changes underneath you. On MPS with kernels installed, ggml-attn becomes the default attention implementation when loading a GGUF file — the same flash-attention kernel llama.cpp uses for decode and prefill. Passing attn_implementation explicitly still takes precedence, but if you benchmark attention variants, your baseline moved without a line of your code changing.

Architecture coverage is one family. The packed path supports Qwen3.5 and Qwen3.5 MoE. Everything else in the legacy list — Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2 and more — routes to the loader that dequantizes.

Model identifiers gain a suffix. Under transformers serve, each .gguf file in a repository is its own model, named as the repository, a colon, then the file name, because one repository holds several quantizations.

The verdict

Wait, with one exception. If your local target is Qwen3.5 on Apple silicon and you already wanted transformers in the loop, the packed path deletes a runtime from your machine and that is a real saving. Everyone else should treat this as a preview: the file you load is the same file, and the memory you spend is not.

The falsifiable claim is that this does not yet replace llama.cpp in a local deployment, and the thing that would show it wrong is a second entry appearing in that kernel's driver family list. That is one command, and it is cheaper than re-reading the announcement next month:

curl -s https://huggingface.co/api/kernels/ggml-org/ggml-quantization | python3 -m json.tool | grep -A 2 supportedDriverFamilies

Today it prints a list with metal in it and nothing else. When it prints a second entry, the question of whether one runtime can cover both the laptop and the server is worth asking again. Until then, the two-stack split is not a legacy habit anyone needs to apologise for — it is what the published kernels support.