To Run Kolibri Locally, the License Is Apache 2.0 and the Runtime Is a Plugin Pinned Below vLLM 0.30

Aleph Alpha released Kolibri on 3 October: an English-German mixture-of-experts model with 78.1B total parameters and 3.46B active per token, full weights on Hugging Face, Apache 2.0. The announcement pitches it as small enough to run on-premise without sending data to a third-party inference service. For a team that wants to run an LLM locally, the license is the easy part. The model card, the config, and its required plugin say where the hard part is.

What shipped

Two repositories: Kolibri-1, with FP8 weights in 128×128 blocks (embeddings, LM head, norms and router stay in bfloat16), and Kolibri-1-BF16. The FP8 repository holds about 78.9 GB of files. The architecture is 50 MoE layers with 384 routed experts, 6 selected per token plus one shared expert, and attention that alternates four sliding-window layers with one full-attention layer.

The model card lists the hardware floor as two A100 80 GB, two H100 SXM5, or one H200, B200 or B300. That is what on-premise means on Aleph Alpha's terms: every configuration it supports is a datacenter GPU node. The card says plainly that the 3B active figure describes compute, not memory, because every expert must stay resident.

Local LLM hardware requirements, from the config

The sliding windows are what make long context cheap here, and the config lets you price it. Only 10 of the 50 layers are full attention, each with 4 KV heads of dimension 128. With the FP8 KV cache the card recommends, that works out to 10,240 bytes of cache per token, before allocator overhead. The 40 sliding layers add a fixed 21 MB per sequence regardless of length.

So a 262,144-token conversation costs about 2.7 GB of KV cache, and a 1,048,576-token one about 10.7 GB. On a two-A100 minimum (160 GB), the weights take roughly half, leaving room for well over a dozen 256k-token caches. The blog's own figure, 18 concurrent 256k-token requests on two H100s, is consistent with that arithmetic.

What the announcement did not say

The runtime is a vendor plugin with a version ceiling. Kolibri's architecture, kolibri1, is not in upstream vLLM: a code search of the vLLM repository for it returns nothing. The card requires aleph-alpha-inference, Aleph Alpha's Apache-2.0 plugin, whose 1.0.0 release on PyPI requires vLLM at least 0.29.0 and below 0.30.0. vLLM 0.30.0 shipped on 22 September. A team already serving other models on 0.30 cannot add Kolibri to that deployment; it runs a second vLLM pinned one minor behind, and owns the upgrade path whenever the plugin lags.

The 1M context is extrapolation past the shipped config. The headline says up to 1M tokens. config.json sets max_position_embeddings to 262,144, which the card calls the native length, the longest one trained. Reaching 1M takes an explicit --hf-overrides flag at serve time, and the card recommends staying at or under 262,144 for latency-sensitive work and complex tasks. Its own RULER table, run on the base model, shows 69.8 at 256k and 63.2 at 1M. Qwen3.5 35B-A3B Base, in the same table, scores higher than Kolibri Base at every length up to 512k and lower only at 1M.

Reasoning defaults to high. The card explains how to pass reasoning_effort but not what happens when you don't. The chat template answers it: an unset reasoning_effort selects the high-effort reasoning instruction, the same branch as passing high. Any client that forwards plain OpenAI-style messages gets the high-effort setting on every call. That is a cost line from the PoC-to-production budget nobody scoped: a pilot that never sets the argument pays for high-effort reasoning on every request, and the fix is one template argument.

The path most local LLM users mean is not there yet

The routes people use to run an LLM locally on their own hardware, llama.cpp, Ollama and mlx-lm, have no merged support. A llama.cpp feature request for Kolibri opened on 3 October is still open. The Ollama repository has an open pull request adding a Kolibri runner on its MLX backend, about 1,400 lines across 19 files. That is the same pattern as the transformers GGUF path that only works when a specific kernel is present: the weights load nowhere until someone writes the architecture.

The community moved faster than upstream. A Hub search on 4 October returned sixteen third-party Kolibri repositories created on release day, some of them plain mirrors. One MLX conversion lists its 4-bit variant at 41 GiB with a peak of 44 to 48 GB, sized for 64 GB Macs, and ships its own kolibri1.py because mlx-lm lacks the architecture. A Q3_K_S GGUF of 33.9 GB ships a 13 KB patch to llama.cpp; a commenter on the llama.cpp issue reports it running on an RTX PRO 6000. Neither is affiliated with Aleph Alpha, and we reproduced neither.

Verdict

Adopt on a GPU node if German is the requirement and you can dedicate a vLLM deployment to it. Wait if the plan is a laptop or an Ollama fleet, because every current path there runs model code that neither Aleph Alpha nor the runtime maintainers have merged. If you are sizing a box for it, our private LLM deployment work starts from exactly this arithmetic.

The claim that could be wrong: through the end of this quarter, the only supported way to serve Kolibri is aleph-alpha-inference on vLLM 0.29.x. A merge of llama.cpp issue 29922's eventual pull request, of Ollama pull request 18780, or a plugin release that accepts vLLM 0.30 would show it wrong.

Before you pilot it:

pip download --no-deps 'aleph-alpha-inference>=1' -d /tmp/aai && unzip -p /tmp/aai/*.whl '*/METADATA' | grep -i requires-dist
pip show vllm | grep -i '^version'

• If your vLLM is 0.30 or later, plan a separate deployment.
• Pass reasoning_effort on every request; set none for extraction and classification.
• Budget KV cache at 10,240 bytes per token per sequence, times concurrent sessions, before you pick a GPU.
• Keep --max-model-len at 262,144 unless your own long-document eval beats the card's 1M RULER score of 63.2.