Chord's INT4 MoE Kernel: Up to 2.15x per Layer, 4 to 8 Percent for a Self-Hosted LLM

The vLLM blog announced Chord on 15 September: an open-source CUDA kernel from Novita AI for the INT4 mixture-of-experts layers in Kimi K2.x, headlined at up to 1.3x on H200 and 2.15x on B300. For a team running a self-hosted LLM of that size, the number to plan with sits further down the same post: 4.1 to 8.0 percent more decode throughput on eight H200s. The repository explains most of that gap, and a version history the post never mentions accounts for more of it.

The claim

Chord replaces the expert matrix multiply for BF16 activations, INT4 weights and group-32 scales. That is the format Moonshot ships: the Kimi-K2.6 config.json declares 4-bit symmetric integer weights with group size 32, pack-quantized through compressed-tensors, across 384 routed experts with 8 active per token.

The post reports per-layer speedups against public Humming, the kernel library vLLM already uses for this path: 1.11 to 1.20x on H200 expert-parallel prefill, 1.17 to 1.33x on H200 tensor-parallel serving, 1.16 to 1.24x on H200 decode, and 1.81 to 2.15x on B300 decode. The code is Apache-2.0, the repository has seven commits with the first on 2 August, and the indexed path loads through vLLM's existing --quantization humming flag.

What the measurement covers

Three boundaries are written into the repository itself.

The B300 ratio is against an untuned baseline. Chord's docs/performance.md says the Blackwell speedups are larger mostly because public Humming ships only its default configuration strategy on that hardware, and tells readers to treat the H200 ratios as the fair comparison. The B300 ratio also decays with load, from 2.15x at 20 tokens per GPU to 1.81x at 50.

The timing covers one operation. The blog's summary table excludes routing, activation and communication. Timing uses the same Triton benchmarking call as upstream Humming, and routing defaults to a random expert assignment rather than production traffic.

There is a point where it flips. On the grouped H200 prefill path at 512 rows per expert, Chord's gate/up stage takes 2318.2 µs against Humming's 2265.0 µs, about 2 percent slower, and the full layer comes out at 1.00x. The blog puts this down to both kernels approaching the same throughput ceiling.

What a self-hosted LLM actually gets

The one end-to-end run served Kimi-K2.6 on 8×H200 with TP8 plus DCP8, an FP8 KV cache and ShareGPT requests. Mean time to first token fell from 2022 ms to 1849 ms, 8.6 percent. Prefill throughput rose 9.6 percent. Decode throughput rose 4.1 percent at batch 8, 5.5 percent at batch 64 and 8.0 percent at batch 128. The post calls this an earlier serving report and gives no vLLM or Humming version for it.

On an eight-GPU node that is worth having. It is not 1.33x, and the distance between a kernel table and a serving bill is the same distance a proof of concept hides, which is why cost control belongs in production terms from day one rather than inherited from a benchmark.

Here is what the announcement does not say. Every kernel table compares against Humming commit 4351af3, tagged v0.1.10 and published to PyPI as humming-kernels 0.1.10 on 2 July. Humming's main branch has 82 commits since that commit. One of them, pull request 57, merged on 14 August, retunes the exact path behind Chord's fair comparison: indexed A16 MoE scheduling on H100 and H200. It first shipped in humming-kernels 0.1.13. The kernel tables predate it, and vLLM still pins an earlier release: version 0.29.0 on PyPI requires humming-kernels 0.1.12.

The claim this post stakes: the H200 per-layer ratios are an upper bound, and against humming-kernels 0.1.13 or later the gap will be smaller than the 1.11 to 1.33x the H200 tables report. Timing the same shapes on H200 with Humming 0.1.13 or later, using Humming's benchmarks/bench_humming.py against Chord's tests/test_w4a16_indexed.py, and getting equal or larger ratios would show this wrong. That run is not published anywhere I read, and I have not run it, since it needs H200s.

The install gives a dependency a second owner

Chord's setup.py packages a top-level humming module so vLLM's loader finds it, and that module's docstring says not to install upstream Humming alongside. But vLLM 0.29.0 on PyPI declares humming-kernels 0.1.12 as a dependency, and Humming's own pyproject.toml packages the same humming module. In a stock vLLM environment, following Chord's instructions leaves two distributions that both ship humming/__init__.py.

The replacement is also narrow. The same docstring scopes Chord's humming to the indexed W4A16 MoE contract and says everything outside it raises NotImplementedError. In an image with Chord installed, any other model routed to the humming backend gets that error instead of a kernel.

What to check before you believe it applies

Choosing which of these controls matters on your hardware is the core of a private LLM deployment, and none of them needs a vendor to answer:

1. Match the checkpoint. The model's quantization_config must show num_bits 4, group_size 32 and format pack-quantized. Anything else fails at load by design.

2. Find out who owns humming. Run this in the serving image:

   python -m pip show -f humming-kernels chord_kernels | grep -E '^(Name|Version|  humming/__init__\.py)'
   

   If humming/__init__.py prints under both names, two packages own one import root.

3. Use two controls, not one. Benchmark end to end on your own traffic against vLLM's pinned Humming 0.1.12, and again against 0.1.13 or later.

4. Set the bar at 4 percent. On H200, adopt only if decode throughput at your production batch size beats the stronger control by at least 4 percent, the lowest end-to-end gain the post reports. Below that you carry a replaced dependency for less than the published floor.

5. On B300, wait. The post lists Blackwell prefill kernels as a future release, and its decode comparison has no tuned baseline.