MXFP4 Quantization Did Not Improve the Model. A Second Teacher Did.

Nothing went out here over the weekend, so this run reaches back six days rather than dressing a stale item as today's news. The story is a compression result posted on 25 August, and it earns the reach because the paper behind it answers the objection its announcement provoked.

What MXFP4 quantization was claimed to buy

Multiverse Computing published a write-up on Hugging Face for a technique called Quantization-Aware Healing, backed by arXiv 2608.20953, submitted on 21 August. The setup: compress GPT-OSS 120B down to 60B, recover it by continued bfloat16 training distilled from the uncompressed teacher, then re-quantize to MXFP4 while distilling from that same teacher a second time rather than from the recovered checkpoint. The MXFP4 stage ran on eight nodes of H200s under FSDP2, at 32k sequence length and global batch 64, for 400 steps.

The headline: the 4-bit model matches or beats its own bfloat16 source on 7 of 9 benchmarks, by 7.4 points on AA-LCR long-context reasoning and 5.6 on AIME 2025, against losses of 0.2 on MMLU-Pro and 1.4 on SciCode. The write-up's title says the compressed 4-bit model outperforms its full-precision original.

The paper is straighter than the announcement

There is no full-precision original in this pipeline, and the paper says so without being asked. GPT-OSS 120B ships with its MoE weights, the large majority of its parameters, in MXFP4 and the rest in bfloat16. OpenAI's model card confirms it from the other side: the models were post-trained with MXFP4 quantization of the MoE weights, and every evaluation on that card was run under the same quantization. The paper calls the released checkpoints MXFP4 throughout. The only 16-bit thing in the comparison is the recovered 60B student, not the teacher.

The limitations section then does the work the write-up skipped. Every number in the paper is a single run, with no seed variance and no confidence intervals, and the authors point out that AIME 2025 has 30 problems. They set a reading rule alongside it: where two systems land within a point or two, treat the result as a match rather than a win. Apply that to their own table and five of the nine are clear wins. The two smallest gains, LiveCodeBench at 1.0 and IFBench at 1.5, sit inside the band, and so do both losses.

They also name the baseline they did not run: the head-to-head against standard quantization-aware distillation from the recovered bfloat16 checkpoint at matched configuration. They call it the single most valuable experiment to add, and say that until it exists their claim about a recovered-teacher ceiling is motivated rather than measured.

The arm the comment thread found

The write-up's comment thread names a second gap the paper does not. A commenter observed that the two checkpoints in the headline comparison did not receive the same training: the bfloat16 60B got one distillation pass, the recovery, while the MXFP4 60B got that pass plus a second one against the 120B teacher. What separates distilling under quantization from simply distilling for longer is the bfloat16 60B given the identical second pass, same data, steps and teacher. That arm does not exist either.

The article's author agreed on the page, accepted that the table does not isolate the effect, said the caveat would be added, and declined to retract because the shipped numbers reproduce. That is defensible: a team choosing between healing under quantization and quantizing afterwards still learns that the 4-bit checkpoint landed above the bfloat16 one rather than below it. It is not evidence that the 4 bits did it.

Two documents, one checkpoint, different numbers

The paper links the model it shipped: HyperNova 60B 2605 on the Hub, Apache-2.0, 60B total with 4.8B active. That card carries an evaluation methodology the paper does not — vLLM 0.13.0, temperature 1.0, top_p 1.0, reasoning effort high, a repeat count per benchmark (AIME25 ten times, GPQA five, LiveCodeBench three), and a named judge for each judged task.

The numbers move accordingly. For GPT-OSS 120B the card reports 93.7 on AIME25 and 74.6 on GPQA Diamond, where the paper reports 80.0 and 69.0 for the same model on the same two benchmarks. Neither is wrong: one is a single run, the other a ten-run average at a documented reasoning effort. It is why a benchmark number lifted off a vendor page is worth close to nothing unless the harness comes with it.

Where the result stops

The paper draws its own boundary and it is narrow. Every experiment is a GPT-OSS MoE transformer quantized to MXFP4 on a single Nemotron and SmolTalk mixture. No Llama, Qwen or Mistral. No NVFP4, INT4 or FP8. The compression operator behind the 60B student is proprietary, and the authors have not checked whether the gains survive layer pruning, SliceGPT or low-rank decomposition. Differ on any of those axes and this is a hypothesis about your models, not a finding.

One result does travel, and it is operational rather than about accuracy. Against quantization-aware training under matched conditions on a 9B model, QAH peaked in roughly 100 steps to QAT's 700, the peaks 0.3 apart at 54.9 and 54.6. QAT then came apart, shedding close to 19 points by step 1,200, while QAH stayed within about two points of its peak. If that holds on your model, a QAT run needs a held-out early-stopping signal or it ships a checkpoint that has already begun to degrade, and a KL-distilled run does not.

Three things to require before accepting any compressed checkpoint:

• The equal-compute arm. If baseline and candidate did not get the same number of distillation steps against the same teacher, the delta is not attributable to the compression method.
• The base model's native weight format, read off its own card, before accepting a full-precision label on anything downstream of it.
• Repeat counts. On a 30-problem benchmark a single run settles nothing, and a delta under two points is not a result.

The claim here is cheap to falsify: run the bfloat16 60B through the identical second pass. If it still lands below the MXFP4 checkpoint, quantization was doing work and this post is wrong. Until then the honest reading is that the second teacher earned the gain. The pattern is the one that keeps enterprise AI from shipping in a different costume: a real number credited to the wrong mechanism, and a summary that drops caveats the work underneath it stated plainly. A compressed vendor checkpoint is a buy rather than a build — price it on a reproduction, not a table.