Zero-Shot Classification on a Local LLM: Jeff Accepts 255 Options and Picks From 26

Zero-shot classification is the small, constant decision inside most AI systems: route this ticket, tag this message, pick this intent. Jeff, released yesterday, does it on a local LLM. It fine-tunes Qwen3.5-0.8B, Qwen3.5-2B and Gemma 4 E2B so that one forward pass returns a calibrated probability for every option you describe, with no generated text to parse. The README reports a median of 28 ms per decision for the 0.8B model on an M4 Max under MLX, and 22 ms on an RTX PRO 6000. The 0.8B weights are 1.7 GB at 16-bit.

The README also says a choice question can carry up to 255 options, and the server enforces exactly that limit. The trained Qwen models do not honour it.

What the repository is, by the numbers

The repository was created on 28 September. At the time of this run it had six commits, all from one author, spanning about four hours, no releases or tags, 518 stars, 17 forks, one open issue and no pull requests. Code is MIT, including the upstream AutoJev recipe it forked; the weights are Apache 2.0, and the Hugging Face license tag on all three models agrees. It uses the same request format as Jev, a hosted decision API from TypeSafe, and states that it is not affiliated with it.

On its own five-benchmark panel of 4,599 questions, the README puts Jeff-Qwen3.5-0.8B at 79.1 overall against a published 83.0 for Jev, and Jeff-Qwen3.5-2B at 83.1. On RAGTruth, the grounding benchmark, the 0.8B scores 86.1. The README itself says that the reasoning-heavy benchmarks stay well below the large models, and that the published Jev figures came from a different sample.

The one open issue is the one that matters

Issue #1, opened overnight by a third party, reports that with more than 26 options neither Qwen model ever selected an option coded AA or later. The repro is 30 city names with the correct one moved between position 3 and position 28. At position 3, Jeff-0.8B picked it with probability 0.998. At position 28, it picked Athens and gave the right answer 0.0004.

The reporter then ran real intent taxonomies: BANKING77, CLINC150 and MASSIVE English. Across 1,458 answers from the two models, none landed past Z. Where the right answer sat in positions 1 to 26, the 0.8B was right 80 percent of the time and the 2B 69 percent. Where it sat in positions 27 to 150, both scored zero. Reversing the option order moved the failures with it. The reporter rebuilt the benchmark panel and reproduced the README scores within 0.6 points, so this is the released models behaving as shipped, on the PyTorch backend. Gemma was not tested.

The issue ends with a hypothesis it could not confirm: the training data may never contain a question wider than 26 options. The published recipe supports it.

Why the benchmark could not see it

The model reads its answer through a 255-row readout layer, one row per answer code, initialised from the base model's output weights for the tokens A to Z and then AA onward. In model.py, scores for positions beyond a question's option count are masked out before the loss. Row 27 is therefore only trained by questions that have at least 27 options.

The script the repository labels the recipe behind the released models, scripts/train_all.sh, mixes public datasets, the RAGTruth and WinoGrande training splits, long documents, voice commands, probability questions and teacher-written synthetic data. Reading the builders it calls: TruthfulQA is capped at 10 options, the voice-intent rows at 6, the MAUD document questions at 8 answers, and the widest synthetic family in families.py has 17, A through Q. None of the builders we read produces a question anywhere near 27 options. BANKING77 was skipped because its rows could not be fetched by the download methods the code allows. The benchmark panel itself is built with the 26 codes A to Z, so a failure past Z is invisible to every benchmark score in the README.

What looks like the fix is already in the tree. sft_synthetic.py plans 10,000 intent questions with menus of 32, 64, 128, 160 and 255 options. train_all.sh never calls it, and the model card's list of its 271k training questions does not mention it. As far as the published recipe and model card show, the released weights were not trained with it.

This is the pattern the proof-of-concept post describes: the evaluation matches the demo, and production brings the shape the demo never had. The README's routing example has three options. BANKING77 has 77.

Who should not use it yet

• Anyone routing into a flat list longer than 26 labels. Shortlist first, with a keyword filter or a two-stage choice by category, so every question Jeff sees has 26 options or fewer.
• Anyone who needs a maintained dependency. One author, one day, no tagged release: pin the commit and the weight revision.
• Anyone whose decision needs reasoning rather than recognition. The README says so itself.

Inside those limits, a 0.8B model that runs locally in tens of milliseconds and keeps text on your own hardware is a real option for intent routing and grounding checks. If that is the architecture you are weighing, it is the kind of decision our private LLM deployment work starts from.

The claim, and what would break it

Claim: the 26-option ceiling is a training-coverage gap in the released checkpoints, not a limit of the readout design, so a checkpoint trained with the sft_synthetic.py menus will pick options past Z at roughly the accuracy it shows inside A to Z. If a retrained checkpoint still scores near zero past position 26, the cause is elsewhere, and the right move is to cap requests at 26 permanently.

Until then, run this over any request you send to Jeff. It prints the name and option count of every question wider than the trained range, and an empty object when there is none:

jq -c '.questions | map_values(.criteria | length | select(. > 26))' request.json

Any key in that object means the question can silently return the wrong answer with high confidence. Split it before it reaches the model.