Decision Model d1-3B Leads Its Size Class on Public Tests. On the Held-Out Split It Ranks 53rd.

A decision model takes a state and a set of typed questions and returns a probability for every allowed answer in one forward pass. Liquid AI released two on 7 October: d1-3B and an experimental d1-omni-600M. The headline claim on both the release post and the model card is that d1-3B is the best decision model under 10B parameters, at 48.57 on Decision Index 0.2.1.

That number is accurate. It comes from an edition the index had replaced the day before. On the current edition, which scores mostly on tests nobody can train on, d1-3B ranks 53rd of 117 entrants.

The claim, stated fairly

The d1-3B card says Liquid AI scored the model with the index's official scorer, and that it was not a leaderboard submission. Every other row in its table is copied from the public 0.2.1 board. That board's data file was generated on 28 September. The rows match it exactly: Winnow-12B at 50.02, JPT-9B at 46.89, Decider 4B at 40.70. Against that table, d1-3B's 48.57 beats every entrant under 10B and also Decider 35B-A3B. Only the 12B Winnow scores higher, which is why the claim is scoped to under 10B. Liquid AI's own release page labels the 48.57 as the public split of 0.2.1.

The model itself is a 3.12B-parameter fine-tune of LFM2.5-VL-3B with a 32,768-token context. The card recommends it for routing, triage, moderation, intent classification and agent guardrails.

What 0.2.1 measures, and what replaced it

Edition 0.2.1 is an average over public benchmarks. The Decision Index Space switched to edition 0.3 in a commit on 6 October. That edition's headline Full score weights the public suite at 20 percent, private tests of the same skills at 50 percent and private tasks from new domains at 30 percent. The two private parts are mapped onto the public scale using untuned stock models. The methodology file gives the reason in plain terms. Most of the score now comes from tests no entrant can train on, so an edge that exists only on public data counts for little.

d1-3B is on that board. Its entry was added on 7 October at 18:25 UTC, about an hour and a half after the model repository's first commit. The entry's note says Liquid AI submitted it, with the public suite run on an AMD MI325X. These are its edition 0.3.1 scores:

• Public suite: 48.99, close to its self-scored 0.2.1 figure.
• Private, same skills: 43.55.
• Private, new domains: 27.96.
• Full score: 40.08, rank 53 of 117.

Decider 4B, the model the release post's own table puts it ahead of, scores 40.33 and ranks 52nd. The board's tie band is 0.9 points, so the two sit inside it. On new domains, Decider 4B scores 40.40 against d1-3B's 27.96.

Where the claim flips

The boundary is the move from public data to private data. On the public suite alone, d1-3B ranks fifth among entrants the board lists below 10B served parameters. On the Full score, 17 of them rank above it. Bespoke Nimble 9B v3 leads that group at 54.67, with 49.05 on new domains. On the new-domain split alone, d1-3B ranks 76th of 117.

The card's own secondary table shows a smaller version of the same pattern. The release post compares d1-3B with Decider 4B on seven public benchmarks, and d1-3B leads on the mean, 82.9 to 81.1. The card's version of that table adds an eighth, HelpSteer2. There d1-3B scores 36.7 against Decider 4B's 42.0, and the means narrow to 77.1 and 76.2. Across the eight rows, each model wins four. Adding a single benchmark halved the lead.

None of this shows Liquid AI trained on public test data, and this post does not claim it did. The board does not say why d1-3B drops on new domains, and nobody outside the board can inspect the private tasks. What the board does show is narrower and more useful. Of the 79 entrants listed below 10B parameters, d1-3B has the widest gap between its public-suite score and its new-domain score: 21 points. For Decider 4B, the gap is under one point.

What survives contact with production

Your routing labels, your ticket taxonomy and your moderation policy are a new domain. No public suite contains them. The closest proxy the board offers for them is the new-domain split, where d1-3B scored 27.96. This is the gap the house post on what a leaderboard can and cannot tell you describes: the public number answers which model is generally strong, and you need to know which model is strong on your data.

Two other things on the card matter before adoption. First, the license is the LFM Open License v1.0. Its commercial-use grant applies only to an entity with less than 10 million US dollars in annual revenue, so above that, commercial use is not licensed. Second, the 8 ms per decision on an RTX 4090 assumes the model is compiled with CUDA graphs. Without that, the card puts a single question at 16 ms. Neither changes the ranking, but both affect whether a pilot can ship.

What you would need to see

The falsifiable version of this post: d1-3B's public-suite lead will not show up on a team's own decision traffic, and its new-domain score is the better guide. Two things would prove that wrong. One is a later index edition where d1-3B's new-domain score reaches Decider 4B's. The other is a held-out evaluation, published by Liquid AI or anyone else, where it leads on data it has never seen.

Before you put d1-3B or any decision model in a request path, run this:

curl -s https://huggingface.co/spaces/multimodalart/jev-decision-index/raw/main/data/v03.json | jq --arg a d1-3b --arg b decider-4b -r '.formula.tie_band, (.models[] | select(.engine==$a or .engine==$b) | [.name, .rank_v, .v, .pub, .same_skills, .new_domains] | @tsv)'

Swap the two --arg values for the engine names of the models you are comparing. Then apply three rules:

1. If a candidate's public score is more than 10 points above its new-domain score, treat the public number as unproven for your traffic. d1-3B's gap is 21.
2. Take 300 decisions from last month's traffic that no prompt, fine-tune or few-shot set has seen. Label them, and score every candidate on exact match. This is the repeatable eval set tied to a business number that the PoC-to-production post lists first among what a pilot skips.
3. Adopt only when the candidate beats your current model by more than the run-to-run spread you measure on that set. A lead smaller than a tie band is not a lead.