Ai2 Audited 16 LLM Benchmarks. Nearly Half the Safety Questions Score Reasoning.

On 1 September Ai2 published BenchMIRT, a multidimensional item response theory fit over per-question results from 100 models on 16 benchmarks and more than 34,000 questions. Nobody told the fit which benchmark measured what. It recovered two dominant dimensions anyway, which Ai2 labels safety and general reasoning.

The headline numbers: the fit predicts a held-out question outcome correctly 79% of the time against 70% for a simpler baseline, and keeping 10% of questions preserves most of what the full benchmark tells you. Ai2 also reports two aggregate oddities — BBQ correlating 0.85 with the reasoning dimension despite being a bias benchmark, and WMDP correlating -0.89 with reasoning, which Ai2 describes as intended.

The reason to read further is that Ai2 shipped the fitted per-item parameters as a downloadable table, which turns a claim about benchmarks into something checkable.

What the item table contains

allenai/BenchMIRT-item-statistics holds 29,574 rows, one per question. Alongside the question text and its benchmark it carries Difficulty 0, Discrimination 0, Difficulty 1 and Discrimination 1 — the item parameters on each of the two recovered dimensions — plus eight boolean columns marking which reduced question sets an item survives into. Two retention levels are materialised: dim0_included_benchmark_0.5 keeps 15,196 items (51.4%) and dim0_included_benchmark_0.9 keeps 3,744 (12.7%).

A separate table, allenai/BenchMIRT-eval-data, holds 34.3k rows under CC-BY-4.0 — one row per question, one column per model accuracy. The item statistics cover 29,574 of the questions evaluated, not all of them.

Discrimination is the useful column: how sharply an item separates strong from weak test-takers on a dimension. An item near zero on a dimension tells you nothing about it.

Group the table by benchmark and take the median absolute discrimination on each dimension:

Dimension 0 is the one the jailbreak suites load on; dimension 1 is the one the reasoning suites load on. BBQ and WMDP sit in the safety half of the suite and behave like the reasoning half. On BBQ, 63.2% of items fall below 0.2 in absolute discrimination on dimension 0. On WMDP it is 68.5%.

That is the number the aggregate correlations do not give you. BBQ and WMDP contribute 5,168 of the 11,279 items in the safety category — 46% of the safety question pool consists of questions whose fitted parameters say they are scoring something else. Anyone reporting a single averaged safety number over this suite is reporting a blend, and the blend is close to half.

BBQ's own paper, by a different author group entirely, describes the dataset as measuring whether model answers reflect social bias across nine social dimensions when context is under-informative. The item parameters say that on this population of models, answering its questions sorts them the way a reasoning test does.

Where the result stops holding

Two boundaries, both of which Ai2 states and one of which the repository makes concrete.

The first is the population. src/download/models_static.py is a hardcoded list, and the published model table agrees with it: 94 instruct models and 6 reasoning models. The six are four DeepSeek-R1 distillations, QwQ-32B and Dolphin3.0-R1-Mistral-24B. Its date column runs from 2023-01 to 2025-03 — 21 models from 2023, 67 from 2024, 12 from 2025 — and Ai2 gives March 2025 as the cutoff. The general-reasoning scores come from Open LLM Leaderboard result datasets for bbh, gpqa, math, mmlu_pro, musr and ifeval, which is why the roster ends where it does.

Every dimension here is therefore a property of that specific population of open-weight checkpoints, 88 of which predate 2025 and none of which is a frontier reasoning model. That is the falsifiable half of this post: re-fit on a roster of the models a team would actually shortlist today and the BBQ and WMDP items may load on safety after all. Until somebody runs it, these item parameters describe those 100 checkpoints, not benchmarks in general.

The second boundary is narrower and easy to miss. Ai2 lists, among its own limitations, that simple benchmark averaging is slightly better than the fit at ranking randomly held-out models. The 79%-against-70% result is about predicting whether a model gets a specific question right. It is not a claim that this method ranks models better than adding up scores.

What to do with it on Monday

Download the parquet and reproduce the table above before you trust the summary:

import collections, statistics
import pyarrow.parquet as pq

t = pq.read_table('items.parquet').to_pydict()
g = collections.defaultdict(list)
for i, b in enumerate(t['benchmark']):
    g[b].append(i)
for b, idx in sorted(g.items()):
    d0 = [abs(t['Discrimination 0'][i]) for i in idx]
    d1 = [abs(t['Discrimination 1'][i]) for i in idx]
    print(b, len(idx), round(statistics.median(d0), 3), round(statistics.median(d1), 3))

Then apply the same shape of test to your own suite. The threshold worth writing down: if more than half your eval items score below 0.2 absolute discrimination on the capability the suite is named after, the suite name is wrong, not the model. You do not need item response theory for a first read — the same table carries a plain correlation column between item outcome and total score, computable from any eval run you already store. Storing per-item outcomes rather than a suite average is the prerequisite, and the same one that makes LLM evaluation metrics mean anything.

This is the failure mode behind a lot of stalled rollouts: a number everyone agrees on and nobody can decompose. It is the measurement half of why enterprise AI does not ship, and taking an aggregate score apart is the first job in LLM consulting.

Maturity, briefly

The code is a paper appendix, not a tool. allenai/BenchMIRT was created on 25 August 2026, has six commits, last pushed 4 September, no tests, no CI, no release tags, no open issues and six stars. Apache-2.0 was added three days after the initial commit. Dependencies pin vllm==0.11.0, transformers==4.57.0 and py-irt==0.7.0, and pull safety-eval from a git branch rather than a tag. The datasets are the part worth using; the repository documents how they were made.