AI Agent Evaluation by Repetition: ThinkingBox's 20-of-20 Column Is Not What Its Own Tool Prints

The claim, and the column it swaps

Microsoft's Copilot Studio team and Hugging Face published ThinkingBox on 3 October. It has 507 business workflows across five domains, each run 20 times from a clean backend and graded on the records the agent leaves behind. The harness also landed in OpenEnv, so teams can now run it and quote their own numbers against the published table.

The headline is a column swap. Rank by pass@1 and Claude Opus 5.5 leads at 67.16%. Count the tasks a model passed on all 20 attempts and Opus 5.5 and Opus 5 tie at 241. Kimi-K3 solves 476 tasks at least once but passes only 68 every time. For AI agent evaluation that's the right correction, and the blog says its 20-of-20 figure is a literal count, with no estimator.

The harness it tells you to run doesn't print that count.

What the tool actually reports

The released aggregator, tb agg in microsoft/thinkingbox, computes pass^k as the mean over tasks of (c/n)^k, where c is passes and n is runs. The function's docstring calls this a deliberately biased estimator. Its stated reason: the unbiased version is zero whenever a task misses a single run, which leaves hard tasks indistinguishable. The release README describes the same figure as an estimate that all 20 attempts succeed.

The arithmetic follows from that. Under (c/n)^20, a task that passed 19 of 20 runs adds 0.36 of a task to the score, and one that passed 18 adds 0.12. So the tool's pass^20 is never below the literal every-run share, and it rises with every near miss. Inconsistent models are where near misses pile up.

The two numbers already disagree in public. The arXiv abstract gives Kimi-K3 a pass^20 of 17.60%, while the blog counts 68 of 507 tasks, or 13.41%. 17.60% of 507 is 89.2 tasks, which can't be a count. Either the two come from different runs or they measure different things. If you run the benchmark and set your tb agg line beside the blog's table, you're comparing your estimate with their count.

What a pass requires

The benchmark repository at tag thinkingbox-bench-v1.0 shows how strict a single pass is. For every task, validate_database compares a SHA-256 hash of the whole final database with a hash of the golden state. Free text is handled sensibly: ticket subject and description are left out of the hash, and timestamps and new-row ids are fixed in the tool code. The fields that carry decisions are hashed: status, priority, type, tags, and nearly all of each domain's custom fields.

We parsed the golden trajectories of all 507 tasks in the release test list. In 506 of them, the expected end state sets hashed ticket fields, a median of ten per task. In 144 the ticket is the only thing that changes, with no refund, booking change, RMA or access grant. 362 tasks also require a write to a business system, and one requires no write at all. The split varies by domain: 16 of 98 retail tasks are ticket-only, against 46 of 104 neobank IT tasks. Every domain policy tells the agent never to use ticket comments, and none of the golden runs writes one. An agent that documents its work in a comment fails on the extra row.

That isn't a defect, it's a definition: a 20-of-20 means the agent chose the same priority, type and tags twenty times, as well as completing the business action.

Where the measurement flips

The boundary is your definition of done. Suppose your operators triage from ticket fields and your reporting runs on them. Then ThinkingBox measures what you care about, and the every-run column is the one to read. If your production check is only whether the refund or the booking change landed, it's different. The benchmark's failures then include label disagreements nobody would page you for, and the every-run count can understate what you'd actually get. For 144 of the tasks, there's no business action to check at all.

A second boundary is about a result's integrity, not its meaning. Issue 35 on microsoft/thinkingbox was filed on 21 September by a reporter outside the project. It's still open with no reply, and the cited code is unchanged on main. It shows the Session Proxy handling a reserved dispatch name before it applies the scenario's tool allowlist. Any direct client of the proxy can call tools the scenario excluded and replace the session's world state through the reserved init tool. The data server's effects tool returns the golden database state, and it sits behind that same dispatch.

We checked both paths Microsoft ships. The harness's agent loop rejects any tool name it didn't advertise to the model. The OpenEnv adapter rejects reserved names and anything outside its allowlist. A custom agent pointed straight at the proxy's MCP endpoint gets neither check from ThinkingBox. The blog says golden state stays on the evaluator side. That's true for the bundled paths, not for every way of running the benchmark.

What to check before you quote a number

The argument in From PoC to Production applies directly: evaluation you can defend means knowing what the number counted. The same model's all-20-runs figure is 13.41% in one Microsoft document and 17.60% in another.

Our central claim: a tb agg pass^20 figure isn't comparable to the blog's every-run column, and for any model with near misses it comes out higher. Kimi-K3's numbers alone would be enough to show the claim wrong in one specific way. If a released per-task results file showed 89 tasks passing all 20 runs, the gap would be a data error, not an estimator choice.

Before a ThinkingBox number goes into a model decision:

1. Report the literal count next to the tool's estimate:

uv run tb agg -f json output_thinkingbox_bench_v1.jsonl | jq '{every_run: ([.per_test[] | select(.passed == .runs)] | length), tasks: (.per_test | length), pass_pow_20: (.metrics.pass_power_k[] | select(.[0] == 20) | .[1])}'

1. Run your agent through the bundled loop or the OpenEnv adapter, not directly against the proxy, until this prints CLOSED:

gh issue view 35 --repo microsoft/thinkingbox --json state --jq .state

1. Weight the per-domain tables by your own work. If yours end in business-system writes, retail and travel, where 82 and 83 tasks require one, say more than the overall row.
2. In production, check terminal state before you commit, the same way the benchmark grades. Building that check into the agent, not only into its evaluation, is most of what AI agent development for systems of record comes down to.