Multi Agent LLM Teams: Five Qwen Passes Beat Fable 5 on Ten Problems, and Solve 47 of 100 With Thinking Off

The multi agent LLM claim, stated fairly

GVS5H is a training-free method from a Persis Capital research lab. Several calls to one model take turns as manager, workers and verifier, and talk only through files on disk. Its repository reports Qwen3.8-27B rising from 66.8% on a single call to 92.4% orchestrated, against 90.4% for Fable 5, on 100 hard LiveCodeBench problems.

On 31 October the Memdoor project published a rebuild of that loop as a workflow. On the first ten problems of the authors' list, five passes each, it scored a mean of 9.8 of 10. The authors' own harness scored 9.6 and Fable 5 alone scored 8.0. The repo publishes all 50 runs: 49 solved, $35.79 in total, a median of about 40 minutes of model time and $0.39 per problem.

What the table covers

Three limits sit in the README's own footnote. It is ten problems, not the authors' hundred. The contests date from late March to early April 2025, so training contamination cannot be ruled out. And the Fable 5 column was not run in this harness: it is a single call per problem read from the authors' result files, and it swings from 7 to 10 across the five passes (7, 8, 10, 8, 7).

That swing matters. The team's column barely moves (10, 10, 10, 10, 9). A 1.8-problem gap between a steady team and a single call that ranges across three problems is what ten problems can show, and it is not a ranking. On the authors' hundred problems, per the braindetox analysis of the paper, the team sits 2.0 points above Fable 5, with a 95% lower bound of minus 0.8. That leaves a small deficit open, and the paper's numbers are self-reported and unreplicated.

What the Memdoor table does establish is replication. The repo ships a test that sends the authors' loop and its own workflow the same scripted model and compares every request, byte for byte. All four cases matched. Its 9.8 against the authors' 9.6 says the method transfers to a different engine and host. That is a more useful finding than the Fable column.

Where it flips: thinking off

The same repo contains a second bench that its authors label as not the paper's setting. It runs the team workflow and a single call on all 100 problems, with thinking off, on the same model. The README gives a reason: with thinking on, the model reasoned past the fast host's 32K output cap on a planning prompt and wrote nothing.

Results, which I recomputed from the per-problem files and which match the README's totals: a single call solves 35 of 100 and the team solves 47. Paired, both solve 30, only the team solves 17, and only the single call solves 5. The README states that nothing in that run reaches the 90.4% Fable 5 figure.

I also split those files at the first ten entries of the bench's ids.json. That ten includes arc196_c, arc196_d and abc399_e, the problems the team-solve README names in its ten-problem run, though the README does not state the ten explicitly. With thinking off, the single call solves 2 of those ten and the team solves 4. On the other ninety, they solve 33 and 43. The hosts and providers differ between the two runs, so this is not a clean ablation. What it does show is that the same loop sits at 4 solved on this ten with thinking off and at a 9.8 mean with it on, so the ten-problem score depends on the thinking setting, and a score quoted without it is incomplete.

The cost side needs the same care. The bench run cost $3.27 for 828 model calls. The 50-run thinking-on set cost $35.79 for 49 solves, about $0.73 per solved problem on actual OpenRouter billing. The braindetox write-up of the paper puts the authors' list-price estimate at $0.52 per solved problem for the orchestrated 27B arm and $0.68 for Fable 5, excluding self-hosting. Those are different bases, so the safe reading is that the two do not differ by an order of magnitude, not that one is cheaper.

Verdict, and what would change it

The claim: the supported result is that a file-ledger loop adds solves over a single call at identical settings (17 versus 5 in the paired data), and that Memdoor reproduces the authors' harness. The unsupported result is the ranking against Fable 5 from ten problems. A third-party thinking-on run on all 100 problems that lands within about a point of 90.4%, with a lower bound above zero, would show the ranking holds. A thinking-on single-call run on the same ten that scores like the team would show the loop contributed little.

For teams weighing a self-hosted model plus scaffolding against a paid frontier API, this is the same question Buy, Build, or Fine-Tune asks: decide with a measurement on your own work, not on the headline. Evaluation by repetition covers the other half of this problem: how a score changes once every run counts, not just the average. When the comparison is yours to run, that is agent evaluation work, and the checklist is short.

Before adopting any agent-team result:

1. Pin the thinking setting, output cap and grader on both arms, and write them next to the score.
2. Use at least 100 tasks from your own tickets, not a published ten.
3. Run each arm at least three times and report the range, since a single arm moved across three problems here.
4. Count the paired outcomes, then run an exact sign test on the discordant pairs:

python3 -c 'from math import comb;b,c=17,5;n=b+c;print(2*sum(comb(n,k) for k in range(min(b,c)+1))/2**n)'

Replace 17 and 5 with your only-team and only-single counts. The example prints 0.0169. Switch only if your own value is under 0.05 and the cost per solved task is acceptable.