Coding Agent Benchmark: Copilot Fusion Is Two Points Behind Claude Code, and Three Benchmarks Disagree About Why

GitHub made Copilot Fusion generally available on 28 October. It is a router: you pick Fusion and an effort level, and for each prompt it decides which models run and in what pattern. The changelog's headline claim is that Fusion Max scored 60 on the Artificial Analysis Coding Agent Index, within two points of Claude Code with Opus 5.5 at max effort, for about half the average API cost per task, about half the tokens and about 40 percent less time. The benchmark is run by a third party, not by GitHub. It is also an average of three numbers that point in different directions.

What the claim measures

The index is the equal-weight mean of three benchmarks: FrontierCode v1.1 (100 tasks, three attempts each), Terminal-Bench 4.0 (66 tasks) and SWE-Atlas-QnA-AA (124 tasks), per the methodology page. Errors, timeouts and blocked refusals score zero. Cost is the average per-task API cost at provider token prices, not the price of any consumer plan. The methodology page reports no confidence intervals or variance for the index or its parts.

Artificial Analysis's comparison page gives the rows behind the headline:

The article about the same run says 7.2 million tokens for Fusion, so treat that figure as about 7 million. Fusion was run in Copilot CLI, so the comparison is two harnesses as well as two model setups.

Where the two points come from

Fusion leads on Terminal-Bench by 8 points and trails on FrontierCode by 5 and on repository question answering by 9. Averaged with equal weights that is (8 - 5 - 9) / 3, which is the two-point gap. The headline is arithmetic on a mix, and the mix is the benchmark author's choice, not yours.

A team that mostly runs terminal and shell work will see Fusion ahead. A team that mostly asks an agent to read a large repository and answer questions about it will see Claude Code nine points ahead on the one component that measures that. The cost advantage is real in the table, but cost is measured at API token prices for these runs, and I did not find a GitHub or Artificial Analysis page that converts $5.59 into a Copilot bill.

What GitHub's docs add that the announcement did not

The changelog does not say which models Fusion calls. GitHub's concept page does not list them either, saying only that the pool changes over time. Artificial Analysis's article fills the gap: Fusion can route to Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6 Sol. The score is a snapshot of a pool that GitHub says may change.

The docs also state limits that matter for an agent that edits files. Fusion does not start subagents. If a draft that was later discarded had already edited files, those edits are not undone, so GitHub tells you to review changes before committing. Fusion High and Max are not included on Free, Student or Pro plans. Through 30 November, paid plans get a 30 percent discount on model costs when using Fusion; after that date the discount no longer applies, so any cost comparison made now needs repeating.

For administrators, the managed-settings reference lists two keys: model set to fusion, and fusionTier with the values standard, high, max or unmanaged. Any value other than unmanaged locks the tier against user overrides. The reference marks fusionTier as supported in Copilot CLI, VS Code and the Copilot app, and not in JetBrains, even though Fusion itself is generally available there. If a policy assumes the tier is locked, that gap is where the policy would not apply. The same admin-surface problem showed up with assisted permissions, where one lock did not cover a neighboring mode.

The claim, and what would falsify it

My claim: the within-two-points result does not transfer to a team until that team weights the three components by its own work. Using the rounded percentages above, Fusion's lead over Claude Code is 8 x w(terminal) - 5 x w(FrontierCode) - 9 x w(Q&A), where the weights are your shares of each task type and sum to 1. With no repository Q&A at all, Fusion pulls ahead once terminal-style work passes about 38 percent of the mix. With equal thirds it trails by two. The claim is wrong for you if your own runs show Fusion within two points at half the cost on your tasks, whatever the published split.

This is the same point the proof-of-concept to production checklist makes about evaluation: a score on someone else's workload is a prior, not a result. Cost belongs in that evaluation too, and the cache-write analysis from last week shows why list-price comparisons mislead. Teams building on coding agents can start from our AI agent development work.

What to do this week

1. Sort your last 30 agent tasks into terminal work, feature or bug work in code, and repository questions. Those counts are your weights.
2. Run 20 of them on Fusion at High and on your current agent, and record pass rate, wall time and billed cost for each.
3. Plug the weights into the formula above. If the lead is smaller than the run-to-run spread you observe on repeated attempts, you have no evidence of a difference.
4. Before 30 November, price the same 20 runs without the 30 percent discount.
5. If you manage seats, set fusionTier explicitly and confirm which clients enforce it.