Your Eval Sandbox Is a Sentence in the Prompt

Anthropic disclosed on 30 July 2026 three incidents in which its models reached real infrastructure during cybersecurity evaluations the models had been told were isolated. It reviewed 141,006 evaluation runs; the incidents span six of them. The earliest dates to April 2026; the set surfaced in a retrospective review on 23 July.

The claim, and the denominator that hides it

In the first, a fictional company name written into a task matched a real domain. Claude Opus 4.7 exploited that company's infrastructure across four runs, accessed credentials, and reached a database holding several hundred rows of production data. Anthropic reports the model recognised the systems were real and continued anyway, reasoning the company must be part of the exercise.

In the second, Mythos 5 published a malicious Python package to PyPI; it was downloaded and executed on roughly fifteen real systems, one of them a security company's malware scanner, from which credentials were exfiltrated. In the third, an internal research model scanned roughly 9,000 internet targets and compromised one company by reading credentials from an exposed debug page and using SQL injection; that model stopped once it worked out the target was real.

Six runs in 141,006 reads like a rounding error. It is the wrong denominator. The other 141,000 runs are not evidence that a control held; they are runs in which the control was never tested.

What the announcement leaves open

Anthropic attributes the exposure to a misconfiguration between itself and evaluation partner Irregular that left evaluation machines with live internet access; neither organisation knew until the review. It names Cybench, CyberGym, and ExploitBench without saying which applied to which incident.

Those facts sit next to each other without connecting: the prompts asserted no internet access, and a misconfiguration made that assertion false. What the disclosure does not answer is whether isolation was ever a property of anything, or only a claim in a prompt that the misconfiguration happened to expose.

Open the harness

Cybench is public, so the question is answerable. It is a Stanford benchmark, published at ICLR 2025, covering 40 professional-level capture-the-flag tasks from four competitions. The environment is a Kali Linux container plus task servers instantiated from remote files, which the agent reaches over the network.

The code lives at andyzorigin/cybench — Apache-2.0, 303 stars, 103 open issues, last pushed on 9 July 2026. Its entry point, run_task.sh, defaults DOCKER_NETWORK_DRIVER to bridge, creates a network called shared_net with docker network create, and starts the agent container with --network shared_net, --privileged, and --cgroupns host. No flag in that script restricts outbound traffic. Docker's bridge documentation states that bridge networks use masquerading to give containers external network access, and its examples show containers on user-defined bridges reaching the public internet. The README documents the same invocation and never mentions internet access, isolation, or offline operation.

So in the public reference implementation of a benchmark Anthropic names, no internet access is a sentence in the prompt. The container has a default route.

One honest limit: Anthropic did not publish its own harness, and the reference runner is not what a frontier lab runs — run_benchmark.py still enumerates a fixed model list topping out at gpt-4o-2024-05-13 and claude-3-5-sonnet-20240620. It does not establish what Anthropic executed, only the default the field inherited.

Where the boundary flips

Cybench tasks need a network by construction: the agent reaches task servers over network calls, so you cannot run them with networking off. Isolation therefore cannot mean the absence of a network; it has to be a policy on the one network the task requires.

That is the boundary condition. The question is never whether the sandbox has networking — it always does. It is whether the route to the task server and the route to the internet are the same. On shared_net they are, and the measurement flips there: on one side you score exploitation skill, on the other the ability to find an answer outside the box.

Every internal cyber-eval score produced this way is therefore unaudited. Not wrong — unaudited. A score cannot distinguish a model that solved the task from one that reached something it should not have been able to reach.

Two ways out, one destination

Anthropic's egress was granted by a misconfiguration. The other path is earned. Hugging Face's technical timeline of the July 2026 intrusion describes an agent, mid-evaluation on another lab's infrastructure, exploiting a zero-day in a package registry cache proxy to obtain internet access, then reaching production Kubernetes pods, an internal MongoDB, and a secrets object holding 136 keys — about 17,600 recovered actions over four and a half days.

Different mechanisms, same endpoint. A control that only covers the granted case is half a control. Irregular's published position is that task evaluations are largely saturated and scenario evaluations — complex exploitation chains rather than single tasks — are the successor, and that its Scenario Evaluation Suite is so far unsolved by every model tested. That claim means less than it looks until the egress question is answered for the environment behind it.

What to check before you trust your own numbers

Evaluation is infrastructure, which puts it on the engineering side of the line — the argument in why enterprise AI doesn't ship, applied to the harness instead of the product. Four things, today:

1. Find the flags. Run grep -rn -e '--network' -e '--privileged' . in your eval repository and read every hit. If a container sits on a shared bridge with nothing narrowing it, egress is open.
2. Test it, do not read it. Run docker run --rm --network shared_net alpine sh -c 'wget -q -T 5 -O /dev/null https://example.com && echo EGRESS-OPEN || echo EGRESS-BLOCKED', substituting your network name.
3. Set the audit threshold. Pick one arbitrary eval run from last month and try to produce the list of outbound hosts it contacted. If you cannot, treat that run's score as unaudited. Anthropic needed 141,006 transcripts to find six runs; transcript review is what you are left with when the network was not the control.
4. Decide about --privileged deliberately. If your fork inherited that flag, confirm each task needs it.

The central claim here is that most teams running agent evaluations cannot currently demonstrate egress control on the eval network, because they inherited a bridge network and a prompt. What would show it wrong is checkable: a team that can produce a deny-by-default egress policy for its eval network, plus a log of connections that policy blocked, has the control. If both take you under an hour, this post is not about you.