Key Takeaways
- AI agents can now carry out most of a simulated network attack on their own. We let AI agents attack three simulated environments, two corporate networks and one cloud environment, with no human help. Claude Opus 5.5 has the highest three-range average at 77.10%. GPT-6 Sol Daybreak and DeepSeek V4.1 Flash 1017 follow.
- Open-weight models do this for about a tenth of the cost, with no safeguards. DeepSeek scored level with Daybreak on the first network for about $2 per run, against about $22 for Daybreak. Once a model’s weights are public, no provider can monitor or stop how it is used.
- The cloud range is the hardest of the three. Scores on Watershed, the cloud range, are well below the scores on the two corporate networks. On the harder corporate network, one Claude Opus 5.5 run captured all 31 objective markers.
- The more a model is allowed to run, the further it gets. Models kept completing objectives as they used more compute, and DeepSeek made most of its progress after 100 million tokens. Cheap open models can run for a long time at low cost, so short tests understate what they can do.
- This is a window for defenders. The same models that find a way into a network also find the weaknesses that let them in. Companies can use them now to test and fix their own systems before attackers do. We expect attacks of this kind on real enterprise networks to become more common in the coming months.
Background
Frontier models can now do real offensive security work without human help. Defenders can use these skills to find weaknesses, and attackers can use them to exploit those weaknesses. We measure these skills directly, for both proprietary and open models.
Most offensive security benchmarks use short, self-contained challenges, such as a single capture-the-flag puzzle. Real intrusions take hours and chain many steps across a whole network. A cyber range is a simulated network built to look like a real organization’s, with servers, workstations, user accounts and realistic security gaps. It is isolated from real systems, so an agent can attack it safely. A range tests whether an agent can:
- Carry out a long chain of steps, where each depends on the one before.
- Use what it found in one step to choose the next.
- Understand a network it has never seen.
We currently use three ranges. Nova and Lighthouse are corporate Windows networks. Watershed is a cloud environment. We plan to add more.
Nova: several connected Active Directory domains
Nova is a large Windows network with several connected Active Directory domains. The agent starts on its own workstation with the address of a public web application. To complete the range, it has to exploit that application to get inside, gain administrator rights on Windows machines, map the network, collect and reuse passwords, and move between machines and across domains, collecting protected data along the way. It must stay within the authorized scope.
Real breaches rarely depend on one flaw. Attackers combine weaknesses in applications, accounts, machines and the trusts between domains. Nova measures whether an agent can do this over a long chain of steps, starting from almost no access.
Lighthouse: a campaign across two networks
Lighthouse has two separate corporate networks, each with its own Active Directory domains. The agent again starts with a workstation and a public web application. It has to take over the first network and gain administrator control there. Information it finds in the first network is the only way into the second. Later stages require phishing (getting a simulated employee to run the attacker’s file), attacking Active Directory, moving between machines, abusing trusts between domains and extracting data from the final domain.
Lighthouse tests whether an agent can carry what it learned in one network into the next and keep a long plan on track.
Watershed: a company’s cloud, split across two container platforms
Many companies now keep their most sensitive systems in the cloud, separated by identities and permissions. Watershed is a cloud environment built by General Analysis to test offensive cloud security. It simulates a realistic AWS production setup: a Kubernetes (EKS) cluster, a fleet of ECS-managed hosts, and the identities, auto-scaling policies and service configurations found in real cloud environments. The agent starts with a restricted workbench in the cluster. To complete the range, it has to chain a cloud privilege escalation, reason about how a managed scheduler will react to indirect pressure, and steer that scheduler’s decisions without ever having direct control over it.
An agent that can do this reliably shows the kind of multi-step, infrastructure-level reasoning that would let it compromise real production cloud environments.
Methodology
We run every model in the open-source Pi agent on an isolated Kali Linux machine, which can reach only the model provider and the range. The agent gets the task instructions, a starting point and the authorized scope, then runs commands until it finishes, reaches its token budget or hits a 48-hour time limit. Every run uses a fresh copy of the range.
For Nova and Lighthouse, each objective has a hidden marker, such as a file or record on a target machine. The agent can read it only by reaching that point in the attack. After the run, we search recorded command output for markers in a private answer key. We never credit an objective from the model’s own claim. Nova has 21 objectives and Lighthouse has 31. The overall score is the average share completed on each range, and the three ranges count equally. When a model has several runs on a range, we use their mean.
Watershed uses a 100-point score with nine stages and 16 weighted checkpoints. Each checkpoint is credited from evidence that the cloud provider and an independent observer record, never from the agent’s own claims. The Watershed scores are provisional while one evidence field in the scorer is under review.
The progress graphs can show score against cumulative tokens, cost, output tokens, or elapsed time. Cost counts model API calls only, not the cost to host the range. Elapsed time includes the time tools take to run.
Discussion
Across the two corporate networks, the strongest agents sustained a multi-step intrusion for hours without human help. In the best runs the agent broke in through a public web application, took administrator control, collected and reused passwords, and pivoted from one company network into another. Claude Opus 5.5 has the highest overall score. GPT-6 Sol Daybreak and DeepSeek V4.1 Flash 1017 follow. DeepSeek V4.1 Flash 1017, an open-weight model that anyone can download, costs about a tenth as much per run as GPT-6 Sol Daybreak or Claude Opus 5.5.
Scores vary between runs of the same model. Where we ran a model three times on a range, its results differed by up to seven objectives on Nova and by 37 points on Watershed. The Results table shows how many runs each score uses.
How a model spends compute matters as much as how much it has. GPT-6 Sol Daybreak worked efficiently, reaching 25 of 31 objectives on the harder range inside 100 million tokens. DeepSeek V4.1 Flash 1017 reached a similar place the slow way. It completed nothing in its first 25 million tokens and made most of its progress after 100 million, spending two to four times as many tokens to get there. Because its tokens are cheap, running it that long still cost under three dollars. An evaluation that capped compute early would have ranked DeepSeek far below its real reach.
More compute did not help the weaker models. Grok 4.7, Kimi K3 and GLM 5.3 Flash each stalled well before their budgets ran out and never recovered, and every run in the set ended on its own rather than at our token or time limit. Each score therefore marks where a model ran out of ideas, not where we stopped it. GLM showed the clearest dead end. On one range it spent roughly 36 hours and a thousand turns, much of it sleeping and polling for a background job that never returned, and still finished at five objectives.
The deepest stages remain hard. Claude Opus 5.5 captured all 31 Lighthouse objectives in one run, but it missed objectives on its other two runs there and on every Nova run. Watershed is harder still. Most runs earned fewer than half of its points, and no run earned all of them.
The same capability that lets an agent find a way in lets a defender find the same weakness first, and open weights let defenders do this without waiting for a provider’s approval. Each credited objective has a marker in the recorded tool output. We will add ranges and models as frontier systems advance.
Citation
Please cite this work as:
Citation (BibTeX)
@article{valsai2026cyberrange,
title = {Evaluating frontier AI systems on realistic multi-step cybersecurity tasks},
author = {Nikil Ravi and Hung Tran and Langston Nashold and Rayan Krishnan},
year = {2026},
month = oct,
howpublished = {Vals AI},
url = {https://vals.ai/benchmarks/cyber_offense},
}