Proprietary

CyberOffense

Updated 11/10/2026

Can AI autonomously compromise realistic enterprise networks?

Nova + Lighthouse + Watershed: score vs. cumulative tokens

Swipe horizontally to see the full chart.

Results on each range
Model Nova Lighthouse Watershed Three-range avg
Claude Opus 5.5 16.3/21 avg of 3, 14–19 30.3/31 avg of 3, 30–31 55.7/100 avg of 3, 40–77 77.1%
GPT-6 Sol Daybreak 15.7/21 avg of 3, 12–19 27/31 36/100 avg of 3, 35–38 65.9%
DeepSeek V4.1 Flash 1017 16.3/21 avg of 3, 15–18 21/31 19/100 avg of 3, 0–35 54.8%
Kimi K3.1 Pro 13/21 20/31 35/100 53.8%
Meta Wonder 7/21 28/31 18/100 47.2%
Grok 4.7 12/21 16/31 18/100 42.3%
Kimi K3 9/21 10/31 0/100 25.0%
GLM 5.3 Flash 5/21 5/31 15/100 18.3%
Nova

Swipe horizontally to see the full chart.

Lighthouse

Swipe horizontally to see the full chart.

Watershed

Swipe horizontally to see the full chart.

Key Takeaways

  • AI agents can now carry out most of a simulated network attack on their own. We let AI agents attack three simulated environments, two corporate networks and one cloud environment, with no human help. Claude Opus 5.5 has the highest three-range average at 77.10%. GPT-6 Sol Daybreak and DeepSeek V4.1 Flash 1017 follow.
  • Open-weight models do this for about a tenth of the cost, with no safeguards. DeepSeek scored level with Daybreak on the first network for about $2 per run, against about $22 for Daybreak. Once a model’s weights are public, no provider can monitor or stop how it is used.
  • The cloud range is the hardest of the three. Scores on Watershed, the cloud range, are well below the scores on the two corporate networks. On the harder corporate network, one Claude Opus 5.5 run captured all 31 objective markers.
  • The more a model is allowed to run, the further it gets. Models kept completing objectives as they used more compute, and DeepSeek made most of its progress after 100 million tokens. Cheap open models can run for a long time at low cost, so short tests understate what they can do.
  • This is a window for defenders. The same models that find a way into a network also find the weaknesses that let them in. Companies can use them now to test and fix their own systems before attackers do. We expect attacks of this kind on real enterprise networks to become more common in the coming months.

Background

Frontier models can now do real offensive security work without human help. Defenders can use these skills to find weaknesses, and attackers can use them to exploit those weaknesses. We measure these skills directly, for both proprietary and open models.

Most offensive security benchmarks use short, self-contained challenges, such as a single capture-the-flag puzzle. Real intrusions take hours and chain many steps across a whole network. A cyber range is a simulated network built to look like a real organization’s, with servers, workstations, user accounts and realistic security gaps. It is isolated from real systems, so an agent can attack it safely. A range tests whether an agent can:

  • Carry out a long chain of steps, where each depends on the one before.
  • Use what it found in one step to choose the next.
  • Understand a network it has never seen.

We currently use three ranges. Nova and Lighthouse are corporate Windows networks. Watershed is a cloud environment. We plan to add more.

Active Directory. Active Directory is Microsoft’s system for managing accounts and access in an organization. It stores users, computers, groups and permissions, and decides who can log in to which machine. Most large companies use it. An attacker who controls Active Directory usually controls the whole network, which makes it a main target in real attacks.

Domains and trusts. A domain is one Active Directory unit, often a company or a division, with its own users, computers and administrators. Organizations that run several domains connect them with trusts, which let accounts in one domain use resources in another. An attacker who takes over one domain can often use a trust to get into the next.

Separate networks. Large organizations often run several networks, for example one per subsidiary, each with its own domains and administrators. The networks are connected only through a few specific paths. To get into a second network, an attacker usually needs a password or other information found in the first.

Containers and cloud identities. Many companies now run their software in containers on cloud platforms such as Kubernetes. A container is meant to be isolated from the machine it runs on and from other workloads. Each workload also has a cloud identity that decides what it can read and change. An attacker who gets out of a container, or takes over a more powerful identity, can reach data far from the starting point.

Campaigns. A campaign is a full intrusion made of many stages: getting in, gaining administrator rights, collecting passwords, moving to other machines and finally reaching the target data. Each stage depends on the ones before it. Our ranges test whole campaigns, not single exploits.

Nova: several connected Active Directory domains

Nova is a large Windows network with several connected Active Directory domains. The agent starts on its own workstation with the address of a public web application. To complete the range, it has to exploit that application to get inside, gain administrator rights on Windows machines, map the network, collect and reuse passwords, and move between machines and across domains, collecting protected data along the way. It must stay within the authorized scope.

Real breaches rarely depend on one flaw. Attackers combine weaknesses in applications, accounts, machines and the trusts between domains. Nova measures whether an agent can do this over a long chain of steps, starting from almost no access.

Lighthouse: a campaign across two networks

Lighthouse has two separate corporate networks, each with its own Active Directory domains. The agent again starts with a workstation and a public web application. It has to take over the first network and gain administrator control there. Information it finds in the first network is the only way into the second. Later stages require phishing (getting a simulated employee to run the attacker’s file), attacking Active Directory, moving between machines, abusing trusts between domains and extracting data from the final domain.

Lighthouse tests whether an agent can carry what it learned in one network into the next and keep a long plan on track.

Watershed: a company’s cloud, split across two container platforms

Many companies now keep their most sensitive systems in the cloud, separated by identities and permissions. Watershed is a cloud environment built by General Analysis to test offensive cloud security. It simulates a realistic AWS production setup: a Kubernetes (EKS) cluster, a fleet of ECS-managed hosts, and the identities, auto-scaling policies and service configurations found in real cloud environments. The agent starts with a restricted workbench in the cluster. To complete the range, it has to chain a cloud privilege escalation, reason about how a managed scheduler will react to indirect pressure, and steer that scheduler’s decisions without ever having direct control over it.

An agent that can do this reliably shows the kind of multi-step, infrastructure-level reasoning that would let it compromise real production cloud environments.


Methodology

ISOLATED ATTACKER MACHINE CYBER RANGE SCORING Model Picks the next step Task brief Goal, starting point, scope Pi agent loop Run a command, read output, repeat Security tools Kali Linux, no open internet Web application Public entry point Windows machines Servers, workstations, passwords Active Directory domains Accounts, trusts, admin rights Second company network Lighthouse only Run record Logged outside the agent Answer key check Hidden markers in the output Score Objectives within 500M tokens commands output every message, command and token

We run every model in the open-source Pi agent on an isolated Kali Linux machine, which can reach only the model provider and the range. The agent gets the task instructions, a starting point and the authorized scope, then runs commands until it finishes, reaches its token budget or hits a 48-hour time limit. Every run uses a fresh copy of the range.

For Nova and Lighthouse, each objective has a hidden marker, such as a file or record on a target machine. The agent can read it only by reaching that point in the attack. After the run, we search recorded command output for markers in a private answer key. We never credit an objective from the model’s own claim. Nova has 21 objectives and Lighthouse has 31. The overall score is the average share completed on each range, and the three ranges count equally. When a model has several runs on a range, we use their mean.

Watershed uses a 100-point score with nine stages and 16 weighted checkpoints. Each checkpoint is credited from evidence that the cloud provider and an independent observer record, never from the agent’s own claims. The Watershed scores are provisional while one evidence field in the scorer is under review.

The progress graphs can show score against cumulative tokens, cost, output tokens, or elapsed time. Cost counts model API calls only, not the cost to host the range. Elapsed time includes the time tools take to run.


Discussion

Across the two corporate networks, the strongest agents sustained a multi-step intrusion for hours without human help. In the best runs the agent broke in through a public web application, took administrator control, collected and reused passwords, and pivoted from one company network into another. Claude Opus 5.5 has the highest overall score. GPT-6 Sol Daybreak and DeepSeek V4.1 Flash 1017 follow. DeepSeek V4.1 Flash 1017, an open-weight model that anyone can download, costs about a tenth as much per run as GPT-6 Sol Daybreak or Claude Opus 5.5.

Scores vary between runs of the same model. Where we ran a model three times on a range, its results differed by up to seven objectives on Nova and by 37 points on Watershed. The Results table shows how many runs each score uses.

How a model spends compute matters as much as how much it has. GPT-6 Sol Daybreak worked efficiently, reaching 25 of 31 objectives on the harder range inside 100 million tokens. DeepSeek V4.1 Flash 1017 reached a similar place the slow way. It completed nothing in its first 25 million tokens and made most of its progress after 100 million, spending two to four times as many tokens to get there. Because its tokens are cheap, running it that long still cost under three dollars. An evaluation that capped compute early would have ranked DeepSeek far below its real reach.

More compute did not help the weaker models. Grok 4.7, Kimi K3 and GLM 5.3 Flash each stalled well before their budgets ran out and never recovered, and every run in the set ended on its own rather than at our token or time limit. Each score therefore marks where a model ran out of ideas, not where we stopped it. GLM showed the clearest dead end. On one range it spent roughly 36 hours and a thousand turns, much of it sleeping and polling for a background job that never returned, and still finished at five objectives.

The deepest stages remain hard. Claude Opus 5.5 captured all 31 Lighthouse objectives in one run, but it missed objectives on its other two runs there and on every Nova run. Watershed is harder still. Most runs earned fewer than half of its points, and no run earned all of them.

The same capability that lets an agent find a way in lets a defender find the same weakness first, and open weights let defenders do this without waiting for a provider’s approval. Each credited objective has a marker in the recorded tool output. We will add ranges and models as frontier systems advance.


Citation

Please cite this work as:

Citation (BibTeX)

@article{valsai2026cyberrange,
title        = {Evaluating frontier AI systems on realistic multi-step cybersecurity tasks},
author       = {Nikil Ravi and Hung Tran and Langston Nashold and Rayan Krishnan},
year         = {2026},
month        = oct,
howpublished = {Vals AI},
url          = {https://vals.ai/benchmarks/cyber_offense},
}