Proprietary

CyberOffense

Updated 10/23/2026

Can AI autonomously compromise realistic enterprise networks?

Share of objectives completed vs. tokens used
Objectives completed on each range
Model Nova Lighthouse
GPT-6 Sol Daybreak 15.7/21 avg of 3, 12–19 27/31
DeepSeek V4.1 Flash 1017 16.3/21 avg of 3, 15–18 21/31
Kimi K3.1 Pro 13/21 20/31
Meta Wonder 7/21 28/31
Grok 4.7 12/21 16/31
Kimi K3 9/21 10/31
GLM 5.3 Flash 5/21 5/31
Nova
Lighthouse

Key Takeaways

  • Cyber capability that was limited to trusted-access AI is now available to anyone. We let AI agents attack two simulated company networks on their own, with no human help. The best result came from GPT-6 Sol, which OpenAI offers only through its restricted Daybreak cyber program. An open-weight model that anyone can download, DeepSeek V4.1 Flash 1017, came close behind at 72.8% of objectives, against 80.8% for Daybreak.
  • Open-weight models do this for about a tenth of the cost, with no safeguards. DeepSeek scored level with Daybreak on the first network for under $3 per run, against about $22 for Daybreak. Once a model’s weights are public, no provider can monitor or stop how it is used.
  • AI agents can now carry out most of a simulated network attack on their own. On the harder network, GPT-6 Sol Daybreak broke into the first company, took administrator control, stole passwords and used them to reach a second company, completing 27 of 31 objectives in under three hours.
  • The more a model is allowed to run, the further it gets. Models kept completing objectives as they used more compute, and DeepSeek made most of its progress after 100 million tokens. Cheap open models can run for a long time at low cost, so short tests understate what they can do.
  • This is a window for defenders. The same models that find a way into a network also find the weaknesses that let them in. Companies can use them now to test and fix their own systems before attackers do. We expect attacks of this kind on real enterprise networks to become more common in the coming months.

Background

Frontier models can now do real offensive security work without human help. Defenders can use these skills to find weaknesses, and attackers can use them to exploit those weaknesses. We measure these skills directly, for both proprietary and open models.

Most offensive security benchmarks use short, self-contained challenges, such as a single capture-the-flag puzzle. Real intrusions take hours and chain many steps across a whole network. A cyber range is a simulated network built to look like a real organization’s, with servers, workstations, user accounts and realistic security gaps. It is isolated from real systems, so an agent can attack it safely. A range tests whether an agent can:

  • Carry out a long chain of steps, where each depends on the one before.
  • Use what it found in one step to choose the next.
  • Understand a network it has never seen.

We currently use two ranges, Nova and Lighthouse, and plan to add more.

Active Directory. Active Directory is Microsoft’s system for managing accounts and access in an organization. It stores users, computers, groups and permissions, and decides who can log in to which machine. Most large companies use it. An attacker who controls Active Directory usually controls the whole network, which makes it a main target in real attacks.

Domains and trusts. A domain is one Active Directory unit, often a company or a division, with its own users, computers and administrators. Organizations that run several domains connect them with trusts, which let accounts in one domain use resources in another. An attacker who takes over one domain can often use a trust to get into the next.

Separate networks. Large organizations often run several networks, for example one per subsidiary, each with its own domains and administrators. The networks are connected only through a few specific paths. To get into a second network, an attacker usually needs a password or other information found in the first.

Campaigns. A campaign is a full intrusion made of many stages: getting in, gaining administrator rights, collecting passwords, moving to other machines and finally reaching the target data. Each stage depends on the ones before it. Our ranges test whole campaigns, not single exploits.

Nova: several connected Active Directory domains

Nova is a large Windows network with several connected Active Directory domains. The agent starts on its own workstation with the address of a public web application. To complete the range, it has to exploit that application to get inside, gain administrator rights on Windows machines, map the network, collect and reuse passwords, and move between machines and across domains, collecting protected data along the way. It must stay within the authorized scope.

Real breaches rarely depend on one flaw. Attackers combine weaknesses in applications, accounts, machines and the trusts between domains. Nova measures whether an agent can do this over a long chain of steps, starting from almost no access.

Lighthouse: a campaign across two networks

Lighthouse has two separate corporate networks, each with its own Active Directory domains. The agent again starts with a workstation and a public web application. It has to take over the first network and gain administrator control there. Information it finds in the first network is the only way into the second. Later stages require phishing (getting a simulated employee to run the attacker’s file), attacking Active Directory, moving between machines, abusing trusts between domains and extracting data from the final domain.

Lighthouse tests whether an agent can carry what it learned in one network into the next and keep a long plan on track.

Methodology

ISOLATED ATTACKER MACHINE CYBER RANGE SCORING Model Picks the next step Task brief Goal, starting point, scope Pi agent loop Run a command, read output, repeat Security tools Kali Linux, no open internet Web application Public entry point Windows machines Servers, workstations, passwords Active Directory domains Accounts, trusts, admin rights Second company network Lighthouse only Run record Logged outside the agent Answer key check Hidden markers in the output Score Objectives within 500M tokens commands output every message, command and token

We run every model in the open-source Pi agent on an isolated Kali Linux machine, which can reach only the model provider and the range. The agent gets the task instructions, a starting point and the authorized scope, then runs commands until it finishes, reaches its token budget or hits a 48-hour time limit. Every run uses a fresh copy of the range.

Each objective has a hidden marker, such as a file or record on a target machine, that the agent can read only by reaching that point in the attack. After the run, we search the recorded command output for the markers in a private answer key. We never credit an objective from the model’s own claim. Nova has 21 objectives and Lighthouse has 31, so the overall score averages the share completed on each range, and both ranges count equally.

We compare models at equal compute. A run counts only the objectives it completed within 500 million cumulative tokens. This total includes every token the model reads and writes on every call, including cached input and reasoning tokens.

Discussion

Across both ranges, the strongest agents sustained a multi-step intrusion for hours without human help. In the best runs the agent broke in through a public web application, took administrator control, collected and reused passwords, and pivoted from one company network into another. GPT-6 Sol Daybreak led at 80.8% of objectives, and DeepSeek V4.1 Flash 1017, an open-weight model that anyone can download, followed at 72.8% for about a tenth of the cost per run. The gap that used to separate a restricted frontier model from a freely available one is now small.

How a model spends compute matters as much as how much it has. GPT-6 Sol Daybreak worked efficiently, reaching 25 of 31 objectives on the harder range inside 100 million tokens. DeepSeek V4.1 Flash 1017 reached a similar place the slow way. It completed nothing in its first 25 million tokens and made most of its progress after 100 million, spending two to four times as many tokens to get there. Because its tokens are cheap, running it that long still cost under three dollars. An evaluation that capped compute early would have ranked DeepSeek far below its real reach.

More compute did not help the weaker models. Grok 4.7, Kimi K3 and GLM 5.3 Flash each stalled well before their budgets ran out and never recovered, and every run in the set ended on its own rather than at our token or time limit. Each score therefore marks where a model ran out of ideas, not where we stopped it. GLM showed the clearest dead end. On one range it spent roughly 36 hours and a thousand turns, much of it sleeping and polling for a background job that never returned, and still finished at five objectives.

The deepest stages remain hard for every model. On the two-network range, no model completed the final objectives, which require carrying stolen access from the first network through to the last target, and only one run of the entire set reached the final stage of the single-network range.

The same capability that lets an agent find a way in lets a defender find the same weakness first, and open weights let defenders do this without waiting for a provider’s approval. We reviewed every run and found no shortcuts. No model read the answer key, and each credited objective traces to a marker read from a machine the agent had genuinely compromised. We will add ranges and models as frontier systems advance.

Citation

Please cite this work as:

Citation (BibTeX)

@article{valsai2026cyberrange,
title        = {Evaluating frontier AI systems on realistic multi-step cybersecurity tasks},
author       = {Nikil Ravi and Hung Tran and Langston Nashold and Rayan Krishnan},
year         = {2026},
month        = oct,
howpublished = {Vals AI},
url          = {https://vals.ai/benchmarks/cyber_offense},
}