ReverseEngBench

Key Takeaways

• ReverseEngBench is the first realistic, contamination-free reverse engineering (RE) benchmark for AI agents, built from more than 5,000 domain-expert hours of clean-room development.
• The benchmark consists of 19 in-house programs averaging over 16,915.8 lines of code, spanning five common RE domains: network protocols, firmware, games, file-format recovery, and malware.
• Programs are paired with a comprehensive 27K-LoC anti-analysis suite of more than ten state-of-the-art protections, over half of which have no public implementations, yielding 262 contamination-free instances and 1,572 deterministically verifiable RE tasks.
• Every agent gets the same professional toolkit inside its sandbox — Ghidra, radare2, GDB, angr, binutils, and dynamic tracers, plus domain-specific tools such as tshark and symbolic-execution and exploitation libraries — driven through a single shell. Disassembly and decompilation appear in nearly every trajectory regardless of how well the model scores, so tool access is not what separates the field.

Background

Much of the software most consequential to cybersecurity reaches analysts only as binaries, without source code. This is true at both ends of the threat landscape: the high-value systems that defenders must protect, and the malicious payloads they must inspect. Proprietary enterprise software, security appliances, and firmware are frequent attack targets yet are typically distributed only in binary form — of the 1,635 vulnerabilities in the authoritative catalog of vulnerabilities exploited in the wild, 46.5% originate from vendors that do not release source code, and Google reports that more than 48% of the zero-days exploited in 2025 targeted enterprise software and appliances. At the other end, attackers deliberately distribute malware as obfuscated binaries to impede inspection, with approximately 732,000 new malicious samples reported each day.

Unlike source code, binary code is represented as raw bytes and is not directly intelligible to analysts. Reverse Engineering (RE) recovers high-level program semantics from this opaque representation, and now underpins binary-first security analysis: before analysts can reason about vulnerabilities, patches, or exploits, they must first determine what a binary does. RE requires capabilities distinct from downstream cybersecurity tasks — interpreting low-level semantics, inferring intent from incomplete evidence, and, in harder cases, overcoming packing and obfuscation. To extend agentic cybersecurity beyond source code, AI agents must be evaluated on RE as a distinct capability.

Constructing an RE benchmark that faithfully reflects real-world practice is challenging for two reasons:

1. Contamination control. RE recovers the semantics of unknown binaries; analysts begin with no prior knowledge of the target. Benchmarks built from public source code break this premise, because the target may appear in pretraining data and be recognizable during evaluation. In RE, even coarse leakage is damaging, as it provides crucial top-down guidance for program interpretation.
2. Real-world scale and protection. RE is fundamentally a program-understanding task, so its difficulty grows sharply with system size and complexity — capability measured on isolated small programs does not extrapolate. High-value targets are also often wrapped in multiple anti-analysis layers, such as sophisticated and sometimes bespoke obfuscation.

These requirements rule out straightforward construction strategies: realistic artifacts are typically derived from open-source projects, which contamination controls exclude, and localized modifications to such projects remain recognizable. ReverseEngBench was therefore built entirely from scratch as a clean-room RE benchmark, with more than 5,000 domain-expert hours invested — to our knowledge, no prior RE benchmark has demanded a comparable investment of expert effort.

Methodology

ReverseEngBench pairs 19 in-house programs (averaging over 16,915.8 lines of code, spanning network protocols, firmware, games, file-format recovery, and malware) with in-house protection settings from a 27K-LoC anti-analysis suite of more than ten state-of-the-art protections, over half of which have no public implementations.

This yields 262 contamination-free instances with realistic difficulty. Each instance defines six scenario-specific, deterministically verifiable tasks that cover the core capabilities required for program understanding, for a total of 1,572 RE tasks.

Compared with existing RE-adjacent benchmarks — which rely on existing CTF challenges or toy programs with easy or textbook protection schemes — ReverseEngBench is RE-centric, contamination-controlled, and built from real-world-scale software with real-world protection complexity.

Results

Fully Solved credits a model with an instance only when it completes all six of that instance’s tasks, while Capability Score is the share of the 1,572 individual tasks it completes. Both metrics are computed over all 262 instances, so instances the agent failed to complete score zero. More models are still being evaluated and will be added as their runs finish.

What makes an instance hard

Every instance is built from a known recipe: the language it was written in, whether it was compiled with optimizations, whether symbols were stripped, whether it was statically or dynamically linked, and whether anti-analysis protection was applied. That lets us ask which of those build properties actually makes a binary hard to reverse engineer. Each panel below varies one property at a time: protection is compared only against the single base that every protected instance is built from (optimized, stripped, statically linked), and the language and build-flag panels use only unprotected instances, so protection does not confound those axes.

Anti-analysis protection here means build-time transformations that leave a program’s behavior unchanged but fight inspection: obfuscated and virtualized control flow that no longer decompiles into readable code, encrypted strings and code that only materialize at runtime, packing, and debugger and tracer detection. Decompiler output degrades and dynamic analysis becomes unreliable, so the agent has to recover semantics from much weaker evidence.

Every agent works in the same shell-only loop with the same reverse-engineering toolkit, so the trajectories are directly comparable. The plot below is derived from the trajectory of every graded instance rather than from the scores.

Early evaluation reveals the limits of current agentic RE: even the strongest cybersecurity-oriented models exhibit limited RE capability. This limitation is most pronounced under in-house anti-analysis protections, which roughly halve success rates. Ablations validate both contamination dimensions — comparing agent performance on private versus public target software, and private versus public protection schemes — and show that symbol information substantially improves performance. Difficulty also varies by language: C is the easiest, likely because of tooling bias and decompilation readability, whereas Rust is the most challenging.

Citation

We are grateful to have co-developed and hosted this benchmark with leading faculty and experts in cybersecurity.

If you use this benchmark in your research, please cite the paper (arXiv preprint, forthcoming).

@misc{spence2026srebench,
title        = {The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark},
author       = {Spence, Jeremy and Assaderaghi, Nicholas and Zhu, Jinhao and Ravi, Nikil and Popa, Raluca Ada and Wei, Guannan and Ding, Yangruibo and Zhang, Zhuo},
year         = {2026},
month        = aug,
howpublished = {Vals AI},
}