Background
ProgramBench evaluates whether models can reconstruct command-line programs from an executable binary and a behavioral specification. Each task asks the model to produce an implementation from scratch that compiles and passes the benchmark’s hidden tests.
The benchmark’s headline metric is intentionally strict: a task is counted as Fully Resolved only when the submitted implementation passes all tests. Like the original benchmark, we also report Almost Resolved, the percentage of tasks where at least 95% of hidden tests pass. Finally, Raw Pass Rate reports the average percent of hidden behavioral tests passed per task.
This benchmark was developed by the ProgramBench team; we’d like to thank them for their efforts in building this benchmark. If you’re interested in learning more about ProgramBench, visit programbench.com.
Results
Results summary
| Model | Raw Pass Rate | Almost Resolved | Fully Resolved |
|---|---|---|---|
| Claude Opus 5.5 | 87.0% | 130 | 37 |
| GPT-6 Astra | 85.4% | 100 | 11 |
| GPT-6.1 Sol | 85.3% | 92 | 6 |
| Gemini 4 Argon | 83.7% | 82 | 5 |
| Claude Fable 5.1 | 82.7% | 107 | 14 |
Pass-rate distribution. Claude Fable 5 and Claude Opus 4.8 each pass at least half of hidden tests on 166 tasks. Fable 5 reaches the 95% threshold on 66 tasks. Opus 4.8 reaches it on 31 tasks.
Task-level scores. Of 200 tasks, 131 reach a 95% pass rate for at least one model. Thirteen are fully solved by at least one model. Every task reaches 25% for at least one model.
Interactive analysis
Loading interactive charts and the task-level heatmap.
Methodology
We evaluate models on the 200 public ProgramBench tasks. Each task gives the model a compiled command-line program and a behavioral specification, then asks the model to produce a cleanroom source-code implementation that matches the original program’s behavior.
All models use the same mini-SWE-agent harness, orchestrated through Valkyrie with model calls routed through model-library. The agent has a bash tool, an offline sandbox, and the ProgramBench cleanroom prompt. It may inspect the provided files and run the executable, but it may not use the internet, look up source code or package registries, wrap the provided binary, reuse object files, or use decompilers, disassemblers, tracing, or instrumentation. The provided binary is execute-only.
We follow the public ProgramBench evaluation setup. Models receive 1,000 steps, a 6 hour wall-clock limit, a 180 second action timeout, and 10,000 character head/tail tool-output truncation. A submission contains source files and a compile script. We compile the submission once, then evaluate each hidden test branch in a fresh container created from the compiled image. First-pass branch evaluations use 10 xdist workers.