Key Takeaways
- The Fully Resolved metric is intentionally unforgiving: Claude Opus 5.5 leads at 37 of 200 tasks (18.50%), the next-best results reach only 14 (7.00%) and 13 (6.50%, Claude Sonnet 5.5), and no other model exceeds 11.
- Despite the strict headline metric, top models pass more than 70% of hidden behavioral tests on average (Raw Pass Rate).
- The gap between the two views is wide: Claude Opus 5 achieves an 82.27% Raw Pass Rate — passing most hidden tests — but fully resolves only 6 of 200 tasks, showing that test coverage differs fundamentally from comprehensive solution correctness.
- Kimi K3 is boom-or-bust: it fully resolves 4 tasks and posts a 62.77% Raw Pass Rate — a large jump over Kimi K2.7 Code (49.44%) — but forfeits 22 tasks to zero scores, mostly submissions that fail to build, keeping its average below the frontier.
- The published 2.00% Fully Resolved for Claude Fable 5 counts refusal-triggered fallbacks as successes, and every one of its 200 tasks was affected (100%), so its entire result on this benchmark is fallback-derived. When those fallbacks are counted as failures instead, its score falls to 0.00%. The site provides a toggle to apply this correction.
Background
ProgramBench evaluates whether models can reconstruct command-line programs from an executable binary and a behavioral specification. Each task asks the model to produce an implementation from scratch that compiles and passes the benchmark’s hidden tests.
The benchmark’s headline metric is intentionally strict: a task is counted as Fully Resolved only when the submitted implementation passes all tests. Like the original benchmark, we also report Almost Resolved, the percentage of tasks where at least 95% of hidden tests pass. Finally, Raw Pass Rate reports the average percent of hidden behavioral tests passed per task.
This benchmark was developed by the ProgramBench team; we’d like to thank them for their efforts in building this benchmark. If you’re interested in learning more about ProgramBench, visit programbench.com.
Results
Results summary
| Model | Raw Pass Rate | Almost Resolved | Fully Resolved |
|---|---|---|---|
| Claude Opus 5.5 | 87.0% | 130 | 37 |
| GPT-6 Astra | 85.4% | 100 | 11 |
| GPT-6.1 Sol | 85.3% | 92 | 6 |
| Gemini 4 Argon | 83.7% | 82 | 5 |
| Claude Fable 5.1 | 82.7% | 107 | 14 |
Pass-rate distribution. Claude Fable 5 and Claude Opus 4.8 each pass at least half of hidden tests on 166 tasks. Fable 5 reaches the 95% threshold on 66 tasks. Opus 4.8 reaches it on 31 tasks.
Task-level scores. Of 200 tasks, 131 reach a 95% pass rate for at least one model. Thirteen are fully solved by at least one model. Every task reaches 25% for at least one model.
Interactive analysis
Loading interactive charts and the task-level heatmap.
Methodology
We evaluate models on the 200 public ProgramBench tasks. Each task gives the model a compiled command-line program and a behavioral specification, then asks the model to produce a cleanroom source-code implementation that matches the original program’s behavior.
All models use the same mini-SWE-agent harness, orchestrated through Valkyrie with model calls routed through model-library. The agent has a bash tool, an offline sandbox, and the ProgramBench cleanroom prompt. It may inspect the provided files and run the executable, but it may not use the internet, look up source code or package registries, wrap the provided binary, reuse object files, or use decompilers, disassemblers, tracing, or instrumentation. The provided binary is execute-only.
We follow the public ProgramBench evaluation setup. Models receive 1,000 steps, a 6 hour wall-clock limit, a 180 second action timeout, and 10,000 character head/tail tool-output truncation. A submission contains source files and a compile script. We compile the submission once, then evaluate each hidden test branch in a fresh container created from the compiled image. First-pass branch evaluations use 10 xdist workers.