Background
Autoform-Hard is the hardest slice of AutoformBench (Meta FAIR, 2026), a benchmark of 341 Lean 4 formalization tasks drawn from 24 graduate- and research-level textbooks in algebraic geometry, arithmetic geometry, number theory, Lie theory, algebraic topology, graph theory, complexity theory and more. Each task gives the agent a Lean project whose target theorems are stated but left as sorry, and asks it to prove them.
The statements were written and audited by PhD-level experts and chosen to go beyond what Mathlib already covers. Solving a task usually means building theory, not writing one proof: accepted solutions in the original study add a median of about 300 lines of Lean, and the hardest run to thousands.
Methodology
The board uses 56 tasks: every task in the benchmark’s 100-task validation set that none of the ten frontier models in the original study solved, except three that duplicate another task. Our audit corrected five of them: three statements were false as written, and two had packaging errors in the starting files. So a few tasks differ from the versions the original models attempted.
- Agent. Every model runs in the same agent harness with a shell in a sandbox containing the Lean toolchain and a prebuilt Mathlib. Each task gets one attempt.
- Limits. Each task has a 3-hour wall-clock limit, excluding provider retry time, and a $200 cost cap at the model’s list price. An agent may give up, but a report that the task is unsolved is rejected until the agent has spent $20 on it.1
- Grading. A submission is accepted only if it passes clean-room grading: the graded theorems are rebuilt from a trusted copy of the task, their statements must match the originals exactly, the benchmark’s definitions must be unchanged, and the proofs may use only the standard axioms (
propext,Quot.sound,Classical.choice). Accepted proofs are then replayed through an independent kernel. Proofs that usesorry,native_decide, new axioms or redefined benchmark objects fail. - Score. The percentage of the 56 tasks whose submission is accepted. A task that ends in an error counts as unsolved.
Footnotes
-
This rule matters. GPT-6.2 Astra reported Smoktunowicz’s gap theorem unsolved after spending about $8; sent back to work, it went on to formalize it. See below to learn how! ↩