Academic

Autoform-Hard

Updated 11/18/2026

Can agents formalize graduate- and research-level mathematics in Lean 4?

As of November 18, 2026, GPT-6.2 Astra ranks first on Autoform-Hard with 87.50%, followed by Claude Fable 5.2 (85.71%) and GPT-6.2 Sol (85.71%).

Autoform-HardCan agents formalize graduate- and research-level mathematics in Lean 4?
Autoform-Hard•Nov 18, 2026
ACCURACY
Lab:
Pinch or Ctrl/Cmd + scroll to zoom. Once zoomed, scroll or swipe with two fingers to pan, or drag to pan. Shift + drag draws a box to zoom into; turn on box zoom to draw one without Shift. Plus and minus keys zoom, arrow keys pan, 0 resets.

Autoform-Hard leaderboard

Rank Model Accuracy Cost / Task Duration / Test
1 GPT-6.2 Astra 87.50% $15.24 49m28s
2 Claude Fable 5.2 85.71% $23.21 1h07m
3 GPT-6.2 Sol 85.71% $8.09 1h13m
4 Gemini 4 Carbon 71.43% $12.38 1h55m
5 Gemini 4 Hydrogen 64.29% $23.25 2h07m
6 Gemini 4 Argon 48.21% $25.74 1h48m
7 Beam 1.0 0.00% - 54m03s
8 MiniMax-M3.1-Flash 0.00% $4.29 2h57m

Background

Autoform-Hard is the hardest slice of AutoformBench (Meta FAIR, 2026), a benchmark of 341 Lean 4 formalization tasks drawn from 24 graduate- and research-level textbooks in algebraic geometry, arithmetic geometry, number theory, Lie theory, algebraic topology, graph theory, complexity theory and more. Each task gives the agent a Lean project whose target theorems are stated but left as sorry, and asks it to prove them.

The statements were written and audited by PhD-level experts and chosen to go beyond what Mathlib already covers. Solving a task usually means building theory, not writing one proof: accepted solutions in the original study add a median of about 300 lines of Lean, and the hardest run to thousands.

Methodology

The board uses 56 tasks: every task in the benchmark’s 100-task validation set that none of the ten frontier models in the original study solved, except three that duplicate another task. Our audit corrected five of them: three statements were false as written, and two had packaging errors in the starting files. So a few tasks differ from the versions the original models attempted.

  • Agent. Every model runs in the same agent harness with a shell in a sandbox containing the Lean toolchain and a prebuilt Mathlib. Each task gets one attempt.
  • Limits. Each task has a 3-hour wall-clock limit, excluding provider retry time, and a $200 cost cap at the model’s list price. An agent may give up, but a report that the task is unsolved is rejected until the agent has spent $20 on it.1
  • Grading. A submission is accepted only if it passes clean-room grading: the graded theorems are rebuilt from a trusted copy of the task, their statements must match the originals exactly, the benchmark’s definitions must be unchanged, and the proofs may use only the standard axioms (propext, Quot.sound, Classical.choice). Accepted proofs are then replayed through an independent kernel. Proofs that use sorry, native_decide, new axioms or redefined benchmark objects fail.
  • Score. The percentage of the 56 tasks whose submission is accepted. A task that ends in an error counts as unsolved.

Footnotes

  1. This rule matters. GPT-6.2 Astra reported Smoktunowicz’s gap theorem unsolved after spending about $8; sent back to work, it went on to formalize it. See below to learn how! ↩