Public
leanprover-community / mathlib4
Benchmark updated: 8/18/202630 Tasks
The math library of Lean 4
Languages
Lean99.6%Python0.3%Shell<0.1%TeX<0.1%Dockerfile<0.1%HTML<0.1%Other<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 23 / 30 | $1.81 | $5/$25 | 25m16s | ||
2 | Mini-SWE-agent | 21 / 30 | $0.34 | $1.25/$4.25 | 4m26s | ||
3 | Mini-SWE-agent | 20 / 30 | $0.93 | $1.5/$7.5 | 4m05s | ||
4 | Mini-SWE-agent | 20 / 30 | $0.45 | $2/$12 | 4m32s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 76.67% at $1.81 per test, compared with Muse Spark 1.2 at 70% and $0.34.
- Gemini 3.6 Flash and GPT-5.6 Terra each score 66.67%; GPT-5.6 Terra costs $0.45 per test versus $0.93 for Gemini 3.6 Flash.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Opus 5
Muse Spark 1.2
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 76.67% for $1.81 per test. Muse Spark 1.2 is the lower-cost option at 70.00% for $0.34 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
GPT-5.6 Terra
Muse Spark 1.2
Gemini 3.6 Flash
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 76.67%, while Gemini 3.6 Flash is fastest at 4m 5s with 66.67% accuracy.