leanprover-community/mathlib4
Languages
The math library of Lean 4
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 23 / 30 | $1.81 | $5/$25 | 25m16s | ||
2 | Mini-SWE-agent | 21 / 30 | $0.34 | $1.25/$4.25 | 4m26s | ||
3 | Mini-SWE-agent | 20 / 30 | $0.93 | $1.5/$7.5 | 4m05s | ||
4 | Mini-SWE-agent | 20 / 30 | $0.45 | $2/$12 | 4m32s |
Key Takeaways
- Claude Opus 5 with Mini-SWE-agent scores 76.67% at $1.81 per test, compared with Muse Spark 1.2 at 70% and $0.34.
- Gemini 3.6 Flash and GPT-5.6 Terra each score 66.67%; GPT-5.6 Terra costs $0.45 per test versus $0.93 for Gemini 3.6 Flash.
Model Comparison
Accuracy
76.67%
Claude Opus 5
70.00%
Muse Spark 1.2
Task outcomes
30 tasks
Cost / test
$1.81
Claude Opus 5
$0.34
Muse Spark 1.2
Cost distribution
Latency
25m 16s
Claude Opus 5
4m 26s
Muse Spark 1.2
Latency distribution
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Claude Opus 5 leads at 76.67% for $1.81 per test. Muse Spark 1.2 is the lower-cost option at 70.00% for $0.34 per test.
Latency Analysis
Average Response Time / Test
Latency separates several models with similarly strong scores. Claude Opus 5 leads at 76.67%, while Gemini 3.6 Flash is fastest at 4m 5s with 66.67% accuracy.
Tasks with failures
| Models | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | |||||||||||
| Muse Spark 1.2 | |||||||||||
| GPT-5.6 Terra | |||||||||||
| Gemini 3.6 Flash |
Task detail
2d83dfaIssue statement
Enhance scripts/bench/repeatedly.py so callers can choose the measurements output file and optionally discard a configurable number of the highest and lowest per-metric iteration results before averaging. Measurements with the same metric emitted more than once by a single run must be summed before that iteration participates in trimming and averaging. Preserve measurement units and forward all trailing arguments to the command being run. Reject configurations where the total number of discarded iterations is not smaller than the iteration count.
View Hidden Tests
diff --git a/scripts/bench/test_repeatedly_valsmith.py b/scripts/bench/test_repeatedly_valsmith.pynew file mode 100644index 0000000000..20ed35a5f1--- /dev/null+++ b/scripts/bench/test_repeatedly_valsmith.py@@ -0,0 +1,49 @@+import json+import subprocess+import sys+from pathlib import Path+++SCRIPT = Path(__file__).with_name("repeatedly.py")+++def test_cli_supports_custom_outfile_and_trimmed_metric_averages(tmp_path):+ outfile = tmp_path / "custom-measurements.jsonl"+ state = tmp_path / "iteration.txt"+ producer = tmp_path / "produce.py"+ producer.write_text(+ """import json, pathlib, sys+outfile, state = map(pathlib.Path, sys.argv[1:])+iteration = int(state.read_text()) if state.exists() else 0+state.write_text(str(iteration + 1))+values = [1, 10, 3, 5]+with outfile.open('a') as stream:+ stream.write(json.dumps({'metric': 'time', 'value': values[iteration], 'unit': 'ms'}) + '\\n')+ stream.write(json.dumps({'metric': 'time', 'value': 1, 'unit': 'ms'}) + '\\n')+ stream.write(json.dumps({'metric': 'memory', 'value': 100 + iteration}) + '\\n')+"""+ )++ subprocess.run(+ [+ sys.executable,+ str(SCRIPT),+ "-n", "4",+ "--drop-highest", "1",+ "--drop-lowest", "1",+ "--outfile", str(outfile),+ sys.executable,+ str(producer), str(outfile), str(state),+ ],+ check=True,+ cwd=tmp_path,+ )++ measurements = {+ item["metric"]: item+ for item in map(json.loads, outfile.read_text().splitlines())+ }+ assert measurements == {+ "time": {"metric": "time", "value": 5.0, "unit": "ms"},+ "memory": {"metric": "memory", "value": 101.5},+ }