Public
torvalds / linux
Benchmark updated: 8/15/202685 Tasks
Linux kernel source tree
Languages
C98%Assembly0.7%Rust0.4%Shell0.4%Python0.3%Makefile0.2%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 63 / 85 | $1.09 | $4/$20 | 5m09s | ||
2 | Mini-SWE-agent | 58 / 85 | $1.43 | $5/$25 | 7m08s | ||
3 | Mini-SWE-agent | 58 / 85 | $2.79 | $10/$50 | 8m02s | ||
4 | Mini-SWE-agent | 56 / 85 | $0.19 | $2/$12 | 101.76s | ||
5 | Mini-SWE-agent | 56 / 85 | $1.43 | $5/$25 | 5m16s | ||
6 | Mini-SWE-agent | 55 / 85 | $0.17 | $2/$6 | 108.81s | ||
7 | Mini-SWE-agent | 54 / 85 | $0.78 | $5/$30 | 3m36s | ||
8 | Mini-SWE-agent | 54 / 85 | $2.12 | $2/$10 | 11m23s | ||
9 | Mini-SWE-agent | 53 / 85 | $0.04 | $0.2/$1.2 | 2m49s | ||
10 | Mini-SWE-agent | 53 / 85 | $3.65 | $3/$15 | 8m38s | ||
11 | Mini-SWE-agent | 53 / 85 | $0.34 | $1.25/$4.25 | 4m52s | ||
12 | Mini-SWE-agent | 48 / 85 | $0.18 | $1/$5 | 2m27s | ||
13 | Mini-SWE-agent | 48 / 85 | $0.64 | $1.5/$9 | 2m43s | ||
14 | Mini-SWE-agent | 48 / 85 | $0.52 | $2/$12 | 2m46s | ||
15 | Mini-SWE-agent | 47 / 85 | $0.32 | $1.4/$4.4 | 8m44s |
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
Kimi K3
GLM 5.2
Gemini 3.5 Flash
Claude Opus 4.8
Claude Haiku 4.5 (Nonthinking)
Claude Fable 5
GPT-5.6 Sol
GPT-5.6 Luna
Gemini 3.1 Pro Preview (02/26)
GPT 5.5
Grok 4.5
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 74.12% for $1.09 per test. GPT-5.6 Luna is the lower-cost option at 62.35% for $0.04 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
GLM 5.2
Kimi K3
Claude Fable 5
Claude Opus 4.8
Claude Opus 4.7
GPT-5.6 Sol
Muse Spark 1.2
GPT 5.5
GPT-5.6 Luna
Gemini 3.1 Pro Preview (02/26)
Gemini 3.5 Flash
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
GPT-5.6 Terra
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 74.12%, while GPT-5.6 Terra is fastest at 1m 42s with 65.88% accuracy.