Public
apache / spark
Benchmark updated: 8/15/202658 Tasks
Apache Spark - A unified analytics engine for large-scale data processing
Languages
Scala68.5%Python16.2%Java6.7%Jupyter Notebook4.3%Other2.4%HiveQL1.9%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 49 / 58 | $2.46 | $4/$20 | 9m48s | ||
2 | Mini-SWE-agent | 48 / 58 | $7.44 | $10/$50 | 16m48s | ||
3 | Mini-SWE-agent | 48 / 58 | $4.55 | $5/$25 | 17m18s | ||
4 | Mini-SWE-agent | 47 / 58 | $0.71 | $2/$6 | 3m47s | ||
5 | Mini-SWE-agent | 47 / 58 | $0.88 | $1.4/$4.4 | 15m39s | ||
6 | Mini-SWE-agent | 47 / 58 | $9.82 | $3/$15 | 16m32s | ||
7 | Mini-SWE-agent | 45 / 58 | $4.96 | $2/$10 | 21m11s | ||
8 | Mini-SWE-agent | 44 / 58 | $4.92 | $5/$25 | 13m26s | ||
9 | Mini-SWE-agent | 41 / 58 | $0.12 | $0.2/$1.2 | 5m21s | ||
10 | Mini-SWE-agent | 38 / 58 | $0.48 | $2/$12 | 3m30s | ||
11 | Mini-SWE-agent | 38 / 58 | $1.26 | $1.5/$9 | 5m14s | ||
12 | Mini-SWE-agent | 38 / 58 | $2.02 | $5/$30 | 6m19s | ||
13 | Mini-SWE-agent | 33 / 58 | $0.70 | $2/$12 | 3m23s | ||
14 | Mini-SWE-agent | 33 / 58 | $0.41 | $1/$5 | 4m36s |
Key Takeaways
- Claude Fable 5 and Claude Opus 4.8 with Mini-SWE-agent each resolve 48 of 58 tasks (82.76%), while GLM 5.2 (Fireworks), Kimi K3, and Grok 4.5 each resolve 47.
- Grok 4.5 with Mini-SWE-agent resolves 47 of 58 tasks for $0.71 per test in 226.98 seconds, compared with GPT-5.6 Sol at $2.46 and 588.17 seconds.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
Claude Opus 4.8
GLM 5.2
Claude Fable 5
Kimi K3
Gemini 3.5 Flash
GPT-5.6 Luna
Claude Haiku 4.5 (Nonthinking)
GPT-5.6 Sol
GPT 5.5
Grok 4.5
Gemini 3.1 Pro Preview (02/26)
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 84.48% for $2.46 per test. Grok 4.5 is the lower-cost option at 81.03% for $0.71 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
Claude Opus 4.8
Claude Fable 5
Kimi K3
GLM 5.2
Claude Opus 4.7
GPT-5.6 Sol
GPT 5.5
GPT-5.6 Luna
Gemini 3.5 Flash
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
GPT-5.6 Terra
Gemini 3.1 Pro Preview (02/26)
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 84.48%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 23s with 56.90% accuracy.