Public
jax-ml / jax
Benchmark updated: 8/15/202654 Tasks
Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
Languages
Python87.1%C++9.9%Starlark2%Shell0.5%Jupyter Notebook0.4%C<0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 43 / 54 | $2.27 | $4/$20 | 10m11s | ||
2 | Mini-SWE-agent | 42 / 54 | $1.05 | $2/$6 | 6m35s | ||
3 | Mini-SWE-agent | 42 / 54 | $2.55 | $5/$30 | 7m49s | ||
4 | Mini-SWE-agent | 42 / 54 | $10.42 | $3/$15 | 18m24s | ||
5 | Mini-SWE-agent | 42 / 54 | $7.26 | $10/$50 | 18m41s | ||
6 | Mini-SWE-agent | 42 / 54 | $4.31 | $2/$10 | 24m52s | ||
7 | Mini-SWE-agent | 41 / 54 | $0.49 | $2/$12 | 3m07s | ||
8 | Mini-SWE-agent | 40 / 54 | $0.15 | $0.2/$1.2 | 6m22s | ||
9 | Mini-SWE-agent | 40 / 54 | $1.56 | $1.5/$9 | 6m34s | ||
10 | Mini-SWE-agent | 40 / 54 | $4.61 | $5/$25 | 15m28s | ||
11 | Mini-SWE-agent | 40 / 54 | $3.89 | $5/$25 | 16m08s | ||
12 | Mini-SWE-agent | 38 / 54 | $1.04 | $1.4/$4.4 | 18m50s | ||
13 | Mini-SWE-agent | 37 / 54 | $1.10 | $2/$12 | 5m21s | ||
14 | Mini-SWE-agent | 33 / 54 | $0.45 | $1/$5 | 5m02s |
Key Takeaways
- Claude Fable 5, Claude Sonnet 5, Kimi K3, Grok 4.5, and GPT 5.5 with Mini-SWE-agent each resolve 42 of 54 tasks, one fewer than GPT-5.6 Sol.
- GPT-5.6 Terra with Mini-SWE-agent resolves 41 of 54 tasks, while GPT-5.6 Luna and Gemini 3.5 Flash each resolve 40 of 54.
- All supplied results cover 54 tasks, so these are not smoke tests or directional results.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
GLM 5.2
Claude Opus 4.8
Claude Fable 5
Gemini 3.5 Flash
GPT-5.6 Luna
Kimi K3
GPT 5.5
Claude Haiku 4.5 (Nonthinking)
GPT-5.6 Sol
Gemini 3.1 Pro Preview (02/26)
Grok 4.5
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 79.63% for $2.27 per test. GPT-5.6 Terra is the lower-cost option at 75.93% for $0.49 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
GLM 5.2
Claude Fable 5
Kimi K3
Claude Opus 4.8
Claude Opus 4.7
GPT-5.6 Sol
GPT 5.5
Grok 4.5
Gemini 3.5 Flash
GPT-5.6 Luna
Gemini 3.1 Pro Preview (02/26)
Claude Haiku 4.5 (Nonthinking)
GPT-5.6 Terra
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 79.63%, while GPT-5.6 Terra is fastest at 3m 7s with 75.93% accuracy.