Public
vals-ai / valkyrie
Benchmark updated: 8/15/202646 Tasks
Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark.
Languages
Python99.4%Makefile0.5%Mako0.1%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 41 / 46 | $1.44 | $4/$20 | 9m14s | ||
2 | Mini-SWE-agent | 40 / 46 | $0.37 | $2/$12 | 3m35s | ||
3 | Mini-SWE-agent | 40 / 46 | $0.07 | $0.2/$1.2 | 6m23s | ||
4 | Mini-SWE-agent | 40 / 46 | $4.48 | $10/$50 | 10m26s | ||
5 | Mini-SWE-agent | 40 / 46 | $3.38 | $5/$25 | 12m13s | ||
6 | Mini-SWE-agent | 39 / 46 | $0.53 | $1.4/$4.4 | 9m27s | ||
7 | Mini-SWE-agent | 39 / 46 | $4.73 | $3/$15 | 11m51s | ||
8 | Mini-SWE-agent | 39 / 46 | $3.43 | $2/$10 | 15m30s | ||
9 | Mini-SWE-agent | 38 / 46 | $0.47 | $2/$6 | 3m48s | ||
10 | Mini-SWE-agent | 38 / 46 | $1.74 | $5/$30 | 8m31s | ||
11 | Mini-SWE-agent | 38 / 46 | $3.10 | $5/$25 | 9m53s | ||
12 | Mini-SWE-agent | 36 / 46 | $1.19 | $1.5/$9 | 6m15s | ||
13 | Mini-SWE-agent | 34 / 46 | $0.62 | $2/$12 | 3m17s | ||
14 | Mini-SWE-agent | 31 / 46 | $0.37 | $1/$5 | 4m41s |
Key Takeaways
- GPT-5.6 Luna with Mini-SWE-agent matches the 40-task tier at $0.07 per test and 382.95 seconds, the lowest cost in that tier.
- GPT-5.6 Terra with Mini-SWE-agent also resolves 40 tasks, with 214.54 seconds and $0.37 per test, faster than the other 40-task results.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
Claude Opus 4.8
Gemini 3.5 Flash
GLM 5.2
Claude Haiku 4.5 (Nonthinking)
Claude Fable 5
GPT 5.5
Kimi K3
GPT-5.6 Luna
GPT-5.6 Sol
Gemini 3.1 Pro Preview (02/26)
Grok 4.5
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 89.13% for $1.44 per test. GPT-5.6 Luna is the lower-cost option at 86.96% for $0.07 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
Claude Opus 4.8
Kimi K3
Claude Fable 5
Claude Opus 4.7
GLM 5.2
GPT-5.6 Sol
GPT 5.5
GPT-5.6 Luna
Gemini 3.5 Flash
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
GPT-5.6 Terra
Gemini 3.1 Pro Preview (02/26)
Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 89.13%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 17s with 73.91% accuracy.