Public
skypilot-org / skypilot
Benchmark updated: 8/13/202630 Tasks
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
Languages
Python89.4%JavaScript7.6%Jinja1.2%Shell1.1%HTML0.4%Go0.1%Other0.2%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 9 / 30 | $0.76 | $1.4/$4.4 | 10m01s | ||
2 | Mini-SWE-agent | 9 / 30 | $4.20 | $5/$25 | 13m17s | ||
3 | Mini-SWE-agent | 9 / 30 | $4.20 | $5/$25 | 13m48s | ||
4 | Mini-SWE-agent | 8 / 30 | $0.07 | $0.44/$1.32 | 5m39s |
Key Takeaways
- The 30-task results are directional: Claude Opus 4.8, Claude Opus 5, and GLM 5.2 (Fireworks) tie at 30%, ahead of DeepSeek V4 Flash at 26.67%.
- GLM 5.2 (Fireworks) with Mini-SWE-agent has the lowest cost per test at $0.76 and shortest latency at 601.19 seconds among the tied leaders.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Opus 5
Claude Opus 4.8
DeepSeek V4 Flash 0731
Cost is the clearest tradeoff in this comparison. Claude Opus 4.8 leads at 30.00% for $4.20 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 26.67% for $0.07 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Opus 5
Claude Opus 4.8
GLM 5.2
DeepSeek V4 Flash 0731
Latency separates several models with similarly strong scores. Claude Opus 4.8 leads at 30.00%, while DeepSeek V4 Flash 0731 is fastest at 5m 39s with 26.67% accuracy.