Public
mirumee / ariadne
Benchmark updated: 8/13/202630 Tasks
Python library for implementing GraphQL servers using schema-first approach. Created with :heart: by Mirumee
Languages
Python99%HTML0.8%Just0.2%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 25 / 30 | $0.37 | $1.4/$4.4 | 7m02s | ||
2 | Mini-SWE-agent | 23 / 30 | $0.05 | $0.44/$1.32 | 6m26s |
Key Takeaways
- GLM 5.2 (Fireworks) with Mini-SWE-agent scores 83.33% versus 76.67% for DeepSeek V4 Flash with Mini-SWE-agent.
- DeepSeek V4 Flash with Mini-SWE-agent costs $0.05 per test and runs in 385.59 seconds, compared with $0.37 and 421.95 seconds for GLM 5.2 (Fireworks) with Mini-SWE-agent.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
GLM 5.2
DeepSeek V4 Flash 0731
Cost is the clearest tradeoff in this comparison. GLM 5.2 leads at 83.33% for $0.37 per test. DeepSeek V4 Flash 0731 is the lower-cost option at 76.67% for $0.05 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
GLM 5.2
DeepSeek V4 Flash 0731
Latency separates several models with similarly strong scores. GLM 5.2 leads at 83.33%, while DeepSeek V4 Flash 0731 is fastest at 6m 26s with 76.67% accuracy.