Public
ladydascalie / core
Benchmark updated: 8/18/202630 Tasks
A collection of packages for building a Go microservice on the LUSH platform
Languages
Go100%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 23 / 30 | $0.01 | $0.44/$1.32 | 6m13s | ||
2 | Mini-SWE-agent | 21 / 30 | $0.09 | $1.4/$4.4 | 2m07s | ||
3 | Mini-SWE-agent | 20 / 30 | $0.04 | $0.4/$1.6 | 83.26s | ||
4 | Mini-SWE-agent | 20 / 30 | $1.07 | $1.5/$7.5 | 3m50s |
Key Takeaways
- GLM 5.2 (Fireworks) with Mini-SWE-agent resolves 21 of 30 tasks at 70%, while Qwen3.7 Plus and Gemini 3.6 Flash each resolve 20.
- Deepseek V4 Flash 0731 with Mini-SWE-agent has the lowest cost per test at $0.01; Qwen3.7 Plus with Mini-SWE-agent is fastest at 83.26 seconds.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Gemini 3.6 Flash
Qwen 3.7 Plus
GLM 5.2
DeepSeek V4 Flash 0731
Cost is the clearest tradeoff in this comparison. DeepSeek V4 Flash 0731 leads at 76.67% for $0.01 per test. No other model in this comparison is cheaper.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
DeepSeek V4 Flash 0731
Gemini 3.6 Flash
GLM 5.2
Qwen 3.7 Plus
Latency separates several models with similarly strong scores. DeepSeek V4 Flash 0731 leads at 76.67%, while Qwen 3.7 Plus is fastest at 1m 23s with 66.67% accuracy.