Public
sgl-project / sglang
Benchmark updated: 8/15/202659 Tasks
SGLang is a high-performance serving framework for large language models and multimodal models.
Languages
Python76.9%MDX6.8%Rust6.1%Cuda4.3%C++2.7%JavaScript1.9%Other1.3%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 46 / 59 | $0.49 | $2/$12 | 2m42s | ||
2 | Mini-SWE-agent | 45 / 59 | $0.12 | $0.2/$1.2 | 4m27s | ||
3 | Mini-SWE-agent | 45 / 59 | $2.28 | $5/$30 | 5m39s | ||
4 | Mini-SWE-agent | 45 / 59 | $0.77 | $1.4/$4.4 | 13m46s | ||
5 | Mini-SWE-agent | 45 / 59 | $7.11 | $10/$50 | 14m58s | ||
6 | Mini-SWE-agent | 44 / 59 | $2.41 | $4/$20 | 8m06s | ||
7 | Mini-SWE-agent | 43 / 59 | $0.66 | $2/$12 | 2m40s | ||
8 | Mini-SWE-agent | 43 / 59 | $0.74 | $2/$6 | 3m46s | ||
9 | Mini-SWE-agent | 43 / 59 | $8.84 | $3/$15 | 13m12s | ||
10 | Mini-SWE-agent | 43 / 59 | $4.25 | $5/$25 | 14m15s | ||
11 | Mini-SWE-agent | 42 / 59 | $1.50 | $1.5/$9 | 7m39s | ||
12 | Mini-SWE-agent | 42 / 59 | $4.69 | $2/$10 | 18m05s | ||
13 | Mini-SWE-agent | 41 / 59 | $4.06 | $5/$25 | 10m59s | ||
14 | Mini-SWE-agent | 33 / 59 | $0.37 | $1/$5 | 3m50s |
Key Takeaways
- Claude Fable 5, GLM 5.2 (Fireworks), GPT 5.5, and GPT-5.6 Luna with Mini-SWE-agent each resolve 45 tasks, one behind GPT-5.6 Terra.
- GPT-5.6 Luna with Mini-SWE-agent has the lowest cost per test among the 45-task results at $0.12; Gemini 3.1 Pro Preview (02/26) is fastest among the 43-task results at 159.84 seconds.
- GPT-5.6 Terra with Mini-SWE-agent is also faster than every other result resolving at least 44 tasks, at 162.20 seconds.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
Claude Opus 4.7
Claude Opus 4.8
Claude Fable 5
Gemini 3.5 Flash
GLM 5.2
Kimi K3
GPT-5.6 Luna
GPT 5.5
Claude Haiku 4.5 (Nonthinking)
GPT-5.6 Sol
Grok 4.5
Gemini 3.1 Pro Preview (02/26)
GPT-5.6 Terra
Cost is the clearest tradeoff in this comparison. GPT-5.6 Terra leads at 77.97% for $0.49 per test. GPT-5.6 Luna is the lower-cost option at 76.27% for $0.12 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Claude Sonnet 5
Claude Fable 5
Claude Opus 4.8
GLM 5.2
Kimi K3
Claude Opus 4.7
GPT-5.6 Sol
Gemini 3.5 Flash
GPT 5.5
GPT-5.6 Luna
Claude Haiku 4.5 (Nonthinking)
Grok 4.5
GPT-5.6 Terra
Gemini 3.1 Pro Preview (02/26)
Latency separates several models with similarly strong scores. GPT-5.6 Terra leads at 77.97%, while Gemini 3.1 Pro Preview (02/26) is fastest at 2m 40s with 72.88% accuracy.