Public
BerriAI / litellm
Benchmark updated: 8/13/202630 Tasks
The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]
Languages
Python84.5%TypeScript12.6%HTML1.5%Rust0.6%HCL0.2%Go0.2%Other0.4%
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 21 / 30 | $1.95 | $1.5/$7.5 | 12m55s | ||
2 | Mini-SWE-agent | 19 / 30 | $0.73 | $1.4/$4.4 | 11m29s |
Key Takeaways
- Gemini 3.6 Flash with Mini-SWE-agent scores 70%, versus 63.33% for GLM 5.2 (Fireworks) with Mini-SWE-agent.
- GLM 5.2 (Fireworks) with Mini-SWE-agent costs $0.73 per test and takes 689.22 seconds, compared with $1.95 and 775.49 seconds for Gemini 3.6 Flash with Mini-SWE-agent.
Cost Analysis
Cost / Test vs. Accuracy
ACCURACYCOST
Average Token Use / Test
Token Usage
InputOutputReasoningCache readCache write
No token usage data available.
Cost is the clearest tradeoff in this comparison. Gemini 3.6 Flash leads at 70.00% for $1.95 per test. GLM 5.2 is the lower-cost option at 63.33% for $0.73 per test.
Latency Analysis
Latency vs. Accuracy
ACCURACYLATENCY
Average Response Time / Test
Response Time
Gemini 3.6 Flash
GLM 5.2
Latency separates several models with similarly strong scores. Gemini 3.6 Flash leads at 70.00%, while GLM 5.2 is fastest at 11m 29s with 63.33% accuracy.