Public

astral-sh / uv

Benchmark updated: 8/15/202653 Tasks

An extremely fast Python package and project manager, written in Rust.

Languages

Rust98.2%Python1.5%Shell0.2%Other0.1%

		Harness			Input / Output Cost
1	Claude Fable 5	Mini-SWE-agent	35 / 53	$7.60	$10/$50	17m30s
2	GPT-5.6 Sol	Mini-SWE-agent	34 / 53	$2.26	$4/$20	9m07s
3	Kimi K3	Mini-SWE-agent	34 / 53	$8.50	$3/$15	15m26s
4	Claude Opus 4.8	Mini-SWE-agent	33 / 53	$4.61	$5/$25	17m50s
5	GPT-5.6 Terra	Mini-SWE-agent	32 / 53	$0.45	$2/$12	3m10s
6	GPT-5.6 Luna	Mini-SWE-agent	32 / 53	$0.11	$0.2/$1.2	4m53s
7	GPT 5.5	Mini-SWE-agent	32 / 53	$2.41	$5/$30	7m02s
8	Grok 4.5	Mini-SWE-agent	30 / 53	$0.94	$2/$6	7m02s
9	Claude Sonnet 5	Mini-SWE-agent	29 / 53	$4.60	$2/$10	19m55s
10	GLM 5.2	Mini-SWE-agent	28 / 53	$0.81	$1.4/$4.4	18m20s
11	Claude Opus 4.7	Mini-SWE-agent	27 / 53	$4.59	$5/$25	15m56s
12	Gemini 3.1 Pro Preview (02/26)	Mini-SWE-agent	25 / 53	$0.91	$2/$12	5m20s
13	Gemini 3.5 Flash	Mini-SWE-agent	24 / 53	$1.56	$1.5/$9	6m18s
14	Claude Haiku 4.5 (Nonthinking)	Mini-SWE-agent	17 / 53	$0.45	$1/$5	6m14s

Key Takeaways

Kimi K3 with Mini-SWE-agent and GPT-5.6 Sol with Mini-SWE-agent each resolve 34 of 53 tasks, one behind Claude Fable 5.
GPT-5.6 Luna with Mini-SWE-agent, GPT-5.6 Terra with Mini-SWE-agent, and GPT 5.5 with Mini-SWE-agent each resolve 32 of 53 tasks; GPT-5.6 Luna is lowest-cost at $0.11 per test.
GPT-5.6 Terra with Mini-SWE-agent is fastest among the 32-task results at 189.52 seconds, while GPT-5.6 Luna with Mini-SWE-agent costs $0.11 per test.

Cost Analysis

Cost / Test vs. Accuracy

ACCURACYCOST

Average Token Use / Test

Token Usage

InputOutputReasoningCache readCache write

Claude Sonnet 5

10.2M

Claude Opus 4.7

6.4M

Claude Opus 4.8

5.0M

Claude Fable 5

4.1M

Gemini 3.5 Flash

3.7M

GLM 5.2

3.5M

Claude Haiku 4.5 (Nonthinking)

2.8M

GPT-5.6 Luna

2.8M

Kimi K3

2.7M

GPT 5.5

2.5M

GPT-5.6 Sol

1.8M

Gemini 3.1 Pro Preview (02/26)

1.4M

Grok 4.5

1.3M

GPT-5.6 Terra

734K

Cost is the clearest tradeoff in this comparison. Claude Fable 5 leads at 66.04% for $7.60 per test. GPT-5.6 Sol is the lower-cost option at 64.15% for $2.26 per test.

Latency Analysis

Latency vs. Accuracy

ACCURACYLATENCY

Average Response Time / Test

Response Time

Claude Sonnet 5

19m 55s

GLM 5.2

18m 20s

Claude Opus 4.8

17m 50s

Claude Fable 5

17m 30s

Claude Opus 4.7

15m 56s

Kimi K3

15m 26s

GPT-5.6 Sol

9m 7s

GPT 5.5

7m 2s

Grok 4.5

7m 2s

Gemini 3.5 Flash

6m 18s

Claude Haiku 4.5 (Nonthinking)

6m 14s

Gemini 3.1 Pro Preview (02/26)

5m 20s

GPT-5.6 Luna

4m 53s

GPT-5.6 Terra

3m 10s

Latency separates several models with similarly strong scores. Claude Fable 5 leads at 66.04%, while GPT-5.6 Terra is fastest at 3m 10s with 60.38% accuracy.