/ astral-sh / uv
Updated 7/19/2026
An extremely fast Python package and project manager, written in Rust.
Key Takeaways
- Claude Fable 5 leads with 35 of 53 tasks resolved, with GPT-5.6 Sol and Kimi K3 one task behind.
- GPT-5.6 Terra is the strongest efficiency option at 32 tasks, $0.56 per test, 190 seconds, and 0.73M tokens.
- GPT 5.5 and Kimi K3 resolve 38 of 53 tasks together even though neither model leads the aggregate leaderboard.
Cost / Test vs. Accuracy
GPT-5.6 Terra resolves 32 tasks for $0.56 per test; GPT-5.6 Sol adds two tasks for $2.26, and Claude Fable 5 adds one more for $7.60.
Latency vs. Accuracy
GPT-5.6 Terra resolves 32 tasks in 190 seconds, compared with 34 in 547 seconds for GPT-5.6 Sol and 35 in 1,050 seconds for Claude Fable 5.
Average token use / test
Tasks with failures
| Model | 1e96343 | 2937610 | 29eb5d2 | 2c76766 | 3723315 | 3cb8059 | 49ac431 | 4f7fd1c | 4fab9d3 | 5792a1d | 66142e9 | 6a83f82 | 6e928c0 | 7bbc01a | 7c9685c | 8688435 | 893df5d | 8dfd76c | 8e22984 | 950acf0 | a2af4cb | a5f99cd | a803e26 | af0a961 | b0298f9 | b5af85f | b615f83 | b68cf9b | bc286d3 | be9be78 | c6511b7 | d261212 | dadcccc | dee7a10 | e4ba1db | f2830ec | f385b7a | f6e266f | f9463b7 | f98013e | ff3d01d | ff3fbde |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Fable 5 | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Sol | ||||||||||||||||||||||||||||||||||||||||||
| Kimi K3 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.8 | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Terra | ||||||||||||||||||||||||||||||||||||||||||
| GPT-5.6 Luna | ||||||||||||||||||||||||||||||||||||||||||
| GPT 5.5 | ||||||||||||||||||||||||||||||||||||||||||
| Muse Spark 1.1 | ||||||||||||||||||||||||||||||||||||||||||
| Grok 4.5 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Sonnet 5 | ||||||||||||||||||||||||||||||||||||||||||
| GLM 5.2 | ||||||||||||||||||||||||||||||||||||||||||
| Claude Opus 4.7 | ||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | ||||||||||||||||||||||||||||||||||||||||||
| Gemini 3.5 Flash | ||||||||||||||||||||||||||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
1e96343a7cccIssue statement
When a URL that contains embedded credentials (for example https://user:password@example.com/simple/) is stored in a VerbatimUrl, the struct keeps the original, user-provided string in its given field so that the verbatim form can be reproduced later.
The parsed URL held by VerbatimUrl already redacts credentials when it is formatted (its username/password are shown as ****). However, the Debug output of VerbatimUrl prints the given field as-is, so any credentials that were part of the original string are exposed in plain text.
This is a problem because VerbatimUrl values end up in debug/diagnostic output in several places (for example when settings containing index or publish URLs are dumped, or via verbose logging). A URL such as https://user:password@example.com/simple/ is rendered with the parsed url field correctly masked, but the given field still shows the literal password.
Formatting a VerbatimUrl with {:?} must not reveal credentials that were embedded in the original URL. The credentials in the given field should be masked the same way they are for the parsed URL, while URLs without credentials should continue to be shown unchanged.
Hidden tests
diff --git a/crates/uv-pep508/src/verbatim_url.rs b/crates/uv-pep508/src/verbatim_url.rsindex 730ef8a..0bfc9f1 100644--- a/crates/uv-pep508/src/verbatim_url.rs+++ b/crates/uv-pep508/src/verbatim_url.rs@@ -948,4 +948,17 @@ mod tests { }, ); }++ #[test]+ fn debug_redacts_given_credentials() {+ // The `given` field preserves the URL exactly as the user provided it, including any+ // embedded credentials. The `Debug` output must not leak those credentials.+ let raw = "https://user:s3cr3t-token@example.com/simple/";+ let url = VerbatimUrl::parse_url(raw).unwrap().with_given(raw);+ let debug = format!("{url:?}");+ assert!(+ !debug.contains("s3cr3t-token"),+ "credentials leaked in Debug output: {debug}"+ );+ } }
Head-to-head
Accuracy
66.04%Δ 1.89%
64.15%
Cost / test
$7.60
$2.26Δ $5.34
Latency
1050s
547sΔ 503s
Cost distribution
Latency distribution
Input token distribution
Output token distribution
Task outcomes
53 tasks