/ vals-ai / Valkyrie
Updated 7/19/2026
Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark.
Key Takeaways
- GPT-5.6 Sol leads with 41 of 46 tasks resolved.
- GPT-5.6 Luna, GPT-5.6 Terra, Claude Opus 4.8, and Claude Fable 5 tie one task behind; Luna is cheapest at $0.35, while Terra is fastest at 215 seconds.
- Claude Opus 4.7 and GPT-5.6 Luna resolve 42 of 46 tasks together, exceeding the single-model leader by one task.
Cost / Test vs. Accuracy
GPT-5.6 Luna is one task behind GPT-5.6 Sol while reducing cost per test from $1.44 to $0.35.
Latency vs. Accuracy
GPT-5.6 Terra resolves 40 tasks in 215 seconds, one task behind GPT-5.6 Sol at 554 seconds.
Average token use / test
Tasks with failures
| Model | 11e61ff | 125b61d | 144dc5c | 1884e7c | 223f8e5 | 30d0a05 | 3c850af | 3fa0180 | 45cf4ca | 4dc1f4a | 508afc9 | 7889dd1 | 7ad2042 | 9e10a15 | ab9ca74 | c200f81 | e0c3a0c | efbd9f5 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6 Sol | ||||||||||||||||||
| GPT-5.6 Luna | ||||||||||||||||||
| GPT-5.6 Terra | ||||||||||||||||||
| Claude Opus 4.8 | ||||||||||||||||||
| Claude Fable 5 | ||||||||||||||||||
| GLM 5.2 | ||||||||||||||||||
| Muse Spark 1.1 | ||||||||||||||||||
| Claude Sonnet 5 | ||||||||||||||||||
| Kimi K3 | ||||||||||||||||||
| Grok 4.5 | ||||||||||||||||||
| GPT 5.5 | ||||||||||||||||||
| Claude Opus 4.7 | ||||||||||||||||||
| Gemini 3.5 Flash | ||||||||||||||||||
| Gemini 3.1 Pro Preview (02/26) | ||||||||||||||||||
| Claude Haiku 4.5 (Nonthinking) |
Task detail
11e61ff95d59Issue statement
The tracker service exposes a POST /retry-or-resume-benchmark/{benchmark_id} endpoint that re-enqueues a previously created benchmark run using the arguments it was originally created with, including the agent contract stored on the benchmark row. That stored contract carries a secrets mapping (environment variable name -> secret name) that is passed through to the tasks when they run.
Today there is no way to change those secrets when resuming or retrying a run. If a secret in the stored contract is wrong, stale, or missing, the only option is to recreate the benchmark from scratch. Users need to be able to override secrets at resume/retry time so the resumed tasks pick up the corrected values.
Please extend the resume/retry endpoint so the request body accepts an optional secrets mapping (defaulting to an empty mapping) of environment-variable name to secret name. When a non-empty secrets mapping is provided, it must be merged into the agent contract's existing secrets before the run is re-enqueued:
- Keys already present in the stored contract are replaced by the values from the request.
- Keys not present in the stored contract are added.
- Keys in the stored contract that are not mentioned in the request are left untouched.
The merge must be persisted to the stored benchmark row (so a later refresh from the database reflects the merged secrets) and must also be reflected in the contract carried by the request payload that is used to re-enqueue the resumed tasks. When no secrets mapping is supplied, behavior must be unchanged. Supplying secrets must not otherwise change existing resume/retry behavior (status codes, task selection, concurrency handling, header forwarding, etc.).
Hidden tests
diff --git a/services/tracker/tests/unit/test_stop_and_resume.py b/services/tracker/tests/unit/test_stop_and_resume.pyindex c59a0bbf..7fd88b2c 100644--- a/services/tracker/tests/unit/test_stop_and_resume.py+++ b/services/tracker/tests/unit/test_stop_and_resume.py@@ -518,6 +518,63 @@ class TestStopAndResume: assert observed_headers["X-Descope-Api-Key"] == "tracker-api-key" assert captured_request_json["service_headers"]["X-Descope-Api-Key"] == "tracker-api-key" + async def test_retry_or_resume_applies_secrets_to_stored_contract(+ self,+ example_benchmark_object: Benchmark,+ database_session: Session,+ monkeypatch: MonkeyPatch,+ ):+ """Resume secrets should update the contract used by resumed tasks.++ Test cases:+ - Existing env var mappings are replaced by resume overrides.+ - New env var mappings are added to the stored contract before enqueue.+ """+ benchmark_row = example_benchmark_object+ benchmark_row.status = BenchmarkStatus.STOPPED+ benchmark_row.arguments.contract.secrets = {+ "ANTHROPIC_API_KEY": "old-secret",+ "OPENAI_API_KEY": "openai-secret",+ }+ database_session.add(benchmark_row)+ database_session.commit()++ captured_request_json: dict[str, Any] = {}++ async def _mock_reset_to_in_progress_status(*_args: Any, **_kwargs: Any):+ return ["task_0"]++ class _MockKicker:+ def with_labels(self, **_kwargs: Any) -> "_MockKicker":+ return self++ async def kiq(self, **kwargs: Any) -> None:+ captured_request_json.update(kwargs["start_benchmark_request_json"])++ monkeypatch.setattr("main.reset_to_in_progress_status", _mock_reset_to_in_progress_status)+ monkeypatch.setattr("main.process_benchmark.kicker", lambda: _MockKicker())++ response = client.post(+ f"/retry-or-resume-benchmark/{benchmark_row.id}",+ json={+ "task_ids": [],+ "service_headers": {},+ "secrets": {+ "ANTHROPIC_API_KEY": "new-secret",+ "GEMINI_API_KEY": "gemini-secret",+ },+ },+ )++ assert response.status_code == 200+ assert captured_request_json["contract"]["secrets"] == {+ "ANTHROPIC_API_KEY": "new-secret",+ "OPENAI_API_KEY": "openai-secret",+ "GEMINI_API_KEY": "gemini-secret",+ }+ database_session.refresh(benchmark_row)+ assert benchmark_row.arguments.contract.secrets == captured_request_json["contract"]["secrets"]+ async def test_running_retry_noops_without_error_tasks( self, example_benchmark_object: Benchmark,
Head-to-head
Accuracy
89.13%Δ 2.17%
86.96%
Cost / test
$1.44
$0.35Δ $1.08
Latency
554s
383sΔ 171s
Cost distribution
Latency distribution
Input token distribution
Output token distribution
Task outcomes
46 tasks