Public

vals-ai/valkyrie

Updated: 8/15/2026

Languages

Python99.4%Makefile0.5%Mako0.1%
14 Models46 Tasks

Scalable, cloud-native infrastructure for evaluating AI agents across any benchmark.

Harness

1

Mini-SWE-agent
41 / 46

$1.44

9m14s

2

Mini-SWE-agent
40 / 46

$0.37

3m35s

3

Mini-SWE-agent
40 / 46

$0.07

6m23s

4

Mini-SWE-agent
40 / 46

$4.48

10m26s

5

Mini-SWE-agent
40 / 46

$3.38

12m13s

6

Mini-SWE-agent
39 / 46

$0.53

9m27s

7

Mini-SWE-agent
39 / 46

$4.73

11m51s

8

Mini-SWE-agent
39 / 46

$3.43

15m30s

9

Mini-SWE-agent
38 / 46

$0.47

3m48s

10

Mini-SWE-agent
38 / 46

$1.74

8m31s

11

Mini-SWE-agent
38 / 46

$3.10

9m53s

12

Mini-SWE-agent
36 / 46

$1.19

6m15s

13

Mini-SWE-agent
34 / 46

$0.62

3m17s

14

Mini-SWE-agent
31 / 46

$0.37

4m41s

Key Takeaways

  • GPT-5.6 Luna with Mini-SWE-agent matches the 40-task tier at $0.07 per test and 382.95 seconds, the lowest cost in that tier.
  • GPT-5.6 Terra with Mini-SWE-agent also resolves 40 tasks, with 214.54 seconds and $0.37 per test, faster than the other 40-task results.

Model Comparison

Accuracy

89.13%

GPT-5.6 Sol

86.96%

GPT-5.6 Luna

Task outcomes

46 tasks

Both
GPT-5.6 Sol only
GPT-5.6 Luna only
Neither
Not attempted

Cost / test

$1.44

GPT-5.6 Sol

$0.07

GPT-5.6 Luna

Cost distribution

$0.00$2.78$5.56

Latency

9m 14s

GPT-5.6 Sol

6m 23s

GPT-5.6 Luna

Latency distribution

0s34m 47s69m 33s

Cost Analysis

Cost / Test vs. Accuracy
ACCURACYCOST

Average Token Use / Test

Token Usage
InputOutputReasoningCache readCache write
Claude Sonnet 5
7.5M
Claude Opus 4.7
3.8M
Claude Opus 4.8
3.4M
Gemini 3.5 Flash
2.5M
GLM 5.2
2.2M
Claude Haiku 4.5 (Nonthinking)
2.1M
Claude Fable 5
2.0M
GPT 5.5
1.6M
Kimi K3
1.5M
GPT-5.6 Luna
1.3M
GPT-5.6 Sol
948K
Gemini 3.1 Pro Preview (02/26)
796K
Grok 4.5
605K
GPT-5.6 Terra
508K

Cost is the clearest tradeoff in this comparison. GPT-5.6 Sol leads at 89.13% for $1.44 per test. GPT-5.6 Luna is the lower-cost option at 86.96% for $0.07 per test.

Latency Analysis

Latency vs. Accuracy
ACCURACYLATENCY

Average Response Time / Test

Response Time
Claude Sonnet 5
15m 30s
Claude Opus 4.8
12m 13s
Kimi K3
11m 51s
Claude Fable 5
10m 26s
Claude Opus 4.7
9m 53s
GLM 5.2
9m 27s
GPT-5.6 Sol
9m 14s
GPT 5.5
8m 31s
GPT-5.6 Luna
6m 23s
Gemini 3.5 Flash
6m 15s
Claude Haiku 4.5 (Nonthinking)
4m 41s
Grok 4.5
3m 48s
GPT-5.6 Terra
3m 35s
Gemini 3.1 Pro Preview (02/26)
3m 17s

Latency separates several models with similarly strong scores. GPT-5.6 Sol leads at 89.13%, while Gemini 3.1 Pro Preview (02/26) is fastest at 3m 17s with 73.91% accuracy.

Tasks with failures

Models
GPT-5.6 Sol
GPT-5.6 Luna
GPT-5.6 Terra
Claude Opus 4.8
Claude Fable 5
GLM 5.2
Claude Sonnet 5
Kimi K3
Grok 4.5
GPT 5.5
Claude Opus 4.7
Gemini 3.5 Flash
Gemini 3.1 Pro Preview (02/26)
Claude Haiku 4.5 (Nonthinking)

Task detail

cc3fd33

Issue statement

The tracker service exposes a POST /retry-or-resume-benchmark/{benchmark_id} endpoint that re-enqueues a previously created benchmark run using the arguments it was originally created with, including the agent contract stored on the benchmark row. That stored contract carries a secrets mapping (environment variable name -> secret name) that is passed through to the tasks when they run.

Today there is no way to change those secrets when resuming or retrying a run. If a secret in the stored contract is wrong, stale, or missing, the only option is to recreate the benchmark from scratch. Users need to be able to override secrets at resume/retry time so the resumed tasks pick up the corrected values.

Please extend the resume/retry endpoint so the request body accepts an optional secrets mapping (defaulting to an empty mapping) of environment-variable name to secret name. When a non-empty secrets mapping is provided, it must be merged into the agent contract's existing secrets before the run is re-enqueued:

  • Keys already present in the stored contract are replaced by the values from the request.
  • Keys not present in the stored contract are added.
  • Keys in the stored contract that are not mentioned in the request are left untouched.

The merge must be persisted to the stored benchmark row (so a later refresh from the database reflects the merged secrets) and must also be reflected in the contract carried by the request payload that is used to re-enqueue the resumed tasks. When no secrets mapping is supplied, behavior must be unchanged. Supplying secrets must not otherwise change existing resume/retry behavior (status codes, task selection, concurrency handling, header forwarding, etc.).

View Hidden Tests
diff --git a/services/tracker/tests/unit/test_stop_and_resume.py b/services/tracker/tests/unit/test_stop_and_resume.pyindex c59a0bbf..7fd88b2c 100644--- a/services/tracker/tests/unit/test_stop_and_resume.py+++ b/services/tracker/tests/unit/test_stop_and_resume.py@@ -518,6 +518,63 @@ class TestStopAndResume:         assert observed_headers["X-Descope-Api-Key"] == "tracker-api-key"         assert captured_request_json["service_headers"]["X-Descope-Api-Key"] == "tracker-api-key" +    async def test_retry_or_resume_applies_secrets_to_stored_contract(+        self,+        example_benchmark_object: Benchmark,+        database_session: Session,+        monkeypatch: MonkeyPatch,+    ):+        """Resume secrets should update the contract used by resumed tasks.++        Test cases:+        - Existing env var mappings are replaced by resume overrides.+        - New env var mappings are added to the stored contract before enqueue.+        """+        benchmark_row = example_benchmark_object+        benchmark_row.status = BenchmarkStatus.STOPPED+        benchmark_row.arguments.contract.secrets = {+            "ANTHROPIC_API_KEY": "old-secret",+            "OPENAI_API_KEY": "openai-secret",+        }+        database_session.add(benchmark_row)+        database_session.commit()++        captured_request_json: dict[str, Any] = {}++        async def _mock_reset_to_in_progress_status(*_args: Any, **_kwargs: Any):+            return ["task_0"]++        class _MockKicker:+            def with_labels(self, **_kwargs: Any) -> "_MockKicker":+                return self++            async def kiq(self, **kwargs: Any) -> None:+                captured_request_json.update(kwargs["start_benchmark_request_json"])++        monkeypatch.setattr("main.reset_to_in_progress_status", _mock_reset_to_in_progress_status)+        monkeypatch.setattr("main.process_benchmark.kicker", lambda: _MockKicker())++        response = client.post(+            f"/retry-or-resume-benchmark/{benchmark_row.id}",+            json={+                "task_ids": [],+                "service_headers": {},+                "secrets": {+                    "ANTHROPIC_API_KEY": "new-secret",+                    "GEMINI_API_KEY": "gemini-secret",+                },+            },+        )++        assert response.status_code == 200+        assert captured_request_json["contract"]["secrets"] == {+            "ANTHROPIC_API_KEY": "new-secret",+            "OPENAI_API_KEY": "openai-secret",+            "GEMINI_API_KEY": "gemini-secret",+        }+        database_session.refresh(benchmark_row)+        assert benchmark_row.arguments.contract.secrets == captured_request_json["contract"]["secrets"]+     async def test_running_retry_noops_without_error_tasks(         self,         example_benchmark_object: Benchmark,