Public

abhishek203/SfAgent

Updated: 8/17/2026

Languages

TypeScript94.2%JavaScript4.7%Python0.3%CSS0.3%Shell0.2%HTML0.1%Other0.1%
2 Models30 Tasks

🙌 OpenHands: Code Less, Make More

Harness

1

Mini-SWE-agent
24 / 30

$0.23

57.08s

2

Mini-SWE-agent
21 / 30

$0.18

2m01s

Key Takeaways

  • Claude Opus 4.8 (High Effort) with Mini-SWE-agent scores 80%, compared with 70% for Claude Haiku 4.5 (Thinking) with Mini-SWE-agent.
  • Claude Opus 4.8 (High Effort) with Mini-SWE-agent costs $0.23 per test and takes 57.08 seconds; Claude Haiku 4.5 (Thinking) costs $0.18 and takes 120.54 seconds.

Model Comparison

Accuracy

80.00%

Claude Opus 4 8 High

70.00%

Claude Haiku 4.5 (Thinking)

Task outcomes

30 tasks

Both
Claude Opus 4 8 High only
Claude Haiku 4.5 (Thinking) only
Neither
Not attempted

Cost / test

$0.23

Claude Opus 4 8 High

$0.18

Claude Haiku 4.5 (Thinking)

Cost distribution

$0.00$0.63$1.27

Latency

57s

Claude Opus 4 8 High

2m 1s

Claude Haiku 4.5 (Thinking)

Latency distribution

0s2m 20s4m 41s

Cost Analysis

Cost / Test vs. Accuracy
ACCURACYCOST

Average Token Use / Test

Token Usage
InputOutputReasoningCache readCache write
Claude Haiku 4.5 (Thinking)
779K
anthropic/claude-opus-4-8-high
141K

Cost is the clearest tradeoff in this comparison. anthropic/claude-opus-4-8-high leads at 80.00% for $0.23 per test. Claude Haiku 4.5 (Thinking) is the lower-cost option at 70.00% for $0.18 per test.

Latency Analysis

Latency vs. Accuracy
ACCURACYLATENCY

Average Response Time / Test

Response Time
Claude Haiku 4.5 (Thinking)
2m 1s
anthropic/claude-opus-4-8-high
57s

anthropic/claude-opus-4-8-high is both the most accurate and fastest model in this comparison at 80.00% and 57s.

Tasks with failures

Models
Claude Opus 4 8 High
Claude Haiku 4.5 (Thinking)

Task detail

1354675

Issue statement

Remote runtime startup can take long enough that callers currently receive no progress feedback. When a status callback is supplied, report the startup lifecycle in order: runtime startup begins, container image construction begins (when an image must be built), the runtime is waiting for the remote client after preparing the start request, and a blank status clears the message once initialization is complete. Supplying no callback must remain supported.

View Hidden Tests
diff --git a/tests/valsmith/test_remote_runtime_status.py b/tests/valsmith/test_remote_runtime_status.pynew file mode 100644index 00000000..2430de2d--- /dev/null+++ b/tests/valsmith/test_remote_runtime_status.py@@ -0,0 +1,136 @@+import ast+import os+import threading+import types+import uuid+from pathlib import Path+++def _load_remote_runtime_class():+    source = Path('openhands/runtime/remote/runtime.py').read_text()+    tree = ast.parse(source)+    remote = next(+        node+        for node in tree.body+        if isinstance(node, ast.ClassDef) and node.name == 'RemoteRuntime'+    )+    methods = [+        node+        for node in remote.body+        if isinstance(node, (ast.Assign, ast.AnnAssign))+        or (+            isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))+            and node.name in {'__init__', 'send_status_message'}+        )+    ]+    isolated = ast.Module(+        body=[+            ast.ClassDef(+                name='RemoteRuntime',+                bases=[ast.Name(id='Runtime', ctx=ast.Load())],+                keywords=[],+                body=methods,+                decorator_list=[],+            )+        ],+        type_ignores=[],+    )+    ast.fix_missing_locations(isolated)++    class Runtime:+        def __init__(+            self,+            config,+            event_stream,+            sid,+            plugins,+            env_vars,+            status_message_callback,+        ):+            self.plugins = plugins or []+            self.status_message_callback = status_message_callback++    class Session:+        def __init__(self):+            self.headers = {}++    class Response:+        status_code = 200+        text = ''++        def __init__(self, payload):+            self.payload = payload++        def json(self):+            return self.payload++    def send_request(_session, method, url, **_kwargs):+        if url.endswith('/registry_prefix'):+            return Response({'registry_prefix': 'registry.example'})+        if url.endswith('/image_exists'):+            return Response({'exists': True})+        if url.endswith('/start'):+            response = Response({'runtime_id': 'runtime-1', 'url': 'http://runtime'})+            response.status_code = 201+            return response+        raise AssertionError(f'unexpected request: {method} {url}')++    namespace = {+        'Runtime': Runtime,+        'AppConfig': object,+        'EventStream': object,+        'PluginRequirement': object,+        'Optional': __import__('typing').Optional,+        'Callable': __import__('typing').Callable,+        'RemoteRuntimeBuilder': lambda *_args: object(),+        'build_runtime_image': lambda *_args, **_kwargs: 'built-image',+        'send_request': send_request,+        'requests': types.SimpleNamespace(Session=Session),+        'logger': types.SimpleNamespace(+            warning=lambda *_args: None,+            info=lambda *_args: None,+            debug=lambda *_args: None,+        ),+        'os': os,+        'threading': threading,+        'uuid': uuid,+    }+    exec(compile(isolated, '<isolated RemoteRuntime>', 'exec'), namespace)+    return namespace['RemoteRuntime']+++def test_remote_runtime_reports_startup_status_in_order():+    remote_runtime = _load_remote_runtime_class()+    messages = []+    sandbox = types.SimpleNamespace(+        api_key='secret',+        remote_runtime_api_url='https://sandbox.example',+        runtime_container_image=None,+        base_container_image='python:3.12',+        runtime_extra_deps=None,+        force_rebuild_runtime=False,+        browsergym_eval_env=None,+        user_id=1000,+        timeout=120,+    )+    config = types.SimpleNamespace(+        sandbox=sandbox,+        workspace_base=None,+        workspace_mount_path_in_sandbox='/workspace',+        debug=False,+        run_as_openhands=True,+    )++    remote_runtime(+        config=config,+        event_stream=object(),+        sid='session',+        status_message_callback=messages.append,+    )++    assert messages == [+        'STATUS$STARTING_RUNTIME',+        'STATUS$STARTING_CONTAINER',+        'STATUS$WAITING_FOR_CLIENT',+        ' ',+    ]