18jeffreyma/OpenHands
Languages
🙌 OpenHands: Code Less, Make More
Harness | Input / Output Cost | ||||||
|---|---|---|---|---|---|---|---|
1 |  Mini-SWE-agent | 19 / 30 | $0.42 | $1.25/$4.25 | 3m16s |
Key Takeaways
- This 30-task result is a single-run benchmark, so it summarizes one observed outcome without a comparison.
- Muse Spark 1.2 with Mini-SWE-agent records 196.21 seconds of latency per test and a 63.33% score.
Cost Analysis
Average Token Use / Test
Cost is the clearest tradeoff in this comparison. Muse Spark 1.2 leads at 63.33% for $0.42 per test. No other model in this comparison is cheaper.
Latency Analysis
Average Response Time / Test
Muse Spark 1.2 is both the most accurate and fastest model in this comparison at 63.33% and 3m 16s.
Tasks with failures
| Models | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Muse Spark 1.2 |
Task detail
7fab698Issue statement
Container cleanup currently hides unexpected failures encountered while stopping matching Docker containers. This makes cleanup appear successful even when a programming error or other non-Docker exception prevented a container from being stopped. Update stop_all_containers so that it continues to tolerate the expected Docker APIError and NotFound race conditions, but lets unexpected stop failures propagate to the caller. The Docker client must still be closed when such a failure occurs.
View Hidden Tests
diff --git a/tests/unit/runtime/impl/docker/test_containers_cleanup.py b/tests/unit/runtime/impl/docker/test_containers_cleanup.pynew file mode 100644index 000000000..9d18a62b0--- /dev/null+++ b/tests/unit/runtime/impl/docker/test_containers_cleanup.py@@ -0,0 +1,29 @@+import importlib.util+from pathlib import Path+from unittest.mock import MagicMock, patch++import pytest+++module_path = (+ Path(__file__).parents[5] / 'openhands/runtime/impl/docker/containers.py'+)+spec = importlib.util.spec_from_file_location('containers_under_test', module_path)+assert spec is not None and spec.loader is not None+containers_module = importlib.util.module_from_spec(spec)+spec.loader.exec_module(containers_module)+++def test_unexpected_stop_error_propagates_and_client_is_closed() -> None:+ client = MagicMock()+ matching_container = MagicMock()+ matching_container.name = 'openhands-session-1'+ matching_container.stop.side_effect = RuntimeError('unexpected stop failure')+ client.containers.list.return_value = [matching_container]++ with patch.object(containers_module.docker, 'from_env', return_value=client):+ with pytest.raises(RuntimeError, match='unexpected stop failure'):+ containers_module.stop_all_containers('openhands-')++ matching_container.stop.assert_called_once_with()+ client.close.assert_called_once_with()