Key Takeaways
- Scores spread widely, from 35.83% to 70.21%. Claude Opus 5.5 leads, ahead of Claude Fable 5.1 (61.25%) and Claude Sonnet 5.5 (58.33%).
- Cost per task ranges from $0.03 to $11.40 and does not track score outside the top of the board. A model can cost more than 47 times as much as another and resolve fewer tasks.
- Network access decides what the benchmark measures. Agents with general internet access found the upstream pull request for the issue and copied its fix, and closing that channel cost the three models we compared 21 to 35 points. Every published score here is a no-network score.
Background
SWE-bench Multimodal asks a model to fix a real GitHub issue in a JavaScript repository. The bug report depends on images: a chart with the wrong axis labels, a diagram editor that drops a connection, or a syntax highlighter that colors a keyword incorrectly. The screenshot or short recording shows what the text leaves out.
Unlike text-only issue resolution, this requires the model to read the image, find the responsible code in an unfamiliar front-end codebase, and make a change that passes the repository’s own tests.
The test split has 480 tasks from eleven repositories.
SWE-bench Multimodal was created by John Yang, Carlos Jimenez, Ofir Press et al. at Stanford University and Princeton University.
Methodology
We run models with mini-swe-agent, the minimal reference agent from the SWE-agent team. The model gets the issue text and a shell, and works turn by turn until it decides it is done. Each task runs in its own sandbox, and the upstream SWE-bench harness grades the agent’s final diff with the repository’s own tests.
Issues reference their screenshots as links. We download each image before the run and attach it to the first message alongside the issue text. We run each model once, and the error bars are the binomial standard error.
The agent has no internet access
A SWE-bench Multimodal instance id contains the number of the upstream pull request that fixed the issue, so the official patch is easy to find. Agents with general network access use it. Every published run uses an agent with no internet access. We download the screenshots beforehand and the agent reads them from disk. Closing the network channel cost each model between 21 and 35 percentage points:
| Model | Diagnostic: with network access | No network |
|---|---|---|
| Muse Spark 1.2 | 62.71% | 41.25% |
| Muse Spark 1.3 | 69.17% | 42.08% |
| Muse Spark 1.3 Max | 77.92% | 43.33% |
You can find more information about SWEbench Multimodal here: swebench.com/multimodal.html