BioMysteryBench

Key Takeaways

• BioMysteryBench clearly separates frontier models, with Claude Opus 5 leading the pack at 79.3% accuracy and $3.29 per task.
• Frontier models solve up to 43% of the tasks that none of the participating human experts solved.
• Failure analysis reveals most models give incorrect answers instead of responding with uncertainty.
• DeepSeek V4 Flash leads the efficiency frontier, costing roughly 85× less per task than Opus 5 while still achieving 64.4% accuracy.

Evaluation

BioMysteryBench is a benchmark developed by Anthropic to measure agentic biological analysis performance. Tasks are constructed by stripping real-world biological data of metadata (e.g. sample origin) and evaluating whether models can recover this information via computational analysis. Tasks may require agents to inspect files, run command-line tools, write analysis code, or query biological databases. Tasks cover areas including sequencing, expression analysis, variant calling, epigenomics, metagenomics, proteomics, and metabolomics.

Up to five domain experts attempted each problem, and the tasks solved by at least one expert were labeled human-solvable. Tasks that no expert solved were labeled human-difficult.

Five tasks sent to Opus, as well as three sent to each of GPT-5.6 Sol and Luna, were refused by biology content filters. These are counted as task failures for the purposes of evaluation.

BioMysteryBench was developed by Anthropic. Its task materials are released under CC BY 4.0 with restrictions against their use for model training, while the biological data remain subject to their original repositories’ policies.

Model behavior

We analyze failure modes using an openai/gpt-5.6-terra judge that reads only the final message from each scored-incorrect trajectory. The view shows each failure type as a share of that model’s incorrect attempts, including provider refusals and filters.

We find that models overwhelmingly choose to give confidently wrong answers instead of expressing uncertainty. OpenAI models are comparatively more likely to remain uncertain or end without a definitive answer, while Kimi K3 has the highest confidently-wrong share in this panel: 93.5% of its incorrect attempts. This is directionally consistent with Kimi’s own report of a “noticeable gap in user experience” compared with other models.

Methodology

Models run with the Terminus 2 agent harness inside sandboxes configured for bioinformatics work. The environment includes common alignment, sequence-processing, quality-control, workflow, R/Bioconductor, and Python tooling. Network access is restricted to the scientific package repositories and biological databases required by the benchmark. Tasks were limited to 200 agent turns and four hours of wall-clock execution.

Each displayed score is the mean of three independent full-benchmark runs, and error bars show the standard error across those three run-level scores. Models use their registry-default reasoning or compute effort; the exact settings used for each row are available from the leaderboard’s settings control.

Each task is evaluated from the agent’s submitted answer by a separately configured judge (we use openai/gpt-5.6-terra). The judge is additionally instructed to flag when models arrive at the answer via cheating, though we don’t observe this in our environment. Task scores are binary: a fully accepted answer scores 1, while judge-rejected answers and model/provider failures or refusals score 0.

Dataset Provider