We evaluated Anthropic’s new Claude Opus 5 across 27 benchmark leaderboards.

• Opus 5 places #2 on both the Vals Index (74.82%) and the Vals Multimodal Index (73.90%), within 0.33 and 0.26 points of Claude Fable 5, respectively.

• Opus 5 takes #1 on 14 leaderboards: SWE-bench Verified (97.00%), IOI (91.67%), Code Migration (57.47%), ProgramBench (3.00% fully resolved), CorpFin v2 (73.19%), Finance Agent v2 (58.63%), Legal Research Bench (55.29%), MedCode (63.57%), MedScribe (90.99%), MortgageTax (72.06%), Public Benefits Bench (76.93%), MMLU Pro (91.59%), MMMU (89.88%), and ProofBench (78.00%).

• Its other coding results include #2 on Terminal-Bench 2.1 (84.64%), Vibe Code Bench (88.40%), and LiveCodeBench (89.03%).

We ran Opus 5 with Claude Opus 4.8 as a server-side fallback for refusals. Counting fallback-assisted results as failures changes Terminal-Bench 2.1 from 84.64% to 81.27%, MMLU Pro from 91.59% to 91.58%, the Vals Index from 74.82% to 74.47%, and the Vals Multimodal Index from 73.90% to 73.58%. Fallbacks on Finance Agent v2 and CyberBench did not change their published scores.

The model has a 1M-token context window and 128k max output tokens. Evaluations were run with compute effort set to “max” except Terminal-Bench 2.1, which used “high” effort, and temperature set to 1.0 where configurable.

Congrats to the Anthropic team on the release!

We launched the Web Search Index, which isolates the effect of the web-search tool itself: each model runs the same legal-research and finance-analysis tasks with its own native provider search versus an independent web-search tool (Exa), holding the model fixed.

• Across all four models tested (Claude Fable 5, GPT-5.6 Sol, Gemini 3.5 Flash, and Grok 4.5), swapping in Exa improved overall accuracy — by roughly 1.5 to 4.3 points.

• The gains concentrate in finance analysis, where every model scored higher with Exa — for example, Grok 4.5 rose +9.8 points (32.1% → 41.9%) and Claude Fable 5 +8.8 points (44.8% → 53.6%).

• Legal research was a wash: native search led for Claude Fable 5 and Grok 4.5, while Exa led for Gemini 3.5 Flash and GPT-5.6 Sol — so the better tool depends on the model.

• Total cost is dominated by model inference: Exa’s flat per-search fee is a negligible share of per-task cost, and native provider search is cheaper on average.

See the full leaderboard, per-domain breakdowns, and search-behavior analysis on the Web Search Index.

We evaluated Google’s Gemini 3.5 Flash Lite on the Vals Index and across our benchmark suite.

• Gemini 3.5 Flash Lite scores 51.00% on the Vals Index, placing #25 of 39 models overall.

• Its Index component results include 68.63% on the SWE-bench Verified subset, 62.24% on the CorpFin v2 subset, and 46.63% on the Finance Agent v2 subset.

• The model scores 32.77% on the Vibe Code Bench Index subset and 50.19% across three full trials of Terminal-Bench 2.1.

We evaluated Gemini 3.5 Flash Lite with temperature 1, high reasoning effort, and up to 65k output tokens. The model supports a 1M-token context window, multimodal inputs, and tool calling.

Results are now available across 25 benchmarks. On the full 500-task SWE-bench Verified evaluation, Gemini 3.5 Flash Lite scores 75.00%; the Vals Index uses the 68.63% subset result above.

We evaluated Google’s Gemini 3.6 Flash on the Vals Index and proprietary benchmarks.

• Gemini 3.6 Flash scores 62.43% on the Vals Index, placing #14 of 38 models overall and finishing within 0.32 points of Gemini 3.5 Flash.

• Gemini 3.6 Flash ranks #1 on the CyberBench Patch track at 84.75%, which measures fixing real-world open-source vulnerabilities.

• Its coding results include 77.45% on the SWE-bench Verified Vals Index subset and 57.88% on the Vibe Code Bench Vals Index subset, improvements of 1.96 and 3.17 points over Gemini 3.5 Flash, respectively.

• Gemini 3.6 Flash scores 73.78% across three full trials of Terminal-Bench 2.1, placing #8 of 43 models. It also places #7 of 74 models on MedCode, scoring 53.15%.

We evaluated Gemini 3.6 Flash with temperature 1, high reasoning effort, and up to 65k output tokens. The model supports a 1M-token context window, multimodal inputs, and tool calling.

Congrats to the Google team on the release!