Mistral Large 3 is #20 on our Vals Index (out of 24).

A main source of failure was the model’s difficulty in calling tools - a common error mode was the model making a function call with a malformed tool name (e.g. inserting the function arguments into the tool name). (2/3)

However, it achieves a six percentage point increase over Magistral Medium 1.2 (09/2025), with comparable performance on the Vals Index to GPT OSS 120B.

Full results will be posted soon.

Claude Opus 4.5 (Thinking) is the new leader on our Vals Index, surpassing its cheaper variant, Claude Sonnet 4.5 (Thinking).

Like other Anthropic models, Claude Opus 4.5 (Thinking) demonstrates strong performance on coding tasks - the thinking and nonthinking variants take the top 2 spots on SWE-Bench. However, the model performs worse than Claude Sonnet 4.5 (Thinking) on our new VibeCodeBench.

Unlike other Anthropic models, Claude Opus 4.5 (Thinking) showcases strong multimodal capabilities, placing first on our Multimodal Vals Index. It also places first on SAGE, surpassing the previous leader, Gemini 3 Pro (11/25). It’s also cheaper than before - it’s a third of the price of Claude Opus 4.1 (Nonthinking) and on some benchmarks, more token efficient than Claude Sonnet 4.5 (Thinking)!

Along with Claude Sonnet 4.5 (Thinking) and Claude Opus 4.5 (Thinking), the top 3 models on our Vals Index are now all from Anthropic. We look forward to other providers catching up!

We just released Vibe Code Bench : the first benchmark that tests whether AI models can actually build complete web applications from scratch.

• GPT 5.1 takes first place with 24% accuracy, improving by 4% over its third-place predecessor GPT 5 (with 20% accuracy).
• Claude Sonnet 4.5 (Thinking) takes second place, but costs more than twice as much than the OpenAI models.
• Despite the much-vaunted release, Gemini 3 Pro (11/25) lags behind in fourth place, with only 14.3% accuracy.

Overall, we see these results as evidence that vibe code has plenty of room to improve. We look forward to seeing this improvement!

We just evaluated Grok 4.1 Fast (Reasoning) and Grok 4.1 Fast Non-Reasoning on all benchmarks!

We found that the reasoning variant consistently outperforms the non-reasoning variant - see our comparison page. Grok 4.1 Fast Non-Reasoning struggles - its best performance is tenth place on our IOI benchmark, while Grok 4.1 Fast (Reasoning) achieves strong performance across the board, placing third on our CorpFin benchmark.

Both models perform similarly to their predecessors - see our comparison pages for both the reasoning and non-reasoning variants. Like their predecessors, both models are also extremely cost-efficient on a per-token basis.

Congratulations to the xAI team for the launch of Grok 4.1!