• We evaluated the open-weight Kimi K2.7 Code across our coding benchmarks, where it is the new #1 open-weight model on SWE-bench Verified (78.20%) and Terminal-Bench 2.1 (67.04%).

• Its SWE-bench score matches Claude Opus 4.6 (Thinking) and GPT 5.4 (xhigh), and narrowly edges GPT 5.3 Codex.

• On Vibe Code Bench, which tests full-stack app building from product specs through implementation and browser-based evaluation, Kimi K2.7 Code scores 47.21% and ranks #3 among open-weight models.

• It also scores 82.05% on LiveCodeBench, ranking #8 among open-weight models.

The headline is long-horizon coding: Kimi K2.7 Code is especially competitive on agentic SWE and terminal tasks, while LiveCodeBench is a relative weaker spot.

• We evaluated Anthropic’s new Claude Fable 5 across our benchmark suite. It resets the top of the leaderboard, taking #1 on the Vals Index (75.15%) and the Vals Multimodal Index (74.15%).

• Fable 5 topped most of the suite: #1 on CorpFin v2 (71.83%), MedScribe (88.52%), LegalBench (88.56%), MMLU Pro (91.50%), MMMU (89.31%), and ProofBench (77.00%).

• Coding is the headline. Claude Fable 5 is #1 on every coding benchmark with a scored Fable 5 result: Vibe Code Bench (90.35%), SWE-bench Verified (95.00%), Terminal-Bench 2.1 (80.52%), LiveCodeBench (89.78%), and IOI (72.25%).

• Vibe Code Bench is the standout: Anthropic now holds the top three spots, and Fable 5 leads the best non-Anthropic model by more than 20 points. Six months ago, no model cracked 20% on this benchmark; Fable 5 now reaches 90.35%.

• Compared with Claude Opus 4.8, Fable 5 improves by roughly 5 points on the Vals Index and roughly 8 points on Vibe Code Bench, even though Opus 4.8 took #1 just a week ago.

The API showed a high rate of refusals, especially on bio and cyber-related questions. Because of this, we ran Fable 5 with Claude Opus 4.8 as a fallback: if Fable 5 refused a task, Opus 4.8 handled the request instead. This mainly affected Terminal-Bench 2.1, GPQA, MMLU Pro, and MMMU.

The model has a 1M-token context window and 128k max output tokens. Evaluations were run with compute effort set to “max” and temperature set to 1.0.

Congrats to the Anthropic team on the release!

We’ve released Public Benefits Bench, built with the Center for Civic Futures and Code for America, which asks whether models can help people navigate SNAP benefits. Across 459 expert-validated scenarios, we found:

• No model is reliable enough to be trusted with benefits guidance yet. Claude Fable 5 leads the strongest condition at 71.7% accuracy — meaning beneficiaries still get incorrect answers nearly 30% of the time — and no model crosses 75%.
• How models use their tools matters more than what they already know. Because SNAP rules shift constantly and vary by county, the models that do best are the ones that ask good follow-up questions and search effectively, not the ones with the most policy baked into their weights.
• The gap is grounding, not raw capability. The authoritative information these questions require — internal manuals, policy notices, and local procedures — sits with state agencies and isn’t available in forms AI can reliably use.

Closing this gap will take collaboration between AI labs and state agencies on narrowly scoped, retrieval-grounded tools — rigorously tested against the real questions beneficiaries ask.

• We evaluated NVIDIA’s new open-weight Nemotron 3 Ultra across our benchmark suite. It ranks #5 among open-weight models on the Vals Index, with a score of 43.99%.

• The model is strongest on domain-specific benchmarks. Among open-weight models, it ranks #3 on TaxEval v2 (73.10%), #4 on CorpFin v2 (65.46%), #5 on Finance Agent v2 (37.53%), and #6 on MedCode (38.62%).

• General reasoning is solid: Nemotron 3 Ultra scores 86.11% on GPQA Diamond, 85.76% on MMLU Pro, and 82.07% on LegalBench, landing between #5 and #7 among open-weight models on each.

• On coding, it scores 85.98% on LiveCodeBench, ranking #3 among open-weight models, and 50.94% on Terminal Bench 2.1, ranking #4 among open-weight models.

The model is text-only, with a 1M-token context window and 128k max output tokens. We ran it via the NVIDIA API with top_p set to 0.95.