We’ve released Public Benefits Bench, built with the Center for Civic Futures and Code for America, which asks whether models can help people navigate SNAP benefits. Across 459 expert-validated scenarios, we found:

• No model is reliable enough to be trusted with benefits guidance yet. Claude Opus 4.8 leads the strongest condition at just 62.1% accuracy — meaning beneficiaries get incorrect answers roughly 40% of the time — and no model crosses 65%.
• How models use their tools matters more than what they already know. Because SNAP rules shift constantly and vary by county, the models that do best are the ones that ask good follow-up questions and search effectively, not the ones with the most policy baked into their weights.
• The gap is grounding, not raw capability. The authoritative information these questions require — internal manuals, policy notices, and local procedures — sits with state agencies and isn’t available in forms AI can reliably use.

Closing this gap will take collaboration between AI labs and state agencies on narrowly scoped, retrieval-grounded tools — rigorously tested against the real questions beneficiaries ask.

• We evaluated NVIDIA’s new open-weight Nemotron 3 Ultra across our benchmark suite. It ranks #5 among open-weight models on the Vals Index, with a score of 43.99%.

• The model is strongest on domain-specific benchmarks. Among open-weight models, it ranks #3 on TaxEval v2 (73.10%), #4 on CorpFin v2 (65.46%), #5 on Finance Agent v2 (37.53%), and #6 on MedCode (38.62%).

• General reasoning is solid: Nemotron 3 Ultra scores 86.11% on GPQA Diamond, 85.76% on MMLU Pro, and 82.07% on LegalBench, landing between #5 and #7 among open-weight models on each.

• On coding, it scores 85.98% on LiveCodeBench, ranking #3 among open-weight models, and 50.94% on Terminal Bench 2.1, ranking #4 among open-weight models.

The model is text-only, with a 1M-token context window and 128k max output tokens. We ran it via the NVIDIA API with top_p set to 0.95.

• We evaluated Alibaba’s new Qwen 3.7 Plus on the Vals Index, where it ranks #13 with a score of 52.33%.

• Its strongest Index component was Vibe Code Bench, where it ranks #8 on the Index subset with a score of 46.94%. On the full Vibe Code Bench leaderboard, the model scores 46.39%.

• On coding benchmarks, Qwen 3.7 Plus matches Qwen 3.7 Max on the SWE-bench Verified Index subset at 66.67%, and scores 52.81% on Terminal Bench 2.1.

• On finance benchmarks, it scores 62.01% on CorpFin v2 and 41.25% on Finance Agent v2, trailing Qwen 3.7 Max while slightly improving on Qwen 3.6 Plus in Finance Agent v2.

The model was run via the Alibaba API at temperature 0.7, with preserve_thinking enabled, a 1M-token context window, and 65,536 max output tokens.

• MiniMax just released their latest open-weights MiniMax-M3 model, with multimodal support. It ranks #6 on the Vals Index (58.94%) and #6 on the Vals Multimodal Index (59.97%), making it the top open-weights model on both.

• It is also the top open-weights performer across multiple domain-specific benchmarks, including LegalBench, MedCode, and Finance Agent v2, where it scored 48.27% — a 20-point improvement from its predecessor, MiniMax-M2.7.

• On coding, we see a big improvement on Vibe Code Bench, where the model scores 47.57%, a massive 35-point increase from MiniMax-M2.7. On SWE-bench Verified (75.00%) and Terminal Bench 2.1 (53.56%), performance was mixed, ranking #17 and #12 respectively.

The model was run via the official MiniMax API at temperature 1, top_p 0.95, and 512k max output tokens.

Congrats to the MiniMax team on a strong release!