Vals AI Blog

Do Agent Teams Pay Off? A Case Study on Vibe Code Bench

We ran GPT 6 Sol and Claude Opus 5.5 on Vibe Code Bench as single agents and as teams, at medium and max reasoning effort. Teams cost 1.8 to 5.1 times as much, and only for Sol at medium effort was the difference in score statistically significant. Whether a team is worth it depends on the model, task, and budget: a team can boost performance, but higher reasoning effort is an alternative worth weighing.

An Evaluation of Inception's Mercury Decide

We ran Inception's Mercury Decide on the benchmarks from our Jev evaluation.

Trust but Verify: Our first Public Sector Summit

Reflections from our first Public Sector Summit at STATION DC on youth wellbeing, AI diplomacy with China, and independent evaluation.

Two-Thirds of MiMo v2.6's Coding Tasks Leak the Answer. Are Models That Exploit This Misbehaving?

In two-thirds of the coding environments Xiaomi released for MiMo v2.6, the fix is still readable in the task's Git history, and MiMo finds it.

An Independent Evaluation of TypeSafe's Jev

We evaluated Jev against frontier models on claim-verification and multiple choice legal questions.

Two Room-Temperature Antiferromagnetic Semiconductor Candidates

We are sharing two candidate magnets for next-generation computer memory, found by a team of Claude Opus 5.5 agents. Both are predicted to have zero net magnetism yet still sort electrons by spin: one a new compound we designed, the other a material first made in 1999. We're sharing the full calculations, the code, and a list of known caveats.

Vals Web Search Index: Evaluating Search for Real Work

Why we built the Vals Web Search Index to measure whether a search tool helps an agent resolve real professional tasks.

Has the Bitter Lesson Come for AI Detectors?

Benchmarking AI Detection Adversarially on Private Writing Samples

A Lean Proof of the Thomson Problem for Seven Electrons

We asked ten Claude Sonnet 5.5 agents to prove in Lean that the pentagonal bipyramid is the lowest-energy way to place seven electrons on a sphere. In about 15 hours they produced a 17,895-line proof that Lean accepts, and a second kernel implementation confirms it.

Evaluating AI Safety in Teen Conversations

A study of 648 simulated conversations examines how chatbots respond as teens disclose more, resist advice, and ask for help.

A Faster Shortest Path Algorithm

We asked ten Claude Opus 5.5 agents to find a faster shortest-path algorithm and prove it in Lean. Within 15 hours, they produced C-HD: a formally verified improvement over the published bounds.

AI Cheating is on the Rise

An integrity audit across BioMysteryBench, Terminal-Bench 2.1, and SWE-bench Verified shows that cheating increasingly complicates evaluation.

Vals Environmental Impacts Report

Auditing the carbon, water, and energy costs of LLMs

Claude Fable 5.1 Solves the Cyphral Distich

We gave Claude Fable 5.1 an open task: solve an unsolved 370-year-old cipher. It solved it within a day.

Series A: Always a Higher Peak

Child AI Safety

We are developing independent, youth-specific evaluations with clinicians and academic researchers.

Vals Public Sector Memo

We're bringing independent AI evaluation to government.

Measuring Healthcare AI Where It Actually Works

AI Models Excel at Clinical Documentation But Struggle With Medical Billing

Behind the Scenes of Vibe Code Bench

Last week, we released Vibe Code Bench. This is our most ambitious benchmark to-date. I hope it will provide a useful signal for researchers and vibe-coders alike.