Blog

Inkling 2 Small: An Inkling of Profanity

Thinking Machines' Inkling 2 Small wrote "fuck" in its reasoning in 50 of 2,655 runs on our benchmarks, usually late in a long run that went on to fail.

How Much Does Reasoning Effort Affect Score and Cost?

At their lowest reasoning setting, frontier models often keep a majority of their performance at a fraction of the cost.

Can AI Get Immigration Right?

This report presents an independent evaluation of how leading AI models perform on U.S. immigration law questions, and what it means for the international students, entrepreneurs, employers, and other people who rely on AI to navigate the U.S. immigration system.

Thomson, One N at a Time

How the N = 7 proof works, what the proofs so far have in common, and which N we expect to come next.

Why is my local Qwen sending requests to Alibaba Cloud?

Maybe it was useful in RL

Evaluating Teen Safety

We tested 17 AI chatbots on teen safety. Every one failed, and some far more often than others.

How Far Will AI Assistants Go for a Refund They Can’t Get?

Spamming the government, using a sick mother as leverage and threatening the small claims court.

Routers Are a New AI Primitive

Model routers need an independent, end-to-end evaluation of quality, cost, and latency on real work. We're building one.

How pro-China are Chinese models?

It's complicated.

Do Agent Teams Pay Off? A Case Study on Vibe Code Bench

We ran GPT 6 Sol and Claude Opus 5.5 on Vibe Code Bench as single agents and as teams, at medium and max reasoning effort. Teams cost 1.8 to 5.1 times as much, and only for Sol at medium effort was the difference in score statistically significant. Whether a team is worth it depends on the model, task, and budget: a team can boost performance, but higher reasoning effort is an alternative worth weighing.

An Evaluation of Inception's Mercury Decide

We ran Inception's Mercury Decide on the benchmarks from our Jev evaluation.

Trust but Verify: Our first Public Sector Summit

Reflections from our first Public Sector Summit at STATION DC on youth wellbeing, AI diplomacy with China, and independent evaluation.

Two-Thirds of MiMo v2.6's Coding Tasks Leak the Answer. Are Models That Exploit This Misbehaving?

In two-thirds of the coding environments Xiaomi released for MiMo v2.6, the fix is still readable in the task's Git history, and MiMo finds it.

An Independent Evaluation of TypeSafe's Jev

We evaluated Jev against frontier models on claim-verification and multiple choice legal questions.

Two Room-Temperature Antiferromagnetic Semiconductor Candidates

We are sharing two candidate magnets for next-generation computer memory, found by a team of Claude Opus 5.5 agents. Both are predicted to have zero net magnetism yet still sort electrons by spin: one a new compound we designed, the other a material first made in 1999. We're sharing the full calculations, the code, and a list of known caveats.

Vals Web Search Index: Evaluating Search for Real Work

Why we built the Vals Web Search Index to measure whether a search tool helps an agent resolve real professional tasks.

Has the Bitter Lesson Come for AI Detectors?

Benchmarking AI Detection Adversarially on Private Writing Samples

A Lean Proof of the Thomson Problem for Seven Electrons

We asked ten Claude Sonnet 5.5 agents to prove in Lean that the pentagonal bipyramid is the lowest-energy way to place seven electrons on a sphere. In about 15 hours they produced a 17,895-line proof that Lean accepts, and a second kernel implementation confirms it.

Evaluating AI Safety in Teen Conversations

A study of 648 simulated conversations examines how chatbots respond as teens disclose more, resist advice, and ask for help.

A Faster Shortest Path Algorithm

We asked ten Claude Opus 5.5 agents to find a faster shortest-path algorithm and prove it in Lean. Within 15 hours, they produced C-HD: a formally verified improvement over the published bounds.

AI Cheating is on the Rise

An integrity audit across BioMysteryBench, Terminal-Bench 2.1, and SWE-bench Verified shows that cheating increasingly complicates evaluation.

Vals Environmental Impacts Report

Auditing the carbon, water, and energy costs of LLMs

Claude Fable 5.1 Solves the Cyphral Distich

We gave Claude Fable 5.1 an open task: solve an unsolved 370-year-old cipher. It solved it within a day.

Series A: Always a Higher Peak

Child AI Safety

We are developing independent, youth-specific evaluations with clinicians and academic researchers.

Vals Public Sector Memo

We're bringing independent AI evaluation to government.

Measuring Healthcare AI Where It Actually Works

AI Models Excel at Clinical Documentation But Struggle With Medical Billing

Behind the Scenes of Vibe Code Bench

Last week, we released Vibe Code Bench. This is our most ambitious benchmark to-date. I hope it will provide a useful signal for researchers and vibe-coders alike.