Vals Router Index

Background

New models arrive every month, giving companies an expanding set of options with different strengths and prices. The frontier is jagged: different models lead different benchmarks on the Vals Index, so even the strongest model is not the best choice for every kind of work. Keeping up means continually re-evaluating which model suits each application, an effort that grows with every workflow a company moves onto AI.

Model routers promise to take on that work, selecting a model for each request and weighing quality against cost. Agentic work raises the stakes: an agent may make hundreds of model calls to finish one task, a router can choose differently for each, and a cheaper step can create more work later. We therefore measure the quality and total cost of the finished task.

The Vals Router Index runs five routers through the same agents, tasks and graders as every model on the Vals Index: OpenRouter Auto, Fireworks FireRouter, Ramp Router, TypeSafe Jev Router and Sakana Fugu Max, which coordinates several models on each task. The tasks come from the Index’s eight benchmarks of agentic work in finance, coding, law and tax, about 1,250 in all. Each router ran with its default configuration, as a customer would get it out of the box, and is compared with every single model on the Index, run on the same tasks.

We built this evaluation on the Vals Index because it tests the kind of work routers are now sold for: long, tool-using agent tasks written and graded by people who do that work. Six of its eight benchmarks are private, and we run them under zero-data-retention agreements, so model providers cannot keep or train on the tasks.

For the comparisons below, a router’s pool means the models it used in our runs. Fugu Max does not disclose which models it calls, so it is left out of every comparison that needs its pool.

We’re eager to evaluate other routers. If you build one and want it evaluated, get in touch with us here.

Results

Routing did not consistently improve quality for the money. None of the five routers sits on the cost and accuracy frontier: each was beaten on both score and cost by at least one single model. Ramp Router came closest. DeepSeek V4.1 Flash 1017 scored about 3.5 points higher for about two thirds of its cost, and MiMo V2.6 Pro matched it for under half.

How the routers compared

Ramp Router cost about a quarter as much as GPT-6.1 Sol, its strongest model, and lost ground mainly on coding. It came within 2.5 points of Sol on six of the eight benchmarks, and nearly all of its gap came from Terminal-Bench (32.3% vs. 55.1%) and Code Migration (52.6% vs. 75.1%).

FireRouter cost less than half as much as Claude Opus 5.5 called directly, but scored 7.7 points lower, and below several cheaper models, including GLM 5.3 from its own pool. The largest gap came on Code Migration, where FireRouter scored 21.2% against 47.5% for Opus 5.5.

Auto ran at its cost-saving default and used three inexpensive models. It cost $0.82 per test, but DeepSeek V4.1 Flash 1017, one of the models it used, scored 9 points higher on its own for about four fifths as much.

Jev drew on twelve models. Its strongest, Claude Sonnet 5.5, scored 13 points higher at about three times the cost, and GPT-6.1 Sol scored 8.3 points higher for about two thirds of the cost. A broader pool did not by itself produce a better result.

Fugu Max had the weakest result of the five, and ten single models beat it on both score and cost. It often spent many steps on tasks it did not solve: on Terminal-Bench it took a median of about 110 steps, yet 45 of the 66 tasks failed in all three runs.

Among the routers themselves, Ramp Router scored about 6 points above Auto and 10 above Fugu Max on the same tasks, and was within the margin of error of FireRouter and Jev at a fraction of their cost.

What the routers chose

The routers used different model mixes on different benchmarks. Within a benchmark, though, harder tasks generally did not get more of a router’s spend on its pricier models.

Ramp Router switched models in 99% of tasks, about three and a half times per task, and always started with GPT-6 Luna. FireRouter relied mostly on Claude Opus 5.5, which answered over 95% of its calls on EMB, Terminal-Bench, Code Migration and HLAB, but only 62% to 76% on the other four, where GLM models took most of the rest. Auto usually kept one of its three models for a whole task, chosen largely by benchmark. Jev drew on the most models, twelve, including all three that Auto used, yet the two switched in about the same share of tasks, roughly a third.

Inside the runs

Less work. Under the routers, models often did far less work than on their own. On Tax Agent Bench, FireRouter’s agents submitted after a median of 7 steps, against 41 for Claude Opus 5.5 called directly, and scored about 22 points lower. On Finance Agent, Vibe Code Bench, Legal Research and Tax Agent Bench, GPT-6.1 Sol generated a fifth to two fifths of its usual output tokens under Ramp Router yet scored within a few points of its own runs. The routers appear to run models at a lower reasoning effort by default, which some models handle better than others.

The final step. On Terminal-Bench, Ramp Router gave the final, submitting step to GPT-6 Luna in 131 of 198 attempts (three on each of 66 tasks), and those attempts passed 25% of the time, against 46% for the 67 that GPT-6.1 Sol submitted. The router may simply hand the final step to GPT-6 Luna on tasks that are already lost, but the pattern raises the question of whether deciding one step at a time gives the decision to submit enough scrutiny.

Where they switched. Switching was concentrated in a few benchmarks. On Finance Agent, Vibe Code Bench, Legal Research and Tax Agent Bench, Ramp Router mostly switched to GPT-6.1 Sol after the first step or so, and stayed there; on the rest it switched more frequently, about every four steps. FireRouter rarely switched outside Code Migration, where its moves to GLM were almost all calls to summarise the conversation once it grew long. Auto and Jev switched most on Vibe Code Bench, Code Migration and Legal Research.

Opportunities for further gains

Two baselines show how each router’s own models could have done if chosen with hindsight, using the same results they are scored on. They show how much room there is for better choices, not what a router could achieve on new tasks.

Because the routers’ model mix varied by benchmark, the simpler baseline chooses one model for each benchmark and uses it throughout. For each router, we picked the combination of pool models with the highest overall score that costs no more per test than the router did. That choice scored 11.0 ± 1.2 points above FireRouter, 9.9 ± 1.2 above Auto and 13.2 ± 1.2 above Jev, each for no more money, and was within the margin of error of Ramp Router. For Ramp Router’s pool, it uses GPT-6.1 Sol for Finance Agent, EMB and Terminal-Bench, and GPT-6 Luna for the rest.

Oracle routing chooses a model for each task instead, sending it to whichever pool model scored best on it and breaking ties toward the cheaper model. To compare within the same budget, we also limit it to each router’s own cost per test: it starts every task on the pool’s cheapest model and upgrades the tasks where a better model adds the most score per extra dollar. Within each router’s budget, oracle routing scored 8 points above Ramp Router, 17 above Auto, 21 above FireRouter and 27 above Jev. FireRouter’s pool reached 77.2% while spending less than FireRouter did, about 13 points above Claude Opus 5.5 on its own, and Jev’s pool, the largest, reached 82.8%.

Methodology

We ran each router on all eight Vals Index benchmarks: Finance Agent, the Excel Modeling Benchmark (EMB), Terminal-Bench 4.0, Vibe Code Bench, Code Migration, Legal Research, Harvey’s Legal Agent Benchmark (HLAB) and Tax Agent Bench. Each router was connected to the Index’s existing agents through the same interface as an individual model, and every task was graded and scored as on the Vals Index v2.2, with the same number of runs per benchmark and the Index’s weights. Single-model results are the Index runs of every model with results on all eight benchmarks. The agents decided when to submit, under the same limits as the Index’s model runs.

Each router ran with its default model pool, preferences for quality and cost, and model settings: OpenRouter Auto at its default low cost tier, which favors inexpensive capable models, FireRouter at its default balanced setting, and Ramp Router with cost-efficient routing. Jev was accessed through OpenRouter. These are evaluations of the routers as shipped, including their settings, rather than isolated tests of their model-selection decisions.

Anthropic’s content filter sometimes refuses a request. The routers had no backup model when that happened, so every score here, for routers and single models alike, counts a refused task as a failure. We also did not include fallbacks for single models, which is why some scores may differ from the Vals Index.

Ramp Router’s costs are what Ramp billed, which came to 10% above list price for its OpenAI models, and Fugu Max’s are its flat token price, which covers its internal model calls.