Routers Are a New AI Primitive

Partners

Every few months, frontier models unlock harder problems and new domains. No company can afford to sit that out, but few can afford frontier prices on every request. Which model should handle each piece of work, and when does the performance of frontier models justify their cost?

At Vals, we evaluate models from across the major labs on their ability to carry out complex work across domains such as finance, software engineering, law, and tax. The Vals Index brings these evaluations together to measure AI’s potential economic impact. From that vantage point, the frontier is jagged: five different models hold the top spots on the Index’s eight benchmarks, and no model ranks in the top three on all of them. Choosing only one model means compromising performance on some tasks.

The range of models to choose from is continuously and rapidly expanding. This past month alone, ten developers released 16 language models tracked by Vals. Each release raises another set of decisions for companies: which workloads would benefit, whether the improvement is worth the cost, and whether it should replace a model already in production. Sticking with one model can mean missing better options; maintaining separate model choices across many workflows is an operational burden requiring continuous engineering and evaluation effort.

Routers have emerged as a new primitive of the AI stack, promising to manage that complexity and help companies benefit from new models without revisiting model selection across every application themselves. The value is both better use of today’s models and a way to keep benefiting from progress across labs.

Every router shares the same fundamental design. It sits in front of a pool of models and, for each request, decides which one should answer, right-sizing the intelligence to the task by weighing quality, cost, and latency. Where routers differ is in how they decide. Some judge the prompt itself, such as the type of task or how hard it looks. Others draw on external signals like benchmark scores or usage statistics. Increasingly, routers make this call at every step of an agent’s work rather than once per task.

An ensemble of models, used well, can beat any single model on the frontier. To measure by how much, we calculated the performance of an oracle router, one that picks the perfect model for every task in hindsight. On the Vals Index, that oracle would beat the best single model by 14 points while costing about 40% less.

Perfect hindsight is impossible, of course, but it shows the full potential of choosing the right model for each task. The real question is how much of that potential today’s routers capture in practice, and existing benchmarks weren’t built to answer it. Most academic router benchmarks score a single choice on a single public question, often from datasets frontier models have long since saturated. When every model in the pool gets the answer right, a router that always picks the cheapest one looks perfect. But routers today are asked to do something much harder, choosing a model at every step of agent work that can run for hours.

Router providers have begun publishing serious evaluations of their own, and we think the natural next step is an independent comparison, with every router run through the same harness on private, long-horizon tasks across real professional domains. Vals has always set out to chart the frontier of AI performance, which is why we’re now measuring how today’s routers compare with single models on quality and cost, with the task-level oracle as a reference point.

From isolated prompts to agent workflows

Academic benchmarks have established a foundation for evaluating model routers. RouterBench (Hu et al., 2024) helped establish an early framework using precomputed model outcomes. RouterArena (Lu et al., 2025) broadened comparisons across domains and difficulty levels, while LLMRouterBench (Li et al., 2026) brought routing methods into a shared framework that includes tool-use and software tasks. More recently, TwinRouterBench (Yang et al., 2026) introduced live coding-agent evaluation, selecting a model at every call and measuring task success and actual API spend.

Today’s agents complete complex work over dozens of steps, where each routing decision affects the context and choices that follow. A cheaper model may save money on one call but introduce an error that takes several more calls to recover from, while a weak routing decision can derail the rest of the task. Predicting how a task will decompose into subtasks is another challenge, and any decision to switch models must account for the cost of losing the existing prompt cache. For long-running sessions, switching to a cheaper model does not automatically bring cost savings.

Isolated prompts miss these consequences, and TwinRouterBench’s live evaluation remains focused on coding. Difficulty matters too: Stanford’s 2025 AI Index identifies benchmarks such as MMLU and GSM8K as saturated, making them less revealing of the accuracy sacrificed for cheaper models. Newer benchmarks address parts of this gap, but public tasks still carry a risk of prior exposure.

The next step is to bring these requirements together: challenging, realistic work across professional domains, live execution that exposes the consequences of each routing decision, and private test data to reduce contamination risk.

What router providers are already measuring

Router providers are also evaluating how their systems perform in practice. OpenRouter compares its own and third-party routers against single models on quality, time, and cost.

The way an agent session unfolds affects its cost. Fireworks evaluated FireRouter through randomized A/B tests on live sessions, accounting for the cost of lost prompt caching when switching models. Not Diamond’s coding-agent evaluation methodology uses simulated users to model long sessions and pauses between user interactions, capturing the costs of growing context and cache expiration.

Benchmarks also shape routing decisions themselves. Ramp’s Benchmark Routing strategy lets customers select and weight benchmarks, then routes requests to the model with the highest weighted score. The relevance of those benchmarks to the customer’s work therefore matters to the routing decision.

These efforts provide useful evidence about router performance. An independent comparison can build on them by running router products through the same harness on shared, private tasks across professional domains, with each product’s configuration documented.

Evaluating routers on the Vals Index

The Vals Index was built to measure the impact of AI on real-world agentic tasks—the very things routers are now aiming to optimize.

The Vals Index is agentic and long-horizon. All eight components evaluate tool-using agents spanning finance, coding, legal, and tax work: Finance Agent v2, the Excel Modeling Benchmark, Terminal-Bench 4.0, Vibe Code Bench, Code Migration, Legal Research Bench, Harvey’s Legal Agent Benchmark (HLAB), and Tax Agent Bench. Many of these tasks are long: the median Terminal-Bench 4.0 task is estimated at four hours of expert work, each Excel Modeling task takes an expert five hours or more, and on Legal Research Bench, some models work for upwards of an hour on certain tasks.

The Index is expert-written and mostly private. Six of its eight benchmarks are private, and their tasks, answers, and grading rubrics are written by people who do this work professionally: financial analysts for the finance benchmarks, practicing lawyers for Legal Research Bench, and tax professionals for Tax Agent Bench. Coding tasks are graded by running tests against what the agent actually built. Our private test set has never been released, and our evaluations are run with ZDR (Zero Data Retention) policies in place, meaning that model providers cannot retain or train on our prompts.

Importantly, no single model leads every component. The model that tops Harvey’s Legal Agent Benchmark ranks 40th on Terminal-Bench 4.0, different models lead other components, and of the top two models, the second costs about 36% more per task than the first. This kind of spread is the opportunity routing can exploit.

Finally, the Index is continuously updated. We regularly evaluate frontier models on it with consistent harnesses, and retire components as they saturate. This means every router can be compared against single-model baselines run in the same harness, along with an oracle that keeps pace with the models being routed between.

Choosing a model to answer a trivia question is a very different problem from choosing a model for every step of a corporate tax research task, and we think router evaluations should reflect that. Concretely, good router evaluations put each router through live, end-to-end agent runs. The router takes the place of a model in the same harnesses and is consulted on every model call, allowing it to retain the current model or select another as the task unfolds. Prompts, tools, and graders stay fixed. Each router runs as shipped, with its default model pool and settings, and that configuration is documented for every run. Task outcomes, cost per task, cache usage, and completion time are compared against baselines run in the same harness (the best and the cheapest single model in the router’s own pool), with the task-level oracle as a reference point.

Beyond picking a single model

If routers are judged primarily on isolated public questions, evaluations will reward a narrower set of capabilities than production agentic work requires. As routing increasingly determines what AI work costs, we believe it deserves to be measured on that work, with the same care as the models themselves.

That is what our router evaluation is designed to do, and we will publish results on a public leaderboard soon. If you build a router and are interested in having it evaluated, get in touch with us here.

References

Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., & Upadhyay, S. K. (2024). RouterBench: A benchmark for multi-LLM routing system. arXiv:2403.12031. https://arxiv.org/abs/2403.12031

Li, H., Zhang, Y., Guo, Z., Wang, C., Tang, S., Zhang, Q., Chen, Y., Qi, B., Ye, P., Bai, L., Wang, Z., & Hu, S. (2026). LLMRouterBench: A massive benchmark and unified framework for LLM routing. arXiv:2601.07206. https://arxiv.org/abs/2601.07206

Lu, Y., Liu, R., Yuan, J., Cui, X., Zhang, S., Liu, H., & Xing, J. (2025). RouterArena: An open platform for comprehensive comparison of LLM routers. arXiv:2510.00202. https://arxiv.org/abs/2510.00202

Stanford HAI. (2025). AI Index Report 2025: Technical performance. https://hai.stanford.edu/ai-index/2025-ai-index-report/technical-performance

Yang, P., Chen, W., Yang, T., Feng, P., Xing, J., Guo, W., Yao, Y., Han, Y., Li, H., Wang, X., Gao, Y., Wang, Z., Xiao, J., Yang, A., Tian, L., Ai, L., Yang, E., & Shi, T. (2026). TwinRouterBench: Fast static and live dynamic evaluation for realistic agentic LLM routing. arXiv:2605.18859. https://arxiv.org/abs/2605.18859