Background
Web search is a critical information-retrieval tool for a wide range of tasks, from simple fact-checking to in-depth academic research. A wave of search tools built for AI agents has emerged recently, from frontier model providers’ native search tools to independent tools like Exa and Keenable, each with its own pricing and capabilities.
The Vals Web Search Index measures what each tool is worth inside a real agentic workflow rather than on isolated queries: an agent uses it as its only window onto the web while working through expert-written tasks from two established benchmarks, FAB v2 (Finance Agent Benchmark) and Legal Research Benchmark.
To isolate each tool’s contribution, we swap the search tool the model has access to while holding the model and the rest of the harness constant. The difference in performance then reflects the marginal boost the search tool provides.
We also remove the domain-specific search tools the original benchmarks provide (such as SEC EDGAR for financial filings and CourtListener for legal cases), so the general web-search tool is the model’s only source of external information.
We’re eager to evaluate other search API providers. If you’re interested in having your product evaluated, get in touch with us here.
Results
We analyze the cost-accuracy tradeoff through the Pareto efficiency frontier: for a given budget, which model-tool combination achieves the highest accuracy. Cost is the total per task: model inference plus the search fee. Overall, GPT-5.6 Sol with Keenable is the most accurate combination (49.1%), with Claude Fable 5 with Exa close behind (48.5%) at less than half the cost. In finance, external search tools hold most of the frontier; in legal, native search holds all of it, led by Fable 5 (49.0%).
GPT-5.6 Sol with Exa is the cost outlier: the most expensive option in every domain (about $18.97 overall, up to $28.28 on legal) and never on the frontier, driven by a few tasks where the model loops through hundreds of Exa calls before answering.
Model cost dominates every combination; the search fee is a minor part. For GPT-5.6 Sol with Keenable it is $0.41 of the $8.84 mean task cost.
GPT-5.6 Sol with Exa shows high cost variance in both domains; with Keenable the same model is tighter (median $3.22 per task), though its tail still reaches about $190 in finance and $281 in legal.
| Category | Native accuracy (± SE) | Exa accuracy (± SE) | Keenable accuracy (± SE) |
|---|---|---|---|
| General Qualitative Analysis | 57.7% ± 3.1 | 59.5% ± 3.0 | 65.8% ± 6.1 |
| General Quantitative Analysis | 67.5% ± 2.9 | 68.9% ± 2.9 | 73.5% ± 5.6 |
| Market Analysis | 36.0% ± 3.4 | 54.1% ± 3.5 | 58.2% ± 7.0 |
| Comparables | 41.8% ± 3.2 | 43.1% ± 3.2 | 48.9% ± 6.4 |
| Precedents | 23.2% ± 3.0 | 29.0% ± 3.2 | 27.5% ± 6.4 |
| Adjustments | 36.0% ± 3.4 | 44.0% ± 3.5 | 54.4% ± 6.9 |
| Earnings Analysis | 59.4% ± 3.5 | 73.3% ± 3.1 | 75.5% ± 6.1 |
| Disclosure Analysis | 53.6% ± 3.4 | 59.6% ± 3.3 | 62.1% ± 6.7 |
| Financial Modeling | 17.8% ± 2.6 | 17.4% ± 2.5 | 19.5% ± 5.3 |
| Category | Native accuracy (± SE) | Exa accuracy (± SE) | Keenable accuracy (± SE) |
|---|---|---|---|
| Statutory Interpretation | 35.7% ± 2.0 | 37.0% ± 2.0 | 43.3% ± 4.1 |
| Application of Precedent to a Fact Pattern | 34.6% ± 2.1 | 31.6% ± 2.1 | 39.5% ± 4.4 |
| Doctrinal Test Identification | 38.3% ± 3.2 | 36.6% ± 3.2 | 42.1% ± 6.6 |
| Regulatory Framework Interpretation | 48.5% ± 3.9 | 54.1% ± 4.0 | 58.5% ± 7.8 |
| Interaction of Multiple Legal Regimes | 32.4% ± 3.9 | 31.7% ± 3.9 | 35.1% ± 8.0 |
| Cross-Jurisdiction Comparison | 32.5% ± 5.2 | 31.6% ± 5.3 | 42.9% ± 11.1 |
| Timeline / Doctrinal Evolution | 36.9% ± 5.3 | 34.5% ± 5.2 | 38.1% ± 10.9 |
| Validity of Precedent | 37.5% ± 5.4 | 36.3% ± 5.4 | 40.0% ± 11.2 |
| Resolving Conflicting Precedent | 26.7% ± 5.8 | 28.3% ± 5.9 | 46.7% ± 13.3 |
| Consensus vs. Disagreement Across Courts | 25.0% ± 6.1 | 26.0% ± 6.3 | 38.5% ± 14.0 |
The native and Exa boxes pool every model run with that tool; Keenable has only been run with GPT-5.6 Sol, so that model’s Exa runs are shown beside it. Sol searches heavily with either tool, more so with Keenable: a median of 58 calls per finance task vs. 49 with Exa, and 100 per legal task vs. 84.
Across all models, the median run makes slightly more native calls than Exa calls (18 vs. 14 in finance, 16 vs. 12 in legal), but Exa has the long tail: a handful of GPT-5.6 Sol runs exceed 1,000 searches, which lifts Exa’s mean above native’s.
Responses are longer with Exa than with native search (1,590 vs. 1,380 words in legal, 790 vs. 735 in finance) and cite 15 to 18% more distinct sources. GPT-5.6 Sol with Keenable is in the same range: 1,395 words and 9.9 sources per legal answer, 610 words and 4.7 sources per finance answer.
Where do those sources come from? The tables below list the five most-cited domains for each tool, as a share of that tool’s citations.
Finance Analysis
| Rank | Native | Exa | Keenable (GPT-5.6 Sol) | |||
|---|---|---|---|---|---|---|
| 1 | sec.gov | 53.7% | sec.gov | 57.6% | sec.gov | 77.4% |
| 2 | yahoo.com | 4.1% | yahoo.com | 3.9% | yahoo.com | 5.4% |
| 3 | q4cdn.com | 3.9% | q4cdn.com | 2.5% | q4cdn.com | 1.1% |
| 4 | google.com | 2.1% | macrotrends.net | 1.2% | stockanalysis.com | 0.7% |
| 5 | macrotrends.net | 1.3% | fool.com | 1.0% | financecharts.com | 0.6% |
| Top 5 | 65.1% | 66.2% | 85.2% |
Legal Research
| Rank | Native | Exa | Keenable (GPT-5.6 Sol) | |||
|---|---|---|---|---|---|---|
| 1 | justia.com | 25.9% | justia.com | 19.2% | justia.com | 28.7% |
| 2 | cornell.edu | 10.3% | cornell.edu | 11.5% | ecfr.gov | 6.8% |
| 3 | virginia.gov | 6.2% | virginia.gov | 5.9% | virginia.gov | 6.6% |
| 4 | texas.gov | 5.7% | vlex.com | 5.8% | house.gov | 5.9% |
| 5 | findlaw.com | 5.1% | texas.gov | 5.3% | texas.gov | 5.8% |
| Top 5 | 53.1% | 47.8% | 53.9% |
All three tools cite similar, authoritative sources: Justia leads in legal and the SEC dominates finance, most heavily with Keenable, where SEC filings are 77% of citations.
Methodology
Agents run on the same harness across both domains, with the web-search tool (native/Exa/Keenable) as the variable under test. Two things distinguish these runs from the original FAB v2 and Legal Research Benchmark evaluations: the tool is swapped (native vs. Exa vs. Keenable), and the domain-specific search tools those benchmarks provide (SEC EDGAR for finance, CourtListener for legal, and the like) are removed, leaving the web-search tool as the agent’s only means of gathering external information. Runs are orchestrated through Valkyrie with model calls routed through model-library.
Grading
- Legal Research: each task in the Legal Research Benchmark carries a rubric of required elements written by legal experts; rubrics range from 1 to 22 items per task (mean 9.47). All-pass is our primary metric: a task counts as correct only if every rubric item is satisfied. Grading is performed by a single LLM judge (GPT-5.4).
- Finance Analysis: tasks use FAB v2’s dealbreaker-gated Partial Credit as the primary metric, category-balanced across the dataset’s nine question categories. A subset of checks in each question is flagged as dealbreakers—load-bearing facts or numbers required for a satisfactory answer—and failing any dealbreaker means the response scores 0% for that question regardless of everything else. All-pass (100% only if every check passes) is reported as a secondary metric. Grading uses the same three-judge jury as FAB v2: GPT-5.4, Gemini-3.1-Pro, and Claude Sonnet 4.6.
For the full rubrics, jury setup, and scoring rules behind any specific grading decision, refer to the original FAB v2 and Legal Research Benchmark methodologies.
Dataset
- Legal: 208 expert-authored legal research tasks, each tagged by the reasoning skill it tests (a task can test more than one): Statutory Interpretation, Application of Precedent to a Fact Pattern, Doctrinal Test Identification, Regulatory Framework Interpretation, Interaction of Multiple Legal Regimes, Cross-Jurisdiction Comparison, Timeline/Doctrinal Evolution, Validity of Precedent, Resolving Conflicting Precedent, and Consensus vs. Disagreement Across Courts.
- Finance: 450 finance-analyst questions across nine categories, scored with FAB v2’s category-balanced methodology and reflecting real equity-research workflows: General Qualitative Analysis, General Quantitative Analysis, Market Analysis, Comparables, Precedents, Adjustments, Earnings Analysis, Disclosure Analysis, and Financial Modeling.
Statistical Analysis
To establish whether the two search tools differ statistically, we fit a mixed-effects model per domain, with search tool as a fixed effect (native as the reference level) and a random intercept per task absorbing task-level difficulty: a logistic model for legal’s All-pass metric (all_pass ~ search_tool + (1 | task_id)) and a linear model for finance’s primary metric (score ~ search_tool + (1 | task_id)). Each model pools the per-task, per-model observations underlying the Results charts above.
The fixed effect of search tool (Exa relative to native) in each model:
| Domain | Outcome (model) | Exa effect | Std. error | Statistic | p |
|---|---|---|---|---|---|
| Finance | Weighted score, 0-100 (linear) | +6.51 points | 0.98 | t = 6.62 | < 0.001 |
| Legal | All-pass, 0/1 (logistic) | −0.02 log-odds | 0.13 | z = −0.17 | 0.86 |