Vals-Smith

Turn Your GitHub Repository into
A Custom Benchmark

Are you considering different models for your project? Generate a custom evaluation suite from your codebase and run the models and coding agents of your choice against it. Bring a public or private repo and see how the frontier performs on your code.

Your data stays private No other services can access your data
Join the Waitlist Read our data policy

How Vals-Smith works

Step 1

Connect your repo

Sign in with GitHub and pick any public or private repository. Vals reads your codebase to map its structure, languages, and test surface — your code never leaves your control.

Step 2

Pick models & harnesses

Choose the models and coding agents you want to evaluate and the harness that defines how they work against your repo. Mix frontier models and agents in a single suite.

Step 3

Review your results

Get a private benchmark scored on your own code: pass rates and per-task breakdowns. Compare models side by side and see which one actually performs on your project.

Benchmarks on Public Repository

Methodology

Vals-Smith identifies merged pull requests in your codebase that represent clear, testable engineering work, and converts each one into a self-contained task with an issue statement, hidden tests, and a reproducible setup. Candidates that depend on brittle infrastructure or unclear behavior are rejected. Every task is validated before entering the benchmark: its hidden tests must fail before the original fix and pass after it. During evaluation, each model receives the issue statement and the repository at its starting point, with no access to the original fix. A task is resolved only if the hidden tests pass and existing behavior does not regress. The final score is the percentage of real repository tasks the model can complete.