The most successful Fireworks customers have one thing in common: they deeply understand their own work. They know what good output looks like, where the hard cases are, and which models actually deliver on them. That understanding is how they protect and grow their competitive advantage, ship the best product, and get the most out of every dollar they spend on AI.
Evals turn that understanding into an advantage you can use. They capture what your best people know (the difference between an answer that’s good enough and one that’s right) and make it something you can measure. Once it’s measurable, you can act on it: pick the right model, train one that does your work better than anything off the shelf, and cut costs without cutting quality.
Vals partners with Fireworks customers to build and maintain those evals, so they keep pace with your work and with every new model.
Why you need custom evals
At Vals, we think of evals in three tranches: global, domain specific, and custom.
Global evaluations, like our Vals Index, capture the ability of a model to broadly do work across our economy. They help us understand the transformative impact across our society - but aren’t sufficient for making informed decisions about which model you should use for a specific task.
Domain specific benchmarks - like our Finance Agent benchmark - aim to assess the jagged frontier of model ability across differing tasks. The best model on our composite index fails to place in the top three for Excel modeling, legal research, and tax work.
And the right model for a domain isn’t the right model for your company or your tasks. We build best-in-class coding benchmarks - but our biggest efficiency improvements came only after we built custom benchmarks based on the exact work our engineers do every day. We’re proud of the savings we’ve delivered to our customers, and are excited to partner with Fireworks to unlock these optimizations for more companies.
What your custom evals unlock
Once you can measure your own work, Fireworks gives you several ways to improve it.
Train a model that’s built for your work. Many companies run a large, expensive workload on a closed frontier model, even though the job is narrow and well defined. That’s often exactly where a smaller open model, trained on your own data with Fireworks RFT, can match or beat the frontier at a fraction of the cost. RFT trains against a grader that scores the model’s outputs, which means the quality of your eval sets the ceiling on what training can achieve. A good eval gives training the right target: the specific behaviors that matter for your work, graded the way your experts would grade them. It also gives you proof, so you can show the trained model is better before it reaches production, and keep checking that it stays better as new models come out.
Use the right-sized model for the task. The most expensive model isn’t always the best one for the job. On our own benchmarks, open models can often compete with the leaders for a small fraction of the cost. Muse Spark 1.3 Max scores 55.29% on Legal Research Bench for an average cost of $0.59. Fable 5.1 achieves the same score at a price of $23.06 per task. Only your evals can tell you when parity exists on the work you care about.
Route each request to the right model. For agentic coding, Fireworks Nexus with FireRouter sends the hard requests to a stronger model and the routine ones to faster, cheaper open models. Evals let you configure the router so it sends tasks to the right model, and let you measure the return you’re seeing.
Start with an eval
We’re proud that Fireworks trusts Vals to work alongside its customers, and excited to be part of its Specialized Intelligence Ecosystem. Fireworks gives companies the tools to specialize AI for their own work. Vals helps them prove what works, so they can adopt it with confidence.
If you’re deciding which model to run, whether to train your own, or how to cut what you spend on AI, start with an eval. Talk to your Fireworks team or reach out to us at partnerships@vals.ai, and we’ll help you measure the work that matters most to your business.