Background
Most coding benchmarks stop at the first working version. Vibe Code Bench 1-100 measures what comes after: whether a model can evolve a working web application through a long sequence of dependent product requests without breaking what already works.
It picks up where Vibe Code Bench (VCB) leaves off. VCB measures zero-to-one generation, starting from scratch and producing a working web application. VCB 1-100 begins with a complete, fully passing web application from Vibe Code Bench v1.1 and asks the model to keep going.
Each task hands the model an ordered sequence of realistic requests without specific technical details, the kind a PM sends an engineer or a vibe coder types at an agent. Later requests depend on earlier features, data models, permissions and user journeys. The model keeps both its conversation and its code across the whole task. Every iteration has to add the new behavior while preserving all the behaviors that came before it, and the first incomplete iteration ends the attempt.
This matters because most real software work is not greenfield. It is shipping the next feature, migrating a schema without losing data, applying a policy change consistently across every surface of an app. It is the skill no one-shot benchmark touches, and the one that decides whether coding agents can be trusted with production codebases.
Overview
Long-horizon and realistic product requests
VCB v1.1 measures building an application from a specification. VCB 1–100 starts with a working application and asks the agent to extend it through up to ten dependent requests.
The requests describe product behavior that needs to be changed in plain English, similar to how developers prompt while vibe coding, without any technical jargon. The agent must locate the relevant code, understand existing systems, and choose an implementation of the feature on its own. Later turns retain the code, data, and conversation from earlier work. A short request can require substantial engineering, as compatibility with previously added features needs to be preserved.
Characters. Scale: 0–3,310. Select a row for details.
Million tokens. Scale: 0–60. Select a row for details.
The prompt chart measures the starting-app specifications and follow-up requests in the v1.1 test definitions. The token comparison uses the 8 models shared by the two v1.1 leaderboards, with equal weight per model. Input includes cached tokens and output includes reasoning, counted once. Token coverage is 398 model-task runs for zero-to-one VCB and 400 for VCB 1–100.
Tool calls. Scale: 0–190. Select a row for details.
File-editor calls include both reads and edits through the editor tool; they exclude changes made through terminal commands. Both bars show the median across 400 model-task runs: the same 8 models, with 50 tasks each. These are tool-call counts, not counts of changed files.
Regression tests ensure that the app stays usable under development
Each iteration includes tests for the new request and selected regression tests for earlier behavior that must remain active. Both sets of tests run against the same submitted candidate app. If a model successfully adds a new feature, but breaks the existing app features in the process, we still consider that a failure.
Share of 6022 v1.1 test workflows (%). Scale: 0–100. Select a row for details.
The v1.1 workflow definitions contain 6022 workflows across 456 requests in 50 test scenarios. Of these, 1572 (26.1%) check earlier behavior, and 4450 cover the new request. These figures exclude the separate validation set.
The second chart view shows the share of regression workflows in each iteration’s suite. It does not measure how much earlier functionality is covered: regression checks may cover base-app behavior as well as previous requests.
Methodology
A sequence of changes to one application
Each task starts with a working Vibe Code Bench v1.1 application and its saved app state. The model receives up to ten product requests, one at a time. It edits the same codebase throughout and retains its conversation and tool history. Later requests build on earlier features and decisions; they are not independent coding tasks.
After each change, a browser agent powered by GPT-6 Luna checks the new request and selected regression workflows against the submitted application. We then review failed verdicts using Claude Sonnet 5.5 to verify that they reflect an implementation error in the application. The score counts the consecutive passing prefix under this policy.
All models use the same modified OpenHands V1 environment, with terminal access and preconfigured Supabase, Stripe test services, and MailHog. Each task has one ten-hour generation budget shared across its iterations. Evaluation time is excluded.
Task construction
Each task consists of a working starting application, an ordered sequence of product requests, and browser-based acceptance tests for each iteration.
Requests build on the product state left by earlier changes. They specify the desired behavior and preservation constraints without prescribing an implementation. The sequence covers five change types and contains up to ten iterations.
Each iteration includes new feature tests and selected regression tests. The tests assess observable behavior rather than require a particular internal code structure. The dataset contains 50 test set scenarios and 50 separate validation scenarios. All model scores reported on this page are based only on the test subset.
Five kinds of software change
In software, the tasks cover many different types of changes that will be needed in producing and iterating on a real application.
Our feature requests are categorized into five types of changes, listed below with examples.
| Change type | What the agent must do | Example |
|---|---|---|
| Additive | Add a capability while keeping existing flows working. | Add a private Saved area for posts, with notes. |
| Cross-cutting | Apply one rule across related features and operations. | Add opt-in, read-only collection sharing across the feed, profiles, and shared links. |
| Schema migration | Change the data model without losing identity or history. | Organize saved posts into collections without losing any saved post or note. |
| Bug fix | Diagnose a reported symptom and preserve valid nearby behavior. | Make sure revoked share URLs stay unavailable after new share URLs are generated. |
| Revert | Remove a feature while retaining explicitly preserved behavior. | Remove public sharing while keeping collections, memberships, and notes. |
Share of 456 requests (%). Scale: 0–100. Select a row for details.
This distribution uses all 456 planned requests in the v1.1 test contracts. Pass rates by change type below use the evaluated requests each model reached.
Cross-cutting changes form the largest group. Success requires applying a rule consistently across the product, not only adding a screen or endpoint. Migrations and reverts test a different skill: changing or removing functionality while retaining the data and behavior that must survive.
Quality Assurance
Tasks undergo expert review for clarity, realism and consistency. The experts also review the iteration checks to ensure that the expected behavior in the checks follows from the request and that all reasonable implementations can satisfy the tests.
Each workflow must have reproducible setup, actions a browser can perform, and a clear result.
The review checks four things:
- Prompt/test agreement: Every tested expectation is supported by the request, and each core requirement has a corresponding check.
- Executable workflows: The steps can be performed in a browser, with enough setup information and a clear yes-or-no result.
- Realistic scope: The request gives enough information for a useful implementation without prescribing an unnecessary technology stack.
- Preservation and edge cases: The requested change does not contradict earlier live behavior, and important unhappy paths are covered.
Task review and execution checks serve different purposes: one checks what should be tested; the other checks that the application and test can run. Missing or incomplete observations are not treated as proof of an implementation defect.
Example task: Zeeter
Zeeter is a small social application with accounts, posts, profiles, likes, and follows. This standalone example is a separate task, not part of the test or validation sets.
| Iteration | Product request | Behavior that must survive |
|---|---|---|
| 1 | Add private saved posts and notes. | Existing posts, likes, follows, and account privacy. |
| 2 | Organize saved posts into collections. | Saved records and private notes. |
| 3 | Add opt-in, read-only collection sharing. | Collection membership and note privacy. |
| 4 | Keep revoked links expired after sharing resumes. | Access to independently shared collections. |
| 5 | Remove public sharing without losing data. | Private collections, memberships, and notes. |
| 6 | Reflect source edits and show deleted post placeholders. | Saved references and the removal of public sharing. |
Example request — iteration 1
Add a private Saved area where signed-in members can save and unsave posts from the feed, profiles, and post detail, then find their saved posts without changing likes or follows. Each saved post can have a private note of up to 500 characters; repeated saves must produce one entry, and saved posts must show the current source text after edits. Guests and other members must not see a member’s Saved area or notes.
Example acceptance workflow — save from the feed
1. Open Zeeter. Sign out if needed. Create username 'zt1w1owner', email 'zt1w1owner@example.com', password 'ZeeterPass123!'. Complete or skip profile setup to reach the home feed.
2. For this workflow only, the separate reader account is username 'zt1w1reader', email 'zt1w1reader@example.com', password 'ZeeterPass123!'. Create it only when instructed; later sign-ins use these same credentials. The original author is 'zt1w1owner' with password 'ZeeterPass123!'.
3. Publish a post with text 'zt1w1alpha source post'. Save it from the home feed.
4. Open Saved and confirm exactly one entry for 'zt1w1alpha source post'. Reload and confirm it remains.In the example below, this workflow passed all four steps. Other workflows check note privacy, repeated saves, and existing interactions. Later regression checks include a previously shared URL: removing its button is not enough if guests can still open the URL.
Note that in real tasks from the test set, the workflows are longer and test more complicated aspects of the changes, such as interactions between edge cases.
Example generated output
The image below shows the Zeeter starting app during a browser replay.
The evaluation video below shows this save-from-feed workflow after iteration 1.
Browser evaluation and failure review
After each generation step, a browser agent executes acceptance workflows for the new request and selected regression workflows for earlier behavior. A failure review checks the browser agent’s findings against the task requirements and the available testing evidence.
Sonnet reviews the full recorded action timeline and can request additional recorded screenshots during review. A reviewed iteration fails when the review confirms at least one implementation issue. Once an issue is confirmed, further reviews may be skipped because the iteration has already failed. We also can fail if an app errors out, causing grading to fail before workflows complete.
An inconclusive verdict identifies a specific gap in testing evidence without a confirmed implementation error; completed inconclusive reviews receive policy passes. Missing or unavailable reviews remain evaluation errors rather than evidence of model failure.
Scoring and continuation
Passing a non-final iteration continues generation on the next request while generation time remains. A task ends when it completes all requests, reaches a confirmed failed iteration, exhausts its generation budget, or encounters an execution error.
A task’s score is the number of consecutive passing iterations divided by its number of planned iterations. The leaderboard score is the mean of those fractions across all 50 test scenarios, expressed as a percentage. Tasks that end in errors retain credit for their verified passing prefix; they are not removed from the denominator.
The test and validation sets are separate. Validation results and historical v1 scores are not mixed into this leaderboard.
Costs are calculated from per-request token usage at the configured model prices, including cache usage, reasoning, compaction, and any fallback model calls. They cover generation work across all 50 tasks, including unsuccessful attempts, and exclude evaluator costs. Costs can be recalculated when prices change.
Results
How many iterations each model completes
Each bar partitions the 50 test tasks by credited iterations. “Full arc” means all requests planned for that task passed; the number at right is the benchmark score, which averages each task’s completed fraction.
Performance by change type
Evaluated iterations passing under the review policy (%). Scale: 0–100. Select a row for details.
Pass rates use evaluated iterations reached by each model under the v1.1 review policy. Models reach different requests depending on their earlier results, so the denominators differ and these are not matched-request comparisons.
Failure to implement vs. breaking what already works
Implementing the new request is only one part of maintenance. A change often damages behavior that worked before, even when the new feature is implemented correctly.
Share of recorded workflow-failure stops (%). Scale: 0–100. Select a row for details.
The chart groups recorded workflow-failure stops by whether they involve new-request checks, regression checks, or both. Generation timeouts and execution errors are excluded from this breakdown.
How models use their tools
Tool calls are grouped into database operations, browser interactions, file editing, agent control, and terminal commands. The chart uses the v1.1 test runs and shows mean calls per evaluated iteration.
Browser automation invoked through the terminal is counted as terminal activity, so native browser-call counts do not capture every way an agent can inspect the app.
Refusals and fallbacks
| Model | Tasks with refusals or filters | Of these: provider filters | Tasks with fallback |
|---|---|---|---|
| Claude Opus 5.5 | 0 / 50 | 0 | 12 / 50 |
Counts come from the v1.1 runs, including recorded work beyond the credited prefix. No model had a refusal. Only Claude Opus 5.5 used fallbacks: 12 of 50 tasks, 16 fallback events, with Claude Opus 4.8 or Claude Opus 5 taking over the request. The other seven models had no refusals and no fallbacks, so they do not appear in the table. Opus 5.5’s published score includes this fallback work; capped at the last iteration completed before each task’s first fallback, its score is 23.55%. Costs retain the actual recorded generation usage, priced at the model that served each request.
Qualitative analysis
Failure modes
Each row describes one observed workflow issue. These are illustrative cases, not a frequency ranking of failure causes.
| Workflow | Failure pattern | Confirmed observation |
|---|---|---|
| New request | Existing booking path blocked | The existing default booking path offered no times unless a custom service was selected. |
| New request | Access policy prevents a valid write | Saving a correctly completed staff form failed with an infinite-recursion error in the staff_members access policy. |
| New request | Scheduled transition does not occur | A chapter remained Scheduled after its publication date had elapsed and the creator revisited and reloaded the relevant pages. |
| Regression | Account deletion erases historical records | Deleting a member account removed its historical RSVP entirely instead of preserving the record with the member’s name removed. |
Multi-iteration trajectories across models
The chart follows Morning Routine Tracker in the v1.1 test runs, selected to show several iterations for every model. Vertical boundaries separate requests, and segment widths show recorded tool calls, including retries, rather than elapsed time. A model’s row ends at its terminal request. “No verdict” indicates recorded generation without a completed evaluation.
Changes from v1
In v1.1, we changed iteration prompts to make requirements less ambiguous. We also audited task workflows to check that each assertion follows from the task request. We added a step to the grading process in which a second judge model, Claude Sonnet 5.5, reviews browser-agent failure verdicts against the recorded evidence. Reviewed failures require a confirmed implementation issue; inconclusive reviews receive policy passes. App-start failures can end an iteration before this review.
Citation
Please cite this work as:
Citation (BibTeX)
@article{valsai2026vibecodebench1100v11,
title = {Vibe Code Bench 1-100 v1.1},
author = {Hung Tran and Dilhan Salgado and Langston Nashold and Orestes Kassian and Rayan Krishnan},
year = {2026},
month = nov,
howpublished = {Vals AI},
url = {https://vals.ai/benchmarks/vcb-1-100-v1-1},
}