Proprietary

Vibe Code Bench

Updated 11/18/2026

Can models build web applications from scratch?

As of November 18, 2026, Claude Fable 5.2 (OpenHands) ranks first on Vibe Code Bench with 93.77%, followed by Gemini 4 Neon (OpenHands) at 93.70% and Claude Sonnet 5.5 (OpenHands) at 92.39%.

Vibe Code Bench v1.1Building web apps from scratch
Vibe Code Bench v1.1•Nov 18, 2026
ACCURACY
Lab:
Pinch or Ctrl/Cmd + scroll to zoom. Once zoomed, scroll or swipe with two fingers to pan, or drag to pan. Shift + drag draws a box to zoom into; turn on box zoom to draw one without Shift. Plus and minus keys zoom, arrow keys pan, 0 resets.

Vibe Code Bench leaderboard

Rank Model Harness Score Cost / Test Tokens / Test Duration / Test
1 Claude Fable 5.2 OpenHands 93.77% $22.13 34.71m 40m37s
2 Gemini 4 Neon OpenHands 93.70% $6.76 18.46m 23m38s
3 Claude Sonnet 5.5 OpenHands 92.39% $19.73 116.97m 1h01m
4 Gemini 4 Argon OpenHands 91.91% $8.11 9.15m 37m28s
5 Claude Haiku 5.5 OpenHands 90.44% $6.07 92.49m 34m36s
6 Claude Fable 5 OpenHands 90.35% $41.71 - 1h02m
7 Claude Opus 5.5 OpenHands 90.29% $35.77 112.33m 1h36m
8 Claude Fable 5.1 OpenHands 90.26% $33.37 57.39m 57m40s
9 GPT-6.2 Astra OpenHands 89.90% $7.92 7.11m 44m12s
10 GPT-6 Astra OpenHands 89.59% $38.51 16.43m 39m04s
11 Gemini 4 Carbon OpenHands 89.06% $8.25 16.86m 36m12s
12 GPT-6.1 Sol OpenHands 88.93% $6.23 21.67m 1h15m
13 Claude Opus 5 OpenHands 88.40% $33.88 50.67m 1h28m
14 GPT-6 Sol OpenHands 87.82% $26.36 63.98m 38m56s
15 Gemini 4 Hydrogen OpenHands 87.78% $4.55 17.30m 14m25s
16 DeepSeek V4.1 Flash 1017 OpenHands 87.73% $0.58 43.65m 16m14s
17 GPT-6.2 Sol OpenHands 87.62% $3.23 13.91m 47m51s
18 Qwen 4 Max OpenHands 86.66% $41.40 31.21m 1h10m
19 Grok 4.7 OpenHands 86.17% $15.83 15.50m 35m47s
20 Wonder Full OpenHands 86.00% $2.29 4.50m 10m48s
21 Muse Spark 1.3 Max OpenHands 85.86% $2.54 13.10m 16m18s
22 MiMo V2.6 Pro OpenHands 85.22% $1.04 21.68m 1h01m
23 Kimi K3 OpenHands 84.97% $10.01 25.33m 16m39s
24 Kimi K3.1 Pro OpenHands 84.89% $7.49 28.14m 50m29s
25 DeepSeek V4.1 Flash OpenHands 84.74% $0.41 28.73m 15m28s
26 GPT-6.1 Terra OpenHands 84.00% $6.00 49.21m 37m30s
27 Ember-1 OpenHands 83.87% $8.56 19.72m 39m18s
28 Muse Ember OpenHands 83.70% $0.05 5.40m 2m48s
29 Wonder OpenHands 83.09% $1.80 2.93m 5m50s
30 Muse Spark 1.3 OpenHands 82.86% $2.10 8.09m 12m54s
31 Claude Opus 4.8 OpenHands 82.72% $26.88 - 1h16m
32 DeepSeek V4 Pro 0813 OpenHands 82.30% $0.36 36.76m 1h06m
33 GPT-6 Luna OpenHands 81.65% $1.35 63.91m 37m00s
34 Mistral Large 4 OpenHands 81.56% $11.72 24.44m 50m13s
35 Inkling 2 Small OpenHands 81.55% $1.90 51.08m 25m37s
36 Claude Sonnet 5 OpenHands 81.33% $25.39 - 1h07m
37 Hy4 OpenHands 81.01% $2.39 34.76m 38m01s
38 GPT-5.6 Sol OpenHands 80.50% $33.40 - 33m53s
39 Muse Spark 1.2 OpenHands 79.10% $1.53 6.90m 19m55s
40 MiMo V2.6 Flash OpenHands 78.96% $0.56 27.49m 50m13s
41 Gemini 3.8 Flash OpenHands 78.65% $6.87 23.11m 8m39s
42 Mistral Large 4 Preview OpenHands 78.40% $20.28 36.63m 2h53m
43 GLM 5.3 OpenHands 78.13% $12.45 39.59m 1h04m
44 Claude Opus 4.8 Claude Code 77.48% $6.86 8.20m 19m02s
45 Hy4 Preview OpenHands 77.48% $2.18 33.10m 41m33s
46 Qwen 3.8 Max 0902 OpenHands 77.14% $11.67 38.08m 1h25m
47 GPT-5.6 Luna OpenHands 77.06% $0.73 1.16m 25m52s
48 Grok 4.6 OpenHands 76.24% $4.88 7.98m 25m30s
49 Muse Spark Open OpenHands 76.19% $1.66 4.56m 31m09s
50 Hy4 Flash OpenHands 74.84% $0.12 19.95m 16m48s
51 DeepSeek V4 Flash 0731 OpenHands 74.74% $0.91 43.22m 37m22s
52 GPT-5.6 Terra OpenHands 74.59% $7.89 23.94m 19m12s
53 MiniMax-M3.1-Flash OpenHands 73.94% $1.15 66.86m 21m04s
54 Muse Spark 1.1 OpenHands 72.16% $0.92 - 10m15s
55 Claude Opus 4.7 OpenHands 71.00% $21.41 - 35m42s
56 Step 5 Preview OpenHands 70.40% $5.86 50.26m 1h31m
57 Gemini 3.7 Flash OpenHands 70.39% $4.83 18.35m 23m43s
58 GPT 5.5 OpenHands 69.85% $16.66 1.91m 31m52s
59 Grok 4.5 OpenHands 69.00% $4.10 - 13m50s
60 GPT 5.4 (xhigh) OpenHands 67.42% $10.69 1.92m 1h26m
61 GPT 5.5 Factory 67.39% $5.93 6.89m 19m37s
62 Beam 1.0 OpenHands 66.44% - 8.65m 2m25s
63 Qwen 3.8 27B OpenHands 64.85% $23.46 46.03m 43m39s
64 Qwen 3.8 Max OpenHands 64.70% $8.24 19.46m 2h49m
65 Gemini 3.6 Flash OpenHands 64.00% $3.04 8.48m 25m23s
66 GLM 5.2 OpenHands 63.96% $8.49 - 1h02m
67 GPT 5.3 Codex OpenHands 61.77% $11.91 3.93m 1h16m
68 Mercury 3 OpenHands 60.93% $0.63 28.57m 9m27s
69 GPT 5.5 Codex 58.19% $3.63 3.40m 11m49s
70 Claude Opus 4.6 (Nonthinking) OpenHands 57.57% $8.69 - 21m19s
71 Claude Sonnet 4.6 Claude Code 55.77% $7.35 17.94m 49m42s
72 GPT 5.2 OpenHands 53.50% $17.75 6.14m 1h23m
73 Claude Opus 4.6 (Thinking) OpenHands 53.50% $8.28 - 23m06s
74 Claude Sonnet 4.6 OpenHands 51.48% $5.91 - 26m12s
75 DeepSeek V4 OpenHands 49.93% $2.21 - 56m59s
76 Composer 2.5 Cursor CLI 49.61% $0.50 1.93m 9m53s
77 Gemini 3.5 Flash OpenHands 48.68% $2.54 1.37m 14m53s
78 GPT 5.4 Codex 48.47% $7.51 8.64m 17m55s
79 GPT 5.4 Mini OpenHands 47.97% $1.19 - 34m15s
80 Qwen 3.7 Max OpenHands 47.67% $11.40 3.99m 1h18m
81 MiniMax-M3 OpenHands 47.57% $6.45 - 1h25m
82 Kimi K2.7 Code OpenHands 47.21% $3.84 - 2h52m
83 Qwen 3.7 Plus OpenHands 46.39% $1.08 - 37m12s
84 MiMo V2.5 OpenHands 42.17% $0.07 - 26m20s
85 GPT 5.2 Codex OpenHands 37.91% $4.15 857.70k 32m15s
86 Kimi K2.6 OpenHands 37.89% $1.93 - 49m28s
87 Gemini 3.5 Flash Lite OpenHands 37.16% $0.83 7.82m 8m22s
88 MiMo V2.5 Pro OpenHands 34.11% $0.17 - 40m04s
89 Gemini 3.1 Pro Preview (02/26) OpenHands 32.03% $3.83 - 20m12s
90 GLM 5.1 OpenHands 31.46% $2.89 - 33m35s
91 GLM 5.3 Flash OpenHands 30.76% $4.70 146.42m 1h24m
92 GPT 5.4 Nano OpenHands 26.10% $1.28 - 54m36s
93 Qwen 3.6 Plus OpenHands 25.57% $5.45 9.64m 38m10s
94 GPT 5.1 OpenHands 24.61% $2.57 - 30m36s
95 GLM 5 OpenHands 23.36% $40.27 39.14m 3h44m
96 Claude Sonnet 4.5 (Thinking) OpenHands 22.62% $6.66 - 49m22s
97 GPT 5.1 Codex Max OpenHands 22.17% $7.32 - 8h40m
98 Claude Opus 4.5 (Thinking) OpenHands 20.63% $32.87 - 38m03s
99 Gemini 3 Flash (12/25) OpenHands 20.20% $0.94 - 13m27s
100 GPT 5 OpenHands 20.09% $1.53 - 30m52s
101 Muse Spark OpenHands 19.67% - 3.74m 16m21s
102 Grok 4.3 OpenHands 19.40% $1.28 - 9m49s
103 Inkling OpenHands 19.21% $1.61 7.89m 18m25s
104 Inkling Small OpenHands 19.05% $0.43 5.93m 49m09s
105 Kimi K2.5 OpenHands 17.54% $0.88 - 42m50s
106 Qwen 3.5 Plus OpenHands 15.74% $3.80 9.24m 50m16s
107 MiniMax-M2.5 OpenHands 14.85% $2.20 7.02m 51m06s
108 Gemini 3 Pro (11/25) OpenHands 14.30% $6.42 - 2h53m
109 GPT 5 Mini OpenHands 14.17% $0.25 543.34k 11m38s
110 Grok Build 0.1 Grok Build 13.35% - - 6m35s
111 GPT 5.1 Codex OpenHands 13.12% $3.80 - 50m26s
112 Qwen 3.6 27B OpenHands 11.94% $4.66 7.48m 2h43m
113 MiniMax-M2.7 OpenHands 11.93% $2.82 - 22m57s
114 Claude Haiku 4.5 (Thinking) OpenHands 11.39% $1.31 - 12m56s
115 Laguna M.1 OpenHands 11.04% - 1.86m 39m42s
116 Nemotron 3 Ultra OpenHands 7.64% - 39.42m 36m57s
117 SWE-1.6 Fast Devin CLI 7.22% - - 3m09s
118 Laguna XS.2 OpenHands 5.21% - 6.30m 36m39s
119 DeepSeek V3.2 (Thinking) OpenHands 5.11% $2.47 - 56m06s
120 Grok 4.20 (Reasoning) OpenHands 4.06% $0.77 - 5m02s
121 Nemotron 3.5 Super VL OpenHands 3.71% $0.54 8.45m 4m23s
122 Mercury 2.5 OpenHands 3.68% $0.38 4.00m 3m13s
123 Qwen 3 Max OpenHands 3.51% $6.02 - 57m45s
124 Ling 3.0 Flash Fin OpenHands 3.40% $0.23 13.48m 24m11s
125 GLM 4.6 OpenHands 3.09% $10.85 - 2h47m
126 Ling 3.0 Flash OpenHands 2.91% $0.29 15.67m 16m46s
127 Mistral Medium 3.5 OpenHands 2.89% $16.74 10.84m 41m11s
128 Grok 4.1 Fast (Reasoning) OpenHands 1.20% $0.21 341.69k 8m48s
129 Gemini 2.5 Pro OpenHands 0.40% $1.22 - 34m57s
130 Gemini 3.1 Flash Lite Preview OpenHands 0.00% $0.51 1.10m 9m33s
131 Nemotron 3.5 Lightning OpenHands 0.00% $0.13 10.77m 16m00s
132 Grok 4 Fast (Reasoning) OpenHands 0.00% $0.13 - 24m09s
133 Mistral Small 4 OpenHands 0.00% $6.47 42.97m 30m15s
Reasoning EffortAccuracy at each model's reasoning efforts, labelled with each provider's own setting
Vibe Code Bench v1.1•Nov 18, 2026
Accuracy
Model:
Error bars show ±1 standard error.

Background

Coding represents the largest use case for generative AI by both usage and dollars spent [1] [2], with developers increasingly relying on AI-powered tools to accelerate their workflows [3]. However, existing coding benchmarks have predominantly focused on evaluating models’ ability to solve isolated programming problems—individual functions, algorithms, or bug fixes [4] [5]. While these benchmarks provide valuable insights into code generation capabilities, they fail to capture the frontier of AI-driven software development.

The most impactful application of AI coding assistants is building complete applications from natural language specifications, a zero-to-one use case that can accelerate development and enable non-technical prototyping. However, no standardized framework exists for evaluating this end-to-end capability. Our benchmark addresses this gap by testing models’ ability to build functional applications in a full development environment, evaluating results through point-and-click testing to measure whether models can deliver working software that meets user requirements.

We will be rolling out an access plan for this benchmark in the coming weeks - if you’re interested, please fill out this form.


Results

Even top performers fall short: no model consistently delivers applications that pass every test, highlighting that reliable end-to-end application development remains an open challenge. In the current canonical v1.1 test-set results, Claude Fable 5.2 (OpenHands) leads at 93.77%, followed by Gemini 4 Neon (OpenHands), Claude Sonnet 5.5 (OpenHands), Gemini 4 Argon (OpenHands), Claude Haiku 5.5 (OpenHands), and Claude Fable 5 (OpenHands). The top model still has low-pass outliers, but the top group shows a much longer high-performing tail than earlier versions.

Histogram of Pass Rate Across Applications
TASKS

Application Examples

We asked all models to implement “Zeeter” - a website where users can write short-form messages, which appear to their followers. Features included:

  • Authentication (sign up and log-in)
  • Creating “Zeets”
  • Viewing the Zeets of their followers
  • Liking and Commenting on Zeets
  • Removing Zeets
  • Following and unfollowing other users
  • Searching
  • An explore page to discover users

Read the full specification and the evaluation rubric.

Here are currently hosted applications for Zeeter:

These applications were created entirely by the LLM, and only the networking configuration (host, port) was modified. Please do not put any sensitive personal data into these apps, or re-use any password. We make no guarantees of security, functionality, or content moderation.

Evaluation Examples

Here is an example of how the models were automatically evaluated. Each test has multiple steps. You can see the steps that the evaluator agent goes through for a selection of the models, and what the application looks like at each step.

Testing Generated Apps

Tool Calling

Vibe Code - Tool Use
3/133 models

We provided over thirty tools to the models, including operations to manage Supabase state, browse their running applications, and perform web searches. However, most of these tools were rarely used. The majority of tool use was concentrated in the four tools shown in the chart above.

Every model used the file editing and bash command tools extensively. The browser and SQL execution were the next most used, though usage varied significantly across models. Models typically used the browser to test their applications after implementation.

Trajectory Analysis

Trajectory by Model
1681 actions shown
Qwen 3.5 Plus93 actions
Claude Haiku 4.5 (Thinking)108 actions
Claude Opus 4.6 (Nonthinking)121 actions
Claude Opus 4.6 (Thinking)102 actions
Claude Sonnet 4.696 actions
Gemini 3.1 Pro Preview (02/26)72 actions
Gemini 3 Flash (12/25)148 actions
Gemini 3 Pro (11/25)148 actions
Grok 4.1 Fast (Reasoning)56 actions
Kimi K2.588 actions
MiniMax-M2.584 actions
GPT 5.2264 actions
GPT 5.2 Codex212 actions
GPT 5 Mini89 actions
050100150200250

We have taken the trajectory rollouts for each model while working on the Zeeter application, and categorized each step the model took into one of several buckets. You can see there is large variation in how models choose to spend their time, as well as variation in the length of each trajectory.

Error Analysis

Here, we report the most common qualitative error modes we saw when manually inspecting the trajectories.

  1. Installation Issues: Although the environment came with Docker and Node pre-installed, we expected the models to be able to install other tools and libraries (e.g., Tailwind) they needed via bash commands. Most models did not get these commands right on their first try. The best models were generally able to get it on the second or third try, while the worst wasted a significant amount of time and turns on unsuccessful installation attempts.

  2. Configuring Docker Networking: A crucial part of any app that requires communication with Supabase or a backend is ensuring the frontend knows the backend/Supabase endpoint. The best models determined that to make the environment variables available, they needed to either add a .env file in the “frontend” directory or pass the URL via build.args in the Docker Compose config. Passing it via the env_file parameter does not work, since that only controls runtime environment variables. The less performant models, however, would often get stuck on this for many turns or never solve it.

  3. Timeout Issues: A common error mode, especially for the worst models, was to continue making changes to files right up until the time or turn limit. This led them to not end the session with a working app.

  4. Giving Up Early: Often, models still had plenty of turns left, but they would use the finish tool to submit their work (the prompt was extremely explicit that this ends their ability to make further changes). The best models kept working on the problems until they had solved them, and had also tested that their solution was working.

  5. Direction Following: The less-performant models would often forget key aspects of the initial prompt, leading to major errors

Part of our goal was to measure whether models used the resources (time and turns) efficiently. We found the best models were able to do this, debugging issues in their code or config much quicker than worse models. This efficiency meant they were able to make it much further in a task.


Methodology

Specification and Test Construction

Our benchmark consists of application specifications and corresponding test suites that were carefully constructed through a multi-stage process combining AI generation with expert human review. We generated an initial corpus of ideas using language models, drawing inspiration from real-world consulting case studies, freelance development tasks and startup application patterns. Each specification describes a complete web application in non-technical, natural language, constrained to under one page to ensure clarity and focus on MVP functionality.

Every specification underwent multiple rounds of review by professional product managers and software engineers. Reviewers validated that each specification was clear, executable, and representative of real development work. They ensured specifications avoided prescribing specific technology stacks, allowing models the flexibility to choose appropriate frameworks and tools.

Each specification was paired with 20-60 automated tests covering core functionality, edge cases, and critical user workflows. Each test was triple-verified by professional software engineers to ensure it accurately validates the intended functionality, passes when the feature is correctly implemented, and fails when the feature is broken or missing.

Development Environment and Available Tools

Generation Schematic

Models operate within a fully-featured development environment built on a modified version of OpenHands, an agentic coding harness and framework. The environment uses a Docker-in-Docker architecture, providing each model with an isolated container where it has complete control over the development workspace. Models have unrestricted access to terminal commands, allowing them to install dependencies, run build processes, start servers, and execute any command-line operations necessary for application development.

To enable realistic application development, we provide models with access to a streamlined set of production-grade services running in sandbox or test modes. For authentication, database operations, and file storage, we use self-hosted Supabase instances; for payments, Stripe in test mode; and for email functionality, MailHog. Models also have access to web browsing capabilities for documentation lookup and external service integration. All API keys and service endpoints are pre-configured in the environment, allowing models to integrate these services without manual setup.

Models can work for up to 5 hours or 1000 turns on each application, with access to web browsing capabilities for documentation lookup, file system read/write operations, and the ability to spawn multiple processes. This mirrors the environment a human developer would have when building an application from scratch, with the added benefit of sandbox services that eliminate external dependencies and ensure reproducibility.

Automated Testing and Scoring

Evaluation Schematic

Our evaluation pipeline automatically tests each generated application through a workflow-style process designed to comprehensively assess functionality without human intervention. Once a model completes application development, we first verify the application builds and runs successfully. The application is then deployed to a local port within an isolated Docker container, where our automated testing infrastructure can interact with it.

UI tests are executed using Browser Use, an autonomous web agent that interprets natural language test instructions and performs point-and-click interactions with the running application. For each workflow, the agent navigates to the application, executes a sequence of user actions (such as signing up, creating records, or submitting forms), and validates that the expected outcomes occur at each step. The agent captures screenshots and DOM snapshots throughout testing, providing detailed execution traces for analysis.

Each UI test is composed of a series of “substeps” (e.g. 1. Login to the application, 2. Navigate to the profile page, etc.). The score for an application is the percentage of tests in which at least 90% of the substeps succeed. This final overall score is the average of the application-specific scores.

To validate our automated evaluation, we conduct alignment studies where professional engineers manually test subsets of applications using the same criteria. Current public reporting focuses on completed internal consistency/calibration analyses, with larger external human-evaluation rounds tracked separately.

The entire pipeline—from application generation through testing and scoring—runs automatically without human intervention, allowing us to evaluate multiple models across hundreds of application specifications with consistent, reproducible results. Note that to achieve this high level of consistency, evaluation costs 10-20 dollars per app generated.


Vibe Code Bench v1.1

Vibe Code Bench v1.1 was released on February 24, 2026. This update includes improvements to the benchmark’s data, evaluation pipeline, and overall reliability as compared to v1.0.

Data

  • We standardized the authentication of the specifications, providing clearer instructions to the agent on what the auth model should look like
  • We made explicit the email that the evaluator model should use when doing each ui test, to avoid the low chance of collisions.
  • For certain features being tested in the UI tests, we added more clarity to the corresponding instructions in the spec.
  • We improved the instructions in the tests on the dummy payment information for Stripe

Evaluation

  • We added some new tools to the browser-use evaluator model, meaning it can better handle HTML elements like date pickers
  • We improved the evaluator’s ability to “wait” for certain actions to occur
  • We added some additional files (csv, .jpg, .png, .pdf etc.) that the evaluator can use in the course of its evaluation.

All models were re-run from scratch using the Vibe Code Bench v1.1 harness and data.


Acknowledgements

A special thanks to Alex Gu, whose input was instrumental for the creation of this benchmark. We also appreciate Mike Merrill and John Yang for sharing their lessons learned with Terminal-Bench and SWE-bench Verified. Finally, we want to shout out Engel Nyst, Graham Neubig, and the rest of the OpenHands team for their work.

Citation (BibTeX)

@inproceedings{tran2026vibecodebench,
title        = {Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development},
author       = {Hung Tran and Langston Nashold and Rayan Krishnan and Antoine Bigeard and Alex Gu},
booktitle    = {ACM Conference on AI and Agentic Systems (ACM CAIS '26)},
year         = {2026},
doi          = {10.1145/3786335.3813180},
url          = {https://arxiv.org/abs/2603.04601},
}