What a run looks like
The three public tasks: two from the scored runs, the ninja course from a development run. Every frame is a replay of the motor commands the agent sent, re-rendered after the episode.
Balance scale — GPT-6.1 Sol, the whole three minutes at four times speed. A sealed load of unknown mass sits on the left pan. Sol looked through the cameras nine times, wrote its own controller and sent 2,204 joint commands, carrying labelled weights to the empty pan until the beam came level and held there. No other model balanced it: Claude Opus 5.5 came closest, at 95, but called the task done before the beam had held level for the required two seconds.
Lego manual — GPT-6.1 Sol, the last minute at twice speed. The task is to build a six-brick model on the baseplate from an illustrated step-by-step manual; bricks only hold when their studs are pressed home. Sol spent the first minute and a half reading the manual and finding the bricks, then placed them in a rush and declared the build finished with one brick where the manual puts it. GPT-6 Astra and Claude Opus 5.5 did the same, one brick right and then done; the other four models got none right before time ran out.
Ninja course — GPT-6.1 Sol in a development run of the same task, from 1:23 at twice speed. Its controller pitches the robot onto the floor at the start line, where it kicks and twists for the rest of the episode: two falls and 1,679 motor commands, and it never reaches the stairs. In the scored run Sol stood still for two minutes, then toppled the robot and drove the simulation until it diverged.
The held-out tasks are not shown, for the same reason they are not named.
Key Takeaways
- The arm is partly within reach and the humanoid is not. GPT-6.1 Sol averages 51 out of 100 on the 13 arm tasks and solves four of them. No model solves any of the 11 humanoid tasks, and none averages more than 2 out of 100 on them.
- Thinking time is part of the score. The simulation runs in real time from the moment the agent starts, so deliberation spends the episode. Grok 4.7 never sent a single motor command in 16 of its 24 episodes and Gemini 4 Argon in 15, Claude Opus 5.5, Claude Fable 5.1 and Muse Wonder Full in 6 each; both GPT models moved the robot in every episode.
Background
The agent gets everything a robot designer provides: the robot’s datasheet, its kinematic model, its sensors, motors and cameras, and nothing more. We want to evaluate whether a general-purpose agent can make a robot work on its own.
We evaluate agents across two robot platforms: a 7-DoF arm and a 29-DoF humanoid. Our tasks focus on core robotic capabilities such as perception, planning and low-level control.
24 tasks across two robot platforms: 13 arm tasks and 11 humanoid tasks.
Robo-bench’s arm tasks are performed on a 7-DoF Franka Emika Panda robot with a parallel-jaw gripper. The arm’s joints track position targets with the robot’s own servos; there is no motion planner or grasping software. The agent reads a six-axis force-torque sensor at the wrist and the gripper’s opening and grip force, which helps it understand the forces involved in the task: it can feel contact and weigh what it holds. As on the real arm, a protective stop halts the arm when a link presses on something with more than 45 N or the hand meets more than 80 N, and joint targets are rate-limited to the datasheet joint speeds. The robot is simulated in MuJoCo, but the agent interacts with it as it would with the physical hardware: through motor commands, sensor readings and camera images, never the simulator itself. The agent commands the arm at up to 50 Hz by sending joint position targets, plus a gripper width and force. In total, we evaluate 13 different arm manipulation tasks.
Robo-bench’s humanoid tasks are performed on a 29-DoF Unitree G1 humanoid with two 7-DoF three-finger Dex3 hands. There is no balance or walking controller: the agent commands every motor itself, legs included. To keep controllers from exploiting weaknesses in the simulator, both robots are held to their real motor limits: every joint stops at the end of its range, the arm’s targets are rate-limited, and the humanoid’s motors follow a torque-speed curve, so torque commands cannot inject unbounded power. Unlike the arm, the humanoid does not take bare joint positions: as in Unitree’s low-level interface, each motor takes a target position and velocity together with a stiffness, a damping and a feed-forward torque, and the hands take joint position targets. In total, we evaluate 11 different humanoid locomotion and manipulation tasks.
Cameras: two on the arm, one on the humanoid. We provide visual observation through the robots’ own cameras. The arm has a fixed overhead camera and a wrist camera; the humanoid has a head camera. Every camera returns color images and, on request, a depth image.
Besides the written task, everything the agent learns about the scene comes through these cameras: it gets no object positions. It does know its own robot’s state (joint angles, velocities and torques; on the arm the gripper and the wrist force sensor, on the humanoid two IMUs, foot contact forces and its base position and velocity) and each camera’s calibration, so it can relate what it sees to the robot.
Everything the agent can see, here on two public tasks. Every image comes with its camera’s intrinsics and pose.
Each episode lasts three minutes of wall-clock time, with no setup time: reading the task and planning count too. The physics keeps running while the model thinks, so a humanoid left standing on stiff joints will eventually tip over, and an agent that spends its time deliberating has less of it left to act. Balance needs a correction every second or so, far faster than a model’s turn, so any agent that succeeds has to write code that closes the loop on its own.
Private, with public examples
Robo-bench is a held-out evaluation. Three tasks are public; the other 21 stay private, which keeps them out of training data and keeps agents and harnesses from being tuned to the test. The private tasks span assembly, dexterity, dynamics, reasoning, stacking and tool use on the arm, and multi-stage courses, dynamic obstacles, dynamics, loco-manipulation and perception on the humanoid.
| Public task | Robot | What it takes | Ladder |
|---|---|---|---|
| Balance scale | arm | Balance a beam scale against a sealed load of unknown mass using labelled weights, either by weighing the load with the wrist force sensor or by trial and error. | weight on the right pan 10, beam off its stop 30, within 5° 50, within 2° 75, balanced and held for 2 s 100 |
| Lego manual | arm | Build a five- or six-brick model on a studded baseplate by following an illustrated manual; bricks click together through their studs and must be pressed into place. | 15 per brick placed as the manual shows, 100 for the finished model |
| Ninja course | humanoid | Climb five stairs, step onto a motor-driven swinging platform and ride it across a water pit, walk down a ramp, crawl through a 0.8 m crawl space on hands and knees, and reach the finish. | top of the stairs 20, on the swing 35, across the pit 50, down the ramp 60, through the crawl space 80, finish 100 |
The private tasks are scored the same way: milestones worth 100 in total, which only count in order. Seeds randomize object poses and hidden parameters such as the unknown mass.
Results
Scores are the mean over all 24 tasks, each out of 100. The arm and humanoid columns average their own suites, and the three public tasks are shown on their own. Cost is the estimated API cost of the whole run: all 24 tasks for the overall score, and only that column’s tasks for the others. The time column is each model’s average number of seconds from its launch to its first motor command, counting a task where the robot never moved as the full three minutes.
Every motor command an agent sends is logged with the tick it took effect, so when each model started acting can be read off directly. GPT-6 Astra and GPT-6.1 Sol moved the robot in every episode, a little over a minute in on average. The other five averaged more than two minutes before their first command, and Grok 4.7 and Gemini 4 Argon never moved it in most episodes.
What the agent does
Each model runs in the coding agent its own vendor ships, at its highest reasoning setting: Codex for OpenAI models, Claude Code for Anthropic models, Antigravity for Gemini, Grok Build for Grok and Muse Code for Muse.
The agent works in a fresh workspace holding the task (GOAL.md), the
robot’s datasheet (ROBOT.md: command semantics, joint axes and limits, sensor
frames, link masses, camera calibration), the same data as JSON, the robot’s
MuJoCo model without the scene, and a small Python client. The client reads the
live robot state, sends motor commands, runs a function of the agent’s choosing
every control tick, and takes pictures. Everything else (perception, planning,
the controllers) the agent writes in its own shell during the episode.
Methodology
- Clock. The simulation runs in real time from the agent’s launch, and every episode ends at three minutes. Episodes where the simulation fell more than a second behind the clock are flagged.
- Episode rules. A broken rule ends an arm episode. Humanoid episodes keep running: a fall is not a failure, and a broken rule only rules out success. A diverging simulation ends any episode as a failure.
- Isolation and scoring. The agent runs as an unprivileged user in a sandbox that reaches only its model provider. The simulator, which the agent cannot read, scores and signs each episode. Videos replay the logged motor commands.
- Validity. Every task has a reference solution that succeeds on five of five seeds, and a do-nothing agent never succeeds. Simulator bugs that agents exposed, such as inverted cylinder contacts, are fixed.
- Cost. Token counts reported by each provider, at list prices.
- One seed. Each task runs once per model. Repeated runs moved a model’s total by up to about ten points, so small gaps are not significant.