GPT-6 Astra scores 46% vs 12% for MolmoAct2, a state-of-the-art robotics VLA, across 200 trials on five bimanual tasks. That's 3.9x higher. 🧵
The media could not be played.
Conversation
Post your reply
We tested GPT-6 Astra and Molmoact2 across five challenging tasks that require both arms.
The media could not be played.
GPT-6 Astra consistently scores better than MolmoAct2 across all five bimanual robot tasks. In 99 of its
100 trials, Astra scored at least as high as MolmoAct2’s best trial on the same task.
The easiest task is to uncap the marker. A perfect score by a human is demonstrated below. This example is not provided in-context to the models.
The media could not be played.
MolmoAct2 always failed to separate the cap, while Astra could separate the cap 25% of the time (5/20).
The media could not be played.
The next task is to pour paper clips into a lifted bowl.
The media could not be played.
MolmoAct2 could lift the bowl but did not attempt pouring the paper clips. Astra poured the paper clips 30% of the time (6/20), but always spilled the paper clips.
The media could not be played.
Next, the model has to move a ruler from one cup to another with a mid-air transfer.
The media could not be played.
MolmoAct2 touched the ruler but did not succeed in lifting the ruler. Astra could do mid-air transfers 10% of the time (2/20) and moved it to the other cup 5% of the time (1/20).
The media could not be played.
The fourth task is to remove the middle notes pad from a stack of three without the top pad touching the table.
The media could not be played.
MolmoAct2 did not approach the notes pads, while Astra touched the pads in all 20 trials, and succeeded in removing the middle pad without the top pad touching the table 1 out of 20 times.
The media could not be played.
The fifth task is to take out an eraser from a box and close the lid.
The media could not be played.
MolmoAct2 did not approach the box. Astra could touch and open the lid. Neither model succeeded in taking the eraser out of the box.
The media could not be played.
Our results across 200 trials show that a LLM outperforms the SOTA open-source robotics VLA on five bimanual tasks. Phillip Isola calls this the beginning of "Robot-Use Agents".
Quote
Phillip Isola
@phillip_isola
Recently, there have been a lot of impressive demos of AI agents, like Claude, controlling robots.
I wrote a short blog post with my thoughts on the advent of these "robot-use agents."
web.mit.edu/phillipi/www/w
I think it's an important change in the trajectory of robotics!
We release the report and the traces of all 200 trials here:
We release the five-task bimanual benchmark here:
We evaluated GPT-6 Astra and MolmoAct2 using Inspect Robots, our open source evaluation framework for physical AI:
Robocurve () is a Public Benefit Corporation measuring frontier robotics capabilities and reporting them to the public. Learn more:
