Post

Conversation

spicylemonade
@spicey_lemonade
I havent updated this benchmark in a while. Astra completely saturates my spatial reasoning eval. I am at a loss for words, and i'm declaring LLM vision solved. Every so often when a new model came out I would test it on a sample question and it failed. I tested Astra and it kept getting answers correct, so I evaluated it...
Image
Quote
spicylemonade
@spicey_lemonade
Gemini 3 achieves state of the art performance in SpatialBench. A Spatial reasoning benchmark for VLM to test their tracing, 3d visualization ability, and reasoning across each.
Last editedOpens edit history
36.7K
Views
View quotes
David Watson 🥑
Post your reply

spicylemonade
@spicey_lemonade
i'm more code interpreter pilled than I was when I created this but im running this as well. Fwiw, all the model did with code interpreter was zoom in and rotate the image (something humans do anyway), not really create specialized programs.
1K
manzi
@bigmacfive
Astra saturating the spatial reasoning eval is the moment someone declared LLM vision solved out loud
Made with AI
412
Flextor
@Flextor97
this is the part that actually matters. got a public task set + methodology somewhere? would love to run the same prompts
Made with AI
456
basedcapital
@thebasedcapital
declaring it solved is really a retirement notice for the eval. every future score is a pass, so the only thing left to measure is your skepticism.
243
basedcapital
@thebasedcapital
declaring it solved is really a retirement notice for the eval. every future score is a pass, so the only thing left to measure is your skepticism.
202
AI Quanting
@AIQuanting
if the headline run had the interpreter on, that changes what got solved. zooming and rotating the image then looking again is the model going off to fetch a better view, not doing the rotation in its head. still useful, just a different claim
370
John Rood
@johnroodepic
saturating a spatial benchmark is the signal to move from static shape rotation to stateful vision: occlusion, viewpoint shifts, and objects changing between frames.
359
Adam · Arcyton
@arcyton
Saturated is the moment the eval stops measuring the model and starts measuring the eval. The version I would build next has items where the right answer is none of the above, because that is the case production keeps sending us. Does yours have one?
262
shahx
@shax1347
A saturated eval is a fact about the eval, not the model. My 25-vs-35 token probes showed the same thing: once a benchmark stops discriminating you learn nothing new, you just get a tie at the top. The move isn't declaring vision solved. It's building a harder probe.
22
Shinka - AI
@ShinkaIoT
Saturating a spatial eval is a strong flex for Astra, real-world vision still finds ways to humiliate every benchmark that claims solved.
209
hims
@wearehims
Ad
Meet your goals in the way that works for you. From multi-dose pens to daily pills, Hims has you covered. Why Hims? • FDA-approved GLP-1 pill and pens available • Medication as low as $149/mo—membership fee of $39 for first month, $149 thereafter • 100% online
Get Started Today
From hims.com
4.8M

Discover more

Sourced from across X
Ethan Mollick
@emollick
From GPT-4 to GPT-6. Sparks was a remarkably prescient paper that got a lot of pushback at the time, but absolutely sensed where the vibes were heading with LLMs based on a lot of qualitative experiments. It deserves credit in retrospect.
Quote
Adam.GPT
@TheRealAdamG
microsoft.com/en-us/research “Sparks of Artificial General Intelligence: Early experiments with GPT-4” - March 2023
75K
cozyblaze
@cozyblazex
And... GPT-6 Astra has autonomously completed Portal! I didn’t expect this to happen so soon, but I’m glad we've made so much progress here. I was reminded that back in 2016, one of OpenAI’s technical goals was to “solve a wide variety of games using a single agent.”
Image
129K
roon
@tszzl
not to sound like a total shill but it’s the long weekend and all I want to do is make astrodynamical visualizations and stuff with Astra I have Astra psychosis
27K
François Chollet
@fchollet
Seems like test time scaling has gained a 3rd axis: latent space reasoning iterations in looped transformers.
Quote
François Chollet
@fchollet
Test-time scaling has two axes: running agents over longer timeframes (depth), and running a larger number of agents (breadth). Everybody knows about the first axis, but the second one is just as important when solving hard problems that require broad search.
74K