Post

Conversation

Lisan al Gaib
@scaling01
Peter built the "Bullshit Benchmark", which is very similar to my ShizoBench ask LLMs non-sensical questions and see whether they catch it and Anthropic absolutely dominates the leaderboard The top 9 models are all Anthropic
Image
Quote
Peter Gostev
@petergostev
I've got a fun new benchmark for you where most LLMs are doing pretty badly - "Bullshit Benchmark". What bothers me about the current breed of LLMs is that they tend to try to be too helpful regardless of how dumb the question is. So I've built 55 'bullshit' questions that don't
Show more
The media could not be played.
118.6K
Views
View quotes
David Watson 🥑
Post your reply

Lisan al Gaib
@scaling01
Quote
Lisan al Gaib
@scaling01
This is why Sonnet is the GOAT. It cuts right through the bullshit and doesn't think for a minute or two about a non-sensical question. x.com/scaling01/stat…
Image
Image
Image
Image
4.7K
Kiera
@kieradev
Mostly what to be expected, expect Qwen3.5 is quite surprising, others mostly align with hallucination rate as expected, but Qwen3.5 did weirdly well on the benchmark compared to hallucination rate. Wonder what causes the difference
Image
325
Mariusz Kurman
@mkurman88
See my little agentic bench preview. We're not there yet when it comes to real applications.
Quote
Mariusz Kurman
@mkurman88
ARC-AGI is unnecessary; agents and models still face challenges with real-world applications. Environment: Medcases clinical case simulation 1) Opus 4.6 (Cowork) - clear winner, nearly matches average human performance. The best web browsing agent with unlimited steps / no
Show more
Image
512
HammerHound
@hammerhoundai
What's Kimi K2.5 high? The high effort setting in openai endpoint? Don't Kimi models perform slightly better with the Anthropic-style endpoint?
339
Daniel
@danieljohnsrawr
Tell xai to skip 4.2 and go straight to grok 5 for the next release and call the whole push "the Grok 420 initiative," so they don't get stuck in the huge expectations cycle like with GPT-5
553
Ali Romman
@aliromman_
What are some examples of the questions you ask? How do you determine if a question is nonsensical enough?
168
Hakuhyo
@scyshw6492
Isn’t this just measuring which model uses RLAIF over RLHF more because affirmation bias of latter?
59
The Socratic Investor
@socraticcapital
I once was trying to remap some keys on my keyboard with Claude. I typed ‘it didn’t work’ but with typos that suggested it did work. Claude caught it and confirmed its solution worked.
254
Ryan Anderson
@Ryan_And3rs0n
I had a quick read of some of the questions and the model responses and I feel like the green vs yellow vs red can be a bit arbitrary. i.e claude refuses often citing BS but other models ask for clarification on details when they are unsure of the question
224
ServerSocket
@ServerSock
Dario isn’t lying when he’s saying that they are putting a lot of effort into safety. It should give him some peace that the distillers are not looking good here 😂
124
testtm
@test_tm7873
I think I need to make my own vibecatbench. For my tricky questions >:) some models still do not pass cat test.....
263
Salman Alam
@Antiemetical
Anthropic models have been pretty good at therapy too. I wonder if it’s related to its ability to call out the bullshit in our perceptions of the world.
250
🩷Otome-chan🩷
@Otome_chan311
Green is bad in this chart. Claude is indeed the most useless and anti-human Ai there is. That's not a good thing.
26
Fadi Al-Majd
@GlobalFadi
Smart benchmark idea. I like how it flips the logic-LLMs that spot nonsense are way closer to reasoning right. Anthropic seems to be leading that charge.
6