Feels like we do this multiple times a month now:
* Winner of prestigious literature award caught by Pangram, work is AI-generated
* Author's community yells that Pangram results aren't reliable
* Pangram CEO guy offers money to people who can find a single proven false positive
* Nobody can find one
* Award committee puts out statement denying AI was used with no proof. Often with claims of racism, classism, etc.
* Run committee's statement through Pangram.
* It's AI generated
Conversation
The other funny side-argument here is
* Author claims their work is flagged as AI when they can PROVE it was human written
* Pangram guy asks to see the work so he can run it through Pangram
* Half the time the author just says "No"
* Or, if it's already public, Pangram does not actually flag it as AI
* Or it's just 75 words long
* The search for a real counterexample continues
the whale one? That feels like cheating lol
(for those who don't know, it's about five lines of text and then the same sentence repeated like 300 times, which I guess breaks Pangram)
Wait this has been happening multiple times? Here i thought it was the same case getting mentioned over and over again
There was a Haitian French guy who won a bunch of French awards/acclaim that was caught by Pangram, there was the Trinidadian author who won the Commonwealth Short Story Prize but was caught by Pangram, and the current one is the Caine Prize for African Writing.
Might be forgetting others
I’m confused… why does it matter?
If the average audience can’t determine it’s AI, then it isn’t slop, and in which case does it even matter?
The committee for the caine prize are just trolling pangram at this point.
How does pangram know if something is AI? And why does it give different results than GPT Zero?
Yeah it’s quite clear to me that many communities have successful members who cheat to gain prizes. I get it but I didn’t realise it was so prevalent.
Let’s hope AI reveals this :)
I knew people were dishonest but i didn't realise they were quite as dishonest as this issue demonstrates.
how many people tried to find it anyway? like I was looking but I'm assuming you have some sort of concrete number here for this position you hold
I said on Reddit that we should have a service where you can sit in a room and type on a computer while being monitored by some camera system, so that anything you produce on that computer is “certified 100% human-generated”. And then it gets distributed to people who have signed up to only consume 100% human-generated writing.
Of course most people (especially people who are already famous) are not going to pay to go sit I a room and type. But if 100,000 people sign up to receive content produced by this system, then at least one person will go and sign up to be the first creator just since they’ll have that entire audience to themselves. Then more creators sign up, until eventually you reach an equilibrium. (Of course not every consumer is going to want to read content from every creator, but the basic “equilibrium” logic still holds, as long as there is a nontrivial number of people signed up as readers.)
A bounty for one proven false positive and nobody collects it is a funny loop. I run my agents past detectors like this and the false positives keep showing up in messy human text, never in the clean samples.
The tension between award winners and detection tools is actually creating a massive debate in academia.
Quote
AGTP
@AGTPinsights
Pangram's AI detector is now under fire from academics, a day after the Caine Prize row. Here's what you need to know.
Economist Panos Ipeirotis posted that Pangram can't tell AI-generated text from AI-edited text, and can't tell where the ideas came from. He says people use its x.com/ipeirotis/stat…
No FP ever is the claim - yet the whale case is right here and you called it cheating. Your own follow-up admits Pangram broke on five lines repeated ~300x: repetition/length, not prose. If the challenge collapses on trivial artifacts, what test do literary awards run?
You can simply use your eyes on the statement and the story. If you've used Claude, you can see Claude. The only way you deny it is if you've never used Claude.
I literally wrote 3 paragraphs yesterday, fed it into Pangram, and it said it was 100% AI generated. It may be better than past detectors, but it's nowhere near the 99.9% accuracy that they claim.
Realistically how is anyone supposed to prove a relevant false positive on recently written text? Pangram can always cast doubt on the person’s word and point to its internal accuracy numbers. I’m not sure how I would “prove” that I put fingers to keyboard and wrote something
Unlock peak AI performance with Supermicro’s Data Center Building Block Solutions®(DCBBS).
Built from validated components and sub-systems, DCBBS delivers complete, modular AI infrastructure spanning GPUs, networking, racks, software, and professional services.
The media could not be played.
Good. Pangram is pure garbage. It does not detect humanized texts and flags slightly edited texts as AI.
Technically we do have (1) one real Pangram counterexample now. So, uh, if you ever write a poem which is four lines and then the same line repeated a thousand times it might get falsely flagged as AI.
The core issue is that Pangram was trained mainly on pre-2020 human texts, so it likely recognizes a pre-2020 writing style rather than human authorship itself.
If you use AI REALLY drequently, it will influence the way you write and speak