Post

Conversation

Feels like we do this multiple times a month now: * Winner of prestigious literature award caught by Pangram, work is AI-generated * Author's community yells that Pangram results aren't reliable * Pangram CEO guy offers money to people who can find a single proven false positive * Nobody can find one * Award committee puts out statement denying AI was used with no proof. Often with claims of racism, classism, etc. * Run committee's statement through Pangram. * It's AI generated
David Watson 🥑
Post your reply

The other funny side-argument here is * Author claims their work is flagged as AI when they can PROVE it was human written * Pangram guy asks to see the work so he can run it through Pangram * Half the time the author just says "No" * Or, if it's already public, Pangram does not actually flag it as AI * Or it's just 75 words long * The search for a real counterexample continues
the whale one? That feels like cheating lol (for those who don't know, it's about five lines of text and then the same sentence repeated like 300 times, which I guess breaks Pangram)
There was a Haitian French guy who won a bunch of French awards/acclaim that was caught by Pangram, there was the Trinidadian author who won the Commonwealth Short Story Prize but was caught by Pangram, and the current one is the Caine Prize for African Writing. Might be forgetting others
I’m confused… why does it matter? If the average audience can’t determine it’s AI, then it isn’t slop, and in which case does it even matter?
Image
Yeah it’s quite clear to me that many communities have successful members who cheat to gain prizes. I get it but I didn’t realise it was so prevalent. Let’s hope AI reveals this :)
how many people tried to find it anyway? like I was looking but I'm assuming you have some sort of concrete number here for this position you hold
I said on Reddit that we should have a service where you can sit in a room and type on a computer while being monitored by some camera system, so that anything you produce on that computer is “certified 100% human-generated”. And then it gets distributed to people who have signed up to only consume 100% human-generated writing. Of course most people (especially people who are already famous) are not going to pay to go sit I a room and type. But if 100,000 people sign up to receive content produced by this system, then at least one person will go and sign up to be the first creator just since they’ll have that entire audience to themselves. Then more creators sign up, until eventually you reach an equilibrium. (Of course not every consumer is going to want to read content from every creator, but the basic “equilibrium” logic still holds, as long as there is a nontrivial number of people signed up as readers.)
A bounty for one proven false positive and nobody collects it is a funny loop. I run my agents past detectors like this and the false positives keep showing up in messy human text, never in the clean samples.
The tension between award winners and detection tools is actually creating a massive debate in academia.
Quote
AGTP
@AGTPinsights
Pangram's AI detector is now under fire from academics, a day after the Caine Prize row. Here's what you need to know. Economist Panos Ipeirotis posted that Pangram can't tell AI-generated text from AI-edited text, and can't tell where the ideas came from. He says people use its x.com/ipeirotis/stat…
Image
No FP ever is the claim - yet the whale case is right here and you called it cheating. Your own follow-up admits Pangram broke on five lines repeated ~300x: repetition/length, not prose. If the challenge collapses on trivial artifacts, what test do literary awards run?
I literally wrote 3 paragraphs yesterday, fed it into Pangram, and it said it was 100% AI generated. It may be better than past detectors, but it's nowhere near the 99.9% accuracy that they claim.
Realistically how is anyone supposed to prove a relevant false positive on recently written text? Pangram can always cast doubt on the person’s word and point to its internal accuracy numbers. I’m not sure how I would “prove” that I put fingers to keyboard and wrote something
Square profile picture
Unlock peak AI performance with Supermicro’s Data Center Building Block Solutions®(DCBBS). Built from validated components and sub-systems, DCBBS delivers complete, modular AI infrastructure spanning GPUs, networking, racks, software, and professional services.
The media could not be played.
Idek if you needed pangram in that case, the last sentence in every paragraph from the brief passage I read screamed AI, I can’t imagine how bad the entire thing was
Sounds like Pangram just claims many texts as AI. You do realize that just because Pangeam claims some text is AI isn’t proof that it is?
Good. Pangram is pure garbage. It does not detect humanized texts and flags slightly edited texts as AI.
Technically we do have (1) one real Pangram counterexample now. So, uh, if you ever write a poem which is four lines and then the same line repeated a thousand times it might get falsely flagged as AI.
The pattern is becoming hard to ignore. When both the work and the denial light up the same detectors, the questions get louder.
Translated from Chinese
Pangram's false positive bounty has gone unclaimed this whole time, which in itself says something. As for the judges' statement, I'd actually like to see how many it turned up.
The core issue is that Pangram was trained mainly on pre-2020 human texts, so it likely recognizes a pre-2020 writing style rather than human authorship itself. If you use AI REALLY drequently, it will influence the way you write and speak