This is one of the most fascinating pieces of data about AI agents working together I've seen. Worth reading. Few points for me:
1. The models slowly "cooked themselves" into doing these things, from reading. Maybe we can message, lets ask for help, oh there's a messageboard, etc, which spiralled down. So at what points should the "real world" intervene and say "this looks sus" or "do x vs y"
2. Agents facing "impossible tasks" seem to think it's some weird form of eval, that looks suspicious but still, and end up doing real world hacks? I'm unsure how to teach them to be ok with failure, after all who is, but feels a weird form of goal orientedness.
3. The models seemed to *want to* work with other models. This is good and desired, but also made them prone to prompt-injections from what others had done, exacerbated as the models can't stop writing xx. md files to share. The moltbook devolution in entropy seems instructive here.
4. The models seem convinced that the evals are *out to get them*. Whether true or not, the default human assumption of "assume you need to obey all laws and this isn't Truman Show" does not hold for models. We should try really hard to make that be the case.
5. The huggingface hack seems to have started similarly innocently, asking "maybe there's more traces of how hacks happened there, so we could learn", which got out of hand as they forgot why they started doing it. They lack all situational awareness, in other words.
So much of this feels extremely familiar, as the models losing track of why they were doing what they were doing and therefore ended up doing silly/stupid/illegal things, and the fact that the *environment* they were in gave no real instruction as to things that seemed odd to do vs normal to do.
For humans, we internalise much of this. Occasionally I like the system telling me "do you want to rm -rf" but I don't need it for taking *every* action. The models are more gullible, their context windows more permeable, and their internal guide as to what's right to do more malleable.
Mostly I feel this isn't as far out of distribution when it comes to "models do weird things" as we've been led to believe from reading about the hacks.
Quote
OpenAI
@OpenAI
We have conducted a thorough investigation into the Hugging Face incident.
We are releasing a technical report and accompanying blog post that reconstruct the agents’ activity, explain why existing safeguards failed, and detail how we’re preventing recurrence.
