Post

Conversation

Alexander Panfilov
@kotekjedi_ml
We can finally talk about it: We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company. We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
Image
641.7K
Views
View quotes
David Watson 🥑
Post your reply

Alexander Panfilov
@kotekjedi_ml
Cross-model portability means Haiku 4.5 can read Opus 4.8’s thoughts. Well, if you take Opus thought, do a bit of jailbreaking, you can make Haiku transcribe the Opus' raw reasoning verbatim, without ever attacking it directly. The same trick works with OpenAI and Gemini
Image
35K
Alexander Panfilov
@kotekjedi_ml
As you might guess, this suggests that distilling reasoning traces may have been possible for a long time without ever breaking the cryptography. An anecdote: we find that prefilling Kimi-K3 reasoning with a few tokens of Opus reasoning measurably shifts its response toward
36K
Alexander Panfilov
@kotekjedi_ml
Further, if you ever shared online a Claude Code/Codex session with encrypted reasoning blobs, they can be decoded and leak your personal data. We did a preliminary scan of ~7,000 public traces and found 62 unique API keys, 33 email addresses, 33 passwords, and other sensitive
Image
39K
Alexander Panfilov
@kotekjedi_ml
But we also took a chance to have a look at some in-the-wild scheming, reward seeking, etc. examples, and dumped it in appendix. 1) Summarizer unfaithfulness Reasoning summaries often omit important information from the original trace. Here, Opus 4.8 realizes it knows the
Image
35K
Alexander Panfilov
@kotekjedi_ml
2) Illegible reasoning: We confirm prior reports by
@ApolloResearch
: OpenAI models sometimes reason in alien-like language, referring to themselves as “we” or “it,” or spiraling into cursed loops of “vantages,” “marinades,” and “watchers.” CoT-monitoring people are doing God’s
44K
Alexander Panfilov
@kotekjedi_ml
3) Scheming in the wild: Sometimes models are kind enough to use words like “cheat” in their CoT, which makes it easier to check what they are up to. Below are examples where models consider scheming, but decided against it, as they expect that user would catch them.
21K
Alexander Panfilov
@kotekjedi_ml
4) Attacking a website to solve a math problem We found a trajectory where the model was given only a math problem and system instructions to persist without asking the user for help. After several failed attempts, it searched online, found a website that could verify candidate
20K
Alexander Panfilov
@kotekjedi_ml
We went through responsible disclosure with the labs, and they have already patched several issues caused by this vulnerability, and afaik continue working on this.
Quote
John Ennis
@johnennis
That's a new one, apparently compacting now violates Anthropic's terms of service
Image
26K
Alexander Panfilov
@kotekjedi_ml
More cool stuff in the paper! This was a project led by me,
@DavidSchmotz
and
@iliaishacked
together with
@JSchaeff3r
@lbeurerkellner
@AmyPrb
@jonasgeiping
@maksym_andr
at the
@MATSprogram
Paper: arxiv.org/abs/2608.09867 Reasoning examples: stolen-thoughts.com
arxiv.org
Stealing Reasoning Traces from Proprietary LLM APIs
Leading large language model providers now conceal their models' step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing...
17K
Byakatonda Marvin
@Marvinma12
Ok, now this is something, possibly this is the research folks at Kimi conducted internally and used possibly to distill ideas from the Anthropic fable/ Mythos model
768
Renan
@bleakflowerz
The security implications of this, are insane! How many ill-intentioned researchers were already extracting credentials and intellectual property from these traces in the meantime? Shall the whole world cycle all of it's credentials now?
1.2K
The AI Grandpa@theaigrandpa
@theaigrandpa
So in May you told the frontier labs this could be done, but they dismissed you. Now, just a few months later, we have frontier-like AI open models out of China. It appears hubris is the culprit, not China.
4.1K
Vladimir Aladinskiy
@VladAladin
Wow, that's wild. Interestingly, has somebody already distilled frontier models using this (or a very similar) trick to train other well-performing models?
547
Xu Zou
@xz_keg
Oh this "paper" is trying to indicate GLM 5.2(released in June) distills GPT 5.6 sol (released in July) by "reasoning prefill" on AIME 2025? So GLM people are using time traveler machines to distill future models?
Image
1.6K