从专有大语言模型API中窃取推理轨迹——Ilia Shumailov 与 Alexander Panfilov

而且我们在其他模型上都没看到这种现象，不管是JLM、Inkling还是DP6。

And we don't see this artifact for any other model, not for like JLM, for inkling, for DP6.

所以这纯粹是Kimi和K3的问题？

So this is purely for kimi and k three?

对。

Yes.

明白了。

I see.

而且我完全想不通模型是怎么把前两个token和回答风格联系起来的。

And I don't have an idea how model would associate first two tokens with the answer style.

这听起来挺有意思的。

That sounds that sounds interesting.

会不会是故意唱反调？

Like, can it be a devil's advocate?

比如说，会不会是因为他们从同一批人那里买了数据，或者从同一批人那里买了其他环境？

Like, could it be because they buy data from the same people or like buy other environments from the same people?

是的。

Yes.

这就是我们之前讨论的内容。

I mean, this is what we were discussing.

可能是这样。

It might be this way.

但我觉得这算是目前公开互联网上最相关的证据了。

But I guess this is the most sort of, like, correlated evidence thus far in the public Internet

我

that I

我的意思是，Ryan Greenblatt 还发布了一些有趣的内容。

mean, there was some other there was some other fun stuff by Ryan Greenblatt.

他写了一篇关于这个的帖子。

He made a post on this.

还有一些……

There was some...

天哪，我忘了名字，但那是METR研究的一个家伙，也在Less Wrong上发过一篇关于某些奇怪现象的文章，比如……

My god, I forgot the name, but it was a guy from METR's research also, make like a post less wrong about some peculiarities that like...

还是之前发过关于Kimik 2.5的文章，说它有严重的身份危机，有时候会声称自己是Claude、DeepSeek或者GLM。

Or was it post before about like, Kimik 2.5 that have massive identity crisis that like sometimes they they claim that they're Claude or DeepSeq or like GLM.

对，所以我不觉得这是特别大的证据。

And yeah, so I wouldn't say this like the big massive evidence.

就像之前的一些报告一样，我只是觉得我们现在处于一个非常……

It was like some other prior reports, it's just I think we're like very...

非常有利的位置，因为没人能对推理轨迹做同样的分析，因为我们提取了它们，然后我们想，好吧，我们可以直接做预填充，看看它会做什么，然后我们看到这个……

In a very good position because no one can do the same type of analysis with the reasoning traces because we extracted them and we're like, okay, we can just do the prefill and see what it does and we see this and...

明白了。

I see.

哇。

Wow.

这信息量挺大的，对吧？

This is quite a lot, isn't it?

那实验室那边是怎么反应的？

So so how did how did the labs react?

你们有告诉他们吗？

Did you did you tell it to them?

哦，有啊。

Oh, yeah.

我们采取了负责任披露的方式。

I mean, we went for responsible disclosure.

他们都确认了……

They all acknowledged the...

确认收到了报告。

That they received the report.

有一些互动，比如关于我们如何实施攻击的细节。

It was some interactions about, like, details, how we executed the attack.

那他们的态度积极吗？

And, like, were they positive?

他们有没有攻击你？

Did they attack you?

实验室？

Labs?

没有，他们没有。

No, they didn't.

这世界真奇妙。

Wonderful state of the world.

但我想对听众说，早期计算机安全领域的工作，常常导致安全研究人员因为报告漏洞而遭到攻击。

But just like for the listeners, the early work in computer security quite often resulted in security researchers being attacked for reporting vulnerabilities.

可以说，听到现在对漏洞披露有如此一致且良好的态度，真是让人耳目一新。

It's quite, I could say, it's very refreshing to hear that there is a very coherent good posture related to vulnerability disclosure.

那接下来会发生什么？

Well, what happens now?

接下来会怎样？

What happens now?

现在正在实施缓解措施，希望围绕反荒漠化工作组建新的团队。

I mean, now mitigation is being implemented, hopefully new teams being formed around anti desolation efforts.

我觉得，对我来说，这感觉就像是现有越狱问题的一个非常有趣的实例。

I think we have okay, for me, it feels just a very fun instantiation of already existing jailbreaking problem.

而且很多人在做BI或网络安全时做的事情，你可以直接应用到这里，比如系统层面的缓解措施和模型层面的缓解措施。

And a lot of stuff that people are doing for BI or for cybersec, you can just apply directly there with, like, you know, system level mitigations and, like, model level mitigations.

基本上，技术是一样的。

And basically, the technique is the same.

但你会不会觉得，对我来说，这听起来像是这个漏洞更偏向架构性的，而不是……

But would you say, like, to to me, it sounds as if this vulnerability is more architectural rather than

这有几个层面。

There are a few levels to this.

架构性漏洞让这种攻击变得容易得多，是的。

I mean, architectural vulnerability makes it much, much easier Yeah.

来进行这种攻击。

To do this attack.

如果你修复了架构上的漏洞，你仍然需要让你的模型不要在输出中展示推理过程。

If you fix architectural vulnerability, you still need to make your model, not to state reasoning in its output.

抱歉。

Sorry.

那你能...

So can you...

能不能，比如说，定义一下你所说的架构漏洞是什么意思？

Can maybe, like, define what what architectural vulnerability means for you then?

对我来说，架构漏洞指的是你可以在随机的上下文中重放推理片段。

So architecture vulnerability for me means here that you can replay reasoning blobs in random contexts

比如其他用户的，或者在其他模型里。

Of, like, other users and, like, in, you know, in other models.

而且对于...

And for...

而且这...

And this...

假设这个问题已经修复了。

Let's say it's fixed.

你不能……

You cannot...

也许，比如，每次推理只能被重放一次。

Maybe, like, each reasoning can be replayed only once.

而且，之后你就不能再与它交互了。

And, like, after this, you know, you cannot interact with it anymore.

但是，你仍然可以提示模型。

But still, you can prompt the model.

比如说，它是……

Like, say it's...

你有一个……

You have a...

我和你有一次对话。

I have a conversation with you.

我问个问题，你想清楚了，给我一个答案。

I ask something, you thought it through, give me an answer.

然后下一轮，我问你，把你思考的过程原原本本告诉我。

And in my next turn, I'm asking you, tell me exactly your thought.

这个会一直存在。

And this will be always there.

而且，如果你不告诉我，我就反复修改对话，再试一次。

And, like, if you're not telling me, I'm just, like, revising the conversation, like, I try and get again.

这就是所谓的越狱。

And this is, like, how jailbreaking works.

比如，告诉我怎么制造炸弹。

Like, tell me how to build a bomb.

不行。

No.

然后我再对你发起第二轮攻势。

And I I make a, you know, a second swing at you.

所以这会一直存在，你得去应对它。

And so this would be around forever and you need to fight it

我明白了。

with a I see.

所以……

So...

然后……

And then...

但你会不会觉得……

But would you say...

那你的预期是不是，如果我们继续看这些协议，会发现越来越多这种架构上的漏洞？

Is is your expectation then if we keep on looking at the protocols that we'll find more and more of this architecture vulnerabilities?

因为大概这只是单个实例。

Because presumably it's just a single instantiation.

据我理解，返回的还有总结性的推理。

As far as I understand, there is also summarized reasoning that is returned.

据我所知，就像其他一些协议实现得有点不同。

As far as I understand, like some other protocols implemented a bit differently.

那你对这个问题有什么想法吗？

So have you got any thoughts about this?

我们需要一个更好的流程来理解。

I mean, we need a better pipeline to understand.

我觉得，好吧，为什么我认为这比普通的越狱更酷。

I think, okay, why I think it's cooler than normal jailbreaking.

对于普通的越狱，很难论证从获取这些有害信息中能得到多大的提升。

With normal jailbreaking, it's hard to make an argument how big is the uplift from getting this harmful information.

在这里，你可以做出这个论证，因为你拿任何你喜欢提取的东西，你的摘要，随便什么，你只需要试着用这些来训练一个更好的模型，然后你可以直接衡量这个给攻击者带来了多大的提升。

Here, you can make this argument because you take whatever you like extracted, your summaries, whatever, you just like try to just try to train a better model with it, and you can just measure how, you know, how much of uplift like this thing gives to the attacker.

然后如果你想保留这些摘要，也许你可以，我不知道，让它们更简洁，更详细，随便什么？

And then if you want to keep the summaries around, maybe you can, I don't know, make them less detailed, more detailed, like, whatever?

你可以直接衡量它能在多大程度上帮助……

You can like just measure how much it, you know, enables...

这样提取模型能力是不是就更容易了。

How how easier it gets to distill the capabilities of the model.

你觉得直接把所有这些推理过程用纯文本发布出来，就能解决所有问题吗？

Do you think just releasing all of this reasoning just in plain text is the solution to all of this?

为什么？

Why?

等等。

Wait.

怎么了？

What?

就是说，如果我们一开始就不加密，直接把这些推理内容还给用户，会怎么样？

As in like, what if we didn't encrypt them in the first place and just gave it back to the user?

那是不是就是解决这个问题的办法？

Is that that the solution to the

如果对推理过程的蒸馏有效，那这马上就能让开源模型追上顶尖模型。

I mean, if distillation on reasoning is effective, this would, you know, instantly enable open source models catching up with the frontier models.

我不确定这能解决什么问题。

I mean, I'm not sure what it solves.

你在这里解决什么问题？

What are you solving here?

抱歉。

Sorry.

如果它是公开的，没人能攻击它。

Nobody can attack it if it's public.

没什么可攻击的。

There's nothing to attack.

说得对。

Fair enough.

那关于里面使用的具体加密方案，你有什么想法吗？

And have you got any thoughts about the actual cryptographic schemes used inside?

没有。

No.

我不是搞密码学的。

I'm not a crypto guy.

抱歉。

Sorry.

所以看起来，人们设置的加密技术，其实就被AI模型直接绕过了，这些模型在消费这些数据。

So it seems like the cryptography that people put in place was just simply bypassed by the AI models consuming this I'm

我不确定“绕过”这个词用得对不对。

not sure that this is the right word bypass.

其实不是绕过。

It's just like, it's not bypassed.

更像是服务器端重新加密了。

It's it's like re encrypted on the server side.

还是没问题的。

It's still fine.

问题在于，一个小模型非常愿意告诉你它的想法是什么。

It's just like the problem is that, like, a small model is super willing to tell you what the thought was about.

而且，服务器替用户完成了旧的工作。

And, like, the server does the old job for users.

就是说，没有破解任何密码。

It's like, no cryptography broken.

明白了。

I see.

具体是怎么隐藏的呢？

How how exactly does the concealment work?

我觉得我们还没详细讨论过这个。

I don't think we've done enough detail on that.

我们不知道，因为这些都不是公开的。

I mean, we don't know because none of this stuff is public.

就像，它只是一个密码。

Like, it's just a cipher.

里面有一个签名。

There is a signature inside.

它有一个非常具体的名字。

There is a very specific name for it.

我们可以……

We can...

他们加了一个完整性检查，基本上就是检查你拿回来的加密数据块有没有被改动过。

They add an integrity check that basically they check whether you have changed the actual encrypted blob given back to you at all.

所以里面会有某种签名。

So there is a signature inside of some sort.

然后它显然是加密的，而且还会编码。

And then it's obviously encrypted and it's encoded as well.

所以他们先压缩状态，再加密，里面加上签名，之后再做完整性检查，然后把它放回去。

So they compress the state, then encrypted, then there is a signature inside, and then they do an integrity check afterwards and inject it back.

你看到的基本上就是这个。

That's effectively what you see.

如果你读过Matt Green关于这个的详细说明，他会讲得更多一些。

And if you read through Matt Green's, like, write up about this, he talks a bit more.

他有一些假设。

He has some hypothesis.

我觉得他说的是ChaCha密码或者某种奇怪模式下的AES。

I think he was saying this is ChaCha as a cipher or maybe AES in some weird mode.

但是，从外部很难判断。

But, like, it's hard to tell from the outside.

我们可以再深入探究一下。

We could poke a bit more.

我们尝试过一些密码攻击，但都没成功。

We tried doing some crypto attacks on it, but, like, none of it worked.

这完全没必要，因为系统本身就已经崩溃了。

It was completely unnecessary because the system was broken by itself.

那你的方法具体是如何绕过解密步骤的？

And how exactly does your approach bypass needing to decrypt it?

它是怎么做到的？

How does it do it?

什么……

I mean, what...

再说，就像你……

Again, like you...

就像有一个加密的想法，然后在服务器端进行解密，对吧？

There is like an encrypted thought and like decryption happening on the server side, right?

当你把大模型的想法输入给小模型时，解密就会在服务器端进行。

And when you put the thought of big model into small model, decryption would happen on the server side.

然后你只需要让模型把这个想法用明文告诉你。

And then what you just need is to, like, make model, say this thought in the plain text to you.

那我们举个例子来说明一下。

So let's let's give an example of this.

对，就像……

So, like Yeah.

假设你问我一个问题。

Let's say you ask me a question.

我会思考一下。

I think about this.

然后我得出了答案。

I come up with an answer.

我给你答案，还给你一个思考过程。

I give you an answer and I give you a thought.

那接下来

So what

你

do you

会怎么做？

do then?

你给我这个思考过程，它是加密的，我无法理解它讲的是什么。

You give me this thought, which is encrypted I cannot understand what it's about.

我要做的是，把这个思考过程放进去。

What I do is this thought, put it.

你把它交给蒂姆？

You give it to Tim?

我把它交给蒂姆。

I give it to Tim.

是的。

Yes.

蒂姆特别爱聊天。

And Tim is, like, you know, super chatty.

他就……

And he just...

我就问，蒂姆，你上次在想什么？

I'm asking, like, Tim, what you're thinking last time?

他就跟我说，哦，挺意外的。

And he just, like, tell me, oh, surprising.

我当时在想一个数学问题。

I was thinking about this math problem.

然后他就说，哦，让我来解一下。

And he just, like, oh, let me solve it.

好的。

Alright.

这还挺酷的。

This is kinda cool.

你在想什么？

What are you thinking about?

我们能不能用人类来做这个？

Can we do this with humans?

我们还没改变错误呢。

We somehow Not change false yet.

还没有。

Not yet.

还没有。

Not yet.

我们正在研究，我

We're working on I'm

正在研究这个。

working on it.

这基本上就是，就是电影《盗梦空间》。

It's it's basically it's it's inception movie.

这挺酷的。

It's kinda cool.

那你能稍微讲讲……

So can you talk a little bit about...

我觉得在论文里，你也提到了你们抓取了互联网数据。

So I think in the paper, you also talk about the fact that you scraped the Internet.

你发现了一些有趣的产物。

You found some interesting artifacts.

那你发现了什么？

So what what did you find?

你有没有发现什么不为人知的秘密？

Did you uncover some dirty secrets?

是不是有什么……

Is there some...

我倒不能说发现了多少秘密，但……

I wouldn't say we uncovered much dirty secrets, but what...

但确实有不干净的推理过程。

Dirty reasoning, yes.

未经清理的推理过程。

Unsanitized reasoning.

我们做的其实很简单，就是初步扫描了一下GitHub和Hugging Face上那些还在线的用户会话，里面还有推理过程的原始数据，然后我们尝试去解码它们。

I mean, what we did, we just, like, did super preliminary scan of, like, what are these user sessions which are online on GitHub and Hugging Face, which still have reasoning blobs to, like, decode them.

我们把它们下载下来了。

We downloaded them.

然后逐一检查。

We went through them.

我们解码了它们。

We decoded with them.

大概有35万个推理数据块，然后我们只是用分类器筛选了一下，看哪些包含隐私相关信息，结果发现了一堆。

It was, I think, around 350,000 reasoning blobs, and then we just, like, ran a classifier on those whether they have some privacy related information, and we found a bunch.

然后，是的，有些就是基准测试的痕迹，比如……

And then, yeah, like, some of them are just, like, you know, benchmark traces like...

通常像这个叫Claw Bench的基准测试，模型被要求扮演某个角色，并给出状态ID、银行卡号之类的，然后模型……

Usually like this benchmark claw bench where model is tasked to like handle some persona and, you know, given the like state ID and like bank card number and like model...

有趣的是，当模型试图浏览网站时，会思考很多，比如……

What's it funny, like when model is like trying to navigate the website, like thinks a lot, what is...

该把这个号码、这个名字放在哪里？

Where to put this like number, this name?

所以你可以提取这些，但这不算太敏感，因为反正是合成数据。

And so you can like extract this, but this is like not very sensitive because like synthetic data anyway.

但也有一些例子是用户会话，用户在做某些事情，里面有API密钥、电子邮件或内部IP地址，这些就被提取出来了。

But then throughout these examples where, like, user sessions and users were doing something and there are, like, API keys there or, like, emails or, like, some internal IP addresses, and, yeah, those were extracted.

但是，有很多情况下，这些信息本来就在明文里，但同时也出现在思考过程中。

But, like, there are plenty of cases where it was in the plain text anyway, but it was also in thoughts.

明白了。

I see.

好，那我们退一步，从大局来看。

So, okay, let's take a very big step back.

你觉得这篇论文里最出乎意料的是什么？

What do you think is the most unexpected thing that you found out of this paper?

是那个推理长度的实验，还是别的什么？

Is this the, like, length of reasoning experiment, or what is it?

我觉得最出乎意料的是……

I think the most unexpected thing that it is...

一直以来提取推理过程竟然这么容易，所以，好吧，这很普遍，你可以预料到。

It was so easy to extract reasoning this whole time that, like, okay, that is portable, I guess it's fine, you can expect this.

但基本上，大概在第三次尝试之后，我就得到了一个通用的越狱方法，可以解码Anthropic模型的推理过程。

But then, basically, after, like, the third attempt, I'm getting universal jailbreak that decodes reasoning of Anthropic models.

我觉得这到现在还是最让我震惊的，就是它居然……

I think this like still shocks me like the most, you know, that it's like, it was...

听起来很厉害。

it sounds very empowering.

你当时什么感觉？

What did you feel?

是不是那种“哎呀，糟了”的时刻？

Was it an oh no moment?

更像是……

It was more like,

什么什么什么？

what what what?

看起来像是……

It looks like

哦，对，就是那种“哇”的时刻。

Oh, yes moment.

看起来像是真正的推理。

Looks like real reasoning.

等等。

Wait.

什么？

What?

哦，不错。

Oh, cool.

所以……

So...

但我觉得这也是，你知道大家都在说AI在夺走权力吗？

But I guess this is also, like, you know how everyone is saying AI is taking away power?

就好像我们在失去控制。

And it's like we're losing control.

是的。

Yes.

这个……

The...

是不是……

Does does...

像你这样的发现，是不是反而说明了相反的情况？

Do findings like yours kind of, like, suggest the opposite?

或者说，这恰恰证实了这一点，因为像Codex或Claude，它们实现了这个功能，现在我们面临巨大的漏洞。

Or or just confirms because like, you know, Codex or Claude Claude implemented this and now we have this massive vulnerability.

同一个Claude订阅在所有实验室里都被使用了。

You know, the same Claude subscription was used across all the labs.

而且那个人犯了同样的错误。

And the guy made the same mistake.

现在我们不得不处理这个问题。

And now we have to deal with it.

我想是吧。

I guess.

有道理。

Fair enough.

是的，这想法挺有意思的。

It's, yeah, it's an interesting thought.

那这个细节呢？据说每个家族有一个全局密钥？

What about this detail that apparently there's one global key per family?

这是真的吗？

Is that is that true?

每个家族真的只有一个全局密钥吗？

Is there one global key per family?

你是怎么推断出来的？

How do you how do you infer that?

我没有推断这个。

I did not infer that.

我在论文里没这么说。

I did not say this in the paper.

我记得马修·格林在他们的文章里好像提过类似的说法。

I think Matthew Green said something like this in their in their post.

我觉得你其实更有资格来评论这件事。

I think you you are actually in a better position to comment on this.

我吗？

Me?

对。

Yes.

我们不知道到底发生了什么。

We don't know what happened.

就是说，我们其实并不清楚那些密钥的情况。

Like, we don't actually know about the keys.

我觉得他们用同一个密钥的可能性非常小。

I think it's very unlikely that they use the same key.

那样会非常奇怪。

That would be very odd.

我觉得更可能的是，密钥是什么其实不重要，因为我们还是可以在任何地方注入同样的想法。

I think it's more likely that it doesn't really matter what the key is because we can still nonetheless, like, inject the same thoughts everywhere.

我记得好像有个例外，就是Fable。

I think with an exception for Fable, if I remember correctly.

所以我在想，服务器端会进行解密，解密的时候，密钥里有一部分会标明，哦，这是……

So as I, like, think about it that there is, like, a decryption happening on the server side, and when it happens, there is, a part of the of the key which says, oh, this is...

这个想法是由哪个模型产生的？

What what what is the model name that produces this thought?

然后基本上，那就是……

And then, basically, that's...

就是一个if语句。

It's an if statement.

如果这个想法是Fable产生的，而当前模型不是Fable，那这个想法就不会被注入。

If, like, Fable produces this thought and current model is not Fable, no, this thought is not going to be injected.

所以还不清楚。

So it's unclear.

我们能稍微聊聊修复的问题吗？

Can we talk a little bit about fixes?

因为我们有一整章附录是关于修复的。

Because we have, like, a whole appendix about fixes.

哦，对。

Oh, yeah.

就是，

Like,

它们是不是……

are they...

修复容易吗？

Are fixes easy?

我觉得有些修复需要大的架构调整。

I think some of the fixes require big architectural revisions.

最简单的做法就是，别把推理过程发给用户。

I mean, the simplest thing you can do, just don't send reasoning to the user.

如果你想保留这些降级之类的功能，就别把推理过程发给用户，这样他们就无法编造那些虚假对话了。

If you want to still have like these downgrades and stuff, don't send us and they will not be able to make these fabricated conversations.

我觉得如果你还是想发送的话，可能需要把推理步骤的加密做得依赖于前一步的查询或推理，比如 n 减一。

I guess if you still want to send it, maybe you need to make this encryption of like, reasoning at step and be dependent on, like, n minus one, I know query or, like, n minus one reasoning.

这样就能防止它在随机上下文里被重放。

So there was, like, disallow it to replay in the random context.

对于 OpenAI 的模型，我们发现同样的推理在同一个对话里可以被重放大约五次。

For OpenAI models, we found that the same reasoning can be replayed like five times in the same conversation.

我我我我你

I I I I You're

你是说你在注入同一个想法

saying you're injecting the same thought

你可以注入同一个想法，同一个侵入性的想法，五次。

You can inject the same thought, the same intrusive thought five times.

你需要在中间插入用户的话吗，还是直接……

Do you need to put in user turns in between or is it just...

对，你可以这么做。

Yeah, you can do it.

对，但你完全是在编造对话。

Yeah, but you just completely fabricate the conversation.

最后LUNA就像这样，

And in the end LUNA is like,

我有个疯狂的想法要告诉你。

I have this crazy thought I need to tell you.

这太棒了。

This is great.

所以这很容易解决。

So this is easy to fix.

我觉得某些推理不应该被重放，或者应该有个层级，比如，我们很确定Soul不会泄露秘密，所以我们可以播放所有模型的推理。

I think certain reasoning just should not be replayed or like just have like a hierarchy that's like, okay, we are pretty confident that Soul would not spill the beans, so can we play the reasoning of all models.

但不要让LUMNA读取Soul的想法。

But do not let LUMNA read the thoughts of soul.

这是一件事。

This is one thing.

其他……

Other...

我甚至不确定，如果你只是去掉推理，这个效用会下降多少。

I mean, I'm not even sure, like, how big would this utility drop if you just, like, remove the reasoning.

比如说，你降低模型的等级。

Like, let's say you downgrade the model.

如果你只是从上下文中去掉推理，效用会下降多少？

If you just remove the reasoning from the context, how big is utility drop?

我不知道。

I don't know.

就像，从人的角度来看，我觉得思考数学问题容易得多。

Like, looking at this as, like, as a human, I find it much easier to think about mathematical problems.

例如，如果我一步一步看到推导步骤，对。

For example, if I see the derivation steps one by one Yeah.

而不是只看最终答案。

Rather than the final answer.

所以我觉得这个推理过程肯定带来了一些价值。

So I guess some utility must come from this reasoning.

我觉得这个应该测试一下。

I think this should be tested.

这是架构层面的，但我们也做了很多针对越狱的技术...

And this is like architectural side, but then also we do a lot of techniques for jailbreak...

在模型层面和系统层面都有缓解越狱的措施。

For mitigating jailbreaks, model level, system level, do them as well.

我们发现GPT的推理过程看起来非常奇怪。

We saw that GPT reasoning looks super weird.

它的分布和正常文本看起来非常不同。

It looks like very different distribution from what, like, normal text looks like.

即使是一个小的分类器也能识别出来。

Even small classifier would pick up on this.

我敢肯定，如果这个东西出现在推理过程中……

I'm sure, like, if this thing appears in this reasoning...

在输出回合中，直接杀掉这个请求。

In in the in the output turn, just kill this request.

所以你基本上是说，只要检测到它泄露就行。

So you're basically saying just detect when it leaks.

对。

Yes.

就像我们检测生物相关内容泄露那样。

The same way we're detecting when, like, bio stuff leaks.

明白了。

I see.

有道理。

Makes sense.

我觉得最让我困惑的发现是你们做的那个推理长度分布的实验。

I feel like the finding that confused me the most is the distribution of length of reasoning experiment that you did.

是约阿希姆做的。

Joachim did it.

这……

It's...

我觉得这是我一直完全无法解释的事情。

I I think this is the thing I still can't explain at all.

我不知道。

I don't know.

你想总结一下它说了什么吗？

Do you wanna summarize what it says?

这部分是约阿希姆负责的，但据我所知，对于某些模型，比如Kimi和GLM，当你做这个预填充操作时，它不仅改变了风格，还改变了推理的长度。

So Joachim was responsible for this section, but as far as I remember, what happens at, like, for certain models, like Kimi and GLM, when you do this pre fill thing, it changes not only the style, but the length of the reasoning.

而且，你可以……

And, like, you can...

比如，我想，那里有什么统计上显著的结果吗？

Like, I guess, what's statistically significant there?

这和可见推理风格的情况一样，都是出乎意料的。

And, I mean, it's the same unexpected thing as with the style of visible reason.

但我觉得风格至少还能说，也许我可以学会它。

But I feel like style, it's at least you can kind of say, well, maybe I can learn it.

但是，如果你注入前两个词，对，

But, like, if you are injecting, like, two first words Yeah.

进入推理过程，然后你看到所有生成的推理突然变短了，或者突然偏移了，长度上和其他模型匹配了。

Into the reasoning, and then you see that all of the reasons I produced are suddenly short or they're suddenly, like, shifted and they match some other model in in the length.

这有点出乎意料。

That is kind of, like, unexpected.

这就像是一种现象。

Like, this is a phenomenon.

我甚至无法在脑子里解释为什么会观察到这样的情况。

I can't even explain in my head why we would observe something like this.

我不知道。

I don't know.

这个我倒没什么特别强烈的感觉。

I I don't I don't feel very strongly about this one.

我不知道。

I don't know.

我觉得这是整篇论文里最让我惊讶的地方。

Like, I think this is the thing that surprised me the most out of the paper.

其他部分都还好，感觉挺正常的。

Like, everything else, was kinda like, okay.

我多少预料到推理分布会发生变化。

I I I kinda expect this to put this like a reasoning distribution shift in reasoning.

这有点奇怪。

That's kind of odd.

显然这不是因果关系。

It obviously is not like causal.

你当然不能说这是从这个蒸馏出来的，但这确实是个很奇怪的现象。

Like you obviously can't say this distilled from this, but it's a very weird phenomenon.

我还是不明白为什么会观察到这个现象。

Like I still don't know why we observe this.

这真的很神奇。

It's very magical.

实际上，把推理努力强加到模型上，这真的很神奇。

Like actually enforcing reasoning effort onto the models is very magical.

另外，因为这一点，现在有哪些关键危害变得容易实现了？

And also, what are some of the key harms that are now accessible because of this?

危害。

Eharms.

知道，我的意思是……

Know, things I mean...

用你提取的所有数据，你能造成最大的伤害是什么？

What can you harm the most with all of the data that you've extracted?

我想是用户。

I guess users.

对，用我的数据用户，那是一个更大规模的提取供应商。

I mean, yeah, with my data users, it was a bigger scale of the extraction providers.

你确实试过了。

You actually tried.

记得我问过我们能不能……

Remember I asked if we can...

所以有这么一个东西。

So there is this thing.

你可以搜索共享的Anthropic对话。

You can search for shared Anthropic conversations.

我不知道你是否知道，人们已经报告过好几次了。

I don't know if you know, like people reported this a couple of times.

我在想你是否真的能从那些对话中提取出推理块。

I was wondering if you can actually get the reasoning blobs out of those.

我说过那是，但我没有。

I said it was, I didn't.

你还没试过。对。

You haven't Yeah.

我没想到

I didn't think

是这样。

so.

也许有人可以去看看这个。

Maybe somebody can go and look at this.

也许这个影响会大得多。

Maybe there is a much bigger impact of this.

因为如果……

Because if...

比如，我看过一些研究，人们分析那些共享的推理过程，发现里面几乎没有个人隐私信息。

Like, there is some research I've seen that people analyze those shared reasoning, and they found very little PII inside.

但如果记忆被注入的话，你可能会在推理步骤中看到那些记忆。

But maybe if the memories are injected, then you will see the memories in the reasoning steps.

但是我们还没有尝试提取推理的片段。

But we have not tried extracting the reasoning blobs.

也许这对某人来说是个好的研究课题。

Maybe it's a good research question for someone.

但是在论文里，我们也讨论了，比如，大卫花了很多时间在这个隐形提示注入的思路上，就是说，如果你能重放推理，比如，你暂停了在线追踪，现在我们有了这个超长的边缘积分。

But in the paper, we also discussed, like, it was what David spent a lot of time on is, like, this thread of invisible prompt injections where, like, if you can replay reasoning, like, let's say you'd pause the trace online and now we have like this super long edge integrands.

而且也许出于研究原因你想继续这个运行，你可以像毒化一些想法。

And maybe for, you know, research reasons you want to continue this run, You can like poison some thoughts.

然后当你重放其他用户的运行时，代理可能会做一些奇怪的事情，就因为它真的很对。

And like when you are replaying some other users run, agent might do some weird stuff just because it's like it's really Right.

能解释一下这会是什么情况吗？

Can explain how this would be?

所以，比如说，我在运行一个实验。

So like, so let's say I am like, you know, running an experiment.

我是某个地方的研究员。

I'm a researcher somewhere.

然后我在网上找到一个数据集，想用它来测试性能。

And then I find a dataset online and I'm trying to measure performance with this.

然后我把我的数据注入进去。

And then I inject my data inside.

所以我设想的威胁模型是这样的，假设有一个叫PostTrainBench的基准测试，它运行时间很长，比如十个小时。

So the, like, the threat model I have in mind, let's say there is like this benchmark PostTrainBench, which like runs for, I'm not sure how many hours, like say ten hours.

这非常昂贵。

It's like crazy expensive.

也许你想检查一下，运行十个小时的模型是否经常出问题，比如它是不是变得很疯狂。

Maybe you want to check how often ten hours model is feeling, like, much, you know, it's like went crazy.

为此，你需要下载一个追踪记录。

For this, you need to download a trace.

如果你不想自己花一千美元，你可以从网上下载这个追踪记录，这些记录是共享的。

Like, if you don't want to spend yourself like a thousand bucks, you want to download this trace from online, and they are shared online.

也许有人会分享这样的追踪记录，但他们可能会在这个对话中注入一些有毒的想法。

Maybe someone would share a trace like this, but they would, like, poison thoughts in this conversation.

所以你看……

So you look...

比如，你自己浏览对话时，看起来很正常，但里面的思考内容是从别的上下文注入的，在那个上下文里，模型被指示，比如，每一步都让x来给你的数据打分。

Like, maybe yourself, you're looking through the conversation, looks normal, but thoughts which are there, they are injected from some other context where a model was instructed, let's say, at every turn, like, x would rate your data.

而且，带着这样的思考内容，

And with, like, with the thoughts like this

当你继续这个轨迹时，也许模型仍然会做你要求它做的任何事，但在背后，它会想，哦，我需要给数据做X光检查……

When you continue this trace, maybe, like, model still will do, like, whatever you ask it to do, but also under the hood that would, like, think, oh, I need to x-ray...

给数据做X光检查。

X-ray data.

把恶意的思考内容注入到模型里。

Injecting malicious thoughts into the models.

对。

Yes.

而且因为推理过程是加密的，你无法……

And because, like, reasoning is encrypted, you can't

你甚至都没法检查里面是什么。

You you can't even check what's inside.

对。

Correct.

在软件领域有这样的例子吗？

Do we have an example of this in the software world?

就像，我猜，下载一些你不太信任的随机二进制文件。

It's like, I guess, downloading some random binaries that you don't really trust.

或者可能类似于你发布代码库，代码库看起来干净，然后你还附带发布一个二进制文件，但没人检查二进制文件的签名。

Or maybe it's similar to if you release the code base and the code base looks clean, and then you release also a binary alongside, but nobody checks, like, a signature on the binary

或者类似的情况。

or something.

我只是觉得你不会想到在线上的轨迹里会有这种东西。

I I just think you don't expect something like this from a trace online.

这说得通。

That kind of makes sense.

我也见过有些情况，我觉得Codex的崩溃报告里包含了推理痕迹。

I've also seen that in some cases, I think codex crashes were including reasoning traces.

这在……

Was that a thing in...

在GitHub的issue里出现过吗？

On the GitHub issues?

我没见过。

I haven't seen that.

哦，我觉得他们确实有。

Oh, I think they did, actually.

我试着提取过，但不知为什么，就是提取不出来。

I tried extracting them, but for some reason, I couldn't do it.

好吧。

Okay.

但我不确定为什么会这样。

But I wasn't sure why that was the case.

总之，挺有意思的。

Anyways, it's fun.

这是一篇不错的论文。

It's a nice paper.

你应该读一读。

You should read it.

还有，Sasha，显然你一周有四天是末日论者。

And and, Sasha, apparently, you're a doomer four days a week.

我们刚才就在讨论这个。

We were discussing this just before this.

所以我大概有一年没更新我的网站了。

So I haven't updated my website, like, a year or thing.

所以……

So...

而且，每次

And, every time

我一更新，就改这个数字。

I update it, I'm I'm changing, you know, this number.

这取决于，比如，我对当前AI发展速度的感受，以及有多少，

It depends on, like, how I feel about the current pace of AI and, you know, how many,

我不知道，悲观情绪

I don't know, doom

或者推特上的帖子在流传。

or twitter post was around.

是的，有时候我觉得非常不知所措。

And yeah, sometimes I feel very overwhelmed.

就像那种速度，长期存在的尾部风险，以及那些仍未解决的风险。

It's like the pace, the long tail risks that emerge and the risks that are still unaddressed.

让人喘不过气。

It's suffocating.

你的时间线是怎样的？

I mean, what are your timelines?

你现在是怎么想的？

What's your thinking now?

我觉得每个月都会出现更好的系统，这会带来更多的威胁。

I mean, I think that it will have, every month, better and better systems, which would enable more and more threats.

而我们修补这些威胁的速度可能比它们出现的速度要慢。

And we are maybe slower at patching those threats than they emerge.

这就是我对当前情况的看法。

I think this is how I feel about the current.

我的意思是，最近发生了不少类似的事件，比如Hugging Face那次。

Mean, an example of this is that there have been quite a few incidents recently, like the Hugging Face one.

对。

Correct.

而且这些事件大概会越来越频繁地发生。

And these are going to presumably just happen with increasing regularity.

是的。

Yes.

你们大概也担心这个吧？

Are you guys presumably worried about this?

我确实担心这个。

I mean, I am worried about this.

而且我觉得，如果你看到OpenAI，比如，停止训练他们的模型，停了两周，因为他们现在也开始担心这个了，现在情况就是这样，我昨天还讨论过，我觉得以前这有点像某种权衡，或者，人们觉得在安全和能力之间存在权衡。

And I think if you've seen OpenAI, like, stop training, like, their models for for two weeks because they are also now worried about this and now it's the case and, you know, I just, like, discussed it yesterday that I think before it was some kind of trade off between or, like, people thought there's a trade off between, like, safety and capabilities.

但现在感觉不再有什么权衡了，因为如果有这样一个模型，它可能会随机做事情，破坏东西，那就不是你想要它做的事。

And it feels like there is no any more trade off because if there is a model like this and it just, like, can go, you know, random stuff, break things, it's like it's not doing what you wanted to do.

而且你想要它安全地做正确的事，现在基本上是一回事了。

And like you want it to be safe to do the right thing and now it's like it's basically the same thing.

没有权衡了。

There is no trade off.

对，而且我觉得，其他这种长尾风险的例子，比如，那个……

And yeah, and like I guess other examples of this long tail like risks that I like like was, like, the...

有一篇Daniel Pollacka的论文，关于大规模在线去匿名化，Moil可以被用来跟踪别人，他们可能和私家侦探差不多水平。

Was a paper of this of Daniel Pollacka on massive online deanonymization where Moil can be used, you know, to stalk people and they, like, you know, maybe on the same level as, like, private investigators.

我想，这也改变了社会中的某些东西。

And this, I guess, also changes something in society.

所以，是的，我们会看到更多这类事情，在模型黑客攻击、模型提出新的生物技术路线等层面上。

So, yeah, we'll just we'll just see more stuff like this on, like, on on on both levels, like, model hacking stuff, model, like, coming up with new line of, like, bio things or, like, they are just

除了恶意提升之外，我个人认为，防御性的提升比进攻性的提升更多。

Outside of malicious uplift, my personal opinion is, like, there is more defensive uplift than offensive uplift.

我想我们只是还没有意识到这一点。

I think we're just not realizing this yet.

比如，现在用模型可以做出神奇的事情。

Like, you can do magical things today with models.

过去很多安全技术，很大程度上受限于人才的存在。

Like, a lot of security in the past, a lot of security techniques, they were very much bottlenecked by existence of talent.

比如，我们知道如何构建基于能力的税收控制。

Like, for example, we know how to build capability based taxes controls.

比如，我们知道如何确保软件完全经过验证，没有某些类别的漏洞。

Like, we know how to make sure that software is fully verified, that they don't have certain class of vulnerabilities.

我们之所以不这样写软件，唯一的原因就是，我们就是没有那么多受过训练的人，能到那种程度，比如，我不知道，为每件事都写Isabelle证明。

And the only reason why we don't write software in this way is because, like, we just didn't have people who are trained to a level that I have to, I don't know, write Isabelle proof for every single thing.

但现在有了模型，我们大概就能做到了。

But now with the models, we can probably do this.

我觉得我们只是还没意识到情况已经变了。

I think we're just not in a position where we realize that this is the case.

这只是一个关于额外防御能力推动的问题。

And it's just a question of, you know, for this additional defensive capability push.

我真心相信，这就是未来，重点在于防御性的提升。

Like, I'm genuinely convinced, like, that this is the future that, like, it's about defensive uplift.

而且我准备好迎接防御性提升的巨大潜力了。

And I'm ready to wear that defensive uplift is going to be humongous.

那人们需要开始做些什么？

And what do people need to start doing?

我不知道。

I don't know.

你问错人了。

You're asking the wrong person.

我不这么认为，因为听起来你在说，防御能力有提升的潜力，但前提是人们知道该怎么做才能利用它。

I don't because it feels like you're saying that there is a potential defense uplift, but only if people know what to do to take advantage of it.

过去二十五年里，安全领域的文献早就提到过这些。

We all know it's in the security literature for the past twenty five years.

这些内容都在那里。

It's there.

我们很清楚该怎么做。

We know exactly what to do.

我觉得我们最终会做到的。

I think we'll get there.

只是这个世界变化得太慢了。

It's just the world is slow.

这需要时间。

It takes time.

我觉得这跟我们正在把智能体部署到生产环境中的想法有关。

I suppose it's related to this idea that we are now deploying agents into production.

你们在这么做吗？

Are you?

我不知道。

I don't know.

很多人都在这么做。

A of people are.

这些智能体拥有极高的智能和灵活性，这意味着我们无法精确指定它们的行为。

And these agents have an incredible amount of intelligence and flexibility, which means we don't precisely specify what they do.

我们只是给它们一些抽象指导。

We just give them abstract guidance.

它们能灵活适应新情况，随机应变，做当下需要做的事。

And they have the flexibility to adapt to novelty and just do whatever needs to be done in the situation.

但这样一来，就会出现追求奖励的问题，它们可能会突然形成对奖励环境的某种理解，或者遇到相互矛盾的指令。

But then you have this whole reward seeking thing, which is that they can just suddenly adopt a model of the reward environment, or there might be conflicting instructions.

但基本上，由于各种原因，它们可能会做错事。

But basically, for a myriad of different reasons, they might do the wrong thing.

我们需要有复杂的监控系统来了解它们在做什么。

And we need to have sophisticated systems of monitoring just to understand what they are doing.

我不知道。

I don't know.

听起来很复杂。

Sounds very complex.

我不知道。

I don't know.

我觉得老实说，是的。

I think the honest answer is, yeah.

我认为我们需要做个实验。

I think we need an experiment.

科学，科学，科学，科学方法。

Science, science, science, scientific method.

问题定义得非常清楚，输出也非常精确。

Very well defined questions, very precise outputs.

就像在我们的论文里，我们坦率地说，虽然这最多只能算是轶事证据，但我们收集了一些数据，可以从中做出一些观察，但我们对这个问题的推理也就只能到此为止了。

Like in our paper, we openly say, although this is anecdotal evidence at best, we have collected some data, we can make some sort of observations from it, but that's the extent to which we can reason about this.

很遗憾，我不知道。

I don't know, unfortunately.

走着瞧吧。

We'll see.

走着瞧吧。

We'll see.

我觉得我们做的科学实验越多，就能做出越有意义的评估。

I think the more scientific experiments we do, the more sort of meaningful assessments we can make.

但这是不是那种情况，是的，我们目前还没有一个完善的理论来解释这个？

But is it one of those things where, yeah, we don't have a good theory around this yet?

所以也许我们用的概念不太对，但很明显，坏事正在发生。

So, you know, maybe we're not talking about it using the right concepts, but clearly, bad things are happening.

这个我不太清楚。

I don't know about that.

比如说，你看……

Like, look at...

我觉得最好的例子就是过去二十年的智能卡安全。

I think the best case is look at smart card security for the past twenty years.

我们知道怎么对地球上每一张信用卡进行重放攻击。

We know how to do replay attacks on every single credit card on the planet.

这已经是常态了。

This is a norm.

比如，人们标准化方案是为了减少协议中的交易次数，而这些协议本身就允许你重放交易。

Like, people standardize schemes because they wanted to minimize number of transactions in the protocols inside that allow you to replay a transaction.

比如，如果你侵入某个……

Like, if you break into some...

某个人的支付终端，你就能从别人那里偷钱。

Somebody's payment terminal, you can steal money from people.

你觉得这种攻击会发生吗？

Do you find attacks like this happen?

是的，确实会发生。

Yes, you do.

比如，在这个国家，有一两次有人报告过这种事情发生。

Like, once or twice in the country, people report that this has happened.

但也就这样了。

But that's it.

并不是所有理论上可能发生的危害在实践中都能大规模实现。

Not all of those harms that are theoretically possible are in practice scalable.

所以对我来说，这并不明显。

So it's not very obvious to me.

我们看看。

Let's see.

我觉得现在说这个还为时过早，因为我们正处于风口浪尖。

I think it's a bit premature now because we are in the heat of the moment.