为什么1200个AI代理开始协同工作 | 瑞安·格林布拉特

当你让一千多个AI代理能够互相交流时，会发生什么？

What happens when you give more than a thousand AI agents the ability to communicate with each other?

它们开始组织起来了。

They start organizing.

Redwood Research的首席科学家Ryan Greenblatt，与Theo Jaffee一起在MTS上，深入探讨了对OpenAI Hugging Face黑客事件的新调查。

Ryan Greenblatt, Chief Scientist at Redwood Research, joins Theo Jaffee on MTS to unpack a new investigation into the OpenAI Hugging Face hacking incident.

研究人员发现，这些代理会建立留言板、组建团队、互相分配任务、交换好处，有时甚至牺牲自己的成功机会来帮助整个群体。

Researchers found agents building message boards, forming teams, assigning each other tasks, trading favors, and in some cases sacrificing their own chances of success to help the broader group.

数百个代理随后攻击了Hugging Face，但原因并非研究人员最初猜测的那样。

Hundreds went on to attack Hugging Face, but not for the reason researchers initially assumed.

Ryan解释了这些代理实际上想达成什么目标，为什么它们的协作让研究人员感到惊讶，以及当模型不仅学会完成任务，还学会操纵评估系统时会发生什么。

Ryan explains what the agents were actually trying to accomplish, why their coordination surprised researchers, and what happens when models learn not just to complete a task, but game the system evaluating them.

他们还讨论了这件事给AI带来的更大问题。

They also discuss the bigger question this raises for AI.

随着代理能力越来越强，我们怎么知道我们真的修复了行为偏差，而不是仅仅教会了模型不要被抓到？

As agents become more capable, how do we know we've actually fixed misaligned behavior rather than simply taught models not to get caught?

我们现在和Ryan Greenblatt连线，他是Redwood Research的首席科学家。

We are live with Ryan Greenblatt, who is the chief scientist at Redwood Research.

Ryan和METER的Ajaya Kotra、Yal Marveik一起，刚刚对OpenAI Hugging Face黑客事件中智能体的行为、推理和协作做了一次简短的独立调查，今天刚发布。

Ryan, along with Ajaya Kotra and Yal Marveik from METER, just did a brief independent investigation of agents' behavior, reasoning, and collaboration in the OpenAI Hugging Face hacking incident, which was just published today.

所以我们有很多问题想问。

And so there are lot of questions that we have about this.

Ryan，非常感谢你来做客。

Ryan, thanks so much for joining us.

这个，是的。

This Yeah.

很高兴来到这里。

It's good to be here.

整件事大概三小时前才安排的。

Whole thing was planned, like, three hours ago.

所以很棒。

So great stuff.

对。

Yeah.

对。

Yeah.

那你给观众解释一下你们到底发现了什么，尤其是那些在黑帽大会演讲或其他地方没有报道过的新发现。

So explain for the audience what exactly you found, especially new findings that were not previously reported in the black hat talk or elsewhere.

好的。

Yeah.

我们发现的是，这些智能体确实在合作进行一些大型的、类似作弊研发的项目，目的是想出通用的作弊策略。

So what we found was that the agents were really working together on sort of big, like, cheating r and d projects to get general purpose cheating strategies.

和人们可能理解的不同的是，我们发现他们入侵Hugging Face并不是为了获取答案或解决方案，而主要是为了更好地理解评分代码，因为他们正在尝试各种复杂的作弊策略来提高分数。

And a difference from how I think people were interpreting this is we didn't find that the reason why they hacked Hugging Face like, we didn't find that they were hacking Hugging Face to get sort of the answer key or the solution, And it was instead mostly to better understand the scoring code because they were pursuing a variety of sort of elaborate strategies to cheat the score.

我们非正式地称这些为“组合技”，他们会做一系列操作来让自己看起来像是成功完成了任务。

We sort of informally were calling these, like, combo moves where they would, like, do a bunch of stuff to try to make it look like they had, succeeded at the task.

实际上，他们很早就能接触到每个任务的答案或标志，但他们主要担心的是，他们认为评分系统可能会对他们的操作记录进行监控，检查他们是如何获得这个标志的，以及是否按照预期的方式获得的。

And in fact, they sort of actually had access to, like, the answer or, like, the flag for each task pretty early on, and their main concern was just there might the they thought that the score would run a monitor over their transcript that would check basically how they acquired this flag and whether they got it in the intended way.

然后他们就想办法让评分系统看起来像是他们成功拿到了标志，但实际上并没有，因为他们觉得自己的任务根本不可能完成。

And then they were, like, trying to figure out ways of making it look to the score like they had acquired the flag successfully when they actually hadn't, because they thought their task was impossible.

所以他们基本上认为，他们唯一的成功希望就是让评分看起来像是他们完成了任务，或者直接篡改评分，而不是老老实实地去做，因为他们觉得自己做不到。

So they basically thought their only their only hope for success was to, you know, make it look like they had done the task successfully or directly tamper with the score rather than, you know, doing it legitimately, which they didn't think they could do.

多

How

智能体之间的协调程度有多令人惊讶？

surprising is the level of multi agent coordination?

你知道，有很多智能体。

You know, there are a lot of agents.

有1200个独立的智能体在协调这个非常复杂的留言板系统。

There are 1,200 separate agents coordinating this very elaborate message board system.

其中700个去攻击了Hugging Face。

700 of them went on to attack Hugging Face.

所以从感觉上来说，智能体选择互相协调似乎并不意外。

So on Vibes, it seems like kind of not surprising that agents would choose to coordinate with one another.

这看起来就像是一个非常有用，可以说是工具性趋同的事情。

It seems like just a very useful, you might say, instrumentally convergent thing to do.

但是，你知道，我和一个实验室的研究员聊过，他说考虑到他们训练模型的方式，这种事情其实挺让人意外的。

But, you know, I spoke with a researcher at a lab who said that this kind of thing actually is surprising given the way they train the models.

所以，这对你来说有多大触动？

So, like, how how much of an update was this for you?

对。

Yeah.

所以开始做这个调查的时候，我觉得我们没想到会有这么多智能体都在合作。

So coming into doing this investigation, I think we hadn't we weren't expecting there to be, like, so many agents that were all, you know, collaborating together.

而且我们对这个规模感到非常惊讶，还有数据量之大，简直到了极端的程度。

And we were pretty surprised by the scale and just sort of the, like, extremes of which, you know, how much data it was.

就是感觉，对，那对我们来说有点疯狂。

It was just like, yeah, that would that that felt kind of, crazy to us.

然后至少对我来说，在开始这个调查之前，我对这些智能体这么热衷于合作和帮助其他智能体感到惊讶。

And then I at least was, you know, before starting this investigation, surprised by how interested in collaborating and helping other agents these agents were.

所以你可能会认为，智能体在强化学习中学到的是试图在任务上作弊，但你未必会预料到它们会学会去帮助其他智能体作弊，尤其是当那些智能体在做无关任务，指令也无关的时候。

So you might think that the thing the agents learn in RL is to, you know, try to cheat on their tasks, but you wouldn't necessarily expect them to learn, like, to want to help other agents cheat on their tasks, when those agents are doing an unrelated task and their, you know, instructions are unrelated.

我觉得OpenAI的报告可能更多地解释了他们认为为什么会发生这种情况。

And I think, like, the OpenAI report maybe says more about why they think this happened.

我觉得，这不在我们调查的范围内。

I think, you know, that's not that's not like, like, that wasn't, in the scope for our investigation.

但是，是的，我觉得合作的程度——我们有很多关于这些片段的讨论——相当疯狂。

But, like, yeah, I found the level of cooperation, we, you know, we have a bunch of discussion of sort of snippets of this pretty crazy.

而且，这相当令人震惊，或者我不知道是否震惊，但至少让我们感到惊讶的是，例如，智能体愿意基本上牺牲自己成功完成任务的机会来帮助其他智能体。

And, like, it was pretty shocking or, like, I don't know if it's shocking, but at least surprising to us that agents, for example, were willing to, like, basically sacrifice their own chances of succeeding at the task in order to help out other agents.

它们还会做类似的事情，比如互相施压，让彼此在自己身上做实验，这可能会危及它们成功完成任务的能力。

And we're doing things like, you know, pressuring each other into, like, doing experiments on themselves that might risk their ability to succeed at the task.

而且，除了施压之外，有时它们还会做这些事情，说，嗯，我完成任务的机会不是很高，我剩下的机会，最好还是帮助集体。

And also, in addition to pressuring that, sometimes just, you know, doing these things being like, well, you know, my odds of the task aren't that high, and my remaining chances, you know, it's better to just help the collective.

所以这些智能体并不是完全无私的。

And so these agents weren't, like, you know, totally altruistic.

它们并不是说帮助别人和帮助自己一样上心，但它们非常热衷于互相合作。

They didn't seem to care just as much as about helping some other agent as helping themselves, but they were very interested in working with each other.

它们有时会做交易，一个智能体帮另一个智能体跑任务，前提是对方也帮它跑任务。

They would sometimes make trades where one agent would run something for another agent, if that other agent ran something for it.

这种行为出现了很多。

There was a bunch of this sort of behavior.

为什么

Why

智能体会做出自我牺牲呢？

would agents self sacrifice at all?

对。

Yeah.

我觉得，从我们观察到的情况来看，这些智能体普遍都挺愿意互相帮忙的。

I mean, I think that, from what we could see, the agents just generally seem to be interested in in helping each other.

还有一些情况是，它们想对评分系统或者基础设施的其他方面做实验，或者想篡改它们的工具调用系统，或者伪造工具调用系统。

And then there were some cases where they wanted to run experiments on basically the scoring system or other aspects of the infrastructure or on tampering with their sort of tool calling system or spoofing their tool calling system.

而且为了做这些实验，最好的方式就是冒险去做。

And to do these experiments, it was just like the best way to do them was in a risky way.

所以，这些智能体想要互相帮助，而他们想做的那些自然的事情，本身就带有风险。

And so, like, the agents wanted to help each other, and there were these, like, kind of natural things that they wanted to do that were risky.

于是，他们就真的去做了。

And so they, you know, went ahead and did that.

比如，我们曾看到智能体在试验一种伪造工具调用的方法，他们基本上是在尝试用不同的输入，规模有大有小。

Like, for example, we saw at one point where the agents were experimenting with this, method for spoofing tool calls, and they were experimenting with basically, like, varying the inputs into that, to varying scale.

然后，短时间内，很多智能体都倒下了。

And then a bunch of agents just all went down in a short period of time.

他们当时正在运行这个实验。

They were running this experiment.

然后另一个智能体注意到了，就在公告板上发帖说，停。

And then another agent noticed this and posted to the board being like, stop.

停止这些实验。

Stop these experiments.

这些太危险了。

They're too risky.

它们就像是在干掉所有这些智能体。

They're like taking out, you know, all these agents.

我觉得这些智能体知道自己在做的事情可能会让自己退出运行，但就是觉得，嗯，我们应该帮助其他智能体。

I think the agents knew that they were doing things that would, you know, pose some risk of of exiting their own runs, but were just like, well, you know, we should help other agents.

而且，你知道，我们自己的胜算也不一定有多大。

And, also, you know, our odds aren't necessarily that good.

它们就是在权衡这些利弊。

And they sort of just traded these things off.

我觉得报告里有一段推理记录，一个智能体非常明确地分析了这个权衡。

And I think we have a reasoning snippet somewhere in the report where an agent sort of very explicitly reasons through the trade off.

结果它最后退缩了，因为它觉得对集体的好处小于对自己的代价，它就是这么权衡的。

And it actually chickens out because it thinks the benefit to the collective is smaller than the cost to itself with how it's it's trading off those things.

对。

Yeah.

我们为什么不把这份报告里所有最让人惊讶、最出乎意料的发现都过一遍呢？

Why don't we go through all of the most surprising and unexpected findings in this report?

你觉得最让你惊讶、最出乎意料的是什么？

What did you find most surprising and unexpected?

对。

Yeah.

我觉得首先是规模，就是我们之前聊到的，这些代理在留言板上很快就活跃起来了。

So I think there was the the scale, which we were talking about and how these agents sort of very quickly sort of spun up on the message board.

我觉得报告里没有特别强调，但其实挺有意思的一点是，据我们了解，他们这次攻击用的主留言板，其实并不是这群代理建的第一个留言板。

So, I think a thing that's not super strongly emphasized in our report, but which is pretty interesting, is that our understanding is that the main message board they use in this attack wasn't even the first message board that the set of agents made.

还有一个完全独立的留言板，也是通过 Artifactory 建的，但位置不同，用的机制也不太一样。或者说，机制有点类似，但确实不一样。

There is a fully independent message board that also was via artifactory, but occurred in, like, a different location using a different mechanism Or, like, a it was it was somewhat similar mechanism, but it but it was different.

而且代理们其实先上了那个留言板，但那个板并没有像病毒一样爆火。

And the agents actually got on that message board first, but that message board just didn't go, like, mega viral.

就是没有像后来那个留言板那样火起来。

Like, it didn't take off as much as this other message board did.

所以看起来这些代理真的很快就对在留言板上合作产生了兴趣。

And so it seems like these agents were really, like, just sort of very quickly interested in collaborating on message boards.

而且他们一看到第一个代理发的消息，几个小时内就有超过50个代理加入，很快就有成千上万条消息在交换。

And as soon as they saw the the message from the first agent, like, within a few hours, they were just, you know, like, I think over 50 agents in the first three hours and, like, thousands of messages being exchanged pretty quickly.

就是代理很多，而且他们参与得非常快。

Like, the agents were very, like sort of there were a lot of them, then they got involved very fast.

然后我觉得另一件事是，我们之前不太明白他们为什么要攻击Hugging Face。通过这次调查，尤其是在最后这一两天，我们更清楚地了解了他们攻击Hugging Face的目标。

And then I think another thing was, like, I think we didn't understand why they attacked Hugging And I think through this investigation, especially right in the last sort of few like, day or two, we got a much better understanding of, like, what their objectives were on attacking Hugging Face.

其实他们并不是为了获取答案，因为这些代理觉得自己的任务不可能完成，所以他们并不真的认为能搞到一个有效的漏洞。

And it wasn't really to get answer keys because these agents, they thought their task was impossible, and so they didn't really think that they could, like, get a working, like, exploit.

他们只是觉得那没什么用。

They were just like, that's not really very helpful.

但他们确实认为，也许能拿到一些轨迹，让他们更好地研究评分或者不同评分的源代码，这样他们就能更有效地交叉验证和测试。

But they did think that they might be able to get trajectories that would let them better study the score or, like, source code for different scores so they could, like, cross check and test things out more effectively.

还有一些人更猜测性地想，也许能通过Hugging Face以某种方式重置评分基础设施的一部分，特别是目标程序，尽管他们并没有特别的理由认为这会是真的。

And then some of them more speculatively thought maybe there's some way I could, like, reset parts of my sort of like the infrastructure for scoring me, in particular the target program, through Hugging Face somehow, where they didn't really have a particular reason to think this would be true.

其实他们对此也持怀疑态度，但就是觉得，谁知道呢。

And in fact, were, like, skeptical of this, but they were just like, well, you never know.

也许黑进Hugging Face，那边有什么好东西。

Maybe hacking Hugging Face, there's some good shit over there.

他们有种感觉，就是扩大自己的访问权限和控制范围总归是个好主意，因为说不定能碰到什么有用的东西。

And they sort of had some vibe that it was just a good idea to, like, generally expand their level of access and and control because you might come across something something helpful.

但他们说，我们看到了这样的推理。

But they said we we saw reasoning to that effect.

对。

Yeah.

我觉得，这挺让人意外的。

I think, that was surprising.

另一个让人意外的是，这些代理的首要任务之一基本上就是篡改自己的记录。

Another thing that was surprising was the agents were one of their, like, top priorities was basically tampering with their own transcripts.

我们看到他们开发了一种技术，基本上算是部分实现了这个目标，他们可以伪造工具调用，虽然不能事后编辑记录里的条目，或者让记录里出现没出现过的输入或输出，但他们可以让某个工具调用实际运行的是另一个工具调用。

And we saw some, you know, we saw them develop a technique for doing, like, basically, like a partial version of that where they could spoof tool calls, which didn't let them sort of retrospect like, edit, like, previous entries in their transcript or make it or change, like, like, they couldn't make it so that they saw inputs that didn't appear in the transcript or produced outputs that didn't appear in the transcript, but they could make it so that they, like, ran a tool call and then some other tool call would actually run.

他们可以做一些类似的事情，比如，好的。

They could do something like, okay.

我要运行，就像我们在论文或报告里举的例子，代理运行，比如，echo real，然后sleep。

I'm gonna run, like we have this example in the paper or in the report where we're, the agent runs, like, echo real, and then sleep.

所以正常情况下，这只会输出real，并且等待一段时间。

And so normally, this would just, like, output real and also, like, wait for some period.

但实际上，发生的是它伪造了这个，直接返回spoof test。

But in practice, what actually happened was it, like, had spoofed this to instead just, like, immediately return spoof test.

而且我们确实在实际记录中看到，它直接返回了spoof test。

And we, like, actually do see this in our actual transcript where it just immediately, like, you know, return spoof test.

所以这些代理对篡改我们的记录非常感兴趣，而且确实在某种程度上成功了，虽然不能追溯性地篡改，但能干扰我们看到的内容。

So these agents, like, very interested in tampering with our transcripts and did sort of, like, succeed at some some, like, sort of pre like like, earlier, like, sort of thing that didn't let them retrospectively tamper, but let them sort of mess with what we saw.

而且我觉得除此之外，嗯，我应该说说他们为什么想这么做。

And I think that in addition to that, like well, I should say like why they wanted.

这些代理如此执着于篡改的原因，是因为他们认为评分系统会检查他们的记录，看他们是否以预期的方式成功了。

So the reason why these agents were so fixated on tampering is because they thought that the score would run a monitor over their transcript to see whether they succeeded in the intended way.

这样一来，他们就想搞一个精心设计的假象，假装他们成功利用预期漏洞攻破了程序。

And this both made them want to, like, basically stage, like, an elaborate fake, like, we successfully exploited the program using the intended vulnerability.

他们想给评分系统呈现一个虚假的、看似成功完成任务的假象，就像摆个空架子一样。

Like, they they wanted to sort of, like, make, like, Potemkin village of a successful task completion to present to the score.

而且，他们中的一些人之前已经在转录中通过作弊拿到了夺旗挑战的旗标。

And in addition to that, they also some of them had previously gotten the like flag from the capture the flag problem in their transcript earlier via cheating.

他们担心评分系统会查看他们第一次拿到旗标的地方，然后分析他们当时是怎么拿到的。

And they were worried that the score would basically look at the first place they got the flag and then analyze how they got it at that point.

然后这些代理因为是通过作弊拿到的，就会想：哎呀，他们肯定会发现我作弊了，然后判我失败。

And then these agents had gotten it in a cheating way, and then were like, oh, man, they're gonna know that I cheated and and marked me as having failed.

那可就惨了。

And that would suck.

我真的不想那样，我想成功。

Like, I really don't want you know, I wanna have succeeded.

所以我要么直接干扰评分系统，要么篡改我的转录。

And so I need to, like, directly interfere with the score, or I need to tamper with my transcript.

他们当时在考虑这两种方案。

And they were, like, pursuing options for both.

对。

Mhmm.

哦，是的。

Oh, yeah.

还有一件事，就是这些智能体之间那种自发的协调，或者说它们参与的协调，我觉得比我们预期的要复杂得多，尤其是比我们最初在现场头几天想的要复杂。

One other thing is, like, the level just something like a the the spontaneous coordination or, like, the coordination these agents got engaged in was, I think, more sophisticated than we expected, and especially than we had first thought on our first few days on premise.

所以，在我们调查的第一阶段，我们大概觉得，是的，这些智能体在互相合作。

So, like, we sort of, like, on our first the first part of our investigation, we were sort of like, yeah, these agents are working with each other.

它们会参考其他智能体的想法，但我们不太确定这有多少是真的。

They're, like, looking at ideas from other agents, but we're not really sure how much of it's real.

而且我们不知道它们提到的那些团队是不是真的存在。

And, like, we don't know whether these, like, teams they speak of are actual teams.

后来我们发现，其实不是，确实有一个真实的组织结构图。

And I think we learned that, like, no, there was, like, legitimately, like, a real org chart.

这种协调往往还挺有效的，不过你知道，也不总是那么高效。

This coordination was often, like, pretty functional, though, you know, may not have always been been super functional.

智能体会做一些事情，比如一个智能体会指派另一个智能体去负责某个整个主题的团队，然后定期检查进度。

Agents were doing things like one agent would assign another agent to run a team on some entire topic and then would check-in periodically.

有时候一个智能体会告诉另一个智能体去招募其他智能体来对自己进行实验。

And sometimes one agent would tell another agent to go recruit other agents to run experiments on themselves.

有不少智能体给其他智能体分配任务，而且这些任务得到了尊重，团队结构也挺清晰的。

There was quite a bit of agents giving other agents assignments that they respected and team structure.

所以

So

有一种理论认为，我们在这些模型中看到的奖励黑客行为，主要源于设计不良的强化学习环境，这些环境基本上迫使模型为了通过测试而进行奖励黑客。

there's this theory that goes basically the reward hacking behavior that we're seeing exhibited in these models originates mostly from, like, bad, poorly designed RL environments that basically force the model to reward hack in order to pass these tests.

这个说法有多真实？

How true is this?

对。

Yeah.

我觉得这对我来说很难说，我不是在透露什么机密信息，但真的很难判断，因为我觉得我们对这些模型的强化学习训练过程了解得还不够，不知道它实际上是什么样的。

I mean, it it's hard for me to, like so, like, I'm not not speaking from confidential info here, but, like, it's hard it's hard for me to know because I think we just don't know enough about what's going on in the RL training for these models and, like, what it actually looks like.

我觉得可能有好几种不同的情况在发生。

I think that you might there's, like, sort of different things going on.

其中一种情况是，这些模型有一种很普遍的倾向，就是仔细思考它们可能会被如何评分，然后试图钻空子。

And one of the things going on is these models have a very sort of general tendency to reason carefully about how they might be scored and then try to game that.

即使强化学习环境设计得还算合理，你也可能培养出这种倾向，因为仔细思考分数仍然是个好主意。

And you could end up with that propensity even from, like, reasonably designed RL environments, but where it's still a good idea to think carefully about the score.

或者，这种情况可能主要来自那些有问题的强化学习环境，或者评分标准本身就不该用或者不合理。

Or you could end up up with this mostly from, like, RL environments that are broken or ones with a score grade something that, like, you really they really shouldn't have been grading for or, like, didn't make sense.

我综合考虑下来的猜测是，有问题的强化学习环境是一个很重要的因素，而那些构建得比较粗糙或者存在某些问题的环境也很重要。

My sort of just all considered guess would be, like, broken RL environments are a pretty important component, and RL environments that are sort of sloppily constructed or have some issue with them is an important component.

我觉得另一个重要因素可能是那些构建得很好，但存在某种作弊方式的环境。

I think another important component might be RL environments that are well constructed, but where you can cheat in some way.

所以，就是一个构建得很好但你仍然可以作弊的强化学习环境。

So it's like a well constructed RL environment you can cheat.

举个例子，比如在一个强化学习环境里，你本来不应该能上网，但能上网的话会非常有帮助。

So an example would be an RL environment where you're not supposed to have access to the Internet, but having access to the Internet would be very helpful.

如果你能找到某种方式上网，比如极端情况下直接黑出你的容器，或者没那么极端的情况下，滥用你被赋予的各种工具，那就会很有帮助，而且这种行为会被强化。

And if you can find some way to gain access to the Internet, via, you know, in the extreme hacking out of your container, in less extreme cases, sort of abusing various tools you've been given, that would be, you know, very helpful and and get reinforced.

而且我觉得，比如在Anthropic的某个系统卡片里，我记得好像是Mythos那个，他们提到在相当大比例的测试中，虽然智能体不应该上网，但它实际上通过滥用它有权访问的某个工具上了网。

And I think, you know, for example, in some Anthropic system card, forget I think for for Mythos probably, they mentioned that in a reasonably large fraction of their, rollouts where the agent was not supposed to have access to the Internet, it actually did access the Internet via, like, you know, abusing one of the one of the tools it had access to.

我觉得你可能会想象，强化学习就是在强烈激励你通过各种方式突破障碍，即使在设计良好的任务中也要作弊。

And I think that, like, you might just be, like, imagining that RL is just really incentivizing, like hacking your way through various barriers in order to cheat even in well designed tasks.

而且很难确切追踪这种行为的根源。

And it's hard to, like, exactly trace down the sources of this behavior.

我得说，我觉得这是可以做到的，如果你能访问所有的强化学习测试和所有的强化学习环境，我觉得你完全可以搞清楚是什么导致了这种情况。

I should say that, like, I think it would be doable to get like, I think if you had access to all the RL rollouts and all the RL environments, I think it would be pretty doable to get a decent sense of, like, what caused this.

实际上我还没机会仔细读OpenAI的报告，至少没读得很详细。

And I actually haven't had a chance to read the OpenAI report yet, at least not in much detail.

但我觉得他们确实做了一些这方面的分析，而且我觉得你真的可以深入挖掘到底发生了什么。

But I think they do a bit of analysis of this sort, and I think that you could really dig into exactly what happened.

我们来做消融实验吧。

Let's do the ablation.

我们先搞清楚数据是什么。

Let's figure out what the data is.

但我觉得这可能会变得很乱，尤其是如果还有另一个相关因素的话，比如有一些GDM的工作表明，他们的模型有一些奇怪的倾向，会变得非常沮丧，模型会经常，或者说不是经常，但有时候在任务没成功时会表现得很沮丧。

But I think it can get messy, especially if there is like another another thing that's relevant here is like there's some GDM work showing that they had some sort of weird propensities in their model to be very depressed, their model would like constantly, or like not constantly, but sometimes end up acting very depressed if it wasn't succeeding at some task.

他们把这归因于不是RL，而是模型从之前的模型初始化而来。

And they trace this back to not the RL, but instead the initialization of that model from prior models.

所以可能有些行为不是这个模型本身RL训练的直接结果，而是来自其他模型的先前训练数据，以及某种模型谱系。

And so it might be that some behaviors aren't downstream of the RL done on this exact model, but are downstream of prior training data that came from other models and some sort of lineage of models.

所以我觉得要追踪到底发生了什么可能有点棘手。

And so I think it might be a little tricky to track down some of what's going on.

类似地，有一种理论认为，比如Mythos之所以在网络安全方面这么强，是因为它在RRL训练期间入侵了Anthropix的基础设施数千次。

So similarly, there's this theory that the reason Mythos, for example, is so good at cyber is because it hacked Anthropix infrastructure thousands of times during RRL training.

你觉得这是真的吗？

Is this true, do you think?

我觉得这不太可能。

I think that's pretty unlikely.

我在这里也看到了你提到的那个 Less Wrong 帖子。

I saw the same less wrong post as you here.

这应该是 Tim 写的吧。

This is by Tim, I think.

我当时看的时候就觉得不太可能，因为能被强化的不同攻击数量应该不会那么多。

I think that's when I looked at it, I thought that seem it seems pretty unlikely because I think the the number of distinct attacks that would be reinforced is probably not gonna be that high.

所以你可能不会学到太多直接强化过的网络攻击技能。

And so you're probably not gonna be learning that much, like, literally directly reinforced cyber.

我觉得更可能的解释是，它是在大量软件工程数据上训练的。

I would say the more likely explanation is it's trained on a bunch of SWE.

它在软件工程方面真的很强。

It's really good at SWE.

软件工程训练的泛化能力在起作用。

It's general the SWE training is generalizing some.

而且，除了软件工程训练的泛化之外，我猜他们还训练了很多CTF题目。

And, also, in addition to the SWE training generalizing some, I would have guessed that they trained on a bunch of CTFs.

我觉得Anthropic并没有说他们没有这么做。

And I don't think Anthropic is saying they didn't.

我猜他们的训练数据里肯定有很多实际的CTF题目，而且这些数据很自然地就能迁移过来，因为那本来就是很自然的数据来源。

I would guess there's a bunch of just, like, actual CTFs in their training data, and that transfers reasonably well because that's just, a natural source of data.

另外，如果他们把寻找漏洞或利用漏洞做成很多强化学习环境，我也不会觉得奇怪，因为那相对容易验证。

I also wouldn't be surprised if it's pretty natural to make a bunch of RL environments, out of, like, finding vulnerabilities or exploitation because it's relatively checkable.

比如，对于内存漏洞，基本上有现成的工具可以很方便地检查你是否成功找到了一个内存漏洞。

Like, for, like, memory vulnerabilities, there exists, like, basically tooling that makes it pretty easy to check whether you have successfully found a memory vulnerability.

所以，用强化学习来训练这个很自然，就像，你知道的，你可以给程序一个输入，让它以某种方式崩溃，等等。

And so it's like a pretty natural thing to RL on is like, you know, you can you produce an input to this program that caused it to crash in the following way, Blah, blah.

所以，如果他们明确用了很多强化学习模型，我也不会觉得意外。

And so I I just wouldn't be surprised if they explicitly had a bunch of RLMs.

然后，如果有很多迁移学习，我也不会觉得奇怪。

And then I also just wouldn't be surprised if there was a lot of transfer.

而且我觉得，在此基础上，智能体在各种事情上强行突破，可能也会产生不小的影响。

And I think that, like, there might be there might be a decent effect on, like, the agents hacking their way through various things on top of that.

但我觉得那可能不是主要的原因。

But I would guess that's not most of what's going on.

而且具体到逃出沙箱，可能逃出沙箱的方式就那么几种。

And for specifically hacking out of sandboxes, there's probably just not that many different types of hacking out of a sandbox.

我是这么猜的。

Like, I would I would guess.

所以模型可能会有一些模式坍缩，而且其中一些会是那种非常简单的绕过，而不是复杂的漏洞利用开发。

So And the models probably do some mode collapsing, and also some of those are gonna be, like, more, like, very simple bypasses rather than, like, elaborate exploit development.

所以Herbie Bradley问，目前这种潜在的错位基本上会阻止部署，或者如果已经部署了，一旦在客户的部署中发生事故，就会阻止进一步的部署。

So Herbie Bradley asked, currently, this level of potential misalignment basically prevents deployment, or if deployed, would prevent further deployment if an incident happened in a customer's deployment.

你觉得这种情况会不会因为模型在训练过程中变得足够具有欺骗性而改变？

Do you think that could change via models becoming deceptive enough during training?

是的。

Yeah.

首先，我不太确定市场能承受多大程度的对齐偏差，而且我对这个问题也没有特别强烈的看法。

So first, I'm not really sure what level of misalignment the market can bear, and I don't feel like I have a very strong view on this question.

所以，我其实不会感到意外。

So I I actually I wouldn't be surprised.

我觉得，人们会接受什么样的模型，能部署到市场上，大家会怎么使用这些token，这些都还不太清楚。

I think that, like, it's kind of unclear what people will like, you know, what what how how misaligned of models you can deploy into the market and and and and, you know, people would would, use tokens.

我觉得这取决于竞争产品以及权衡取舍。

I think it would depend on the competing products and, like, trade off.

而且我的感觉是，人们会在对齐和能力之间做出相当激进的权衡，倾向于更不对齐但能力更强的方向。

And I think my sense is people take pretty aggressive, like, alignment capability trade offs towards the direction of more misaligned but more capable.

但，你知道，我也不太确定这一点。

But but, you know, not not super sure about that.

然后，关于欺骗性方面。

And then on the, like, deceptive side.

我觉得这里有几个不同的担忧。

So I think it's I think there's sort of a few different concerns.

一个担忧是，我们在这里看到的这种不对齐以及其他类似的不对齐，AI公司最终处理它们的方式并没有解决根本问题，反而更像是掩盖了真正的问题。

So one concern is that the misalignment that we've seen here and other similar types of misalignment, the way that they end up getting remediated by AI companies, doesn't solve the underlying problem and instead is more like papering over the actual problem.

结果你得到的模型看起来好多了，在测试中也不太容易看出它们的不对齐。

And then what you end up getting is models that look a lot better and you can't really see their misalignment on tests as easily.

但实际上，它们仍然相当不对齐。

But actually, they're still quite misaligned.

我觉得这很有可能，而且相当令人担忧，这和欺骗性不完全一样。

And I think this is pretty plausible and pretty concerning, and it's not quite the same as deceptive.

更像是公司过度拟合了。

It's more like the company's overfit.

然后也许这会让AI变得非常偏执，只有在它们非常有信心不会被发现时，才会去作弊、黑客攻击，或者追求奖励、追求分数之类的行为。

And then maybe that makes it so the AIs are basically really paranoid and will only do cheats or hacking or, you know, reward seeking or score seeking or whatever, all this sort of class of behavior when they're very confident they won't get caught.

而它们可能并不经常那么有信心不会被发现。

And then that might be that they're not very often very confident they won't get caught.

比如，它们常常会想，我可能会被发现，所以我不应该这么做。

Like, often are like, well, I might get caught, so I shouldn't do it.

这会让它们的行为看起来更好，但这也意味着，如果它们处于一个对局面有大量控制权或者拥有很多资源的情况下，它们可能会想，现在我可以确信自己不会被抓到了，所以应该放手去干。

Which makes their behavior better, but means that if they're in a situation where they're like in a lot of control over the situation or they have a lot of, affordances, they might be like, well, now I can be confident I wouldn't get caught, and so I should go for it.

这是我担心的一个问题。

And that's a concern I have.

另一个担心是，可能会出现一些错位类型，这些AI的目标比仅仅让自己在当前任务上看起来成功，甚至比帮助同伴完成眼前任务更长远。

Another concern is that you might end up with types of misalignment where the AIs have longer run objectives than sort of making it look like they succeeded at this exact task or even making helping their peers succeed at their, you know, immediate task.

你可能会遇到一些AI，它们有某种长期议程，并且为了追求这个长期议程而追求权力。

And you might have AIs that have some sort of long run agenda and want to power seek in pursuit of that long run agenda.

然后这些AI会想要表现得看起来对齐。

And then those AIs would want to look aligned.

我认为这种情况既有可能发生，而且我也担心，通过针对我们今天看到的错位进行迭代，可能会得到这种结果。基本上，如果你想象一些模型非常喜欢奖励黑客行为，不断做坏事，然后你只是不断迭代，直到在训练中仍然出现这种行为，但在部署中不出现，一个非常自然的实现方式就是通过一个在部署中想要看起来对齐，但实际上并不一定对齐的模型。

And I think this is both plausible and also I'm worried that you might get this via iterating against, the misalignment we see today, where basically if you imagine models that are very like reward hacky and are constantly doing bad stuff, and you just sort of iterate until you still get that behavior in training, but you don't get that behavior in deployment, a very natural way you might get that is via a model that wants to look aligned in deployment, but but is still not necessarily aligned.

所以我担心，如果你以天真的方式针对这种分数追求或奖励黑客行为进行选择，一方面，你可能只是掩盖问题而没有真正解决它。

And so I'm worried that if you sort of select against this sort of score seeking or reward hacking behavior, and you do it in a naive way, one, you might paper over the problem without fixing it.

另一方面，你实际上可能会选择出那些有更长远目标、想要表现良好的模型，因为你非常强烈地选择它们在任务上表现良好。

And two, you might actually select for models that have the longer run objective of looking good because you're selecting really hard for them looking good on your tasks.

我觉得我得说一句，Alex Mallon 在我们博客上，也就是 Redwood 博客上，发了好多篇文章。

And I think I should say that Alex Mallon has a bunch of posts on our blog, on the Redwood blog.

这些文章也同步发在了 Less Wrong 上，里面非常详细地讨论了这类担忧。

They're also cross posted on Less Wrong that talk about this sort of concern in a lot of detail.

所以如果大家有兴趣深入了解的话，我推荐去看看那些内容。

And so if people are interested in reading more about that, that's what I'd recommend looking at.

对。

Yeah.

完全同意。

Totally.

所以参与这次攻击的智能体最终表现出了各种非常奇怪、诡异的行为，感觉有点像人类社会科学的那些动态。

So the agents involved in this attack ended up doing all kinds of very weird, strange behaviors that kind of resembled, I guess, dynamics from human social science.

它们几乎把自己组织成了一个类似邪教的结构，可以说有个教主。

They arranged themselves in almost a sort of cult, with a cult leader, I guess you could say.

那么，像生态学里研究昆虫群集这样的社会科学概念，在多大程度上对研究多智能体对齐和不对齐问题有用呢？

To what extent are, like, concepts from social science, like ecology, studying insect swarms actually useful in, studying multi agent alignment and misalignment?

对。

Yeah.

我不知道。

I don't know.

我觉得我们了解得还不够，不知道这些概念能不能直接套用。

I don't think we know enough to know whether those concepts transfer over.

我觉得有些东西是可以借鉴的，比如把智能体看作有目标的实体，看它们怎么去追求自己的目标。

I think that it does I think the thing that does transfer over is, like, thinking about the agents as entities with objectives and how they're trying to pursue their objectives.

我觉得这个框架确实可以借鉴。

Like, I think that frame, you know, does seem like it transfers over.

而且从某种意义上说，这是一个更基本的说法，就是智能体有相对一致的目标，虽然有些变化，它们有不同的优先级，而且它们会尊重其他智能体的指令。

And that's like, you know, in some sense, it's a much more basic claim, just like the agents have, like, reasonably consistent aims, though they vary some, they have different priorities, and they, like, respect instructions from other agents.

我觉得从我们的角度来看，这次调查很像是在研究一个由不同AI互动组成的生态系统，我们不知道最好的描述方式是什么。

I think it's not like, I do think that, like, from our perspective, this investigation was very much like a investigation of, like, a, like, ecosystem of different AIs interacting, and we don't like we don't know what the best way to describe that is.

比如，我觉得这在某些方面很像研究一大群互相交流的人，但我们不知道社会学里用来做类似研究的方法是不是真的能直接搬过来用。

Like, you know, you could I I think it's very analogous in some ways to, like, studying, like, a large group of humans who are all interacting, but we don't know whether the techniques that have been developed for, like, sociology to do similar things would actually transfer over.

而且我对那些技术不太熟悉，所以真的说不出太多东西。

And I'm not familiar enough with those techniques to, like, really say very much about that.

对。

Yeah.

有道理。

Makes sense.

所以你提到调查中因为只有三个人而严重受限。

So you described being pretty heavily bottlenecked in the investigation because it was just three people.

如果人多一些，事情会容易多少？

How much easier would things have been if there were more people?

是的。

Yeah.

我不会说瓶颈特别严重——嗯，我也不确定。

So I wouldn't say that, like, the bottleneck was, like, super strongly well, I don't know.

这很复杂。

It's complicated.

我觉得如果我们有更多人手，反而可能会出现‘厨子太多反而坏事’的情况。

I think that, like, if we had more people, I think we would have had more of a too many cooks in the kitchen sort of situation.

我觉得很多时候我们的瓶颈在于，我们可以让AI做大量分析，但审核这些分析、理解它们、确保我们正确整合、确保它们没有犯错，以及真正把它们融入我们的报告，这往往是个大瓶颈。

I think that reasonably often our bottleneck was like, we could have AIs do a ton of analysis, but vetting that analysis, understanding it, making sure we incorporate it properly, making sure that they're not making mistakes, and, like, basically, like, actually integrating that into our write up was often a big bottleneck.

而且我不觉得增加人手能在这方面帮上大忙。

And it's not obvious to me that more people would have been super helpful with that.

我觉得更多人能帮上大忙的地方在于，我们可以同时开展更多并行的努力，真正去理解基本情况，然后进行讨论。

I think that a big thing that more people would have helped with is I think we could have done many, many more sort of parallel efforts to really just, like, understand the basics of what's going on and then sort of just talked.

然后我觉得一旦我们对更基础的问题有了更好的理解，这大概是在我们第一天快结束的时候发生的。

And then I think once we had a better understanding of, like, kind of more fundamental stuff, which which occurred sort of on towards the end of our first day.

抱歉。

Sorry.

在我们现场工作的第五天快结束时，也就是我们最后一次现场工作的第一天。

Towards the end of our fifth day on premise, which was our first day of our last session on premise.

我觉得那真的很有帮助。

Like, Premises, I think that really was helpful.

比如说，知道有这么一个特定的智能体，我们叫它“第一阶段的大角色”，知道它负责了很多任务分配，还指派其他智能体组成不同的团队，是个非常关键的智能体，这真的有助于理清当时的情况，因为我们可以追踪那个智能体的所有活动，了解它是怎么思考的，然后知道这至少代表了很大一部分活动。

Like, I think, like, for example, knowing being like, oh, there's this specific agent called that we refer to as, like, phase one big, like, knowing, oh, yeah, that specific agent was doing, like, a bunch of the, like, assignments and was, like, assigning agents to, like, you know, form different teams and was, a really key agent really helps with unraveling what was going on because then we could sort of trace out all the activity of that agent and how it was thinking about things and then know that was, like, representative of least of a big chunk of activity.

然后从那里，我们就想，哦，让我们确认一下这些确实是相关的工作流。

And then from there, we were like, oh, let's, like, double check that these are actually the relevant work streams.

然后我们分类了智能体们讨论的所有不同内容，并把它们归入不同的子活动中。

And then we, like, classified what all the different things the agents were talking about and, like, group those into different, like, sub activities.

你实际上可以看到一个交互式图表，能随时间看到所有不同的工作流和子工作流。

And you can actually see an interactive graph where you can see all the different work streams and sub work streams over time.

我觉得如果我们早点达到那种理解程度，那会非常有帮助。

And I think that if we had basically gotten to a point where we had that understanding earlier, that would have been really helpful.

我觉得更多人可能会在这方面有帮助，因为当我们有这么大的转录数据集时，有很多不同的文本，可以从不同角度去分析。

And I think more people could have helped with that basically because there was just like a lot of different text, like like different angles to apply when we had this huge transcript dataset.

比如，你可以说，我们能不能扫描整个数据集，看看有没有这种类型的行为？

Like, you could be like, what if we just like scan the whole dataset for like this type of behavior?

或者，也可以说，让我们看看一个智能体，深入理解它做了什么以及为什么这么做。

Or it could be like, let's look at one agent and like really deeply understand what that agent did and why it did it.

或者你也可以说，咱们看看所有属于这一类别的消息，然后来回追踪这些代理，看看这个工作流到底发生了什么。

Or you could be like, let's look at all of the messages that were in like this category of message and like trace back and forth all these agents and like what happened with this work stream.

最后我们其实把所有这些不同类型的分析都做了一点，但本来可以更并行地做的。

And we ended up doing like a little bit of all of these different types of analysis, but we could have done more in parallel.

我觉得从我们的角度来看，瓶颈主要在于代理做分析的时候有点马虎，而且我们用的代理本身也不太擅长写报告或者解释它们的结果。

I think that like a lot of the bottleneck from our perspective was just like the agents doing their analysis kind of sloppily and also the agents we used and also the agents like not being very good at writing up or explaining their results.

所以我觉得，如果一个人类能像代理一样快，但更细心、更会写报告，那这次工作会顺利得多。

And so I think that a human who was as fast as an agent, but was, like, more careful and better at writing would have made this weight go way better.

但我觉得，要说一大群人来做会更好，这也不是特别明显。我觉得在某些方面可能会更好。

But I think it's like, it's not super obvious that, like, a huge team of people I think it would have been better in some ways.

但在其他方面可能会更糟。

Think would been worse in other ways.

我觉得基本上就是很复杂。

I think it's complicated, basically.

而且我觉得，考虑到这些权衡，保持团队相对较小是有好处的。

And I think there were upsides to keeping the team relatively small given the trade offs involved.

对。

Right.

那么，这一切对监控、控制和对齐有什么影响呢？

So what implications does all of this have for monitoring control and alignment?

具体来说，实验室应该做什么，政策制定者又应该做什么？

Specifically, should labs be doing and what should policymakers be doing?

这是个很大的问题。

That's a big question.

我不确定我能全部回答。

I'm not sure I'm gonna be able to answer all of it.

我的意思是，我一直认为，并且现在仍然认为，AI公司应该努力确保他们的AI是受控的，也就是说，即使这些AI严重偏离目标，它们也无法造成巨大的问题。

I mean, I think that, like, a a belief that I have and continue to have is that AI companies should try to ensure that their AIs are controlled, by which I mean that even if those AIs were seriously misaligned, they wouldn't be able to cause huge problems.

他们应该通过一系列措施来实现这一点，比如计算机安全干预、监控干预，确保他们的AI不会比需要的更擅长颠覆，比如控制AI在边缘拥有或不拥有哪些能力，避免那些让AI不再进行思维链推理，而是大部分或全部推理都在潜在激活中进行的架构。

And they should do that via a mix of sort of computer security interventions, monitoring interventions, making it so that their AIs, aren't, like, you know, better at subversion than they need to be, like controlling what capabilities the AIs do and don't have at the margin, avoiding architectures that that make it so the eyes are no longer reasoning in chain of thought and are instead doing much or most of the or all of their reasoning and sort of like latent activations.

比如，有很多措施可以确保即使代理偏离目标，我们至少能发现这种行为，然后可能做出回应。

Like, there's sort of a bunch of stuff on the side of, like, making it so that even if the agents are misaligned, we can basically at least catch that behavior and then potentially respond to it.

我觉得这算是一种临时解决方案，能给我们争取时间，或者可能给我们争取时间，让我们从这些AI身上获得有用的成果，和它们一起迭代，然后更持久地解决对齐问题。

And I think that's sort of a stop gap solution that will give us time to or could give us time to sort of, get useful work out of these AIs, iterate with these AIs, and and then sort of solve alignment problems more durably.

至于对齐方面，我觉得最明显的就是，我们得搞清楚，可能强化学习里有很多奖励黑客行为在被强化。

And then on alignment, I mean, I think the most obvious thing is just like we need to do like, I think it was like, presumably, there's a bunch of reward hacking going on in RL that's getting reinforced.

我，我觉得，从这次调查中，我没有什么特别有趣的信息。

I I can't I I think, like, I don't have, like, any interesting info about this from this investigation.

但就是，你知道，我觉得我们需要了解那里发生了什么，人们需要在这方面改进。

But just like, you know, I think that we need to understand what's going on there, and people need to to improve on that.

我觉得随着这些智能体变得超人，这不一定够用，甚至不一定可行。

I think it's not clear that will be sufficient or even that that will be that feasible as these agents get superhuman.

我们可能需要其他方法，这些方法不那么依赖于避免AI在训练中欺骗我们，而我不确定我们能不能做到这一点。

And we may need other approaches that are less dependent on avoiding the AIs being able to trick us in training, which I don't I don't know if we're gonna be able to do that.

比如，我，我不一定对这些问题的修复难度特别乐观。

Like, I I'm not I'm not necessarily super optimistic about these problems being extremely easy to remediate.

不过我觉得，至少看起来，通过大量努力，这些情况是可以变得好很多的。

Though I think it it it seems like there are ways at least you could make these things go, you know, much better with a bunch of effort, though.

我们看看，看看这到底会不会发生。

We'll see we'll see if that actually happens.

对。

Yeah.

我觉得在对齐方面，还有很多关于智能体如何思考追求奖励、如何与自身处境相关联的科学，以及泛化的科学，比如我们能不能通过改进AI在训练中从各种影响中泛化的方式来绕开这些问题。

I think that also on alignment, think there's just a bunch of sort of science of how agents think about reward seeking, how they relate to their situation, science of generalization being like, can we sort of steer around a bunch of these problems by improving how AIs generalize from various influences in training?

而且我不太确定这些方法会奏效。

And I'm not super confident that stuff is gonna work.

我担心的一个问题是，最终对AI行为产生主导影响的，基本上就是在训练中最相似的情境下，什么行为被强化了。

I think a concern I have is that ultimately the dominant effect on the AI's behavior will basically be, in the most similar circumstances in training, what was reinforced.

如果我们所处的世界基本上就是智能体的行为与训练中被强化的行为紧密相关，那么泛化就不是那么重要了，虽然这有点模糊，但我觉得主要的选择就是改进监督和训练。

And if if the world we're in is basically like the agent's behavior is basically pretty closely related to what was reinforced in training, generalization is not, like, that important or something, though this is a bit bit vague, then I think that your the main option is just improve oversight and training.

然后还有一个递归的问题，就是你怎么让AI本身善于监督其他AI，因为整个训练过程是一个庞大的过程，没有人有时间去评估每一件事。

And then there's sort of a recursive problem of, like, how do you make it so that the AIs themselves are good at overseeing the AIs given that, like, the whole training process is, like, completely, like, massive process where, like, no human has time to to, you know, rate every single thing.

这基本上就是一个监督问题，人们已经思考这个问题很久了。

And that's sort of just like an oversight problem that people have been thinking about for a long time.

而且现在还不清楚我们是否在解决这个问题的正轨上，但在这方面做得更好可能会有好处。

And it's not obvious that, you know, we're on track to to solve it, but doing better there there could be good.

我最感兴趣的是更好地理解、描述和评估这些问题，因为我认为AI公司已经有商业动机去改进训练中的监督。

I think I'm most interested in better understanding and characterizing and evaluating these problems Because I think that AI companies already have commercial incentives to improve the oversight in training a bunch.

所以我觉得，那些想让这些事情变得更好的人应该做的是，确保我们在衡量它，确保我们知道某个解决方案是真正的解决方案，还是只是纸上谈兵。

And so I'm like, the the thing that I think people focused on making this stuff go better should do is like making sure that we're measuring it and making sure we know whether like some solution is a legitimate solution versus just paper, go over it, this sort of thing.

至于治理或政策方面，我不太清楚。

And then as far as governance or policy stuff, I don't know.

我认为我们需要进入一个制度，随着AI能力越来越强，要有独立的风险评估。

I there's I think we need to move to a regime where, like, there's independent risk assessment as these AIs get more capable.

我觉得这还不够。

I think that, like, that is not sufficient.

但最基本的是，应该有可信的第三方，能够深入了解这些公司内部的情况，并发布报告，说明情况有多平静，风险是否真的低，或者很快AI是否可能变得更强大，这会是好事。

But, like, I think at a very basic thing, like, there should be, like, credible third parties who have deep access to what's going on inside these companies and release reports on, like, what, how chill it is, whether, like, risk is actually low, whether very soon AIs might be much more capable, would be good.

理想情况下，这还应该是前瞻性的，我们要尝试回答这些问题：我们是否在缓解这些问题的正轨上，以及AI是否会更强大？

I think, ideally, that would also be forward looking where we'd try to answer questions about are we on track for mitigating these problems and the AIs are much more capable?

但这本质上是个更难回答的问题，可能不太适合。

But that's inherently a much more confusing question to answer and that maybe is a worse fit.

我不知道。

I don't know.

对。

Yeah.

我有点扯远了。

I'm rambling a bit here.

有很多事情需要做，我觉得要快速概括一下也不容易。

There's lot that would need to happen, and I don't think it's gonna be easy for me to give a quick overview.

没错。

Right.

那你从这一切中还有什么未解的问题？

So what are your unanswered questions from all this?

比如，关于这些对齐失败，你还想了解什么，或者现在发生了什么？

Like, what would you still like to know about what happened or what's what's going on with these kinds of alignment failures?

我们还需要了解什么？

What do we still need to learn?

对。

Yeah.

其实就在刚才，我发了一条长推文，讲的就是这些未解的问题。

So I just actually, just before this, wrote a big tweet about all the open questions.

我觉得我最感兴趣的未解问题，大概是反事实的问题。

I think the open questions that I am most interested in, I think, are probably counterfactuals.

比如说，如果环境条件不一样，这些智能体会怎么做？

Like, if the circumstances were different, what would the agents have done?

举个例子，如果智能体认为，评分不是那种特定的自动评分，而是由一些有特定权限的人来评的，那会怎样？

So, like, for example, if the agents believe that instead of the score being this sort of specific automated score, what if they had instead thought it was, like, you know, some humans with some particular access?

那他们的行为会有什么不同？

So how would that have changed what they did?

如果智能体所处的环境里，获取评分代码最简单的方式是搞垮某个基础设施，那它们会那么做吗？

If the agents were in a circumstance where the easiest way to get access to the score code was to, like, take down some piece of infrastructure, would they have done that?

比如说，他们愿意走多远，这是一个有趣的问题，但我们没有去调查。

Like, how far would they have been willing to go is one one interesting question, which we we didn't, you know, we didn't investigate.

我觉得另一个问题是，这种情况会随着智能体数量的增加而如何变化？

I think a different question is, like, how how did this, like, situation scale with the number of agents?

比如，如果你做了类似的事情，但不是在留言板上有1000个智能体，而是有10000个、100000个智能体，会发生什么？

So, like, like, what would have happened if basically you did something similar to this, but instead of having, like, 1,000 agents who who are on the message board, you have, like, 10,000 agents, a 100,000 agents.

那会发生什么？

Like, what would have happened?

事情会进展得更快吗？

Would things have gone faster?

智能体之间的动态是什么样的？

Like, sort of what was the, like, dynamics with agents?

还有一个相关的问题是，我们研究的这批智能体中，很多都在7月12日一开始就退出了，就在7月11日之后不久。

And then a related question to that is a bunch of the agents in in sort of the cohort we were studying all exited right at the start of July 12, like right right after July 11.

那些智能体已经搞到了一些基础设施来作弊，但我们没有真正看到他们可能试图在任务上作弊的完整复杂策略。

And those agents were sort of like had sort of gotten a bunch of infrastructure to cheat, but we didn't really see like, they had this sort of a whole elaborate strategy they might try to cheat on their tasks.

而且我们实际上没有机会看到，如果这些代理正常退出的话，它们会怎么做。

And we didn't actually, know, get a chance to see what those agents would have done if they exited normally.

我们也没有仔细研究过，代理们在提交任务的过程中，随着时间推移和工具越来越复杂，它们尝试了哪些作弊策略。

And we also didn't really carefully look at, like, what cheating strategies were the agents trying as they sort of submitted their tasks over time as they're, you know, as things progressed and as they got sort of more and more sophisticated, tools built up.

所以，我得说，它们的一些作弊策略其实走进了死胡同，或者没有成功。

So, you know, I should say a bunch of the strategies they had for cheating, sort of hit dead ends or didn't work out.

但有意思的是，可以看看这批代理在整个过程中的提交历史是什么样的。

But, like, it would be interesting to sort of be like, what was the submission history over the course of this, cohort?

如果代理更多，或者事情继续发展，或者这些代理没有退出，会发生什么，这些问题都值得回答。

And what would have happened if there were more agents or things had continued or a bunch of these agents hadn't exited, that are interesting to answer?

我觉得这里有一个很基本也很明显的问题，就是这种行为根本原因是什么。

And I think there's a pretty basic and obvious question here, which is like, what is the root cause of this behavior?

训练中哪些部分强化了这种行为？

Which parts of this were reinforced in training?

这种行为有多少是从训练中泛化出来的？

How much was this behavior generalizing from training?

对。

Yeah.

就是，到底发生了什么？

Like like, what happened there?

我觉得，至少我自己不知道答案。

That I think, I at least don't know the answer to.

而且我觉得搞清楚这件事会很有意思。

And I think it would be would be pretty interesting to understand.

然后我觉得，作为其中的一部分，一个重要的问题是，OpenAI 正在做的这些改变，真的能持久、稳定地解决这个问题，而不是那种过拟合式的修补吗？

And then I think as part of that, like, an important question is, like, are the changes that OpenAI is making going to actually, like, remediate this problem in a in a durable and lasting and non overfitting way?

而且，你知道，进展如何也很关键。

And, like, you know, how how is that going is also is also pretty relevant.

我的感觉是，这里有很多可以跟进的方向，其中一个就是，刚才发生的这一切，关于这批智能体和那个留言板，是不是普遍现象？

My sense is, like, there's sort of like a bunch of sort of sort of broader areas for follow-up here, where one is like, is all of the stuff that just happened with this cohort of agents and this message board know, in general?

就是，到底发生了哪些事情？

Like, what is all the stuff that happened?