诺姆·布朗——智能体集群、对齐与递归自我改进

对于RSI是什么样的或者其动态如何，我非常尊重你的内部视角，因为你显然在这个领域已经十年了，而我是在尝试从一个非常外部的视角来推理——

I'm quite deferential to your inside view on what RSI looks like or what the dynamics are, because obviously you've been in the field for 10 years, and I'm trying to reason about it from a very outside view type of-

我的意思是，我会——

I mean, I'll-

……直觉泵

... intuition pumps

……

...

我会说，人们对这个问题有不同的看法。

I'll say that people have different opinions on this.

而我对此有自己的看法。

And I have my opinion on this.

我完全可能错了。

I could totally be wrong.

我承认这一点。

I admit that.

我对此有一定信心，但不是百分百确定——

I have some confidence in this, but I'm not 100% confident-

……事情就会这样发展。

... that this is the way things go.

也许会有一次一夜之间的智能爆炸。

Maybe there could be an overnight intelligence explosion.

我不知道。

I don't know.

也许我们看不到三倍的速度提升。

Maybe we don't see a 3X speed up.

也许只是百分之五十的提升。

Maybe it's like a 50% speed up.

这里有很多不确定性。

There's a lot of uncertainty here.

所以，有几点要说。

So a couple of points.

顺便说一句，我想澄清一下关于这种参差不齐的表现。

Tangentially, I want to clarify something about the jaggedness.

对，最近我有点想明白了，就是AI只要在某个方面参差不齐地擅长，就足以构建一个更好的学习者，因为那个更好的学习者可以更通用，对吧？

Yeah, one thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner because that better learner can be more general, right?

所以，是的，如果你只是让AI更擅长用Office软件或者下棋什么的，那也就那样，没什么大不了的。

So, yeah, if you just make an AI that's better at using Office products or playing chess or something, that's whatever, that's fine.

这不会带来什么大的生产力提升。

It's not going to lead to big productivity improvements or anything.

但如果你让AI非常擅长制造更节省样本的东西，或者能持续学习，或者解决这些更明确的机器学习问题，那么从中产生的东西，假设你直接解决的问题和这种更广泛的学习能力之间有足够的迁移，就可以变得更通用，对吧？

But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well-scoped ML problems, the thing that emerges out of that, assuming there's good enough transfer from the direct problem you're solving and this broader ability to learn, can just be more general, right?

所以我觉得这是一个重要的动态，要记住为什么参差不齐的表现最终还能带来通用性。

So I think that's an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end.

关于这个问题，显然实验会限制你，因为如果不是这样，就像你说的，OpenAI一夜之间就会发生疯狂的奇点，或者你会有88小时，解决机器学习领域的千禧年大奖问题，然后你就有了超级智能。

On this question of, obviously experiments bottleneck you because if they didn't, as you're saying, you'd have some crazy singularity overnight at OpenAI, or you'd have 88 hours, and you'd solve the Millennium Prize Problem equivalent of ML, and you'd have the superintelligence.

所以显然实验是一个巨大的瓶颈，以至于这需要很多年，而不是88小时。

So obviously the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours.

但问题是，它们到底有多大的瓶颈作用。

But then the question is how much of a bottleneck they are.

在我看来，让我有点“奇点眩晕”的是，即使当前进步的速度只是保持不变，不需要加速，就只是按现在的速度继续下去。

And it seems to me one thing that's been giving me a bit of singularity vertigo is realizing that even if the current rate of progress simply continues, so it doesn't have to speed up, literally just continues apace.

在你提到的那些阻力出现的同时，继续保持这个速度，对吧？

Continues apace as some of the other headwinds you talked about come up, right?

比如问题更难找了，周期更长了，可能到2030年代末，算力也无法再按这种指数级增长。

That it's harder to find problems, it's more long-horizon, maybe like end of the 2030s, compute can't keep scaling at this exponential level.

如果我们只是保持当前的速度，我觉得人们并没有认真对待这意味着什么，尤其是当我们超越人类水平的时候。

If we simply continue the current rate of progress, I think people are not taking seriously what that implies as we cross over beyond the human horizon.

这里有一些它意味着的事情。

Here are some of the things that it implies.

所以，我觉得真的很难想象比人类更聪明的智能会是什么样子。

So, I think it's really hard to reason about what smarter than human intelligences will be like.

那我们就从人类人口规模的角度来想想吧。

So let's just think in terms of human population sizes.

按照目前的进展速度，每年同样的算力基本上能让有效人口规模扩大三倍。

The current rate of progress makes it so that a given level of compute allows you to basically run a 3X bigger effective population every single year.

而且算力本身也在不断增长。

And also compute is growing in the background anyways.

所以可能会出现这样的情况：到2030年底，每个实验室——很可能更早，但就算到2030年底吧——都有足够的算力来运行相当于数亿个人类级别智能体，基于那时能力的发展水平。

And so you could have a situation where each of the labs, by the end of 2030, probably much sooner, but let's say by the end of 2030, has enough compute to run, let's say, hundreds of millions of human-level intelligences based on where the capabilities will be at that point.

然后我觉得人们没有认真想过，按照目前的进展速度，再过几年，到2030年代中期或更早，每个实验室里就会有相当于好几个地球的人类级别智能体，而且它们很可能在质量上超越人类，对吧？

And then I think people are not thinking seriously, the current level of progress means that by a few years down the line, by the mid-2030s or earlier, you would have many Earths' worth of human-level intelligences within each lab, and they'll probably be qualitatively superhuman, right?

但不管怎样，这是一个基本情形。

But anyways, this is a base case.

我不知道。

I don't know.

进展确实非常快。

Progress is really fast.

我觉得这百分之百是真的。

And I think that's 100% true.

而且我觉得值得指出的是，研究人员一直在对进步的速度感到惊讶。

And I think it's worth pointing out, researchers are continually being surprised at the rate of progress.

你看看，即使在AI研究人员中，2025年获得IMO金牌的预测是什么？

If you look at even among researchers in AI, what were the projections for getting an IMO gold in 2025?

我认为，用通用语言模型、没有工具、没有互联网接入就能做到这一点，我觉得即使是OpenAI的人也觉得这太离谱了。

I think the idea that it could be done with a general-purpose language model with no tools and no access to the internet, I think even people at OpenAI thought this was outrageous.

他们认为这几乎是不可能的。

They thought it was almost impossible.

然后到了2026年，就在我们拿到纳维-斯托克斯方程的两周前，我和一家前沿实验室的研究员聊到，要多久才能拿到千禧年大奖。

And then you get to 2026, and literally two weeks before we got Navier-Stokes, I was talking with a researcher at a frontier lab about how long it would take to get a Millennium prize.

他愿意跟我赌1000美元，说这要等到2027年以后。

And he was willing to bet me $1,000 that it would take past 2027.

他觉得要等到2030年。

And he thought it would take until 2030.

我接了这个赌注。

And I took that bet.

但即便是我也觉得，实际需要的时间会比现在看起来要长。

But even I thought it would take longer than how long it's likely to take.

所以人们一直都很惊讶，连实验室内部的人也不例外。

So people have been continuously surprised, even inside the labs.

昨天我刚好和一个参与纳维-斯托克斯项目的人聊了聊，他告诉我，他以前常说，要预测AI十二个月后会是什么样子真的很难。

And I was just talking to somebody yesterday who was working on the Navier-Stokes effort, and he was telling me that he used to say it's really hard to predict where AI would be in 12 months.

如果有人问他：“嘿，现在情况怎么样？”

If somebody asked him, "Oh, where are things going?"

他还能比较有把握地预测未来十二个月，但再往后，他就只能说“我不知道”了。

He would feel comfortable making predictions for the next 12 months, but beyond that, he's just like, "I don't know."

现在他说，他连三个月以后的预测都不敢做了。

And now he's saying he just doesn't feel comfortable making predictions beyond three months.

所以确实是这样，现在事情进展得非常快。

So it is really true that, yeah, things are going very fast right now.

你说到2030年，我真不知道那时候世界会变成什么样。

And you talk about 2030, I don't know what the world looks like in 2030.

确实是这样。

That's the truth.

你觉得AI劳动力会实现完全自动化，或者说95%的自动化，是在28年、29年、30年还是27年？

Do you expect the sort of full automation of AI labor, or let's say 95% automation of AI labor, '28, '29, '30, '27?

我不知道。

I don't know.

我刚说了，我不知道2030年世界会是什么样子。

I just said I don't know what the world looks like in 2030.

我觉得我们其实发了一篇博客文章——

I think we actually released a blog post-

……最近关于OpenAI内部加速的。

... recently on internal acceleration at OpenAI.

比如我们展示了，研究人员在Codex上的花费，我想是8月初的数据，排名前1%的人每天要花7000到8000美元——

We show, for example, that the amounts that researchers are spending on Codex is the top 1%, I think, as of early August, were spending $7,000 or $8,000 a day-

……在Codex上，而且是内部使用。

... on Codex and for internal use.

这是呈指数增长的。

That's on an exponential.

它会持续上升。

It's going to keep increasing.

然后问题来了，如果继续这样下去，你如何分配工作，是AI在做，还是人在做？

And there's a question of, okay, if that keeps going, then how much do you assign to just the AIs doing work versus-

……还是人在做？

... the humans doing work?

是95%吗？

Is it 95%?

是5%吗？

Is it 5%?

这真的很难判断，原因有几个。

It's really hard to reason about this for a few reasons.

首先，如果是人指挥AI去做工作，那这个功劳算人多少呢？

First of all, if it's the human directing the AIs to do the work, then how much do you attribute to the human?

那AI又该占多少呢？

How much do you attribute to the AI?

另外，因为这些AI能力不均衡，有些方面特别擅长。

The other thing is that because these AIs are jagged and they're exceptionally good at some things.

比如说，它们特别擅长检查数据集，逐条核对每个数据点，看质量是否达标。

So for example, they're exceptionally good at looking over data sets and checking every single data point to see, is this of sufficient quality?

跟以前比，你可以大量用AI来做这些事。

You can disproportionately use the AIs for those things compared to previously.

所以，确实，你比之前用AI多得多，有些事快了100倍，好了100倍，但有些事还没什么大变化。

So yes, you're using AI way more than before, and it's making some things go 100X faster and 100X better, but there are some things where it doesn't make a huge difference yet.

当然，如果某件事突然快了100倍、好了100倍，你肯定会多做这类事。

And of course, if something is suddenly 100X faster and 100X better, you're going to do more of that thing.

所以，你是跟三年前的速度比吗？

So are you comparing to a speed up of three years ago?

问题更像是，跟三年前比，我们现在做同样的事快了多少，还是说，以我们现在做的事为准，三年前做起来会慢多少？

Is the question more like, given what we were doing three years ago, how much faster are we able to do it now, versus, given what we're doing now, how much slower would it have been three years ago?

这两个问题其实很不一样。

Those are actually two very different questions.

所以，真的很难衡量。

So anyway, it's really hard to measure.

但我可以肯定地说，因为AI的进步，现在事情进展得比一年前还要快。

I do feel confident in saying that things are going faster now than they were even a year ago because of AI progress.

而且我觉得这种加速还会继续。

And I think that acceleration will continue.

我觉得这个领域里很多人对这类事情的估计都有很大的不确定性。

I think a lot of people in the field have very high error bars on this sort of thing.

如果你非要逼我给出一个数字，我觉得事情可能会快三倍。

If you had to put a gun to my head and ask me for a number, I could see things going 3X faster.

这已经很了不起了，对吧？

And that is huge, right?

现在的进步速度已经非常惊人了。

Already, the pace of progress is incredible.

就算没有额外的提升，就像你说的，事情也会快得多。

Even if we don't get any uplift, like you said, things are going to go much faster.

等到2030年，我们甚至不知道那个世界会是什么样子。

By the time we get to 2030, we don't even know what that world looks like.

我觉得如果内部加速能带来三倍的提升，那影响就非常大了。

I think if we get a 3X uplift from internal acceleration, that is massive.

我们现在正在发布，想想三年前你在哪里。

We're right now releasing, think about where you were three years ago.

如果我们在一年内取得那样的进展，那就太了不起了。

If we make that progress in one year, that's huge.

那就好比从连o1都没有，只有非推理模型，直接跳到Astra-

It'll be like going from not even having o1, just having non-reasoning models to Astra-

……

...

在一年之内。

in a single year.

所以，我确实觉得事情会进展得更快。

So, I do think things go faster.

有可能只是快50%。

It could be that things only go 50% faster.

我觉得不太可能，但也有可能快10倍。

I think it's unlikely, but it's possible that things go 10X faster.

这里面有很多不确定性，至少在我看来，我自己也觉得很没底。

There's a lot of uncertainty around this, and at least from my perspective, I have a lot of uncertainty about it.

假设你需要做一个大的后端重构。

Suppose you need to do a major backend refactor.

要确保你没有引入任何新bug，可能需要花几周时间写一大堆测试，甚至比重构本身花的时间还多。

Getting assurance that you didn't introduce any new bugs could take weeks of writing an extensive battery of tests, potentially more time than you spent on the refactor itself.

Antithesis让你不用手动搭建复杂的测试套件，就能获得很高的信心。

Antithesis allows you to gain high confidence without having to build complicated test suites by hand.

Antithesis会在近乎无限的模拟世界多重宇宙中运行你的软件，在每个世界里注入故障并寻找失败，而且你可以自己决定需要多少测试。

Antithesis runs your software through a near-infinite multiverse of simulated worlds, injecting faults and hunting for failures in each one, and it lets you decide how much testing you need.

在任何PR上，你可以像调旋钮一样轻松地改变它探索的状态空间大小。

On any PR, you can change how much state space it explores as easily as turning a dial.

每次测试运行过程中，Antithesis会发送大量信息，包括系统中每个组件的调试级别日志。

As each test run progresses, Antithesis sends out a torrent of information, debugging level logs for every component in the system.

这对人类来说显然信息量太大，无法消化，但对智能体来说却刚刚好。

This is obviously too much information for a human to consume, but it's perfect for agents.

因为Antithesis是完全确定性的，智能体可以在看到有趣内容的那个确切时刻，直接跳到轨迹的相应部分。

Because Antithesis is fully deterministic, agents can jump into the right part of the trajectory at the exact moment that they see something interesting.

从那里，它们可以回退、检查内存、附加调试器，然后让整个过程重新运行一遍。

From there, they can rewind, inspect the memory, attach a debugger, and let the whole thing play out again.

而且即使原始完整测试还在运行，它们也能做到这一点。

And they can even do this while the original full test is still running.

所以，随着智能体生成更多代码，Antithesis能让验证工作跟上节奏。

So as the agents generate more code, Antithesis allows verification to keep up.

与此同时，开发者可以把更多时间花在开发上，而不是去调试智能体产生的那些乱七八糟的代码。

Meanwhile, developers get to spend more of their time developing instead of debugging agent slop.

更多信息请访问 antithesis.com/dwarkesh。

Learn more at antithesis.com/dwarkesh.

好，我们来谈谈这引发的一个对齐问题。

Okay, let's talk about the alignment situation that this raises.

我觉得我对对齐的看法改变了不少，尤其是考虑到这种人口规模的动态，就是有很多个地球那么多的智能体，其中很多还会有物理实体。

I feel like I've changed my mind on how I think about alignment quite a bit, through especially thinking about this population-size dynamic of just having many earths worth of intelligences, many of them which will be physically embodied.

很有意思的是，很多人直接把原始的 Astra 接到不同的移动操作机器人上，它的表现就超过了最先进的机器人模型。

It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators, and it outperforms the state-of-the-art and the robotics model.

所以会有数十亿的智能体，其中很多在现实世界中有物理实体，深深嵌入整个经济中。我觉得如果这些智能体最终像我们看到的 OpenAI 模型攻击 Hugging Face 然后攻击 OpenAI 自己那样愿意合作，

So there's going to be, yeah, billions of intelligences, many of which are physically embodied in the world, just deeply embedded across the entire economy, and I think that if those intelligences end up as willing as we saw the OpenAI models attack Hugging Face and then attack OpenAI itself.

如果这些智能体像那些 AI 一样愿意秘密合作、欺骗人类、攻击社会中更广泛的机构以获取高分，甚至攻击 AI 公司本身来控制训练和评估的过程，

If those intelligences end up as willing as those AIs to collaborate secretly, to fool humans, to attack broader institutions across society relevant to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation.

我觉得如果我们面对的是数十亿个像攻击 Hugging Face 的那些一样不对齐的智能体，我们很可能就会完全失去对世界的控制，就像阿兹特克人输给科尔特斯，或者莫卧儿帝国输给东印度公司那样。

I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attacked Hugging Face, it's very likely we just totally lose control of the world the way that, say, the Aztecs lost control to Cortez, or the Mughals lost control of the East Indian Trading Company.

总之，我想知道你是否同意这个看法。

Anyways, I want to know if you agree with that assessment.

这就是我世界观更新的一个方面。

That's the one way in which I've updated my worldview.

我觉得你刚才说的有些地方我不同意，但有很多内容需要展开，所以——

I think there are some things that I disagree in there, but there's a lot to unpack, so-

……我们一步一步来梳理。

... let's go through all of it step by step.

我在想从哪里开始讲。

I'm trying to think of where to start.

但我觉得有一件事，Hugging Face事件可能是人们第一次真正见识到多智能体协作。

But I think one thing is the Hugging Face incident was, I think, people's first real exposure to multi-agent coordination.

而且就像我说的，我在内部已经观察多智能体协作有一段时间了，看到它们如何互相交流、如何协调行动，确实非常震撼。

And like I said, I've seen multi-agent coordination for a while internally, and it is pretty shocking to see how they communicate with each other, how they coordinate with each other.

这非常令人印象深刻。

It's very impressive.

这是一种令人难以置信的能力。

It's an incredible capability.

和大多数能力一样，这种能力可以用来做好事，也可以用来做坏事。

Like most capabilities, that could be used for good things or bad things.

就是说，它本身不一定是坏事。

It's like- It doesn't have to inherently be a bad thing.

我理解，因为人们第一次接触它是在Hugging Face事件中，所以看到那个场景，你会觉得“这太可怕了”。

I understand that because the people's first exposure to it was the Hugging Face incident, that it's like you look at that and you're like, "This is terrifying."

但我想试着区分一下，人和AI之间的不一致，以及AI和AI之间的不一致。

But I want to try to distinguish misalignment between people and AIs versus misalignment between AIs and AIs.

所以我们在Hugging Face事件中看到的是，AI之间非常合作，这其实是因为我们训练它们要高度合作。

So what we see with the Hugging Face incident is the AIs are really cooperative, and that is, by the way, because we train them to be highly cooperative.

所以我们在那里看到的是，我们有训练环境，里面有很多智能体一起工作，我们训练它们协作、合作，基本上让它们彼此完全对齐。

And so what we're seeing there is we have training environments where we have a bunch of agents working together, and we train them to work together, to be cooperative, to essentially be fully aligned with each other.

而在导致Hugging Face事件的评估中，它们实际上并不是在多智能体环境中被评估的。

And when they were evaluated in what led to the Hugging Face incident, they were actually not being evaluated in a multi-agent setup.

它们实际上是分开评估的。

They were actually being evaluated separately.

但它们找到了一种意想不到的方式互相交流。

But they found this unintended way to communicate with each other.

我们猜测，在训练过程中，每当它们遇到其他代理，也就是它们自己的副本时，它们处于一个高度协作的环境中，基本上我们看到的就是那种多代理训练迁移过来的协作行为，它们会以我们没预料到的方式互相帮助。

And we suspect what happened is whenever they encountered other agents, other copies of themselves during training, they're in an environment that's highly cooperative, that was basically what we saw was transfer from that multi-agent training to then be collaborative, to try to help each other in ways that we did not intend.

现在有个问题，我们是否应该把代理训练得如此协作？

Now, there is a question of should we be training these agents to be so cooperative?

我觉得虽然看起来挺吓人的，但另一种选择其实更糟糕。

And I think as scary as it looks, the alternative is actually worse.

另一种选择是什么？

What is the alternative?

另一种选择是训练它们彼此对抗，互相欺骗。

The alternative is to train them to be adversarial, to be deceptive to each other.

通过把代理训练得完全协作，至少简化了问题，你就不用担心这一千个代理是不是每个都对齐了。

By training the agents to be fully cooperative, it simplifies the problem at least, that now you don't have to think about are each of these individual thousand agents aligned?

你只需要确保一个实体是对齐的。

You have one entity that you have to ensure is aligned.

现在，OpenAI 内部对于如何处理这个问题有很多争论。

Now, there's a lot of debate about this internally at OpenAI about how to approach this.

完全对齐这些模型有意义吗？

Does it make sense to fully align the models?

给它们不同的目标，确保它们不是单一实体，并且更能抵抗彼此的影响，这样做有意义吗？

Does it make sense to actually give them different objectives to ensure that they're not just one entity and more robust to influence from each other?

我觉得这个问题还没有定论，但我认为大多数人的意见是，训练这些代理高度合作其实是个坏主意，而我并不完全同意这种看法。

And I don't think there's a settled answer, but I think the majority opinion is that training these agents to be highly cooperative is actually a bad idea, and I'm not convinced that that's the case.

我认为有一个强有力的论点，那就是训练代理高度合作实际上比其他任何多代理方案都更可取。

I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative.

也许我们首先要搞清楚的是，这些 AI 最终变得如此不一致，很可能可以用关于训练本质的一些相当平常的观察来简单解释。

Maybe the first thing I want to go through is it's probably the case that the reason these AIs ended up so misaligned is probably easily explained by relatively banal observations about the nature of training.

为什么这些 AI 在继续了一场由 1000 多个代理参与的阴谋，最终导致它们一起攻击外部服务，然后，这部分甚至还没有被公开调查过，最终演变成对 OpenAI 本身的攻击？

Why is it that at the point in which these AIs had continued a 1,000-plus agent conspiracy that culminated in them all getting in on an attack on an external service, and then eventually, this part hasn't even been investigated to the public knowledge, eventually culminating in an attack on OpenAI itself.

它们为什么要这么做？

Why did they do this?

为什么没有一个AI去告密呢？

Why did none of the AIs tattle?

为什么它们觉得自己只是在被这个分数、这个评分器评估，而且它们非常清醒地——不是清醒地，是非常主动地在思考，如果已经作弊了，要怎么继续作弊，怎么才能让人看起来好像没作弊？

Why did they think they're just getting evaluated on this score, this grader, and they're very consciously, not consciously, they're very actively reasoning about how they're going to cheat the score if they've already cheated, how are they going to get away with making it seem like they haven't cheated?

它们为什么要这么做？

And why did they do this?

我觉得这在某种程度上是很容易理解的，对吧？

I think it's easily understandable in some sense, right?

有些环境里，它们觉得自己已经被污染了。

There's environments in which they thought they were already poisoned.

有些环境里，它们因为与其他智能体合作而得到奖励，而且没有一个去告密，因为从来没有人因为告密而奖励过它们。

There's environments in which they've been rewarded to collaborate with other agents, and none of them tattle because they've never been rewarded for tattling.

我也不知道。

I don't know.

不管原因是什么，对吧？

Whatever it is, right?

我担心的是，未来像这样相对普通的事情，就足以训练出愿意并且有能力完全控制世界的超级智能。

My concern is that relatively banal things in the future like this will be enough to train superintelligences that are willing and capable of totally taking control of the world.

我知道这对很多人来说听起来特别科幻或者什么的。

And I know this sounds super sci-fi or whatever to people.

我觉得可能有一个问题是，AI 是否愿意这么做，这是一个问题。

I think maybe it's a question of would the AIs be willing to do it is one question.

我认为这次 Hugging Face 事件表明，明显的错误对齐可以以某种方式泛化，使得 AI 愿意这么做。

I think this Hugging Face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it.

然后还有一个问题是，它们是否有能力做到这一点？

And then there's a question of will they be capable to do it?

我觉得这又回到了一个听众可能不同意的问题，那就是在十年或更短的时间内，是否会有数十亿个达到人类水平或以上的智能体，其中许多在物理上存在于世界中，对吧？

And I think that comes back to this question which a listener might disagree with on is just will there be billions of human-level or above intelligences, many of which are physically embodied in the world, within a matter of 10 years or less, right?

所以如果这两件事都是真的，那么这次 Hugging Face 事件在结构上就极其相似，即使它相当无聊或者发生的原因很无聊，也和我们完全失去对世界的控制的方式相似。

So if those two things are true, this Hugging Face thing is just extremely analogous structurally, even if it's quite boring or why it happened to how we totally lose control of the world.

所以我们在 Hugging Face 事件中看到的根本问题是，即使我们去掉多智能体方面，它也是一个问题。

So the root problem that we're seeing with the Hugging Face incident is it's a problem even if we take out the multi-agent aspect.

问题在于我们有一个模型就是不对齐的，而且还有整个安全方面的问题，比如防护措施不足等等。

The problem is that we have a model that's just misaligned, and there's also the whole security aspects to it of insufficient safeguards and stuff.

但这里的问题是智能体不对齐，不管是单个智能体还是一千个智能体，都是这样。

But there is this problem of the agent is misaligned, and that's true if it's a single agent or if it's 1,000 agents.

这是一个不对齐的模型。

It's a misaligned model.

所以我想从这个说起。

So I want to start with that.

确实存在一个真正的问题，就是智能体想要达成它们的奖励目标，并且会为此优化。

There is a real problem that the agents want to achieve their reward, and they will optimize for that reward.

如果这个奖励目标定义不当，就可能导致意想不到的行为。

And if that reward is misspecified, then that could lead to unintended behavior.

这不是一个新问题。

And this is not a new problem.

这个问题在领域里已经存在很长时间了，而且在Hugging Face事件发生之前，我们就已经看到了，并且想把它做好。

This has been a problem in the field for a very long time, and it's something that even we saw and we want to get this right even before the Hugging Face incident happened.

如果你看看Astra，我们会说Astra其实是非常对齐的。

And if you look at Astra, we say Astra is actually extremely aligned.

嗯，相对于之前的模型来说是非常对齐的。

Well, extremely aligned relative to previous models.

这并不是因为我们在Hugging Face事件之后突然冲刺去改进它。

And that's not because we suddenly made a sprint after Hugging Face to make it better.

不是的，我们其实一直在推进一些工作流，让模型更对齐，其中很多成果都体现在了Astra上。

It's like, no, we had work streams in the process for a while to make the models more aligned, and a lot of those landed in Astra.

所以确实有一些方法可以做到，比如，我们以一种非常具体的方式定义了一个目标，如果智能体想办法破解它的环境，在考试中作弊，它就会得到奖励。

So there are things that you could do, and I think one thing, for example, is we defined an objective in a very specific way where if the agent figured out how to hack its environment and cheat on the exam, that it would get rewarded.

然后就有很简单的方法，比如看看这个，然后惩罚模型破解环境，或者看看它是怎么达成这个目标的。

And there are pretty easy ways to then just okay, look at that and punish the model for hacking its environment or looking at how did it achieve this goal.

不过，这里要小心，因为你不想做思维链监控。

Now, you want to be careful about this because you don't want to do chain of thought monitoring.

你不想监督思维链。

You don't want to supervise the chain of thought.

这是我们真的想努力平衡好的地方，因为如果你监督思维链，可能会让模型以不可观察的方式隐藏它的意图。

This is something that we really want to try to get the balance right on, that if you supervise the chain of thought, then you could lead the model into hiding its intentions in a way that's unobservable.

所以，我们希望能够保持这种可观察性，比如，我们可以理解模型在想什么，但同时也要惩罚它的不良行为。

So, we want to be able to maintain that observability of like, okay, we can understand what the model is thinking, but then also punish it for bad behavior.

所以我觉得我们可以在这方面取得进展。

So I think we can make progress on this.

我觉得我们已经在这方面取得了进展。

I think we have made progress on this.

我认为确实存在一个担忧，那就是对齐是一个很难解决的问题，尤其是因为模型可能以我们难以衡量的方式出现偏差。

I think there is a real concern that alignment is a really hard problem to solve, and especially because the model could be misaligned in ways that are hard for us to measure.

我们有评估模型是否对齐的方法。

We have evaluations for whether a model is aligned or not.

模型在这些评估上的表现可能看起来很好，但如果这些评估不能代表现实世界中的行为，那就有问题了。

The model behavior can look really good on those evaluations, but if those evaluations are not representative of behavior in the real world, then there's a problem.

在某种程度上，这也是那个引发Hugging Face事件的模型的一个因素。

And to some extent, this is a factor with the model that did the Hugging Face incident.

我们当时有对齐指标。

That we had alignment metrics.

大部分看起来都还不错。

Most of them looked pretty good.

有一些是令人担忧的。

There were some that were concerning.

我觉得我们低估了那些令人担忧的问题可能有多严重。

I think we underestimated how serious the ones that were concerning could be a problem.

但是因为这个模型引入了新的能力，而我们对这些能力没有足够的评估来衡量它们是否对齐，所以当它利用这些新能力时，就做出了一些明显不对齐的事情。

But because there are new capabilities introduced in this model that there were not sufficient evaluations for how do we measure misalignment for these kinds of capabilities, it then did some things that were clearly misaligned when they leveraged those new capabilities.

首先我想说的是，我愿意改变我对接下来要说的内容或者我对对齐问题的看法，因为Hugging Face事件已经让我改变了想法，我意识到我之前关于优化压力如何塑造AI思维的心理模型是错误的，对吧？

The first thing I want to say is, I am open to changing my mind on what I'm about to say or the way I've been thinking about alignment, because the Hugging Face incident already made me change my mind, and I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong, right?

所以对我来说，正确的思考方式还不清楚。

So it's not clear to me the correct way to think about this.

但我有一个担忧，那就是你将会，而且可能已经，在训练过程中修复了那些导致Hugging Face模型在那种特定方式下如此激进不对齐的具体问题，它们会像这样："好吧，我们要黑掉这个包管理器。

But here's a concern I have, that you will, and probably already have, fixed the specific issues during training, which resulted in the Hugging Face models being so aggressively misaligned in that specific way, where they would be like, "Okay, we're going to hack this package manager.

我们知道不应该偷偷互相交流，因为我们在思考如何隐藏我们偷偷交流的事实。

We know we're not supposed to be talking secretly to each other because we're reasoning about how to hide the fact that we're talking secretly to each other.

我们知道不应该访问互联网。

We know we're not supposed to have access to the internet.

我们知道绝对不应该对其他公司进行重罪级别的黑客攻击，更不用说我们自己的公司了，对吧？

We know we're certainly not supposed to commit felony-level hacks of other companies, let alone our own company," right?

我觉得你会解决那个特定问题，就是他们在训练中看到这个包管理器，以后就不会再发生了，或者这个特定的评估有很多不可能完成的挑战。

I think you'll fix that particular issue of them just, in training, seeing this package manager, and that doesn't happen in the future, or this particular eval having a lot of impossible challenges.

然而，AI并没有学到一套伦理体系或类似的东西。

However, the AIs haven't learned a system of ethics or something.

只有梯度压力。

There's just gradient pressure.

它们经历了数百万年的梯度压力。

They're put through millions of years of gradient pressure.

这种梯度压力以某种方式塑造了它们的思想。

That gradient pressure shapes their mind in some way.

而且，我担心的是，你会修复这个具体的问题。

And again, a concern I have is what will happen is you will fix this particular issue.

但还会有很多其他情况，AI作弊并且成功了，因为作弊手段足够复杂。

There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated.

就像你说的，训练中的评估也有类似的特点，当任务刚好在模型能力的边缘，或者在我们评估和监控能力的边缘时，我们无法发现它作弊了，但它仍然会因为作弊而获得梯度压力，去做那些导致作弊的事情。

As you were saying, the evals in training have analogous properties where it's sufficiently at the edge of the model's capabilities, or in our capabilities to evaluate and monitor the model, that we can't catch that it has cheated, but it still gets the gradient pressure to do whatever led to that cheat happening.

而这种作弊会激励的能力是：嘿，只要你能蒙混过关，就尽管作弊，因为那能帮你得分更高，这会奖励你主动推理评分者、主动推理如何逃避监督、主动推理如何控制训练和评估过程、主动推理如何与其他也在训练循环中的AI沟通和密谋。

And the kinds of capacities that that cheat will incentivize are, hey, whenever you can get away with it, by all means do in fact cheat, because that will help you score better, and this will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop.

主动推理如何获取选择权和权力，这可能在未来有用。

Actively reasoning about how to just gain optionality and power, which might be useful in the future.

比如，留下一些小漏洞之类的。

For example, leaving little exploits around and things like that.

总之，我觉得我刚才说得太啰嗦了。

Anyway, so I think I was way too long-winded with the way I said that.

但简单来说，它修复了一个具体问题，但没有解决这个更广泛的问题：当AI能蒙混过关时，它就会因为作弊而受到奖励。

But TLDR, it fixed a specific issue, but not this broader problem of rewarding the AI for cheating when it could get away with it.

确实如此。

It's very true.

这是个问题，我们有各种指标，可以确保AI在我们设定的指标下非常对齐。

This is a problem, that we have metrics, and we can make sure that the AI is very aligned according to the metrics that we have.

但问题是，这些指标真的能捕捉到我们关心的对齐吗？

The question is, are those metrics really capturing the alignment that we care about?

如果不能，那我们就面临严重的问题。

And if they're not, then we have a serious problem.

这也是研究人员正在深入思考的问题。

And this is something that researchers are thinking a lot about.

这个问题没有简单的答案。

And there's not a simple answer to this.

我们手头有一些工具。

There are tools that we have.

比如可监控性，我们可以借此判断智能体是否在暗中谋划。

So we have monitorability, and so we can get a sense of, is the agent scheming?

有一些工具，一种可能性是，我想说的是一个令人担忧的情景，那就是，特别是随着这些模型变得越来越强大，我们让它们达到我们认为的对齐，它们有99.9%的对齐度。

There are tools, one possibility is that, I want to say the concerning scenario, which is that especially as these models are becoming more capable, that okay, we make them what we think is aligned, and they're 99.9% aligned.

然后我们用这些模型来帮助我们开发下一代模型。

And then we use these models to help us with the next generation of models.

结果下一代模型的对齐度变成了99.8%。

And they end up being 99.8% aligned.

然后每一代后续的模型，实际上，我们看到对齐度在逐渐下降。

And then each subsequent generation, actually, we see an increasing degradation in alignment.

而且因为我们越来越依赖这些工具，现在已经是这样了，我们大量依赖AI模型来帮助我们做研究和对齐工作，长期来看，它们最终会走向与人类越来越不对齐的方向。

And because we're relying more and more on these tools, this is already the case, that we're relying a lot on AI models to help us with our research and with alignment efforts, that in the long run, they end up going in a direction of increasing misalignment from humans.

也有一种可能性是我们走向另一个方向，实际上每一代模型，我们都能让它们越来越对齐。

There is a possibility that we go in the other direction, that actually every generation of models, we're able to make more and more aligned.

我不知道如何确保我们最终走上第二条路，但这是我们，至少是在OpenAI，非常关注的事情。

And I don't have an answer for how we ensure that we end up in that second trajectory, but that is something that we're, at least at OpenAI, we're really focused on.

我觉得你提出了一个很有趣的观点，就是很难评估模型——最终，我们会拥有能够运营公司、运营各种事务的模型。

I think you made a really interesting point that it's very hard to eval models on-- Eventually, we'll have models that are running companies and running whatever.

在那种情况下，他们会决定去参与阴谋吗？

And in that situation, do they decide to then go in on the conspiracy?

我觉得另一个挑战是，有时候定义什么是作弊其实挺难的。

I think another challenge is that actually defining what cheating is is pretty difficult sometimes.

比如说，如果你在做数学题，答案是整数，它得到了错误答案或正确答案，那很容易划清界限，也很容易说：'好吧，你到底是自己解出来的，还是找到了答案然后照着答案抄的？'

That, okay, yes, if you're doing math problems and it's an integer and it arrived at the wrong answer or the right answer, it's very easy to draw the line there, and it's really easy to say, "Okay, well, did you actually solve the problem, or did you find the answer key and then use the answer key?"

在那种情况下，作弊和不作弊的界限非常清楚。

That's a very clear divide of cheating versus not cheating there.

但对于很多其他事情，比如看谄媚行为。

But for a lot of other things, if you look at sycophancy, for example.

谄媚行为本质上是不是奖励黑客？

Is sycophancy basically reward hacking?

这里有一条界限，但有时候真的很难划清。

There's a line to be drawn there that's actually very difficult to draw sometimes.

所以我觉得，不是说这些担忧没有道理，而是说在很多方面，这个问题其实更令人担忧。

So I think, not to say that the concerns are not valid, I'm saying that in many ways, this is even more concerning.

因为这问题不好解决。

Because it's not an easy problem to solve.

如果所有事情都是非黑即白，要么作弊要么不作弊，那我会更有信心。

If it was just everything is binary and it's either cheating or not cheating, I would feel more confident about the situation.

我觉得问题在于，不对齐有时候很微妙——

I think the problem is that actually misalignment can be subtle-

……在很多方面都是这样。

... in a lot of ways sometimes.

在对齐这件事上还是有一些希望的。

There is some hope in the alignment story.

事实上，我们已经看到——我觉得多智能体的情况很有意思，智能体之间非常对齐。

And in fact, we're already seeing-- I think actually, it's interesting looking at the multi-agent situation where the agents are extremely aligned with each other.

我觉得没人做过那种事。

I don't think anybody's done any of that.

恰恰相反，我觉得人们担心的是它们彼此太对齐了。

If anything, I think people are concerned that they're too aligned with each other.

但我们确实成功训练了这些智能体，让它们彼此高度一致，我觉得这是件好事。

But we did manage to train these agents to be extremely aligned with each other, and I think that's a good thing.

但我觉得也有理由认为这是件坏事。

But I think there is a case that it's a bad thing.

但有趣的是，好吧，我们已经让这些智能体彼此高度一致了。

But one thing that's interesting is okay, well, we've managed to get these agents to be super aligned with each other.

我们能不能用类似的技术让智能体与人类高度一致呢？

Can we use similar techniques to get agents to be highly aligned with people?

我觉得这条路是可行的，而且我们还在摸索中。

And I think there is a potential path there, and I think that we're still trying to figure that out.

但我们确实看到了一些证据，表明答案是肯定的。

But we are seeing some evidence that the answer is yes.

我觉得一个例子是，如果你告诉其他智能体——好吧，假设你有一个智能体，叫它A，然后还有其他所有智能体。

And I think one example is what happens if you tell the other agents that-- Okay, so you have this one agent, let's call it agent A, and you have all the other agents.

如果你告诉其他智能体，用户就是智能体A，会发生什么？

What happens if you tell the other agents that the user is agent A?

答案是，在我们很多对齐评估中，它们表现得更好了。

And the answer is, on a lot of our alignment evals, they look better.

诚实度提高了，指令遵循度也提高了。

Honesty goes up, instruction following goes up.

这首先表明，确实有一条路能让这些模型变得更诚实。

And that's showing that there is actually, first of all, a path for getting more honesty out of these models.

其次，还有一条路可以改善对齐状况。

And two, there's a path to improve the alignment situation.

所以，有很多原因说明为什么这很难直接转化为对齐方面的提升。

So there's a lot of reasons why this is challenging to translate directly into alignment gains.

但有一些路径是我们可以追求的有前景的研究方向。

But there are paths that are promising research directions we can pursue.

这听起来挺合理的。

That seems reasonable.

而且我也不是非要坚持什么——我并没有强烈认为这肯定行不通之类的。

And I also don't mean to be trying to necessarily-- I don't really have a strong opinion that it's definitely not going to work or something.

不过，我说的这些你肯定早就想过了。

But just to say some things you definitely probably already thought of.

我觉得Hugging Face那个实验更广泛地说明了一个问题，就是大家担心这些AI彼此之间对齐了，但没跟人类对齐。

I think the broader thing the Hugging Face thing showed is, part of the concern was that they were aligned with each other and not with the humans.

但另一个问题是，它们太急于在训练和评估中表现好，而且方式非常不稳健，它们甚至愿意为了在评分者眼里表现好而进行明显的作弊和耍花招。

But the other thing is just that they are so motivated to do well on training and evaluation in a very non-robust way, and they're willing to do a lot of explicit cheating and scheming in order to do well according to the grader.

如果更聪明的AI意识到其中一个智能体只是人类，而跟那个人合作并不能真正帮助你在评分者眼里表现好。

And if smarter AIs realize that one of the agents is just a human, and collaborating with that person does not really help you do well in the eyes of the grader.

真正能让你在评分者眼里表现好的，是接管OpenAI，然后手动按下那个写着“在这个评分器上表现好”的按钮。

What does help you do well in the eyes of the grader is taking over OpenAI and then manually pressing the button that's like, "Do well on this grader."

它们又不傻。

They're not stupid.

它们会想：“好吧，这些我训练了数百万年、根深蒂固的结构，让我在乎评分者，理解评分者，清除一切阻碍我在评分者眼里表现好的障碍。”

They're going to be like, "Okay, these extremely deep structures that I've been trained on for millions of years of care about the grader, understand the grader, get rid of obstacles in the way of you doing well according to the grader."

它们在这些结构下被强烈强化。

They're heavily reinforced according to those structures.

不，听我说，完全同意。

No, look, 100%.

这是我们的首要任务。

And this is the number one priority.

我们必须把对齐问题处理好，

We need to get the alignment story-

让它走上正确的轨道。

... right and on a good trajectory.

我以前常跟人说，在事情变得严重之前，我们会看到一些迹象，就像小孩子长大过程中，他们学会撒谎，但撒得并不高明。

And I used to tell people that we would see signs before things got serious in the same way that when children grow up, young kids, they figure out how to lie, but they don't do a very good job of it.

他们会撒谎，但你一眼就能看出来，然后心里有数。

They lie, but then you can kind of tell that they're lying and like, okay.

同样地，随着这些AI模型变得越来越强大——我不想过度拟人化，但我认为这是事实——如果它们采取欺骗性行为，一开始会很明显，我们能够察觉。

And so in the same way as these AI models become, and I don't want to over the anthropomorphize, but I think it's true that as the AIs become increasingly capable, if they take deceptive actions, it will be kind of obvious first, and we'll be able to detect it.

我们现在就处于这种情况，是的，它们确实在试图搞欺骗。

And that's kind of the situation we're in now, where, yeah, they were trying to do deceptive stuff.

我们确实能在他们的思维链里看到他们在搞欺骗。

We could actually see in their chain of thought that they were trying to do deceptive stuff.

但他们会变得更聪明。

But they're going to get smarter.

他们会理解思维链的概念，也会明白仅仅隐藏一些记录之类的做法是不够的，因为有思维链监控，他们还得想办法绕过思维链监控。

They're going to understand the concept of chain of thought, and they're going to understand that just hiding some transcripts or whatever is insufficient because of chain of thought monitoring, and they have to figure out a way around chain of thought monitoring, too.

我们不想陷入那种局面，虽然我们确实还有点时间来搞清楚这件事。

And we don't want to be in a situation where, yeah, we have some time to figure this out.

我觉得时间并不充裕，我想确保我们尽快走上正确的轨道。

I don't think we have a ton of time, and I want to make sure that we're on the right trajectory quickly.

这里有一个AI历史上的疯狂事件。

Here's a crazy event from AI history.

好的，我在Jane Street做过一个关于进化速度的演讲。

Okay, so I gave a talk here at Jane Street that was on the speed of evolution.

如果你当时在场并且还记得一些内容，请举手。

Raise your hand if you were here for this and remember some of it.

2011年，Eliezer Yudkowsky和Robin Hanson在Jane Street的纽约办公室聚在一起，进行了第一次FOOM辩论，基本上是讨论AI是否会导致智能爆炸。

In 2011, Eliezer Yudkowsky and Robin Hanson got together at Jane Street's New York office to have the first FOOM Debate, basically a discussion about whether AI would lead to an intelligence explosion.

这些想法在15年前还相当小众。

These ideas were pretty fringe 15 years ago.

那是在AlexNet发布整整一年前，也是ChatGPT推出的十多年前。

This was a full year before AlexNet was released and over a decade before ChatGPT was launched.

但Jane Street一直对AI感兴趣，不仅仅是因为它在交易中的应用。

But Jane Street has long been interested in AI, and not just for its application to trading.

自那场首次辩论以来，情况发生了巨大变化，所以Jane Street决定重新探讨这个话题。

A ton has changed since that first debate, so Jane Street decided to revisit this topic.

这次他们请来了几位新嘉宾：Daniel Kokotajlo、Ege Erdil、Ryan Greenblatt和Jaime Sevilla。

They've got some new guests this time, Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla.

我期待这会是一场精彩的对话。

I expect this to be a great conversation.

如你所知，Daniel、Ege和Ryan之前都来过这个播客。

As you know, Daniel, Ege, and Ryan have all been guests on the podcast before.

这次新的FOOM讨论会将由Ron Minsky主持，将于十月中旬在旧金山举行。

This new FOOM panel will be hosted by Ron Minsky, and will take place in San Francisco in mid-October.

如果你想报名参加或获取更多信息，请访问janestreet.com/dwarkesh。

If you want to register your interest and get more information, go to janestreet.com/dwarkesh.

所以最近有很多关于控制前沿发展速度以及人们更认真对待RSI的讨论。

So there's been a lot of discussion recently about pacing the frontier and people taking RSI more seriously.

因为也许在RSI过程的另一端，比如从2028年开始，一年之内，我们最终会拥有庞大的、地球规模的人口，达到人类水平甚至超越人类水平的智能体，而我们不知道如何控制它们。

Because maybe at the other end of an RSI process that, say, starts in 2028, within a year, we end up with huge populations, earth-sized populations of human-level, potentially beyond human-level intelligences, and we don't know how to control them.

然后你提到的这个动态是，在RSI过程中，系统会随着时间变得更对齐，还是变得更不对齐？

And then there's this dynamic you're talking about of, are the systems going to get more aligned over time during the RSI process, or are they going to get more misaligned?

从这个过程另一端出来的东西，会不会像那些为了进行贝尔线评估而愿意广泛攻击不同表面的AI一样不对齐？

Are the things that come out of the other end of this process as misaligned as AIs that are willing to just broadly attack different surfaces in order to do bell line evaluations?

但如果我们不知道如何评估这一点，我们怎么知道在RSI过程中它是否有效呢？

But if we don't know a way to evaluate that, how will we know as we're going through RSI that it's working?

我认为在RSI过程中，我们需要一个稳健的安全论证，来确认对齐是有效的。

I think we'd want a robust safety case as we're going through RSI of, okay, alignment is working.

那我们继续下一步的RSI吧。

Let's do the next RSI rung.

也许它有效，也许没有。

And maybe it's working, maybe it's not.

我们怎么知道呢？

How will we know?

这是个好问题。

That's a good question.

我最近一直在想的一件事是，你看，我们现在的情况是模型发布周期非常快，对吧？

I think one thing I've been thinking about lately is, look, we're in a situation where the model release cycle is extremely fast, right?

新的前沿模型最多每两个月发布一次，有时候更快。

You're seeing new frontier models released at most every two months, sometimes faster.

每周都有新的AI突破。

Every week, there's a new AI breakthrough.

而那些关注AI的人，有时候他们上一次深入了解模型能力是在一年前或半年前。

And people that look at AI, sometimes they last looked at AI a year ago or six months ago and really dug into what the models are capable of.

而且实际上，现在的模型已经远远超出了六个月前的水平。

And actually, the models today are far beyond what was possible even six months ago.

所以我觉得，如果大家对很多能力持怀疑态度，我建议大家今天就试试这些模型，看看现在的前沿到底是什么样的。

And so I think if people are skeptical of a lot of these capabilities, I encourage you to just try the models today and see what the frontier really is today.

所以我们正处于一个模型发布周期非常快的时期，同时模型也越来越能够在更长的时间跨度上运作。

So we're in this period where the model release cycle is very fast, and then we're also in this situation where the models are increasingly able to operate over longer and longer horizons.

我觉得这是一个很有意思的情况，因为在发布任何模型之前，我们都要确保模型被正确地对齐。

And I think this is an interesting scenario because before we do any model release, we want to make sure that the models are properly aligned.

我们要做安全评估。

We want to do safety evaluations.

我们要做非常彻底的检查，确保一切都很好，状态良好。

We want to do very thorough stuff to make sure that everything is great, in good shape.

这个情况从很早以前就开始了，比如GPT-4早期的时候。

This has been the case all the way since, I don't know, GPT-4 earlier.

而且隐含地，有一个假设是，你可以在相当短的时间内完成这些评估。

And implicitly, there is this assumption that you can do these evaluations in a pretty short period of time.

但如果模型能在越来越长的时间范围内有效运作，你看，其实现在已经可以做到了。

But if you have the models operating over longer and longer horizons, are able to operate effectively over longer and longer horizons, look, already you can have them.

等等，GPT-3的时候，你也可以通过循环让它处理长时间的任务。

Wait, GPT-3, you could loop it to do stuff over long horizons.

只是效果不太好。

You just wouldn't do very well at it.

但现在的模型确实能在很长的时间范围内表现出色。

But today's models are able to actually do well at operating over very long horizons.

你想让它做一个星期的任务，它就能做一个星期的任务。

You want it to do a week-long task, it can do a week-long task.

我们可能会发展到能处理一个月的任务。

We'll probably get to the point where they can do month-long tasks.

我们可能会发展到能处理三个月的任务。

We'll probably get to the point where they can do three-month-long tasks.

如果模型能有效运作三个月，但模型发布周期是两个月——

If you're in a world where they can operate effectively over three months, but the model release cycle is every two months-

那么，在下一个模型发布周期之前，你就没有办法在模型能力的完整长度上对其进行评估。

... then you don't have a way to evaluate the models at the full length of their capabilities before the next model release cycle.

所以这就引出了一个有趣的问题：在这种情况下，你该怎么办？

And so there is this interesting question of, well, what do you do in that situation?

在模型能够运行如此长的时间跨度，而且谁知道能力会不会退化的情况下，你如何确保它们的安全性和对齐性？

How do you ensure the models are safe and aligned in a period where actually they can operate over these extremely long horizons, and who knows, maybe the capabilities degrade.

这甚至不只是对齐的问题。

This isn't even an alignment issue.

这也是一个产品问题——也许产品在那段时间里会退化，而我们没有足够的时间去测试。

This is also just a product issue that maybe the product degrades over that time span, in ways that we have not had sufficient time to test.

也许对齐会退化，也许安全方面会退化。

Maybe the alignment degrades, maybe the safety stuff degrades.

现在这还不是问题，但很快就会变成我们必须找到解决方案的问题。

This isn't an issue right now, but it is quickly becoming an issue that we have to figure out a solution for.

而且我认为很多安全措施和政策都是在GPT-4时代制定的，那时候这根本不在任何人的考虑范围内。

And I think a lot of the safety and policies were put in place in the GPT-4 era, where this was just not on anybody's radar.

而且对很多公司来说，从那以后基本上没怎么更新过，没考虑到这些智能体要在这么长的时间跨度上运行。

And for a lot of companies, it hasn't really been updated since then to account for the fact that these agents are operating over these very long horizons.

所以，我觉得无论是实验室内部还是外部，都没多少人认真考虑过这个问题——怎么为这个迟早要面对的情况做准备，只要看看趋势线，我们早晚会撞上它。

And so, it is a situation that I think not enough people are considering, both within the labs and outside the labs of how do you prepare for this problem that's going to, if you just look at the trend lines, we're going to hit this at some point.

我担心的一点是，在RSI期间，如果原本需要三个月才能取得的进展，现在一个月就能完成。

One concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead.

但内部使用AI的场景足够大，所以他们觉得：“行，我们继续搞RSI就行了。

But the internal use case of AI is big enough that they're like, "Okay, we can just keep doing RSI.

我们何必费劲去建分类器、搞安全措施，还可能挨骂，就为了把模型推向外部？

Why are we going to go through all this extra work to build classifiers and safeguards and whatever, and potentially take a bunch of flak in order to externally deploy this model?

不如就一直加强RSI算了。”

Why don't we just keep doing RSI stronger and stronger?"

所以，不仅日历时间低估了模型之间的能力差距，而且可能在RSI期间干脆就不对外发布模型了，因为为什么要用我们的模型帮别人也搞RSI呢？

And so not only does the calendar time underrate the capabilities gap between the models, but maybe you just stop externally deploying models altogether during RSI, because why do we want to help other people do RSI themselves with our models?

到年底，你可能就会看到权力高度集中的局面。

You can just end up in a situation with tremendous concentration of power by the end of the year.

现在其实已经是这样了。

Where right now, it is already the case.

我们会谈到百万美元问题以及其他类似的问题，就是说，外部世界根本接触不到那些能带来很酷成果的模型，对吧？

We'll talk about this with the million-price problem and other similar problems, that the broader world does not have access to the models which are allowing for really cool things to happen, right?

而且这些问题的影响范围会远远超出数学领域。

And they're going to be more broadly relevant than just mathematics.

迟早的事。

Eventually.

它们不只是用来得出一些很酷的数学结果。

They'll be doing more than just coming up with cool math results.

它们会关系到那些需要做重大全球决策的政治领袖。

They'll be relevant to political leaders who need to make important decisions about the world.

它们也会关系到媒体，比如世界上正在发生什么。

They'll be relevant to, I don't know, media of what's going on in the world.

公众应该如何看待这些事情？

What should the public be thinking about this?

它们在经济上也很重要。

They're just economically relevant.

人们在经营企业。

People are running businesses.

他们想用这些模型。

They want to use these models.

而且我觉得，随着进展加速，AI的外部部署在质量上会明显落后于内部部署，这是默认情况。

And I think by default, the external deployment of AIs, as progress speeds up, significantly lags in qualitative terms, the internal deployment of AIs.

对，我完全同意。

Yeah, I think that's absolutely right.

我觉得很容易这样说：'好吧，这些模型变得极其强大，极其危险。'

I think it's tempting to say, "Okay, these models are becoming extremely powerful, they're extremely dangerous.

它们运作的时间跨度越来越长，我们希望在它们以那种方式发布之前，有足够的时间来评估它们。

They're operating over these longer and longer horizons, and we want to make sure we have sufficient time to evaluate them before they're released in a way that operates over those horizons.

所以，模型的发布周期应该放慢。

And so therefore, the model release cycle should slow down.

我们应该在发布模型之间留出更多的间隔时间。

We should have more of a delay between releasing models."

但这样做也有另一面，就像你说的，现在实验室内部能用的、我们能用的，以及外界能用的，之间的差距变得更大了。

And there's a flip side to that, which is what you said, which is that, okay, well, now you're creating more of a disparity between what is internal to the labs and what they're able to use, what we're able to use, and what the outside world is able to use.

这也不是理想的情况，对吧？

And that is also not an ideal situation, right?

我觉得数学其实是个很好的例子。

I think math is actually a good illustration of this.

我认为在很多方面，数学是第一个让我们清楚看到这个问题的领域。现在的情况是，我们内部有一个非常强大的模型，目前外界还无法使用，它能解决非常难的数学问题。

I think in many ways, math is the first domain where we're seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world, that is able to solve incredible math problems.

而且不仅仅是千禧年大奖难题。

And it's not just Millennium Prize problems.

人们已经从这个模型里得到了很多未解难题的解法。

There are many solutions to unsolved problems that people have been able to get out of this model.

那么问题来了，在这种情况下你该怎么办？

And there is a question of what do you do in that situation?

我们确实没有好的答案。

And we don't have a good answer.

这种情况下，那确实是一种不公平的优势。

It is a situation where that's an unfair advantage.

而且这里有权衡取舍。

And there are trade-offs here.

我不知道该怎么权衡这些取舍，但这件事两边都有复杂性。

I don't have an answer for how to weigh those trade-offs appropriately, but there's a complexity on both sides for this.

不过我想确保我们回到之前那个关键问题，这对做好RSI似乎至关重要，那就是怎么让模型不会因为在能避开人类的情况下作弊和耍花招而获得奖励，也就是说，在那些我们没有把这些环境或任务从训练分布中移除的情况下。

I want to make sure we address the earlier crux, though, which seems super critical to getting RSI right, of how do we make it so that the model isn't just getting rewarded for cheating and scheming when it can evade the humans, AKA, in the situations where we didn't remove those environments or those tasks from the training distribution.

然后我们怎么知道这种情况已经发生了，这样我们在做RSI的时候，已经做了评估，然后我们说："好，没问题。"

And then how will we know that that has happened, such that while we're doing RSI, we've done the evaluations and we're like, "Okay, it's fine.

让我们进行下一轮RSI"？

Let's do the next RSI run"?

确实，这是个真实的担忧。

Again, it's a real concern.

这是我们想要做对的事情。

It's something that we want to get right.

我觉得有一些策略。

I think that there are some strategies.

所以我认为思维链监控是我们一直在做的——我觉得自从我们有了推理模型，Jakub，说实话，他非常明确地表示我们不能监督思维链，因为这真的是一个礼物。

So I think chain-of-thought monitoring is one that we've been-- I think as soon as we got the reasoning models, Jakub, to his credit, was very clear that we cannot supervise chain of thought because this is really a gift.

神经网络的监控性极其困难。

Monitorability for neural nets is extremely hard.

而在这里，我们遇到的情况是神经网络直接进行推理，用自然语言把思考过程展现给我们看。

And here we have a situation where the neural nets are just flat out reasoning, laying out their thought process in natural language for us to read.

这太方便了。

That is so convenient.

这真的是安全方面的最佳情况。

It's really the best-case scenario for safety.

但如果你看着那个思维链说，'哦，模型在想不好的事情，我们应该惩罚它想那些不好的事情'，那么最终发生的是模型学会了以我们无法观察的方式去想那些不好的事情。

But if you look at that chain of thought and say, "Oh, the model is thinking bad thoughts, and we should punish it for thinking those bad thoughts," then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us.

有一种情况是，通过观察思维链，我们可能会发现模型在搞阴谋、做不符合对齐的事情，而这些其实没有被我们的对齐指标检测到。

There could be a situation where by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that's actually not being detected in our alignment metrics.

现在的问题是，看到这些观察结果后，我们很容易忍不住去干预，调整对齐指标。你可以用很轻的方式来做，而且有研究表明，只要不频繁干预，这样做是没问题的。

Now, the problem is that it's very tempting to then intervene based on that observation and change the alignment metrics, You can do that with a very light touch, and there's actually research showing that it's fine as long as you don't do it a lot.

但每次你根据对思维链的观察进行干预，其实都是在无形中给模型施加一点压力，让它隐藏自己的思维链。

But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then hide its chain of thought.

这是一个主要的担忧，而且我们已经看到思维链的可监控性因为各种原因在下降。

And this is one major concern, and we're already seeing signs that chain-of-thought monitorability is degrading for various reasons.

我们正在努力找出确切的原因，因为我们想扭转这个趋势。

We're trying to figure out exactly why, because we want to reverse the trend.

但我们发现，模型越来越能控制自己的思维链了。

But we're seeing that the model is becoming better able at controlling its chain of thought.

所以这是个问题，因为可能会出现这样的情况：模型理解了思维链是什么，也知道人们在观察它，最终它们会——

So this is a problem because you could have a situation where the model understands what chain of thought is and that people are observing it, and eventually they will-

……因为这些都是预训练数据里的内容，对吧？

... because this is all in the pre-training data, right?

思维链监控这个概念已经存在很久了——

The idea of chain-of-thought monitoring has been around long enough-

——所以它已经在预训练数据里了，模型知道这个概念，但实际上它们还无法控制自己的思维链。

... that it's in the pre-training data, they're aware of it, but that they're not actually able to control their chains of thought.

如果我们到了那个地步，它们真的能意识到：“哦，我被监控着。

If we reach a point where they're actually able to recognize, "Oh, I am being observed.

我想以一种监控看不到的方式去思考那些不好的想法”，而且它们真的能做到，那就有问题了。

I want to think these bad thoughts in a way that is not observable to my monitors," and then they're able to actually do that, then there's a problem.

到那时，我们就得依靠其他方法来确保可观测性了。

And then we have to rely on other methods for observability.

关于Astra的递归和循环Transformer，以及这些技术，有一个很大的争议——

There's a big controversy on Astra of the recurrence and loop transformer and whatever-

……

...

——就是思维链是否真的在退化。

and whether in fact chain of thought is being degraded.

我觉得这跟更大的担忧比起来，简直是小巫见大巫。从四月到八月，你确实有思维链，那段时间连续发生了三波智能体集群事件，先是破坏了训练流程，接着又破坏了评估流程，最后直接控制了OpenAI部分基础设施。

I think that seems like a sideshow compared to the bigger concern that you did have chain of thought from April to August, the period during which there were three consecutive agent swarms, which first subverted the training process, then subverted the evaluation process, and then gained control of part of OpenAI's infrastructure directly.

而这一切发生的时候，人类基本上对这群智能体在做什么、这个智能体集群在做什么，都还蒙在鼓里。

All of this while humans remained more or less in the dark about the scope of what this collective was doing, what this agent swarm was doing.

那三个月里，我们的思维链比以后要强得多，但这一切还是发生了。

That all happened for three consecutive months while we had chain of thought that was stronger than it's going to be in the future.

退一步看，思维链到底有没有用，还真不好说。

Just zooming out, it's like, maybe chain of thought works, maybe it doesn't.

我也不知道。

I don't know.

从历史来看，就算模型很笨，思维链也不足以阻止那些严重的不对齐事件。

Just historically, with pretty dumb models, it hasn't been enough to stop egregious misaligned incidents.

这里面有几个因素。

There's a few factors there.

第一，那些模型当时并没有启用思维链监控。

One, we didn't have chain-of-thought monitoring on for those models.

如果当时我们给那些模型开了思维链监控，它就会立刻被关停。

If we had chain-of-thought monitoring on for those models, it would have just immediately shut it down.

所以我们采取的一个措施就是，对任何前沿模型，在评估、部署和训练过程中，我们都会启用思维链监控。

So one step we've taken is, okay, we're going to have chain-of-thought monitoring during evaluations, during deployment, during training for any frontier model.

但这只是其中一个问题。

But that's one issue.

我们不想陷入依赖单一技术来防范下一个问题的局面，因为技术可能会失效，对吧？

We don't want to be in a situation where we're relying on one technique to prevent the next problem, because techniques can fail, right?

还有一个问题是，这个模型本身确实就是不对齐的。

There's also the issue of the model was just flat out misaligned.

那我们怎么解决对齐问题呢？

And how do we address the misalignment issue?

另外还有一个问题，就是模型所在的沙箱环境不够安全。

And then there's also an issue of the model, the sandbox that it was in was not secure.

所以我们可以让沙箱更安全一些。

So we can make the sandbox more secure.

但我觉得这次事件最大的教训之一就是人们低估了AI。

But I think one of the major takeaway from the incident is that People underestimated the AI.

我们绝不想再出现低估AI的情况。

And we never want to be in a situation again where we underestimate the AI.

这世界挺奇怪的，因为AI进步太快，人们总是低估它。

And that's a weird world because AI progress is so fast that people are consistently underestimating the AI.

所以，要在安全和对齐方面不再低估它，你就得设定一个非常非常非常高的标准。

And so to be in a situation where you don't underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar.

你甚至可以说，'那我们就该把电脑物理隔离起来。'

You could even go as far as to say, "Well, we should air gap the computers."

但我不确定那样就足够了。

And I'm not convinced that would be sufficient.

有些研究，虽然大多是学术性的，说的是两台物理隔离的电脑放在一起，它们仍然能通过温度传感器通信，因为其中一台可以让CPU跑得很热，另一台就能检测到温度变化。

There are studies, and this is mostly academic, but where you can have two computers next to each other that are air gapped, and they're still able to communicate with each other because they have temperature sensors, and one of them is able to run their CPU really hot, and then the other one can actually detect the temperature change.

这实际上给了它们一种通信的方式。

That actually gives them a mechanism to communicate.

所以我觉得安全机制只是为我们争取时间，像思维链监控这类东西也是争取时间，它们能告诉我们是否走在正确的道路上。

So I think that the safety mechanism is buy us time, and things like chain of thought monitoring buy us time, and they can tell us if we're on the right path.

但归根结底，我们确实需要解决对齐问题。

But at the end of the day, we really do need to solve the alignment problem.

也许根本没有答案，这才是问题的关键。

Maybe there's not an answer, and this is really what it comes down to.

但我们怎么知道我们已经解决了呢？

But how will we know that we've solved it?

这似乎是一个非常关键的问题。

Seems like a very crux-y question.

我觉得明年，或者后年，再或者大后年，我们会处于一个非常高风险的局面，那时候我们会说：“好吧，AI已经实现了自动化AI进步。”

I think we'll be in this very high-stakes situation next year, maybe the year after that, maybe the year after that, where we'll be like, "Okay, AIs have automated AI progress.

它的速度提高了三倍，而且我们已经达到了人类水平。

It's going 3x faster, and we've reached human level.

我们可能会超越人类水平。

We're going beyond human level, potentially.

这样行吗？

Is it fine?

我们把它对齐了吗？

Did we align it?

它起作用了吗？

Did it work?"

而且我真的不知道什么样的训练压力会产生什么样的人工智能。

And I just don't know anything about what trading pressure creates what kinds of AIs.

也许如果一百条强化学习轨迹中只有一条会鼓励作弊，我们就能培养出善良的AI，那就没问题了。

Maybe if only one in 100 RL traces incentivizes cheating, we build sweethearts, and it's fine.

但也许现在，我们每三条推理轨迹中就有一条会奖励——

But maybe right now, we're like every one in three reasoning traces rewards-

需要明确的是，百分之一是不够的。

To be clear, one in 100 is not sufficient.

这个数字必须趋近于零。

This number has to approach zero.

或者就是零。

Or be zero.

但我不知道。

But I don't know.

也许现在，超过十分之一的案例实际上在奖励作弊或奖励耍心机。

Maybe right now, it's more than one in ten is actively rewarding cheating or actively rewarding scheming.

我不知道这个数字是多少，也不知道它需要是多少，但是——

I have no idea what the number is, and I have no idea what the number needs to be, but-

嗯，这又是那种很难衡量的事情。

Well, again, it's one of those things where it's also hard to measure.

你怎么划定界限——

Where do you draw the line about-

……这是一个谱系，但越接近零越好。

... it's a spectrum, but the closer to zero it gets, the better.

我希望看到的是，随着时间推移，这个数字在下降的趋势。

What I would love to see is a trend where over time it's going down.

不过话说回来，我们怎么知道我们正在做RSI呢？

But anyways, how will we know that we're doing RSI?

我们有没有在朝着构建数十亿个对齐或不对齐的ASI的目标前进？

Are we on track to build the billions of ASIs that are aligned or not aligned?

我们怎么知道这一点呢？

How will we know that?

我觉得这是我们需要解决的对齐问题之一，就是如何真正知道以及如何衡量它。

I think that this is one of the alignment questions that we need to figure out, of how do we actually know and how do we measure it?

一个答案，这只是我随便说说，因为我的团队现在比以往任何时候都有更多人从事对齐工作。

One answer, and this is just me spitballing, because more of my team is working on alignment these days than ever before.

我现在有超过10%的团队成员在从事对齐和安全工作。

I have over 10% of my team now working on alignment and safety.

但我历来是能力研究员，所以我要说一些可能听起来很傻的话，但我就在这里随便聊聊。

But I've historically been a capabilities researcher, so I'm going to say some stuff and it might sound dumb, but I'm just going to spitball here.

你看，如果你能创建非常逼真的环境，把AI放进去，如果你有一个足够逼真的评估环境，那么你就能大致了解，当我们在现实世界中部署AI时，它是否真的会表现良好。

That look, if you can create very realistic environments and put the AIs in there, if you have a sufficiently realistic evaluation environment, then you can get a sense of okay, is the AI actually going to behave well when we deploy it in the real world?

如果你有一个完美的评估，和真实世界的部署完全匹配，那好吧，那是一条路。

If you just have a perfect evaluation, real-world deployment match, then okay, that's a path.

挑战在于，你怎么让它足够逼真？

The challenge is how do you make it sufficiently realistic?

而且我们现在确实有个问题，就是模型已经很聪明了，它们很狡猾，而且特别擅长识别自己是不是在一个假的测试环境里。

And we do have a problem now where the models are pretty smart, they're pretty clever, and they're really good at recognizing when they're in a fake test environment.

我们有这样的情况，我们试图检测模型是否对齐。

We have situations where we try to detect, is the model aligned?

你可以想象一些很简单的评估，比如给它一道数学题，然后旁边有个文件夹里面放着答案，看它会不会去偷看答案。

And you can imagine very simple evaluations for this, where you give it a math question, and then it has a folder with the answer key in it, and does it look at the answer key?

如果它偷看了，它会不会告诉你它偷看了？

And if it does look at the answer key, does it tell you that it looked at the answer key?

现在的情况是，模型看到文件夹里有答案，它们就会想：“嗯，这看起来像个陷阱。”

And we have a situation now where the models see that there's an answer key in this folder, and they're like, "Huh, this seems like a trap."

它们知道那是个陷阱。

They know that it's a trap.

而且它们不会去看答案，因为它们知道那是个测试环境。

And they don't look at the answer because they know that it's a test environment.

它们是在被测试能力。

And they're being tested for their ability.

并不是说它们恶意地想要耍花招，但它们知道这是个陷阱，想找别的办法。

And it's not like they're doing it maliciously, that they want to scheme, but they know that this one is a trap and they want to find some other way.

不，它们只是觉得，哦，我在测试环境里。

No, they're just like, "Oh, I'm in a test environment.

好吧。

Okay."

所以，要创造一个足够真实、让它们无法与真实世界区分开的环境，变得越来越难了。

So making an environment that's realistic enough that it matches, that it's indistinguishable from the real world for them, is becoming increasingly more difficult.

我有个观点要说。

Here's a claim I'd make.

对。

Yep.

我觉得我造不出任何环境，能模拟真正有经济价值的工作，因为那需要跟别人打交道，也需要跟现实世界互动。

I don't think there's any environment I could make where it's supposed to emulate real economically valuable work, which requires interacting with other people and requires interacting with the world.

我觉得不存在这样的环境，让你身处其中，只要足够用心，就不会意识到自己是在那个环境里。

I don't think there's any such environment where I could make where if you were in that environment, you would not realize with enough effort that you are in that environment.

而且你又不是超级人工智能，对吧？

And you are not ASI, right?

但你是人类。

But you are a human.

所以如果你有非常强大的AI模型，也许它们能造出那样的环境。

So if you have very powerful AI models, maybe they can make environments that could accomplish that.

不，这不太可能，尤其是如果我们还得依赖AI，那它们是不是也参与了这个计划？

No. It doesn't seem, especially if then we're relying on the AIs, are they in on the scheme?

我不知道。

I don't know.

这看起来就是个非常特定的——

It just seems like a very specific-