达里奥·阿莫代伊（Anthropic首席执行官）—— 每一次AI突破背后的隐藏模式

我不认为这需要大量额外的计算资源。

I don't expect that to require a substantial amount of additional compute.

我认为这可能是由于对强化学习的思考方式有误，低估了模型自身已经学到的东西。

I think that that was probably an artifact of, yeah, kind of thinking about RL in the wrong way and underestimating how much the model had learned on its own.

关于我们是否会在某些领域达到超人水平，而在其他领域则不会，你怎么看？

In terms of, are we gonna be superhuman in some areas and not others?

我认为这很复杂。

I think it's complicated.

我可以想象，我们可能在某些领域无法达到超人水平，比如那些涉及物理世界具身化的问题。

I could imagine that we won't be superhuman in some areas because, for example, they involve embodiment in the physical world.

那么接下来会发生什么？

And then it's like, what happens?

AI会帮助我们训练出更快的AI，而这些更快的AI反过来解决这些问题吗？

Do the AIs help us train faster AIs, and those faster AIs wrap around and solve that?

你真的不需要物理世界吗？

Do you not need the physical world?

这取决于你指的是什么。

It depends what you mean.

我们是否担心对齐灾难？

Are we worried about an alignment disaster?

我们是否担心滥用，比如制造大规模杀伤性武器？

Are we worried about misuse, like making weapons of mass destruction?

我们是否担心人工智能取代人类进行研究？

Are we worried about the AI taking over research from humans?

我们是否担心它达到某种经济生产力阈值，从而能够完成普通人所能做的各种任务？我认为这些不同的阈值有不同的答案。

Are we worried about it reaching some threshold of economic productivity where it can do what the average these different thresholds, I think, have different answers.

尽管我怀疑它们都会在几年内相继出现。

Although I suspect they will all come within a few years.

让我问一下

Let me ask

关于这些阈值。

about those thresholds.

如果Claude是Anthropic的一名员工，它的薪资应该值多少？

So if Claude was an employee at Anthropic, what salary would it be worth?

它到底在多大程度上显著加速了AI的发展？

What is it, like, meaningfully speeding up AI progress?

它

It

在我看来，它在大多数领域就像个实习生，但在某些特定领域表现得更好。

feels to me like an intern in most areas, but then some specific areas where it's better than that.

再说一次，我认为比较困难的原因之一是，它的形态和人类并不相同，对吧？

Again, I think one thing that makes the comparison hard is like the form factor is kind of like not the same as a human, right?

比如，如果你像这些聊天机器人一样行事，我们其实不会——我的意思是，也许我们可以进行这样的对话。

Like, you know, if you were to behave like one of these chatbots, like we wouldn't really I mean, I guess we could have this conversation.

它们的设计初衷更倾向于回答单个或少数几个问题。

It's like, but, you know, they're they're not really they're more designed to answer single or a few questions.

对吧？

Right?

而且，你知道，它们没有拥有长期过往经验的概念。

And and like, you know, they don't have the concept of having a long life of prior experience.

对吧？

Right?

我们这里谈论的是，我过去经历过的一些事情。

We're talking here about, you know, things that that I've experienced in the past.

对吧？

Right?

而聊天机器人并没有这些。

And chatbots don't don't have that.

所以有很多东西是缺失的，因此很难进行比较。

And so there's there's all kinds of stuff missing, and so it's hard to make a comparison.

但我不确定。

But I don't know.

它们在某些方面像实习生，但在某些领域又表现得特别突出，可能比在座的任何人都更出色。

It it they they feel like interns in some areas and kind of then they have areas where they spike and are really savants where they may be better than they may be better than anyone here.

但关于智能爆炸的整体图景，我的前一位嘉宾卡尔·舒尔曼有一个非常详细的模型，作为一个真正会看到这种情况发生的人，你觉得从实习生到初级软件工程师的转变合理吗？

But does the overall picture of something like an intelligence explosion you know, my my former guest is Karl Schulman, and he has this, like, very detailed model of an does that, as somebody who would actually see that happening, does that make sense to you as they go from interns to entry level software engineers?

这些初级软件工程师的生产力会提升，你的

Those entry level software engineers increase I your

我认为人工智能系统会变得更高效，首先它们会提升人类的生产力，然后逐渐达到与人类相当的水平，最终在某种意义上成为科学进步的主要推动力，这个过程在某个时刻会发生。

think the idea that the AI systems become more productive, and first they speed up the productivity of humans, then they kind of equal the productivity of humans, and then they're, in some meaningful sense, the main contributor to scientific progress, that that happens at some point.

我觉得这个基本逻辑对我来说是合理的，尽管我怀疑当我们深入细节时，实际情况可能会和我们预期的有点奇怪和不同。

I think that basic logic seems likely to me, although I have a suspicion that when we actually go into the details, it's gonna be kinda like weird and different than we expect.

所有这些详细模型都像是，我们在思考错误的东西，或者我们对一件事是对的，但对其他十件事都错了。

That all the detailed models are kind of, you know, we're thinking about the wrong things, or we're right about one thing and then are wrong about 10 other things.

所以我不确定。

And and so I I don't know.

我觉得我们最终可能会进入一个比预期更奇怪的世界。

I think we might end up in, like, a weirder world than we expect.

当你把所有这些综合起来，你对人类水平人工智能出现时间的估计是什么样的？

When you add all this together, like your estimate of when we get something kind of human level, what does that look like?

我的意思是，这 again 取决于阈值。

I mean, again, it depends on the thresholds.

是的。

Yeah.

你知道，从某人观察这些模型的角度来看，即使你和它聊上一个小时，它也基本上就像一个受过良好教育的人类。

You know, in terms of someone looks at these, the model, and you know, even if you talk to it for an hour or so, it's basically like a generally well educated human.

我认为，这可能并不遥远。

That could be not very far away at all, I think.

比如，这可能在两三年内就会发生。

Like that could happen in two or three years.

比如，如果我再看一遍，我认为唯一可能阻止它的因素是我们达到某些安全阈值，我们内部有安全标准之类的测试。

Like if I look at, again, I think the main thing that would stop it would be if we hit certain, and we have internal tests for safety thresholds and stuff like that.

所以，如果一家公司或整个行业决定放慢步伐，或者我们能够促使政府出台一些限制措施，以安全为由减缓进展速度，那才是它不会发生的主要原因。

So if a company or the industry decides to slow down or we're able to get the government institute restrictions that kind of, you know, that moderate the rate of progress for safety reasons, that would be the main reason it wouldn't happen.

但如果你只看规模化所需的后勤和经济能力，我认为我们离那个目标其实并不远。

But if you just look at the logistical and economic ability to scale, I don't think we're very far at all from that.

现在，这可能并不是模型具有存在性危险的临界点。

Now that that may not be the threshold where the models are existentially dangerous.

事实上，我怀疑它还远未达到那个程度。

In fact, I suspect it's not not quite there yet.

它可能也不是模型能够接管大部分人工智能研究的临界点。

It may not be the threshold where the models can take over most AI research.

它可能也不是模型严重改变经济运行方式的临界点。

It may not be the threshold where the models seriously change how the economy works.

我认为在此之后情况就变得模糊了，所有这些临界点可能都会在之后的不同时间点发生。

I think it gets a little murky after that, and all of those thresholds may happen at various times after that.

但从基础技术能力来看，它听起来就像一个普遍受过良好教育的人类。

But I think in terms of the base technical capability of it, it kinda it kinda sounds like a reasonably generally educated human across the board.

我认为这个目标可能已经非常接近了。

I think that could be quite close.

为什么它可能通过了受过教育者水平的图灵测试，却仍无法参与或替代人类在经济中的作用呢？

Why would it be the case that it could be sound pass the Turing test for an educated person, but not be able to contribute or substitute for human involvement in the economy?

有几个原因。

A couple reasons.

一个是，你知道，技能门槛还不够高。

One is just, you know, that the threshold of skill isn't high enough.

对吧？

Right?

比较优势。

Comparative advantage.

就像，即使有个人在每项任务上都比普通人强，这也没关系。

It's like, it, like, doesn't matter that I have someone who's better than the average human at every task.

比如在AI研究方面，我需要找到一种足够强大的工具，能显著加速那上千名最顶尖专家的工作。

Like, what I really need is like for AI research, like, you know, I need to basically find something that is strong enough to substantially accelerate the labor of the thousand experts who are best at it.

因此，我们可能会达到一个点，这些系统的比较优势并不明显。

And so we might reach a point where the comparative advantage of these systems is not great.

另一个可能是，我认为存在一些神秘的摩擦因素，这些在朴素的经济模型中看不到，但当你去接触客户时就会发现，比如你跟客户说：嘿，我有个很酷的聊天机器人。

Another thing that could be the case is that I think there are these kind of mysterious frictions that kind of don't show up in naive economic models, but you see it whenever you're like, when you go to a customer or something and you're like, hey, I have this cool chatbot.

原则上，它可以完成你的客服机器人或公司这部分所做的所有工作。

In principle, it can do everything that your customer service bot does, or that this part of your company does.

但问题是，我们该怎么把它整合进去呢？

But like, the actual friction of like, how do we slot it in?

我们该怎么让它正常运行？

How do we make it work?

这包括一方面，它在公司内部如何以人性化的方式运作，以及经济中如何克服这些摩擦。

That includes both kind of like, just the question of how it works in a human sense within the company, like how things happen in the economy and overcome frictions.

另一方面，就是具体的工作流程是什么？

And also just like, what is the workflow?

你实际上该如何与它互动？

How do you actually interact with it?

说一个聊天机器人看起来像是在完成某项任务，或帮助人类完成任务，和说这个系统已经部署、十万人都在使用它，是完全不同的两回事。

It's very different to say, here's a chatbot that kind of looks like it's doing this task that, or helping the human to do some task, as it is to say like, okay, this thing is deployed and 100,000 people are using it.

通常，现在很多人正急于部署这些系统。

Often, like right now, lots of folks are rushing to deploy these systems.

但我认为，在许多情况下，他们并没有以最高效的方式使用这些系统。

But I I think in many cases, they're not using them in anywhere close to the most efficient way that they could.

你知道，这并不是因为他们不够聪明，而是因为要理清这些事情需要时间。

You know, not because they're not smart, but because it takes time to work these things out.

因此，我认为当事物变化如此迅速时，必然会存在各种摩擦。

And so I think when things are changing this fast, there are gonna be all of these frictions.

而且我认为，这些是复杂的现实，无法被模型完全捕捉。

And I think, again, these are messy reality that doesn't quite get captured in the model.

我不认为这改变了基本的图景。

I don't think it changes the basic picture.

我不认为这改变了这样一个观点：我们正在积累一个雪球效应——模型帮助模型变得更好，加速人类能做的事情，最终，主要由模型来完成工作。

I don't think it changes the idea that we're building up this snowball of like, the models help the models get better and do what the humans and can accelerate what the humans do, and eventually, it's mostly the models doing the work.

只要你足够宏观地看，这确实在发生。

Like, you zoom out far enough, that's happening.

但我对任何精确的数学或指数预测持怀疑态度，即它将如何发展。

But I'm kind of skeptical of kind of any kind of precise mathematical or exponential prediction of how it's gonna be.

我觉得一切都会很混乱，但我们知道这并不是一种比喻性的指数增长，而且它会迅速发生。

I think all gonna be a mess, but I think what we know is it's not an metaphorical exponential, and it's gonna happen fast.

我们之前讨论过的那些不同的指数增长，最终会如何相互作用呢？

How do those different exponentials net out, which we've been talking about?

一个是扩展定律本身是幂律，随着参数增加，边际损失逐渐递减，或者类似的情况。

So one was the scaling laws themselves are power laws with decaying marginal loss per parameter or something.

另一个你提到的指数增长是，这些工具本身可以参与到人工智能研究的过程中，从而加速研究。

The other exponential you talked about is, well, these things can get involved in the process of AI research itself, speeding it up.

所以这两种是某种程度上相互对立的指数增长。

So those two are sort of opposing exponentials.

它们最终是呈现超线性增长，还是亚线性增长？

Does it net out to be super linear or sub linear?

你还提到，智能的分布可能只是变得更广泛。

And also you mentioned, well, the distribution of intelligence might just be broader.

所以当我们达到两三年后这个节点时，是不是就会突然之间，飞速发展？

So should we expect after the after we get to this point in two to three years, it's like, voom, voom.

比如，那看起来像什么？

Like, does that look like?

我的意思是，我觉得这非常不清楚。

It's I mean, I think it's very unclear.

对吧？

Right?

所以我们现在已经到了这样一个阶段：如果你看损失值，扩展规律已经开始出现弯曲。

So we're already at the point where if you look at the loss, the scaling laws are starting to bend.

我的意思是，我们已经在多家公司发布的模型卡片中看到了这一点。

I mean, we've seen that in published model cards offered by multiple companies.

所以这根本不是什么秘密。

So that's not a secret at all.

但当它们开始弯曲时，每一丁点的熵——也就是准确预测——就变得更加重要了，对吧？

But as they start to bend, each little bit of entropy, right, of accurate prediction becomes more important, right?

也许这些最后的一点点熵，就像爱因斯坦会写的物理论文那样，而不是其他物理学家会写的那种。

Maybe these last little bits of entropy are like, well, this is a physics paper as Einstein would have written it, as opposed to as some other physicist would have written it.

因此，很难从这一点评估其重要性。

And so it's hard to assess significance from this.

就实际性能而言，这些指标显然仍在相对线性地上升。

It certainly looks like in terms of practical performance, the metrics keep going up relatively linearly.

希望它们原本就是不可预测的。

Hopefully, they were always unpredictable.

所以，这一点很难看出来。

So so it's it's hard to see that.

而我认为推动加速的最主要因素，是越来越多的资金涌入这个领域。

And then, I mean, the thing that I think is driving the most acceleration is just more and more money is going into the field.

人们看到这里蕴含着巨大的经济价值。

Like, people are seeing that there's just a huge amount of, you know, of of economic value.

因此，我预计在最大模型上的投入金额将增加约一百倍。

And so I expect the price, the amount of money spent on the largest models to go up by like a factor of 100 or something.

再加上芯片变得更快、算法不断优化，因为现在有如此多的人在研究这个问题。

And for that then to be concatenated with the chips are getting faster, the algorithms are getting better because there's so many people working on this now.

所以，我在这里并不是在做规范性陈述。

And so again, I'm not making a normative statement here.

这是应该发生的事情。

This is what should happen.

我甚至没有说这一定会发生，因为我认为这里涉及重要的安全和政府问题，我们正在积极研究这些问题。

I'm not even saying this necessarily will happen because I think there's important safety and government questions here, which we're very actively working on.

我只是说，如果放任不管，经济就会朝这个方向发展。

I'm just saying like left to itself, this is what the economy is gonna do.

我们稍后会谈到

We'll get to

这些问题一会儿再说。

those questions in a second.

但你怎么看待Anthropic对这一行业规模扩大的贡献？我的意思是，我们有一个观点认为，有了这些投资，我们可以在Anthropic专注于安全问题；还有另一种观点认为，你提升了整个领域的关注度。

But how do you think about the contribution of Anthropic to that increasing in the scope of this industry where I mean, there's an argument we make that, listen, with that investment, we can work on safety stuff at Anthropic, another that says you're raising the salience of this field in general.

是的。

Yeah.

我的意思是，这全是成本和收益的问题。

I mean, it's all it's all costs and benefits.

对吧？

Right?

成本并不是零。

The costs are not zero.

对吧？

Right?

所以我认为，成熟地看待这些问题的方式是，不要否认存在任何成本，而是要思考这些成本是什么，收益又是什么。

So I think a mature way to think about these things is, you know, not not to deny that there are any costs, but to think about what the costs are and what the benefits are.

我觉得我们在某种程度上是相对负责任的，因为去年底和今年初发生的那场重大加速，并不是我们引起的。

You know, I think I think we've been relatively responsible in the sense that, you know, the big acceleration that that happened late last year and and beginning of this year, like, we didn't cause that.

我们并不是那个促成它的人。

We weren't the ones who did that.

老实说，我认为如果你看看谷歌的反应，那可能比其他任何事情都重要十倍。

And honestly, I think if you look at the reaction of Google, that that might be 10 times more important than anything else.

一旦事情发生，生态系统发生变化后，我们就做了很多事来保持在前沿。

And then kind of once it had happened, once the ecosystem had changed, then we did a lot of things to kind of stay on the frontier.

所以我不知道，这就像其他任何问题，对吧？

So I don't know, it's like any other question, right?

你试图做那些成本最高、成本最低、收益最大的事情。

It's like you're trying to do the things that have the biggest costs and that have the lowest costs and the biggest benefits.

这会导致你在不同时期采取不同的策略。

And that causes you to have different strategies at different times.

我们在讨论智能相关话题时，我有个问题：是的。

One question I had for you while we were talking about the intelligence stuff was Yes.

听我说，作为一名科学家，你怎么看待这些系统几乎记住了整个人类知识库这一事实？

Listen, as a scientist yourself, is it what do you make of the fact that these things have basically the entire corpus of human knowledge memorized?

据我所知，它们还从未做出过任何能导致新发现的全新关联。

And as far as I'm aware, haven't been able to make a single new connection that has led to a discovery.

而如果一个中等智力的人拥有这么多记忆，他早就注意到：这个东西会导致那个症状。

Whereas if even a moderately intelligent person had this much stuff memorized, they'd notice, oh, this thing causes this symptom.

另一件事也会导致这个症状。

This other thing also causes this symptom.

这里就有一个医学疗法。

There's a medical cure right here.

我们难道不该期待这类事情吗？

Shouldn't we be expecting that kind of stuff?

我不确定。

I'm not sure.

我的意思是，我觉得，我不知道，像‘发现’、‘创造力’这样的词，我学到的一个教训是，在庞大的计算洪流中，这些想法往往变得模糊、难以捉摸，难以追踪。

I mean, I think, I don't know, these words discovery, creativity, it's one of the lessons I've learned is that in kind of the big blob of compute, often these these ideas often end up being kind of fuzzy and elusive and hard to track down.

但我认为这里确实有些东西，那就是我认为这些模型确实展现出了一些能力，比如用科马克·麦卡锡或芭比的风格写一首十四行诗。

But I think I think there is something here, which is I think the models do display Again, the kind of like write a sonnet in the style of Cormac McCarthy or Barbie or something.

这确实体现了一定的创造力。

There is some creativity to that.

而且我认为它们确实会建立起普通人也会做出的新关联。

And I think they do draw new connections of the kind that an ordinary person would draw.

我同意你的看法，目前确实还没有出现什么重大的科学发现。

I agree with you that there haven't been any kind of like, I don't know, like, I would say, like big scientific discoveries.

我认为这是因为模型的能力还不够高。

I think that's a mix of like, just the model skill level is not high enough yet.

对吧？

Right?

上周我参加了一个播客，主持人说，我不知道。

Like I was on a podcast last week where where the host said, I don't know.

我经常使用这些模型。

I play with these models.

它们有点平庸。

They're kind of mid.

对吧？

Right?

它们能拿到B或者B-这样的成绩。

Like, they get, you know, they get a b or a b minus or something.

而且我认为，随着规模的扩大，这种情况将会改变。

And and that that, I think, is gonna change with the with the scaling.

我认为有一个有趣的观点，那就是模型拥有的优势是它们知道的东西比我们多得多。

I do think there's an interesting point about, well, the models have an advantage, which is they know a lot more than us.

你知道，即使它们的技能水平还不够高，它们难道不应该已经拥有这种优势吗？

You know, like, should should they have an advantage already even if even if they their skill level isn't isn't isn't quite high?

也许你正是想表达这个意思。

Maybe that's kinda what you're getting at.

我对这个问题其实没有明确的答案。

I don't really have an answer to that.

我的意思是，显然在记忆、事实和建立联系方面，模型已经领先了。

I mean, it seems certainly like memorization and facts and drawing connections is an area where the models are ahead.

我认为，你可能需要这些联系，也需要相当高的技能水平。

And I do think maybe you need those connections and you need a fairly high level of skill.

我认为，特别是在生物学领域，无论好坏，生物学的复杂性使得当前的模型已经掌握了大量知识。

I do think, particularly in the area of biology, for better and for worse, the complexity of biology is such that the current models know a lot of things right now.

而这就是你做出发现和建立联系所需要的。

And that's what that's what you need to make discoveries and draw.

这不像物理学，你得思考并推导出公式。

It's not like physics where you need to, you know, you need to think and come up with a formula.

在生物学中，你需要了解很多知识，没错。

In biology, you need to know a lot of Right.

因此，我认为这些模型掌握了很多知识，但它们的技能水平还不足以将这些知识整合起来。

And so I do think the models know a lot of things and they have a skill level that's not quite high enough to put them together.

我认为它们正处在能够将这些要素整合起来的临界点上。

And I think they are they are just on the cusp of being able to put these things together.

关于这一点，上周你在参议院作证时提到，这些模型距离可能引发大规模生物恐怖袭击还有两到三年的时间。

On that point, last week in your Senate testimony, you said that these models are two to three years away from potentially enabling large scale biotourism attacks or something

比如可以

like Can

你能更具体地说明一下吗？当然，不能提供可能被滥用的信息。

you make that more concrete without obviously giving the kind of information that would help?

但这是像一次性获取如何武器化某种东西的方法吗？

But is it like one shotting how to weaponize something?

你需要微调一个开源模型吗？

Do you to fine tune an open source model?

也就是说，这实际上会是怎样的？

Like, what would that actually

我认为有必要澄清这一点，因为我们曾在参议院听证会上发布过一篇博客文章，我觉得很多人没理解我们的观点，或者没理解我们做了什么。

I look think it'd be good to clarify this because we did a blog post in the Senate testimony and like, I think various people kinda didn't understand the point or didn't didn't understand what we'd done.

所以我认为，今天——当然在我们的模型中，我们努力防止这种情况，但总会有越狱漏洞。

So I think today, and of course in our models we try and prevent this, but there's always jailbreaks.

你可以向模型询问各种关于生物学的问题，让它们说出各种可怕的内容。

You can ask the models all kinds of things about biology and get them to say all kinds of scary things.

是的。

Yeah.

但这些可怕的内容往往是你在谷歌上也能查到的，因此我对此并不特别担心。

But often those scary things are things that you could Google, and I'm therefore not particularly worried about that.

我认为这反而阻碍了人们看清真正的危险——有人只是说，我问了这个模型一些关于天花的问题，它就给出了答案。

I think it's actually an impediment to seeing the real danger, where someone just says, oh, I asked this model to tell me some things about smallpox, and it will.

这其实并不是我真正担心的事情。

That is actually kind of not what I'm worried about.

所以我们花了大约六个月的时间，与世界上最具专业知识的人合作，研究生物攻击是如何发生的。

So we spent about six months working with basically some

其中

of

你知道，要实施这样的攻击，你需要什么，以及我们该如何防范？

the folks who are the most expert in the world on how do biological attacks happen.

他们非常深入地研究了整个流程：如果我要做一件坏事，这并不是一次性的操作，而是一个漫长的过程，包含许多步骤。

You know, what would you need to conduct such an attack, and how do we defend against such an attack?

这不仅仅是‘我问了模型一页信息’那么简单。

They worked very intensively on just the entire workflow of, if I were trying to do a bad thing, it's not one shot, it's a long process, there are many steps to it.

它不是仅仅因为我问了模型这一页信息就完事了。

It's not just like I asked the model for this one page of information.

而且，不深入细节的话，我在参议院证词中提到的是，有些步骤你可以在谷歌上直接找到信息。

And again, without going into any detail, the thing I said in the Senate testimony is like, there are some steps where you can just get information on Google.

有些步骤是我所说的‘缺失’的。

There are some steps that are what I'd call missing.

这些信息分散在众多教科书中，或者根本不在任何教科书里。

They're scattered across a bunch of textbooks, or they're not in any textbook.

它们属于隐性知识，而不是显性知识。

They're kind of implicit knowledge, and they're not really like, they're not explicit knowledge.

它们更像是：我得执行这个实验流程，但如果我做错了怎么办？

They're more like, I have to do this lab protocol, and like, what if I get it wrong?

哦，如果发生了这种情况，我的温度太低了。

Oh, if this happens, my temperature was too low.

如果出现那种情况，我就需要加入更多这种特定的试剂。

If that happened, I needed to add more of this particular reagents.

我们发现，这些关键的缺失环节，目前模型还做不到。

What we found is that for the most part, those key missing pieces, the models can't do them yet.

但我们发现，有时它们确实可以。

But we found that sometimes they can.

当它们能做到时，有时仍然会幻觉，而这正是让我们保持警惕的地方。

And when they can, sometimes they still hallucinate, which is the thing that's kind of keeping us safe.

但我们看到了足够多的迹象，表明这些模型能够很好地完成这些关键任务。

But we saw enough signs of the models doing those key things well.

如果我们观察当前最先进的模型，并回溯到之前的模型，分析这一趋势，可以看出再过两三年，我们将会面临一个真正的问题。

And if we look at state of the art models and go backwards to previous models, we look at the trend, it shows every sign of two or three years from now, we're gonna have a real problem.

是的。

Yeah.

尤其是你提到的那一点，在对数尺度上，从每100次中成功一次，变成每10次中成功一次，确实如此。

Especially the thing you mentioned, though, on the log scale, go from, like, one in 100 times it gets a ride to one in 10 to Exactly.

所以，你知道，我这一生中见过很多这样的‘grocks’。

So, you know, I've seen many of these, like, grocks in my life.

对吧？

Right?

我亲眼见证了GPT-3学会算术、GPT-2勉强超过随机水平地完成回归任务，以及当我们使用Claude时，在这些关于有益、诚实、无害的测试上取得的进步。

I was there when I watched when GPT-three learned to do arithmetic, when GPT-two learned to do regression a little bit above chance, when, you know, when we got, you know, with Claude and we got better on like, you know, all these tests of helpful, honest, harmless.

我见过很多这样的‘Grox’。

I've seen a lot of Grox.

这并不是让我感到兴奋的一种进展，但我相信它正在发生。

This is this is unfortunately not one that I'm excited about, but I believe it's happening.

所以有人可能会说：你不是曾作为合著者参与了OpenAI发布的那篇关于GPT-2的文章吗？当时他们说，因为担心这个模型会被用于不良用途，所以不发布权重或详细信息。

So somebody might say, listen, you were a coauthor on this post that OpenAI released, GPT two, where they said, you know, we're not gonna release the weights or the details here because we're worried that this model will be used for something, you know, bad.

现在回头看，认为GPT-2能造成什么实际危害的想法简直可笑。

And looking back on it, now it's laughable to think that GPT-two could have done anything bad.

我们是不是太过担忧了？

Are we just, like, way too worried?

这种担忧对……来说根本没有道理。

This is a concern that doesn't make sense for

这确实很有趣。

It is interesting.

回头看那篇帖子的实际内容可能很有价值。

It might be worth looking back at the actual text of that post.

我不太记得具体内容了，但它应该还挂在互联网上。

So I don't remember it exactly, but it it should it you know, it's it's it's still up on the Internet.

它提到类似这样的内容：我们选择不发布权重，是因为担心被滥用。

It says something like, you know, we're choosing not to release the weights because of concerns about misuse.

但它也说，这是一次实验。

But it also said, this is an experiment.

我们不确定此时是否有必要或正确这样做，但我们希望确立一种认真思考这些问题的规范。

We're not sure if this is necessary or the right thing to do at this time, but we'd like to establish a norm of thinking carefully about these things.

你可以把它看作是20世纪70年代的阿西洛马会议，当时人们刚刚开始研究重组DNA技术。

You could think of it a little like the Silamur Conference in the 1970s, right, where it's like they were just figuring out recombinant DNA.

当时并不一定意味着重组DNA会被用来做真正有害的事情。

You know, it was not necessarily the case that someone could do something really bad with recombinant DNA.

只是这些可能性开始变得清晰起来。

It's just the possibilities were starting to become clear.

至少，这些话体现了正确的态度。

Those words, at least, were the right attitude.

现在我认为还有一件事，人们不只是评判这篇帖子，他们还在评判整个组织。

Now I think there's a separate thing that, like, you know, people don't just judge the post, they judge the organization.

这是一个制造大量炒作的组织，还是一个有公信力的组织，或者类似的情况？

Is this an organization that produces a lot of hype or that has credibility or something like that?

所以我认为这在一定程度上产生了影响。

And so I think that had some effect on it.

我想你也可以问，人们是否会不可避免地将其解读为：你无法传达任何比‘这东西很危险’更复杂的信息。

I guess you could also ask, is it inevitable that people would just interpret it as like, you can't get across any message more complicated than this thing right here is dangerous.

你可以就这些问题展开讨论，但我想，当时在我和参与此事的其他人脑海中最基本的想法是，正如帖子中所体现的：我们其实并不确定。

So you can argue about those, but I think the the basic thing that was in my head and the head the head of others who were who were who were involved in that, and, you know, I think what what is what is evident in the post is like, we actually don't know.

我们对什么是危险的、什么不是危险的，有着相当宽泛的猜测。

We have pretty wide air of ours on what's dangerous and what's not.

所以我们应该，嗯，希望建立一种谨慎行事的规范。

So we should, you know, like, we we want to establish a norm of being careful.

我想顺便说一下，我们现在拥有的证据要多得多。

I I think, by the way, we have enormously more evidence.

我们现在看到的这类grocs也多得多了。

We've seen enormously more of these grocs now.

所以我们已经很好地调整了判断，但仍然存在不确定性。

And so we're well calibrated, but there's still uncertainty.

对吧？

Right?

在所有这些陈述中，我都说过，比如两三年后，我们可能会达到那个程度。

In all these statements, I've said, like, in two or three years, we might be there.

对吧？

Right?

这存在相当大的风险，我们不想冒这个险。

There's a substantial risk of it, and we don't wanna take that risk.

但你知道，我不会说这是100%的。

But, you know, I wouldn't say it's it's a 100%.

可能是五五开。

It could be fifty fifty.

好的。

Okay.

让我们谈谈网络安全，除了生物恐怖主义之外，这是Anthropic一直强调的另一个方面。

Let's talk about cybersecurity, which in addition to Bioterrorisk is another thing Anthropic has been emphasizing.

你们是如何避免云微架构泄露的？

How have you avoided the cloud microarchitecture from leaking?

因为你知道，你们的竞争对手在這種安全性方面做得不太成功。

Because as you know, your competitors have been less successful at, this kind of security.

我不能评论其他人的安全措施。

Can't comment on anyone else's security.

我不清楚里面发生了什么。

Don't know what's going on in there.

我们做的一件事是，有一些架构上的创新，能让训练更加高效。

A thing that we have done is, so there are these architectural innovations, right, that make training more efficient.

我们称它们为计算倍增器，因为它们的效果相当于提升了计算能力。

We call them compute multipliers because they're the equivalent of improving, they're like having more compute.

我们的计算倍增器，再说一遍，我不想透露太多细节，因为这可能让对手找到对策，所以我们只让真正需要知道的人了解每个计算倍增器的具体内容。

Our compute multipliers, again, I don't want to say too much about it because it could allow an adversary to counteract our measures, but we limit the number of people who are aware of a given compute multiplier to those who need to know about it.

因此，只有极少数人掌握所有这些机密。

And so there's a very small number of people who could leak all of these secrets.

能泄露其中某一项机密的人则相对更多一些。

There's a larger number of people who could leak one of them.

但这种做法正是情报机构或地下抵抗组织常用的隔离策略。

But this is the standard compartmentalization strategy that's used in the intelligence community or resistant cells or whatever.

在过去几个月里，我们已经实施了这些措施。

We've, over the last few months, we've implemented these measures.

所以我不敢妄言说‘这种事情永远不会发生在我们身上’，那样会不吉利。

So I don't wanna jinx anything by saying, oh, this could never happen to us.

但我认为，这种事情发生的可能性会小得多。

But I think I think it would be harder for it to happen.

我不想再深入细节了。

I don't wanna go into any more detail.

顺便说一下，我鼓励其他所有公司也这么做。

And, know, by the way, I'd encourage all the other companies to do this as well.

竞争对手架构泄露对Anthropic的帮助非常有限，对任何人都没有好处，

It's as much as like competitors' architectures leaking is is narrowly helpful to Anthropic, it's not good for anyone in

从长远来看。

the long run.

对吧？

Right?

所以这方面的安全非常重要。

So security around this stuff is really important.

即使你们有所有这些安全措施，以你们目前的安全水平，能否阻止一个专注的国家级行为者获取Claude 208？

Even with all the security you have, could you, with your current security, prevent a dedicated state level actor from getting the Claude two zero eights?

这取决于他们有多专注，我会这么说。

It depends how dedicated is what I would say.

我们的安全主管以前负责Chrome的安全工作，而Chrome被广泛应用于AT&T和各类应用中，他喜欢从成功攻击Anthropic需要多少成本的角度来思考这个问题。

Our head of security, who used to work on security for Chrome, which very widely used in ATT and application, he likes to think about it in terms of how much would it cost to attack Anthropic successfully.

再说一遍，我不想详细说明我认为攻击需要多少成本，那样等于在给他人提供线索。

Again, I don't want to go into super detail of how much I think it will cost to attack, and it's kind of inviting people.

但我们的目标之一是，攻击Anthropic的成本要高于自己训练一个模型的成本，当然这并不能完全保证安全，因为你还需要人才。

But like, one of our goals is that it costs more to attack Anthropic than than it costs to just train your own model, which doesn't guarantee things because, you know, of course, you need the talent as well.

所以你可能还是会去尝试，但你知道，攻击本身是有风险的。

So you might still but, you know, but but attacks have have have risks.

还有外交上的代价。

The diplomatic costs.

而且，你知道，攻击会消耗国家行为体可能拥有的极其有限的资源。

And, you know, And they use up the very sparse resources that nation state actors might have in order to do the attacks.

顺便说一句，我们还没达到完美，但以我们公司的规模来看，我认为已经达到了非常高的安全标准。

So we're not there yet, by the way, but I think we're to a very high standard compared to the size of company that we are.

如果你看看大多数150人规模公司的安全措施，我觉得根本没法相提并论。

Like, I think if you look at security for most 150 person companies, like, I think there's just no comparison.

但如果一个国家行为体将窃取我们的模型权重作为最高优先级，我们能抵抗得住吗？

But could we resist if if it was a state actor's top priority to steal our model weights?

不能。

No.

他们会成功的。

They would they would succeed.

这种情况会持续多久？

How long does that stay true?

因为随着时间推移，模型的价值会不断攀升。

Because at some point, the value keeps increasing and increasing.

这个问题的另一部分是：训练Claude 3或Claude 2的方法究竟算什么类型的秘密？

And another part of this question is that what kind of a secret is how to train cloud three or cloud two?

比如核武器，我们身边到处都是间谍。

Is it with nuclear weapons, for example, we have lots of spies.

你只需把设计图带出去，那就是内爆装置，而这正是你需要的一切。

You just take a blueprint across and that's the implosion device, and that's what you need.

在这里，它是否更像你提到的生物学中的那种隐性知识？

Here, is it more tacit like the thing you were talking about biology?

你需要了解这些试剂是如何工作的。

You need to know how these reagents work.

是仅仅像你拿到了蓝图、微架构和超参数一样吗？

Is it just like you got the blueprint, you got the microarchitecture and the hyperparameters?

那你就可以开始了

There You're good to

有些东西就像一个简单的公式，而其他一些则更复杂。

are some things that are like a one line equation, and there are other things that are more complicated.

我认为最好的方式是进行隔离。

And I think compartmentalization is the best way to do it.

限制知道某件事的人数。

Just limit the number of people who know about something.

如果你是一家千人公司，而每个人都了解每一个秘密，那么我敢保证，你一定有内鬼。

If you're a thousand person company and everyone knows every secret, like, one, I guarantee you have some you have a leaker.

第二，我敢保证你有个间谍，一个真正的间谍。

And two, I guarantee you have a spy, like a literal spy.

好的。

Okay.

我们来谈谈对齐问题。

Let's talk about alignment.

我们来谈谈机制可解释性，这是那个……是的。

And let's talk about mechanistic interpretability, which is the branch Yes.

你们正是专精于这一领域的。

Of which you, you guys specialize in.

在回答这个问题时，你或许需要先解释一下什么是机制可解释性。

While you're answering this question, you might wanna explain what mechanistic interpretability is.

但更广泛的问题是：从机制上讲，对齐到底是什么？

But just, the broader question is, mechanistically, what is alignment?

是不是把模型锁定在一个仁慈的特质上？

Is it that you're locking in the model into a benevolent character?

你们在禁用欺骗性电路和流程吗？

Are you disabling deceptive circuits and procedures?

也就是说，当你对模型进行对齐时，具体发生了什么？

Like, what concretely is happening when you align a model?

我认为，和大多数事情一样，当我们实际训练模型以实现对齐时，并不清楚模型内部发生了什么，对吧？

I think as with most things, you know, when we actually train a model to be aligned, we don't know what happens inside the model, right?

有多种方式训练模型实现对齐，但我认为我们其实并不清楚其中发生了什么。

There are different ways of training it to be aligned, but I think we don't really know what happens.

我的意思是，对于一些当前的方法，我认为所有涉及某种微调的当前方法，都有一个共同特点：我们可能担心的底层知识和能力并不会消失。

I mean, I think for some of the current methods, I think all of the current methods that involve some kind of fine tuning, of course, have the property that the underlying knowledge and abilities that we might be worried about don't disappear.

只是，模型被训练成不去输出这些内容而已。

It's just, you know, the model is just taught not to output them.

我不知道这是否是一个致命缺陷，或者这只是事情必须如此的方式。

I don't know if that's a fatal flaw or if, you know, or if that's just the way things have to be.

我不清楚机制层面内部究竟发生了什么，而我认为这正是机制可解释性的核心目的——真正理解模型内部各个电路层面的运作。

I don't know what's going on inside mechanistically, and I think that's the whole point of mechanistic interpretability, to really understand what's going on inside the models at the level of individual circuits.

最终问题解决了，这个解决方案会是什么样子？

Eventually, it's solved, what does the solution look like?

在什么情况下，如果你是Claude，你做了机制可解释性分析，然后说：我满意了？

Where what is it the case where if you're Claude for, you do the mechanistic interpretability thing and you're like, I'm satisfied.

它已经对齐了。

It's aligned.

你看到了什么？

What is it that you've seen?

是的，我认为我们目前还不知道。

Yeah, so I think we don't know that yet.

我认为我们还不足以了解这一点。

I think we don't know enough to know that yet.

我的意思是，我可以给你描述一下这个过程看起来是什么样，而不是最终结果是什么样。

I mean, I can give you a sketch for what the process looks like as opposed to what the final result looks like.

所以我认为可验证性是这里的主要挑战，对吧？

So I think verifiability is a lot of the challenge here, right?

我们有各种声称能够对齐AI系统的方法，并且在当前任务上确实取得了成功。

We have all these methods that purport to align AI systems and do succeed at doing so for today's tasks.

但问题总是在于，如果你有一个更强大的模型，或者模型处于不同情境下，它还会对齐吗？

But then the question is always, if you had a more powerful model or if you had a model in a different situation, would it be aligned?

因此，我认为，如果你有一个可以扫描模型并直接说‘我知道这个模型是对齐的’的预言机，这个问题就会容易得多。

And so I think this problem would be much easier if you had an Oracle that could just scan a model and say like, okay, I know this model is aligned.

我知道它在每种情况下都会做什么。

I know what it'll do in every situation.

那样的话，问题就会简单得多。

Then the problem would be much easier.

我认为我们目前最接近这一点的是机制可解释性。

And I think the closest thing we have to that is something like mechanistic interpretability.

但它离胜任这项任务还差得很远。

It's not anywhere near up to the task yet.

但我想说，我把它看作是一个扩展的训练集和扩展的测试集。

But I guess I would say, I think of it as almost like an extended training set and an extended test set.

对吧？

Right?

我们所做的一切，所有这些对齐方法，都是训练集。

Everything we're doing, all the alignment methods we're doing are the training set.

对吧？

Right?

你可以在这些方法中进行测试，但它们真的能应对分布变化吗？

You you know, you can you can run tests in them, but will it really work out a distribution?

在其他情况下，它们真的有效吗？

Will it really work in another situation?

机制可解释性是唯一在原则上可能做到这一点的方法，尽管我们离这个目标还很远，但原则上，它更像是对模型的X光检查，而不是对模型的修改，对吧？

Mechanistic interpretability is the only thing that even in principle, and we're nowhere near there yet, but even in principle is the thing where it's like, it's more like an x-ray of the model than modification of the model, right?

它更像是一种评估，而不是干预。

It's more like an assessment than an intervention.

因此，我们需要某种机制，既能拥有一个扩展的训练集——也就是这些对齐方法，又能拥有一个扩展的测试集——就像对模型进行X光扫描，判断哪些有效、哪些无效。

And so somehow we need to get into a dynamic where we have an extended test set, an extended training set, which is all these alignment methods, and an extended test set, which is kind of like you x-ray the model and say like, okay, what works and what didn't?

这种方式超越了你所进行的那些经验性测试，对吧？

In a way that goes beyond just the empirical tests that you've run, right?

你在说，模型在这种情况下会做什么？

Where you're saying, what is the model going to do in these situations?

它是内在能力上能做什么，而不是仅仅从现象上看它做了什么？

What is it within its capabilities to do instead of what did it do phenomenologically?

当然，我们必须对此保持谨慎。

And of course, we have to be careful about that.

对吧？

Right?

我认为非常重要的一点是，我们绝不应该为了可解释性而训练，因为我认为这会削弱这种优势。

One of the things I think is very important is we should never train for interpretability because I think that is that's taking away that advantage.

对吧？

Right?

你甚至会遇到类似验证集与测试集的问题：如果你反复查看X光图像，就可能造成干扰。

You even have the problem similar to validation versus test set, where if you look at the X-ray too many times, you can interfere.

但我认为这是一个更弱的选择，我们确实应该关注它，但这是一个更弱的过程。

But I think that's a much weaker opt we should worry about that, but that's a much weaker process.

这不是自动化优化。

It's not automated optimization.

我们应该像对待验证集和测试集一样，确保在运行测试集之前不要过多地查看验证集。

We should just make sure as with validation and test sets that we don't look at the validation set too many times before running the test set.

但同样，这是人为的压力，而不是自动化压力。

But again, that's manual pressure rather than automated pressure.

因此，我们需要某种解决方案，在训练集和测试集之间建立一种动态关系，即我们尝试各种方法，并通过模型并未针对优化的某种正交方式来真正检验它们是否有效。

And so some solution where it's like, we have some dynamic between the training and test set, where it's like, we're trying things out and we really figure out if they work via way of testing them that the model isn't optimizing against, some orthogonal way.

比如，我认为我们永远无法保证万无一失，但可以建立某种流程，把这些事情结合起来——当然，不能以愚蠢的方式。

Like if I think of, and I think we're never gonna have a guarantee, but some process where we we do those things together, again, not in a stupid way.

有很多愚蠢的做法会让你自己骗自己。

There's lots of stupid ways to do this where you fool yourself.

但我们需要某种方式，将对齐能力的长期训练与对齐能力的长期测试结合起来，真正发挥作用。

But, like, some way to put extended training for alignment ability with extended testing for alignment ability together in a way that actually works.

我还是不明白你的直觉，为什么你觉得这个方法很可能有效，或者值得去探索。

I I still don't feel like I understand the intuition that why you think this is likely to work or this is a promising to pursue.

让我更具体地问一个问题，抱歉这个类比有点牵强。

And let me ask the question in a sort of more specific way and excuse the tortured analogy.

假设你是一位经济学家，想要理解经济运行的规律。

But listen, if you're you're an economist and you wanna understand the economy.

是的。

Yeah.

于是你派一大批微观经济学家去研究，其中一位研究餐饮业如何运作。

So you send a whole bunch of microeconomists out there and one of them studies how the restaurant business works.

另一位研究旅游业如何运作。

One of them studies how the tourism business works.

还有一位研究烘焙行业如何运作。

You know, one of them studies how the baking works.

最后他们聚在一起，你依然无法预测五月会不会出现经济衰退。

And at the end, they all come together, and you still don't know whether there's gonna be a recession in May or not.

为什么这不像那样，我们明明理解了在两层变换器中归纳头是如何工作的？

Why is this not like that where you have an understanding of we understand how induction heads work in a two layer transformer?

我们理解，比如说，模算术。

We understand, you know, modular arithmetic.

这些加起来怎么就能说明这个模型是否想杀死我们？

How does this add up to does this model want to kill us?

换句话说，这个模型从根本上来说想要做什么？

Like, what does this model fundamentally A

关于这一点，有几点要说。

few things on that.

我的意思是，我认为这些问题问得正是时候。

I mean, I think that's like the right set of questions to ask.

我认为我们最终希望的并不是理解每一个细节。

I think what we're hoping for in the end is not that we'll understand every detail.

但再次强调，我会用X光或核磁共振的类比。

But again, I would give the X-ray or the MRI analogy.

我们能否处于这样一种状态：能够观察模型的宏观特征，并判断这个模型的内部状态和意图是否与其外在表现的行为截然不同？

That we can be in a position where we can look at the broad features of the model and say, is this a model whose internal state and plans are very different from what it externally represents itself to do?

对吧？

Right?

这是一个模型吗？它的大部分计算资源都被用于做那些看起来极具破坏性和操纵性的事情，而我们对此感到不安。

Is this a model where we're uncomfortable that far too much of its computational power is devoted to doing what looked like fairly disruptive and manipulative things.

当然，我们还不确定这是否真的可能，但我认为至少有一些积极的迹象表明这是有可能的。

Again, we don't know for sure whether that's possible, but I think some at least positive signs that it might be possible.

再说一遍，模型并不是有意在对你隐瞒，对吧？

Again, the model is not intentionally hiding from you, right?

最终可能是训练过程本身将这些信息隐藏了起来。

It might turn out that the training process hides it from you.

我可以想到一些模型极其聪明的例子。

And I can think of cases where the model's really super intelligent.

它会以某种方式思考，以至于这种思考方式会直接影响它自身的认知。

It, like, thinks in a way so that it, like, affects its own cognition.

我怀疑我们应该思考这个问题。

I suspect we should think about that.

我们应该考虑所有可能性。

We should consider everything.

我怀疑，我们可以大致这样理解：模型是通过正常方式训练的，仅达到略高于人类的水平，这可能是一个我们应当检查的理由，也可能是合理的假设——模型的内部结构并非有意针对我们进行优化。

I I I suspect that it may roughly work to think of the model as if it's trained in the normal way, just getting to just above human level, it may be a reason we should check, it may be a reasonable assumption that the internal structure of the model is not intentionally optimizing against us.

我举一个类似人类的类比。

And I give an analogy like to humans.

实际上，通过查看某人的核磁共振成像，你就能比随机猜测更准确地预测他们是否是心理变态者。

So it's actually possible to, you know, to look at an MRI of someone and predict above random chance whether they were a psychopath.

几年前确实有一个关于一位研究此领域的神经科学家的故事。

There was actually a story a few years back about a neuroscientist who was studying this.

他查看了自己的扫描结果，发现自己是个心理变态者。

And he looked at his own scan and discovered that he was a psychopath.

然后他身边的所有人都说：不，不，不。

And then everyone everyone in his life was like, no, no, no.

这很明显。

This is obvious.

你简直就是个混蛋。

Like, you're you're a complete asshole.

你肯定是心理变态。

Like, you must be a psychopath.

而他对此完全毫无察觉。

And and he was total totally unaware of this.

基本观点是，可能存在一些宏观特征，心理变态或许是一个很好的类比。

The basic idea that there can be these macro features that like psychopath is probably a good analogy for it.

他们就像是：我们所担心的那种模型，表面上很迷人，目标明确，但内心却非常黑暗。

They're like, this is what we'd be afraid of, model that's kind of like charming on the surface, very goal oriented, and very dark on the inside.

表面上，他们的行为可能看起来和别人一样，但他们的目标却截然不同。

On the surface, their behavior might look like the behavior of someone else, but their goals are very different.

有人可能会问：你之前提到实证的重要性，这是为什么？

A question somebody might have is, listen, you mentioned earlier the importance of being empirical.

在这种情况下，你是在试图估算它。

And in this case, you're trying to estimate it.

听好了，这些激活信号可疑吗？

Listen, are these activations sus?

但这件事我们是否能以实证方式处理，还是我们需要一个非常扎实的、基于第一性原理的理论依据，来证明这不仅仅是因为这些模型的MRI图像与‘坏’行为相关？

But is this something we can afford to be empirical about a or do we need a very good first principle theoretical reason to think, no, it's not just that these MRIs of the model correlate with being bad.

我们需要一些深层的数学证明，说明确实如此。

We need just some deep root math proof that this So is

这取决于你所说的‘实证’是什么意思。

it depends what you mean by empirical.

更好的说法应该是‘现象学的’。

I mean, better term would be phenomenological.

我不认为我们应该纯粹停留在现象学层面，比如，这里有一些非常危险模型的大脑扫描图，那里也有一些大脑扫描图。

I don't think we should be purely phenomenological in like, you know, here are some brain scans of like really dangerous models and here are some brain scans.

我认为机制可解释性的整个理念，是去探究底层的原理和电路。

I think the whole idea of mechanistic interpretability is to look at the underlying principles and circuits.

但我想我是这样看待这个问题的：一方面，我一直以来都支持以我们所能达到的最细致层次来研究这些电路。

But I guess the way I think about it is like, on one hand, I've actually always been a fan of studying these circuits at the lowest level of detail that we possibly can.

这样做的原因是，这正是积累知识的方式。

And the reason for that is kind of that's how you build up knowledge.

即使你最终的目标是这些特征太多、太复杂，归根结底，我们试图构建的是一个更宏观的理解。

Even if you're ultimately aiming for there's too many of these features, it's too complicated, at the end of the day, we're trying to build something broad, and we're trying to build some broad understanding.

我认为，构建这种理解的方式是通过做出大量这样的具体发现。

I think the way you build that up is by trying to make a lot of these very specific discoveries.

你必须先理解这些基础构件，然后才能利用它们来得出宏观的结论，即使你无法弄清楚所有细节。

Like you have to understand the building blocks, and then you have to figure out how to kind of use that to draw these broad conclusions, even if you're not gonna figure out everything.

我觉得你最好去问问克里斯·奥拉，他对此会有更详细的见解。

I think you should probably talk to Chris Ola, who would have much more detail.

这只是我对这个问题的宏观思考。

This is my kind of high level thinking on it.

克里斯·奥拉掌控着可解释性研究的方向。

Like, Chris Ola controls the interpretability agenda.

你知道，他才是决定解释性工作方向的人。

Like, you know, he's he's the one who decides what to what to do on interpretability.

这是我对此的高层次思考，但肯定不如他的深入。

This is my high level thinking about it, which is not gonna be as good as his.

Anthropic 的乐观前景是否依赖于机制可解释性对能力提升有帮助？

Does the bull case on Anthropic rely on the fact that mechanistic interpretability is helpful for capabilities?

我完全不这么认为。

I don't think so at all.

不过，从原则上讲，机制可解释性确实有可能对能力提升有帮助。

Now I do think, think in principle, it's possible that mechanistic interpretability could be helpful with capabilities.

如果真是这样，我们可能因为各种原因选择不公开讨论这一点。

We might, for various reasons, not choose to talk about it if that were the case.

在 Anthropic 创立之初，我或者我们任何人并没有这样想过。

That wasn't something that I thought of, or that any of us thought of at the time of Anthropic's founding.

我们当时觉得自己就是擅长扩大模型规模、并在这些模型之上做安全工作的人。

Mean, we thought of ourselves as like, we're people who are like good at scaling models and good at doing safety on top of those models.

而且你知道，我们认为我们团队中擅长这方面的人才密度非常高。

And like, you know, we think that we have a very high talent density of folks who are good at that.

我的观点一直认为，人才密度胜过人才数量。

And you know, my view has always been talent density beats talent mass.

所以这更像是我们的看涨理由。

And so that's more of our bull case.

人才密度胜过人才数量。

Talent density beats talent mass.

我不认为这取决于某个特定的因素。

I don't think it depends on some particular thing.

现在其他人也开始做机制可解释性了，我非常高兴看到这一点。

Like others are starting to do mechanistic interpretability now, and I'm very glad that they are.

我们的变革理论之一，讽刺的是，就是要让其他组织更像我们。

That a part our theory of change is paradoxically to make other organizations more like us.

人才密度当然很重要。

Talent density, I'm sure, is important.

但Anthropic强调的另一点是，要进行安全研究，你必须拥有前沿模型。

But another thing Anthropic has emphasized is that you need to have frontier models in order to do safety research.

而且，当然，你还得真是一家公司。

And of course, like actually be a company as well.

目前的前沿模型，有人可能会猜，比如GPT-4的开销高达一亿美元左右。

The current frontier models, something somebody might guess, like GPT-four o'clock to like $100,000,000 or something

就是这样。

like that.

从非常宽泛的角度来看，这个数量级大致没错。

That general order of magnitude in very broad terms is not wrong.

但你知道，我们距离你所谈论的这些技术，还有两到三年的时间。

But, you know, we're two to three years from now, the kinds of things you're talking about.

为了跟上这种发展，我们需要越来越多的数量级提升，而且如果情况是这样——比如说，你对站在前沿感到羞愧的话。

We're talking more and more orders of magnitude to keep up with that and to if it's the case, let's say, you're embarrassed to be on the frontier.

我的意思是，在什么情况下，Anthropic会去和这些巨无霸竞争，以维持同样的规模？

I mean, what is the case in which Anthropic is, like, competing with these Leviathans to stay on that same scale?

我的意思是，我认为这是一个充满权衡的状况。

I mean, I think it's a I think it's a very it's a situation with a lot of trade offs.

对吧？

Right?

我觉得这并不容易。

I think it's I think it's not easy.

我想回到之前的话题，也许我可以逐个回答这些问题。

I guess to go back, maybe I'll just like answer the questions one by one.

对吧？

Right?

那么回到为什么安全如此依赖规模这个问题上。

So like to go back to like, you know, why is safety so tied to scale?

对吧？

Right?

有些人并不这么认为，但你看一看，哪些领域已经实际应用了安全方法，哪怕只是在某些情况下有效，即使我们不认为它们普遍适用。

Some people don't think it is, but like, just look at like, you know, where have been the areas that, you know, I don't know, like safety methods have like been put into practice or worked for something, for anything, even if we don't think they'll work in general.

我会回想起所有那些想法，比如辩论和放大。

I go back to thinking of all the ideas, something like debate and amplification.

2018年我们在OpenAI撰写相关论文时，觉得人类反馈可能行不通，但辩论和放大能让我们超越它。

Back in 2018, when we wrote papers about those at OpenAI, was like, well, human feedback isn't quite gonna work, but debate and amplification will take us beyond that.

但如果你实际去观察，我们确实尝试过辩论，却发现我们严重受限于模型的质量——要让两个模型进行足够连贯的辩论，以便人类能够评判，从而让训练过程真正起作用，你需要的模型水平至少要达到甚至超越当前前沿。

But then if you actually look at, and we've done attempts to do debates, we're really limited by the quality of the model, where it's like for two models to have a debate that is coherent enough that a human can judge it so that the training process can actually work, you need models that are at or maybe even beyond on some topics the current frontier.

你可以不处于前沿，就能想出一种方法或一个点子。

Now you can come up with a method, you can come up with the idea without being on the frontier.

但对我来说，这仅仅是需要完成的工作中非常小的一部分。

But for me, that's a very small fraction of what needs to be done.

想出这些方法非常容易。

It's very easy to come up with these methods.

想出类似‘问题是X，也许解决方案是Y’这样的想法非常容易。

It's very easy to come up with like, oh, the problem is X, maybe a solution is Y.

但我真正想知道的是，这些方法在实践中是否有效，哪怕只是针对我们今天已有的系统。

But I really wanna know whether things work in practice, even for the systems we have today.

我想知道这些方法在实践中会出现哪些问题。

And I wanna know what kinds of things go wrong with them.

我只是觉得，通过实际尝试，你会发现十种新想法和十种新的出错方式。

I just feel like you discover 10 new ideas and 10 new ways that things can go wrong by trying these in practice.

这种经验性学习，我认为还没有得到应有的广泛理解。

And that empirical learning, think it's just not as widely understood as it should be.

我对类似宪法AI的方法也有同样的看法。

I would say the same thing about methods like constitutional AI.

有些人说，这无关紧要。

And some people say, oh, it doesn't matter.

比如，我们知道这种方法行不通。

Like, we know this method doesn't work.

它无法实现纯粹的对齐。

It won't work for pure alignment.

我对这一点既不赞同也不反对。

I neither agree nor disagree with that.

我觉得这有点过于自信了。

I think that's just kinda overconfident.

我们发现新事物、理解哪些方法有效、哪些无效的方式，是通过不断尝试和探索。

The way we discover new things and understand the structure of what's gonna work and what's not is by playing around with things.

这并不意味着我们应该盲目地认为，这里有效的方法在那里也一定有效。

Not that we should just kind of blindly say, oh, this worked here and so it'll work there.

但你真的会开始理解其中的模式，尤其是规模定律。

But you really start to understand the patterns, with the scaling laws.

甚至机制可解释性，这可能是我看过的、在没有前沿模型的情况下取得大量进展的领域。

Even mechanistic interpretability, which might be the one area I see where a lot of progress has been made without the frontier models.

我们看到OpenAI几个月前发布的工作中，使用非常强大的模型来帮助自动解释较弱的模型。

We're seeing in the work that say OpenAI put out a couple months ago, that using very powerful models to help you auto interpret the weak models.

这当然不是可解释性研究的全部，但却是其中一个重要组成部分。

Again, that's not everything you can do in interpretability, but that's a big component of it.

我们自己也发现它很有用。

And we found it useful too.

因此，你一次又一次地看到这种现象：扩展与安全就像两条相互缠绕的蛇，比你想象的还要紧密，对吧？

And so you see this phenomenon over and over again, where it's like, you know, the scaling and the safety are these two snakes that are like coiled with each other, always even more than you think, right?

就可解释性而言，我想三年前我并没觉得这一点在可解释性上也会如此成立，但不知为何它偏偏就是成立的。

You know, with interpretability, like I think three years ago, I didn't think that this would be as true of interpretability, But somehow it manages to be true.

为什么？

Why?

因为智能是有用的。

Because intelligence is useful.

它在许多任务中都很有用。

It's useful for a number of tasks.

它其中一个有用之处在于，能够判断和评估其他智能体，也许将来甚至能用于对齐研究本身。

One of the tasks it's useful for is figuring out how to judge and evaluate other intelligence, and maybe someday even for doing the alignment research itself.

鉴于这一点是真实的，

Given how that's true,

那么这对Anthropic意味着什么？当两到三年后，这些庞然大物进行价值高达一百亿美元的训练时？

what what does that imply for Anthropic when in two to three years, these Leviathans are doing, like, $10,000,000,000 training runs?

第一个选择是，如果我们无法做到，或者保持在前沿成本太高，那么我们就不要这么做。

Choice one is if it if we can't or if it costs too much to stay on the frontier, then we shouldn't do it.

我们不会使用最先进的模型。

And we won't work with the most advanced models.

我们会看看在不那么先进的模型上能获得什么成果。

We'll see what we can get with models that are not quite as advanced.

我认为那里还是能获得一些价值的，不是零价值，但我有点怀疑这种价值是否真的很高，或者学习速度是否足够快，足以支持这项任务。

I think you can get some value there, like non zero value, but I'm kind of skeptical that the value is all that high or the learning can be fast enough to really be in favor of the task.

第二个选择是，你只是找到一种方法。

The second option is you just find a way.

你只是接受这些权衡。

You just accept the trade offs.

我认为这些权衡比表面看起来更积极，因为有一种我称之为‘向上竞争’的现象。

And I think the trade offs are more positive than they appear because of a phenomenon that I've called race to the top.

我稍后可以详细讲讲，但目前先搁置一下。

I could go into that later, but I'll just let me put that aside for now.

然后我认为第三个现象是，当事物发展到那个规模时，可能会伴随着出现相当严重的危险的概率开始变得不可忽视。

And then I think the third phenomenon is, you know, as things get to that scale, I think this may coincide with starting to get into some nontrivial probability of very serious danger.

同样，我认为首先出现的问题会是滥用，比如我之前提到的生物技术方面的风险，但我觉得我们目前还没有达到那种自主性水平，不需要担心两年内就会出现对齐问题，但这个问题可能紧随其后。

Again, I think it's gonna come first from misuse, the kind of bio stuff that I talked about, but I don't think we have the level of autonomy yet to worry about some of the alignment stuff happening in like two years, but it might not be very far behind that at all.

这可能会导致单边、多边或政府强制的决策，即我们支持的、不以最大速度推进的决定。

You know, that that may that may lead to unilateral or multilateral or government enforced, which we support, decisions, not to scale as fast as we could.

这最终可能是正确的做法。

That may end up being the right thing to do.

所以，说实话，我其实希望事情朝这个方向发展。

So I I you know, actually, that's kind of like, I kind of hope things go in that direction.

这样一来，我们就不用在‘我们不在前沿，无法像希望的那样开展研究或影响其他机构’和‘我们身处前沿，必须接受那些总体积极但利弊交织的权衡’之间做出艰难抉择。

And then we don't have this hard trade off between we're not in the frontier and we can't quite do the research as well as we want or influence other orgs as well as we want, or versus we're kind of on the frontier and have to accept the trade offs, which are net positive, but have a lot in both directions.

好的。

Okay.

关于滥用和对齐问题，正如你所说，两者都是问题。

On the misuse versus misalignment, those are both problems, as you mentioned.

但从长远来看，你更担心哪一个？

But in the long scheme of things, what are you more concerned about?

比如三十年后，你认为哪个问题会更严重？

Like thirty years down the line, which do you think will be considered a bigger problem?

我觉得用不了三十年，但我对两者都感到担忧。

I think it's much less than thirty years, but I'm worried about both.

我不知道。

I don't know.

如果你有一个理论上能够独自掌控世界的模型，而你能控制这个模型，那么很显然，如果这个模型只服从一小部分人的意愿而忽视其他人，这些人就能利用它为自己夺取世界控制权。

If you have a model that could, in theory, take over the world on its own, If you were able to control that model, then it follows pretty simply that if a model was following the wishes of some small subset of people and not others, then those people could use it to take over the world on their behalf.

错位的预设本身就意味着，我们同样应该对滥用保持高度警惕，因为其后果可能相当严重。

The very premise of misalignment means that we should be worried about misuse as well with similar levels of consequences.

但一些比你更悲观的人可能会说，滥用其实已经是比较乐观的情形了，因为你至少已经知道如何让模型听命于坏人。

But but some people who might be more doomer y than you would say misuse is you're already working towards the optimistic scenario there because you've at least figured out how to align the model with the bad guys.

现在你只需要确保它听命于好人就行了。

Now you just need to make sure that it's aligned with the good guys instead.

你为什么认为你能达到让AI与坏人对齐的阶段？你难道没看到已经发生了什么吗？

Why do you think that you could get to the point where it's aligned with the bad you know, you haven't already saw the

我想，如果你认为对齐问题是完全无解的，那你可能会说，好吧，反正我们完蛋了，所以也不用担心滥用问题。

I I guess if you had the view that, like, alignment is completely unsolvable, then, you know, then you'd be like, well, I don't you know, we're dead anyway, so I don't wanna worry about misuse.

这完全不是我的立场。

That's not my position at all.

但你也应该想想，什么样的计划才真正能成功，从而让事情变得更好？

But but also, like, you should think in terms of, like, what's a plan that would actually succeed that would make things good?

任何真正能成功的计划，无论对齐问题有多难解决，都必须同时解决滥用问题和对齐问题。

Any plan that actually succeeds, regardless of how hard misalignment is to solve, any problem any plan that actually succeeds is gonna need to solve misuse as well as misalignment.

随着AI模型变得越来越强大、越来越快，它们将引发国家间力量平衡的重大问题。

It's getting to solve the fact that like, as the AI models get better, faster and faster, they're gonna create a big problem around the balance of power between countries.

它们还将引发一个重大问题：单个个体是否可能做出一些坏事，而其他人却难以阻止？

They're gonna create a big problem around, is it possible for a single individual to do something bad that it's hard for everyone else to stop?

任何真正能导向美好未来的实际解决方案，都必须解决这些问题。

Any actual solution that needs to leads to a good future needs to solve those problems as well.