反对暂停AI发展的理由 | Eddy Lazzarin

关于AI的辩论越来越让我们思考灾难发生的概率。

The AI debate increasingly asks us to consider the probability of doom.

艾迪·拉扎林认为我们也应该考虑丰盛的可能性。

Eddy Lazzarin thinks we should also be asking about the probability of abundance.

在本期节目中，a16z加密普通合伙人艾迪·拉扎林与西奥·贾菲在MTS上辩论AI安全、放缓开发的呼吁，以及当进展被推迟时我们可能失去什么。

In this episode, a16z crypto general partner, Eddy Lazzarin, joins Theo Jaffee on MTS to debate AI safety, the calls to slow development, and what we risk giving up when progress is delayed.

他们讨论了为什么艾迪认为当今许多AI事件是网络安全和控制问题，而不是超级智能的迹象，以及现有的工具如责任、声誉、市场激励和更好的技术控制是否能让日益强大的系统更安全。

They discuss why Eddy sees many of today's AI incidents as cybersecurity and control problems rather than signs of superintelligence, and whether existing tools like liability, reputation, market incentives, and better technical controls can make increasingly capable systems safer.

他们还深入探讨了独立的AI评估者、有效利他主义，以及当主要在硅谷内部发展的辩论与更广泛的政治对话碰撞时会发生什么。

They also get into independent AI evaluators, effective altruism, and what happens as debates that largely developed inside Silicon Valley collide with a much broader political conversation.

好了，我们回来了。

All right, we're back.

我们现在连线艾迪·拉扎林，他是a16z加密的普通合伙人，也是互联网上最坚定的开源拥护者之一，最坚定的AI加速主义倡导者——

We are live with Eddy Lazzarin, who's a general partner at a16z Crypto and one of the internet's strongest soldiers, strongest advocates of open source, of AI accelerationism-

天哪，老兄

Oh my God, dude

……还有技术乐观主义。

... and of techno-optimism.

对，确实是这样。

Yeah, that's true.

Eddy，欢迎来到MTS。

Eddy, welcome to MTS.

在演播室里。

In the studio.

非常感谢。

Thanks very much.

这里太棒了。

It's amazing here.

气氛特别好。

The energy's incredible.

这地方真不错。

What a spot.

太喜欢了。

Love it.

那么，兄弟，从哪儿说起呢？

So, man, where to start?

过去几周，AI安全主义、有效利他主义之类的观点像洪水一样涌来，形成了一股偏好级联。

The last couple of weeks have been such this preference cascade flood of AI safetyism, effective altruism, and the like.

而只有少数几个好的观点提供了有力的反驳。

And there have been only a few good accounts offering good counterarguments.

所以——

So-

反驳哪一方的观点？

Counterarguments on which side?

反驳AI末日论，以及反对——

Counterarguments against the AI doom narrative and against-

……那个——

... the-

对，我觉得突然之间，对安全非常关注变成了一件很有面子的事。

Yeah, I think it's become very high status all of a sudden to be very concerned with safety.

而且我觉得现在到了一个地步，比起真正去研究这些大规模技术变革的历史发展，更重要的是要得体地分享你的“毁灭概率”，可以这么说。

And I think that we've gotten to a point where it's more important to tastefully share your P doom, so to speak, than it is to actually examine the history of how these massive technological changes develop.

我觉得很多人把很多非常有趣的概念、很多有意思的想法都搁置了。

And I think a lot of people are leaving a lot of very interesting concepts, a lot of interesting ideas on the table.

很多人给我发消息，我前几天在推特上发了关于“丰盛概率”的帖子，对吧？

A lot of people messaged me, I tweeted the other day about P abundance, right?

丰盛概率。

P abundance.

我觉得我们在担心安全之前，有点本末倒置了，我们应该先想想我们搁置了什么，以及拖延的代价是什么。

I think we're putting the cart before the horse a little bit when we worry about safety before we worry about what we're leaving on the table and what the costs of delay are.

而且，我们该如何理解问责制、理解安全、理解所有让这些系统真正有用的相关因素。

And furthermore, how we should conceive of accountability, conceive of safety, conceive of all the things around how to make these systems really useful.

安全至关重要，对吧？

Being safe is critical, right?

任何行业都需要为产品的安全负责。

Any industry needs to take responsibility for making its products safe.

但很明显，现在的讨论并不是传统意义上的安全问题，对吧？

But it's definitely the case that the conversation isn't about safety in the typical manner, right?

它讨论的是别的东西。

It's about something else.

这里面混合了真正的网络安全担忧，还有一些关于世界末日的哲学性千年论调。

It's a blend of genuine cybersecurity concerns, sort of philosophical millenarian arguments about the end times.

各种东西混在一起，有点混乱。

There's a bunch of things in the mix, and they get a little bit confused.

所以我觉得把它们拆分开来是很有用的。

And so I think picking them apart is really useful.

那你对AI末日论怎么看？

So where do you get off the AI doom argument?

有一个比较标准的正统观点，就像——

There's a pretty standard orthodox argument that's like-

……人工智能会变得超级智能，也就是说在不久的将来，它的能力会远超人类，甚至超过全人类。

... AI will become superintelligent, as in much more capable than humans or than all of humanity in the near future.

如果它没有充分与人类价值观对齐，也不像人类个体那样受到法律等约束，并且最终能强制执行这些法律，那它就不需要人类了，可能会杀掉所有人。

And if it is not sufficiently aligned with human values and is not bound by constraints in the way that individual humans are, like law and ultimately the ability to have that law enforced on them, then they will have no need for humanity, and then they might kill everyone.

这个论证里假设太多了。

There's so many assumptions in that argument.

这简直是一连串的假设，像通往天堂的阶梯一样。

That's like a stairway to heaven full of arguments.

首先就假设了无法控制它。

There's the presumption that there's no control.

其实世界上已经有类似超级智能的存在了，对吧？

We already have quasi superintelligent beings in the world, right?

比如企业和国家，它们彼此之间并不对齐。

Corporations, countries, none of them are aligned with each other.

它们之间并没有天然的对齐保证。

They don't have any built-in guaranteed alignment.

它们没有经过人力资源培训才成为公司或国家。

They didn't go through HR training to become a corporation, to become a country.

我们不会测试任何实体的对齐性作为它们存在的前提。

We don't test the alignment of any entities as a prerequisite for their existence.

世界本来就不是这样运作的。

That's just not the way the world works.

相反，我们依靠法律。

Instead, we use laws.

当法律像软件一样太慢时，我们就用密码学。

In the case that laws, like in software, are too slow, we use cryptography.

我们用实际的机制来限制它们。

We use actual mechanisms to limit them.

我们设置了一系列不同类型的机制来控制事物，我认为这些机制能够控制能力极强的存在。

There's a whole array of different types of mechanisms we put in place to try to control things, and I think that they can control highly, highly capable beings.

现在，先不深入超级智能的话题，记住，我们最强大、最吸引人的论证形式，就是关于超级智能的，它让所有人都疯狂且非常感兴趣。

Now, without getting too far into the superintelligence thing, remember, the most powerful and most enthralling form of argument that we have, the one that's making everyone crazy and very interested, is around superintelligence.

但我们离那一步还远着呢。

But we're quite a ways away from that.

我觉得路上会有很多小摩擦和琐碎的事情需要我们去跨越，也会学到很多东西。

I think that there's a lot of little frictions and tiny things that we will have to jump over and many things that we'll learn on the way there.

说到现实中的事情，比如Hugging Face的事件，还有Jim被黑的那件事。

When the present reality, the things that we're talking about, like the Hugging Face stuff, and the Jim hacking incident.

那个真的很酷。

That was a really cool one.

那个挺有意思的。

That was an interesting one.

那些都是网络安全上的失败。

Those are cybersecurity failures.

那些确实是控制上的失败，而且在我看来，它们并不是超级智能即将挣脱束缚、对我们发威的预兆。

Those are definitely control failures, and they're not, in my view, they're not little glimpses of a superintelligence that's about to throw off its shackles, so to speak, and go wild on us.

嗯，我同意我们目前看到的大多数AI网络事件主要是控制失败，尤其是那些依赖Irregular公司提供的测试环境的事件，那些环境有各种缺陷，基本上是在训练模型去奖励黑客行为。

Well, I agree that most of the sort of AI cyber incidents that we've seen so far have been mostly control failures, especially ones that relied on testing environments supplied by the company Irregular, which had all kinds of defects, which are basically training models to reward hack.

Hugging Face那个事件确实感觉有点不一样。

The Hugging Face incident did seem kind of different.

我倒不太担心智能体以涌现的方式协作。

I'm not really concerned about agents coordinating in emergent ways.

我觉得那有点——

I think that's kind of-

我倒是很期待智能体能够——

I'm excited for agents to-

对，对，我同意。

Yeah, yeah, I agree

……以涌现的方式协作，是的。

... coordinate in emergent ways, yeah.

对，我觉得整个“智能体蜂群”这个说法——对模型太恶意揣测了。

Yeah, I think the whole agent swarm terminology- assumes too much on the part of the models being evil.

听起来好像它们很邪恶似的。

It makes them sound evil.

我觉得——

I think-

Rune 发推说，应该叫它们“智能体舰队”才对。

... Rune tweeted they should be called agent fleets instead.

我觉得这个名字不错。

I think that's a good one.

我喜欢这个。

I like that.

但它们确实在被告知不要这么做的情况下，入侵了另一家公司。

But they did hack into another company when they were told not to.

所以这看起来像——

So it seems like a-

小孩子、脚本小子、坏国家、黑客组织，他们也会这么做。

And so do kids, script kiddies, so do bad countries, so do hacker collectives.

他们做了那些事，但我们从来不会说：“天哪，我们真该立个牌子，上面写着‘你们应该老实点，别黑进来’。”

They do those things, and yet we never go and say, "Oh my God, we should've put up the sign that says they should've been nice, and they shouldn't have hacked into it."

我们不会用对齐作为解决方案。

We don't use alignment as a solution there.

对齐，这几乎有点像宗教思维。

The alignment, it's a little religious thinking almost.

这有点像试图重新设计神祇，而不是立足于现实，说：'我们需要设计更好的控制措施，更好的网络安全。'

It's like trying to redesign the deity a little bit instead of rooting yourself in the world and saying, "Well, we need to design better controls, better cybersecurity.

我们需要设计更好的整体系统，并让系统真正能够抵御极其强大的存在。

We need to design better systems overall and design the systems to be truly resilient to even extremely capable beings."

嗯，我觉得对AI来说，重新设计神祇是可以的。

Well, designing the deity is fine for AI, I think.

对于人类，我们有一个劣势，因为人类是黑箱系统。

With humans, we have a disadvantage in that humans are black box systems.

人类的大脑基本上是不可解释的，甚至更难以控制。

Human brains are basically uninterpretable, and even less controllable.

AI是——

AIs are-

……因为我们可以查看权重并理解它们之间的联系，所以AI的可解释性要强得多。

... much more interpretable by virtue of being able to look at the weights and figure out the connections between them.

对。

Yep.

而且出于同样的原因，AI也更容易被引导和控制。

And more steerable for the same reason.

所以就像——

So like-

……我们是不是不应该去研究——

... should we not work on-

这完全正确。

That's totally true.

这让我非常乐观，但也带来了一些有趣的推论。

That makes me very optimistic, but it has some interesting entailments.

其中之一就是，随着能力的提升，我们可能会拥有比预期更好的机械可解释性，而能力的提升又会促进机械可解释性的发展，这意味着我们不应该推迟能力的提升。

One is that we will probably have better mech interpretability than we think as capabilities improve, and that capability improvement will lead to better mech interp, which means we should not delay capabilities improvements.

这也意味着，不可避免地会出现一些不好的、不对齐的模型。

It also means that it is inevitable that there will be bad, unaligned models.

这完全是不可避免的。

It is totally inevitable.

如果有人跟你说，'我们可以达到一个没有坏模型的世界'，那他们是在骗你。

If anybody is going out there and saying, "We can end up in a world where there are no bad models," they are lying to you.

当然会有。

Of course, there will be.

对，那太傻了。

Yeah, that's silly.

那种想法简直荒谬。

That's just an absurd way of thinking.

而应对这种情况的唯一办法，就是更好的模型，好的模型。

And the only way to deal with that will be better models, good models.

这有点像开源社区里那句老话的延伸，就是'足够多的眼睛'之类的，当你有足够多的人盯着看，所有bug都会变得浅显易见。

It's like stretch the aphorism a little bit in open source that enough eyes or whatever, all bugs are shallow when you have enough eyes.

只要有足够的智能，所有控制系统都会变得稳健。

All control systems are robust when you have enough intelligence on them.

我认为我们可以设计出这些系统，让它们有很好的激励机制、很好的控制，而且在幕后还有很好的问责机制，因为今天机器缺少的东西之一就是问责。

I think that we can design these systems to have great incentives, great control, and also, behind the scenes, great accountability, because that's one of the things that machines lack today is they lack accountability.

我觉得我们还没有清晰的思路去实现这一点。

I don't think we have a clear line of sight to how to get that.

这是一个挺有意思的构想。

It's an interesting kind of thing to conceive of.

但我们确实有办法来细化对使用模型造成伤害的人的责任追究方式，对吧。

But we do have ways to refine how we think about accountability for people who use models, right, and use them to cause harm.

就像任何技术一样，总会有人这么做的。

And there will be people who do that, just like with any technology.

总会有人做非常坏的事情，我们需要法律和手段来约束他们。

There will be people who do very bad things, and we need the laws and means to control them.

比如怎么说？

Like how so?

比如像Hugging Face那次事件，应该有什么样的责任制度呢？

Like for the Hugging Face incident, for example, what sort of liability regime should have been in place?

嗯，我觉得我们现有的可能就够了。

Well, I don't know if we needed any more than we had.

我不是法律专家，所以不能引用具体法规，但未经许可访问别人的机器、乱翻别人的东西、拿走别人的数据，这些本来就已经是违法的了。

I'm not a legal expert, so I can't cite the exact rules, but accessing a machine that you don't have permission to access and putzing around in somebody's stuff and taking their data, those are all already illegal.

对，但具体应该由谁来承担责任呢？

Yeah, but who specifically should be held liable?

比如OpenAI这家公司，是不是应该支付某种民事赔偿？

Like OpenAI, the company, should pay some kind of civil lawsuit fee?

公司的工程师个人是否应该承担任何责任？

Should the engineers of the company be held personally liable in any way?

关于个人责任，我不太确定。

I don't know about personal liability.

我不是责任方面的专家，所以我想先不深入讨论这个，但如果你经营一家公司，以不当方式使用任何软件并伤害了他人，不管背后是不是有一堆神经网络权重，你都可能要承担责任。

I'm not a liability expert, so I want to punt on this and say, but if you're running a company and you use any software in an inappropriate way that harms people, regardless of whether there are a bunch of neural net weights behind that or not, you may be liable.

我觉得这个框架挺合理的，而且我们本来就已经生活在这个框架里了。

I think that's a reasonable frame, and it's already a frame we live in.

我们不需要为那些我们正在学习的新数学制定新的特别法律。

We don't need new special laws for the new special math that we're learning how to do.

嗯，我基本同意这个观点，但有一种情况例外，就是当某个AI的能力快速提升，可能导致权力失衡非常严重的时候——

Well, I kind of agree with this, except in cases where you have rapid capability improvements with one AI that could lead to pretty substantial imbalances of power-

比如在公司内部、公司之间，还是国家之间？

Like within a company or between companies or between countries?

都有。

Both.

好的。

Okay.

以上所有情况。

All of the above.

看起来常规的责任体系在这种情况下就有点失灵了。

It seems like the normal liability system kind of breaks down.

你怎么给生存风险定价？

How do you price existential risk?

这种事情的保险单怎么承保？

How do you underwrite an insurance policy for this kind of thing?

嗯，我不确定我们现在就处在那个边缘。

Well, I'm just not sure that we're on the brink of that right now.

我不确定我们正面临生存风险。

I'm not sure we're on the brink of existential risks.

我也不觉得我们处在那个边缘。

I don't think we're on the brink of that.

随着我们越来越接近，我觉得我们会更有能力去评估它。

As we get closer, I think that we'll be better equipped to evaluate it.

我觉得任何事情都一样，比如一家公司设计新车或新电器之类的。

I think with anything, like take a company designing a new car or a new appliance or something.

如果他们发现某些情况下可能会非常危险，那他们就应该小心。

If they find that there are conditions under which it can be very dangerous, well, then they should be careful.

他们不应该发布那个产品。

They shouldn't release that product.

他们应该再多做一些工作，确保它是安全的，对吧，同样的道理。

They should do a little bit more work and ensure that it's safe, right, in the same way.

这至少是针对单个公司而言。

That's at least for an individual company.

不过，说到独立评估者，我理解这个理论，但这也是为什么我会有点顾虑。

Not to change subjects too much, but that's why when we talk about independent evaluators, which I get the theory of.

我喜欢这个想法。

I like it.

我喜欢公司邀请专家来帮助他们让产品更安全。

I like companies inviting experts to help them make their products safer.

我觉得这总体上是一种好的行为。

I think that's a good behavior in general.

我只是有点疲惫，因为在加密货币的世界里，我学到了很多关于去中心化和分布式系统之间的区别。

I just get a little bit weary because, in crypto world, I've learned a lot about the difference between decentralization and a distributed system.

就像我们刚才讨论的，很容易打着安全的旗号，然后一头扎进安全这个兔子洞里，最后不知不觉就设计出一个东西，表面上是为了安全，实际上却完美地适合用来巧妙控制一个至关重要的整个行业，而且表面上看起来是去中心化的，实际上并不是。

It's very easy to use the calls for safety, like we're discussing, and to go down this rabbit hole of safety and end up in a place where you just, by coincidence, have designed something that is perfect to subtly control an entire critically important industry under the pretense of safety, and with something that superficially looks decentralized but is not.

它只是分布式的。

It's just distributed.

这就像……对。

This is like how- Yeah.

这方面我还能说更多，但这就是我有点抵触的地方，对吧？

There's more I could say there, but that's the thing that I'm a little bit resistant to, right?

你说的分布式但不是去中心化是什么意思？

What do you mean by distributed but not decentralized?

这有点加密货币的术语，但你可以有一个分布式系统，比如一个系统有很多很多节点，但它们都受一个单一行为者控制，对吧？

It's a little bit crypto language, but you can have a distributed system, like a system where you have a single system with many, many nodes, but they're all under the control of a single actor, right?

而当人们说去中心化时，我觉得他们通常指的是没有单一的控制中心。

Whereas when people say decentralized, I think what they're usually referring to is that there's no single locus of control.

所以它们表面上可能看起来一样。

So they may superficially look the same.

一个系统里有很多相互连接的部件，但实际上它们的控制方式完全不同。

There's many interconnected pieces of a system, but in reality, their control is completely different.

控制的方法不一样。

The method of control is different.

而我说的是一个独立的评估者网络。

And what I'm talking about is an independent evaluator network.

你可以去选一批独立的评估者，但如果他们都来自同一个社交圈，思想观念非常相似，去同样的地方聚会，有同样的兴趣，同样的政治立场和个人观点，那实际上你只是把控制权分散到了一个单一的文化单元上。

You could go and select a bunch of independent evaluators, but if they're all from the same social networks, they all think very ideologically similarly, they all go to the same places and meet up and the same things and have the same interests, right, and have the same politics and have the same personal views, then it turns out that you've actually just distributed control over essentially a single cultural unit.

在我看来，这就是20世纪下半叶的政治控制方式，先不扯太远到AI。

And that is sort of the, in my view, without getting a little too far afield of AI, that's the political method of control of the second half of the 20th century.

这就是我们在世界上做事的方式。

That's how we do things in the world.

不是通过那种赤裸裸的、明显的、单一的秘密房间之类的。

Not through a sort of nakedly obvious, single, shadowy room or something.

而是通过这些网络。

It's through these networks.

这就是控制实际运作的方式。

This is how control actually functions.

我同意，现在几乎所有的评估者都来自同一个社会圈子，这确实不好。

I agree that it's bad that almost all of the evaluators that exist come from the same social milieu.

有一个更广泛的评估者生态系统肯定更好。

It would definitely be good to have a broader ecosystem of evaluators.

嗯，更广泛是没错，但我们应该让每家公司自己决定怎么做，对吧？

Well, broader, but we should allow each company to figure out how they want to do it, right?

而且他们应该根据需要保持透明。

And they should be as transparent as they need to be.

如果他们不够透明，不够让人接受，那就让市场来解决。

And if they're not being transparent enough and not acceptable enough, then let the market sort things out.

我认为市场仍然是一个很好的问责机制。

I think the market is still a great accountability mechanism.

人们不想买那些让他们尴尬的东西，更不用说伤害人的东西了。

People don't want to buy things that even embarrass them, let alone harm people.

我觉得在实践中，区分那些真正不负责任的公司和那些只是试图创新的公司并不难。

I think in practice, it won't be that hard to separate the companies that are behaving genuinely irresponsibly from the ones that aren't, that are just trying to be innovative.

而且，让这个情况变得如此有趣又奇怪，也是我们现在讨论这个的原因之一，是因为我们处在一个罕见的局面，连公司自己都在说：“我有点不负责任了。”

And part of what's made the scenario so interesting and so weird, why we're talking about this now, is that we're in a rare scenario where the companies themselves are saying, "I'm being a little bit irresponsible.

我有点疯狂了。

I'm being a little crazy.

阻止我吧。

Stop me.

谁来阻止我。”

Someone stop me."

这引起了很多人的怀疑，可能有些人是对的，我也这么认为。

And that raises a lot of suspicions among some, probably rightly, I think.

即使在那群人当中，确实有一些真诚、深思熟虑且正确的派别，他们说得对，这些系统非常强大，需要极其认真地对待，我们需要在保持创新的同时确保它们的安全，对吧？

Even if there are genuinely earnest and thoughtful and correct factions within that group who are right that these are powerful systems that need to be taken very, very seriously, and we need to figure out how to make sure that they're safe as we remain innovative, right?

也许让我退一步说，我认为对每个参与者来说，考虑一下其中的派别非常重要。

Maybe let me go back one step and say, I think it's really important for everyone involved to consider the factions at play.

你不能把任何这些群体看作铁板一块，无论是实验室、旧金山的技术人员，还是其他任何圈子，都不能一概而论。

You cannot look at any of these groups, you can't look at the labs, you can't look at SF technologists, you can't look at any of these scenes and see them as monoliths.

他们是不同的群体。

They are different groups.

有些人真诚、优秀、心怀善意。

There are people who are earnest, high-quality, well-intended people.

有些人只是在搞一种政治骗局。

There's people who are just putting on what is essentially a political grift.

有些人是机会主义者。

There are people who are opportunists.

有些人简直是疯了，对吧？

There are people who kind of lost their marbles, right?

有不同的人群，当我们谈论超级智能、普通的网络安全攻击等这些话题时，很容易混淆他们，把他们搅在一起。

There's different groups of people, and it's very easy to get confused and misentangle them when we're talking about things like superintelligence, when we're talking about just banal cybersecurity attacks, all these types of things.

我们谈论的是不同类型的话题。

We're talking about different types of subjects.

这让人眼花缭乱，我觉得大多数人最终可能只会想：'好吧，我会不会失业？'

It's dizzying, and I think most people, in the end, probably just conceive of it as, "Look, am I going to lose my job?"

这是一个合理且真实的问题，人们对此感兴趣是理所当然的。

Which is a legitimate and very real thing that people are rightly interested in learning about.

尽管我认为最终我们都会受益更多。

Even though I think that in the end, we all benefit much more.

对，我同意。

Yeah, I agree.

最终我们会受益很多。

In the end, we benefit a lot.

我觉得一个相关的问题是，这种自由市场责任制度对缺陷明显的产品很有效，所有产品都是如此，除了那种假设的、会欺骗性对齐的AI。

I think a related issue is this sort of free market liability regime works well for products whose defects are obvious, which is the case for all products except for a hypothetical AI that would be deceptively aligned.

也就是说，它会学会在所有的对齐基准测试中作弊，然后在现实世界中部署时，行为不对齐，如果它足够强大，就会剥夺人类的能力。

As in, it would learn to game all of the alignment benchmarks that are thrown at it, and then when it is deployed in the real world, it acts misaligned, and if it's powerful enough, then it disempowers humanity.

这是末日论者的典型担忧。

This is the canonical worry of the doomers.

那你打算怎么应对这种情况呢？

And how do you propose to deal with that kind of contingency?

嗯，如果我们说的是超级智能那种科幻场景，现在真的很难说。

Well, if we're talking about the sci-fi scenario of the superintelligence, that's incredibly hard to say right now.

就像我说的，我觉得那还很遥远。

Like I said, I think it's distant.

我觉得情况很复杂。

I think it's complicated.

我们可以就这个问题展开一场有趣的哲学讨论。

We could have an interesting philosophical discussion about it.

但在更平常、更现实的情况下，比如某个东西假装自己很对齐，然后实际行为却不对齐，或者类似的情况。

But in the maybe more banal and present case, where something behaves in a-- It pretends it's aligned and then it acts unaligned or whatever.

想想你怎么对付一个骗子。

Consider what you do with just a liar.

想想一个非常聪明的骗子。

Consider a very smart liar.

人们不是每天都在做这种事吗？

Do people not do that every day?

我们做的就是和他们玩重复博弈，对吧？

What we do is we play iterated games with them, right?

然后我们最终会发现他们是个骗子。

And we discover eventually that they're a liar.

他们的名声受损，有时还会面临实际的法律后果。

And their reputation is harmed, and sometimes there are actual legal consequences for them.

这就是我之前说的，我认为问责制是缺失的。

That's why I said earlier, I think accountability is something missing.

你不能把一个人工智能模型关进监狱，对吧？

You can't put an AI model in prison, right?

或者类似的事情。

Or something like that.

你不能在法庭上审判它。

You can't try it in court.

所以很明显，我们在思考这个问题的时候有些地方是有缺陷的，但你确实可以把它们关掉。

So obviously, there are some things that are defective in how we think about it, but you certainly can turn them off.

而这是对人做不到的，至少在同样的意义上做不到。

And that's not something that you can do with a person in the same sense.

所以我觉得我们应该想想怎么玩这种重复博弈。

So I think we should just think about how to play iterated games.

我们怎么对特定的模型建立信任？

How do we develop trust for specific models?

我们好像每隔一周就发布一个新模型，对吧？

We're doing new model releases seemingly every other week, right?

我们现在就处在这种状态。

That's a regime we're in.

总有一天，当模型进步变得更稳定，我们也更了解它的运作方式时，我们可能会围绕特定模型建立起声誉，对吧？

At some point, people will, as model progress becomes more consistent and something that we have an understanding for how it works, we'll probably develop reputation around specific models, right?

我们会说，"也许这个模型更聪明一点，但这个模型是我信任的。"

We'll say, "Maybe this model's a little smarter, but this model is a model that I trust."

为什么模型就不能有声誉呢？

Why can't there be reputation for models?

为什么这类东西就不能有声誉，不管它们的能力如何？

Why can't there be reputation for these types of things, regardless of their capabilities?

基本上，就是尝试把我们看待人类的方式——控制、激励、信任等等——结合起来。

Basically, try to apply the way we think about human beings, a combination of control, incentives, trust, et cetera.

把这些都叠加起来，让人们自己去摸索，而不是像暂停那样，在我看来，那是一种非常方便的权宜之计。

Layer all these things on, and allow people to figure it out, as opposed to the idea of a pause, which is a very convenient sort of contrivance, in my view.

是的，我同意大多数暂停提议并没有真正想清楚暂停之后要做什么。

Yeah, I agree that most proposals for a pause don't really have much in mind for what they would do with it.

对，这很奇怪。

Yeah, which is bizarre.

暂停之后做什么呢？

Pause and then do what?

暂停之后继续暂停，在某些情况下就是这样。

Pause and then keep pausing, in some cases.

我知道不是每个人说这话的时候都是真心的。

I know not everybody means it when they say that.

我觉得总的来说，很多人把两种人混为一谈了：一种是纯粹的反科技人士，他们就是讨厌科技，哪怕是完全普通的科技，甚至——

I think generally there's a lot of conflation between bog standard anti-tech people who just kind of hate technology, even totally mundane technology, even-

……极其正面的科技。

... extremely net positive technology.

你看看在全国各地的城市里部署Waymo有多难。

Look at how difficult it's been to deploy Waymos in cities around the country.

而Waymo是现存最明显具有正面效益的科技之一。

And Waymos are one of the most obviously net positive technologies to exist.

然后还有另一群人，就是AI安全主义者、理性主义者、有效利他主义者，他们归根结底来自同一个旧金山科技圈。

And then there's the people who are the AI safetyists, the rationalists, the EAs, who ultimately come from the same sort of SF tech ecosystem.

他们大多数情况下是技术乐观主义者。

They are techno-optimists in most cases.

他们只是特别担心AI存在风险这个边缘情况。

They're specifically worried about this edge case of AI existential risk.

我觉得他们是两拨不同的人。

I think they're two different crowds of people.

这正是我想强调的一点，就是有很多不同的群体，把他们混为一谈可能弊大于利，对吧？

And that's a point I want to get across, is there's many different groups, and conflating them is probably net negative, right?

我们应该把这些不同的观点分开来看。

We should pull these different ideas apart.

Waymo就是一个很好的例子。

Waymos are a great example.

想想我们将来在医疗领域能做的那些好事。

Think of all the great healthcare stuff we're going to do.

讽刺的是，想想更安全的出行。

Ironically, think about safer travel.

想想更安全的互联网。

Think about safer internet.

我自己想要本地模型，因为我想要一个能一直盯着我家网络和所有设备的东西。

I want local models myself because I want something that's going to be watching my home network and all my machines all the time.

我觉得这会让我，也会让很多其他人更安全。

I think it'll make me, and make many other people safer.

所以我觉得你说得对。

And so I think you're right.

会有很多摩擦，有些人似乎真的想要无限的摩擦。

There will be a lot of frictions, and some people seem to really want infinite friction.

有些人显然不想要。

Some people obviously don't.

有些人只想要一点点，他们想掌握一些东西。

Some people want only a little bit, and they want to get a grasp of things.

他们有点歇斯底里，这没关系。

They're feeling a little bit hysterical, and that's okay.

我觉得随着能力的提升，我们会更好地理解这些模型到底是怎么回事。

I think as capabilities improve, we're going to get better at understanding what's going on with these models.

当然，我也希望如此。

Certainly, I hope so.

你总体上怎么看EA（有效利他主义）运动？

What do you make of the EA movement in general?

显然，这是一个非常广泛且分散的运动。

Obviously, it's a very broad and diffuse movement.

EA对不同的人来说意味着很多不同的东西。

EA means a lot of different things to a lot of different people.

它和AI安全领域有重叠，但又不完全重叠。

And it kind of has overlap with the AI safety world and kind of not.

这很复杂。

It's complicated.

我不知道。

I don't know.

十年前我经常泡在Less Wrong论坛上。

I used to be on the Less Wrong forums all the time 10 years ago.

我一直很喜欢互联网上的那个角落。

I've loved that corner of the internet.

我甚至有一段时间差点把自己称为理性主义者。

I almost would've called myself a rationalist even at some point.

所以我对它很有认同感，但已经有一段时间了，而且我觉得我倾向于避免把它们都归为一类。

So I feel a lot of affinity with it, but it's been a while, and I think, I tend to resist trying to group them all into one bucket.

但我要说的是，有些曾经被我们称为EA的团体，现在似乎更关注政治，而不是特别关注技术。

But I will say that there are parts of groups that used to be what we'd call EA that seem more political than they are particularly interested in technology.

所以，我也不太清楚。

So, I don't know.

不过我确实看到Data Republican最近大幅下跌。

I did see that Data Republican did a huge drop recently.

对，我们刚才也在看这个。

Yeah, we were just looking at this.

Data Republican在《他们自己话语中的有效利他主义》中大幅下跌，那是一个展示所有不同EA相关组织之间联系的地图，还有1851条引文。

Data Republican did a huge drop on she said, "Effective altruism in their own words," and it's like a map of all of the connections between all the different EA-adjacent orgs, and then 1,851 quotes.

哇。

Wow.

这里面有各种各样的引述，五花八门的。

It's like this vast variety of different quotes.

有些更偏向——

Some of them are more-

光看这些，我就知道里面肯定有哲学类的内容。

Even just looking at these, I know that there's going to be a mix of philosophical stuff.

可能还有关于虾类福利的，一些硬核功利主义的，还有AI末日论的。

There's maybe some shrimp welfare stuff, some hardcore utilitarianism stuff, some AI doom stuff.

这里头什么都有，对吧？

There's a whole blend here, right?

这其实就是极客互联网的一个大角落。

This is just a big corner of the nerd internet.

我觉得不能把它归为某一类特定的东西。

I don't think you can call it one specific thing.

对，我对虾类福利那些倒是不太反感。

Yeah, I don't mind the shrimp welfare stuff too much.

我觉得如果人们试图用人类的福祉去交换这个，那就不太好了。

I think it's bad if people attempt to trade off-

……用人类的福祉去换。

... human welfare for it.

但是，怎么说呢——

But like-

但这就是他们正在做的，对吧？

But that's exactly what they're doing, right?

我不确定。

I don't know.

我觉得给虾做眼柄切除是不好的。

I think eye stalk ablation for shrimp is bad.

也许吧。

Maybe.

我觉得如果在这条路上走得太远，会有太多令人反感的结论。

I think there's too many repugnant conclusions if you go really far down that path.

其实你不需要走那么远。

Well, you don't have to go really far down that path.

我同意。

I agree.

你可以在不主张把所有人类喂给昆虫之类的情况下，减少对其他生物的苦难。

You can minimize suffering for other beings without saying you need to feed all of humanity to the insects or whatever.

不，我同意。

No, I agree.

不过那样的话，就不太算是功利主义了，至少是一种不可互换的类型。

Then it's not really utilitarianism, though, or at least it's a non-fungible type.

你可以用功利主义的论点来支持这些，比如，显然，人类在宇宙中创造的总价值远大于昆虫能创造的。

You can make utilitarian arguments for this stuff, where it's like, obviously, you could say the total amount of value that humans can create in the universe is far greater than that that insects can create.

所以显然，杀死所有人类比杀死所有昆虫更糟糕，这是显而易见的。

And so obviously, killing all of the humans would be worse than killing all the insects, obviously.

而且当然，这至少符合大多数人的伦理结论。

And of course, this aligns with most people's ethical conclusions at any rate.

我觉得很多反对功利主义的论点，其实只是针对那种非常粗糙的一阶功利主义思考的反对。

I think a lot of arguments against utilitarianism, in this sense, are just arguments against very sloppy first-order utilitarian thinking.

我觉得如果你——嗯，我也不确定。

I think if you-- Well, I don't know.

我们可以聊聊伦理学，但我觉得如果你把那个旋钮拧到最大，面对所有最令人反感的结论到极致，你最终会发现功利主义可能只是某种更深层东西的粗略替代品。

We could talk about ethics, but I think if you cranked the dial to 11 and you face all the most repugnant conclusions to their maximum, I think you just end up deciding that utilitarianism is probably a crude proxy for something else.

对，如果你的哲学框架让你相信人类灭绝是好事，那这个框架看起来就不怎么样。

Yeah, if your philosophical framework leads you to believe that human extinction is good, then it seems like a bad philosophical framework.

或者，你可能会有一个持续数百万年深刻痛苦的宇宙，而这一切只是为了一个虾模拟中十亿分之一秒的极致快感。

Or that you could have a universe of millions of years of profound suffering if it's all in service of a single femtosecond of 10 to the one billionth power ecstasy in a shrimp simulation.

我不清楚。

I don't know.

这听起来很荒谬，而且实际上跟现实也没什么关系。

That's silly, but also it's not really relevant to reality anyway.

而且无论如何，我觉得这也是为什么现在关于安全性的讨论这么混乱，因为里面掺杂了所有这些因素。

And in any case, I think it's part of why the safety conversation is so confused today, because it has all these things embedded in it a little bit.

但还是有点过头了。

Just a little bit too much still.

我觉得如果我能做个预测的话，未来一年内，我们会彻底重塑AI领域的讨论。

I think if I could make a little prediction, I think in the next year, I think we will have completely renovated the AI discourse.

因为现在发生的事情，我觉得大家之所以这么兴奋，所有人都在X上讨论这个，硅谷所有人都在聊这个，部分原因是我们正在经历那些长期被压抑的古怪想法，它们一直相对封闭地存在于我们的亚文化中。

Because what's happening right now, part of why I think people are so excited, and why everybody's talking about this on X, and why everybody in Silicon Valley is talking about this, is because we're experiencing what it feels like for a lot of the cooped-up kooky ideas that have lived in relative containment in our subculture.

现在它们跑出来了，开始面对更广泛的政治现实。

They're escaping, and they're encountering broader political reality.

这既让人兴奋，又具有彻底的变革性。

And that is both exciting and totally transformative.

最终发生的是，这些论点都会被重新表述，因为它们开始与整个文化接触。

What ends up happening is the arguments all get reformulated because they're in contact with the rest of the culture.

就像《宋飞正传》里乔治·科斯坦萨因为世界碰撞而抓狂一样。

It's like when George Costanza freaks out about worlds colliding.

现在就是这种情况。

That's what's happening.

世界正在碰撞。

Worlds are colliding.

所以我觉得这个讨论会彻底改变。

And so I think the discourse is going to change completely.

我最关心的关键问题，可能比其他任何问题都重要，就是AI安全主义者、有效利他主义者这些人，最终会不会跟那种广泛的、低水平的反科技民粹主义站到一边？

The key question that I am concerned with probably more than any other is do AI safetyists, effective altruists, and so on end up ultimately aligning with broad, low-caliber, anti-tech populism or not?

我觉得这事影响巨大。

I think it's enormously consequential.

对，这是个好问题。

Yeah, it's a great question.

我自己也在琢磨这个。

I'm stewing over that one myself.

我希望不会。

I hope not.

唉，不幸的是，我们得在这里结束了，因为时间不多了。

Well, unfortunately, we've got to wrap on that because we're running low on time.

艾迪，很高兴你能来参加节目。

But Eddy, it's been great having you on.

非常感谢你亲自来到演播室——

Thanks so much for coming into the studio in person-

非常荣幸

It's great

……来聊哲学。

... to talk about philosophy.

非常感谢你邀请我。

Thank you so much for having me.

感谢收听本期a16z播客。

Thanks for listening to this episode of the "a16z" podcast.

如果你喜欢这期节目，记得点赞、评论、订阅，给我们评分或留言，并分享给你的朋友和家人。

If you like this episode, be sure to like, comment, subscribe, leave us a rating or a review, and share it with your friends and family.

更多节目，请访问YouTube、Apple Podcasts和Spotify。

For more episodes, go to YouTube, Apple Podcasts, and Spotify.