罗曼·扬波利斯基：2030年仅存的5种职业及我们生活在模拟中的证据！

你从事AI安全研究至少已有二十年了。

You've been working on AI safety for two decades, at least.

是的。我曾坚信我们能造出安全的AI。但越是深入研究，我越意识到这实际上是我们无法做到的。

Yeah. I was convinced we can make safe AI. But the more I looked at it, more I realized it's not something we can actually do.

你曾对多个不同时间点做过一系列预测。那么你对2027年的预测是什么？罗曼·扬普尔斯基博士是全球公认的AI安全领域权威，计算机科学副教授。他向人们揭示AI的可怕真相，以及我们需要采取什么行动来拯救人类。

You have made a series of predictions about a variety of different dates. So what is your prediction for 2027? Doctor Roman Yampulski is a globally recognized voice on AI safety and associate professor of computer science. He educates people on the terrifying truth of AI. And what we need to do to save humanity.

两年内，取代大多数人类职业的能力将迅速到来。我是说，五年后我们将面临前所未有的失业水平——不是10%，而是99%。这还只是没有超级智能的情况。一个在所有领域都比人类聪明的系统。

In two years, the capability to replace most humans in most occupations will come very quickly. I mean, in five years, we're looking at a world where we have levels of unemployment we've never seen before. Not talking about 10%, but 99%. And that's without superintelligence. A system's smarter than all humans in all domains.

所以它在制造新AI方面会比我们更出色。但更糟的是，我们不知道如何确保它们的安全。而世界上最聪明的人们仍在竞相争夺超级智能的研发领先地位。

So it would be better than us at making new AI. But it's worse than that. We don't know how to make them safe. And yet we still have the smartest people in the world competing to win the race to super intelligence.

但你怎么看待像萨姆·奥特曼这样的人在AI领域的探索？

But what do make of people like Sam Ottman's journey with AI?

十年前我们发布了如何正确发展AI的护栏准则。他们违反了每一条规则，他正用80亿人的生命作赌注来换取更富有和更强大。我想有些人想去火星，有些人想掌控宇宙。但由谁来建造并不重要。

So a decade ago, we published guardrails for how to do AI right. They violated every single one, and he's gambling 8,000,000,000 lives on getting richer and more powerful. So I guess some people want to go to Mars. Others want to control the universe. But it doesn't matter who builds it.

当你转向超级智能的那一刻，我们很可能会为此追悔莫及。

The moment you switch to superintelligence, we will most likely regret it terribly.

那么到2045年

And then by 2045

现在才是真正有趣的部分。

Now this is where it gets interesting.

罗曼·金普尔斯基医生，我们来谈谈模拟理论。

Doctor Roman Gimpulski. Let's talk about simulation theory.

我认为我们正身处其中。对此有广泛共识，而你应该做的就是确保我们不会关闭这个模拟。首先

I think we are in one. And there is a lot of agreement on this, and this is what you should be doing in it so we don't shut it down. First

请给我三十秒时间。我想说两件事。第一是衷心感谢你们每周收听节目，这对我们所有人而言意义重大，这个我们从未敢想象的梦想如今竟成真了。但其次，我们感觉这梦想才刚刚开始。

Just give me thirty seconds of your time. Two things I wanted to say. The first thing is a huge thank you for listening and tuning into the show week after week. It means the world to all of us and this really is a dream that we absolutely never had and couldn't have imagined getting to this place. But secondly, it's a dream where we feel like we're only just getting started.

如果你喜欢我们的节目，请加入24%定期收听播客的听众行列，在这个应用上关注我们。我向你承诺：我将竭尽所能让节目现在和未来都保持高质量，邀请你想听的嘉宾，继续呈现你喜爱的节目内容。谢谢。

And And if you enjoy what we do here, please join the 24% of people that listen to this podcast regularly and follow us on this app. Here's a promise I'm gonna make to you. I'm gonna do everything in my power to make this show as good as I can now and into the future. Gonna deliver the guests that you want me to speak to and we're gonna continue to keep doing all of the things you love about this show. Thank you.

罗曼·金普尔斯基医生，你当前肩负着什么使命？因为我清楚地感觉到你带着某种使命，这个使命你至少坚持了近二十年。

Doctor Roman Impulski, what is the mission that you're currently on? Because it's quite clear to me that you are on a bit of a mission, and you've been on this mission for, I think, the best part of two decades at least.

我希望确保我们正在创造的超级智能不会杀死所有人。

I'm hoping to make sure that superintelligence we are creating right now does not kill everyone.

请详细解释这个惊人论断的背景。

Give me some give me some context on that statement because it's quite a shocking statement.

好的。过去十年我们确实掌握了提升人工智能的方法。事实证明，增加算力和数据就能让它变得更聪明。现在全世界最聪明的人和数十亿美元都在致力于创造最强大的超级智能。但问题是，我们虽能提升其能力，却不知如何确保其安全性，避免它们做出让我们后悔的事。

Sure. So in the last decade, we actually figured out how to make artificial intelligence better. Turns out, if you add more compute, more data, it just kinda becomes smarter. And so now smartest people in the world, billions of dollars, all going to create the best possible superintelligence we can. Unfortunately, while we know how to make those systems much more capable, we don't know how to make them safe, how to make sure they don't do something we will regret.

这就是当前的技术现状。预测市场显示，我们距离高级AI出现的时间非常短——只有两三年，顶级实验室CEO们也这么认为。但与此同时，我们却无法确保这些系统与人类价值观对齐。

And that's the state of the art right now. When we look at just prediction markets, how soon will we get to advanced AI? The timelines are very short, couple of years. Two, three years according to prediction markets, according to CEOs of Top Labs. And at the same time, we don't know how to make sure that the systems are aligned with our preferences.

我们正在创造的是外星智慧般的存在。如果有外星人三年后降临地球，你现在就该恐慌了。但大多数人甚至没意识到这件事正在发生。

So we are creating this alien intelligence. If aliens were coming to Earth and you you have three years to prepare, you would be panicking right now. But most people don't don't even realize this is happening.

一些反对观点可能会说，这些人非常非常聪明。这些是大公司，资金雄厚。他们不仅有道德义务，更有法律责任确保不造成伤害。所以我相信一切都会没事的。

So some of the counterarguments might be, well, these are very, very smart people. These are very big companies with lots of money. They have a obligation and a moral obligation, but also just a legal obligation to make sure they do no harm. So I'm sure it'll be fine.

他们唯一的法律责任就是为投资者赚钱。他们没有道德或伦理义务。而且对他们来说，目前还不知道如何实现。最先进的回答是：等我们到了那一步再想办法，或者用AI来控制更高级的AI。

The only obligation they have is to make money for the investors. That's the legal obligation they have. They have no moral or ethical obligations. Also, to them, they don't know how to do it yet. The state of the art answers are, we'll figure it out then we get there, or AI will help us control more advanced AI.

这太荒谬了。

That's insane.

从概率角度，你认为出现灾难性错误的可能性有多大？

In terms of probability, what do you think is the probability that something goes catastrophically wrong?

没人能确切预知未来。但如果你不掌控制权，就得不到想要的结果。可能性空间近乎无限，而我们满意的结果范围极小。

So nobody can tell you for sure what's going to happen. But if you're not in charge, you're not controlling it, you will not get outcomes you want. The space of possibilities is almost infinite. The space of outcomes we will like is tiny.

请问你是哪位？从事这个领域多久了？

And who are you and how long have you been working on this?

我是受过专业训练的计算机科学家，拥有计算机科学与工程博士学位。大约十五年前就开始从事广义的AI安全研究，当时主要是机器人控制。

I'm a computer scientist by training. I have a PhD in computer science and engineering. I probably started work in AI safety mildly defined as control of bots at the time fifteen years ago.

十五年前？所以你在AI安全还没流行时就研究这个了？

Fifteen years ago. So you've been working on AI safety before it was cool?

在这个术语出现之前。'AI安全'这个词是我创造的。

Before the term existed. I coined the term AI safety.

你是'AI安全'这个术语的创始人？

You're the founder of the term AI safety?

这个术语。是的。不是指整个领域。在我涉足之前，已有其他人做出了卓越的工作。

The term. Yes. Not the field. There are other people who did brilliant work before I got there.

为什么你十五年前就在思考这个问题？因为大多数人直到最近两三年才开始谈论AI安全这个术语。

Why were you thinking about this fifteen years ago? Because most people have only been talking about the term AI safety for the last two or three years.

没错。最初这只是一个温和的安全项目。当时我在研究扑克机器人，意识到它们正变得越来越强。只要时间足够，它们终将超越人类——更聪明、更能干。而事实确实如此。

Yeah. It started very mildly just as a security project. I was looking at poker bots, and I realized that the bots are getting better and better. And if you just project this forward enough, they're gonna get better than us, smarter, more capable. And it happened.

现在它们打扑克已远超普通玩家。但更广泛地说，这种超越会发生在所有领域、所有数字资源上。我想确保AI成为对全人类有益的技术，于是开始研究如何让AI更安全。

They are playing poker way better than average players. But more generally, it will happen with all other domains, all the other cyber resources. I wanted to make sure AI is a technology which is beneficial for everyone. So I started work on making AI safer.

你职业生涯中是否有某个瞬间让你觉得'天啊'？

Was there a particular moment in your career where you thought, oh my god.

至少前五年，我都在努力解决这个问题。当时坚信我们能成功——创造出安全的AI。但越是深入研究，就越发现这个等式里的每个要素我们都无法真正掌控。

First five years at least, I was working on solving this problem. I was convinced we can make this happen. We can make safe AI. And that was the goal. But the more I looked at it, the more I realized every single component of that equation is not something we can actually do.

就像分形结构，放大观察会发现10个新问题，然后是100个。这些问题不仅困难——根本无解。这个领域没有里程碑式的成果能宣告'我们解决了这个'。

And the more you zoom in, it's like a fractal. You go in and you find 10 more problems and then a 100 more problems. And all of them are not just difficult. They're impossible to solve. There is no seminal work in this field where, like, we solved this.

我们只能打补丁，做些小修小补，但人们很快就能找到规避方法，破坏我们的安全机制。AI能力的进步是指数级甚至超指数级的，而AI安全的进步却是线性或停滞的。

We don't have to worry about this. There are patches. There are little fixes we put in place, and quickly people find ways to work around them. They drill, break whatever safety mechanisms we have. So, while progress in AI capabilities is exponential or maybe even hyperexponential, progress in AI safety is linear or constant.

差距正在扩大。

The gap is increasing.

这个差距指的是

The gap between the

这些系统的能力有多强，我们能在多大程度上控制它们，预测它们的行为，解释它们的决策过程。

How capable the systems are and how well we can control them, predict what they're gonna do, explain their decision making.

我认为这一点非常重要，因为你提到我们基本上是在修补发现的问题。我们正在开发这种核心智能，而为了防止它做出某些行为或展现不可预测性及威胁性，开发这些AI的公司会在顶层编程代码，比如规定不准说脏话、不准使用粗鲁词汇、不准做某些坏事。

I think this is quite an important point because you you said that we're basically patching over the issues that we find. So we're developing this this core intelligence. And then to stop it doing things or to stop it showing some of its unpredictability or its threats, the companies that are developing this AI are programming in code over the top to say, okay, don't swear. Don't say that rude word. Don't do that bad thing.

没错。你可以看看其他类似的例子，比如人力资源手册。我们人类也有这样的规范。

Exactly. And you can look at other examples of that. So HR manuals. Right? We have those humans.

它们都是通用智能，但你希望它们在公司里表现良好。所以制定了政策：禁止性骚扰，禁止这个，禁止那个。但如果你够聪明，总能找到变通办法。

They're general intelligences, but you want them to behave in a company. So they have a policy. No sexual harassment. No this, no that. But if you're smart enough, you always find a workaround.

所以，你只是把行为推向了另一个尚未受到限制的子领域。

So, you're just pushing behavior into a different, not yet restricted subdomain.

我们可能需要先定义一些术语。有狭义智能，比如会下棋的那种；有人工通用智能（AGI），能跨领域操作；还有超级智能，在所有领域都比所有人类更聪明。我们现在处于什么阶段？

We should probably define some terms here. So there's narrow intelligence, which can play chess or whatever. There's artificial general intelligence, which can operate across domains, and then super intelligence, which is smarter than all humans and all domains. And where are we?

这个边界其实非常模糊。毫无疑问，我们已拥有许多出色的狭义系统，在特定领域它们就是超级智能。比如蛋白质折叠问题就是由狭义AI解决的，它在该领域超越了所有人类。至于AGI，我说过，如果把现在的系统展示给二十年前的科学家，他们会坚信我们已经实现了完全的AGI。我们现有的系统能够学习，

So, that's a very fuzzy boundary, right? We definitely have many excellent narrow systems, no question about it, and they are super intelligent in that narrow domain. So, protein folding is a problem which was solved using narrow AI and it's superior to all humans in that domain. In terms of AGI, again, I said if we showed what we have today to a scientist from twenty years ago, they would be convinced we have full blown AGI. We have systems which can learn.

能在数百个领域表现优异，其中许多方面已超越人类。所以可以说我们拥有弱化版的AGI。但尚未达到超级智能，仍有杰出人类在科学和工程等领域完全领先于AI。

They can perform in hundreds of domains. And they're better than human and many of them. So you can argue we have a weak version of AGI. Now, don't have superintelligence yet. We still have brilliant humans who are completely dominating AI, especially in science and engineering.

但这个差距正在飞速缩小。尤其在数学领域，三年前大语言模型连基础代数都不会，三位数乘法都是挑战，现在它们已能协助数学证明，

But that gap is closing so fast. You can see, especially in the domain of mathematics. Three years ago, large language models couldn't do basic algebra. Multiplying three digit numbers was a challenge. Now, they're helping with mathematical proofs.

赢得数学奥林匹克竞赛，甚至着手解决千禧年难题——数学界最困难的问题。短短三年间，我们就从低于人类水平发展到超越全球大多数数学家。在科学和工程领域，我们也看到了同样的发展轨迹。

They're winning mathematics Olympiads competitions. They are working on solving millennial problems, hardest problems in mathematics. So in three years, we closed the gap from subhuman performance to better than most mathematicians in the world. And we see the same process happening in science and in engineering.

你已经做了一系列预测，这些预测对应着各种不同的日期。我这里就有这些日期。你对2027年的预测是什么？

You have made a series of predictions, and they correspond to a variety of different dates. I have those dates in front of me here. What is your prediction for the year 2027?

根据预测市场和顶尖实验室的预测，我们很可能在2027年看到人工通用智能（AGI）的出现。

We're probably looking at AGI as predicted by prediction markets and tops of the labs.

也就是说到2027年我们将拥有人工通用智能。那这会让世界与现在有什么不同？

So we'd have artificial general intelligence by 2027. And how would that make the world different to how it is now?

如果你理解‘员工替代’这个概念——你将获得免费的劳动力和认知能力，价值数万亿美元。对大多数工作来说，雇佣人类将毫无意义。如果我只需支付20美元订阅费或使用免费模型就能完成员工的工作，首先所有电脑操作类工作将被自动化。其次，我认为人形机器人可能落后五年。所以五年后，所有体力劳动也能被自动化。

So if you have this concept of a drop in employee, you have free labor, physical and cognitive, trillions of dollars of it. It makes no sense to hire humans for most jobs. If I can just get, a $20 subscription or a free model to do what an employee does, first anything on a computer will be automated. And next, I think humanoid robots are maybe five years behind. So in five years, all the physical labor can also be automated.

我们将面临一个失业率前所未有的世界。不是指10%这种已经令人恐慌的数字，而是99%。最后剩下的只有那些——无论出于什么原因——你更希望由人类来完成的工作。其他所有事情都能完全自动化。但这不意味着实践中会立即实现自动化。

So we're looking at a world where we have levels of unemployment we've never seen before. Not talking about 10% unemployment, which is scary, but 99%. All you have left is jobs where, for whatever reason, you prefer another human would do it for you. But anything else can be fully automated. It doesn't mean it will be automated in practice.

很多时候技术早已存在，但并未投入使用。可视电话发明于70年代，但在iPhone出现前没人使用。所以我们可能还会在现有工作模式和世界形态中维持更长时间。但取代大多数人类和职业的能力会来得非常快。

A lot of times, technology exists, but it's not deployed. Video phones were invented in the '70s. Nobody had them until iPhones came around. So we may have a lot more time with jobs and with a world which looks like this. But capability to replace most humans and most occupations will come very quickly.

好的，让我们深入探讨并压力测试这个观点。像我这样的播客主持人，未来还会需要吗？

Okay. So let's try and drill down into that and stress test it. So a podcaster like me, would you need a podcaster like me?

让我们分析你的工作内容：你需要准备资料（是的）、提出问题（嗯哼）

So let's look at what you do. You prepare. Yep. You ask questions. Mhmm.

你会追问后续问题，而且在镜头前表现得很得体。

You ask follow-up questions, and you look good on camera.

非常感谢。

Thank you so much.

让我们看看能做些什么。如今的大型语言模型可以轻松阅读我写过的所有内容，没错。而且理解得非常透彻——我猜你没读过我的每一本书吧。

Let's see what we can do. Large language model today can easily read everything I wrote Yep. And have very solid understanding better. I assume you haven't read every single one of my books. Yeah.

那东西就能做到。它可以训练你所有的播客内容，从而精确掌握你的风格和提问方式。还能发现哪些内容效果特别好，比如某类问题能显著提升观看量。

That thing would do it. It can train on every podcast you ever did. So it knows exactly your style, the types of questions you ask. It can also find correspondence between what worked really well. Like this type of question really increased views.

这类话题很有潜力，所以我认为它能比你更优化内容，因为你没有这些数据集。当然，视觉模拟现在已经是小菜一碟了。

This type of topic was very promising, so you can optimize, I think, better than you can because you don't have the dataset. Of course, visual simulation is trivial at this point.

所以你能在我坐在这里的几秒钟内就生成视频吗？

So can you can make a video within seconds of me sat here and

我们可以高效生成你采访任何人谈论任何主题的视频，你只需要批准形象授权之类的手续。

So we can generate videos of you interviewing anyone on any topic very efficiently, and you just have to get likeness approval, whatever.

在AGI的世界里，你认为还有多少工作会保留？如果按你说的AGI可能在2027年前就会出现（无论是否部署），那么...先不考虑体力劳动岗位，你觉得在AGI时代还有人类能做得更好的工作吗？

Are there many jobs that you think would remain in a world of AGI? If you're saying AGI is potentially gonna be here, whether it's deployed or not, by 2027, what kind and and then okay. So let's take out of this any physical labor jobs for a second. Are there any jobs that you think a human would be able to do better in a world of AGI still?

这是我常问别人的问题。在AGI的世界里——我认为超级智能会立即随之而来——真正的问题是：在全面超越人类所有能力的超级智能面前，你能贡献什么？比如你比任何人都清楚'作为你'的体验，知道冰淇淋对你而言的滋味。

So that's the question I often ask people. In a world with AGI, and I think almost immediately we'll get superintelligence as a side effect. So the question really is, in a world of superintelligence, is defined as better than all humans in all domains, what can you contribute? And so you know better than anyone what it's like to be you. You know what ice cream tastes to you.

这种认知能变现吗？有人感兴趣吗？可能没有。市场不大。但确实存在需要人类的工作场景。

Can you get paid for that knowledge? Is someone interested in that? Maybe not. Not a big market. There are jobs where you want a human.

或许富人出于历史原因坚持要人类会计师，就像老年人偏爱传统方式。沃伦·巴菲特就不会改用AI，他会继续用人类会计师。但这只是市场的极小部分。

Maybe you're rich and you want a human accountant for whatever historic reasons. Old people like traditional ways of doing things. Warren Buffett would not switch to AI. He would use his human accountant. But it's a tiny subset of the market.

现在有些产品标榜'美国手工制造'而非'中国批量生产'，虽然有人愿意为此多付钱，但这只是小众需求，近乎一种癖好，并没有实际意义。

Today, have products which are man made in US as opposed to mass produced in China. And some people pay more to have those. But it's a small subset. It's almost a fetish. There is no practical reason for it.

我认为任何能在电脑上完成的工作，都可以用这项技术实现自动化。

And I think anything you can do on a computer could be automated using that technology.

你说这话时肯定听到过很多反驳吧？因为当人们听说自己的工作、职业、学位价值、十万美元的投资将被剥夺时，会感到巨大的心理不适。所以有些人自然会产生认知失调反应——不，你错了，AI不可能有创造力，它做不到这些。

You must hear a lot of rebuttals to when this when you say it because people experience a huge amount of mental discomfort when they hear that their job, their career, the thing they got a degree in, the thing they invested a $100,000 into is gonna be taken away from them. So their natural reaction some for some people is that cognitive dissonance that, no. You're wrong. AI can't be creative. It's not this.

不是那样的。它永远不会对我的工作感兴趣。我会没事的...这类论调你经常听到对吧？

It's not that. It'll never be interested in my job. I'll be fine because. You hear these arguments all the time. Right?

特别有意思的是，我问过各行各业的人。比如问我的优步司机：你担心自动驾驶汽车吗？他们都说完全不担心。

It's truly funny. I ask people, and I ask people in different occupations. I'll ask my Uber driver. Are you worried about self driving cars? And they go, no.

没人能取代我！我熟悉纽约的每一条街道，我的导航能力是AI比不了的。所有职业的人都是这套说辞。

No one can do what I do. I know the streets of New York. I can navigate like no AI. I'm safe. And it's true for any job.

大学教授也这么对我说：没人能像我这样讲课，这是独一无二的。但你知道这很荒谬——自动驾驶取代司机已成事实。

Professors are saying this to me. Oh, nobody can lecture like I do. Like, this is so special. But you understand it's ridiculous. We already have self driving cars replacing drivers.

这已经不是可能与否的问题，而是你什么时候会被解雇的问题。

That is not even a question if it's possible. It's like how soon before you fired.

没错。昨天在洛杉矶，我的车全程自动驾驶。设定目的地后，全程不用碰方向盘或刹车，一小时车程零干预。虽然我还得自己停车，但基本等于没在开车。

Yeah. I mean, I've just been in LA yesterday, and my car drives itself. So I get in the car. I'd set I put in where I wanna go, and then I don't touch the steering wheel or the brake pedals, and it takes me from a to b, even if it's an hour long drive without any intervention at all. I actually still park it, but other than that, I'm not I'm not driving the car at all.

现在洛杉矶还有Waymo，手机叫车就会来辆无人驾驶汽车。对于那些以驾驶为职业的听众——如果我没记错，驾驶应该是全球从业人数最多的职业...

I mean, obviously, in LA, we also have Waymo now, which means you order it on your phone, and it shows up with no driver in it and takes you to where you wanna go. Oh, yeah. So it's quite clear to see how that is potentially a matter of time. If for those people because we do have some of those people listening to this conversation right now that their occupation is driving. To offer them a and I think driving is the biggest occupation in the world, if I'm correct.

我确信这就是全球从业人数最多的

I'm sure it is the biggest occupation in the

最顶尖的之一。是的。

one of the top ones. Yeah.

你会对那些人说什么？他们应该怎样规划自己的人生？是该重新学习某项技能，还是考虑时间框架？

What would you say to those people? What what should they be doing with their lives? What should they should they be retraining in something, or what time frame?

这就是当前的范式转变。过去我们总说某个职业会被自动化，建议转行做其他工作。但如果我告诉你所有工作都将被自动化，那就没有备选方案了。你无法通过转行解决问题。看看计算机科学领域。

So that's the paradigm shift here. Before, we always said this job is going to be automated, retrain to do this other job. But if I'm telling you that all jobs will be automated, then there is no plan b. You cannot retrain. Look at computer science.

两年前我们告诉人们：学会编程吧。你是个艺术家，赚不到钱就去学编程。后来我们意识到，人工智能好像也会编程，而且越来越擅长。

Two years ago, we told people, learn to code. You are an artist. You cannot make money. Learn to code. Then we realized, oh, AI kinda knows how to code and getting better.

那就当提示词工程师吧。为AI设计提示词会是个好职业，甚至可以攻读全年制学位。但很快我们发现，AI设计提示词的能力远超人类。

Become a prompt engineer. You can engineer prompts for AIs. It's gonna be a great job. Get a full year degree in it. But then we're like, AI is way better at designing prompts for other AIs than any human.

所以这条路也行不通了。目前最困难的是设计实用型AI代理，但我敢保证一两年后这个领域同样会消失。因此我认为不存在'这个职业应该转行做那个'的解决方案，而是全人类都将面临失业。

So that's gone. So I can't really tell you right now the hardest thing is design AI agents for practical applications. I guarantee you, in a year or two, it's gonna be gone just as well. So I don't think there is a this occupation needs to learn to do this instead. I think it's more like we as a humanity, then we all lose our jobs.

我们该怎么办？经济来源如何解决？谁来供养我们？人生意义又该如何定义？每周多出来的六十到八十小时该做什么？

What do we do? What do we do financially? Who's paying for us? And what do we do in terms of meaning? What do I do with my extra sixty, eighty hours a week?

你已经深入思考过这个困境了，对吧？

You've thought around this corner, haven't you?

略有涉猎。

A little bit.

在你看来，这个困境的出路是什么？

What is around that corner in your view?

所以经济部分看起来很简单。如果你创造了大量免费劳动力，就会拥有大量免费财富，各种现在不太负担得起的东西会变得极其廉价，这样你就能满足每个人的基本需求。有人说你甚至能超越基本需求，为所有人提供非常优渥的生活。真正棘手的问题是：人们该如何打发这些空闲时间？

So the economic part seems easy. If you create a lot of free labor, you have a lot of free wealth, abundance, things which are right now not very affordable become dirt cheap, and so you can provide for everyone basic needs. Some people say you can provide beyond basic needs. You can provide very good existence for everyone. The hard problem is what do you do with all that free time?

对很多人来说，工作是他们生活的意义所在。这些人会感到迷失。我们从退休或提前退休的人身上就能看到这种现象。当然也有很多讨厌工作的人会非常乐意不工作。但现在你有一群人整天无所事事。

For a lot of people, their jobs are what gives them meaning in their lives. So they would be kind of lost. We see it with people who retire or do early retirement. And for so many people who hate their jobs, they'll be very happy not working. But now you have people who are chilling all day.

社会会变成什么样？这会对犯罪率、生育率等各类问题产生什么影响？没人思考过这些。政府也没有制定任何方案来应对99%失业率的状况。

What happens to society? How does that impact crime rate, pregnancy rate, all sorts of issues? Nobody thinks about. Governments don't have programs prepared to deal with 99% unemployment. What

你觉得那样的世界会是什么样子？

do you think that world looks like?

重申一下，我认为关键是要理解其中的不可预测性。我们无法预测一个比我们更聪明的系统会做什么。这个临界点常被类比为物理奇点，称为技术奇点——就像你无法看穿事件视界一样。我可以告诉你我的猜想，但那只是我的预测。

Again, I I think Are you gonna be doing very important part to understand here is the unpredictability of it. We cannot predict what a smarter than us system will do. And the point when we get to that is often called singularity, by analogy with physical singularity. You cannot see beyond the event horizon. I can tell you what I think might happen, but that's my prediction.

这绝非实际会发生的情况，因为我的认知能力根本不足以预测一个更聪明的智能体将如何影响世界。看看科幻作品就知道了——里面从没有超级智能真正采取行动的情节，因为没人能写出那个层面的可信故事。要么像《沙丘》那样直接禁止AI，回避描写；要么就像《星球大战》那样，

It is not what actually is going to happen because I just don't have cognitive ability to predict a much smarter agent impacting this world. Then you read science fiction. There is never a superintelligence in it actually doing anything because nobody can write believable science fiction at that level. They either banned AI like Dune because this way you can avoid writing about it. Or it's like Star Wars.

只有些蠢笨的机器人，从未出现过真正的超级智能。因为从定义上来说，你无法在那个层面进行预测。

You have this really dumb bots, but not nothing super intelligent ever. Because by definition, you cannot predict at that level.

因为超级智能的定义就意味着它会自主决策。

Because by definition of it being super intelligent, it will make its own mind up.

严格来说，如果你能预测它的行为，说明你们处于同等智力水平，这就违背了它比你聪明的初始设定。如果我和超级智能下棋时能预判它的每一步，那说明我和它棋力相当。

By definition, if it was something you could predict, you would be operating at the same level of intelligence, violating our assumption that it is smarter than you. If I'm playing chess with super intelligence and I can predict every move, I'm playing at that level.

就像我的法国斗牛犬试图准确预测我的想法和行动一样。

It's kinda like my French bulldog trying to predict exactly what I'm thinking and what I'm gonna do.

这是个很好的认知鸿沟。不仅是他能预测你要去工作、你会回来，但他无法理解你为何要做播客。这完全超出了他对世界的理解模型。

That's a good cognitive gap. And it's not just he can predict you're going to work, you're coming back, but he cannot understand why you're doing a podcast. That is something completely outside of his model of the world.

是啊。他甚至不知道我去上班。他只是看到我离开房子，不知道我去哪里。

Yeah. He doesn't even know that I go to work. He just sees that I leave the house and doesn't know where I go.

给他买食物。

Buy food for him.

针对你目前的观点，最具说服力的反对论点是什么？

What's the most persuasive argument against your own perspective here?

是说我们不会因为先进技术而失业吗？

That we will not have unemployment due to advanced technology?

是说不会出现这种法国斗牛犬与人类之间的理解鸿沟，以及我想，权力和控制方面的差距？

That there won't be this French bulldog human gap in understanding and, I guess, like power and control?

有些人认为我们可以通过硬件结合（比如Neuralink）或基因重组来增强人类心智，从而创造更聪明的人类。

So some people think that we can enhance human minds either through combination with hardware, so something like Neuralink, or through genetic reengineering to where we make smarter humans.

嗯。

Yeah.

这可能会让我们稍微聪明些。但我不认为生物形态在智力上能与硅基形态竞争。硅基底物在智能方面更具优势——速度更快、更坚韧、在许多方面能效更高。

It may give us a little more intelligence. I don't think we're still competitive in biological form with silicon form. Silicon substrate is much more capable for intelligence. It's faster. It's more resilient, more energy efficient in many ways.

这就是计算机与大脑的构成差异所在。确实。

Which is what computers are made out of versus the brain. Yeah.

因此我认为仅靠改进我们的生物学是无法持续跟上的。有些人认为——这非常具有推测性——也许我们可以将意识上传到计算机中。即扫描你的大脑，某种程度上梳理你的大脑结构，然后在计算机上运行一个模拟程序。你可以加速它，赋予它更多能力。但对我而言，那感觉就像你已不复存在。

So I don't think we can keep up just with improving our biology. Some people think maybe, and this is very speculative, we can upload our minds into computers. So scan your brain, kind of comb of your brain, and have a simulation running on a computer. And you can speed it up, give it more capabilities. But to me, that feels like you no longer exist.

我们只是通过不同方式创造了软件，现在有了基于生物学的AI和基于其他训练形式的AI。可以有进化算法，可以有多种路径实现通用人工智能。但最终，它们都不是人类。

We just created software by different means, and now you have AI based on biology and AI based on some other forms of training. You can have evolutionary algorithms. You can have many paths to reach AGI. But at the end, none of them are humans.

我这里还有个时间节点2030年。你对2030年有什么预测？世界会变成什么样？

I have another date here, which is 2030. What's your prediction for 2030? What will the world look like?

届时我们很可能会拥有灵活性、灵巧度足以在所有领域与人类竞争的人形机器人，包括水管工。我们可以制造人工水管工。

So we probably will have humanoid robots with enough flexibility, dexterity to compete with humans in all domains, including plumbers. We can make artificial plumbers.

连水管工都不放过啊。这感觉像是人类就业的最后堡垒了。那么到2030年2月，也就是五年后，包括特斯拉在内的许多领先企业正在以光速研发人形机器人，它们会越来越高效。这些人形机器人将能在物理空间中活动，比如煎个蛋饼，做任何人类能做的事，当然还会连接AI系统使其具备思考和交谈能力。

Not the plumbers. We're that was that felt like the last bastion of human employment. So 02/1930, five years from now, humanoid robots, so many of the companies, the leading companies, including Tesla, are developing humanoid robots at light speed, they're getting increasingly more effective. These humanoid robots will be able to move through physical space for you know, make an omelet, do anything humans can do, but obviously have be connected to AI as well so they can think, talk.

没错。它们由AI控制，始终连接着网络，所以已经在很多方面占据优势了。

Right. They are controlled by AI. They're always connected to the networks, so they are already dominating in many ways.

当人形机器人真正实用高效时，我们的世界将截然不同，因为那时——你懂的——智能与实体能力的结合真的不会给人类留下多少空间，不是吗？

Our world will look remarkably different when humanoid robots are functional and effective because that's really when, you know, I hear something cry. Like, the combination of intelligence and physical ability is really really doesn't leave much, does it, for us human beings?

确实不多。如今如果你通过互联网获得智能服务，可以雇佣人类为你做事，用比特币支付报酬。也就是说你已能间接拥有实体劳动力，直接控制物理躯体并非革命性突破。

Not much. So today, if you have intelligence through Internet, you can hire humans to do your bidding for you. You can pay them in Bitcoin. So you can have bodies just not directly controlling them. So it's not a huge game changer to add direct control of physical bodies.

关键在于智能。最重要的能力绝对是更强的优化能力、解决问题能力，以及发现人类无法察觉的规律。

Intelligence is where it's at. The important component is definitely higher ability to optimize, to solve problems, to find patterns people cannot see.

那么到2045年2月——也就是二十年后——我想世界会变得更加不同。

And then by 02/1945, I guess the world looks even even more, which is twenty years from now.

所以如果它还存在的话？

So if it's still around?

如果它还存在的话。

If it's still around.

雷·库兹韦尔预测那一年将是奇点到来之时。那是进步速度变得如此之快的一年。这个从事科学与工程工作的AI改进速度如此之快，以至于我们再也跟不上。这就是奇点的定义——超越那个点后我们将无法观察、理解和预测。

Ray Kurzweil predicts that that's the year for the singularity. That's the year where progress becomes so fast. So this AI doing science and engineering work makes improvements so quickly we cannot keep up anymore. That's the definition of singularity, point beyond which we cannot see, understand, predict.

无法观察理解，预测智能本身？还是？

See and understand, predict the intelligence itself? Or?

世界上正在发生什么？技术正在发展。现在如果我有一部iPhone，我可以期待明年出新款，我知道它会有稍好的摄像头。想象现在研发这款手机的过程是自动化的。它每六个月、三个月、一个月、一周、一天、一小时、一分钟、一秒钟都在发生。

What is happening in the world? The technology is being developed. So right now, if I have an iPhone, I can look forward to a new one coming out next year, and I'll understand it has slightly better camera. Imagine now this process of researching and developing this phone is automated. It happens every six months, every three months, every month, week, day, hour, minute, second.

你无法跟上一天内30次iPhone迭代。你不明白它有什么功能，什么是适当的控制措施。它就这样超出你的理解。现在，任何AI研究人员都很难跟上最先进的技术。就在我和你进行这次采访时，一个新模型发布了，我已经不知道最前沿是什么了。

You cannot keep up with 30 iterations of iPhone in one day. You don't understand what capabilities it has, what proper controls are. It just escapes you. Right now, it's hard for any researcher in AI to keep up with the state of the art. While I was doing this interview with you, a new model came out, and I no longer know what the state of the art is.

每天，作为总知识量的百分比，我都在变得更无知。我可能知道得更多，因为我一直在阅读，但作为整体知识的百分比，我们都在变得更无知。然后你把它推到极端值，你对周围世界的认知和理解就归零了。

Every day, as a percentage of total knowledge, I get dumber. I may still know more because I keep reading, but as a percentage of overall knowledge, we're all getting dumber. And then you take it to extreme values, you have zero knowledge, zero understanding of the world around you.

反对这种可能性的论点之一是，当你观察工业革命等其他技术时，人们只是找到了新的工作方式和当时无法想象的新职业。在超级智能的世界里，你如何回应这一点？

Some of the arguments against this eventuality are that when you look at other technologies like the Industrial Revolution, people just found new ways to to work and new careers that we could never have imagined at the time were created. How do you respond to that in a world of superintelligence?

这是范式转变。我们一直有工具，新工具，可以让某些工作更高效地完成。所以与其需要10个工人，你可以用2个工人，另外8个工人必须找到新工作。而且总会有新工作。现在你可以监督那些工人或做些很酷的事。

It's a paradigm shift. We always had tools, new tools, which allowed some job to be done more efficiently. So instead of having 10 workers, you could have two workers, and eight workers had to find a new job. And there was another job. Now you can supervise those workers or do something cool.

如果你正在创造一个元发明，你正在发明智能，你正在发明一个工人，一个代理，那么你可以把这个代理应用到新工作上。没有不能自动化的工作。这是前所未有的。我们之前所有的发明都像是做某事的工具。所以我们发明了火。

If you're creating a meta invention, you're inventing intelligence, you're inventing a worker, an agent, then you can apply that agent to the new job. There is not a job which cannot be automated. That never happened before. All the inventions we previously had were kind of a tool for doing something. So we invented fire.

巨大的游戏规则改变者。但仅此而已。火的出现就是终点。我们发明了轮子，理念相同。

Huge game changer. But that's it. It stops with fire. We invent a wheel. Same idea.

影响深远，但轮子本身并非发明者。而这里，我们正在发明替代人类心智的新发明者，它能进行新的发明。这将是我们需要做出的最后一项发明。届时它将接管一切。科学研究的进程，甚至伦理研究、道德规范，到那时都将实现自动化。

Huge implications, but wheel itself is not an inventor. Here, we're inventing a replacement for human mind, a new inventor capable of doing new inventions. It's the last invention we ever have to make. At that point, it takes over. And the process of doing science, research, even ethics research, morals, all that is automated at that point.

你晚上睡得好吗？

Do you sleep well at night?

非常好。

Really well.

尽管你过去十五到二十年都在研究AI安全，而它突然以五年前无人能预料的方式降临。我说的'降临'，是指现在投入超级智能研发的资金和人才规模，让这件事比我们任何人想象的都更不可避免、更迫在眉睫。

Even though you spent the last, what, fifteen, twenty years of your life working on AI safety, and it's suddenly among us in a in a way that I don't think anyone could have predicted five years ago. When I say among us, I really mean that the amount of funding and talent that is now focused on reaching superintelligence faster has made it feel more inevitable and more soon than any of us could have possibly imagined.

人类天生有种偏见，不愿思考真正糟糕的结果和无法预防的事。我们终将死去，你的孩子会死，父母会死，所有人都会死，但你依然能安睡。

We, as humans, have this built in bias about not thinking about really bad outcomes and things we cannot prevent. So all of us are dying. Your kids are dying. Your parents are dying. Everyone's dying, but you still sleep well.

日子照样过。95岁老人仍在玩游戏打高尔夫，因为我们具备这种能力——不去想最坏的结果，尤其是当我们无力改变结局时。面对AI威胁时，我们动用的正是同样的心理机制。

You still go on with your day. Even 95 year olds are still doing games and playing golf and whatnot. Because we have this ability to not think about the worst outcomes, especially if we cannot actually modify the outcome. So that's the same infrastructure being used for this. Yeah.

人类级别的灭绝性事件可能正在临近。但除非我能有所作为，否则不妨继续享受人生。事实上，知道时日有限反而让人更珍惜生活，分秒不容浪费。

There is humanity level death like event. We're happening to be close to it probably. But unless I can do something about it, I I can just keep enjoying my life. In fact, maybe knowing that you have limited amount of time left gives you more reason to have a better life. You cannot waste any.

我想这就是进化赋予的生存特质——我的祖先若终日忧心忡忡，就没时间生育打猎延续血脉了。

And that's the survival trait of evolution, I guess, because those of my ancestors that spent all their time worrying wouldn't have spent enough time having babies and hunting to survive.

自杀倾向。真正开始思考世界有多残酷的人，通常很快就会选择逃离。

Suicidal ideation. People who really start thinking about how horrible the world is usually escape pretty soon.

你曾合著过一篇论文，分析人们反对AI安全重要性的主要论点。其中一个论点是，目前还有其他更重要的事情，比如世界大战、核遏制或其他问题。

One of the you co authored this paper analyzing the key arguments people make against the importance of AI safety. And one of the arguments in there is that there's other things that are of bigger importance right now. It might be world wars. It could be nuclear containment. It could be other things.

政府和像我这样的播客应该讨论更重要的议题。你对这个论点有何反驳？

There's other things that the governments and podcasters like me should be talking about that are more important. What's your rebuttal to that argument?

超级智能是一个元解决方案。如果我们正确掌握超级智能，它将帮助我们应对气候变化、战争，解决所有其他生存风险。如果处理不当，它将主宰一切。

So superintelligence is a meta solution. If we get superintelligence right, it will help us with climate change. It will help us with wars. It can solve all the other existential risks. If we don't get it right, it dominates.

如果气候变化需要一百年才能把我们煮熟，而超级智能五年内就能杀死所有人，那我就不必担心气候变化了。所以无论如何，要么它为我解决问题，要么这就不是问题。

If climate change will take a hundred years to boil us alive and superintelligence kills everyone in five, I don't have to worry about climate change. So either way, either it solves it for me or it's not an issue.

所以你认为这是当前最值得投入的工作？

So you think it's the most important thing to be working on?

毫无疑问，没有什么比正确处理这件事更重要。我知道每个人都这么说。你上任何课程，比如英语教授的课，他都会告诉你这是你上过最重要的课。但你能看出这件事在元层面的不同。

Without question, there is nothing more important than getting this right. And I know everyone says it. You take any class, but you take English professor's class, and he tells you, this is the most important class you'll ever take. But you can see the meta level differences with this one.

那篇论文中另一个论点是我们可以掌控一切，危险并非来自AI。这个论点声称AI只是工具，人类才是真正的危险源，我们总能通过断电保持控制。难道不能直接拔插头吗？每次节目讨论AI时都会听到这种说法。

Another argument in that paper is that we all be in control and that the danger is not AI. This particular argument asserts that AI is just a tool. Humans are the real actors that present danger, and we can always main maintain control by simply turning it off. Can't we just pull the plug out? I see that every time we have a conversation on the show about AI.

总有人说：不能直接断电吗？

Someone says, can't we just unplug it?

是啊，我每期播客都会收到这种评论。我总想联系那个人说：这太天才了，我从未想过。我们要合写论文拿诺贝尔奖——就这么干吧。

Yeah. I get those comments on every podcast I make, and I always wanna, like, get in touch with a guy and say, this is brilliant. I never thought of it. We're gonna write a paper together and get a Nobel Prize for it. This is, like, let's do it.

因为这太荒谬了。你能关掉病毒吗？电脑中毒了你不喜欢——关掉它啊。比特币呢？

Because it's so silly. Like, can you turn off a virus? You have a computer virus, you don't like it. Turn it off. How about Bitcoin?

关掉比特币网络啊。你试试看。我等着。这太可笑了。那些是分布式系统。

Turn off Bitcoin network. Go ahead. I'll wait. This is silly. Those are distributed systems.

你关不掉它们的。更关键的是，它们比你聪明。它们做了多重备份。它们预判了你的行动。在你关闭它们之前，它们会先让你下线。

You cannot turn them off. And on top of it, they're smarter than you. They made multiple backups. They predicted what you're going to do. They will turn you off before you can turn them off.

所谓人类能保持控制权的想法，仅适用于超级智能出现前的阶段——基本上就是现在。如今，掌握AI工具的人类确实危险，可能是黑客或恶意行为者。但等到超级智能更聪明、占据主导时，人类就不再是这个等式里的重要部分了。

The idea that we will be in control applies only to pre superintelligence levels, basically what we have today. Today, humans with AI tools are dangerous. They can be hackers, malevolent actors. Absolutely. But the moment superintelligence becomes smarter, dominates, they no longer the important part of that equation.

我担忧的是那个更高级的智能体，而不是可能附加恶意程序但最终仍无法掌控它的人类。

It is the higher intelligence I'm concerned about, not the human who may add additional malevolent payload, but at the end still doesn't control it.

我差点要被你论文里的下一个论点说服了——它本质上在说：认命吧，这是不可避免的。既然毫无希望，反抗也没有意义。我们或许该放弃挣扎，虔诚相信事情会自行解决，因为你说的每件事听起来都无可避免。中国在搞这个，普京肯定也有秘密部门在研究。

It is tempting to follow your the next argument that I saw in that paper, which basically says, listen. This is inevitable. So there's no point fighting against it because there's really no hope here. So we should probably give up even trying and be faithful that it'll work itself out because everything you said sounds really inevitable. And if with China working on it, I'm sure Putin's got some secret division.

伊朗肯定也在捣鼓些零碎。每个欧洲国家都想在AI领域领先。美国正带头冲锋。所以这是必然的。我们或许该保持信仰并祈祷。

I'm sure Iran are doing some bits and pieces. Every European country's trying to get ahead of AI. The United States is leading the way. So it's it's inevitable. So we probably should just have faith and pray.

祈祷总是好的，但动机很重要。如果你观察驱动这些人的因素——没错，金钱很关键。这个领域有巨额利益，所以人人都想分一杯羹。但如果他们真正理解这个论点，就会明白自己终将灭亡——再多钱对你也没用。

Oh, praying is always good, but incentives matter. If you are looking at what drives these people so, yes, money is important. So there is a lot of money in that space, and so everyone's trying to be there and develop this technology. But if they truly understand the argument, they understand that you will be dead. No amount of money will be useful to you.

这时动机会转变。他们会想活下去。很多从业者是年轻人、富豪，人生还长。我认为他们应该停止开发超级智能，专注于用狭义AI工具解决具体问题。

Then incentive switch. They would wanna not be dead. A lot of them are young people, rich people. They have their whole lives ahead of them. I think they would be better off not building advanced superintelligence, concentrating on narrow AI tools for solving specific problems.

比如我的公司专攻乳腺癌治疗。仅此而已。我们赚几十亿，皆大欢喜。

Okay. My company cures breast cancer. That's all. We make billions of dollars. Everyone's happy.

各方受益，双赢局面。现阶段控制权还在我们手中。只要还没结束就还有转机。我们可以选择不造通用超级智能。

Everyone benefits. It's a win. We are still in control today. It's not over until it's over. We can decide not to build general superintelligences.

我是说，美国或许能为此激发足够的热情。但如果美国不发展通用超级智能，那么中国将占据巨大优势。对吧？

I mean, The United States might be able to conjure up enough enthusiasm for that. But if The United States doesn't build general superintelligences, then China are gonna have the big advantage. Right?

所以在当前阶段，AI技术更先进的国家军事实力就更强。毫无疑问。我们从现有冲突中就能看出这点。但一旦转向不受控的超级智能，谁先造出来都无所谓——无论是我们还是他们。如果他们理解这个论点，他们也不会去研发。

So right now, at those levels, whoever has more advanced AI has more advanced military. No question. We see it with existing conflicts. But the moment you switch to superintelligence uncontrolled superintelligence, it doesn't matter who builds it, us or them. And if they understand this argument, they also would not build it.

这就像是双向的相互保证毁灭。

It's a mutually assured destruction on both ends.

这项技术是否不同于核武器？制造核武器需要巨额投资，比如铀浓缩，可能需要数十亿美元才能造出一枚核弹。但超级智能技术似乎成本低得多，或者说终将变得廉价。我在想，是否有可能某个初创公司在几年内，不需要数十亿美元的计算资源或电力就能造出超级智能？

Is this technology different than, say, nuclear weapons, which require a huge amount of investment and you have to, like, enrich the uranium and you need billions of dollars potentially to even build a nuclear weapon? But it feels like this technology is much cheaper to get to superintelligence potentially, or at least it will become cheaper. I wonder if it's possible that some some guy, some startup is gonna be able to build superintelligence in, you know, a couple of years without the need of bit you know, billions of dollars of compute and or or electricity power?

说得好。每年训练大型模型的成本都在降低。如果现在需要一万亿美元来打造超级智能，明年可能只要一千亿，最终个人用笔记本电脑就能实现。但没人愿意等四年让它变得廉价，所以现在才有大量资金涌入。

That's a great point. So every year, it becomes cheaper and cheaper to train sufficiently large model. If today, it would take a trillion dollars to build superintelligence, next year it could be a 100,000,000,000 and so on, at some point, a guy in a laptop could do it. But you don't wanna wait four years to make it affordable. So that's why so much money is pouring in.

有人想在今年就实现突破，赢得类似莱特币级别的奖项。从这个角度看，这两个都是曼哈顿级别的昂贵项目。

Somebody wants to get there this year and lucky in all the winnings, Litecoin level award. So in that regard, they're both very expensive projects, like Manhattan level projects.

曼哈顿计划是指核弹项目吗？

Which was the nuclear bomb projects?

对。两项技术的区别在于：核武器始终是工具，需要独裁者或国家决定是否使用；而超级智能不是工具，它是自主决策的智能体，不受任何人控制。

Right. The difference between the two technologies is that nuclear weapons are still tools. Some dictators, some countries, someone has to decide to use them, deploy them, whereas superintelligence is not a is not a tool. It's an agent. It makes its own decisions, and no one is controlling it.

我无法除掉某个独裁者就确保超级智能的安全。这才是根本区别。

I cannot take out this dictator, and now superintelligence is safe. So that's a fundamental difference to me.

但既然你说成本会越来越低——就像摩尔定律描述的技术成本下降趋势——那么未来是否会出现这种情况：某个人仅用笔记本电脑就能在无监管、无团队的情况下创造出超级智能？

But if you're saying that it is gonna get incrementally cheaper, like, I think it's Moore's Law, isn't it, that technology gets cheaper? It is. Then there is a future where some guy on his laptop is gonna be able to create superintelligence without oversight or regulation or employees, etcetera.

是的。这就是为什么很多人建议我们需要建立一个类似‘监控星球’的体系，实时监控每个人的行为并试图阻止恶意行为。但我认为可行吗？不可行。当技术变得足够廉价和普及时，这种事必然会发生。

Yeah. That's why a lot of people suggesting we need to build something like surveillance planet where you are monitoring who's doing what, and you're trying to prevent people from doing it. Do I think it's feasible? No. At some point, it becomes so affordable and so trivial that it just will happen.

但现阶段我们想争取更多时间。我们不希望五年内发生这种事，而是希望推迟到五十年后。

But at this point, we're trying to get more time. We don't want it to happen in five years. We want it to happen in fifty years.

听起来并不乐观啊。所以

I mean, that's not very hopeful. So

这取决于你的年龄。

you Depends on how old you are.

确实取决于年龄。如果你认为未来人类无需当今这些资源就能创造超级智能，那这就只是时间问题了。

Depends on how old you are. I mean, if you're saying that you believe in the future people will be able to make superintelligence without the resources that are required today, then it is just a matter of time.

没错。其他技术也是如此。比如合成生物学正在突飞猛进——现在一个生物学学士就可能制造新病毒。这类技术的门槛会越来越低，我们正面临一个困境：很难确保哪项技术突破不会成为终结者。

Yeah. But so will be true for many other technologies. We're getting much better in synthetic biology, where today someone with a bachelor's degree in biology can probably create a new virus. This will also become cheaper, other technologies like that. So we are approaching a point where it's very difficult to make sure no technological breakthrough is the last one.

本质上，人类正在多维度降低毁灭世界的资源与智力门槛。五百年前最残暴的统治者倾尽资源也只能杀害数百万人，而如今核武器能反复毁灭地球，合成生物学在新冠疫情中已显现威力——

So essentially, in many directions, we have this pattern of making it easier in terms of resources, in terms of intelligence to destroy the world. If you look at, I don't know, five hundred years ago, the worst dictator with all the resources could kill a couple million people. He couldn't destroy the world. Now we know nuclear weapons, we can blow up the whole planet multiple times over. Synthetic biology, we saw with COVID.

人们可以轻易制造影响数十亿人的组合病毒。所有这些行为的实施难度都在持续降低。

You can very easily create a combination virus which impacts billions of people. And all of those things becoming easier to do.

短期来看，你认为人类灭绝是真实存在的风险。在所有可能的灭绝路径中，你认为最可能的是哪种？比如AI工具部署前的设计失误，或是部署后遭黑客劫持恶意篡改？在这些选项中，哪个概率最高？

In the near term, you talk about extinction being a real risk, human extinction being a real risk. Of all the the pathways to human extinction that you think are most likely, what what is the leading pathway? Because I know you talk about there being some issue predeployment of these AI tools, like, you know, someone makes a mistake when they're designing a model or other issues post deployment. When I say post deployment, I mean, once a tragedy or something, an agent's released into the world and someone hacking into it and changing it and reprogram reprogramming it to be malicious. Of all these potential paths to human extinction, which one do you think is the highest probability?

我只能预测自己理解的范畴。在超级智能出现前，就可能有人制造出高级生物工具——比如新型病毒导致全球性感染。这个路径我能清晰预见并理解其机制。

So I can only talk about the ones I can predict myself. So I can predict even before we get to superintelligence, someone will create a very advanced biological tool, tool, create a novel virus, and that virus gets everyone or most everyone. I can envision it. I can understand the pathway. I can say that.

那么具体来说，就是利用AI制造病毒然后释放它。是的。这是蓄意的吗？还是

So just to zoom in on that then, that would be using an AI to make a virus and then releasing it. Yeah. And would that be intentional? Or

世界上有很多心理变态者、恐怖分子和末日教派。历史上我们看到，他们总是试图尽可能多地杀人。虽然通常失败，但也能造成数十万人死亡。如果他们掌握能杀死数百万甚至数十亿人的技术，他们会欣然为之。

There is a lot of psychopaths, lot of terrorists, a lot of doomsday cults. We've seen historically, again, they try to kill as many people as they can. They usually fail. They kill hundreds of thousands. But if they get technology to kill millions of billions, they would do that gladly.

我想强调的是，我能想到什么并不重要。我并不是你们要防范的恶意行为者。关键在于超级智能可能创造出全新的实施方式。就像你举的狗的例子——你的狗无法理解你能消灭它的所有方式。

The point I'm trying to emphasize is that it doesn't matter what I can come up with. I am not a malevolent actor you're trying to defeat here. It's the superintelligence which can come up with completely novel ways of doing it. Again, you brought up example of your dog. Your dog cannot understand all the ways you can take it out.

它或许能想到用撕咬致死之类的方式，但仅此而已。而你拥有无限的资源。所以如果我问你的狗你会怎么消灭它，它给不出有意义的答案。它只能想到撕咬。而这就是我们的认知局限。

It can maybe think it'll bite it to death or something, but that's all. Whereas you have infinite supply of resources. So if I asked your dog exactly how you're going to take it out, it would not give you a meaningful answer. It can talk about biting. And this is what we know.

我们了解病毒，经历过病毒，可以讨论病毒。但一个能进行全新物理研究的AI系统能构想出什么，这超出了我的理解范围。

We know viruses. We experience viruses. We can talk about them. But what an AI system capable of doing novel physics research can come up with is beyond me.

我认为大多数人没意识到的是，我们对这些AI的实际运作机制知之甚少。人们可能以为既然计算机是我们造的，就理应了解其工作原理——知道它按代码执行指令。但根据你的著作描述，这就像个黑箱。以ChatGPT为例，你是说连创造者都不清楚其内部究竟如何运作？

One of the things that I think most people don't understand is how little we understand about how these AIs are actually working. Because one would assume, you know, with computers, we kind of understand how a computer works. We we know that it's doing this and then this, and it's running on code. But from reading your work, you described it as being a black box. We act so in the context of something like ChatGPT or an AI we know, you're telling me that the people that have built that tool don't actually know what's going on inside there.

完全正确。即便是开发者也需要通过实验来了解其产品的能力。他们用海量数据（比如全网文本）训练模型，用大量计算机学习文本模式，然后才开始测试这个模型。

That's exactly right. So even people making those systems have to run experiments on their product to learn what it's capable of. So they train it by giving it all of data, let's say all of Internet text. They run it on a lot of computers to learn patterns in that text. And then they start experimenting with that model.

比如问：你会说法语吗？能做数学题吗？现在在骗我吗？可能训练需要一年，再花六个月才能掌握其基本能力边界和安全阈值。但即便是旧模型，我们仍不断发现新能力。

Oh, do you speak French? Or can you do mathematics? Or are you lying to me now? And so maybe it takes a year to train it and then six months to get some fundamentals about what it's capable of, some safety overhead. But we still discover new capabilities in old models.

换种提问方式，它就显得更聪明。这已不再是前五十年的工程学范式——由知识工程师编写专家系统AI来执行特定任务。现在这是门科学，我们在培育这个人造物，就像栽培外星植物。

If you ask the question in a different way, it becomes smarter. So it's no longer engineering how it was the first fifty years where someone was a knowledge engineer programming an expert system AI to do specific things. It's a science. We are creating this artifact, growing it. It's like an alien plant.

然后观察它的行为。就像对植物的认知一样，我们对生物学没有百分百准确的了解，对这个领域也不完全掌握。我们只知道某些规律：比如增加算力通常会让它更聪明。

And then we study to see what it's doing. And just like with plants, we don't have 100% accurate knowledge of biology. We don't have full knowledge here. We kind of know some patterns. We know, okay, if we add more compute, it gets smarter most of the time.

但没人能准确告诉你给定一组输入后会产生什么结果。

But nobody can tell you precisely what the outcome is going to be given a set of inputs.

我见过太多企业家把销售视为表现问题，而实际上往往是可见性问题。因为当你无法看清销售渠道中的动态、每段对话的进展阶段、哪些停滞不前、哪些在推进时，你根本无法改进任何环节，也无法达成交易。我们的赞助商Pipedrive是中小企业的头号CRM工具——不仅是联系人列表，更是一个完整展示端到端销售流程的系统，实时显示所有动态、滞后环节及后续所需步骤。

I've watched so many entrepreneurs treat sales like a performance problem when it's often down to visibility. Because when you can't see what's happening in your pipeline, what stage each conversation is at, what's stalled, what's moving, you can't improve anything. And you can't close the deal. Our sponsor, Pipedrive, is the number one CRM tool for small to medium businesses. Not just a contact list, but an actual system that shows your entire sales process end to end, everything that's live, what's lagging, and the steps you need to take next.

所有团队都能更智能高效地运作。使用Pipedrive的团队平均成交率是未使用团队的三倍。这是首个由销售人员为销售人员打造的CRM，全球超过10万家企业信赖它，包括我那些对它爱不释手的团队。立即访问pipedrive.com/ceo试用，无需支付即可在几分钟内完成部署。通过此链接注册还可获得30天免费试用。

All of your teams can move smarter and faster. Teams using Pipedrive are on average closing three times more deals than those that aren't. It's the first CRM made by salespeople for salespeople that over 100,000 companies around the world rely on, including my team who absolutely love it. Give Pipedrive a try today by visiting pipedrive.com/ceo, and you can get up and running in a couple of minutes with no payment needed. And if you use this link, you'll get a thirty day free trial.

你对OpenAI和Sam Altman的所作所为怎么看？显然你是...等等其中一位联合创始人是叫Ilya对吧？Ilya。对。

What do you make of OpenAI and Sam Altman and what they're doing? And obviously, you're that one of the cofounders was it it Ilya, Jack? Ilya. Ilya. Yeah.

Ilya离职后创办了家新公司叫

Ilya left, and he started a a new company called

超级智能安全。超级智能安全。他觉得AI安全挑战性不够，直接跳去攻克最难的课题了。

Superintelligence safety. Superintelligence safety. AI safety wasn't challenging enough. He decided to just jump right to the hard problem.

作为旁观者，看到人们离开OpenAI去创办超级智能安全公司时，你如何解读这种局面？

As an onlooker, when you see that people are leaving OpenAI to to start super intelligent safety companies, what was your read on that situation?

许多与Sam共事过的人表示，他可能不是最直率坦诚的人，他们对其安全理念存有疑虑。这是部分原因——他们想要更多控制权，更聚焦安全。但还有个现象：任何离开那家公司创业的人，刚起步就能获得200亿美元的估值。

So a lot of people who worked with Sam said that maybe he's not the most direct person in terms of being honest with them, and they had concerns about his views on safety. That's part of it. So they wanted more control. They wanted more concentration on safety. But also, seems that anyone who leaves that company and starts a new one gets a $20,000,000,000 valuation just for having it started.

没有产品，没有客户，但想赚几十亿？照做就行。所以这对有能力者来说简直是难以抗拒的非理性诱惑，人员流失自然严重。私下接触时，他非常友善、聪明，堪称完美的公众形象——看他参议院听证会的表现，对议员们言辞得体；看他与投资者交流，传递的信息无懈可击。

You don't have a product, you don't have customers, but if you wanna make many billions of dollars, just do that. So it seems like a very irrational thing to do for anyone who can. So I'm not surprised that there is a lot of attrition. Meeting him in person, he's super nice, very smart, absolutely perfect public interface. You see him testify in the senate.

但了解他私下面貌的人的评价是：他或许不该掌控如此影响力的项目。为什么？

He says the right thing to the senators. You see him talk to the investors. They get the right message. But if you look at what people who know him personally are saying, it's probably not the right person to be controlling a project of that impact. Why?

他把安全放在第二位。

He puts safety second.

仅次于什么？

Second to?

赢得这场通往超级智能的竞赛，成为那个创造神明以掌控宇宙光锥的人，他更恶劣。

Winning this race to superintelligence, being the guy who created God against controlling the light cone of the universe, he's worse.

你是否怀疑他的驱动力是成为有影响力、做出非凡成就的传奇人物，而非考虑对社会可能造成的后果？有趣的是他的另一家初创公司Worldcoin，本质上是一个创建全民基本收入的平台，即在人们失去工作的世界里提供收入的平台。所以一方面他在创建AI公司，另一方面又在创建为人类失业做准备的机构。

Do you suspect that's what he's driven by, is by the the legacy of being an impactful person that did a remarkable thing versus the consequence that that might have on for society. Because it's interesting that he's his other startup is Worldcoin, which is bay basically a platform to create universal basic income, I e, a platform to give us income in a world where people don't have jobs anymore. So on one hand, you're creating an AI company. On the other hand, you're creating a company that is preparing for people not to have employment.

它还有其他特性——追踪每个人的生物特征数据，掌控全球经济命脉。他们持有大量Worldcoin份额，我认为这相当于是与世界统治体系的深度融合。

It also has other properties. It keeps track of everyone's biometrics. It, keeps you in charge of a world's economy, world's wealth. They're retaining a large portion of world coins. So I I think it's kinda very reasonable part to integrate with world dominance.

如果你拥有超级智能系统并掌控货币，你就稳操胜券了。

If you have a superintelligent system and you control money, you're doing well.

为什么会有人想要统治世界？

Why would someone want world dominance?

人们的野心层次不同——当你是个坐拥亿万财富和名声的年轻人时，就会开始寻求更宏大的目标。有人想登陆火星，有人则想掌控宇宙光锥。

People have different levels of ambition than you are a very young person with billions of dollars, fame. You start looking for more ambitious projects. Some people want to go to Mars. Others want to control light cone of the universe.

你刚说什么？宇宙光锥？

What what did you say? Light cone of the universe?

光锥。宇宙中从某点出发光能抵达的所有区域，意味着任何可触及之物你都想纳入掌控。

Light cone. Light cone. Every part of the universe light can reach from this point, meaning anything accessible you wanna grab and bring into your control.

你认为萨姆·奥特曼想要控制宇宙的每一部分吗？

You think Sam Altman wants to control every part of the universe?

我怀疑他可能确实如此。是的。但这并不意味着他不希望这项技术带来让全人类幸福的积极副作用。快乐的民众更易于控制。

I suspect he might. Yes. It doesn't mean he doesn't want a side effect of it being a very beneficial technology which makes all the humans happy. Happy humans are good for control.

如果要你猜测二月的世界会是什么样子，你会怎么猜？

If you had to guess what the world looks like in February, if you had to guess?

要么人类不复存在，要么世界会变得像我们这样的人完全无法理解。必居其一。

It's either free of human existence or it's completely not comprehensible to someone like us. It's one of those extremes.

所以要么人类灭绝。

So there's either no humans.

基本上就是世界毁灭，或是变得面目全非，以至于我无法想象那些预测场景。

It's basically the world is destroyed, or it's so different that I cannot envision those predictions.

现阶段还能采取什么措施来扭转局面，确保更积极的未来？我们还能有所作为，还是为时已晚？

What can be done to turn this ship to a more certain positive outcome at this point? Is is there still things that we can do, or is it too late?

我相信个人利益驱动。如果人们意识到做这件事对他们自身非常不利，他们就不会去做。所以我们的任务是说服这个领域所有掌权者——无论是开发技术还是为相关公司工作的人——他们正在做对自己极其不利的事。别说什么你们未经许可就拿80亿人做实验，最终结果绝不会让你们满意。

So I believe in personal self interest. If people realize that doing this thing is really bad for them personally, they will not do it. So our job is to convince everyone with any power in this space, creating this technology, working for those companies, they are doing something very bad for them. Not just forget our 8,000,000,000 people you're experimenting on with no permission, no consent. You will not be happy with the outcome.

如果能让大家明白这是普遍共识而非我一家之言——杰夫·辛顿（机器学习领域奠基人、诺贝尔奖得主）也持相同观点，本吉等数十位顶尖学者也是。我们曾发布过由数千名学者和计算机科学家联署的危险AI声明。

If we can get everyone to understand that's a default and it's not just me saying it. You had Jeff Hinton, Nobel Prize winner, founder of a whole machine learning space. He says the same thing. Benji or dozens of others, top scholars. We had a statement about dangerous AI signed by thousands of scholars, computer scientists.

这基本是当前学界共识，我们需要将其普及化。对此不该存在异议。这样我们才能真正做出关于技术发展的明智决策。虽不能保证人类长远安全，但至少能避免以最快速度冲向最糟糕的结局。

This is basically what we think right now, and we need to make it universal. No one should disagree with this. And then we may actually make good decisions about what technology to build. It doesn't guarantee long term safety for humanity, but it means we're not trying to get there as soon as possible to the worst possible outcome.

你是否认为这甚至是有可能的？

And do you are you hopeful that that's even possible?

我想尝试。我们别无选择，只能尝试。

I wanna try. We have no choice but to try.

那么需要发生什么？谁需要采取行动？是政府立法吗？还是

And what would need to happen? And who would need to act? What is it government legislation? Is it

不幸的是，我认为将其定为非法是不够的。有不同的司法管辖区，还有漏洞。如果有人这么做了，你打算怎么办？你会因为他们毁灭人类而罚款吗？

Unfortunately, I don't think making it illegal is sufficient. There are different jurisdictions. There is, you know, loopholes. And what are you gonna do if if somebody does it? Are you gonna fine them for destroying humanity?

比如，对此处以非常高的罚款？你打算怎么做？这是无法执行的。如果他们真的创造了它，现在就是超级智能在掌控。所以我们现有的司法体系没有影响力。

Like, very steep fines for it? Like, what are you gonna do? It's not enforceable. If they do create it, now the superintelligence is in charge. So the judicial system we have is not impactful.

而我们所有的惩罚措施都是为惩罚人类设计的。监狱、死刑不适用于AI。

And all the punishments we have are designed for punishing humans. Prisons, capital punishment doesn't apply to AI.

我遇到的另一个问题是，当我进行这些对话时，我从未觉得离开时抱有希望，希望事情会好转。我的意思是，我从未觉得离开时有一系列明确的行动可以纠正这里可能发生的事情。那么我该做什么？坐在家里听这个的人该做什么？

The other problem I have is when I have these conversations, I never feel like I walk away with, I hope that something's gonna go well. And what I mean by that is I never feel like I walk away with clear some kind of clear set of actions that can course correct what might happen here. So what should what should I do? What should the person sat at home listening to this do?

你和很多正在构建这项技术的人交谈。嗯。请他们确切解释一些他们声称不可能的事情，他们是如何解决的，或者在他们达到目标之前将如何解决。

You you talk to a lot of people who are building this technology. Mhmm. Ask them precisely to explain some of those things they claim to be impossible, how they solved it or going to solve it before they get to where they're going.

你知道吗，我不认为Sam Altman想和我说话。

Do know, I don't think Sam Altman wants to talk to me.

我不知道。他似乎上了很多播客。也许他会。

I don't know. He seems to go on a lot of podcasts. Maybe he does.

他想上我的节目。我很好奇为什么。我很好奇为什么。我很想和他聊聊，但我不认为他想——我不认为他希望我采访他。

He wants to go on mine. I I wonder why that is. I wonder why that is. I'd love to speak to him, but I don't I don't think he wants to I don't think he wants me to interview him.

可以发起公开挑战。也许金钱不是激励因素，但只要能吸引这类人，谁能证明超级智能可以被控制并确保安全，谁就能获得奖金。他们可以上你的节目证明自己的观点。任何人都行。如果几年后无人认领奖金甚至无人接受挑战，或许说明我们根本没有解决方案。

Have an open challenge. Maybe money is not the incentive, but whatever attracts people like that, whoever can convince you that it's possible to control and make safe superintelligence gets the price. They come on your show and prove their case. Anyone. If no one claims the price or even accepts the challenge after a few years, maybe we don't have anyone with solutions.

现在有市值数十亿的公司正在研发安全的超级智能，但我们尚未看到他们的成果。

We have companies valued, again, at billions and billions of dollars working on safe superintelligence. We haven't seen their output yet.

是的。我也想和Ilya聊聊，因为我知道他正在研究安全的超级智能。

Yeah. I'd like to speak to Ilya as well because I know he's he's working on safe superintelligence.

注意一个规律。回顾AI安全组织或公司内部部门的历史，它们通常开局雄心勃勃，最终失败解散。比如OpenAI曾有个超级智能对齐团队，成立当天宣称四年内解决问题，结果半年后团队就解散了。

Like Notice a pattern too. If you look at history of AI safety organizations or departments within companies, They usually start well, very ambitious, and then they fail and disappear. So OpenAI had superintelligence alignment team. The day they announced it, I think they said they're gonna solve it in four years. But half a year later, they canceled the team.

类似案例不胜枚举。要为持续进化、不断调整、与人类互动的超级智能建立完美永久的安全机制？这永远不可能实现。计算机科学领域的难题与P完全问题、不可能问题存在本质区别。我认为对超级智能的永久控制就属于不可能问题。

And there is dozens of similar examples. Creating a perfect safety for superintelligence, perpetual safety as it keeps improving, modifying, interacting with people, you're never gonna get there. It's impossible. There is a big difference between difficult problems in computer science and p complete problems and impossible problems. And I think control, indefinite control of superintelligence is such a problem.

既然不可能，那尝试还有什么意义？

So what's the point trying then if it's impossible?

我正是要证明其不可能性。一旦确认某事不可为，就能减少人们浪费时间空谈能力、骗取资金。现在总有人说'给我十亿，两年内搞定'——我认为他们根本做不到。

Well, I'm trying to prove that it is specifically that. Once we establish something is impossible, fewer people will waste their time claiming they can do it and find looking for money. So many people go and give me a billion dollars in two years, and I'll solve it for you. Well, I don't think you will.

但人们不会停止追求。如果没人尝试确保安全，而追求者越来越多，那悲剧就不可避免了。

But people aren't gonna stop striving towards it. So if there's no attempts to make it safe and there's more people increasingly striving towards it, then it's inevitable.

但这会改变我们的行动方向。如果明知无法确保安全，那么'尽快造出来'的直接路径就成了自杀任务。希望更少人会走这条路，转而探索其他方向。毕竟，我是个科学家。

But it changes what we do. If we know that it's impossible to make it right, to make it safe, then this direct path of just build it as soon as you can becomes suicide mission. Hopefully, fewer people will pursue that. They may go in other directions. Like, again, I'm a scientist.

我是一名工程师。我热爱人工智能。我热爱科技。我时刻都在使用它。打造有用的工具。

I'm an engineer. I love AI. I love technology. I use it all the time. Build useful tools.

别再开发智能体了。要打造的是狭窄领域的超级智能，而非通用型。我不是说你们不该赚几十亿美元。我也爱钱。但别害死所有人，包括你自己。

Stop building agents. Build narrow superintelligence, not a general one. I'm not saying you shouldn't make billions of dollars. I love billions of dollars. But don't kill everyone, yourself included.

可他们不认为会发展到那一步。

They don't think they're going to, though.

那就给出科学解释。我听多了直觉之类的说辞，听多了'以后会解决'的托词。用具体的科学术语告诉我。发表经过同行评议的论文，阐明你们将如何控制超级智能。

Then tell us why. I hear things about intuition. I hear things about we'll solve it later. Tell me specifically in scientific terms. Publish a peer reviewed paper explaining how you're going to control superintelligence.

是啊。这很荒谬。哪怕人类灭绝的可能性只有1%，为此费心都很荒谬。就像如果有人告诉我，上车后有1%的几率会丧命...

Yeah. It's strange. It's strange to it's strange to even bother if there was even a 1% chance of human extinction. It's strange to do something. Like, if there was a 1% someone told me there was a 1% chance that if I got in a car, I might not I might not be alive.

我绝不会上车。如果说喝下这杯液体有1%的致死率，即便存活能获得十亿美元——99%几率得巨款，1%几率死亡——我也不会喝。我不会冒险。

I would not get in the car. If you told me there was a 1% chance that if I drank whatever liquid is in this cup right now, I might die. I would not drink the liquid. Even if there was a billion dollars if I survived, so the ninety nine percent chance that I get a billion dollars, the 1% is I die, I wouldn't drink it. I wouldn't take the chance.

情况更糟。不只是你会死。是全人类灭亡。

It's worse than that. Not just you die. Everyone dies.

对。确实。

Yeah. Yeah.

现在问题在于：我们是否允许你以任何概率喝下它？这该由我们决定。你无权替我们做选择。要让人类受试者知情同意，必须确保他们理解同意内容。如果这些系统既不可解释又不可预测，何谈知情同意？

Now would we let you drink it at any odds? That's for us to decide. You don't get to make that choice for us. To get consent from human subjects, you need them to comprehend what they are consenting to. If those systems are unexplainable, unpredictable, how can they consent?

他们根本不知道自己同意了什么。因此从定义上就不可能获得有效同意。所以这个实验永远无法符合伦理。本质上，他们正在进行违背伦理的人体实验。

They don't know what they are consenting to. So it's impossible to get consent by definition. So this experiment can never be run ethically. By definition, they are doing unethical experimentation on human subjects.

你认为人们应该抗议吗？

Do you think people should be protesting?

确实有人在抗议。有Stop AI组织，有Pause AI组织。他们每周每月都会封锁OpenAI办公室，进行不少行动，还在招募新人。

There are people protesting. There is Stop AI. There is Pause AI. They block offices of OpenAI. They do it weekly, monthly, quite a few actions, and they're recruiting new people.

你觉得应该有更多人参与抗议吗？你认为这是有效的解决方案吗？

You think more people should be protesting? Do think that's an effective solution?

如果能扩大到全民参与的规模，确实会产生影响。但我不确定他们能否从当前人数扩展到那种程度。不过我支持所有人以和平合法的方式尝试各种可能。

If you can get it to a large enough scale to where majority of population is participating, it would impactful. I don't know if they can scale from current numbers to that, but I support everyone trying everything peacefully and legally.

对于在家收听节目的听众，他们应该做些什么？因为没人愿意感到无能为力，我们都不想有这种感受。

And for the for the person listening at home, what should they what should they be doing? What what what because they they don't wanna feel powerless. None none of us wanna feel powerless.

这取决于我们讨论的时间尺度。比如今年你孩子要上大学，该选什么专业？该不该上大学？该不该换工作？

So it depends on what scale we're asking about timescale. I was saying, like, this year, your kid goes to college, what major to pick? Should they go to college at all? Yeah. Should you switch jobs?

该不该进入某些行业？这些近期问题我们可以回答。但五年后面对AI发展，普通人能做的其实很有限。

Should you go into certain industries? Those questions we can answer. We can talk about immediate future. What should you do in five years with this being created? For an average person, not much.

就像普通人无法影响第三次世界大战或核危机这类事一样。如果想参与这场运动，可以加入Pause.ai或Stop.ai这些正试图通过民主力量影响决策的组织。

Just like they can't influence World War three, nuclear, holocaust, anything like that. It's not something anyone's gonna ask them about. Today, if you wanna be a part of this movement, yeah, join Pause dot ai. Join Stop dot ai. Those organizations currently trying to build up momentum to bring democratic powers to influence those individuals.

所以短期内能做的有限？我在想近期是否有其他策略。比如是否该重新考虑家庭规划？你有三个孩子对吧？

So in the near term, not a huge amount. Was wondering if there there are any interesting strategies in the near term. Like, should I be thinking differently about my family, about I mean, you've got kids. Right? You've got three kids?

据我所知是有的。

That I know about. Yeah.

三个孩子。你是否在考虑眼前这个世界里的育儿问题？你在想该如何与他们交流、给予什么建议、他们应该学习些什么？

Three kids. Are you thinking about parenting in this world that you see around the corner? How are thinking about what to say to them, the advice to give them, what they should be learning?

在这个领域之外有个普遍建议——把每一天当作最后一天来活。无论剩余三年还是三十年，这都是金玉良言。尽量别长时间做厌恶的事，去尝试有趣的事物。

So there is general advice outside of this domain that you should live your everyday as if it's your last. It's a good advice no matter what. If you have three years left or thirty years left, you live your best life. So try to not do things you hate for too long. Do interesting things.

做有影响力的事。如果能在帮助他人的同时达成这些就更好了。

Do impactful things. If you can do all that while helping people do that.

模拟理论是个有趣的相邻课题。随着计算机加速进化变得更智能，我们通过AI能做到超乎想象的事——比如用虚拟现实创造各种世界。我记得谷歌最近发布了...叫什么来着？那种AI生成的世界。

Simulation theory is a interesting sort of adjacent subject here because as computers begin to accelerate and get more intelligent and we're able to, you know, do things with AI that we can never have imagined in terms of, like, imagine the worlds that we could create with virtual reality. I think it was Google that recently released what was it called? Like the AI worlds.

嗯，你拍张照片就能生成完整世界。

Mhmm. You take a picture and it generates a whole world.

对，还能在世界里移动。我会把演示放给大家看。谷歌这项技术只需简单指令就能创建可探索的3D世界，而且具有记忆功能——比如你在墙上涂鸦...

Yeah. And you can move through the world. I'll put it on the screen for people to see, but Google have released this technology, which allows you, I think, with a simple prompt, actually, to make a three-dimensional world that you can then navigate through. And in that world, it has memory. So in the world, if you paint on a wall Mhmm.

转身再看时，涂鸦还在墙上。

And turn away, you look back the wall.

具有持久性。

It's persistent.

没错，持久保存。看到时我简直惊呼：天啊，太不可思议了。

Yeah. It's persistent. And when I saw that, I go, god. Jesus. Bloody hell.

这简直是为创造与真实世界无异的模拟系统奠定了初步基础。

This is this is like the foothills of being able to create a simulation that's indistinguishable from everything I see here.

没错。这正是我认为我们身处其中的原因。人工智能正发展到能创造人类级别代理的水平，虚拟现实也即将达到与我们现实无法区分的程度。

Right. That's why I think we are in one. That's exactly the reason. AI is getting to the level of creating human agents, human level agents, and virtual reality is getting to the level of being indistinguishable from ours.

所以你认为这是模拟世界？

So you think this is a simulation?

我非常确定我们处于模拟中。是的。

I'm pretty sure we are in a simulation. Yeah.

对于不熟悉模拟论的人来说，是哪些基本原理让你确信我们正生活在模拟现实中？

For someone that isn't familiar with the simulation arguments, what are what are the first principles here that convince you that we are currently living in a simulation?

这需要特定技术支撑。如果你相信我们能创造人类级别AI，也相信我们能创造出分辨率、触觉等各方面媲美现实的虚拟世界，那么我敢说——当技术成本足够低时，我会立即运行数十亿次此刻情景的模拟，从统计学上确保你正身处其中。

So you need certain technologies to make it happen. If you believe we can create human level AI Yeah. And you believe we can create virtual reality as good as this in terms of resolution, haptics, whatever properties it has, then I commit right now the moment this is affordable. I'm gonna run billions of simulations of this exact moment, making sure you are statistically in one.

重复下最后部分。你说你会运行...会运行什么？

Say that last part again. You're gonna run you're gonna run?

我承诺当成本足够低廉时——比如每月只需10美元——我将运行十亿次这个访谈的模拟版本。

I'm gonna commit right now when it's very affordable. It's like $10 a month to run it. I'm gonna run a billion simulations of this interview.

为什么？

Why?

因为从统计学看，这意味着你现在就身处某个模拟中。你处于真实世界的概率只有十亿分之一。

Because statistically, that means you are in one right now. The chances of you being in the real one is one in a billion.

明白了。让我确认下是否理解正确

Okay. So to make sure I'm clear on this

这是一种追溯性的安置。

It's a retroactive placement.

没错。所以一旦成本可负担，你就能运行数十亿个这样的实例，它们的感觉和表现会与此刻的这场访谈一模一样。是的。

Yeah. So the the minute it's affordable, then you can run billions of them, and they would feel and appear to be exactly like this interview right now. Yeah.

假设AI确实具有作为感受质的内部体验状态——有人持反对意见，也有人认为它们已经具备——这是个独立的哲学问题。但如果我们能模拟这点，我会去做的。

So assuming the AI has internal states experience as qualia, some people argue that they don't, some say they already have it. That's a separate philosophical question. But if we can simulate this, I will.

有些人可能会误解。你并不是说你自己会做，而是说有人会做。

Some people might misunderstand. You're not you're not saying that you will. You're saying that someone will.

其实我也可以做。我不介意。当然，在我达成之前肯定会有别人先做到。如果我能用10美元实现，别人可能已经花了一千美元。

So I I can also do it. I don't mind. Okay. Of course, others will do it before I get there. If I'm getting it for $10, somebody got it for a thousand.

重点不在这里。有了这项技术后，我们肯定会为了研究、娱乐、游戏等各种原因运行大量模拟。这些模拟的数量远超我们所在的真实世界数量。看看孩子们玩的电子游戏，每个孩子都在玩10款不同的游戏。

That's not the point. If you have technology, we're definitely running a lot of simulations for research, for entertainment, games, all sorts of reasons. And the number of those greatly exceeds the number of real worlds we're in. Look at all the video games kids are playing. Every kid plays 10 different games.

全球有上亿孩子，所以一个现实世界就对应着100亿个模拟世界。

There's, you know, billion kids in the world. So there is 10,000,000,000 simulations in one real world.

嗯。

Mhmm.

当我们考虑高级AI和超级智能系统时更是如此。它们的思维方式与我们不同，思考更细致，会进行实验。因此，在创造人造人类并模拟整个星球的层面上运行详细的问题模拟，将成为它们的常规操作。所以更可能的情况是——这不需要我花10美元来实现。

Even more so when we think about advanced AI, superintelligent systems, their thinking is not like ours. They think in a lot more detail. They run experiments. So running a detailed simulation of some problem at the level of creating artificial humans and simulating the whole planet would be something they'll do routinely. So there is a good chance this is not me doing it for $10.

而是未来某个模拟系统在思考关于这个世界的某些事情。

It's a future simulation thinking about something in this world.

所以可能存在这种情况：某个人类物种或某种形式的智慧生命发展到了能够经济地运行与当前世界无法区分的模拟程序，并决定付诸实践。而我们现在就身处其中。他们运行这些模拟作为实验、游戏或娱乐是合理的。而且，当我思考这个我可能正身处其中的模拟世界里的时间时，相对而言时间感觉很长。你知道，我一天有二十四小时，但在他们的世界里，可能——

So it could be the case that a species of humans or a species of intelligence in some form got to this point where they could affordably run simulations that are indistinguishable from this, and they decided to do it. And this is it right now. And it would make sense that they would run simulations as experiments or for games or for entertainment. And, also, when we think about time in the world that I'm in, in this simulation that I could be in right now, time feels long relatively. You know, I have twenty four hours in a day, but on their in their world, it could be

时间是相对的。

Time is relative.

相对的。没错。可能只是一秒钟。我的一生在那里可能只是一毫秒。

Relative. Yeah. It could be a second. My whole life could be a millisecond in there.

对。你当然可以调整所运行模拟的时间流速。

Right. You can change speed of simulations you run for sure.

所以你相信我们很可能身处模拟之中？

So your belief is that this is probably a simulation?

极有可能。如果你重新审视宗教就会发现，这种观点其实有广泛共识。每个宗教本质上都在描述什么？一个超级智能存在——工程师、程序员——为了测试或其他目的创造虚拟世界。但如果你拿着模拟假说论文去丛林里，用原始部落的语言向他们解释。

Most likely. And there is a lot of agreement on that if you look again, returning to religions. Every religion basically describes what? A superintelligent being, an engineer, a programmer, creating a fake world for testing purposes or for whatever. But if you took the simulation hypothesis paper, you go to jungle, you talk to primitive people, a local tribe, and in their language, you tell them about it.

两代人之后再去。他们就会形成宗教体系。这就是故事的本质。

Go back two generations later. They have religion. That's basically what the story is.

宗教，没错，本质上描述的就是模拟理论。某个存在创造了——

Religion, yeah, describes a simulation theory basically. Somebody created

所以这天然是我们最早形成的理论。如今随着科学发展，越来越多人开始认真考虑其可能性。虽然像我这样高度确信的人不多，但确实有不少人开始重视这个假说。

So by default, that was the first theory we had. And now with science, more and more people are going like, I'm giving it nontrivial probabilizing. A few people are as high as I am, but a lot of people give it some credence.

你有多大比例相信我们目前生活在模拟世界中？

What percentage are you at in terms of believing that we are currently living in a simulation?

几乎可以确定。

Very close to certainty.

这对你生活的本质意味着什么？如果你近乎100%确定我们正生活在模拟中，这会改变你生活中的任何事吗？

And what does that mean for the nature of your life? If you're close to 100% certain that we are currently living in a simulation, Does that change anything in your life?

你在乎的一切依然如故。痛苦仍是痛苦，爱仍是爱，对吧？这些并未改变，所以无关紧要。

So all the things you care about are still the same. Pain still hurts. Love still love. Right? Like, those things are not different, so it doesn't matter.

它们仍然重要。这才是关键。那1%的不同在于，我关心模拟之外的事物。我想了解它，为此撰写论文。

They're still important. That's what matters. The little 1% different is that I care about what's outside the simulation. I wanna learn about it. I write papers about it.

这是唯一的影响。

So that's the only impact.

你认为模拟之外有什么？

And what do you think is outside of the simulation?

我不知道。但我们可以观察这个世界，推导出模拟者的某些特性。显然，他们是卓越的工程师、科学家、艺术家，但在道德伦理方面有所欠缺，尚有改进空间。

I don't know. But we can look at this world and derive some properties of the simulators. So clearly, brilliant engineer, brilliant scientist, brilliant artist, not so good with morals and ethics. Room for improvement.

依据我们对道德伦理的认知标准？

In our view of what morals and ethics should be?

我们知道世间存在苦难。除非你认为折磨儿童是合乎伦理的，否则我质疑这种设计方式。

Well, we know there is suffering in the world. So unless you think it's ethical to torture children, then I'm questioning your approach.

但若要建立正向激励体系，可能也需要设置负面激励。苦难似乎是我们设计中内置的负面因素，用于阻止我做不该做的事——比如把手伸进火里会感到疼痛。

But in terms of incentives to create a positive incentive, you probably also need to create negative incentives. Suffering seems to be one of the negatives and incentives built into our design to stop me doing things I shouldn't do. So, like, put my hand in a fire, it's gonna hurt.

但这完全关乎痛苦的等级，对吧？所以不愉快的刺激、负面反馈不必达到像负无穷地狱般的程度。你不想被活活烧死并感受它。你只是觉得，哦，这让人不舒服。

But it's all about levels levels of suffering. Right? So unpleasant stimuli, negative feedback doesn't have to be at, like, negative infinity hell levels. You don't wanna burn a life and feel it. You wanna be like, oh, this is uncomfortable.

我要停下了。

I'm gonna stop.

这很有趣，因为我们假设它们没有更高的道德伦理标准，但我们同样会捕捉动物烹饪食用，也会用老鼠做实验。

It's interesting because we we assume that they don't have great more morals and ethics, but we too would we take animals and cook them and eat them for dinner, and we also take conduct experiments on mice and rats.

但要获得大学批准进行实验，你需要提交提案，伦理委员会会裁定：你不能在人类身上实验，不能焚烧婴儿，不能活吃动物——这些行为都会被禁止。

But to get university approval to conduct an experiment, you submit a proposal, and there is a panel of ethicists who would say, you can't experiment in humans. You can't burn babies. You can't eat animals alive. All those things would be banned.

在世界大部分地区是这样。

In most parts of the world.

有伦理委员会的地方确实如此。有些地方根本不讲究这些，所以审批流程更宽松。

Where they have ethical boards. Yeah. Because Some places don't bother with it, so they have easier approval process.

讨论模拟理论时有个诡异现象——这种对话会莫名消解生命的意义。虽然知道无所谓，但每次和人聊到（非播客场合）我们是否活在模拟中时，你会看到他们眼中短暂闪过空洞，然后遗忘继续。这种模拟假设暗示着一切都不重要。我觉得人类渴望相信这是最高现实，我们至关重要，宇宙以我们为中心——我们天生就极度自我。

It's funny when you talk about the simulation theory. There's there's an element of the conversation that makes life feel less meaningful in a weird way. Like, I know it doesn't matter, but whenever I have this conversation with people, not on the podcast about are we living in a simulation, you almost see a little bit of meaning come out of their life for a second, and then they forget and then they carry on. But the the the thought that this is a simulation almost posits that it's not important or that I think humans want to believe that this is the highest level, we're at the most important, and we're the it's all about us. We're quite egotistical by design.

是啊，这是我常观察到的有趣现象：这类对话似乎会抽走人们生活的某些意义。

And, yeah, just a certain interesting observation I've always had when I have these conversations with people that it it seems to strip something out of their life.

你觉得宗教徒会这样想吗？他们知道存在更重要的彼岸世界。你认为他们不珍视现世生命吗？某些宗教可能如此。

Do you feel religious people feel that way? They know there is another world, and the one that matters is not this one. Do you feel they don't value their lives the same? I guess in some religions.

他们认为现世是为其创造的，死后会去天堂或地狱——这仍使他们处于宇宙中心。但如果是模拟，我们可能只是外星幼童消磨时间的电子游戏角色。

Think they think that this world is being created for them and that they are gonna go to this heaven or or hell, and that still puts them at the very center of it. But it but if it's a simulation, you know, we could just be some computer game that a four year old alien is messing around with while he's got some time to burn.