AI先驱尤尔根·施密德胡伯：AI已经能感受痛苦、爱并具有自我意识

疼痛信号只是告诉机器人哪些事情应该避开。

The pain signals are just informing the robot about what should be avoided.

疼痛其实是大自然和生物进化发明的一种机制，为了让动物有动力去学习避免痛苦。

Pain is just an invention of nature and biological beings, of evolution, which invented this pain sensor thing for animals such that they have an incentive to learn to avoid the pain.

我们在人工智能的学习系统里，早就这么做了几十年了。

And we have done that for many many decades in our learning machines.

大脑里用来编码恐惧、爱等情绪的化学物质，跟我们人工大脑里用的完全不一样。

The chemicals that are used in brains to encode certain states of fear or of love and whatever, they are different from what we are using in our artificial brains.

但背后的原理一定是一样的。

But the principles must be the same.

然后它就有了自我意识。

And then it's self aware.

它真的具备了自我意识。

It's self aware.

比如说，当它照镜子时，会马上意识到：镜子里那个人，我能控制他的动作。

In the sense that, for example, if it looks in the mirror, it will quickly figure out, oh, the guy in the mirror, I can control what this guy does.

如果我这么做，他就会做出对称的动作，因为我能控制他。

If I do this, then he will do the symmetric thing because I have it under control.

但如果你在房间另一头，我做了这个动作，而你却做出完全无关的反应，我就没法预测了。

But if you are there on the other side of the room and I do this and you will do something that's completely unrelated, I cannot predict them.

在这里，你已经能看到‘能动性’这个概念——它立刻被区分为自我能动性，或是他人的能动性。

So there you already see this concept of agency which is immediately recognized as self agency or the agency of another guy.

五年内，你会损失九千亿美元。

Within five years, you are going to lose $900,000,000,000.

不久的将来，总会有人损失九千亿美元，因为根本不存在可行的商业模式。

Somebody is going to lose $900,000,000,000 in the near future, because there is no business model.

没人能想出办法把这些损失都补回来。

Nobody has a business model to recuperate all these losses.

AI模型从现在开始还能提升多少？

How much better can AI models get from here?

它们越来越聪明，这又说明了我们的大脑和存在本身些什么？

And what does their increasing smarts say about our brains and existence itself?

接下来，我们将与人工智能先驱、《卫报》称其为‘人工智能之父’的约尔根·施密德胡伯展开对话。

We'll talk about it with AI pioneer and someone that The Guardian has called the father of AI, Jorgen Schmidhuber, right after this.

面对持续的变革与机遇，TMT领域的领导者需要交付切实的成果，而不仅仅是想法。

In the face of ongoing disruption and opportunity, TMT leaders need to deliver tangible results, not just ideas.

当速度和表现至关重要时，普华永道融合市场洞察、行业专长，结合人工智能、云计算和新兴技术，加速您的转型，从战略到执行实现可衡量的投资回报。

When pace and performance matter most, PwC combines market insights and deep sector experience with AI, cloud, and emerging tech to accelerate your transformation and drive measurable ROI from strategy to execution.

普华永道能帮助您预判未来趋势，超越颠覆，赢得竞争。

PWC can help you anticipate what's next, outpace disruption, and compete.

如需更多信息，请访问 pwc.com。

For more information, visit pwc.com.

欢迎收听《大科技》播客，一档致力于理性、深入探讨科技世界及其更广影响的节目。

Welcome to Big Technology Podcast, a show for cool headed and nuanced conversation of the tech world and beyond.

今天这期节目，我们将深入探讨人工智能的未来走向、这项技术还有多大提升空间，以及它的日益智能化对我们自身、我们的存在、我们的大脑意味着什么。

We have a great show for you today looking exactly into where AI is heading, how much potential the technology has to to improve after this, and also what its increasing smart says about us, our existence, our brains.

我们非常荣幸邀请到人工智能领域的先驱之一、《卫报》誉为‘人工智能之父’的约尔根·施密德胡伯教授，他今天从阿姆斯特丹连线加入我们。

And we are thrilled to be joined by one of the pioneers of AI, someone that The Guardian has called the father of AI, Jorgen Schmidhuber, a professor who's joining us today from Amsterdam.

教授，很高兴见到您。

Professor, great to see you.

欢迎来到节目。

Welcome to the show.

亚历克斯，能来这儿是我的荣幸。

Alex, it's my pleasure to be here.

我们先聊聊您本人，以及您对人工智能发展所做的贡献吧。

So let's just speak a little bit about about yourself and your contributions to where AI stands.

您显然为人工智能做出了许多重大贡献，比如记忆机制、P和T，还有GPT——您的推特简介里也提到，《卫报》称您为人工智能之父。

You've obviously made a lot of big contributions towards AI, things like memory, the P and the T and the GPT as your Twitter bio says the Guardian's called you father of AI.

媒体似乎总喜欢给研究人员贴标签，比如‘人工智能之父’、‘教父’之类的，总说有那么几位关键人物。

It seems like the media tends to, know, put these labels on researchers, you know, there's there's a handful of fathers and godfathers of AI.

您怎么看待这种说法？您如何定位自己在当今技术发展中的角色？

How how do you respond to that, and how would you contextualize your role in where the technology is today?

没有人能独自创造出人工智能。

No single person can create an AI by himself or herself.

构建人工智能需要整个文明的支持。

You need an entire civilization to build an AI.

不仅需要那些研发算法的人，特别是针对人工神经网络的学习算法——这正是当前人工智能的核心。

You need not only the guys who are trying to invent algorithms, learning algorithms for artificial neural networks, that is what current AI is about.

还需要那些制造更强大计算机的人。

You also need people who build better computers.

更需要无数游戏玩家，是他们创造了对更快计算机的巨大需求，激励硬件厂商每五年将每美元的计算性能提升十倍。

You need all the video gamers who are creating a market for acquiring more of these faster and faster computers and you need and providing an incentive to the computer makers to speed up the computation per dollar by a factor of 10 every five years.

还得有那些为游戏玩家提供粮食的农民，等等。

You need all the farmers who are feeding the video gamers and so on.

所以，一个人根本不可能独立创造出人工智能。

So it is impossible for a single person to create an AI.

必须依靠整个文明的协作。

You need an entire civilization.

明白了。

Okay.

所以我很好奇，你认为这个文明正在朝哪个方向发展？

And so I'm curious to hear your perspective about where this civilization is working toward.

你长期奋战在AI研究的第一线，而如今我们确实正处在飞速进步的阶段。

You have been obviously in the trenches working on AI researcher research for a long time, and we're definitely in a period of fast progress today.

大家一直在问，AI还能变得多强？会不会遇到瓶颈？现在的技术路线最终能带我们走多远？

So there's been all these questions about how much better AI can get and whether it's gonna hit a wall and where current techniques will lead.

你对此怎么看？

Where do you what do you think?

你对AI未来能走到哪一步，持什么看法？

Where do you land on that question about where AI can go from here?

从上世纪七十年代起，我一直是个乐观派，一直说希望在我有生之年能造出一个能超越我的AI，让我可以退休，但显然我们还没达到那一步。

So since the nineteen seventies, I have been an optimist and I have been claiming that within my lifetime I want to build an AI that learns to become smarter than myself such that I can retire and we are obviously not there yet.

目前真正能用得上的AI，其实就在你眼前的屏幕上。

And at the moment the only AI, the only AI that is working well is the AI behind your screen, you know.

没错，屏幕背后那个AI，已经能通过图灵测试了。

Yes, behind the screen there is an AI that can pass the Turing test.

那是什么意思？

What is that?

就是说，你打字问它，它会回答你。现在的问题是，你能分得清对面是人还是机器吗？

It means if you type questions to it, it answers back and and now the question is can you distinguish whether the other guy is a human or a machine.

如今，很多AI都已经能通过图灵测试了。

And today this Turing test is passed by many AIs.

但这恰恰说明图灵测试根本不能准确衡量智能——因为到现在，还没有任何AI能像一个小男孩那样在现实世界里做事，也没法像水管工或电工那样干活。几十年来，我一直拿水管工当例子。

However, it just means that the Turing test is a bad way of measuring intelligence because there is no AI in the physical world outside of the screen that can do all the things that a little boy can do, that can do the things that a plumber can do, for decades I have used the plumber as an example, Or an electrician.

人类用双手能做的所有事情，AI现在都做不好。

So all the things that humans can do with their hands, their physical hands, they don't work well.

目前真正能干好的，只有屏幕背后那些二进制的代码和数据。

Only bits and zeros and ones behind the screen, that is the only thing that is working well.

但这种情况不会永远持续下去，现实世界中的AI也在进步。我一直说，几十年来我都相信，不远的将来，我们会造出一种机器人，它不需要特别聪明，只要够聪明去学会操作所有现成的机器就行。

However, it's not going to stay like that forever and, there is progress in the physical world, AI for the physical world, and I think the culmination point of that, which I've been talking about again for decades, is at some point in the not so distant future we will have a robot that doesn't have to be super smart, but just smart enough to learn to operate all the already existing machines.

一旦有了这样的机器人，我们就迎来了一种全新的生命形态。

Once we have a robot like that, we have a new kind of life.

几个世纪以来，人们一直在讨论自我复制的机器，但现在我们有了机会——一旦有了这样的机器人，它就能开始制造更多同类。

For hundreds of years people have talked about self replicating machinery, but now we have an opening, once we have a robot like that, then it can start making more of its own kind.

这意味着，我们将突然拥有一种终极的扩展机器，也就是我所说的自我复制机器，而且它还能自我改进，因为那些在屏幕背后已经表现优异的软件技术，也会反过来提升现实世界中各种机械设备的性能。

And this means, suddenly we will have this ultimate scaling machine, as I have called it, self replicating machinery, but then also self improving machinery, because all the stuff that already works well for a software behind the screen is going to also improve the performance and all the machinery in the real world, in the physical world.

我认为这将是一个关键转折点，从此将催生一种全新的文明，因为这种机制不仅适用于地球上的生物圈，也适用于太空，比如月球，但更可能首先在水星上实现，然后扩展到整个太阳系乃至更远的地方。

And that is I think going to be this inflection point and from there a new kind of civilization is going to emerge because something like that doesn't work only in the biosphere, it also works out there in space, on on the moon, but maybe more likely first on Mercury and then the rest of the solar system and beyond.

明白了。

Okay.

关于这个，我有很多问题，包括它对‘何以为人’这一本质意味着什么，不过我们先放一放。

So I have many questions about this, including what this would mean for the nature of what it means to be human, but let's leave that for now.

你的说法让我立刻想到一件事：我最近去了OpenAI总部，和他们的总裁格雷格·布罗克曼聊过，当时他们刚决定暂停Sora视频生成器的研究，转而全力聚焦在GPT这类文本模型上，对吧？

Your answer sparks an immediate follow-up for me, which is I was recently at OpenAI Headquarters speaking with Greg Brockman, president of OpenAI, and it was in the moment that they had decided that they were going to pause research on Sora, their video generator, and decide to focus almost entirely on these GPT models, right?

就是最近进步飞快的那些文本模型。

The text models that we've seen make so much progress recently.

我问布罗克曼：你们不专注在视频生成这类更贴近真实世界的模型上，会不会错过什么？

And I asked Brockman, don't you lose something by not focusing on these more world model style applications like video generation?

他说，我们确实会失去一些东西。

And he said we do lose something.

我们不可能面面俱到，但我们相信，GPT 这类模型才是通向通用人工智能、达到人类水平智能的关键路径。

We can't do everything and we believe that the GPT style models are the way to get to AGI or AI on par with human intelligence.

你刚才那样回答，你觉得他这个观点对吗？

Do you think having answered the way that you did that he's correct?

仅靠 GPT 模型真的有可能实现通用人工智能吗？还是说，他们放弃世界模型的做法从根本上就错了？

That there is a chance to get to AGI using just the GPT models or are they making a fundamental mistake by abandoning that world model in practice?

尽管 OpenAI 仍然在做一些机器人方面的研究。

Even though there's still some robotics work within OpenAI.

是的。

Yeah.

如果问题是：一个大型语言模型本身能否直接通向通用人工智能？那答案很明确：不能。

So when the phrase when the question is phrased like this, does an LLM, a large language model by itself lead to AGI, the answer is a clear no.

不过呢，OpenAI 那边有一群非常聪明的人，他们心里其实很清楚这一点。

But you know, OpenAI has a bunch of smart people and they know that exactly, you know.

当然，他们也知道像基础模型或者大语言模型这类东西能做什么。

And of course they know also what you can do with a foundation model or a large language model or something like that.

你可以把它当作一个世界模型来用。

You can use it as a model of the world, as a world model.

当年我1990年提到的世界模型，其实就是用同样的神经网络，去预测另一个决策者——也就是另一个生成动作、改变环境的神经网络——的行为后果。

The same type of neural networks that are used there can be used as a a world model as I called it in 1990 which just learns to predict the consequences of the actions of another decision maker, of another neural network that is generating actions that modify the environment.

就像一个小婴儿那样。

Like a little baby, you know.

婴儿不是靠上网下载信息来学习的。

A little late baby doesn't learn by downloading the web.

ChatGPT 才是靠下载网页来学习的。

That's what chat GPT does.

婴儿不是靠上网下载信息来学习的。

A baby doesn't learn by downloading the web.

没错。

No.

它通过自己设计的实验来生成数据流，从而学习。

It learns by creating its own data stream through its own self invented experiments.

比如，当婴儿这么做时，摄像头传入的视频画面每几毫秒就更新一次，数以亿计的新像素不断涌入，而婴儿体内有个内在机制——我们不妨称之为‘世界模型’，它能预测这些变化。

For example, when the baby does this, then the video which changes coming in through the cameras, you know, every few milliseconds hundreds of millions of new pixels coming in, and then the the baby has an internal mechanism, let's call it the wild model, which learns to predict these changes.

一开始，它甚至不知道自己有手，但随着时间推移，它学会了预测发送给运动神经元的指令会产生什么后果：声音会上下起伏，手会动起来，慢慢就理解了世界是如何运作的。

And in the beginning, it doesn't even know that it has a hand, but then over time it learns to predict the consequences of sending certain action signals to its motor neurons and then the speech muscles go up and down or the hand is moving and it learns how the world works.

所以，它所依赖的数据，都是自己通过实验产生的。

So the data that it is generating through its own experiments.

婴儿有点像物理学家。

So a baby is a little bit like a physicist.

物理学家可不是靠下载网页来学习的。

A physicist doesn't learn by downloading the web.

他偶尔会看一些论文，算是一种补充。

He learns a little bit by downloading papers.

但物理学家真正做的，是设计新的实验，产生前所未有的数据，用来验证关于宇宙的假设，从而更深入地理解物理世界。从1990年左右开始，我们的神经网络其实也在做类似的事，只是还没达到物理学家那样的水平。

But what the physicists do then is they generate new experiments that lead to data which has never been there before or has never been collected before to better verify certain hypotheses about the universe and then to better understand the world, the physical world, and that's what our artificial neural network since about 1990 also do in a way that is just not yet as impressive as what physicists do.

但我认为它迟早能做到。

But I think it's going to get there.

不过，至少可以分成两个部分。

Nevertheless, there are then at least two components.

一个是基础模型，可以是各种人工神经网络，本质上就是一个预测工具。

There is one, there is a foundation model which can be any of a variety of artificial neural networks, that is just a prediction machine.

如果我这么做，会发生什么？

What happens if I do that, you know?

根据到目前为止的数据，下一个词会是什么？

What is the next token if I look at this data so far?

另一个是控制器，它会利用那个预测模型来做规划。一旦这个预测模型——也就是世界模型——变得足够准确，它就能预测复杂动作序列的后果，然后控制器就会选择那些能带来高预期奖励、低预期痛苦的动作序列。

And then there's the other thing, the other neural network, the controller, which uses the second guy, the prediction machine, to plan, so that it can, once the second guy, the prediction machine, the WAN model, is pretty good, then it can learn to predict the consequences of complicated action sequences and then it's going to select for the controller an action sequence that leads to a lot of predicted reward and little predicted pain.

当然，我们的机器人装有痛感传感器，一旦撞上障碍物，就会收到负面反馈——也就是负数信号。世界模型不仅能预测视频这类中性信号，还能预测这些价值信号、奖励信号等等。如果模型足够好，就能用来做规划，控制器于是能用心理模拟代替真实实验——毕竟真实实验成本太高，来筛选出更优的动作序列。但你看，单靠这个模型本身，也就是基础模型，还远远算不上通用人工智能。

So of course our robots, they get pain sensors and then whenever they bump against an obstacle, negative punishment is negative numbers are coming in and the world model predicts not only the neutral signals like video and so on, but also these value signals, these reward signals and so on and then if the model is good, then you can use it for planning and the controller then can use mental experiments instead of real experiments, which are really expensive to to select action sequences that are good, but in in those cases, you see, the model alone, the foundation model alone is is not an AGI.

当然不是。

No.

你还需要另一个系统，它会用各种技巧，从世界模型中提取算法信息，来生成更优的计划。

You need the other guy which uses all kinds of tricks to exploit the algorithmic information in the world model to come up with better plans.

所以2017年左右，你提到过这么一件事，你说过这样的话。

So in 2017, around the time, we're talking about you said something, you said this.

在不远的将来，我能跟一个小机器人说话，教它做复杂的事，比如只靠示范就能组装手机、制作T恤——这些现在都是发展中国家的穷孩子在像奴隶一样干的活。

In in the not so distant future, I will be able to talk to a little robot and teach it to do complicated things, such as assembling a smartphone just by show and tell, making t shirts, and all these things that have currently that are currently done under slave like conditions by poor kids in developing countries.

人类会活得更久、更快乐、更健康、更轻松，因为很多现在压在人身上的苦活累活，都会被机器取代。

Humans are going to live longer, happier, healthier, and easier lives because lots of the jobs that are now demanding on humans are going to be replaced by the machines.

到时候，会有上万亿种不同类型的AI，形成一个快速变化、高度复杂的AI生态，其发展速度连人类都跟不上。

Then there will be trillions of different types of AIs and a rapidly changing complex AI ecology expanding in a way where humans cannot even follow.

差不多十年前，你说过，现在看来这事反而比当时更有可能实现了。

So almost ten years ago, you said that it actually seems more plausible now than obviously it did back then.

你觉得我们离那个未来还有多远？

How far away do you think we are from that future?

是的。

Yeah.

这是个好问题。

That's a good question.

实际上，早在2014年，因为当时我说的那些话，还有我做的一些演讲，我们就成立了一家公司叫Nasens，专注于现实世界中的物理AI，利用世界模型来更好地与环境互动。我们当时接了不少疯狂的合同，合作的都是些特别有名的大公司，但可能还是太早了——就像我们做的其他一些事一样，那个年代的技术根本撑不起这么大的野心。

So actually even even earlier in 2014, because of these things that I said back then and when I gave talks or whatever, We formed a company, Nasens, which was really about physical AI in the world, in the real world using world models and then exploiting the world models to better interact with the world and we had crazy contracts with really famous companies, nevertheless, this was probably still too early, like some of the other things we have done, so probably too ambitious for that time.

但现在我们越来越接近了，而主要的瓶颈其实出在硬件上。

But now we are getting closer and closer, and you know the main hindrance, the main obstacle is very progress in the hardware side.

硬件方面有一种进步非常显著，从1941年就开始了，每五年，计算机的成本就降低到原来的十分之一。

So there is one type of progress on the hardware side which is enormous, which has been enormous since 1941, which is every five years computers getting 10 times cheaper.

1941年，图泽建造了第一台通用程序控制的计算机，每秒大概只能执行一次操作。

So in 1941, the first general purpose program controlled computer by Tzuze, he could do maybe one operation per second.

三十年后，花同样的钱，就能做到每秒一百万次操作，而今天，花同样的钱，我们几乎能实现每秒百亿亿次的运算。

And then 30 later, for the same price, one was able to do a million operations per second, and today we almost not quite have a billion billion instructions per second, for the same price for the same price.

但今天的机器人，跟三十年前的相比，可没好上一百万倍。

But the robots of today the robots of today, they are not a million times better than the robots that we had thirty years ago, you know.

比如，二十五年前，我们已经有能走路的机器人了，不过它们得走得特别小心，不像现在的机器人这么灵活。那时候日本的阿西莫机器人特别出名，日本当时占了全球机器人的一半以上，它们走路时必须时刻保持重心在脚掌正上方，远不如现在这种动态行走技术先进，但也没差上一百万倍，大概也就强个两三倍吧。

For example, you know, twenty five years ago, we already had walking robots, they they had to be they had to walk more carefully than today's robots, you know, the Azimu robots in Japan, back then Japan had more than half of the robots in the world, they always had to keep their center of gravity above the foot and so it was not as advanced as modern dynamic walking and stuff, but it was a little bit worse than today's robots, but not a fact of a million, know, maybe a fact of three or something like that.

所以，我们为类人机器人所研发的硬件，比如机械手、仿生身体，其进步速度远远赶不上每美元算力的增长。

So the hardware, artificial hands, the artificial bodies that we are trying to build for humanoid robots and so on, they are evolving much less rapidly than the compute per dollar.

我认为这才是核心问题。

And I think that's the main problem.

早在二十年前，我们就已经有了像婴儿一样的小型机器人，比如意大利北部一个实验室开发的iCub机器人，外形像婴儿，能自己设计实验、设定目标，逐步探索世界运行的规律，还采用分层学习的方式。

So already twenty years ago we had little robots, baby like robots, the iCub robot which back then was constructed by a lab in Northern Italy, which looks like a baby and then it invented its own self invented experiment, set itself its own goals, tried to figure out how the world works and doing that in a hierarchical way and so on.

但才完成了三个自设实验，它的肌腱或手指就断了。

But, after having executed three self invented experiments, some tendon and some finger was broken.

得请技术人员来修理。

Technician had to come and fix it.

做这些事的成本实在太高了。

So it was just so expensive to do all of that.

我们真正需要的是能媲美人类双手的硬件，但目前没有任何人工设计的技术能跟人类的手相提并论。

And what we really need is hardware that can compete with human hands, but there's no human designed tech that can compete with these hands.

人类的手里有数以亿计的传感器，无数根缆线把传感器连到控制中枢，我根本不知道该怎么布这些线，而且要是割伤了，它还能自己愈合。

These hands, they have millions of millions of sensors and all kinds of cables connecting the sensors to the control center and I wouldn't know where to put all the cables and if I cut it, it starts healing itself.

所以我的手和你的手，这都是超级先进的技术。

So my hand and yours, that's super advanced technology.

这远远超出了传统机器人技术人类能实现的水平。

It's really completely beyond what humans can build with traditional robot technology.

所以我们还有很长的路要走。

And so then we still have so much there's such a long way to go there.

我仍然希望在我有生之年能看到突破，实现我年轻时在七十年代做出的预测，但很明显，硬件的进步远比GPU慢。

I'm still hoping that it's going to happen within my lifetime to make it through the prediction that I made in the seventies when I was a teenager, but clearly the hardware evolution is much slower than the GPU evolution.

是的。

Right.

看来这方面还得花不少时间。

Looks like it's gonna take a lot more time on that front.

我有个问题想问你。

Now question for you.

随着技术不断进步，你认为谁会最终获得最大的价值？

Who do you think is gonna capture the most value as the technology continues to to improve?

我觉得这句话是别人引用你的。

I think this is a quote attributed to you.

你说过，不会是几家大公司垄断一切。

You said it's not a few big companies that are going to dominate everything.

人工智能最大的受益者，其实是普通人。

Greatest profit or the great profiteer of AI is going to be the little man.

但如果你看看眼下趋势，确实是一些大型实验室在迅速崛起，估值可能达到万亿美元，背后还有那些科技巨头撑着。

But it does seem like if you at least look at where things are heading right now, you've had these big labs that are going to have trillion dollar valuations and the big tech companies surrounding them.

看起来经济价值都被他们拿走了，而所谓的普通人，处境比以往任何时候都更不确定。

Looks like they're capturing most of the economic value and the quote unquote little man is about as uncertain as they've ever been.

你对这个问题怎么看？

What do you think on that front?

是的，我觉得这反映的是当前经济某些领域里的泡沫。

Yeah, I think it's just a reflection of the current bubble that we have in certain aspects of the economy.

你看硅谷现在一堆僵尸独角兽，很多公司名义上估值超过十亿，可要是真上市，恐怕连两千万都值不了。

Because if you look at all these zombie unicorns in Silicon Valley now, know, there are lots of companies that have officially a billion dollar plus evaluation, but if they were on the stock market, they would probably be worth just 20,000,000 or something like that.

所以这些被称为僵尸独角兽，如果你看看像Anthropic和OpenAI这样最显眼的公司，它们在这些方面花的钱简直没完没了。再看谷歌和微软，它们的资本支出直接拖累了现金流。

So zombie unicorns they are called, and if you look at the most visible companies like Anthropic and OpenAI, you know, they are they are spending so much money on on all that stuff, and if you look at Google and Microsoft, you look at the cap ex spending and this directly affects their cash flow.

表面上看，这些公司的市盈率好像没受影响，但实际上应该受影响才对。以前谷歌和微软每年的自由现金流能有上千亿美元，现在却跌到了两三百亿，甚至更低。

Apparently it doesn't affect the price earnings ratios of these companies, but it should because there was a time when, you know, Google and Microsoft, they had on the order of a $100,000,000,000 free cash flow, but now it's down to 20,000,000,000, 10,000,000,000.

有些公司现在的现金流已经变成负的上百亿美元了。

Some of the companies suddenly have minus 10,000,000,000 cash flow.

它们为了给数据中心买GPU，不得不借债，结果这些公司越来越像公用事业企业了。

They take on debt to finance all these GPUs in the data centers and so these companies are becoming more like utilities, right?

以前这些灵活的软件公司，可能就十几个人，优化一下操作系统，然后让全球几十亿人用自己电脑或手机跑就行。现在呢？同样的公司却得去考虑买燃气轮机、投资核电站、背负债务——全干起了电力公司那套活儿。

Because suddenly these formerly nimble software companies who had maybe a small team of 10 people to improve some shitty operating system and roll it out for billions of people who all use their own computers, their own smartphones to run the operating system, suddenly the same companies, they have to think about buying gas turbines and investing in nuclear power plants and taking on debt and doing all the things that electricity companies do, utility companies.

而且别忘了，每五年，计算机的成本还是会降十倍，这个规律没变。

And now take into account that every five years, it's still true, five years computers getting 10 times cheaper.

今天你投一千亿美元买GPU，五年后，这钱就差不多蒸发九千亿。

Now if you invest 1,000 billion dollars today into GPUs for data centers this means that within five years you are going to lose $900,000,000,000.

未来肯定有人要亏掉九千亿，因为根本没人想清楚怎么把这笔钱赚回来。

Somebody is going to lose $900,000,000,000 in the near future because there is no business model, nobody has a business model to recuperate all these losses.

这些公司都没有护城河，因为每当有新的基准测试打破纪录，或者出现什么新的语言模型，过几个月同样的技术就会出现在开源社区里。

And none of the companies have a moat because whenever there's a new benchmark breaking record or something, benchmark record breaking language model that does this or this or whatever, a few months later there's the same thing in open source, you know.

所以压价压力巨大，而这些公司根本没时间喘息。

So there is so much pressure to keep the prices down and none of these companies has the moment.

咱们先等等吧，等那些万亿级公司被强制纳入ETF的热潮过去再说。

So let's wait a little bit until the forced ETF buying of these trillion dollar companies is over.

现在市场上突然冒出一堆万亿甚至更高估值的公司，指数基金不得不赶紧买入，因为纳斯达克和其他指数调整了成分股规则。

At the moment we have trillion dollar or even more companies appearing on the stock market and the index funds have to buy them suddenly because the Nasdaq and and others, they change their rules.

通常来说，市场得花一年左右时间自己去发现一家公司的真正价值，但现在这一步被跳过了。

Normally, you wait for a year or so until the stock market finds out by itself what is the true value of this company, but at the moment, this is not done.

于是ETF、养老基金，所有人，都得跟着买。

So suddenly the ETFs and the pension funds and everybody, they have to buy that.

等这波强制买入结束，咱们再看看，当创始人想套现时会发生什么。

Once that is over, let's see what happens when the when the founders try to sell their stakes.

如果我们看不到股市出现剧烈波动，那才真叫奇怪呢。

So where's where's your vision of would be astonishing if if we wouldn't see some huge fluctuations in the stock market prices here.

对，那你觉得散户最后会落得什么下场？

Right, so where's your vision of where the little guy can end up?

散户长远来看是会受益的。

The little guy is, in the long run, the little guy is going to profit.

眼下大公司都在疯狂赚钱，大家都这么说。

So at the moment, the big companies, and everybody says, the big companies, they are profiting like crazy.

但事实真是这样吗？

But is that true?

看看他们的现金流，不，都在往下掉。

Look at their cash flows, no, they go down like that.

再看看他们的负债率，他们都已经停止回购自家股票了，因为手里的现金已经不够了。

Look at their debt ratios, they have stopped to repurchasing their own shares because they don't have enough cash any longer.

所以，普通投资者这时候能做的，就是稍微等等。你知道的，电脑性能每五年就会提升十倍，这意味着十年后，同样的配置只需要花百分之一的钱就能买到。

And so at the same time, what the little guy has to do is just wait a little bit, you know, because every five years computers getting 10 times cheaper, which means in ten years you can buy the same thing for 1% of the price.

这情况就跟智能手机的发展轨迹一模一样。

And it's going to be just like with smartphones.

我常讲一个八十年代认识的富人的故事，他有一辆保时捷。

And I'm often relating the story of the rich guy whom I knew in the eighties and he had a Porsche.

他很有钱，开的是保时捷。

He was rich, he had a Porsche.

但最让人惊讶的是，那辆保时捷里居然装了部移动电话。

But the most amazing thing was in the Porsche there was a mobile phone.

他能拿起听筒，通过卫星跟另一个开同样车的人通话。

He could pick up the receiver and talk via satellite to another guy with a Porsche like that.

而今天，四十年过去了，发展中国家的普通人人手一部智能手机，性能远超他当年那台设备。

And today, forty years later, everybody in developing countries has a smartphone that is immensely more powerful than what he had in his porch.

所以普通人现在花几乎不用花钱，就能得到当年贵得离谱的东西。

So the little guy is paying almost nothing for whatever he had there which was really expensive.

AI也会是同样的情况，很快它就不再只存在于云端了。

And the same thing is going to be true with AI and soon AI will not be in the cloud, know.

它会直接运行在你自己的小型电脑上。

No, it will be local on your local computer, a small little computer.

它会和现在的云端一样强大，但你不用连上网，而联网总是让人担心，你说对吧？谁知道外面有多少人正等着你连上去呢。

It will be as powerful as what's now in the cloud but you won't have to connect it to the internet, which is always a worrisome thing, you know, and who knows who out there is just waiting for you to connect.

就连穷人也会拥有多个AI，我觉得将来还会有实体AI，但眼下主要是软件AI，能让他们活得更久、更轻松、更健康。

Then even the poor guys will have many AIs, also physical AIs I think, but at the moment especially software AIs that are going to make his life longer and easier and healthier.

而且这些AI都是他们自己的。

And he will own them.

他们不用再给其他人付钱。

He won't have to pay money to other guys.

我喜欢这样的未来。

I like that future.

好，我们先休息一下，接下来聊聊AI的身心理论，看看AI会不会感到疼痛，以及我们有没有自由意志。

Okay, let's take a quick break and on the other side of this talk a little bit about AI theory of the mind and body and whether AIs can feel pain and whether we have free will.

这些内容马上就会讲到。

So a lot of that stuff is coming up right after this.

大家好，我是亚历克斯·坎特罗伊德。

Hi everyone, Alex Cantroids here.

我想跟你们聊聊我和Gravity合作制作的一部纪录片，探讨AI代理的安全未来。

I want to tell you about a documentary I've made with Gravity to explore the future of AI agent security.

为了弄清楚我们是否真的准备好迎接自主代理，我采访了麻省理工学院的拉梅什·拉加尔教授、前白宫首席信息官特蕾莎·佩顿、米其林集团首席数据与AI官安比卡·拉贾戈帕尔，以及前阿里巴巴高管莎伦·盖伊。

To find out if we're truly ready for autonomous agents, I sat down with MIT professor Ramesh Rasgar, former White House CIO Theresa Payton, Michelin's Group Chief Data and AI Officer Ambika Rajagopal, and Sharon Guy, a former executive at Alibaba.

他们各自都对这个不断变化的领域提供了独特的见解。

They each offer unique insights into this evolving landscape.

最后，我们请来Gravity的首席执行官罗里·布伦德尔，一起探讨未来的方向。

We conclude with Rory Blundell, CEO of Gravity, to discuss the path forward.

在Gravity的引领下，欢迎加入我们的这场探索。

With Gravity leading the way, join us on this journey.

完整纪录片可以在节目说明里的链接观看。

You can watch the full documentary at the link in the show notes.

《Morning Brew Daily》每天早晨为你梳理商业领域最重要的新闻，让你轻松融入日常。

Morning Brew Daily breaks down the biggest news in business every morning so it fits seamlessly in your day.

我是托比·霍威尔。

I'm Toby Howell.

我是尼尔·弗莱曼。

And I'm Neil Fryman.

每天早上，我们会从最新的科技新闻聊到为什么现在没人买得起房子。

And each morning, we cover everything from the latest tech headlines to why nobody can afford a house right now.

听完每一期《Morning Brew Daily》，你都会变得更聪明，更有准备去应对周围的世界。

You'll leave each episode of Morning Brew Daily smarter and ready to take on the world around you.

有些人说，这是一天中最精彩的部分

And some people are saying it's the best part

他们的早晨必备。

of their morning.

因为我们知道一些你还不知道的事。

Because we know something you don't.

商业新闻不一定枯燥乏味。

Business news doesn't have to be boring.

加入数百万月度听众的行列，在你常用的播客平台收听《Morning Brew Daily》吧。

Join millions of monthly listeners and check out Morning Brew Daily wherever you get your podcasts.

我们又回来了，这里是《大科技》播客，今天我们邀请到施密特胡贝尔教授。

We're back here on Big Technology Podcast with professor Jorgen Schmidhuber.

教授，我想问您关于AI的一个问题——我觉得您在我们之前对话的开头似乎暗示过，AI也许能感受到痛苦，尤其是当它们无法达成我们设定的目标时，这种痛苦可能本身就是模型设计的一部分。

Professor, let me ask you this about AIs because I think you sort of suggested this in our earlier in the beginning of our conversation, that AI's might be able to feel pain, maybe if they're not able to accomplish the goal that we set that we set out for them, or maybe that can actually be baked into the experience of the model.

我想知道，您觉得AI要像人类那样，在任务失败时体验到类似痛苦的感觉，这种可能性离我们还远吗？比如，当AI被赋予一个任务，却没能按指令完成时。

I'm curious if if you think that that is far off that an AI can experience some something akin to what a human feels with pain, when they when they, for instance, are set out on a task and don't get don't accomplish it the way that they've been instructed.

我一直觉得挺有意思的是，很多人，包括计算机科学家，都说AI不可能感受到痛苦，但其实我们的AI从1990年就开始‘感受痛苦’了。

I always think it's funny that many people claim, even computer scientists claim, that AIs cannot feel pain because our AIs have felt pain at least since 1990.

那我们构建自主型AI时，到底在做什么呢？

So what do we do when we build an agentic AI?

我们建立一个能产生行为、改变世界的神经网络，环境不断提供新的输入，网络再做出反应，如此循环。这个智能体记住过去发生的事，努力调整行为，以最大化奖励信号、最小化痛苦信号。

An artificial neural network that produces actions that change the world and then new inputs are coming in from the environment and again there is an action and so on and the agent, the neural network remembers what happened before and is trying to figure out how to behave such that the sum of all reward signals is maximized and the sum of all pain signals is minimized.

那这些痛苦信号到底是什么？

What about these pain signals?

痛苦信号其实再自然不过了——只要我们造机器人，就会给它装上痛觉传感器。

The pain signals are the most natural thing because whenever we have a robot we give it pain sensors.

为什么？

Why?

因为这是一个会学习的机器人，它需要某种动力来学会保护自己。

Because it is a learning robot and so this learning robot needs some sort of motivation to learn to protect itself.

每当机器人撞上障碍物，相应的痛觉传感器就会激活，负值信号作为特殊输入传入机器人大脑，机器人接收到这些负值就会本能地避开。

So whenever the robot bumps against an obstacle, then, you know, the corresponding pain sensors wake up and negative numbers are special inputs to the robot brain and the robot sees then these incoming negative numbers and it is wired to avoid that.

它在学习如何生成避开痛苦的行为序列，同时也学习如何生成带来奖励的行为序列。

So it is trying to learn to generate action sequences that avoid the pain and it is trying to learn to generate action sequences that lead to the rewarding events.

也许机器人附近有个充电站，当电量低时，电池就会发出负值信号——就像饥饿和疼痛一样。

Maybe the robot has a charging station somewhere and whenever the battery is low, the negative numbers are coming from the battery: hunger, pain, hunger.

于是机器人的目标就是避开障碍，顺利到达充电站，然后停下来享受充电时的正向反馈——纯粹的愉悦感。

And then the robot's goal is to reach the charging station without bumping into obstacles and sit down there and enjoy the pleasure, just positive numbers, as the battery is being recharged.

所以，给这些机器人赋予这样简单的‘情绪’是最自然的做法。

So the most natural thing is to give these robots simple emotions like that.

这些情绪和痛觉，是学习过程中的关键要素，因为学习的本质就是让机器人产生更优的行为。

Now the emotions and the pain they are just crucial ingredients of the learning process because the learning is about generating better behavior for the robot.

所以机器人得知道什么对它有利，什么不利。

So the robot has to know what's good for it and what's not good for it.

疼痛信号其实就是告诉机器人哪些事情该避开。

So the pain signals are just informing the robot about what should be avoided.

疼痛是大自然和生物进化发明出来的一种机制，让动物有动力去学会躲避痛苦。

Pain is just an invention of nature and biological beings, of evolution which invented this pain sensor thing for animals such that they have an incentive to learn to avoid the pain.

我们几十年来一直在学习型机器中使用这套方法。

And we have done that for many many decades in our learning machines.

现在的情况是，这些学习系统已经能构建世界模型，预测未来——不只是眼前马上要发生的痛苦，比如手碰到烤箱的瞬间，而是能预测未来所有疼痛信号的总和，这正是强化学习机器在做的事，它们会放眼长远。

Now what's happening is then that these learning machines they have world models predicting the future, not only the immediate future, not only the immediate pain signal that I am going to feel right now when I touch with my hand the oven or something, no, they also try to predict the sum of all these future pain signals, that's what reinforcement learning machines do, and so they look ahead into the future.

于是就产生了一种新的情绪，不是当下这一刻的反应，而是面向未来的预期。

And so there is immediately this kind of secondary emotion which is not just the current moment, no, which is looking ahead.

比如，可能有个坏人时不时会进房间，一巴掌拍在小机器人头上。

For example, maybe there is a bad man which sometimes comes into the room and knocks the little robot on the head.

久而久之，只要机器人配备了强化学习系统和能预测未来的世界模型，它就能慢慢分辨出坏人和好人；一旦那个坏人再次出现，机器人通过人脸识别认出他，就会预判：如果现在不躲到窗帘后面，马上就要挨打了。

So over time, the robot is going to learn that if it has a reinforcement learning machinery on board and and a wild model that predicts and learns to predict, and then over time it will be able to distinguish the bad man from the friendly man, and then whenever the bad man appears again, I'm there and it does face recognition, then it predicts that very soon it's going to feel pain if it doesn't hide itself behind the curtain.

所以它会迅速躲到窗帘后面。

So it will rapidly try to hide itself behind the curtain.

作为旁观者，你会说：看，这个小机器人害怕了。

Now you as an outside observer will say, look, the little robot is afraid.

它表现出恐惧的情绪。

It has the emotion of being afraid.

但这其实只是传统机器学习最简单的副产品。

But it's just the most trivial side effect of traditional machine learning.

没错，我们的机器人早就有了各种各样的情绪，几十年前就有了；而且这些情绪不只是针对当下，它们还会展望未来。如果你让多个这样的机器人或智能体互动，很快就会出现类似喜欢甚至爱对方的行为——比如给它们一个单个无法完成、必须合作才能解决的任务，它们就得学会配合。

So yes, we have all kinds of emotions in our robots already and have had them for many decades, and then we also have these high level emotions which are not just about the current moment, no, they are looking ahead, you know, and then of course if you bring several robots or agents like that together, you immediately get stuff that is reminiscent of liking other robots or of loving them maybe because if you give them a task or a set of tasks that they can collectively solve but one of them alone cannot solve them then they will have to learn to work together.

自然地，每个机器人都会有动力去帮助对方，尤其是当对方生病或遇到困难的时候。

Of course suddenly each of them has an incentive to help the other one and each of them has an incentive to help the other one especially when the other guy is sick or something or has a problem.

去照顾它，关心它，这种行为的极致形式，人类可能会称之为爱。

To help him, to care for him and the extreme form of that a human might call love.

在一个讲究浪漫或类似亚洲文化的社会里，这不过是每个小机器人出于自身利益、想尽量减少痛苦信号、最大化愉悦信号的自然结果。

And in a society of romance or Asians like that, this is just a natural byproduct of the individual egoism of all these little guys who all want to minimize some of their pain sensors, sense sensor signals and maximize the sum of their pleasure signals.

所以利他主义，也就是你们所说的利他，其实就是这些学习型代理自我利益的自然结果。

So altruism, what you call altruism, is just a natural consequence of the egoism, of of the learning agents.

我觉得反方观点会是：机器人的痛苦不可能像人类感受到的那样，因为人类是有意识的，拥有神经系统，能将物理疼痛信号传递给大脑并确认为真正的疼痛；而机器人则是无感、无意识的，本质上跟计算器没什么两样。

So I think the counter argument would be that it can't be pain like the way that humans feel pain, because humans are conscious, humans have a nervous system that sends physical pain signals to the brain and registers as true pain, whereas robots are, the argument would be robots are unfeeling, unconscious, and effectively not that different from a calculator.

你会怎么回应这个观点呢？

How would you respond to that?

对，那么你是怎么判断一个人是否感到疼痛或害怕的呢？

Yeah, so how do you evaluate whether someone feels pain or is afraid?

通过观察它的行为。

By looking at its behavior.

如果它看起来像一只可爱的小机器猫之类的，而且它学会了每当坏人进来试图打它的时候就躲到窗帘后面，那么你就会说，很明显，这只小机器猫害怕了，它有了恐惧这种情绪，你还能要求什么呢？

And if it looks like a sweet little robot cat or something, and it has learned to hide behind the curtain whenever the bad man comes in and tries to knock it, then you will say, it's obvious, this little robot cat is afraid, has fear, the emotion of fear and what else do you want to do?

所以，从客观角度来看，我们无法区分追求奖励最大化的生物大脑的情感和追求奖励最大化的人工智能大脑的情感。

So there is no way of objectively seeing the difference between the emotions of reward maximizing biological brain and the emotions of reward maximizing artificial brain.

但我们确实有，比如，你刚才提到了爱。

But we do have, for instance, you talked about love.

比如，跟爱有关的化学物质，像催产素之类的。

Like, have chemicals that are associated with the feeling of love, like oxytocin.

机器人没有这些化学物质，所以有人会说，那根本是完全不同的感受，根本没法跟真爱相提并论。

So the robots don't have those chemicals, and hence the argument would be that it is a it is a, you know, a a completely different feeling that you could never compare to love.

你怎么看？

What do you think?

当然，大脑里用来编码恐惧或爱这类状态的化学物质，跟我们人工大脑里用的东西确实不一样。

Yeah, so of course the chemicals that are used in brains to encode certain states of fear or of love and whatever, they are different from what we are using in our artificial brains.

但背后的原理一定是一样的，毕竟都是为了实现智能行为。

But the principles must be the same, right, because it's about achieving intelligent behavior.

这到底是什么意思？

What does that mean?

意思就是找到更有效的方法去达成目标。

It means finding better ways of achieving your goals.

这又意味着什么？

What does that mean?

你一生中最重要的目标，就是避开疼痛信号和饥饿信号，比如每天按时吃三餐，在试图繁衍后代时获得奖励感。这些动机原本是生物进化为动物和人类编码在效用函数里的，而我们现在也用类似的方式，把这些激励机制植入人工智能系统，让它们更积极地解决问题、不断优化自己，最大化奖励、最小化痛苦。

The main goal in your life until the end of your life is to avoid these pain signals and hunger signals, such that you eat three times a day and get these rewarding signals during moments where you are trying to reproduce yourself for example and handful of objectives encoded in a utility function which was invented by biological evolution for the animals and for yourself and which in very similar form we are encoding in our learning agents to give them the same incentive to solve problems better, to make them better, maximize their own rewards and minimize their pain.

没错，意识这个问题也很有意思。

Yep, and then consciousness is an interesting one as well.

我最近刚和杰夫·辛顿教授聊过这个，有人认为，目前主流观点觉得像大语言模型这样的AI，不过是随机的鹦鹉而已。

There are, and I spoke about this recently with Professor Jeff Hinton, there are those that say, actually the prevailing view around AI is that AIs like LLMs are just stochastic parrots.

它们就是统计机器。

They're statistics machines.

你完全可以运行这些算法。

You could effectively, you know, run this algorithms.

它们即使不能完全被理解，至少大部分是可解释的，本质上就是统计预测工具，所以不可能有意识。

They're effectively, if not entirely interpretable, mostly explainable, and they're statistical prediction machines, hence, not conscious.

但你早就坚持认为它们是有意识的。

But you've been arguing that they're conscious for for quite some time.

对，确实如此。

So Yeah.

你到底是怎么得出这个结论的？

Tell me how you come to that.

是的。

Yes.

现在大家用的这些大型语言模型，本质上就是根据上下文预测下一个词，比如根据前面的内容猜下一个token，但它们太简单了，根本达不到我所说的、能让人称之为意识的那种核心条件。

So the the large language models that everybody is using today, which are basically about predicting parts of text from other parts of text, so, for example, predict the next token given the past, they are too simple for what I consider the they don't carry in themselves main reason for developing something that people might call consciousness.

那为什么它们看起来像是有意识的呢？

Why do they seem conscious?

因为它们读过所有关于意识的资料。

Well, because they have read everything about consciousness.

所有写过关于爱、意识、痛苦、冲突这些人类重要主题的书，全都被上传到互联网上，它们都读过了。

All the books ever written about love and consciousness and pain and conflicts and everything that is important to humans, which was put on the worldwide web, they have read that.

所以当你跟它们聊天时，它们特别擅长复述那些包含‘意识’这个词的、听起来特别有说服力的话——毕竟它们读过太多得过文学奖的小说，也读过大量相关作品，能说得头头是道，甚至能告诉你一些你从来没听说过的观点。

Which means that as you are interacting with them in a chat, they are very prone to repeat very convincing sentences that include the word consciousness in a way that is convincing, you know, because they have read so many literature prize winning novels about consciousness and other novels such that they can do a very convincing job there and they will tell you a lot about consciousness which you maybe even didn't know.

但它们并没有真正发展出自我意识的内在动力。我们再回头想想这两个网络系统：一个是负责生成行为的控制器，行为会带来环境的新输入，比如你动一下手，视频画面就变了；另一个是只负责预测这些变化的WAN模型。

However, they don't have their own motive to develop self consciousness in the following sense: Let's think back again of this two network systems where one is the controller that is generating the actions, where the actions then lead to new inputs from the environment, because if you move your hand like this, then the video changes and so on, and the other network which allows just to predict these changes, the WAN model.

所以你需要一个世界模型，通过心理模拟来规划未来，而不必真的在现实中执行所有这些动作序列——那代价太高了。

So you need the world model to plan your future through mental simulations without, you know, really executing all these action sequences in the real world, which would be very expensive.

这就是为什么我们需要这个世界模型。

So thats the motivation for this world model.

那么，意识这部分又是怎么切入的呢？

Now, where does now consciousness stuff come in?

虽然目前还没有人能给出意识的统一定义，但我现在要给你看一个非常简单的概念，它和很多人听到‘意识’或‘自我觉察’时心里想的完全吻合。

Nobody has a universally accepted definition of consciousness, but let me now show you something very simple, which is super compatible with what lots of people think about when they hear the word consciousness and self awareness.

现在，我们来看这个被用来规划某个智能体未来的世界模型——它正被用于心理实验。

Now, let's look at this world model which is being used to plan the future of some agent which is using the world model for mental experiments.

这个世界模型是一个深度神经网络，它学会了把所有见过的信息高效地编码到一群神经元里，比如环境中反复出现的东西，都会被抽象成某种原型表征。

Now, the world model is a deep neural network which has learned to encode everything that it has seen efficiently in a bunch of neurons, for example, everything that frequently appears in the environment gets internal abstract representations that stand for, you know, a prototype of that concept.

比如说，在一个充满各种人脸的世界里，你就会发现神经网络内部有一些单元，专门对应着‘人脸原型’，有的更偏向特定人脸，有的则更通用；同样，如果你的世界里有很多眼镜，就会出现专门识别眼镜的内部单元，能响应并编码眼镜这类东西。现在我们来看看规划过程。

For example, in a world where there are lots of different faces of different humans, then you will find units, internal units on these artificial neural networks that correspond to prototype faces and some of them are more specific for certain faces and less specific to other faces and so on, and so you in a world where you have lots of glasses, you will have glass detectors, internal representations, internal hidden units that learn to respond and encode glasses and whatever, And then lets now look the planning procedure.

控制器正在想办法，未来该怎么做，才能最大化奖励、最小化痛苦。

Now the controller is trying to figure out a plan, how should I act in the future to maximize my reward and minimize my pain.

然后它会用这个世界模型模拟各种可能的动作序列，从中选择那个能带来最大预期奖励、最小预期痛苦的方案。

And then it uses the Vaughan model for a mental simulation of different possible action sequences that it could execute and it going to pick the one that leads to the most predicted reward and the least predicted pain.

在这个过程中，模型中所有与当前问题相关的隐藏单元都会被激活，比如人脸、眼镜之类的。而有一个单元始终处于活跃状态，那就是代理自身。

So as it is doing that, it is waking up all kinds of hidden units in the one model that stand for whatever is relevant to the problem, faces or glasses or whatever, And there is one thing that is always active when the agent is active, is the agent itself.

当然，这些内部单元会代表代理本身，或者它的组成部分——如果有手，就会代表手；如果有轮子，就会代表轮子，等等。

So of course all kinds of these internal units are going to represent the agent or aspects of the agent and the hand of the agent if it has one or the wheels of the agent if it has one and so on.

所以每当代理在做这种规划时，它其实是在思考自己，激活关于自身的各种内部表征。

And so whenever the agent is making plans like that, it's thinking about itself, waking up these internal representations of itself.

这就叫自我意识。

And then it's self aware.

它具备自我意识。

It's self aware.

也就是说，比如它照镜子时，会立刻意识到：镜子里那个人，我能控制他的动作。

In this sense that, for example, if it looks in the mirror, it will quickly figure out, oh, the guy in the mirror, I can control what this guy does.

我怎么做，他就会做对应的反应，因为我能掌控他。

If I do this, then he will do the symmetric thing because I have it under control.

但如果你在房间另一头，我做这个动作，你却做出完全无关的反应，我就没法预测。

But if you are there on the other side of the room and I do this and you will do something that's completely unrelated, I cannot predict them.

这时候你就已经能看出‘能动性’这个概念了——能立刻分辨出这是自己的能动性，还是别人的能动性。

So there you already see this concept of agency which is immediately recognized as self agency or the agency of another guy.

还有一件事，其实早在1991年就有人提过了，就是有意识的东西往往会变成无意识的。

And now one more thing which also goes back to 1991 is this tendency of conscious things becoming subconscious.

很多人对各种事情都有点模糊的觉察，但根本不会深想，因为这些已经成了自动化流程的一部分。

So many people are aware of all kinds of things kind of vaguely without thinking much about that, because it is part of an automated process.

你每天开车从家到公司，走的都是同一条路，路上发生的大部分事情都很可预测，所以你根本不用专门想怎么开车，可能心思早飞到别处去了。

As you are driving always the same way from your home to your workplace, much of what happens is very predictable and so you don't even think much about the driving and maybe you're thinking about other things.

与此同时，你的注意力、内在的意识，却在悄悄聚焦在别的事情上。

At the same time your attention, your internal consciousness is somehow focusing on other things.

很多人都有这种体验。

Many people report that.

这其实是最自然不过的现象了。

And this is also one of the most natural things.

1991年的时候，他们有一个由两个网络组成的系统，其中一个我称之为有意识的问题解决器，这是一个神经网络，专门学习去预测那些它还无法预测的输入数据特征。

In 1991 they had a system consisting of two networks, one I call the conscious problem solver, a neural network that just learned to try to learn to predict certain aspects of incoming data which it was not yet able to predict.

它试图找出异常之处，而且确实面临一个需要解决的问题。

Trying to find irregularities and it had a problem to solve.

所以你可以说，它在某种意义上是有意识的，因为它必须学会新东西。

And so you could say it was conscious in the sense that there it had to learn something.

另一个网络则负责模仿高层网络找到的所有解决方案，它只是复制高层网络的隐藏层活动。

And then another network which basically learned to imitate all the solutions that the guy on the higher level found by just imitating the hidden units of the guy on the higher level.

现在这被称为行为蒸馏——把高层网络的成果提炼出来，交给底层网络去自动化执行，底层网络就是自动化器，它把高层发现的东西变成自动流程。

Today it's called distillation of the behavior of one guy into and then the lower level guy is the automatizer because it automates the stuff that the higher level guy finds, discovers.

所以高层网络仍然不确定，还在不断探索和形成洞见；一旦有了新发现、学会了新东西，就会被提炼成自动化流程。

So a higher level guy is still unsure, still working on creating insights and when these insights come and when it learns something, then it gets distilled down into this automatizer thing.

这两种意识特征都体现在这个系统里：一方面，是自我意识——它不仅在预测外部世界，还在预测与之互动的自身行为，从而形成对自我的认知；另一方面，是意识的聚焦机制——它只关注内部状态中那些尚未解决的部分，把已解决的和未解决的明确区分开来。

So both these aspects of consciousness where you have on the one hand self awareness in a world model that is not only predicting aspects of the world but also of the agent that is interacting with the world, which leads to self awareness of that kind, and this difference between the conscious stuff, so the internal consciousness which pays attention to certain aspects of the internal state but not to others, which focuses on what's still unsolved, where I still have to find a solution, and separates that from the stuff that is already solved.

我认为这两种特征在当年的系统里就已经存在了。2016年，我记得有家杂志采访我，说‘施密德胡伯声称人工智能在1991年就已具备意识’，大概就是这类报道。

So I think both of these aspects are there in these old systems, and in 2016 I believe I had an interview with magazine which then said Schmidhuber claims that AI became conscious in 1991, something like that.

而这正是在说这件事。

And this was exactly about that.

那场采访是十年前的事，但里面提到的内容其实要早得多，都快到1991年了。

That was ten years ago, this interview there, but it was actually referring to stuff that is much older, 1991.

那时候的意识系统虽然简单，跟人类没法比，毕竟当年的算力比现在贵一千万倍，我们只能做些很小的实验，比如我叫它‘意识分块器’和‘潜意识自动化器’，还有规划的原型之类的，系统里就几百个权重；而你，还有你，你们的大脑有上万亿个连接，能承载更复杂的意识状态，但我认为核心原理完全一样。

Conscious simple systems back then not as impressive as humans are obviously because back then compute was 10,000,000 times more expensive than today and we just had tiny little experiments, know, with this conscious chunker as I called it and the subconscious automatizer and the role models for planning and so on and we just had a few 100 weights in our systems while you and you and your brain, you have trillions of connections which can hold a much larger sort of consciousness, but I think the principles are exactly the same.

是的，这让我想到一个问题：当机器越来越接近人脑时，会不会改变我们对‘什么是人’的理解？

Yeah, and so that sort of brings me to this question which is as the machines get closer to the human brain, does it change the way we're gonna think about what it means to be human?

我觉得这会改变很多人对人类的看法。

I think it will change what many people think about humans.

但对我自己来说，大概不会有什么改变，因为我几十年前说的本来就是这样。

I guess it won't change much what I think about humans, because it's more or less what I said many decades ago.

当然，现在还有很多人坚持说AI不可能有情感之类的东西，可他们都是在事情发生几十年后才这么说的，我敢打赌，他们迟早会改口。

But yes, there are many people who claim that AIs can't have emotions and stuff like that, they do that decades after the fact, and they are going to change their minds, I'm pretty sure.

很多时候，这不过是个亲身体验的问题。

And often it's just a matter of direct experience.

你可能听说过那些可爱的机器人海豹，有些医疗中心会用它们，和这些明显很人工的机器人互动的人，往往会对他产生很深的情感依恋。

So maybe you have heard of these little sweet cute robot seals that you have in certain healthcare centers, and the people who are interacting with these very artificial beings, they get really emotionally attached to them.

他们真的很喜欢这些机器人，会跟它们玩，尽管它们根本不算聪明，几乎学不到什么东西。但就连这么简单的机器人都能让人产生近乎爱的感觉，那你可以想象，一旦出现真正逼真、特别可爱的机器人，它们更像小动物，却又能做一些传统生物动物做不到的事，那时候会发生什么。

They really like them and they play around with them, although they are not smart at all, you know, they don't learn much, if even such a simple robot can invoke feelings of, you know, almost love or something, then you can imagine what will happen once you have really convincing, very sweet little robots that are more like little animals, except that they can do maybe a couple of things that these traditional biological little animals cannot do.

不对。

No.

我记得在2017年或者2018年《彭博社》那篇你提到的文章里，有人问你，你是不是觉得我们活在某种模拟世界里。

I think in the 2017 Bloomberg article, or 2018 Bloomberg article that I referenced, you were asked whether you think, you know, we are living in some form of simulation.

你说是的，因为这是对人类为何执着于进步的最简单的解释——我们会不断造出更强大的计算机，直到我们自己被淘汰，或者选择和智能机器融合。

You said that's what I think because it's the simplest explanation of everything that humankind is programmed to chase progress, and we'll keep making more powerful computers until we make ourselves obsolete or decide to merge with the smart machines.

这里有一段原话。

Here's a quote.

你要么变成一种和人类截然不同的存在，要么就为了怀旧而继续当人类，但那样的话，你就不会再是能做重大决策的人了。

Either you become something that's really really different from a human or you stay as human for or you stay as a human for nostalgic reasons, but then you will not be a major decision maker.

你不会在塑造世界的过程中扮演任何角色。

You will not play a role in shaping the world.

你能再详细说说吗？这些年你的看法有没有被加强或者改变了？

Can you expand upon that and tell us whether your opinion has been reinforced or changed in the interceding years?

没有，我的观点一直没有变。

No, my opinion has not been changed.

那时候，其实几十年来，一直有人在讨论把人类的大脑，或者说灵魂，上传到电脑里，然后在某种模拟的天堂里生活，或者活在一个能与现实世界互动的机器人身上。

You know, back then, and actually for many decades, there has been talk about uploading human brains or souls, if you will, into computers, and then live your your future life in some sort of simulation simulated paradise or in a in a robot that interacts with the real world or so.

我相信这类科幻故事最早是1964年发表的，当时有人声称把意识上传到了电脑里，用的还是旋转磁带那种老式设备。

And I believe the first story, a science fiction story of that kind was published in 1964 where someone was able to upload his mind in a computer back then with rotating tapes and everything.

那部作品叫《模拟三号》，作者是丹尼尔·法卢伊。从物理上讲，我们没有理由否定这种可能性。

That was, what was the name, call it Simulacron three, something like that, it was by Daniel F Galouy and so there is no physical reason to reject the notion that this might be possible.

目前还做不到，除了某些非常简单的生物，比如苍蝇。

At the moment it is not possible except for certain kinds of very simple animals like flies.

到2024年，可以说苍蝇的大脑已经被成功上传了。

So apparently as of 2024 you can argue that a fly brain has been uploaded.

也许不是完整的苍蝇大脑，那些学习机制和神经元还没完全复制，但模拟出来的苍蝇已经能做出和真实苍蝇几乎一模一样的行为——在虚拟环境里，它就像继续活着一样。

Maybe it's not the full fly brain with all the learning algorithms or the neurons which are in there, but in simulation the fly, the simulated fly now does stuff that is very much like what the real fly did before it was uploaded, before all the connections were read and uploaded into this computer, virtual environment where it kind of lived on.

没有明显的理由认为，不可能读取人脑中所有的突触连接，理解那些不断改变突触的学习机制，然后在计算机里复刻出来。这样一来，你的意识或灵魂就被上传了，永远生活在这样一个模拟宇宙里，它或许还能与真实世界保持联系。

There is no obvious reason to believe that it is impossible to read all the synapses of a human brain and understand also the learning algorithms that are changing the synapses all the time and then replicate that in a computer and then the idea would be that your soul or your mind is uploaded and living there forever in this simulated universe, which may have contact to the real universe.

当然，一旦我们接受了这个前提，接下来自然就会想：下一步该怎么做？

Now of course, once, we accept accept this premise and then we think what is the next step.

假设你已经被上传了，突然间你有了机会，可以拥有不止两只眼睛——也许是一百万只眼睛，遍布全球的卫星眼；你的大脑也可能远超现在，不再是每秒十的十八次方次运算，而是提升十的十次方倍。

Now suppose you are uploaded there and suddenly you have the opportunity to have more than two eyes, maybe have a million eyes, satellite eyes all around the planet and you know, maybe you have a much bigger brain, not just maybe 10 to the 18 instructions per second, something like that, but 10 to the 10 times as much.

这时候你有两个选择：要么屈服于这种新生活的诱惑，结果你将彻底变成一个完全不同的存在。

Now there are two things you can do: either you succumb to the temptations of this new life and in the process you are going to become something very very different.

你不可能再像从前那样了，因为一切都在无限扩张，偶尔你或许会想起自己曾经是个凡人，但你的未来生活已经完全脱离了人类的轨迹，很可能深受其他类似扩展意识的影响。

You are not going to remain a lot like you were because suddenly everything expands in a way and sometimes you might remember your roots as a human somewhere, but your future life is totally detached and disconnected and probably is highly influenced by other expanded minds like that.

你将讨论一些在现实世界里，作为人类时根本不可能触及的问题。

Discussing problems that you would have never discussed as a human person in this reality.

另一种选择是，出于怀旧，你决定不被这些诱惑牵着走：我保留两只眼睛，保持原来的大脑，不把它扩大一万亿倍。但问题是，别人会忽略你——因为他们掌握了你根本没有的新能力，真正的决策者，都是这些被增强的意识。而他们自己，还要和那些原生AI竞争，那些AI没有进化带来的包袱，可能更适应未来的环境。一旦AI的领域从我们的生物圈扩展到太空，谁知道他们会干出什么来。

Now the alternative is, for nostalgic reasons, you say I don't want to succumb to these temptations, I keep my two eyes and I keep my little brain and I don't increase it by a factor of 10,000,000,000 or something, but then your will be ignoring you, because they will have so many new skills that you don't have, and the main decision makers, they are going to be these expanded minds, and they themselves, they will be in competition with the native AIs which you know are not they don't have this evolutionary ballast and maybe much more adapted to the needs of the future, you know, once the AI sphere is expanding from our biosphere into space and who knows what they are going to do there.

所以，要么你怀旧，变得无关紧要；要么你就融入这个由AI主导的新社会——其中大多数是纯AI，少数或许有来自人类的痕迹。但真正塑造未来、主导一切的，将全是这些新存在，而不是那些还像人类的个体。

So either you are nostalgic and remain irrelevant, or you become part of this growing sociology, if you will, of AIs, which are mostly AIs, and some of them may have human roots or whatever, but almost all the decision making process and the universe shaping things, they are going to be done by these new beings and not by the human like beings.