Rich Sutton 和 Khurram Javed：为什么 AI 模型会停止学习，以及如何重新开始

你可以通过提供更多上下文来改变模型的状态。

You can change the state of the model by giving it more context.

所以它已经学会了，如果状态不同，如果状态有新的内容，它就会用这个来做下一个预测，但模型正在遗忘。

And so it had already learned that if the state is different, if the state says something new, then it will use that to make the next prediction, but the model is unlearning.

Cursor 的 Tab 自动补全模型，你知道的，

Cursors, tab, autocomplete model is, you know

对，

Yeah,

所以它确实会根据……更新，

so It does get updated based on

那些模型。

Those models.

那些权重会改变。

Those weights change.

那些权重会改变。

Those weights change.

这两个是例子。

Those are two examples.

比如 Cursor 的 Tab 补全，还有我觉得 Composer 也在更新。

Like, Cursors, tab, and I think the Composer, they were also updating.

对。

Yep.

这两个都是持续学习的例子。

Those are two examples of continual learning.

但它可以做得更好。

But it can be much better.

据我了解，他们的方法是利用大量用户在使用 TAB 功能时产生的数据。

So the way they do it, as far as I understand, is a lot of people are using TAB.

他们会收集来自成千上万甚至数百万用户的数据，然后基于这批数据对模型进行一次更新。

They collect all this data, so coming from millions of users or thousands of users, and then they do one update of the policy from this batch data.

这种方法可行，但想象一下，如果我想教这个模型一些特定的内容呢？

So this could work, but imagine I want to teach this model something specific.

我不想和十万个其他人争论他们想教模型什么。

I don't wanna fight with 100,000 other peoples about what they wanna teach their models.

我想教我的模型一些非常具体的东西，而且我想在我的模型版本上做这件事。

I wanna teach my model something very specific, and I want to do it to my version of the model.

我不在乎模型从别人那里学到的共享知识。

I like, I don't care about the shared knowledge that the model has coming from other people.

所以这样做效率很低。

And so it's a very inefficient way of doing it.

看起来目前的做法是，有一些基础技能可能是在权重中学习的，对所有人都一样，然后个性化是通过上下文来实现的。

It seems like the way that this is currently done is that there's fundamental skills maybe that are learned in the weights that are common to everybody, and then there's personalization that happens in the form of context.

对吧？

Right?

对。

Yep.

对。

Yep.

这不是学习应该有的正确思维模式吗？

That not the right mental model for how learning should work?

比如，所有的上下文都应该存在权重里吗？

Like, should should all the context live in the weights themselves?

上下文也可以存在状态里。

So context can be in the state too.

可能两者都有，但你仍然需要能够更新权重。

Could be both, but you still need to be able to update the weights.

所以，我给你举个例子，一些很好的应用案例是关于人类残疾的。

So if I give you an example, some really good use studies are for, with human disabilities.

当人类经历一些改变他们思维或感官的事情时，你可以看到他们适应。

When human go through something that changes their mind or or some sensors, you can see them adapt.

比如，我们有本体感觉。

So for example, we have proprioception.

我们有内部传感器告诉我们身体的位置，我们用这个来走路。

We have internal sensors that tell us where the how the body is positioned, and we use this for walking.

有些人的这种能力会完全丧失，彻底失去，然后他们就完全没法走路了，因为那正是他们行走机制的基础。

There are cases where people lose this ability completely, completely, and then they can't walk at all, because that is literally the foundation of their walking policies.

这种能力已经深深扎根在大脑里，但经过两三年，他们可以通过看着自己的脚来重新学会走路。

It is ingrained in the brain, but then over the course of two, three years, they can learn to walk again by looking at their feet.

也就是通过视觉反馈来实现这一点。

So visual feedback through that.

所以大脑的可塑性极强，它能学会很多东西。

So brain is insanely plastic in the sense that it can learn a lot of things.

有些情况可能持续了二十年，当它不再成立时，系统可以去更新并摒弃旧有的认知。我认为这种能力非常实用，也是我们希望系统具备的。

Something that has been true for twenty years, when it stops being true, it can go and update that and get rid of that, And that is the capability, I think, that's extremely useful we would want in our systems.

从人类婴儿或动物的学习方式中，我们能学到什么？你从中汲取了多少灵感？

What is there for us to learn from how human babies or animals learn, and how much inspiration do you take from that?

我们确实借鉴了很多。

Well, we take a lot of inspiration.

但我们并不要求人工智能必须像自然系统那样运作，比如像婴儿、人或动物那样行为，它更多是作为一种灵感来源。

We don't take it as a requirement that AI has to behave like the natural system, like babies or people or animals, but it's a source of inspiration.

从动物学习中获取灵感，但不是限制。

Inspiration but not constraint from animal learning.

这和《苦涩的教训》不一致。

Inconsistent with The Bitter Lesson.

对。

Yeah.

你认为我们应该从生物学习的方式中汲取哪些灵感，而这些灵感在当今的系统中并不存在？

Where do you think we should most seek to draw inspiration from the way that biological learnings works that is not present in today's systems?

我觉得我现在只是在发表意见，但这些意见都是显而易见的。

I feel like I'm just giving opinions now, but they're just obvious opinions.

所以我认为很明显，没有动物是通过监督学习来学习的，因为我们没有关于肌肉应该如何抽搐的示例，而那是我们的输出。

So I think it's apparent that no animal learns by supervised learning because we don't get examples of how our muscles should twitch, and that's our output.

但学校里的所有学习都是监督学习。

But all of school is supervised learning.

我知道，绝对不是。

I know, absolutely not.

但即便如此，学校也只是我们所学的一小部分。

But even if it was, school is like a tiny fraction of what we learn.

比如我们学会看、学会走路、学会世界如何运作。

Like we learn to see and we learn to walk and we learn how the world works.

但即使在校园里，也没人告诉我们该怎么抽动肌肉。

But even in school, no one tells us how we should twitch our muscles.

我获得的知识技能来自监督学习，所以……

The knowledge skills I acquire were from supervised learning So in

我不想说向他人学习、从他人那里传承不重要。

I don't want to say that learning from others, transmission from others is not important.

这极其重要，语言也极其重要。

It's like extremely important and language is extremely important.

但我们缺少的是什么呢？

But what are we missing?

没有监督学习。

There is no supervised learning.

没有人给我们设定目标。

There's no targets that are given to us.

你知道，你会听到正确答案，比如，法国的首都在哪里？

You know, you hear the right answer is, you know, where is what's the capital of France?

然后我们知道答案是巴黎。

And we know the answer is Paris.

好，但没人告诉我'巴黎'这个词该怎么读。

Okay, but no one tells me how I should pronounce Paris.

你说答案是巴黎，我听你说，也听到你的话，然后我会自己做出一些其他的肌肉动作来发出'巴黎'这个答案。

You say the answer is Paris and I listen to you and I hear your words and I will make some other muscle motions to produce the answer Paris.

这其实并不算是真正的监督学习。

It's not literally supervised learning.

总之，对。

Anyway, yeah.

所以我觉得确实是这样。

So I think it's really true.

我是说，好吧，不管怎样，首先，学校并不是关键。

I mean, well, anyway, the first thing is the school is irrelevant.

比如说，你知道，松鼠不会去学校学这个。

Like, you know, squirrels don't go to school and learn that.

它们可能会。

They might.

动物不是那样学习的。

Animals don't learn that way.

而且学校是非常特殊的东西，直到几百年前我们甚至都没有，你知道，我不确定具体时间。

And school is a very special thing that even we didn't have up until, you know, I don't know, a few hundred years ago.

但这不是唯一进化出来的方式，它只是……

But isn't it only where it evolved It's

不是智能的本质部分。

not part of the essence of intelligence.

把学校当作学习的主要例子是一种干扰，因为那并不是我们过去做的事情。

It's a distraction to think of that as your primary example of learning is this thing which we didn't do

作为动物来说。

as animals.

我真希望她还在世，能跟我父母说说这件事。

I wish she had been around to tell my parents that.

他们逼我去上学，去应付那些各种各样的

Forced me to go to school and deal The with all the

事情就像松鼠很擅长从树上跳下来，但松鼠可不会证明数学定理。如果我想学怎么证明数学定理，我就得去学校。

thing is like squirrels are wonderful at jumping off trees but squirrels can't prove math theorems and if I want to learn how to prove a math theorem, I go to school.

对。

Yeah.

他们也没有DVD和iPod这些玩意儿。

They also don't have DVDs and iPods.

你知道的，有很多东西都是这样。

You know, there are a lot of things.

他们能做一些我们做不到的事情。

They can do things that we can't do.

数学定理，是的，而且它们不会下棋。

Math theorems, yeah, and they don't play chess.

你知道吗，这有点像莫拉维克悖论。

You know, it's sort of like Morvik's paradox.

有些我们认为是高智商的事情，对计算机来说其实很容易，而那些日常的事情，比如移动、有意识地看东西等等，反而很难。

They're these advanced things that we think of as really intelligent, but, they're sort of easy for computers to do as opposed to all these regular things that are hard, like moving and seeing with attention and everything.

我认为监督学习是件好事。

I think supervised learning is good thing.

你知道，顺便提一下，我喜欢寻找显而易见的东西。

You know, just mention, I like to think look for obvious things.

没有人能通过给我们例子来教我们如何抽动肌肉，因为他们根本做不到。

No one tells us how to twitch our muscles by giving us examples because they couldn't possibly.

因为我们必须自己抽动肌肉。

Because we have had to twitch our muscles.

我们必须自己摸索出来。

We've had to figure that out.

是的。

Yeah.

而且他们的答案会是错的，对吧？

And their answer would be wrong, right?

所以如果我移动我的嘴巴、舌头和声带，完全按照里奇发“巴黎”这个词的方式，我确信发出来的声音会非常不同。

So if I move my mouth and my tongue and my vocal cords exactly the same way that Rich does to pronounce Paris, I'm sure a very different sound would come out.

所以从某种意义上说，里奇或者任何人都不知道用我的身体发出正确声音的方法。

So in some sense, Rich or no one knows the right way of producing a sound with my body.

只有我知道。

Only I know that.

是的。

Yeah.

在我看来，许多最原始的、感觉运动的能力，尤其是与物理世界中的运动相关的能力。

It seems to me that many of the most I guess the most raw, like, sensory motor capabilities, especially related to movement in the physical world.

我同意你的看法，这似乎确实是本质上需要从经验中学习的东西。

I agree with you that that seems something that is inherently learned from experience.

不过在我看来，有些更高层次的抽象让我们更接近人类之所以伟大的地方。

It seems to me, though, that there are higher levels of abstraction that bring us closer to, you know, what makes humans great.

而这些大部分并不存在于这种低层次的感知运动学习中。

And much of that doesn't live in this low level of sensorimotor learning.

你的世界模型，我猜，是不是涵盖了从感知运动学习一直到更高层次的所有内容？

Does your world model, I guess, span sensorimotor learning all the way up?

对。

Yeah.

这就是我们的目标。

That's the ambition.

当然。

Absolutely.

而且顺便说一句，松鼠也能做一些非常抽象的事情。

And squirrels, by the way, can do some enormously abstract things.

松鼠能做的最酷的事情是什么？

What's the coolest thing a squirrel can do?

它总能钻进你的喂鸟器，不管你怎么设障碍。

Well, it can always get into your bird feeder no matter what obstacles you put in the way.

你知道的，它能找到新的跳跃和攀爬方式，做各种各样的事情。

You know, it can find new ways to jump and climb and do lots of things.

它们的轨迹计算能力相当不错。

You calculate trajectories pretty well.

动物似乎很擅长理解物理世界，而我们人类在想象自己飞向太空时，却要进行那些复杂的心理计算。

Animals are pretty good at understanding the physical world without the mental calculations that we think we are doing when we think about launching ourselves into space?

它们能在坠落时实时做出正确的反应，从而避免受伤。

Breaking a fall, they can do it in real time in the right way to prevent injuries.

好吧，有道理。

Okay, fair enough.

我觉得这只是一个程度问题。我倾向于认为其他动物和人类非常接近。

I think it's just a question of degree between I like to think that other animals, are are very close to humans.

刻意强调我们与动物的不同之处，我觉得有点傲慢了。

I think it's hubristic to try to emphasize what we do differently, you know, how we're different from animals.

最好还是看共同点。

It's better to see the commonalities.

而且我觉得我们只是程度上的差别。

And I think we are just a question of degree.

是程度上的差别，当然，社会和文化给了我们很大的优势。

It's degree and, of course, society and culture give us big advantages.

语言给了我们很大的优势。

Language gives us big advantages.

我能就这一点追问一下吗？

Can I just push on some of this?

对。

Yeah.

好。

Good.

因为我想先退一步，索尼娅。

Because I want to back up, Sonya.

所以我相信动物和儿童从经验中学习，并且通过经验学习能做到令人难以置信的事情。

So I believe animals and children learn from experience and do incredible things learning from experience.

当我儿子两三岁或四岁时，我就想，哇，这真的很有趣，我儿子能学会这些东西，而没有人真正教他怎么做。

When my son was two or three or four, I'm like, wow, this is really interesting that my son can learn these things without nobody really teaching them how to do these things.

但同时，索尼娅所说的，是让人类独一无二的能力，比如能去外太空，造火箭。

But at the same time, what Sonya is saying is like, makes human uniquely human to be able to go to outer space, build a rocket.

这些不是百分之百从经验中学来的，因为在发射火箭之前，你实际上必须进行抽象思考，这种方式不是从所谓的经验中学来的，因为你不知道它是否会成功。

Those are not things that are learned 100% from experience because before you launch the rocket, you actually have to abstract thinking through it in a way that is not learned from quote unquote experience because you don't know if it's going to work or not.

你必须想象它。

You have to imagine it.

我们如何教机器想象以前不存在的事物？

How do we teach a machine to imagine things that were not available before?

这可能是我们试图推动的事情，因为我们还不太理解。

That's probably the thing that we're trying to push on because we're not quite understanding We're

我们同意你的看法。

gonna agree with you there.

你得能规划。

You have to be able to plan.

你得能想象。

You have to be able to imagine.

对。

Yeah.

你觉得一千年前的人类，在他们还没做出我们说的这些事之前，他们有那么聪明吗？

Would you say that humans a thousand years ago, before they had done most of the things that we're talking about, were they as intelligent?

比如说，如果那个时代的人接触到这种新文化，他们能学会同样的技能并开始做有用的事情吗？

If, for example, someone from that era was exposed to this new culture, would they be able to get the same skills and start doing useful things?

即使在过去的几万年里，我不认为人类的大脑进化了很多。

Even over the last ten thousand years, I don't think the human brain has evolved that much.

本质上还是同样的机器。

Fundamentally the same machine.

本质上还是同样的机器，但我们积累了上万年的知识。

Fundamentally the same machine, but we've built up ten thousand years of knowledge.

我通过上学，在监督学习下，就能学到一万年的知识，这比通过经验学习要快得多。

I get to learn ten thousand years of knowledge by going to school through supervised learning, and I get all that much, much faster than trying to learn through experience.

对。

So Right.

所以我觉得你说得完全对。

So I think you're, like, totally right.

所以我们希望我们的系统能从经验中学习，而它们的经验之一就是接触我们的文化，然后从中学习。

So we we would want our systems to learn from experience, and part of their experience would be getting exposed to our culture and then learning from about our culture.

它们应该从中学习。

They should learn from that.

这都没问题，但让我们谈谈有人做出范式转变的事情。

That's all good, but let's talk about when someone goes and does a paradigm shifting thing.

大家都举爱因斯坦的例子，但我觉得还有很多其他例子。

So everyone gives the example of Einstein, but I think there are many examples.

学习也是这样的，就像学习东西和编程东西的区别。

Learning is that too, like looking at learning thing versus programming things.

所以当这些范式转移发生时，我认为是一个人类积累了所有这些知识，然后基于他们的经验构建新的抽象概念，用它们进行规划，进而发现新知识。这种提出新抽象概念、学习更多模型并用它们进行规划的能力，在我们目前的系统中完全缺失。你既可以在人类知识的边缘观察到这一点，也可以在感觉运动流的层面研究这个问题。

So when these paradigm shifts happen, I would say it's a human who has accumulated all this knowledge, and then from their experience, they're building new abstractions, they're planning with them, and then they're discovering new knowledge, and that skill of coming up with new abstractions and then learning more models and planning with them, that skill is totally missing in our current systems, and you can expose this at the edge of human knowledge, but you can also study this problem at the sensory motor stream level.

所以我们并不是在质疑这个原则。

So we're not arguing with the principle.

我们需要形成抽象概念，以便能在高层级进行推理。

We need to form abstractions so we can reason at a high level.

你们正在接近做那件我说过我们绝对不该做的事，也就是争论：是先验知识重要，还是获取知识重要？

You guys are coming close to doing that thing that I said we should never do, which is argue, is prior knowledge important or gaining knowledge important?

你们刚才说的就是这个意思。

You know, that's what you guys just said.

你说你还是要继续学习新东西。

You said you're still going to have to learn things.

你还提到，可以从自己的文化背景和既有知识里获取养分。

And you were saying, Oh, I can get things from my culture and from prior knowledge.

但这些来源之间不应该互相冲突。

But these should not fight for each other.

关于范式转换这一点，我们怎么制造出能理解何时该转换范式的机器呢？

On the exact thing around paradigm shifts, how do we create a machine that understands when to shift the paradigm?

对，通过它自己的经验来思考，对吧？

Yeah, think through its experience, right?

所以它必须通过

So it would have to, through

自身的经验来实现。它不能依赖人类的知识，因为我们假设人类看到的是某种范式，而我们希望有一种不同的看待事物的方式。因此，它必须通过自身经验找到一种更好的方法。

its own experience, it can't rely on human knowledge because we're assuming the humans see one paradigm, and we want a different way of looking at things, and so through its experience, it has to find something that is better.

也许它的泛化能力更强，预测也更准确。

Maybe it generalizes better and makes better predictions.

也许它在其他方面表现更好，但必须是通过它自身的经验获得的。

Maybe it's better in some other ways, but it has to be through its own experience.

我们领域目前还没看到的一个重大挑战，或者说尚未具备的能力，就是学会一个模型，然后基于这个模型进行规划。

The big challenge that we don't see in our field, the ability we don't see in our field yet, is the ability to learn a model and then plan with a model.

我们可以做数学计算，也能下AlphaGo，因为那些游戏的规则模型我们是已知的。

We can do the math things, we can do AlphaGo because the games, we know the model.

我们知道每一步棋是怎么走的。

We know how the moves work.

在数学里，我们知道运算符是什么。

And in math, we know what the operators are.

我们知道Lean能让我们从一个知识状态推进到证明的下一个状态。

We know lean will take us from one state of knowledge to the state of the proof to the next state.

但如果必须学习模型，就没有——我直说了。

But if we have to learn the models, there are no I'm going to say it.

也许有个奇怪的例子，或者反例，但在我看来，在我们这个领域，还没有先学习模型再用模型做规划的例子。

There's probably maybe a weird example, a counterexample, but I can see that there's no instances of learning the model and then planning with the model in our field.

至少不是那种自我发现的抽象概念。

At least not with, like, self discovered abstractions.

所以有人说，我就学习下一秒或下一毫秒发生什么的模型，但这不是我们模型的工作方式。

So there are people who say, I'm just going to learn a model of what happened in the next second or next millisecond, but that's not how our models work.

我们的模型更抽象。

Our models are more abstract.

我们的模型很不一样。

Our models are quite different.

所以我喜欢你在这里做的事情的一点是，你不是坐在那里空谈或者抱怨世界的现状。

So one of the things I like about what you're doing here is you're not just sitting around pontificating or lamenting the state of the world as it is.

你非常注重行动。

You're very action oriented.

这就是你创办公司的原因。

It's why you started the company.

那我们就开始聊聊这个吧。

So let's let's start talking about that a bit.

2022年，Rich，你提出了一个非常具体的12计划，即阿尔伯塔AI研究计划。

In 2022, Rich, you laid out a very specific 12 plan, the Alberta plan for AI research.

也许跟我们说说那个计划吧。

Maybe tell us about that.

阿尔伯塔计划之所以产生，是因为我们确实有一些总体想法，但我们也需要把它们分解成更小的部分。

So the Alberta plan came about because we did have general ideas, but we also needed to convert them into smaller chunks.

所以这十二个步骤，就是想把这些具体的想法一个个提炼出来。

And so the 12 steps are the attempt to crystallize particular chunks.

其中有一个非常重要的早期步骤，就是第二步：持续深度学习。

There's a very important early step, step two, which is continual deep learning.

我们觉得这一步几乎是最重要的，因为它能带动其他所有环节。

And we think that one is like almost the most important because it unlocks everything else.

如果你能做到持续深度学习，你就能不断更新你对世界的模型认知。

If you could do continual deep learning, you could then continually update your model of the world.

然后如果你知道怎么把抽象做对，后面的那些步骤全都是关于怎么正确地进行抽象。

And then if you knew how to do the abstraction rights, like the second half of the steps are all about how to get the abstractions right.

并不是说把抽象做对就行——我的意思不是说要去找到正确的抽象，因为没人能说得清什么是正确的抽象。

Not only my abstractions right, what I mean, I don't mean get the right abstractions because no one can say what the right abstractions are.

那取决于它所处的世界。

That depends on the world that you're in.

你的智能体必须学会在任何它所处的世界里，掌握正确的抽象方式。

Your agent would have to learn the correct abstractions for whatever world it's in.

所以，可能这两点就是关键。

And so maybe those are the two key things.

你得找到正确的抽象概念，然后还要能进行持续的深度学习。

You have to find the right abstractions and then you have to be able to continual deep learning.

我觉得领域里很多人都意识到我们需要模型，需要用它来规划，但抽象概念决定了模型应该基于什么条件。

I think that a lot of the people in the field realize that we need models, we need to plan with them, but the abstractions tell us what the model should be conditioned on.

那么，模型到底应该预测什么呢？

What should you So what should the model predict?

你打算怎么做？

What are you gonna do?

然后就会发生一些事情。

And then something is gonna happen.

更重要的是，这些会从哪里来呢？

And more importantly, where would that come from?

所以我特别喜欢用精英运动员的例子来说明。

So I really like the example of elite athletes.

如果你问精英运动员他们是怎么做某些事情的，他们会用一些奇怪的专门术语来描述非常具体的动作。

If you ask elite athletes about how they do certain things, they would have weird niche terminology for doing very specific things.

他们会说，你知道，我做这个动作，如果他们交流的话，会给它起个名字。

They were like, you know, I do this thing, and they would have a name for it if they communicate.

有时候他们独自练习时，甚至都不给动作起名字。

Sometimes they don't even have a name for it if they're just doing it alone.

那他们是怎么想出这些抽象概念的呢？

So how did they come up with those abstractions?

从某种意义上说，这正是缺失的关键部分，而阿尔伯塔计划的后半部分回答了这个问题。

That's, in some sense, a crucial thing that's missing that the later half of Alberta plan answers.

我们能聊聊持续深度学习这部分吗？

Can we talk about the continual deep learning part?

这到底是目前存在的算法上的差距，还是仅仅是实际部署、基础设施和数据隐私方面的差距？

Is it an algorithmic gap that exists today, or is it just a practical deployment infrastructure data privacy gap?

因为如果我想做，比如说，根据用户交互来朴素地更新权重，我现在就能做到。

Because if I wanted to do, call it naive updating of weights based on user interaction, I can do that today.

那你觉得，要实现持续深度学习，我们最缺的是什么？

And so what in your opinion is the biggest thing that we're missing to kind of get to continual deep learning?

对，这绝对是一个算法上的差距。

Yeah, so it's absolutely an algorithmic gap.

你可以做那种简单粗暴的做法，但那样会遇到各种问题。

You can do the naive thing, but then you'll see all sorts of problems.

比如说，如果你说，我拿一个样本，然后用这个样本来更新整个模型，你就会遇到一个问题，就是模型里之前学到的所有知识都会受到负面影响。

So for example, if you say, I'm gonna take one sample, and then I'm gonna update my whole model with that one sample, you will run into this problem that now all of the previous knowledge in the model, it's impacted negatively.

目前我们绕过这个问题的方法，其实就是Cursor的做法。

And the way currently we get around this is exactly what Cursor does.

他们不用单个样本。

They don't use one example.

他们用的是来自大量用户的大批量数据。

They use a large batch coming from a lot of users.

所以在那些能用这种方式的场景下，你就能做持续学习了。

So in use cases where you can have that, you can do continual learning.

但大多数使用场景下，你并没有那种条件。

Well, most use cases, you don't have that.

大多数情况下，你只有单一的数据流，如果直接用那种简单粗暴的方法，就会彻底摧毁你之前的知识，破坏性非常大。

Most use cases, you have a single stream of data, and then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very destructive way.

灾难性遗忘。

Catastrophic forgetting.

对。

Yep.

确实如此，是的。

That is so, yeah.

但这完全是可以解决的。

But it's totally curable.

你必须写出来。

You have to to write out

那怎么解决呢？

What's the cure?

对，没错。

Well, yeah, exactly.

那怎么治呢？

What does it cure?

首先，你需要做所谓的步长优化，这意味着网络中的每个权重都要有独立的步长。

Well, you know, first you need to do what we call step size optimization and it means every weight in your network has to have a separate step size.

有些权重更新得快，有些更新得慢。

So some will move fast, some will move slow.

而且你需要为每个权重元学习这些步长。

And you will have to meta learn these step sizes for each weight.

网络中的大部分权重步长都很小，这样在训练新样本时，它们不会被破坏。

Most of your network will have weights that have tiny step sizes so then when you train on a new example, they don't get destroyed.

只在合适的地方更新。

Happens just to the right places.

其次，你需要使用某种在特征空间中的生成和测试方法。

And then secondly, you have to use some form of generate and test which is in feature space.

所以你要提出新的特征或新的单元，而不依赖梯度。

So you come up with new features or new units without following gradients.

因为梯度是一个很慢的过程。

Because gradients are a very slow process.

你只有在确定方向有帮助时才移动，这总是很慢，而且无法让你变得更复杂，也无法持续学习。

You only move in a direction if you know it's the helpful one and that's always going be very slow and doesn't give you a path to grow more and more complex and to have sustained learnings.

你需要有东西能提出一堆新单元，然后从那里开始。

You need to have something that just proposes a bunch of new units and then goes from there.

我想我可以具体说一个东西，让它更具体，就是我们有一种算法叫持续反向传播。

Guess so there is a specific thing I can say to make it at least concrete, which is to say we have this algorithm called continual backprop.

我们几年前发表在《自然》杂志上。

We used published in Nature a couple years ago.

它和反向传播完全一样，但你会种下新的单元种子，这些单元用随机权重初始化。

And it is exactly like backprop, but you also plant new seeds of units that are newly initialized with random weights.

反向传播只在初始时有随机权重，然后随着时间推移，所有的随机性和多样性都会被消耗掉。

Backprop only has random weights at the beginning of time and then as you go on, all that randomness, all that variety from the randomness gets used up.

而持续反向传播则不断注入一些随机性，一些生成和测试，一些生成，然后反向传播的操作就是测试者。

And with continual backprop, you keep injecting a bit of randomness, a bit of generate and test, a bit of generate and then the operation of backprop is the tester.

所以你需要那个。

So you need that.

如果你把这些很好地结合起来，我认为你将迎来新一代的、大大优越的持续深度学习。

And if you put those together really well, I think you'll have a new generation of massively superior continual deep learning.

这就是我们未来几年希望做的事情。

That's what we hope to do in the next couple of years.

太好了。

Wonderful.

你认为这些算法能应用于当前人们扩展大型语言模型并试图让它们进行持续学习而不发生灾难性遗忘的情况吗？

Do you think that these algorithms can be applied to the current state of affairs with people scaling LLMs and trying to get them to do continual learning without catastrophic forgetting?

是的。

Yeah.

当然可以。

Absolutely.

我觉得是这样，你不能拿一个现有的模型说，我直接用这些算法来更新它，因为这些算法是元学习如何学习的。

I think it's so I don't think that you could take an existing model and say, I'm gonna just start updating it with these algorithms, because these algorithms meta learn how to learn.

所以实际上，你必须从头开始学习。

So really, you have to say, I'm going to learn from scratch.

比如说，我学习一个新的基础模型，但我会用这些新算法来学习。

So let's say I learn a new foundation model, but I'm going to learn with these new algorithms.

这些新算法，除了学习知识，它们还会学习如何学习未来的东西。

These new algorithms, in addition to learning the knowledge, they're also going to learn how to learn future things.

所以它们同时学习两件事，然后我觉得你就能学习新东西而不会灾难性遗忘。

So they're learning two things at the same time, and then I think you would be able to learn new things without catastrophic forgetting.

这是你们公司目前在做的最激进的事情吗？从现在的状态来看，要同时做这两件事。

Is that the most radical thing that you're trying to do in your company in terms of, from the current state of affairs, to try to do these two things at the

同时进行？

same time?

最激进的事情。

Most radical thing.

我觉得这要追溯到，

I think that's This goes back to,

我没疯，是别人都疯了。

I'm not crazy, everyone else is crazy.

这也许不算特别激进。

That's perhaps not totally radical.

在2016到2018年间，有很多人都在探索这些想法。

There was a point in 2016 to 2018 where a lot of people were exploring these ideas quite a bit.

他们当时是在更有限的场景下做的，比如他们会说，我们有一系列问题，然后在这个特定情况下，我们会这么做，而我们是希望从单一的经验流中学习，所以我们的方法应该更通用。

They were doing it in a much more limited setting, so they would say, We We have a distribution of problems, and then in this specific case, we'll do it, whereas we want to do it from a single stream of experience, so our method should be more generally applicable.

所以我觉得很多人都探索过这个，但没有人是在通用场景下探索的，也就是最终算法能适用于所有地方。

So I think many people have explored this, but no one has explored this in the general setting where the resulting algorithm would be applicable everywhere.

那么，你的新公司最想做的、别人没在做的最激进的事情是什么？

So what would be the most radical thing that your company, your new company is trying to do that other people are not doing?

最有野心的事情是什么？

What's the most ambitious thing?

记得，我不觉得自己奇怪，所以我不想

Remember, I don't think I'm weird so I don't want

说最雄心勃勃的事是什么？

say What's the most ambitious thing?

最

The most

雄心勃勃的事，我觉得是要掌握从细微之处到宏观层面的全部知识，比如思考如何把飞机从一个城市飞到另一个城市。

ambitious thing I think is to try to have the full spectrum of knowledge both about the tiny things and about the big things, like thinking about how you take an airplane from one city to another.

这确实是个很大的目标，就像你提到的太空飞行例子那样，但大家更容易理解，因为我们很多人坐过飞机，而且我们在生活的方方面面都在使用抽象概念，甚至连松鼠也会用抽象思维。

That's a very big it's like your spaceflight example but it's just kind of more common sense to think about because we all many of us take airplanes and all of us use abstractions in all kinds of our life and even the squirrels use abstractions.

所以，要把从微观到宏观的知识都涵盖进来，用统一的方式处理，并且能让系统自我维护。

So to have that spectrum of knowledge from the small to the big and to treat it in a uniform way and to be able to have it self, maintaining.

你知道，关键问题永远是：当你有一个基于知识的系统时，是什么保证了里面的知识是正确的？

You know, the big question is always, you have your knowledge based system and what keeps the knowledge in it correct?

那在大语言模型里，靠什么来保证知识的正确性呢？其实人们做了大量的后训练工作，确保内容准确无误，然后就把模型冻结住了。

Well, what keeps the knowledge correct in a large language model is, well, people did a lot of post training and they made sure it was correct and then they freeze it after that.

所以这就是让它保持正确的原因。

So that's what keeps it correct.

但实际上，我们的思维总是在变化，但总有某种东西让它保持有序和连贯，让它回到一个好的状态，而不是飘向疯狂的地方。

But really, our minds, we're always changing things and yet something keeps it organized and coherent and settling back into a good place rather than drifting off into crazy land.

我认为，我们最大的抱负就是拥有一个自我一致、能够不断自我训练并保持连贯的思维。

That is, think, our biggest ambition to have a mind that is self consistent and can keep training itself and making it coherent.

我喜欢这个想法。

I love that.

我能问一下吗？

Can I ask?

这似乎是一个非常宏大的愿景，而且把所有这些东西统一到一个思维中的想法也太雄心勃勃了。

It almost seems that it's such an ambitious vision and the idea that all these things can be unified into a single mind is so ambitious.

这是可以实现的。

It's within reach.

我觉得这是可以实现的。

I think it's within reach.

现在是2026年，我们的电脑已经非常快了。

Here it's 2026 and our computers are so fast.

这个目标太宏大了，以至于遥不可及吗？

Is so ambitious that it's out of reach?

我们是否已经隐约知道如何实现所有步骤？

Do we have already inklings of how all the steps can be done?

我觉得我们有了愿景和一些初步的想法。

I think we have a vision and inklings.

我觉得这并不离谱。

I don't think it's inappropriate.

你的愿景涉及一个万亿参数模型，功耗只有20瓦。

Your vision involves a trillion parameter model with 20 watts.

这看起来相当有野心。

That seems pretty ambitious.

这确实很有野心。

That is ambitious.

从某种意义上说，以目前的技术来看，这其实也是不可能的。

In some sense, with current technology, would say it's also impossible.

比如，仅用现在的内存技术，把万亿参数存进内存里，能耗恐怕就超过20瓦了。但我们真正想表达的是，情况正在好转，计算成本在降低，能效也在提升。

Like, just storing a trillion parameters in memory would probably use more than 20 watts of energy with current memory technologies, but we are really thinking of, okay, things are getting better, computation is getting cheaper, it is getting more energy efficient.

那五到十年后呢？

So where would be in five to ten years?

我觉得只要算法得当，五到十年内，我们完全有可能实现这一目标。

And I think five to ten years with the right algorithms, and we can totally be in a world where this would be possible.

所以五到十年相当于摩尔定律的两个数量级。

So five to ten years is two orders of magnitude of Moore's Law.

如果我们每18个月性能翻一番，这就是标准的提升幅度。

It's a standard improvement if we double every 18.

十年时间就能带来两个数量级的提升。

Ten years would give you two orders of magnitude.

所以如果Khurram的说法成立，那今天你应该能用20瓦的功耗做到这一点；而两个数量级之后，就是2000瓦的功耗水平。

And so for Khurram's statement to be plausible, then today you should be able to do it for what, 20 watts, two orders of magnitude, 2,000 watts.

如果今天能用2000瓦做到，那么十年后你就能用20瓦做到。

If you can do it with 2,000 watts today, then in ten years, you'll be able to do it for 20 watts.

你觉得能用2000瓦做到吗？

You think you can do it for 2,000 watts?

研究实验室里有很多人能用到的功率远不止这个数。

You have lots of people at research labs that have access to way more than that.

是的，我觉得即使现在，只要算法合适，我们也能比这更高效。

Yeah, I think we can be more efficient than that even now with the right algorithm.

如果我们能比这更高效，那为什么现在没做到呢？

If we can be more efficient than that, then why aren't we?

又不是人们就想把钱都花光，把我们所有的钱都花掉。

It's not like people just want to spend all their money, spend all our money.

看起来他们确实想这样。

It seems like they want to.

对。

Yeah.

难道不是吗？

Doesn't it?

我觉得他们就是用大量能源来显示自己是真正的男人。

Think that's how they show they're real men, by using lots of energy.

至少在我看不同的研究团队时，我甚至没看到有人相信这是可能的。

At least when I look at different research groups, I don't even see anyone believing in it's possible.

而且我觉得如果你不相信，你就不会去解决技术问题并克服它们。

And I think if you don't believe in it, you're just not gonna work on the technical problems and work through them.

是因为不可能，还是因为系统中有太多浪费？

It that it's not possible or it's that there's so much waste in the system?

到底是哪一个？

Which one is it?

有没有人知道如何高效地做到这一点？

Is there someone who knows how to do it efficiently?

是的。

Yeah.

而且同一个实验室里还有十倍的人在做其他各种事情。

And then there's 10 times the number of people in the same lab doing all these other things.

所以十个人里有九个都在浪费？

And so nine out of 10 people are wasting?

在

In

某种意义上，我觉得我们被困在了一个局部最小值。

some sense, the way I think about it is that we are stuck in a local minnow.

所以如果我们想转向这种新算法，事情在变好之前几乎不可能不变糟。

So if we want to move towards this new kind of algorithms, it is almost impossible that things will not get worse before they get better.

所以当我们开始探索这些新方向时，你不会一开始就得到最先进的性能，但这是因为这是一个不同的范式，而那条路最终会以更高的能量达到类似的性能。

So when we start exploring these new directions, you're not gonna get state of the art performance from day one, but it is because it is a different paradigm, but that path leads to similar performance at a higher energy.

而那些大实验室，他们太执着于他们喜欢的产品，对他们来说，不可能去走一条事情会越来越糟的路。

And these big labs, they are so locked into a product that they like, it is not possible for them to pursue a path where things get worse, worse.

因为他们当前的范式允许他们继续扩展，而这个新范式，他们必须赌一把，而且他们

Because their current paradigm allows them to keep scaling, and this new paradigm, they have to take a bet, and they

而且他们还得解决一些技术难题，这些难题我们思考了很多年。

And they have to figure out some technical things that are difficult that we have thought about it for many years.

我们认识一些人，他们思考这些问题很多年了，我跟他们聊过之后，觉得这事儿是可行的，但确实需要花很长时间去琢磨这些挑战。

We know people who have thought about these things for many years, and when I talk to them, it makes sense that it's doable, but you need to think about those challenges for a long period of time.

如果Oak一切顺利的话，公司会变成什么样？

If everything goes right with Oak, what happens with the company?

你在打造什么样的公司？

What kind of company are you building?

如果一切顺利，我们会实现这个架构。

If everything goes right, we implement the architecture.

我们就能实现真正的持续学习，还能形成抽象概念，从而进行规划和推理，实现真正的智能。

We can have genuine continual learning and we can form abstractions so that we can do planning and reasoning we so do like true intelligence.

然后，你知道，到那时候很难想象具体会发生什么。

And then, you know, it's hard to imagine just exactly what will happen by then.

但我觉得人类会变得无关紧要，对吧？

But I think Humans will become irrelevant?

我完全不这么认为。

I don't think that's true at all.

我们也不这么认为。

We don't either.

我觉得世界会变得令人兴奋，对人类来说甚至更有趣、更精彩。

I think the world becomes exciting and even more exciting and interesting for humans.

但特别是，我觉得你得想想那些大型语言模型。

But in particular, I think there are the you have to wonder about the large language models.

它们可能会面临风险。

They might be at risk.

当这一切最终发生时，我相信它们会有不错的表现。

When this eventually happens, I'm sure they'll get a good run.

它们已经取得了很大的成功。

They've already had a good They've been very successful.

而且我要说清楚，大型语言模型是一项了不起的科学突破。

And let me say, just for clarity, that large language models are an amazing scientific breakthrough.

神经网络在语言运用上的突破，完全出乎意料。

A breakthrough in the skillful use of language by neural networks is totally unanticipated.

你知道，语言一直是符号方法的保留地，现在它们彻底改变了人们对这个领域的看法。

You know, it was always a holdout for symbolic methods in language and they have totally changed how that's thought about now.

是的。

Yeah.

所以这是一个巨大的突破。

So it's a big breakthrough.

所以让我感到沮丧的是，我们不得不只是庆祝在人工智能这个子问题上取得的巨大进展，并享受这一点。

So it's frustrating to me that we have to, you know, just celebrate that we've made this great progress in the subset of the problem of AI and enjoy that.

相反，它不得不假装是整个人工智能。

Instead, it has to pretend to be all of AI.

所有的智能不仅仅是流利地使用语言。

All of intelligence is not fluid, capable use of language.

还有更多的东西。

There's so much more.

这是很重要的一部分。

It's an important part.

你知道，这大概占智力的20%或四分之一。

You know, it's like 20% or a quarter of intelligence.

还有更多。

There's more.

我们还没完成。

We're not done.

对。

Yeah.

如果一切顺利，你想象中是否有一个单一的智能体，既能学会从树枝上荡来荡去，又能制造宇宙飞船，还有我们今天讨论的各种事情？

If everything goes right, are you imagining that there's a single mind that can do everything from learn how to swing from tree branches to make a spaceship to, you know, all these various things we've talked about today?

是同一个智能体，同一组权重就能做所有这些事情，还是说……

Is it a single mind and is it a single set of weights that can can do all these things or is it a

单一的设计。

single design.

好的。

Okay.

会有很多这样的系统。

There'll be many minds.

所以它是一种单一的设计，能应对不同的环境。

So it's a single design that reacts to different environments.

而这种系统的不同版本，会因为各自的经历不同而学到不同的东西。

And it it it and different versions of that mind would learn different things because they have different experience.

这又回到了大世界假说：值得学的东西是无限的，所以一个系统不可能学到所有的东西。

This sort of goes back to the big world hypothesis that there are infinitely many things to learn, so one system cannot learn infinitely many things.

而且，我觉得Rich之前也说过，如果你有两个这样的系统，比如世界上最大的两个系统，那它们显然无法相互模拟，因为它们的复杂度是相当的。

And, like, I think Rich already mentioned this, but if you have two of these systems, if you have two of the largest systems in the world, then it is trivial that they cannot model each other because they're equally complex.

所以，单个系统永远不可能达到学会所有东西的程度。

So a single system would never be able to get to a point where it can learn everything.

永远都会是多个系统从各自的经历中去学习。

It would always be multiple systems that are learning from their own experience.

你们在招人吗？

You guys are hiring?

你们在找什么样的人？

What kind of people are you looking for?

我们在招人。

We are hiring.

最初的团队，大部分我们心里已经有数了。

The initial team, most of it we already have in our mind.

这些人都是长期思考过这些想法的人。

So these are people who have thought about these ideas in the in many, you know, for a long time.

我们会采取稍微不同的方法，因为这是一个不同的范式。

And we are gonna take a slightly different approach because this is a different paradigm.

快速扩张没有意义，因为从某种意义上说，每个我们招进来的人都需要看到我们所看到的，而并不是每个人都能看到。

It doesn't make sense to become large very quickly because in some sense, everyone we hire has to come to see what we see, and not everyone sees that.

所以我们会从小规模开始，慢慢发展到几个人，然后再继续扩展。

So we're going to start small, slowly grow to maybe a handful or two or three, and then go from there.

我们希望能高度一致。

We want to be super aligned.

对，要高度一致。

Want to be super aligned.

这样我们才能高效协作，推动进展并实现规模化

So that we can be very productive working together and scaling the

发展。

progress.

没错。

Absolutely.

真的，非常

Very, very

酷。

cool.

太棒了。

Wonderful.

我很喜欢这次对话。

I love this conversation.

感谢你抽出时间分享你的想法，你对强化学习和算法设计的未来有着非同寻常的深刻思考。

Thank you for taking the time to share what You you're up are an unusually deep thinker about where reinforcement learning and algorithmic design will go.

今天能和你一起探讨这些，真的非常愉快。

And it was a true pleasure to get to explore it together with you today.

所以，谢谢你。

So thank you.

非常感谢。

Thank you very much.

谢谢你，Raj。

Thank you, Raj.

这是我们的荣幸。

It's our pleasure.