构建自动化AGI实验室：Core Automation的Jerry Tworek和Rohan Anil

比如说我踢足球，这跟强化学习就非常非常像。

If I play football, for example, it looks very, very closely to reinforcement learning.

我反复踢球，每次稍微调整一下，看看是不是大致达到了我想要的效果，这里面就有自我强化的过程。

I kick a ball a lot of times, and every time I adjust it a little bit and I see if it roughly matches what I what I wanted and either there's some self reinforcement happening.

但学数学就是完全不同的体验了。

When I learn mathematics, it's very different type of thing.

更像是阅读一些难懂的概念，然后在脑子里反复琢磨，直到豁然开朗，直到把它们串联起来。

It's it's like reading about hard concepts and thinking about them very deeply inside my head until things click and until until have them connected.

这两种方式在某种程度上都是从经验中学习。

And both of those in some way are learning from experience.

只是它们非常不同。

They are just very different.

我们现在可能比以往任何时候都投入更多的计算资源在从经验中学习上，但强化学习并不是从经验中学习的终点。

We probably are spending the most compute crap than ever on learning from experience, but learn reinforcement learning is is not the end of learning from experience.

未来几年，研究人员会提出更好的方法来利用这些数据。

And there will be better approaches that researchers will be coming up in the coming years on how to how to use that data.

杰瑞、罗汉，非常感谢你们今天来做客。

Jerry, Rohan, thank you so much for joining us today.

你们两位是Core Automation的创始人，这是目前旧金山最热门的AI实验室之一。

The two of you are the founders of Core Automation, one of the hottest neo labs in San Francisco right now.

在创立Core Automation之前，你们领导了AI时代一些最重要的研究项目。

And before starting Core Automation, you led some of the most important research projects of the AI era.

杰瑞，你曾是OpenAI的副总裁，负责草莓和推理团队等工作。

Jerry, you were VP at OpenAI, where you worked amongst other things on running the strawberry and reasonings teams.

罗汉，你曾是Gemini四位预训练负责人之一，之前还在Google Brain主导了大量基础AI研究，并且是谷歌和Anthropic的“救火队员”。

And Rohan, you were one of two of the four pre training leads at Gemini, and before that led a lot of the fundamental AI research at Google Brain and were the fix it guy across Google and then at Anthropic.

所以你们两位对前沿研究领域的现状，比大多数人看得都多。

And so between the two of you, you've seen more than your fair share of what the world looks like in terms of doing frontier research.

因此我非常期待深入探讨。

And so I'm very, very excited to dig in.

先从你开始吧，杰瑞。

Let's start with you, Jerry.

你最近发了一条挺犀利的推文。

You tweeted a very spicy take recently.

要取代Transformer，第一步是深刻理解它到底把我们带到了多远。

The first step to replacing transformers is appreciating deeply how far they were able to carry us.

这是在给Transformer唱挽歌吗？

Is that a eulogy for the transformer?

这话是什么意思？

What does that mean?

非常感谢你邀请我们，索尼娅。

Thank you very much for inviting us here, Sonya.

感觉我最近接受采访，大部分时间都在解释自己的推文，解释我到底想表达什么。

I feel like a lot of my interviews these days is explaining my tweets and what did I what did I mean.

但所谓‘理解Transformer’，就是搞清楚它擅长什么。

But Appreciating transformer means, like, understanding what it does well.

这样你就不会去解决它已经解决得很好的问题了。

So you are not solving the problems that it is solving well.

你得把注意力放在它的弱点上。

You have to focus on its on its weaknesses.

你得理解它的优点和缺点，现在很多人在架构方面的工作，都只是想让Transformer更便宜、更高效。

You have to you have to understand good parts and bad parts, and it's very easy in a lot of the work what people are doing in architectures is trying to make transformers cheaper and trying to make transformer more efficient.

我很少看到有人思考怎么让Transformer更强大、更有表现力。

I I very rarely see people thinking about how do we make transformers more powerful, trying to do more expressive.

但是，看到它的弱点和看到它的优点，其实几乎是同一回事。

But but, like, seeing someone weak parts and seeing someone strong parts are almost almost almost the same thing.

只是对Transformer的形态有了更深入的理解。

It's just just understanding the shape of transformer a little bit more.

嗯，我觉得我们现在就处在这个阶段。

Well, I think right now we are in this stage.

我们已经非常擅长训练非常大的模型了。

We got really, really good at training really, really big models.

我们掌握了两种算法。

We mastered two algorithms.

我们掌握了大规模预训练，也掌握了大规模强化学习。

We mastered pre training at a large scale, and we mastered reinforcement learning at a large scale.

我经常问自己，机器学习的下一个突破是什么。

And I'm asking myself a lot what is next in machine learning.

我认为目前，制约更好模型和更智能系统的瓶颈，正是架构本身。

And I think at this moment, what the bottleneck is to better models and to smarter systems is the architecture itself.

现在是时候重新审视我们过去六年一直在走的那条路了——不断给两个基本操作（混合专家模型和注意力机制）增加参数。

It is this moment to revisit the train we've been riding for the last six years of trying to add more and more parameters to essentially two of the same operations, which is MOE and and attention.

当我思考这个问题时，比如我们现在处于什么位置、在做什么，我经常想到的是 Codex 和 Cloud Code 为我们带来的改变。

And when I am when I'm thinking about it, like, what's what like, where we are today and what we are doing, I am thinking a lot about about, like, what Codex and what Cloud Code are doing for us.

我真的很感激这些系统，以及我们在这六年规模化过程中构建的编码、工作流自动化和产品体系。

And I'm really, really appreciative of those systems and of the coding and of the workflow automation and of the systems of the products that we have today that we essentially have built over those those six years of scale.

我认为这是思考的第一步。

And I think I think this is this is the first step of thinking.

如果我们想寻找替代方案，就需要先看清我们现在的位置、解决了哪些问题，才能开始思考下一阶段是什么、还有哪些问题没有解决。

Like, what is the if we wanna work on the replacement, we need to, like, see where we are, what problems we have solved to, like, start seeing what the next stage is, what problems we haven't solved yet.

我们到底缺了些什么？

What what what kind of are we missing?

每次我用Codex成功完成一个任务时，我也会开始想，为什么我当时没有更努力地去推动那件事呢？

And this is this is kind of whenever whenever I use Codex and I am successful at a task, I also start thinking, why didn't I why didn't I try to push that thing harder?

每天来上班，我用Codex做很多事情，但我还是得来上班。

Whenever whenever I come to work, there are a lot of things I do with Codex, but I still come to work.

我还是会让它帮我做一些事情。

I still ask it to do certain things for me.

我总是在问自己，为什么那里还需要我？

And I'm always asking myself, why why am I even needed there?

为什么核心自动化这个名字和概念，就是我们想要实现任务自动化。

Why is Core Automation is name and its concept is we wanna we wanna be automating tasks.

为什么那些事情还没有被自动化？

And why why why are those things not yet automated?

为什么Codex不能替我做所有事情？

Why why why is not Codex doing everything for me?

这其实就是我们研究想要探索的方向。

And this is this is the question of, like, the research where we want to go.

通过这项研究，我一直在思考我们需要什么样的模型、什么样的系统、什么样的特性——那些我们今天还不具备的东西，这就是我最近经常在想的问题。

And with that research, I'm trying to think what kind of models, what kind of what kind of systems do we need, what kind of qualities do we need, what that that that we don't we don't have today, and that's that's what I'm thinking a lot these days.

而你提出的这个前提——架构才是问题所在——我觉得是个反主流的观点。

And you have this starting premise of the architecture is the issue, which I think is a contrarian point of view.

那是什么让你得出这个结论的呢？

So what what led you to that point of view?

你看到了什么，让你觉得架构才是问题的根源？

What did you see that made you think the architecture was the issue?

从根本上说，这就是问题所在。

It's fundamentally what is the issue.

这其实又回到了之前那个推论。

It's it's it, like, comes back from the from from the previous implication.

我认为问题在于，模型是在实验室里训练的，却要在现实世界中部署。

What what I think is the issue is that the models are being trained in the lab and are being deployed in the in the real world.

那就是……

That is that that that

根本的根本矛盾所在。

is the fundamental fundamental tension that is that is there.

而且，我个人的经历也让我有点失望。

And, like, a bit of a bit of my disappointment can comes from my my my personal story.

当初我们在OpenAI开始研究并推进强化学习规模化的时候，我基本上就相信，强化学习的规模化是通往AGI道路上必不可少的一步，从我一开始在OpenAI工作起就这么认为。

Whenever we were starting the research and progress on scaling up reinforcement learning at OpenAI, I basically believe that scaling up reinforcement learning is a necessary stepping stone on a path to AGI since since since I started, like, working working at OpenAI.

我一直是个强化学习的极端拥护者。

And I was always reinforcement learning maximalist.

我一直坚信这是我们需要专注的方向。

I was always believed this is what we need to focus on.

这是我们必须做的事情。

This is what we need to do.

我看到大语言模型从GPT-3到GPT-4不断被扩展到越来越高的水平，但我们在强化学习上投入的却依然很少。

I've seen LLMs being scaled up to for to to higher and higher levels for GPT three to GPT four, and we're still doing very little RL.

我内心一直有个信念，觉得一旦我们开始规模化强化学习，就能解决一切问题。

And I had this this internal belief that the moment we start scaling up RL, we'll we'll we'll solve everything.

我们就能解决所有难题。

We'll we'll we'll be able to solve solve all the problems.

后来我们终于开始规模化强化学习了。

And we eventually started solve scaling up RL.

我当时就在那里。

I was I was just in there.

我身处其中。

I was I was in the center of it.

我当时想，就是现在了。

I was thinking here we are.

如果你在2024年问杰瑞，我们什么时候能实现通用人工智能？

If you ask Jerry in 2024, when do we get AGI?

我会说，2025年就是那一年。

I would say 2025 will be will be that year.

这就是我们解决一切问题的地方。

This is where we solve everything.

然后我看到我们一个接一个地训练模型。

And I saw us training model after model.

这个模型变得越来越好。

This model was getting better and better.

所有的基准测试分数都在上升。

All the benchmark scores were going up.

那当时我们是不是也解决了所有现实世界的问题？

And did we also solve all the real world tasks at that moment?

可惜，可惜并没有。

Unfortunately, unfortunately not.

我们还有工作要做。

We we we still have work.

我意识到这里有一个细微的区别：我们用来评估模型的所有基准测试，本质上和我们训练模型时用的东西是一样的。

And I realized there was this bit of distinction as all the benchmarks that we are evaluating our models, they were essentially the same thing as we were training the models on.

所有的评估和训练任务就像是同一枚硬币的两面，但现实世界的分布和任务要混乱得多、模糊得多，也大不相同。

Like all the evals and training side tasks are the same sides of the coin, But the real world distribution and real world task is much messier, much murkier, much more much more different.

我们的训练数据并没有真正复现现实世界的使用场景。

Our our training data didn't really replicate the real world use cases.

尽管我们基本上把所有任务都优化到了极致，但如果你去问任何一个训练模型的人，嘿，你们的主要问题是什么？

And despite us basically maximizing all the task, if you see ask anyone training models, hey, what is one of your main issues?

我没有足够难的任务。

I don't have hard enough tasks.

我不是没有东西可以训练模型，但我们仍然没有覆盖现实世界的全部分布。

I don't I don't have what to what to train our model on, yet we are still not covering the the entirety of the of the real world distribution.

由此，我的结论是，我们需要让模型在测试时也能学习。

From that, my conclusion is we need to have models that learn at test time.

我们需要让模型与用户一起，在他们的数据、他们的现实任务、现实世界的分布中学习。

We need to have models that learn with users on their data, their real world task, on the real world distribution.

那么，当被问到为什么我们今天还没有做到这一点时？

And there when you are asked why why why why don't we have that today?

为什么Transformer模型不能在任何地方学习呢？

Why are transformers not not learning any anywhere?

实际上，在测试阶段我们有两种学习方式。

And there are essentially two types of learning that we could be doing at test time.

一种是Transformer的上下文学习，它基本上没有灾难性遗忘的问题。

We could be doing in context learning essentially of transformers, which is it it it doesn't have fundamental problems of catastrophic forgetting.

它没有这个毛病。

It doesn't have that issue.

而且数据效率很高，这点很不错。

It is pretty data efficient, so that that is great.

但它的扩展性不太好。

But it's not very scalable.

我们能用的量是有限的。

We only can have so much of it.

它受到限制，而且从机制上讲，构建上下文时你到底在做什么也有更多限制，不过我们也许可以之后再讨论这个。

It is limited and has some more even more of mechanical limitations of what actually are you doing when you build context, but maybe we can come back to it later.

但我们现在有上下文学习，这种方式非常有限，能处理的数据量也很少。比如我用Codex的时候，大概用了二十分钟左右，

But we have in context learning which is very limited and very small amount of data whenever I'm using Codex roughly after around twenty minutes of usage.

我就得压缩一下，然后把它移走，其实能处理的数据量并不多。

I need to I need to compact it and move and move it afterwards, which is not that much, not that much data.

如果我们只能学二十分钟的内容，那确实不算多。

If we have all all we can learn is for for twenty minutes, it's not that much.

第二种方法是微调。

And the second thing is fine tuning.

我们可以尝试持续微调模型，但这样就会出现灾难性遗忘的问题。

We could try to continuously fine tune our models, but then those have the issues of catastrophic forgetting.

而且数据效率非常低，这两个问题都不太好解决。

We have issues of very low data efficiency, and neither of those are very solvable.

这两个问题都不太容易找到解决办法，很多人一直在尝试，如果真那么容易解决的话，早就有人解决了。

Neither of those are very easy to to find ways people have been trying if there was were easy to solve some already solve it.

所以我个人认为，我们需要找到一种算法，能够进行元学习，并且可以在架构层面表达出来，让它能够代表学习的过程是什么样的。

So my personal belief is we need to find an algorithm that we can we can meta learn, as we can express on the architectural layer that can represent how does how does learning look like.

学习到底是怎么样的，才能适应更长时间跨度呢？

How does learning look like that can work on much, much longer horizons?

你觉得这个架构会像Transformer那样吗？

Do you expect the architecture will look transformer like?

因为据我从高层那里了解，OpenAI一直在尝试扩展强化学习，但直到Transformer出现，才似乎有了一个可扩展的世界模型基础，让强化学习得以规模化。

Because my my my understanding from the from the chief seats is that, you know, OpenAI had been trying to scale up reinforcement learning for a long time, and it wasn't until the transformer came about that it seemed like there was an even kinda scalable prior on the world upon which to even scale RL.

那你打算怎么思考扩展这个新范式呢？

And so how do you even go about trying to think about scaling up this this new regime?

对。

Yeah.

这是个好问题。

It's a it's a great question.

我觉得这两件事是同时发生的。

I think those two things happened at the same time.

但要说有什么变化，主要还是经济层面的原因，因为从技术上讲，LSTM也是可以扩展的。

But if anything that that happened there was was mostly about about economics because technically you can scale up LSTMs.

只是没人敢往那个方向走，而且它们的扩展性差得多。

Just no one no one really really dared to go in that direction and they did scale much much more poorly.

在那篇关于扩展规律的论文里，他们对比了LSTM和Transformer的扩展效果。

They're they're they're scaling in a scaling loss paper, there is presented a comparison of LSTMs and transformers.

对。

Yeah.

从根本上说，Transformer的扩展性更好。

And fundamentally, the scaling of transformers was better.

如果当初没有发明Transformer，我们可能还在扩展LSTM，也能搞出一些模型。

There is a world where we never invented transformers, and we will be scaling LSTMs, and we'll having some models.

但因为训练成本高得多，产品效果也差得多，用户体验会很糟糕，可能没人愿意花那么多钱去训练那些巨型LSTM，因为市场回报根本看不到。

But because they would be much more expensive to train and much less impressive as a product, we would have would have just as worse experience and maybe no one would be able to convince people to spend as many dollars training those gigantic LSTMs because we wouldn't get a market return.

Transformer最了不起的地方——这也是为什么我们得深深感激它——在于它的经济价值。

The the majestic thing about transformer, which goes back to, like, why why why do we have to appreciate transformers so deeply is that transformers are economically valuable.

它的训练成本低于它带来的收入，这就是机器学习的魔力所在。

The them the cost of training them is lower than the revenue that they that they generate, which is magic of machine learning.

这本身并不能保证什么。

It's not guaranteed by itself.

但对于LSTM来说，可能就不会是这样，这种情况是有可能发生的。

But for for LSTMs, it probably wouldn't be that way, which which which may happen.

不过，从很多方面来看，大多数架构都是可以扩展的。

But you can in many ways, can scale most of the architectures.

我觉得，以前人们不搞规模化，很大程度上是因为OpenAI之前的研究人员对规模化很抵触。

I think I think a lot of reasons why people didn't scale things before was because researchers before OpenAI had a lot of reluctance to scaling.

规模化常常被视为不科学，而算法研究的方向是，我们如何变得越来越高效，如何在相同的计算预算下，得到越来越好的结果。

It was often seen as unscientific, and research in algorithm was was providing how do we become more and more efficient, how do we, for the same compute budget, get better and better result.

而OpenAI当时做了一个有点反主流的赌注，他们说，嘿，我们不在乎算法是不是越来越高效。

And it a bit of a contrarian bet by OpenAI at that moment to try to say, hey, we don't care about better and better algorithms.

我们在乎的是算法能不能更好地扩展。

We care about more and more scalable algorithms.

我们如何投入更多的计算资源，获得更好的结果，为此OpenAI在很长一段时间里，反复受到社区里很多人的批评。

And how we pour more and more compute and get better results, which OpenAI was criticized repeatedly by many people in the community for a long time.

但多亏了这一点，我们才有了今天这些模型。

But thanks to that, we have we have the models that we that we have today.

而且我认为有很多架构是可以被扩展的。

And I think there are tons of architectures that can be scaled up.

我是Core Automation团队的一员，我们的信念是，很多架构研究在太小的规模上进行了太长时间。

And I am part of the Core Automation's mission, and our our belief is that a lot of architectural research happened at too small scale for too long time.

很多人都在说，嘿。

A lot of people are trying to say, hey.

我们先在小数据集、小计算量的条件下测试架构，等它证明了自己之后，再去看它能不能扩展。

Let's let's try to first try our architecture on a on a on a small dataset on a small in a small computer regime and then see where it scales only after you prove itself.

但比如说，做强化学习的时候，你知道要想得到任何有意义的结果，需要一定量的计算资源才能看到模型的能力。

But for example, when you do work on reinforcement learning, you know that to get to any interesting results, need certain level of compute to even see the capabilities in the model.

强化学习需要一定的计算基础才能开始发挥作用。

Reinforcement learning needs a baseline of of of ability to only to only start working.

所以从我的角度来看，可能有很多架构都需要一个计算量的基线，才能开始做出任何有趣、有用的事情。

So so where I am coming from, probably there are many architectures that need a baseline of compute to even start doing anything anything interesting, anything useful.

那我能问一个可能有点敏感的问题吗？

Can I ask you then maybe a touchy question?

请说。

Please do.

如果你需要

If you need

一个计算基线，这听起来像是大研究实验室里才适合做的工作。

a baseline of compute, that sounds like a job that would be well served inside of a big research lab.

那为什么要自己开公司来做这个呢？

Why start a company to go do this?

这是个很好的问题，我觉得在很多方面，这很可能是个时机问题。现在市场处在一个非常特殊的位置，那些最大、最成功的实验室，不管是巧合还是命运使然，正处在有史以来最激烈的市场竞争中，这让他们不太愿意尝试不同的路径或替代方案。

It's a it's a it's a great question, and it's I think in in many ways, it's likely a timing thing timing issue market is right now in a in a very specific place where the biggest and the most successful labs by coincidence or by fate are probably in the most competitive market fights ever right now, which makes them not very keen on trying different paths, trying alternatives.

如果Transformer能盈利，而且你可以投入更多精力和资源去扩展它，以便在下一个季度胜出，那很难把大量注意力和精力放在一个可能更好、或者可能在一两年内重新定义这个领域的东西上。

If transformer is profitable and if you can spend more efforts and more resources scaling transformer to win in the next quarter, it's very hard to put at least a lot of attention and a lot of energy to work on something that will that will maybe better or maybe or maybe will will redefine the field in a year or two.

所以我认为那些最大的实验室——我基本上都跟他们聊过——对尝试Transformer的替代方案并没有太大兴趣。

So so I I think the big biggest labs, and I talked to basically all of them, don't don't have that much interest in trying the alternatives to to transformer.

而那些不是最大的实验室，都在拼命模仿最成功的实验室的做法，每个人都在训练同样的编程代理。

And the labs that are not the biggest are doing whatever they can to do what the what the what the most successful labs are doing, and everyone is trying to train the same the same coding agent.

你看看上周发布的产品，每个人都在推出编程代理。

If you look at the last week's releases, everyone everyone is is trying to release a coding agent right now.

我觉得我们需要不同的路径和不同的方法。

And and I think we need different paths and different and different approaches here.

所以，这就是我们在生态系统中想要填补的空白。

So so that's what the the niche in the ecosystem we are trying to trying to fill in.

罗翰，变压器发明的时候你就在谷歌大脑工作吧？

And Rohan, you were at were at Brain when the transformer was invented.

你同意杰瑞对变压器的评价吗？

Do do you agree with Jerry's eulogy for the transformer?

是的。

Yes.

从某种意义上说，当阿西什、纳瓦和其他人首次提出变压器时，我同时也在做在线蒸馏的研究，我们在同一个内部研究会议上展示了成果。

In some sense, like, once the first when the transformers, Ashish, Nava, and others came up with it, I had worked on my work on online distillation around the same time we presented it at the same internal research conference.

当时在公司内部并没有引起太大轰动，只有少数人真正理解了它。

It wasn't a big deal internally, there was only a few people who actually got it.

很多人觉得，哦，这不过是又一篇论文罢了。

A lot of people were like, oh, it's yeah, it's another work.

而且大家还在找各种方法，同时它最初的研究也聚焦在一个很实际的问题上，那就是翻译。

And people were finding ways to and it was also very focused on, at least the original work, very focused on a real problem, which was translation.

所以他们解决了翻译任务中LSTM的瓶颈问题。

So they solved, like, the beta LSTM on translation.

后来OpenAI接手了——我是说，在谷歌内部，确实有诺姆和其他几个人对扩展语言模型很感兴趣。

And it took OpenAI I mean, internally at Google, there was definitely, like Noam and a few others, definitely interested in scaling language models.

我觉得直到GPT-2和GPT-3出现，我们才看到Transformer的优势真正发挥出来。

I think it is until GPT-two and GPT-three that we saw the benefit of transformers working quite well.

至少在我看来，架构的本质就是如何分配计算资源。

At least the way I think about architecture is how do we spend computation?

而Transformer是一种非常高效的计算分配方式。

And transformer is one way, very efficient way to spend computation.

但现在看整个行业，我们大部分算力都花在推理阶段，也就是生成token上。

But now that I look at the industry, a lot of our computation is inference time and spending it on tokens.

我想问个问题：如果我想优化架构，是不是应该同时考虑预训练和强化学习？

Let me ask this question, like if I want to optimize for a better architecture, want to look at both pre training and RL together.

我希望找到一种架构，能比现在的思维链逐token生成更高效地利用算力。

And I would like to find architectures that spend computation much better than current chain of thought token generation.

更全面地看，我觉得预训练相当于构建了一个有特定上下文长度的Transformer，而强化学习介入后会发现，这样还不够。

In at a like, to give a much better overview, I think of, like, pre training as built the transformer with certain context length, and RL comes in and it's like, well, that's not sufficient.

我需要更多算力。

I need more computation.

那就通过每次加一个token来实现。

Let me do it via adding one token at a time.

从推理的角度看，这效率很低。

This is quite inefficient from inference perspective.

你是一次只生成一个token。

You're doing one token at a time.

所以大部分解决方案都在寻找更好的推测解码方法。

So most of the solutions have been finding to do better ways of speculative decoding.

这就像是在给问题贴创可贴——我们选了一种一次只能生成一个token的方式。

So it's like a band aid to a problem that we've picked something that can only generate one token at a time.

也就是自回归解码。

So auto recursive decoding.

Transformer在计算资源分配上确实存在问题。

There is problems with the transformer in terms of how do we spend the computation.

很长一段时间里，我觉得业界主流都在训练超大规模密集模型，大概花了两年到三年时间才把架构优化到我们现在习以为常的程度——比如稀疏性和混合专家模型，以及如何高效训练它们，这些对很多人来说当初并不明显。

For the longest time, I think most of the world was training very large dense models and it took like the industry like roughly two to three years to get to refine the architecture to what we now take for granted was not obvious to a lot of people, sparsity and mixtures of experts, and getting good training efficiencies with them.

对吧？

Right?

那么问题来了，Transformer到底哪里不对劲？

So then you can ask, like, what's wrong with the transformer?

嗯，如果计算深度不够，我们该怎么提升计算深度呢？

Well, it's if the computational depth is poor, how do we increase computational depth?

提出这个问题，就能打开大约20个新方向，让我们思考如何修改机制来融入它。

And just posing that question opens up, like, 20 new directions on how we can modify the mechanism to incorporate it.

所以我觉得，做这样的工作需要时间。

So I see, like, to do work like this, it takes time.

而且通常来说，过去的基础研究要花五到六年才能落地到工业界，这很大程度上取决于组织是否认识到它的重要性。

And usually, like, fundamental research in the past have taken, like, five, six years to land into industry, and it's largely from organizational, knowing that it is important.

这就是一场赌注。

This is the bet.

就像Jerry内心坚信强化学习是必要的，但谷歌绝对没有这种信念。

Like, just like Jerry had the inner belief that RL is needed, absolutely do not have that belief at Google.

我当时是个预训练最大化主义者，预训练了最大的

I was a pretraining maximalist, pretrained a biggest

很合适。

are a good fit.

对吧？

Right?

事情就是这样。

Things are.

对吧？

Right?

所以那种内心的信念。

And then so that inner belief.

其次，你的架构需要在硬件上高效运行。

And second is you need your architecture to run efficiently on hardware.

对。

Mhmm.

理论上最优的架构对谁都没用。

A theoretically optimal architecture is not useful to anyone.

它只有在实践中才有意义。

It is something when it comes into practice.

所以你需要从研究起步，到把它产品化，编写内核和所有东西，完成整个端到端的流程。

So you need the research inception to getting it productionized and getting kernels and everything written, the end to end loop.

目前只有少数地方有整合团队在做这件事。

And there's only few places right now which have integrated teams doing that.

而且我认为我们团队的建设方式是把专家聚在一起，而不是分散在不同的部门，这样我们就能加速推进，让每个人都能从全局角度看待问题。

And I think we have built a team in a way that puts the experts together, not in different silos that we are accelerating on having everybody look at the problem holistically from end to end.

所以我对这一点非常乐观。

So that's where I'm quite bullish.

这也是我在这里的原因。

That's why I'm here.

目前的机制相当糟糕，如果放任不管，我担心我们可能需要更长的时间才能取代Transformer。

The current mechanisms are quite poor, and it will if if you leave it to the world, I am afraid that it'll take us like a much longer time horizon before we replace the transformer.

而且我觉得很多人已经在抱怨token成本太高了。

And I think a lot of folks are already complaining a lot on token costs.

看起来好像我们并没有在抱怨。

And that seems like as some We're not complaining.

我的意思是，从某种角度来说，是的。

I mean, in terms of like yeah.

没错。

Exactly.

我来自谷歌那种思维方式，我们得服务几十亿人。

I come from the Google mindset where we had to, like, serve billions of people.

所以，找到更高效的架构，能符合延迟和可服务令牌数量的要求。

So, like, finding more efficient architectures that fit have like a deadline on latency and the number of tokens that you can serve.

所以当我看到这个，世界上能用上前沿技术的人非常少，必须有人或某个团队来加速改进，我们正在努力做这件事。

So like when I look at that, like the amount of like the world that can use like frontier tech is very little and someone or some group has to like, accelerate and make this better, and we are taking that shot at doing that.

当前技术只是简单扩大规模，仍然只适用于一部分人。

The current technology just scaled up is still only relevant to, like, a subset of humans.

对。

Yeah.

这就是我们押注的方向。

And this is the bet we're making.

我听到你提到的一个问题是，变压器的计算深度不够好。

So one of the things I heard you say was the problem with transformers is the computational depth is poor.

对。

Yeah.

如果这是问题的关键，那请告诉我们，

If that's the crux of the issue, tell us what does

这是什么意思？

that mean?

为什么会这样？

Why is that the case?

你怎么解决？

How do you fix it?

我可以给你一个见解。

I can give you, like, one insight.

我们训练的大多数Transformer模型其实都很浅。

Like, most transformers that we train are quite shallow.

最多也就100层深。

It's at most like a 100 layers deep.

深度之所以叫深度学习，是因为你想要更深的表征。

Depth is like it's called deep learning because you wanted deeper representations.

关于增加深度，有人做过实验，但没人真正证明我们能学到极深的表征。

There has been experiments on going into depth, but no one has actually shown, us learning extremely deep representations.

思维链推理和用强化学习让模型自己进行思维链，是增加计算深度的一种方式，因为每增加一个词元，就相当于多了一条路径。

Chain of thought reasoning and RL to do chain of thought by model itself is one way to increase computational depth because every token you add, you add like one more pathway.

这样一来，就能突破预训练架构给你设定的那个瓶颈。

And there has been so then you can get out of like this bottleneck that the pre trained architecture has set you up on.

你原本只能处理层数乘以序列长度那么多。

You can only do layer number of layers times sequence length.

现在你可以增加序列长度，结果会强很多。

Now you can increase the sequence length and you get much stronger results.

你可以做推理时的规模扩展。

You can do inference time scaling.

但推理时规模扩展的问题是，模型现在需要生成更多词元才能得到更好的结果，而且一次只能生成一个词元。

Now the issue with inference time scaling is that models now have to produce more tokens to get better results, and that's very one token at a time.

从这里可以看出，我们可以直接解决很多这些问题，这也是我们正在研究的一个方向，对吧，就是让这个过程更高效。

And from this, you can see, like, you can directly address many of these things, and this is like a subset of work that we are looking at, right, making this much more efficient.

你对基于Transformer的架构有什么预测？

What's your forecast for the transformer based architecture?

如果它不是最终形态，它能带我们走多远？

If it's not the end state, how far can it get us?

我们什么时候会看到它开始达到瓶颈？

When do we start to see it topping out?

我觉得这一切都归结于我们训练Transformer的目的，以及我们能拿它们做什么。

I think it all it all comes back to what we are what we are training transformers for and what we can what we can do with them.

我们做预训练，这非常擅长把互联网上的所有知识提炼到Transformer里。

We're doing pretraining, which is very good at distilling all the knowledge from the Internet into transformers.

然后我们可以通过强化学习来训练它们，基本上就是把我们想要的所有工作流程都嵌入到Transformer里。

And then we can RL them through, which is we basically can bake all the workflows that we want into a transformer.

所以Transformer的极限在于，我们已经把人类的所有知识、关系、以及它们如何协同工作、如何组合，都放进了模型里。

So what what what we're transformer is, like, capped out is we have all the knowledge of of of humanity in the model together with the relationships and how do they how do they work together, how how they can be combined.

基本上，只要有训练数据的任务，我们都可以把它放进模型里。

And, basically, any task that we have training data for, we can we can put into the model.

这可能会是一个巨大的模型，用大量的算力去训练全世界所有的数据。

And this could be gigantic model train with a lot of compute on all the on all the data in the world.

然后，如果我们停止训练那个模型，会发生什么？

And then if we ever stop training that model, what would what would happen?

我觉得，关于Transformer，有一个问题很值得经常问一问。

Like, the the question worth asking often, I'm thinking about transformer.

如果OpenAI和Anthropic停止训练新模型，我们就用现在这个Transformer，说这就是最终版本了，那会怎么样？

What would happen if OpenAI and Entropic stopped training new models and we got we got the transformer we have today and say this is this is it.

那就是……

This is this is this is

我们现有的最好的模型。

the best the best model we have.

几个月过去，一年、两年、很多年过去，这个模型会变得越来越没用。

Months pass, year year year year's pass, and the model is getting less and less useful.

也许实验室真的记录了地球上每个人的行为、任务和环境，然后把它们都放进模型里，放进学习环境里。

Maybe the lab really, like, recorded of every human on earth what they were doing and what their what their tasks were and their environments and put them put them in the model, put them in the in the learning environment.

但如果这些情况发生了变化，那又会怎样呢？

But then the but then then then what happens if the if anything of that changes?

如果世界上出现了新事件，这些新事件之间有了新关系，出现了新类型的任务、新的代码库、新的工具，那么Transformer的很多实用性和价值都来自于训练数据中已有的东西。

If there are new events in the world, if those new events have new relationships between them, if there are there are new types of tasks, if there are new code bases, new tools to use, transformers are getting a lot of their usefulness and all value through the things that are valuable have to be present in training.

而当这些新东西不在训练数据里时，它们就表现不佳。

And when they are not, they suffer.

它们有一定的适应能力，但适应范围不大，也不够灵活。

There's some ability to adapt, but it's not very not very big and not very not very flexible.

所以在我看来，这就是Transformer能力的上限，从很多方面来说，我认为它只是我们使用的一个工具。

So in my mind, this is this is kind of the level where the where the transformers top, which in many ways what I think is a is a tool to use for us.

如果我们知道有人了解Transformer的局限性，他们就可以调度这个模型。

If we if we if we kind of know if there's a human who knows the limitations of a transformer, they can they can schedule that model.

他们可以写一个提示，说明你想要完成的任务是什么。

They can they can write a prompt of what is what is it the task that you want.

对。

Mhmm.

通过我们正在进行的训练，你可以在这方面做得非常成功。

And by doing the training we are doing, you can you can, like, get very successful at that.

模型失败的任何任务，你都可以通过添加训练数据来弥补，然后模型就能成功完成。

And any task the model fails, you get added through training data, and the model and the model can can succeed.

但这个循环必须由实验室为你训练模型来完成。

But that that loop is has to go for the lab training training the model for you.

如果模型本质上需要在实验室里训练，你觉得这有多大程度上是目标，还是说你想以某种方式更新模型，而不必每次都回到实验室？

And if the model that that fundamentally needs to be trained in the lab, like, how much do you think of it that this is this is the goal, or or or you would want to be able to update the model somehow, not having to to go back there?

你读过Rich Sutton和David Silver的那篇论文《经验时代》吗？

Have you read the Rich Sutton and David Silver have this paper, the the age of experience.

你读过吗？

Have you read it?

我很好奇你有多大程度上同意，或者有没有不同的看法。

I'm curious how much you agree or if you have any different opinions.

你的观点在哪些地方不同呢？

Where do your opinions diverge?

强化学习并不是什么特别新的方法，也不是什么特别新鲜的事。

Reinforcement learning is not a particularly new approach, particularly particularly new thing to do.

所以从某种意义上说，经验时代其实一直都在，人们也一直在批评预训练，因为预训练很明显是另一种看待模型的方式，就是我们有静态数据。

So in some way in some way, age of experience, I think I think always has been there, and people have been criticizing a bit pretraining because pretraining very clearly is is the other way of of looking at the models, which is like we have static data.

那些数据大部分是别人生成的。

That data is mostly generated by others.

虽然我个人觉得，现在的预训练很大程度上是在把其他模型的知识蒸馏到新模型里，因为互联网上的大部分 token 都来自 AI，但预训练显然就是行为克隆、模仿、压缩互联网数据。

Although I I have this personal view that pretraining today is largely distilling other models, all their models into the new model because most of the tokens in the Internet are are coming from from from AI, but there there's clearly pretraining, which is which is behavioral cloning, which is mimicry, which is which is compression of of of of Internet data.

但强化学习并不是人们没想过或者没做过的事。

But reinforcement learning is not something that people haven't been thinking and people haven't doing.

强化学习以前用来解决一些问题，后来用来下围棋、玩星际争霸、打 DOTA，现在又用来解决编程问题。

Reinforcement learning was used to solve back in the day, then it used to solve Go, StarCraft, DOTA, to solving programming right now.

每次归根结底都是模型自己生成经验，然后从这些经验中学习。

And every every time it comes down to model writing its own experience and learning learning from that experience.

但有一点很清楚，我觉得有趣、也让人困惑的是，强化学习并不是从经验中学习的唯一方式。

But it is very clear and what I think is interesting and what I think is still is still perplexing to people that reinforcement learning is not really the only way to learn from experience.

而且未来还会有更多，我可以说，更多算法层面的创新，本质上就是关于我们如何从经验中学习，因为强化学习只是其中一种方法。

And there there will be more, and there will be there will be a little bit more of I think you you can call it algorithmic, but essentially innovation of how we learn from experience just because just because reinforcement learning is only one way to do it.

它只是一个数学公式。

It's a mathematical formulation.

特别是现在我们在使用它时，它非常依赖那些并行展开来降低方差，以及比较模型在不同平行世界中的表现，但这并不是我们人类学习经验的方式。

And especially right now how we are using it, it really likes those parallel rollouts for variance reduction and for for comparing how the model does in in in parallel versions of the world, which is not not how we do, not how we learn from experience.

我们从经验中学习要高效得多，也丰富得多。

We learn from our experience much more efficiently and much more.

我们会以多种多样的方式利用这些经验。

We use we use those in many in many, many ways.

有时候我会试着跟别人解释，大脑是怎么运作的，我们是怎么学习的。

At some moment, I've been trying to explain to people, like, what what brain does, how how we learn.

大脑里并不是只有一种学习算法。

It's not that there's one learning algorithm in a brain.

我觉得其实有好几种，它们会协同工作。

I think there are multiple actually, and they work together.

但比如说我踢足球的时候，那看起来就非常接近强化学习。

But if I play football, for example, it looks very, very closely to reinforcement learning.

我反复踢球，每次稍微调整一下，看看是不是大致达到了我想要的效果，然后就会产生某种自我强化。

I kick a ball a lot of times, and every time I adjust it a little bit and I see if it roughly matches what I what I wanted and either there there is some self reinforcement happening.

而当我学数学的时候，那完全是另一回事。

When I when I when I learn mathematics, it's very different type of thing.

就像阅读一些难懂的概念，在脑子里反复琢磨，直到突然想通，直到把它们联系起来。

It's it's like reading about hard concepts and thinking about them very deeply inside my head until until things click and until until I have them connected.

这两种方式在某种程度上都是从经验中学习。

And both of those in some way are learning from experience.

只是它们非常不同。

They are just very different.

所以总结一下我对从经验中学习的看法，那就是我们其实已经在这方面做了很久了。

So so summarizing my thinking of the learning from experience is that we've been doing it for a while.

我们可能比以往任何时候都投入更多的计算资源来从经验中学习，但强化学习并不是从经验中学习的终点。

We probably are spending the most compute than ever on learning from experience, but reinforcement learning is is not the end of learning from experience.

未来几年，研究人员会提出更好的方法，让我们在更丰富的场景中利用这些数据。

And there will be there will there will be better approaches that will researchers will be coming up in the coming years on how to how to use that data in in a richer richer settings.

有意思。

Interesting.

Rohan，我很好奇。

Rohan, I'm curious.

既然你的很多工作都围绕优化和效率，那么我们要怎样才能让学习算法在计算和数据效率上实现数量级的提升呢？

Since a lot of your work has been around optimization and efficiency, how do we get to orders of magnitude more efficient, I guess more compute efficient and more data efficient learning algorithm?

这得从测量开始。

It had to start with measurement.

我认为目前我们定义的预训练，本质上就是压缩。

I think pre training, as we define it, right now is about compression.

我们看困惑度，然后衡量如何降低它，结果发现扩展规模、增加参数数量、投入更多计算资源是有效的方法，而且每次按对数尺度增加计算量，这些指标就会获得一点点提升。

We look at perplexity and then measure, how do we decrease the perplexity and then we find that scaling and increasing parameter count and putting more compute, is the way and every time we increase, compute in log scale, we get this epsilon more improvement in these metrics.

我觉得，这对于建立先验知识来说还行，但看待问题的方式不对。

This is, I think, this is fine for building the prior, but I think this is the wrong way to look at the problem.

我们应该从整体来看。

We should be looking at the end to end.

我们训练这些模型到底是为了什么？

What are we training these models for?

要看最终的结果。

Look at the outcome.

比如说，我训练好这个模型，然后交给Jerry。

Like, for example, I train this model and give it to Jerry.

Jerry一用强化学习，我辛辛苦苦建立的那些困惑度指标就全毁了。

Jerry will do RL and destroy all the perplexity metrics that I have created.

对吧？

Right?

所以，这算是我们目前能想到的最好的解决办法了。

Like so then it's sort of like it was the best way we had so far to attempt to solve the problem.

而且我觉得，各大实验室和其他团队在创造智能方面做得非常出色，这些智能非常有价值，也让我的工作变得特别有趣。

And I think the labs and everyone else have done a great job in producing intelligence that's super valuable and makes my work so much fun.

但这只是达到那个目标的启动过程。

But it was the bootstrap process to get there.

我们必须把预训练和强化学习结合起来，这才是能带来一个数量级提升的地方。

We have to combine pre training and RL together, And that's, like, where one order of magnitude improvement would come from.

这本质上是一种训练流程。

And that's like a training procedure.

你也可以说它是一种学习算法。

You can say it's a learning algorithm.

说到优化，我的故事是从2016年在谷歌做逻辑回归优化开始的，当时被这个问题深深吸引，然后参与了一些求解器的开发，就是我们以前叫SIBL的东西，那是谷歌在神经网络兴起之前使用的大规模线性求解器，后来被神经网络取代了。

In terms of optimization, my story is I started optimization at Google for logistic regression back in 2016, got nerd sniped by it, worked on some solvers for what we used to call SIBL, which was the large scale linear solver that was used at Google before neural network took off and then replaced this.

然后我问自己，我想在神经网络领域做什么？

Then I asked myself, what do I wanna work on with neural network?

答案很明确，我想理解训练算法，并让它变得更好。

And it was quite clear, like, I wanna understand the training algorithm and make it better.

然后有一天，维尼特·古普塔突然出现在我办公桌前。

And then someone, Vineet Gupta, just showed up one day at my desk.

他说，听说你特别擅长写优化方法。

It's like, I heard you're really good at writing optimization methods.

我们有个想法，就是在白板上推演出来的，后来变成了Shampoo算法，你能帮我们实现并规模化，确保它在神经网络训练中能大规模运行吗？

We have this idea that, you know, like, worked out on a whiteboard and what turned out to be the Shampoo algorithm, can you help us implement scale, make sure it works at large scale for neural network training?

所以在我做这个项目的时候，我一直感谢我的经理，他从头到尾都支持我，直到我2024年离职。

So when I was working on this, I thanked my manager, who has supported it throughout that until my end of my tenure, 2024.

但总的来说，这个领域的大多数人并不怎么看好这个想法。

But largely, the community and most of the other people were not as excited by this idea.

而对我来说，这是最激动人心的事，因为我觉得，我是在投入计算资源，让训练变得更好。

And for me, this was the most exciting thing because I was like, I'm putting in computation and making training better.

这是我必须搞明白的事情。

This is the thing I have to figure out.

我会花多少时间就花多少时间，直到把它做成。

I will spend as much time I would take to do it.

然后大家就开始猜测，哎，上限在哪里呢？

And then people were making this assumption, oh, why like, what's the upper bond?

其实你还可以继续用Adam。

You could still use Adam.

这也没问题。

That's fine.

就是说，你完全可以把时间花在其他地方，而不是优化上。

Like, why are we you could spend all the time on everything else, not optimization.

但从某种意义上说，优化就像——你有一个模型，你在优化它，你想让它变得更好。

But in some sense, optimization is like like, you have a model, you're optimizing it, you wanna optimize it better.

现在联系到我们之前讨论的架构问题，实际上，我们在架构方面做了很多工作，就是为了让这些指标能够训练起来。

Now connected to some of the stuff that we talked about architecture, what has happened is that a lot of the work that we've done in architecture is to make these metrics train.

从某种意义上说，这就像硬币的两面，优化和架构是相辅相成的。

And in some sense, it's like two sides of this coin, and optimization and architecture go together.

你可以用更强的优化器，训练一个更难优化的模型，从而获得更好的性能。

You could have a stronger optimizer, train a much more harder to optimize model and get better performance.