AI研究人员辩论我们距离递归自我改进还有多远

对。

Yeah.

对。

Yeah.

也许退一步看，我觉得这就是未来AI研究的计划，你觉得这能行吗？或者你同意这种描述吗？

Maybe taking a step back, here's what it seems to me that the plan for AI research going forward is, and you tell me if you think it's going to work or if you agree with this characterization.

所以，这个赌注是，我们会在数百万个不同的环境中，跨越数百种不同的领域，扩大我们的强化学习和虚拟现实训练。

So the bet is that we will scale up our RL VR training across millions of diverse environments, across hundreds of different kinds of domains.

最终出现的将是一个智能体，它学会了这些基本技能，或者说不仅仅是基本技能，比如坚持不懈、在上下文中对信息进行分诊，最终实现与其他智能体合作的端到端优化，等等。

And what will emerge at the other end is an agent which has like learned these basic skill or less than basic skills around being persistent, being able to triage information in context, eventually having like end to end optimization of working with other agents and things like that.

这样的智能体在特定情境下会非常高效地利用样本。

And such an agent will be very sample efficient within the context.

你知道，你做过研究，关于如何真正扩展上下文学习，让它变得任意长，但你只是不断扩展它。

You know, you've done research on how you actually scale up in context learning to make it like arbitrarily long, but you just keep scaling it up.

所以，最终出来的东西，基本上就像一个可以随时上岗的远程员工，能工作一周或一个月。

And so what comes out the other end will something will be something that it basically functions like a drop in remote worker over the course of a week or a month.

首先，你同意这就是实验室们在下的赌注吗？

First of all, do you agree that that is a bet the labs are making?

第二，这够吗？

And second, is it is that enough?

就是说，基本上是在数据中心里的模拟环境里学习如何学习，然后部署到现实世界，但实际上并不从现实世界的部署中学习，只是从数据中心里的模拟环境中学习这些元技能？

Like basically learning how to learn within the simulacra within a data center and then but getting deployed into the real world, but not actually like learning from real world deployment, only learning these meta skills from the simulated environments in the data center?

对。

Yeah.

我觉得现在很难分清实验室的努力有多少是直接用于递归自我改进，又有多少是用于制造通用智能模型，以便继续部署、赚取收入，来资助下一次大规模训练。

I think it's now hard to separate out like how much of the lab's effort is going towards like direct RSI versus like making generally intelligent models that they can continue to deploy collect revenue to fund the next big training run.

对于后者，我觉得那大概就是他们在下的赌注。

I think for the latter, like yes, that's probably just the bet they're making.

而且过去几年这些环境的发展方向是非常清楚的。

And it's very clear like the pattern of like where these environments are going over the last few years.

比如说，Anthropic 的环境演变就是一个很明显的例子。

I mean like Anthropics lineage of environments is like a very clear example of this.

比如说，你知道，一开始我们只专注于编程，我们会在这方面变得非常非常擅长。

Like, you know, first like we just focus on coding and like we're gonna get really really good at that.

然后，我们从编程中得到的任务范围，这可能是最容易实现的目标，因为互联网上有大量数据可以用来创建环境，比如他们自己的内部数据，可以转化为环境。

And then the task horizon that we've got from coding which is probably the lowest hanging fruit in terms of like data available on the internet to create environments like their own internal stuff that they can turn into environments.

然后我们会推广开来，接下来会进军金融领域，那里有大量的Excel数据，以及所有类似的东西，都可以用于强化学习训练。

Then we're gonna generalize, we're gonna go after finance next and literally just so much Excel data and all that sort of stuff in the RL training.

然后是PowerPoint，这是工作经济中的长尾部分，这似乎效果很好，而且很多其他实验室，甚至开源实验室，现在都意识到那是正确的做法。

And then it's PowerPoints, it's this long tail of the working economy and that seemed to work really well and like a lot of the other labs and thing, even the open source labs have now realized that that was the correct that.

那么，这有什么影响呢？

And so but what is the implication from that?

当我邀请Dario上播客时，我问他的问题是，如果你真的期望模型在岗位上学习的能力像人类一样，那为什么还要试图把所有这些技能，比如处理PowerPoint之类的，都预先内置进去呢？

When I had Dario on the podcast, the thing I asked him was, if you truly expect models, which will be human like in their ability to learn on the job, Why would you try to bake in all these skills of like working with PowerPoint or something?

难道你不期望模型在部署后就能自己学会这些吗？

Wouldn't you just expect the model to be able to pick that up on while it's deployed?

所以，是的，这里有多种不同的解释。

And so yeah, there's multiple different explanations.

一个原因就是，我们预期模型很快就能达到那个水平，但还没到那一步。

One is just that this is we expect models to get there soon but they're not there yet.

那为什么不把这些技能直接融入到模型训练里呢？

So why not amortize these skills into the model training?

另一个原因是，我们并不是专注于让它在广泛部署的工作中表现特别好。

Another is that we're not concentrated on making it really good at widely deployed work.

我们只是希望它在RSI上非常出色，这就像是我们的一个收入来源，这样我们就能把赚到的钱再投入到真正擅长RSI开发的模型中，然后一旦奇点到来，最终出来的那个东西就会擅长所有那些对当前一代模型来说像是瓶颈的事情。

We just want it really good at RSI and this is just like a way up for us to like get revenue so that we can pour it back into a model that is actually like really good at doing RSI development and then like once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models.

对，约翰，不知道你怎么看，就是你怎么理解这些模型里为什么有这么多特定任务的知识，如果路径是这样的泛化的话。

Yeah, John, don't know if have a taste on like what how once you construe why there is so much task specific knowledge in these models, if the if the path is like this kind of generalization.

对。

Yeah.

我的意思是，如果模型在上下文学习方面足够好，理论上你就不需要专门训练它们处理金融。

I mean, if the models were good enough at learning in context, then in theory you need to train them on finance.

它们就能自己边读边学，读所有相关的书，然后弄清楚在相应的管辖范围内该怎么做所有事情。

They would just be able to figure out, read all the books on the fly and figure out to do everything in the appropriate jurisdiction.

对，而且你可以说，做这么多领域特定的训练，就是为了让它们更高效。

Yeah, and you could argue that you need to do a lot of this domain specific training just to make them more efficient.

所以即使它们聪明到能当场搞定，你还是可能想做一堆强化学习，把这些直觉都烧进权重里，这样模型在运行时会更高效。

So even if they were smart enough to figure this out on the fly, you still might wanna do a bunch of RL and bake all these intuitions into the weights, so the model would be more efficient at runtime.

对。

Yeah.

是的，我觉得实际上，模型提供商确实是一个领域一个领域地去强化，试图在价值最高的领域里加强模型。

Yeah, I'd say in practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest value domain.

我觉得这也是为什么模型变得这么厉害的原因之一。

And I'd say that that's one of the answers to why the models have gotten so much better.

就是因为模型提供商覆盖了很多高价值领域和最常见的技能类型。

It's just because the model providers have covered a lot of the high value domains and the most common types of skills.

我觉得另一件事就是，同时做这两件事其实成本不高。

I mean, I think another thing is just that, like, it's not that expensive to do both at the same time.

对吧？

Right?

因为模型规模很大。

Because, like, the models are massive.

它们有足够的参数空间，可以轻松学会所有东西。

They can easily afford in terms of that parameter to, like, learn everything.

而且很可能存在某种迁移学习，即使金融不是特定领域，这些信息对RSI也很重要，只是如何找出重点、如何培养品味、如何进行长期工作的通用元学习能力，可能是可以泛化的。

And there is likely some transfer in sort of even even if finance is not specific, the information is important for RSI, just the general meta learning of how to figure out what's important, how to have taste, how to do long horizon work is potentially generalized.

而且世界上RSI的数据也不多。

There's not that much RSI data in the world as well.

生成这些数据挺难的，需要大量努力，如果你能利用其他数据来分摊成本，从中获得一些迁移，你已经有海量的计算资源和庞大的参数空间，为什么不也这么做呢？

It's kind of hard to generate, that requires a lot of effort to if you can sort of amortize in this other data, get some transfer from it, you already have masses of compute and massive parameters space, so why not do that as well?

这显然是直接卖模型的商业动机。

It's obviously the direct commercial incentive of selling your model.

有道理。

That makes sense.

哦，我想补充一点，就是关于当前这种从模拟到现实的做法是否会成为主流，还有一个问题。基本上，你观察现实世界的任务是什么样的，然后尝试创建一堆可以在数据中心模拟的环境，然后对它们进行强化学习。

Oh, I'll add that I mean, there's one question about whether this current paradigm of doing like sim-to-real will be the dominant one So basically, you look at what the real world tasks, are like, and then you try to create a bunch of environments that can be simulated, like in the data center, and, you can do RL on them.

而且我觉得，显然，这种方法已经非常成功了，但它也有很多弱点，因为

And I think, obviously, has been very successful successful, but it has a lot of weaknesses because a

很多

lot of

东西就是很难模拟，尤其是那些需要实时和一群人互动的场景。

things are just kind of hard to simulate, especially if they involve interacting with a bunch of humans in real time.

对。

Yeah.

所以有人质疑，模拟到现实这种方法会不会永远占据主导地位。

So there's some question about sim- like whether sim- to real will be the dominant framework forever.

我觉得在样本效率还比较低的情况下，模拟到现实肯定是主导框架。

I think sim- to real has to be the dominant framework, while sample efficiency is kind of low.

因为现在你需要和人类进行成千上万次的互动，没有人会愿意坐在那里，基本上就是参与强化学习的训练循环。

Because like, right now you need like thousands and thousands of interactions with the humans, and no human is gonna sit there and deal with this, basically be in the loop of RL training.

对。

Yeah.

所以我们现在得模拟这些情况来获取样本，但显然，如果样本效率大幅提升，你会期望从实际部署中学习成为更重要的部分。

And so we kind of have to simulate that now to get But the samples you obviously if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it.

不过还有其他办法，比如你可以进行离策略学习，不必使用所有轨迹，即使不重新模拟一切，也有可能从中学习到一些东西。

Though there are also other things you could do, like you can learn off policy, you can not take all the traces and even without re simulating everything, you can potentially learn something from them.

Jane Street 刚刚推出了一项新竞赛，这是他们迄今为止最有野心的一次。

Jane Street just launched a new competition and it's their most ambitious one yet.

设计一个协议仿真器 ASIC。

Design a protocol emulator ASIC.

基本上，如果你有一块想测试的芯片，你可以把它连接到这个 ASIC 上，然后这个 ASIC 会模拟真实的流量。

Basically, if you have a chip that you wanna test, you can connect it to this ASIC, and then this ASIC will simulate realistic traffic.

这样，你就能看到芯片如何响应，而无需将其接入实时系统。

That way, you can see how the chip responds without having to plug it into a live system.

Jane Street 在寻找灵活、通用的设计，而不是单一协议的仿真器。

Jane Street is looking for flexible, general purpose designs, not single protocol emulators.

我和他们聊天时，他们建议我先从实现三个非常常见的协议开始，分别是 UART、SPI 和 I2C。

When I was chatting with them, they suggested that I start off by trying to implement what are apparently three very common protocols, UART, SPI, and I2C.

Jane Street还提到，他们希望更有雄心的设计也能涉及低速USB和以太网，以及其他任何能发挥你芯片特定架构优势的协议。

Jane Street also mentioned that they hoped that more ambitious designs will also tackle low speed USB and Ethernet and any other protocols that flex your chip's specific architecture.

重要的是，你的设计应该是可重新编程的，而不是把一堆特定协议硬塞到芯片上。

Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip.

如果你的ASIC流片后出现了新协议，你的芯片仍然需要能够处理它。

If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it.

具体怎么做由你决定。

How exactly does it is up to you.

但有一个硬性限制。

But there is one hard constraint.

你的设计必须针对开源的130纳米工艺节点。

Your design must target an open source 130 nanometer process node.

这是因为Jane Street会为最新颖的提交支付流片费用，并将实体芯片寄给获胜者。

That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners.

比赛开放至2027年1月18日，强烈鼓励团队合作。

The competition is open till 01/18/2027, and working in Teams is highly encouraged.

去janestreet.com/thorcache下载模板代码，开始动手吧。

Go to janestreet.com/thorcache to download the template code and get started.

我想多问一点这个，因为有点奇怪，你有50%的计算资源花在推理上，而这些推理并没有直接帮助模型变得更好。

I wanna ask more about this because it's sort of weird that you have 50% of compute that's spent on inference that is not directly helping the model become better.

比如，你最终期望数字思维的一个关键优势是，不像人类能有五十年的真实世界经验，一个模型通过它的所有实例，能经历数百万年的部署，覆盖经济中各种有经济价值的工作。

Like one of the key advantages you'd expect eventually digital minds to have is unlike a human who gets to have fifty years of like real world experience, a model will get to through all its instances, get to experience millions of years of deployment across all kinds of economically relevant work in the economy.

而现在，这些数据在某种意义上并没有真正帮助模型变得更好。

And right now that data is just not in a meaningful sense helping the model get better.

这看起来太明显了，最终模型应该能从这些数据中学习，一旦它们做到了，你就会拥有一种几乎像是广泛部署的智能爆炸的东西，因为模型在所有这些部署实例中吸收了如此多的信息。

Like it just seems so obvious that eventually models should be able to learn from this data and once they do, you would have something that almost feels like a widely deployed intelligence explosion because the model is assimilating so much information across all these deployed instances.

但你预计这种蜂巢思维般的疯狂事情什么时候会开始呢？

But when do you expect this kind of hive mind kind of crazy shit to be start

我觉得大体上，在最基本的层面上，这已经在发生了。

I think broadly, like at a very basic level, this is already happening.

对吧？

Right?

比如说，就在下一代模型里。

Like, just in the next generation of models.

所以现在，你总是可以把你的部署数据拿过来，放到未来模型的预训练或中期训练里。

So like right now, you can always see take your deployment data and put this in the pre train or the mid train of like future models.

尤其是如果你做了一些过滤，或者某种判断、标注，或者最近那种合成处理。

Especially if you do like some kind of filtering or some kind of like judgment or annotation or like recent, you know, synthesization of that.

你觉得这在多大程度上解释了模型一代比一代进步的原因？

How much do you think that explains the generation over generation improvement?

我觉得解释了不少。

I think it explains like quite a bit.

尤其是，我是说，这个，你知道，我不确定那些实验室会不会这么做，因为理论上他们声称不会用用户的数据来训练。

Mean especially like, I mean this is, you know, I don't know whether the labs do this because theoretically they claim not to train on people's data.

但中国的实验室百分百会这么做，而且他们确实得到了这个优势，显然在部署上也是。

But like the Chinese 100% do and like they definitely get this advantage both like obviously deploying.

这基本上就是蒸馏的过程，他们取出模型，获取一些部署数据，通过调用模型得到其中一部分，然后用这些数据来训练下一代模型。

Is This basically what distillation is, they take out the models, they get some of their deployment data, they get some fraction of that by pinging the model, and then they train their next generation models on it.

而且他们也能在自己的模型上做到这一点，完全没有理由不这么做。

And they can suddenly do it on their own models as well, there's no reason not to whatsoever.

我

I

完全同意这个观点。

completely agree with this.

我觉得如果你把视野放得足够远，这确实正在发生。

I think if you zoom out far enough, this is definitely happening.

就像你想象的，我们都在想象的，这就是持续学习的终极目标。

Like you're picturing this like, and we're all picturing this, this is like what continual learning, like the holy grail is.

就像是一个非常非常有机的实时循环，一个模型获得经验，然后实时更新，从中学习。

It's like this very, very organic live loop of like an individual model, like getting an experience and like live updating on the spot and learning from that.

当你深入到那个细节层面时，很多事情就会出问题。

And a lot of things break when you zoom into that level of granularity.

但大实验室在做这个，闭源模型也在做这个。

But the big labs are doing this, the closed models are doing this.

也有一些早期迹象表明，有人用开源模型在做这件事，而且频率快得多。

There's also early signs of life of people using open source models doing this at a much faster cadence.

一个很好的例子可能就是Composer。

So a good example is probably Composer.

你有某种模型，能够做到这一点，或者Harvey也在用法律代理做同样的事情。

You have some sort of model and are able to or Harvey's doing the same thing with legal agents.

它从你为特定任务拥有的数据中获取非常具体的情境，还有用户抱怨的事情，以及你从特定部署中提取的所有反馈，很多这些公司相比大实验室的优势在于，他们能非常好地利用这些数据。

It is getting very specific environments from the data that you have for that particular task and things that users are complaining about and all the feedback that you're somehow extracting from your specific deployments and a lot of these companies have the advantage over the big labs in that they can use this data really really well.

然后他们会创建环境，会对Kimi K3进行大规模的后期训练。

And then they will create environments, they will do a big post train of Kimi K3.

他们会去部署它。

They'll go deploy it.

他们可能也会做一些在线学习，就像Composer那样，基本上长期进行在线强化学习。

They might do some online learning as well like Composer did online basically like re:Inforce for a long time.

所以是的，仍然有人类参与其中，仍然有人类在说，好的，这些是我们关心的信号。

So yeah, there's still a human in the loop, there's still a human saying okay, are signals we care about.

我们打算这样从我们手头的数据里创建环境。

Here's how we're gonna create environments from the data that we have.

而且这个节奏可能比你想象的要慢一些。

And there's like still a longer cadence than maybe the one that you're thinking of.

但确实是在发生的。

But like it really is happening.

而且最终这个循环会变得越来越快。

And like eventually that loop will become like faster and faster.

我觉得Composer这事儿挺有意思的，因为这就是那种模型，在Cursor里，人们按Tab键或者不按Tab键，来决定模型建议的下一个补全。

I mean the composer thing is interesting because this is where the model like in cursor people like press tab or they don't press tab on the next completion that the model suggests.

基于这个，Composer每天都能更好地预测下一个。

And based on that every single day, composer gets better at like predicting the next

那是旧的Tab模型。

So that was that was the old tab model.

他们其实对真正的生成模型也做了同样的事，不只是Tab模型。

Like they actually did the same thing for the actual, not just like the tab model but the actual like generative model.

哦，这挺有意思的。

Oh, that's interesting.

这确实很难，因为做在线强化学习的时候，你没有对照组，对吧？

And it's hard because when you do online reinforcement learning, you don't have groups, right?

你只有一个用户说了一句话，然后你得到一次反馈。

You just have one user saying one thing and then you get one rollout.

所以你就面临一个很大的方差缩减问题。

And so like you have a big variance reduction problem.

而Kursar对这个问题的回答有点模糊，就是说，哦，我们有很好的启发式方法，能够估计这个回答比平均水平好多少或者差多少。

And like Kursar's kind of fuzzy answer to this was like, oh you know we have very good heuristics which are able to estimate how much better than average this response was or how much worse than average this response was.

然后他们会做一次大的强化更新，至于判断模型是否变差，他们的办法就是看它在cursor bench上的表现有没有提升，如果提升了，就每五个小时部署一次新模型，如果没有提升，就把那个版本扔掉。

And then they would do this big reinforce update, and then their solution to whether it got worse or not was if it improved on cursor bench, they would deploy the new model every five hours, and if it didn't, they would throw that version out.

有意思。

Interesting.

对。

Yeah.

我觉得你最大的问题其实是，对于自然数据，你根本不知道奖励函数应该是什么。

Think your biggest problem is actually just not knowing what the reward function should be for natural data.

如果你用一些表面的信号，比如他们是否接受了代码或编辑，那可能会以某种方式被奖励黑客利用。

And if you use some kind of superficial signal like did they accept code, the edit, that might get reward hacked in some way.

但这看起来是模拟到现实的一个更大的问题，任务的时间跨度越长，就越难在数据中心里模拟，对吧？

But this seems like a bigger issue with the sim-two real thing where the longer and longer horizon tasks get, the harder they are to simulate within a data center, right?

在我看来，即使在编程领域，我们可能已经快要到这样一个地步：没有一个长达一年的编程任务最终不需要你去跟客户沟通，或者跟公司互动，或者跟用户互动。

It seems to me already potentially at least even in coding, we're getting to over the point where there's like not some year long coding task that doesn't eventually require you to like talk to a client or interact with the company or interact with users.

如果你想想我们希望AI能胜任的所有事情，你最终希望超级智能的人能经营企业，或者创办新企业并盈利，或者在市场上做日交易赚钱，或者打赢官司。

And if you think about the gamut of things we would want AI to be capable at, you want eventually super intelligent people to like run a business or like start a new business and make it profitable or like have a profitable day trading in the markets or win a court case.

这些都是很难在数据中心里模拟的事情。

And these are all things which are very hard to simulate in a data center.

比如，学习的一个固有部分就是与现实世界互动。

Like an inherent part of the learning there is interacting with the real world.

所以也许他们可以通过模拟到现实的迁移来学习如何在这些事情上做得更好。

And so maybe they they may learn how to get better at these things from, like, the transfer between sim to real.

但换个角度想，也许你真的需要从这些互动中获取权重更新，才能在这些事情上做得更好。

But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them.

如果真是这样，如果迁移学习不够强，确实需要权重更新，那么模型样本效率低这个问题可能就更严重了。

And then if that is the case, if transfer isn't strong enough and you need do need weight updates, then the fact that the models are quite sample inefficient is like maybe a deeper problem.

我之所以好奇这个，是因为我觉得默认情况下，我看不出未来十年内怎么不会出现一些疯狂的递归现象。

And the reason I'm curious about this, I feel like by default, I don't see how you don't get some kind of crazy recursive stuff from prevent within the next ten years.

但可能不会发生的一个原因是，就权重更新而言，样本效率似乎远远落后于人类，对吧？

But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans, right?

比如，人类从出生到成年看到的数据量，和模型从冷启动到训练完成看到的数据量相比，可能差了一百万倍。

Like plausibly million fold behind humans in terms of how much data a human sees from birth to adulthood versus how much a model sees from, you know, like cold start to like finish it finishing training.

所以，是的，这就是我想说的。

And so, yeah, this is all to say.

首先，模拟和我们在现实世界中希望AI完成的那些极其长期、极其复杂的真实任务之间，会不会有迁移？

First of all, is there is there gonna be a transfer between simulations and extremely long horizon, really complicated real shit that we want the AI to do in the real world?

如果没有，那是不是意味着这些模型缺乏样本效率的问题最终会给我们带来麻烦？

And if not, does that really mean that like the the lack of sample efficiency in these models comes to bite us?

我觉得，可能我区分两类任务的方式是，一类是模型能做得好的，另一类是模型还会继续挣扎的，区别就在于任务是不是累积性的，或者你是不是面对一个非平稳的分布，需要不断学习和重新处理很多事情。

I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is cumulative or you have this non stationary distribution you have to keep learning and relitigating a bunch of stuff.

所以，累积性任务的一个例子可能就是RSI。

So maybe an example of a cumulative task might be RSI.

比如，理论上有可能用一个不到一百万token的Python文件，从头训练出一个能够递归自我改进的模型。

Like it's theoretically possible to maybe have less than a million token Python file which from scratch trains a model that is capable of recursive self improvement.

而且你每发现一个东西，就像在沙地上画了一条线，你可以守住它。

And every discovery that you make is kind of a line in the sand that you hold.

比如，如果对于RSI来说，我们不需要发现新的注意力变体之类的。

Like if it's true that for RSI we don't need to discover a new attention variant or whatever.

就像你一旦发现了注意力机制，一旦发现了专家混合，一旦发现了gRPO，你就把它加到训练栈里，它就在那里了。

Like once you've discovered attention and then once you've discovered a mixture of experts and once you've discovered gRPO, you just add that to the training stack and that's there.

一个很好的例子就是5.6的sole训练，5.6 terra，或者OpenAI让它训练的那个。

And a good example of this is 5.6 sole training, 5.6 terra or whichever one OpenAI told it to train.

它不需要回头再去发现retention。

It didn't have to go back and discover retention.

基本上它可能就会调用一堆脚本，比如预训练脚本和后训练脚本，然后就那样做了。

Like it basically probably would have called a bunch of like scripts which is like pre training dot sh and post training dot sh and just did that.

所以这就是一个累积型任务的例子。

So like that's an example of like a cumulative task.

我觉得现实世界，以及为什么人们这么关注持续学习，是因为它并不是真正的累积型任务。

I think the real world and the reason like people are thinking so much about like continual learning is it's it's not really a cumulative task.

想象一下，在律师事务所里，有一个代理扮演法律助理的角色。

Like imagine like in a law firm you have an agent acting as a legal associate.

那是一个非常非平稳的分布。

Like that's a very non stationary distribution.

你需要在上下文中容纳所有重要人物之间的关系，而这些关系还在不断变化。

You have to be able to fit in your context like all the relationships between all the important people at that company which are also changing all the time.

你还要处理所有那些隐性的做事方式、去哪里找信息等等。

You have like all these implicit ways about how things are done, where to find information, etcetera.

而这就没有像RSI那样清晰的累积型任务的例子了。

And like that's not as clean of an example of accumulative tasks like RSI is.

所以我觉得任务之间会有这种划分，但你知道，如果实验室意识到这一点，而且他们确实相信RSI是累积性的，也就是说我们不需要回头去发现什么全新的架构之类的，对。

So I think that there will be this breakdown between tasks but you know, it's like if the labs realize that and they they do believe that RSI is cumulative in the sense that like we don't need to go back and discover some brand new like architecture or whatever Yeah.

那么也许越来越多的精力和算力会集中在这方面，而不是在……

Then maybe more and more effort and compute gets focused on that versus the

真遗憾，RSI碰巧比所有其他任务都容易。

It's so unfortunate that RSI happened to be easier than apparently all.

对，不知道你们对此有什么想法。

Yeah, don't know if you guys have thoughts on this.

对，我想说，现在的模型在很多方面都比人类弱，其中一些可能和特定情况下的样本效率有关。我的意思是，在某些情况下，模型非常样本高效，比如在上下文中学习，但在某些中等长度的情况下，它们可能不那么样本高效，因为人类可以比模型更高效地进行某种权重更新。

Yeah, would say there's, models are, today's models are weaker than humans in a lot of different ways, and some of them might have to do with sample efficiency in a certain regime, I mean, in some regimes, models are very sample efficient, like learning in context, but then there might be some medium length regime where they're less sample efficient because humans can do some kind of weight update more efficiently than models.

所以我认为在某些情况下样本效率较低可能是弱点之一，但还有其他完全不同的弱点来源。

So I think being less sample efficient in certain regimes might be one of the sources of weakness, but then I think there are other sources of weaknesses that are completely different than that.

比如，思维多样性不如人类，或者在某些长期判断上表现不佳。

For example, having lower diversity of thought than humans, or being bad at certain kinds of long horizon judgments.

我的意思是，我觉得很多人所说的品味，其实是一种长期有效的行为，而且人们已经意识到它长期有效。

I mean, I think a lot of what people call taste is something about behavior that works in the long run and that people have realized works in the long run.

不是所有方面，但品味的一部分是这样的，尤其是在软件工程这类领域，我觉得很多品味就是关于哪些系统在项目长期运行中更容易维护、表现更好。

Not everything, but like some aspect of taste, And especially for something like software engineering, like I think a lot of taste is like what are the systems that are gonna be maintainable and work well in the long run of this project.

所以，是的，

So, yeah,

我觉得

I think

人类的弱点限制了RSI以及其他方面的发展，这些弱点多种多样，有些和样本效率有关，有些则无关。

the weaknesses of humans which limit RSI along with other things are, yeah, there's a variety of them and some of them are related to sample efficiency and some of them aren't.

也许一个有趣的思想实验是，如果你能给模型一个上下文窗口，比如一万亿个token，或者你需要把你在RLHF之前的所有经验都放进去。

Maybe an interesting thought experiment is like, if you were able to give a model like a context window of, I don't know, a trillion tokens or whatever you would have needed to fit in like your experience prior to like let's say RLHF.

而且它把所有这些经验都放在上下文窗口里，并且拥有和一百万token时一样的样本效率和上下文学习能力。

And it's got all that experience in the context window and it has the same sample efficiency and in context learning ability as it does at a million tokens.

你觉得品味问题就此解决了吗？

Do you think taste is then solved?

它能否做出和你一样的判断，还是说除了更长的上下文窗口和相同的样本效率之外，还有根本性的缺失？

Would it be able to make the same judgments that you did or is there something fundamentally missing apart from just a longer context window with the same sample efficiency?

是的，我的意思是，它得经过训练才能从那个上下文里学习。

Yeah, I mean it would have to be trained to learn from that context.

所以我不确定，是的，要么它得经过训练，学会从那个上下文里做出正确的更新，要么……

So I'm not sure, yeah, either it would have to be trained to learn the right update to make from that context or

所以你得把所有的都吸收进去？

So you you have to take all in?

比如你整个一生的研究经验。

Like your whole life research experience.

我的意思是，你还是需要数据来训练它处理长上下文，对吧？

I mean, you still need the data to train it long context, right?

就算理论上你能有一万亿的上下文，你也需要一万亿长度的数据来训练它。

Like even if you could theoretically get like a trillion context, you would need a trillion lengths of data to train it.

就像现在，你有八个上下文，呃，在倾倒……

Like right now, you have, like, eight context e coms dumping

如果你有那个的话……

a if you had that

理论上，我觉得是的。

In theory, think, yes.

其实，这归根结底就是，品味能不能从较短时间跨度的事件中学到。

I mean, this really just comes down to the question of, like, how meta learnable is taste from, like, shorter horizon episodes.

而且，我觉得没有明显的理由说它需要很长时间，因为人类在没有很多长事件的情况下也能培养出品味。

And, like, I feel like there's no obvious reason it's super long because, like, humans somehow develop taste with not having many long episodes.

我们不会活到一万岁。

Like, we don't live to be, 10,000.

我们发展得很快。

We have, like know, we we develop pretty quickly.

对吧？

Right?

所以，你想，即使在博士阶段，第一年的博士生和最后的博士后或类似的人之间的差别，也就大概五年。

And so, like, you know, if you think about, like, even, like, in a PhD, the difference between, like, a first year PhD student and, like, a final, like, postdoc or something, that's, like, five years maybe.

他们总共可能只做了十到三十个研究项目，但不知怎么的，他们从相对较短的一系列小事中很快就培养出了品味。

And they've only done maybe ten, fifth, 30 research projects in total, but somehow they develop taste quite quickly from a relatively short succession of small things.

所以理论上，这样发展是可能的。

And so theoretically, it's possible to develop it like that.

AI显然会有更多的经验来发展这种元学习的品味。

The AI obviously will have vastly more experience in which to develop tastes like meta learned.

然后问题就是，这种品味能很好地推广到非常长期的任务上吗？

And then it's like, how well does that generalize to like really long horizon things?

我觉得这个问题目前真的还没有解决。

Is I think the question which I think is really unsolved at this point.

我们不知道。

Like, we don't know.

回到这个问题，最终应该会有一个阶段，AI从每次部署的每个实例中学习到大量东西。

Going back to this question, eventually, should be a regime where AIs are learning a ton from each individual instance of deployment that they have.

嗯，目前可以说有一个元模糊的过程，模型通过部署来改进，但我觉得这是一个非常弱的反馈循环。

Well, currently, you could say there's a meta fuzzy process by which models do improvement for deployment, but I feel like it's a very weak very weak feedback loop.

你觉得在不久的将来，会不会出现这种非常快速的集体智慧学习？

Do you see this around the horizon where there's this hive mind kind of learning that's very rapid?

如果是这样，具体是怎么发生的呢？

If so, how exactly does it happen?

实际上，我想说的是，我们会不会有一个能从所有部署经验中学习的群体智能？

Actually, would say that around will we get a hive mind that learns from all of its deployment experience?

我觉得这很大程度上其实是激励问题，而不是技术问题。

Mean, a big part of that is actually about incentives rather than being a technical question.

所以，公司不会希望模型提供商从他们的部署中学习，因为那可能会削弱他们业务的优势。

So like companies aren't gonna wanna have the model provider learn from all of their deployment because that might just reduce the advantage of their business.

我觉得经济因素可能会推动的不是更新一个大的共享模型，而是像模块一样的东西被插入进去。

I think that maybe the economics of this will pressure not necessarily weight updates to one big like common shared model, but like kind of like modules that get subbed in.

比如一个很明显的例子就是LoRA，但也可能是别的东西，比如有很多工作尝试把任意长度的上下文压缩到固定大小。

So like a very obvious example, this is a LoRa, but it might be something else like, you know, there's been a lot of work to try and fit like an arbitrary context length into a fixed size.

就像所有那些线性注意力之类的东西，还有像“卡带”那样的，本质上是经过训练的高度压缩的KV缓存，用来容纳大量信息。

Like this is all the linear attention stuff and all that sort of stuff and like cartridges which are essentially KV cases trained to be very very compressed KV cases to fit in a lot of information.

这是另一个公司可能愿意参与的例子，如果这些模块被插入到模型中，而不是真正改变底层模型本身的话。

That's another example of something that companies may be willing to sign up for if that gets subbed into the model and it's not like actually changing the base underlying model itself.

所以，实时从你的数据中学习有很多不同的方式，而后面那些方式其实对大实验室没什么帮助，因为它们只是些模块。

So there's many different versions of learning from your data in real time and the latter ones are not really helping the big labs because they are just these modules.

但我认为经济压力会迫使实验室先走那条路，然后才能开始做这个。

But I think the economic pressure will force the labs to go down that path first before they can embark on this.

哪种经济压力？

Which economic pressure there?

因为我觉得，即使你有一堆卡带或者LoRa之类的东西，你还是可以把所有这些痕迹拿过来，然后蒸馏一下，倒进你下一代模型的预训练里。

Because I feel like even if you have a bunch of cartridges or laws or whatnot, can still just take all these traces and just like distill this, dump this to the pre training of like your next generation of models.

对。

Yeah.

所以，大实验室可能获得的一种更间接的学习方式，这对他们来说显然仍然非常有价值。

So it may be a more indirect form of learning that the big labs are getting and that's obviously still really valuable to them.

但我无法想象一个世界，我们会一开始就直接把所有精确的数据都用来训练这一个大型模型，而

But I can't imagine a world in which we start off with like, you know, we're going to just directly train this one big model on all the exact data that

不，我们不会。

we're No.

我觉得肯定会经历几个阶段，因为我是说，这假设有一个不连续的事件，突然我们解决了，比如持续更新，但实际上我认为更可能是，那些模块让你在部署时能专门化。

Think it will definitely go through stages because I mean this is assuming there's like one discontinuous event where suddenly we fix like wait updates continuously and in practice I think it's much more likely to be like, the cartridges and stuff allow you to specialize in deployment.

然后你生成轨迹，把它们放进你的模型里。

Then you generate traces, you put that in your model.

三个月后，出一个在这些方面更好的模型。

Three months later, come out with a model which is better with this stuff.

你再专门化它，再整合它。

You specialize it again, you consolidate it again.

然后最终，我们会让这个跳跃越来越快。

And then eventually, we'll just make this leap faster and faster.

不再是每三个月发布一个模型。

Instead of every three months, we release a model.

现在变成每周，然后每天，然后每小时，到那时我们基本上就已经解决了。

Now it's like every week and then every day and then every hour at which point we basically have obviously solved it.

对。

Yeah.

而且我觉得这一点也很重要，因为你刚才问的是，我们离当前这种范式能实现这个目标还有多远。

And I think this is a good point as well because you asked like kind of how far off the current paradigm we are from being able to do this.

我们在这方面做过一些研究，而且很多人也做了大量研究。

We've done a bit of research to this and people have done a lot of research.

在大规模的情况下，当你把足够的噪声过滤掉，并且有足够大的数据块时，这种把数据放入中间训练、创建我们自己的环境的外循环过程，在某种持续学习机制下确实是有效的。

At a really large scale, when you wash out enough noise and you have large enough patches, this outer loop process of putting data into mid training, creating our own environments, it does work in some sort of continual learning regime.

但问题是，当你把镜头拉近到微观层面，比如我有一个模型，我想为一家律师事务所之类的持续更新它，而且数据量相对较小，还要非常频繁地更新。

But the problem is when you zoom in close enough at a micro level, it's like I've got one model and I'm trying to update it again for like a law firm or something and I'm trying to do that very continuously like with a relatively small amount of data.

这时候所有的方法都有点失效了。

Like all the methods kind of break down a bit.

比如，如果我只用成功的轨迹，不管是在策略还是离策略，对模型做监督微调，在非常迭代的机制下，比如做几百次这种微调，你会看到灾难性遗忘，会忘记之前学到的信息，这些信息是在基础模型之上更早学到的，而且还会看到通用能力和通用性能的下降。

So like if I SFT the model on just like successful traces off policy on policy, eventually in the very iterative regime, like when you're doing hundreds of these micro updates, you see both catastrophic forgetting, you see forgetting of previous information I've learned on top of the base model that was much earlier on and I see degradation of general use, general capabilities.

在策略蒸馏似乎能把这个问题推迟一些，但最终还是会遇到同样的问题。

On policy distillation seems to push this horizon out a little bit but it still eventually succumbs to the same thing.

强化学习在这方面不是很好，它擅长获取能力，但不擅长获取知识。

And RL is not very good at it is good at getting capabilities in but it's not as good as getting like knowledge in.

而且就是那种非常明确的知识，比如，好的，这个人在这个律师事务所做这个，这是我们发现的一个非常具体的流程。

And like just this very explicit knowledge of like, okay, like this person does this at this law firm and like this is a very specific process we find.

而且你得投入大量的计算资源来创建合适的环境，才能把这些知识灌输进去。

And you have to put in a lot of compute to create the right environments to get the the knowledge in

强化学习。

the RL.

你觉得这里根本的问题——为什么你在其他技能上会变差，或者出现遗忘之类的——你觉得这根本上是容量的问题，还是技术的问题？

Do you think that the fundamental issue here, why you get worse at these other skills or there's forgetting and stuff, do you think it's fundamentally an issue of capacity or it's an issue of techniques?

两者

A little

都有一点。

bit of both.

我觉得像SFT，甚至像蒸馏，比如在策略蒸馏，可能破坏性太大了。

Think like SFT and even like distillate, like on policy distillation can be like way too destructive.

强化学习之所以这么好，是因为它对模型的改变非常非常小，而且有很多证据支持这一点。

The reason RL is so nice is because it changes a very very small amount about the model and there's like a lot of evidence for why this is the case.

所以它只是用非常非常小的损失值来微调模型，让它达到正确的状态。

And so it kind of just tweaks it in this very very very small like loss value to like get it into the right point.

但这也限制了RL能做的事情，比如你实际上能对模型做多大的改变。

But that also then limits what you can do with RL, like how much you can actually change the model.

嗯，抱歉。

Well, I'm sorry.

你是说，这可能不是赢家通吃的原因，是因为很难把那么多信息压缩进去。

You're saying like the reason this isn't the winner take all potentially is that it's just like very hard to distill that much information into

基础模型？

the base model?

在不破坏整数版本中的某些东西的情况下。

Without ruining something in an integer version.

对。

Yeah.

比如，把它蒸馏到另一个基础模型里很容易。

Like, it's easy to distill it into like a different base model.

我觉得这主要是技术问题。

Like, this is where I think it's mostly technique.

并不是说模型没有能力，绝对不是。

It's not like it's definitely not like just like there isn't capacity.

比如说，如果你有一个模型，你知道，带着所有这些数据，然后你用同样大小的模型，从头开始预训练，把所有这些内容都加进去，那它会更好。我觉得现在很多情况就是这样。

Like, if you had some model, you know, with all this data and you take like literally the same size model and pre train it from scratch with like all of this stuff in mid training, it will be better, and I think that's a lot of what's happening today.

所以，确实存在一个瓶颈，阻止我们一直训练同一个模型，而不是从旧模型里提取所有数据，然后从头训练一个新模型。

And so it's very much like there's a bottleneck that stops us from just keeping training the same model forever versus just getting all the data from the old model and training a new model from scratch.

这正是查理说的，是塑性和灾难性遗忘的某种结合。

And this is exactly as Charlie was saying, some combination like plasticity and catastrophe forgetting.

而且，如果你只是天真地在非平稳数据上训练，因为你在不断添加新数据，这基本上会打乱数据分布。

And it like the it, you know, if you just naively train on like non stationary data because you're adding new data as you go, basically this is messing with the data distribution.

所以旧的东西就被遗忘了。

So like the old stuff is just forgotten.

我们确实没有很好的方法来阻止这种情况发生。

And we don't really have good methods to like stop that from happening.

而且也许在极限情况下，你就会被从头训练模型并加入所有这些新信息所限制。

And maybe at the in the limit you're like just bottleneck by retraining the model from scratch with all this new information.

是的。

Yes.

当然，这非常昂贵。

Which of course is like very expensive.

比如，从头训练模型确实很贵。

Like, training model from scratch is is expensive.

但你反正也会这么做。

But you're gonna do that anyways.

我的意思是，也不一定。

And I mean, not necessarily.

我是说，也许最终如果你有持续学习的能力，你就不需要训练新模型了。

I mean, like, maybe eventually if you have continual learning, you never train a new model.

你只需要一个模型，它不断学习，然后不断扩展。

You just, like, just have a model and it keeps learning and then it keeps expanding.

对吧？

Right?

但是，可能有一些很深的技术原因，让这件事变得非常困难，因为那些……

But but there's there might be like some deep technical reason why that's very difficult because of these

我觉得，这就是问题所在。

like I mean, that's the question.

这就是问题所在。

That's the question.

对。

Yeah.

我觉得我们已经推迟了很多从头训练的工作。

I think we have pushed back like how much from scratch we need to do.

现在确实有可能在预训练模型的基础上，做很好的中期训练，有点像持续学习，再加上从训练后期不同检查点来的强化学习，这看起来更像是一种调优学习，肯定不是那种拿最新的模型，做几个很小的更新，然后迭代下去，永远不会丢失……

Like, it is definitely possible now to take like the pretrained base and like do very good mid training on top of that, like, kinda continuously plus some RL from, like, different checkpoints that are later on in the training and, like, that's looking more like a tuning learning, certainly not the case of, like, you know, take the most recent model, apply a couple of very small updates, and, like, iteratively, like, never lose

……它。

it.

但是，这不是让我有点困惑吗？因为这难道不就是训练或后训练过程中实际发生的事情吗？

So but isn't this like I'm a bit confused because isn't this literally what happens during training or during post training or something?

你只是有一个已经经过大量训练的模型，然后你提炼出一些经过进一步强化学习的分支。

You just have like you have a model that's already gone through so much training and then you like distill some fork that's been further RL ed or something.

这不就是实际发生的事情吗？

Isn't isn't that literally what happens?

而且为什么

And like why

我觉得这仍然是在足够大的规模上进行的，所以你会洗掉很多噪音。

is that It's still at a large enough scale I think that you're washing out a lot of like the noise.

而且你不是只专注于一个分布，正如Beren所说，如果你只专注于一个任务，那现在是非常……对吧？

And you're not just focused on one distribution which as Beren said is like, that is now a very if you're just focusing on one task, right?

那

That's

不是，我的意思是，在最终的情况下，你会做，我不知道，有数十亿的部署实例。

not I mean in the eventual regime you'd be doing, I don't know, there's billions of deployed instances.

你同时从所有实例中学习，所以希望有些噪声能被冲淡。

You're learning from all of them at once and so hopefully there's some washing out

噪声之类的东西能被冲淡。

of noise and stuff from that.

对吧？

Right?

也许这就是规模效应。

Maybe that's scale.

对。

Yeah.

对。

Yeah.

我是说，我觉得，确实像查理说的那样，你可以长时间进行持续的中期训练，而且有了中期训练数据，你可以回滚到某个检查点。

I mean, I I think, like, definitely as as Charlie was saying, like, can do continual mid training for, a long time and you can, like, roll back to a checkpoint given you mid training data.

但同时，你也不能无限期地这样做。

But at the same time, like, you can't do this like indefinitely.

比如说，如果你一直用同一个基础模型继续做中期训练，总有一天会达到一个瓶颈。

Like, if you just keep continuing mid training the same base forever, it just like get it does it's sort of asymptote at some point.

也就是说，你不能在那个基础上无限地学习新东西。

Like, you can't just learn new stuff in that base.

这就是为什么人们最终会训练新的基础模型。

And this is why people end up training new bases.

否则，你只会一直用同一个基础模型做中期训练，永远不变。

Like, otherwise, you would just keep mid training the same base forever.

每次我录完采访，都会立刻把所有的想法倒进Slack里。

Whenever I finish recording an interview, I immediately brain dump all my thoughts into Slack.

比如，什么最有趣，什么应该剪掉。

Things like, what was the most interesting and what should get cut?

这样我的编辑们就能掌握所有需要的背景信息，开始剪辑这一集。

This ensures that my editors have all the context they need to start editing the episode.

但这些脑力倾泻并没有明确的时间戳，而且我的未剪辑录音有好几个小时长。

But it's not like these brain dumps have any clear timestamps, and my unedited recordings are many hours long.

光是找到我提到的那些具体片段，就可能花掉编辑大量时间。

It can take a ton of editor time to even find the exact moments that I was referencing.

所以我们决定试着在我们的聊天里加一个Grok Bot制作人。

So we decided to try adding a Grok Bot producer to our chat.

现在，每当我的编辑上传一集粗剪，Grok Bot就会在自己的电脑上打开转录稿并开始工作，通常在我看到消息之前就已经开始了。

Now whenever one of my editors puts a rough cut of the episode, Grok Bot opens a transcript on its own computer and starts working, usually before I've even seen the message.

它会读取我发到Slack里的笔记，并在转录稿中高亮出相关的片段。

It takes the notes that I dropped into Slack and it highlights the relevant snippets in the transcript.

它还会使用我整理的一个包含我所有偏好的大案例文件，这样它就能提出可能的剪辑建议。

It also uses a big case file that I've compiled with all my preferences so it can suggest potential edits.

当它完成后，它会给我发送它认为最值得保留的片段候选，这样我就可以在手机上审阅所有内容。

And when it's done, it sends me its top clip candidates so that I can review everything from my phone.

这个效果真的很好。

This has worked really well.

能像给编辑发短信一样发送非正式消息，然后转录稿立刻反映出我的偏好，这真的帮了大忙。

Being able to send informal messages like I'm texting my editor and then having the transcript immediately reflect my preferences has just been so helpful.

你自己去 x.ai 斜杠 bot 试试 Grok Bot 吧。

Try Grok Bot yourself at x dot a I slash bot.

好的。

Okay.

现在咱们来聊聊数据吧。

Let's talk a bit about data now.

我一直对一个问题很感兴趣，就是 AI 的进步有多少其实只是靠数据的进步来解释的。

So I'm generally interested in this question of how much of AI progress is just explained by data progress.

这不一定意味着自动化会很困难，但那是另一个问题。

Doesn't mean it will be necessarily hard to automate, but that's a separate question.

那么，是否存在某种数据分布，如果你用当前架构去训练，就能得到一个超级智能，它在每个领域都完全碾压人类专家？

So is there some data distribution which if you trained current architectures on would result in a super intelligence that totally dominates human experts across every single field.

我们说的是预训练数据加上后训练数据，还有环境之类的吗？

Are we talking about, like, pre training plus, post training data, like, environments as well?

而且，我觉得这种东西的存在是显而易见的。

And, like, I think the existence of this is obvious.

问题就在于我们能不能创造出合适的环境。

It's just, like, whether we can create the right environments.

对。

Yeah.

而且在最简单的情况下，我们可以直接训练它输出那个Python文件，这个文件本身就能训练出真正的超级智能。

And in the trivial case, we could just train it to output the Python file which like trains the actual superintelligence.

就像是怎么去记忆那种方式。

Like just how to memorize in the way it's

是的，可能有一系列强化学习环境可以构建出来，这样你就能得到一个至少和人类研究员一样优秀的AI研究员。

Like, yes, there's probably like a ladder of RL environments that is possible to construct such that you would get a AI researcher which is at least as good as a human researcher.

但是每往上爬一级，付出的努力会呈指数级增长。

But the effort to climb each successive rung grows like kind of exponentially.

这就是你需要权衡的两件事，决定我们多快能达到那个最终阶段，也就是它变得更好的时候。

And that's going to be the two things that you have to trade off against as to whether how fast we're going to hit that final rung where it's better.

我觉得这一点很清楚。

I think that's fairly clear.

而且我觉得，我们在创建自己的环境方面还处于相对早期的阶段。

And I think there's like, we're still relatively early in our own environment creation.

我们利用了很多不对称性来创造好的环境。

There's a lot of asymmetries that we exploit in order to create good environments.

其中一个我们之前讨论过的不对称性，就是有些环境更容易来回切换。

So one of those asymmetries which we've talked about before is there's environments where it's easier to go backwards and forwards.

我的意思是，定义这种复杂的数据生成过程很容易，而且这是你从模型中隐藏的那种潜在变量。

What I mean by that is it's very easy to define this complex data generating process and this is this kind of latent variable you keep hidden from the model.

你可以生成任意复杂的环境，而模型必须花费大量不可简化的代币和不可简化的工作量来弄清楚那个数据生成过程是什么。

You can generate arbitrarily complex environments and the model has to do a lot of irreducible token spend and irreducible work to figure out what that data generating process was.

在不对称性方面，比如你可以从现实世界注入信息，所以像Anthropic通过数万个人类和LML结合发现了一个漏洞，然后把它变成一个非常非常简洁的环境，单个LM理论上可以在几百万个代币内找到它。

There's asymmetries in terms of like you can inject information from the real world so like Anthropic finds a bug through like tens of thousands of human and LML was combined and like turn that into a very very neat environment which in a single LM could theoretically find within like a few million tokens.

所以有所有这些不对称性，我们在挑选利用，而且我们依赖这种任务视野的泛化。

So like there's all these asymmetries which we're cherry picking and like we're counting on like kind of this task horizon generalization.

但我觉得，是的，这最终会达到收益递减的点。

But I think, yeah, again, this is just gonna hit diminishing returns at some point.

比如说，到了某个阶段，创建这些环境本身就会遇到收益递减的问题，因为想出这些环境并不容易，你不能总是依赖那种从后往前比从前往后更容易的过程。

Like at some point there's diminishing returns and how hard it is to create those environments in the first place, like coming up with them because you can't necessarily just like have these really like these process where it's easier to go backwards than forwards.

实际上，你得坐下来，像人类一样，构建一个时间跨度足够长的东西，这本身就是一个非常复杂的任务。

Like you actually have to sit down and construct something that looks like with humans, like a long enough time horizon, like it's gonna be a really complex task to create.

而且，对于智能体来说，执行这些任务也会遇到计算和时间上的瓶颈。

And then there's also gonna be like the compute and time bottlenecks for the agent to actually do those tasks.

对。

Yep.

所以，我觉得你会看到这条曲线开始趋于平缓。

So like, I think you're just gonna start seeing this like curve to flatten out.

我看到过一些内容，说有人对Talkie模型进行了微调，这个模型只训练到1930年的数据，然后用在现代编码智能体的数据上，结果它在SweepBench上的表现比Claude 3 Opus还要好。

I saw something about how someone fine tuned the talkie model, which is only trained on data up to 1930 on this, like, modern coding agent data, and and it did better than Claude three Opus on sweepench.

所以，这个对代码一无所知的模型，经过中等规模数据的微调，就能在编码智能体任务上表现得比那个大得多的预训练模型更好，这真的很惊人。这也说明，一旦你有了正确的专家行为示例，把它复制到一个相对较弱的模型上，其实出奇地容易。

So so this model that has, like, no knowledge of code whatsoever can be fine tuned on a moderate amount of data and, like, behave better as as a coding agent than this much larger pre trained model is pretty crazy, and it kinda shows you that, like once you have an example of like the right expert behavior, it's actually surprisingly easy to like copy that into a like relatively weak model.

是的。

Yeah.

但有个反例，最近有篇论文，他们只训练到五年级的数学水平。

But a counter example to like that kind of is there was a paper recently where they trained it up to like fifth grade maths.

嗯哼。

Uh-huh.

还有小学英语之类的，所以算是个不错的语言模型。

And also like primary school like English and stuff, so it was like a decent language model.

他们试着用强化学习让它做高中和大学数学，但差距太大了，根本爬不上去。

And they tried to RL it to do like late high school and college maths and the gap was just too large, like they couldn't get it to climb at all.

但如果一级一级来，比如七年级数学、八年级数学这样，那肯定能爬到十二年级。

But if you did successive rungs of like, you you're seven maths and then you're eight maths and so on, you could obviously climb to year 12.

所以还是那个问题：梯子上的每一级之间距离有多大，以及造这些梯子有多难？

So like, again, it's just like what is the distance between the rungs on those letters and how hard is it to create?

对。

Yeah.

这又回到了强化学习信号的问题上。

And this just comes back to like the RL signal problem.

现在的强化学习在探索方面其实不太擅长。

Like, RL is not very good at like exploring right now.

所以如果模型在128次尝试里都找不到方向，那它基本上很难获得进步的信号。

And so if you if the model can't like get in like, you know, a 128 rollouts, it's very unlikely to get signal to like progress.

这就是为什么在强化学习里我们需要课程学习，而在预训练阶段就不需要，因为预训练根本不存在这个问题。

And this is why like in RL we need like curricula, whereas in pretraining we don't because that's not a problem for pretraining at all.

对。

Yeah.

而且预训练的数据和后续训练的数据也是不一样的。

And again pretraining data is different to post training data.

我想随着时间推移，人类的参与会越来越少，但这并不能改变一个事实，就是你仍然受限于能从现实世界中提取多少信号。

And I imagine as we continue on, humans will be involved less and less but that doesn't change the fact that you're bottlenecked on how much signal you can extract from the real world.

所以世界上确实有很多信号，这是事实。

So like there's a lot of signal in the world and that's true.

有人在处理电子表格任务，有人在处理法律任务，诸如此类，但以目前模型的能力边界来看，世界上到底有多少信息真正有助于提升模型的能力呢？

There's people doing spreadsheet tasks, there's people doing legal tasks and all this sort of stuff but the capability frontier of where the models are at now, like how many bits in the world are actually really relevant to improving the model's capabilities?

有多少新的数学问题被解决，而这些问题是当前模型还够不着或者理解不了的？

How many new maths problems are being solved that are just beyond the reach or grasp of the current models?

有多少新的编程问题被创造或解决，而这些问题超出了当前模型的能力范围？

How many new coding problems are being created or solved that are beyond the reach of the current models?

我觉得这就是为什么会出现收益递减，因为即使是整个世界，也没有给你提供那些能帮你跨入下一个能力层次的有用信息。

And I think that's why the diminishing returns kicks in because even the world as a whole is not giving you the bits that are useful for tipping you into the next basin of capability.

对。

Mhmm.

是的。

Yep.

我完全同意这个观点。

I totally agree with this.

这真的是一个信号来源的问题。

It's really a question of where the signal is coming from.

在预训练中，信号其实已经存在于基础数据里了。

And so the signal doesn't in in pretraining, the signal is, like, already in common core.

对吧？

Right?

比如说，在预训练阶段，你关心的那些任务，问题不只是获取信号那么简单。

Like, for the tasks that you care about in pretraining, the problem is it's not just, like, getting signal at all.

而是要过滤掉所有存在的噪音。

It's, like, filtering out all the noise that exists.

对。

Yeah.

而且这个过程其实挺容易自动化的。

And that's quite an automatable process.

但是，随着模型越来越好，当我们进入中期训练和后训练阶段，信号在我们已有的原始数据里根本不存在了。

But, like, as the models get better, as we enter enter mid training and post training, the signal just doesn't exist anywhere in the original data we have.

无论怎么过滤，也找不到——千禧年大奖难题的隐藏证明可不在共同核心标准里。

No amount of filtering will get this there's no hidden proof of the Millennium Prize problem sitting in Common Core.

我们不可能靠过滤就把它找出来。

We can just filter until we see it.

对吧？

Right?

所以到了那个阶段，你就得从别的地方获取数据了，要么直接让人类写出他们的推理过程，要么创建环境，让人类决定该创建什么样的环境、这些环境的目标是什么，或者利用部署中已有的真实人类数据进行某种训练。

And so at that point, you have to get bits some other way, either from humans directly asking them to write out their reasoning or by creating environments where humans decide what environment should be created, what the objectives of these environments are, or some kind of training on like the human data that exists in deployment.

就是说，你总得从某个地方拿到这些数据。

Like, you have to get the bits from somewhere.

对。

Yeah.

对。

Yeah.

有个问题是，预训练阶段的进步有多少是由数据驱动的。

There's a question of how much of the progress in pre shredding is being driven by data.

是的。

Yep.

我和普林斯顿的学生Jerry Hahn一起做了个研究，基本上我们把2019年至今的所有训练方法和2019年至今的所有数据集都两两组合进行了训练。

I I I did this investigation with Jerry Hahn, who's a student at Princeton, where we basically trained all the recipes from 2019 till now pairwise with all the datasets from 2019 till now.

所以你的意思是，用最新的数据集比如Ultra Fine Web来训练GPT-2，然后用最新的训练方法或者开源的训练方法在Pile或者一些旧数据集上训练，然后做整个网格搜索，看看能达到什么样的能力水平，在这个网格中，计算量能减少多少？

So you're saying like GPT two on the newest dataset like Ultra Fine Web and you train Delphi which is the newest training recipe or the open source training recipe on like the pile or some old dataset and you do like the whole grid and you see the getting to some level of capabilities, how much less compute does it take across this grid?

然后你发现，数据似乎能解释九倍的计算效率提升，而架构改进在很小的规模上只解释了三倍的计算效率提升。

And you see that the data seems to explain like nine x of a compute efficiency gain but the architecture improvements explains like a three x compute efficiency gain at a very small scale.

所以如果在大规模上也成立，大部分预训练计算效率的提升来自更好的数据，那这种情况能持续多久呢？

And so to the extent that that is true at large scale that most of the pre training computer efficiency gains are coming from better data, how much can that continue?

比如，能不能一直过滤数据，不断生成更多合成数据，直到……对。

Like, can you keep just filtering data more and more and building more and more synthetic data until yeah.

你觉得这种预训练的进展还能持续多久？

Do you have a sense of how much this kind of pre training progress can continue?

我觉得我的直觉是，嗯，低垂的果实已经差不多摘完了，感觉我们拿到了互联网这块大蛋糕，但互联网本身不一定在按同样的速度增长。

I think my my prior is that like, again, the low hanging fruit is like somewhat exhausted with like feels like we got the Internet as this big block and like, there's it's not like the Internet is necessarily like growing at the same rate.

互联网上有用的东西在按同样的速度增长。

All the useful stuff on the internet is growing at the same rate.

我们可能还能再降0.1%的损失，但肯定没有已经发生的那么多了。

We've probably got a bunch of 0.1% loss drops to go, definitely not as many as have currently occurred.

不过你们发现这两方面加起来总共提升了27倍，这真的很有意思。

But that's also really interesting that you find this cumulative 27 times improvement across both.

我记得好像是Epoch还是谁估算过，自2019年以来每年提升三倍，那算下来就是三的七次方，也就是超过两千倍的提升。

I think like it was Epoch or someone who estimated like three times a year since 2019 which would imply something like three to the seven, like over 2,000 times improvement.

那这中间差的一百倍左右是从哪儿来的呢？

So where's that missing a 100 times or whatever coming from?

这大概能给你一个很好的信号，说明有多少

That probably gives you a good signal of how much

是来自后训练。

of this is post training.

我觉得解释只能是，很多计算效率的提升是跟规模相关的，而我们是在极小的规模下研究的。

I think the explanation has to be that a lot of the compute efficiency gains are scale dependent and we're studying at extremely small scale.

这就引出一个问题：数据带来的计算效率提升和算法带来的计算效率提升，哪个更依赖规模？

And that raises a question of do the data compute efficiency gains or the algorithmic compute efficiency gains have more scale dependence?

我不知道你们对此有没有什么先验判断。

I don't know if you guys are prior on that.

我们就是没有足够的算力去研究那个问题。

We just didn't have enough compute to investigate that question.

我是说，就简单来说，对吧？

I mean, like, just naively, right?

理论上，架构的规模依赖性其实挺清楚的。

Like, theoretically, the scale dependence of the architecture is, like, fairly well known.

对。

Yeah.

而且你可以给它拟合一条直线。

And, like, you can fit a straight line to it.

但要是把预训练、后训练和中训练的数据结合起来，我完全不知道该怎么处理。

Whereas, like, I would have no idea how to do that for, like, combining pretraining plus post training data and mid training data.

有意思的是。

Funny enough.

我觉得数据其实随着规模变大更重要。

I feel like data is actually more important with scale.

我觉得架构有点像一次性的事情。我觉得说只是提高百分之几的效率其实有点误导，因为架构的作用是让你进入一个全新的阶段，这是旧架构无法达到的。

I feel like architecture is a kind of like a one time I feel like combining saying just like an x percent efficiency again is kind of misleading because what an architecture does is let you reach a qualitatively new regime which you couldn't reach with the old architecture.

然后在这个阶段里，显然数据是决定性的主要因素。

And then within that regime, obviously, the data is like the primary thing determining it.

但是，你知道，如果我们说没有像GQA这样的东西，我们整天都在做全注意力。

But like, you know, if we say it didn't have like even like GQA, we're doing like full attention all day.

我们就没法做到百万级别的上下文，我们本来期望能做到百万上下文。

We wouldn't be able to do like a million we like we'd expect to do a million context.

然后正因为如此，我们永远无法使用那些实际上在百万上下文下的数据。

And then like because of that, we couldn't we could never use the data which is like actually at a million context.

所以我们就无法获得这些能力，即使你只是简单地看，在两千上下文下这个架构并没有解锁什么，那么数据看起来会比实际上重要得多。

And so we couldn't get these capabilities even though, like, if you just do a naive, like, how much does this do at, like, two k context where the architecture isn't unlocking anything than, the data, you know, will look much more important than in some sense it is.

对吧？

Right?

我不太确定这些东西真的只是这样乘数式的增益。

It's unclear to me that these things are, like, really just, like, multiplicative gains in this way.

我明白了。

I see.

抱歉。

Sorry.

但这对数据的规模依赖性意味着什么呢？

But then but what does this take away for the scale dependence of data?

所以，关于规模依赖性，我觉得我们现在很多中期训练和后期训练的数据，实际上随着规模增大效果会更好。

So, I mean, on scale dependence, I think, like, a lot of the, like, mid training and post training data we have now is, like, actually gets better with scale.

因为其中很多数据，比如超长上下文环境相关的数据，真的需要大模型才能利用起来。

Because, like a lot of it, like the very long context horizon environment stuff, really requires like big models to be able to like make use of them.

对。

Yeah.

而且，如果你试图用一亿参数的模型去训练三条基准轨迹，它不会有任何进展。

And like this is not know, if you try and train like your 100,000,000 parameter model on like three bench traces, it's not gonna get anywhere.

它不会像训练一个规模合理的模型那样，展现出同样的改进效果。

Like it's not gonna show you the same kind of improvement that you would get if you train an actual sensible sized model on it.

对。

Yeah.

而且现在也很难，因为架构变化太多了。

And it's hard as well now because so many of the architecture changes.

你看Kimi，比如DeepSeq。

You look at Kimi for instance, DeepSeq.

他们做这些架构修改，不只是为了降低预训练损失，还考虑到模型在现实世界中会怎么被使用。

They're doing these architectural modifications with not just dropping the pre training loss in mind, but for instance how the models are gonna be used in the real world.

推理效率，在DeepSeq模型里做某种压缩注意力，不一定是为了预训练本身，而是说，我们考虑的是模型将来怎么用。

The inference efficiency, having some form of compressed attention in the deep sea models is not necessarily geared around, this is fundamentally a pre It's training just like, okay, we're considering how the models are gonna be used.

对，对，对。

Right, right, right.

嗯，我好奇一个问题，想了解未来，就是随着我们进入更依赖强化学习的阶段，参数规模会怎么发展。

Well, one question I'm curious about to understand the future is how parameter scaling will go as we're getting into more of a RL heavy regime.

比如，我也不太清楚。

Like, I don't know.

我不太清楚历史上增长的速度有多快。

I don't know how fast historically.

对。

Yeah.

你可以看看开源架构，看看参数规模增长得有多快，可能前沿开源模型每年大约翻一倍。

You can look at sort of open source architectures and see how fast parameters have been scaling and maybe it's like roughly 2x every year for frontier open source models.

而且即使前沿闭源模型也有大约1000亿或2000亿的活跃参数。

And to the extent that like even frontier closed source models have like 100B or 200B active parameters.

你觉得这个速度会保持每年翻倍吗，还是说在强化学习模式下，你还需要节省推理计算资源，而且可能存在一个阈值效应，当容量足够大时，无限增加参数就不那么重要了？

Do you think that like keeps 2x in year over year or another RL regime where you also want to conserve compute on rollouts that and also maybe there is like a threshold effect where you have enough capacity and at that point does increasing parameters arbitrarily doesn't matter as much.

你们觉得到2030年，前沿模型的活跃参数会达到多少？

Do you guys have a sense of in 2030 how many active parameters will a frontier model have?

是的。

Yeah.

我觉得未来几年，我们会专注于做越来越长的强化学习推理，因为推理效率非常重要。

I think for the next few years we're going to be like because we're so focused on doing longer and longer horizon rollouts for RL where like inference efficiency matters a lot.

对。

Yeah.

感觉模型本身的能力还没到瓶颈，真正的瓶颈还是环境。

It feels like the the models aren't necessarily saturated on their ability to that where the bottleneck is still the environments.

所以我们可能会看到一点平台期。

And so we might see like a little bit of plateau.

我有个感觉，就是像Mythos和GPT这些模型，其实比大家说的那种一万亿参数要小得多。

Like I I I have a feeling that, you know, like mythos and and the GPT models are much smaller than like, you know, the 10,000,000,000,000 parameter range that people are talking about.

就算只是粗略地和开源模型比一比，也能得出这个结论。

Even just naively comparing them to open source models you can probably back out that conclusion.

所以，对，未来几年我觉得参数数量不会大幅增长，但这里面确实有很多需要权衡的地方。

So yeah probably for the next few years I wouldn't imagine a huge growth in the number of parameters but again there's so many different things to trade off here.

模型的大小取决于你有多少预训练数据，以及你要训练的RL环境的难度。你理想情况下想达到一个最优的点，就是能在最难的环境上取得不错的成绩，但如果模型再大，可能就不划算了，因为推理成本会高很多。所以这里有很多考量。

You decide the size of your model based on how much pre training data you have and then the difficulty of the RL environments that you've got to train on and you ideally want to get to the optimal point where you can get like a decent pass at one or something on like the hardest environments you have and like it wouldn't make sense to like make a bigger model pass there because then you're just paying like much more inference swaps So when you need there's lot of emphasis.

这取决于Macore和内部团队能多快提升RL环境的复杂度。

It like depends on how quickly you know like Macore and then in house these guys can scale up the complexity of the RL environments they're training on.

我觉得模型会继续变大，因为大家在扩大算力，GPU也越来越大。

I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger.

但我觉得它们具体会变大多少，某种程度上取决于那些不太明显的缩放规律。

But I would say exactly how much they get bigger depends a bit on the scaling laws in non obvious ways.

有一点是，我认为数据效率会比计算效率更能推动人们采用的具体架构，因为我们现在已经接近高质量预训练数据不足的阶段，所以这可能会影响你想让模型变得多稀疏。

So one thing is that I think, like, data efficiency is gonna be a bigger driver than compute efficiency of, like, the exact architectures people use, now that we're getting to the regime where we're sort of running low on, like, high quality pre training data, so that might affect how sparse you want to make the model.

另外，我觉得我们对稀疏性的理解还不够深入。参数和激活参数是两种不同的资源，虽然稀疏性确实有所增加，但还不清楚它是否会继续增加，可能会有一个最佳点。

And then I also think we don't understand sparsity that well, and it's like parameters are a different resource than active parameters, but it's like sparsity has definitely increased a bit, but it's not clear that it's gonna keep increasing with outbound, there might be some kind of sweet spot.

有一种观点认为，稀疏性可能会降低数据效率，因为你可能需要在多个专家中学习同样的内容，不过这一点还有争议。

There's an argument that sparsity should make data efficiency worse because you might have to learn the same thing on multiple experts, though that's debatable.

所以我觉得我们还没有足够好的缩放规律理论，无法真正理解为什么稀疏性有帮助，以及它会有多大帮助，是否会达到某个平台期。

So I don't think we have a good enough theory of scaling laws that we we really understand why sparsity is helping and to what how much it'll help, and if that'll, like, plateau at some point at a certain level of sparsity.

那么你能具体说明一下数据效率的影响吗？听起来你会说应该减少稀疏性，但这对参数规模还有其他影响吗？

And so can you spell out exactly what the implication of data efficiency would be on, so it sounds like you'd say well, there should be less sparsity but what are the other implications on parameter scaling?

我想说的是，你主要关注的缩放规律不是优化计算效率，所以你在架构上做的所有选择都会带来不同的缩放规律。传统上，你会根据计算量来看某种包络线，也就是性能与计算量的关系。对。

I guess just that the scaling law you're necessarily looking for the most, you're not trying to optimize compute efficiency, so you have all your, choices you can make on the architecture, and, each of these gives you a different scaling law, then, like, traditionally you would look at some kind of envelope based on compute, so you would look at performance versus compute Yeah.

然后取最好的模型的包络线。

And take the envelope of the best models.

但如果我们是基于数据来做这个决定，也就是说我们假设可以投入大量计算，而数据是横轴而不是计算，那么我们就会得到一组不同的最优模型，或者说是处于那个前沿的不同模型。

But if we're making that decision based on data, so it's like we're assuming we can spend a lot of compute, and like we're sort of data is on our x axis instead of compute, then we just get a different set of Optima or a different set of models that are on that frontier.

对。

Yeah.

而且我也不认为过去几年我们真的每年都把模型规模翻倍了。

And I I also don't think that we've necessarily doubled the size of the models every year for the last few years.

人们训练万亿参数模型已经至少好几年了。

People have been training 1,000,000,000,000 parameter models for at least a few years.

甚至有一个开源的叫Falcon，但Periodic Labs的Liam，我想是昨天在推特上发帖说，OpenAI早期的一个实验是在训练一个万亿参数的模型，而且非常稀疏。

There was even an open source one called Falcon but Liam from Periodic Labs I think posted yesterday on Twitter about how an early experiment at OpenAI was training a 1,000,000,000,000 parameter model that was very, very sparse.

哦对，那是他们在OpenAI之前做的。

Oh yeah that was what they did before OpenAI.

那就像，哦，在谷歌做Switch Transformer之前。

That was like, Oh before, they Google the switch transformer,

对。

yeah.

所以它在知识方面非常强，但在推理方面很差，因为它太稀疏了。

So it was very, very good at knowledge but terrible at reasoning because it was so sparse.

所以感觉我们一直在1000亿到2万亿参数这个范围内玩了一段时间，肯定不是那种漂亮的线性增长。

And so it feels like we've been playing in this 100,000,000,000 to up to 2,000,000,000,000 parameter range for at least a little bit and it certainly hasn't been this nice linear increase.

对。

Yeah.

我觉得有两件事，就像Charlie说的，推理效率对强化学习的 rollout 非常重要。

I mean, I feel like there's two things, so as as Charlie was saying, like, inference efficiency is super important for RL rollouts.

所以这会大大降低活跃参数的数量。

And so, like, this will really push down active parameters quite a lot.

然后我觉得总参数也很大程度上取决于硬件。

And then I think the total parameters really depends a lot on the hardware as well.

所以你真的需要非常高的内存带宽和显存大小，才能实际运行几万亿参数的大模型。

So like you really need to get like very high memory bandwidth and like VRAM size to like actually be able to serve like multi trillion parameter models.

所以你看，现在大家还在大量使用H100之类的硬件。

And so like, you know, right now, you know, people still still using a lot of like h one hundreds and stuff.

随着大家转向GB系列，我们就能获得更强的扩展能力，真正去服务更大规模的RL输入。

And so as everyone moves to GBs and then we'll get like more actual the ability to scale and actually serve and do large RL inputs at larger scales.

数据这个问题我觉得挺有意思的，因为从直觉上看，更大的模型在数据点上的样本效率要高得多。

The data question I think is interesting because naively larger models are much more sample efficient in the actual data points.

所以即使模型还没饱和，还是越大越好，因为更大的模型泛化能力更强，在同样的数据量下能达到更低的损失。

And so even if you're not saturating the model, it's still best to go bigger because the models larger models generalize better and get to a better loss for the same amount of data.

所以现在我觉得我们其实数据很多，瓶颈不在数据，而在算力。

And so right now, I think we kind of have a lot of data, that's not the constraint rather than computing.

所以我们用的是小模型，推理效率很高。

So we're having small models, which are like very inference efficient.

但如果算力不再是瓶颈，可能又会回到更大的模型，虽然它们可能还没完全饱和，但因为规模大，泛化能力更强。

But if computers are no longer the bottleneck, it might come back to larger models which are like under sort of under saturated, but like they have this generalization ability because they're much larger.

如果你只看John Schulman的缩放定律，然后最大化参数数量，对。

If if you just look at like the basic John Schulman scaling law and you just maximize out parameters Yeah.

实际上，要达到同样的损失，你所需的数据量减少得非常少。

It actually decreases the amount of data you need to get to the same loss very little.

是的。

Yes.

如果参数无限增加，我认为所需的数据量增长不到10倍。

If you go to infinity on parameters, the amount of data you need, I think, goes on less than 10 x Mhmm.

只是因为，那种，幂律的性质。

Just because the nature of, like, the the power of

我们现在处于Chinchilla定律中数据过多的那一边。

We're we're now on the way too much data side of the Chinchilla laws.

对吧？

Right?

所以现在我们比Chinchilla更过度训练，因此当我们数据快用完时，很容易回到Chinchilla最优点的位置。

So right now we overtrain more of our into Chinchilla and so we could easily get back to a point to which as we're running out of data, move back to like the Chinchilla optimal point.

我们甚至有点在过度训练，也就是模型训练不足的那一边。

We're even like a bit on the overtraining, like, you know, undertraining model side.

但确实，即使有这些新芯片上线什么的，未来几年我们还是会严重受限于算力，所以那不一定能……

But surely, like, even with these new chips that come online and stuff, like, we're just gonna be so compute bottlenecked for the next few years that that won't necessarily

……对，这很有可能。

be I this could well yeah.

这真的取决于训练和推理算力的比例。

This depends on, like, the ratio you have, like, training an inference compute, really.

如果你主要受限于数据而不是算力，那就应该把模型做大。

It's like if you're super bottlenecked on data and not on compute, you should go bigger.

如果你主要受限于算力，那就应该把模型做小。

If you're super bottlenecked on compute, you should always go smaller.

然后，对。

And then, like, yeah.

你还可以用算力来生成合成数据，所以这真的很难预测。

You can also use compute to generate synthetic data, so it's like one of these very hard things to predict.

对。

Yeah.