持续学习时代的8大预测

我之前在别的地方解释过，为什么我认为真正的持续学习是必要的。

So I've explained elsewhere why I think actual Continual Learning is needed.

我认为，如果AI只能从一次会话到另一次会话在文件上做标记，它们不可能像人类一样胜任整个工作。

I don't think you can have AIs that perform whole jobs as competently as humans if they are forced to just write marked on files from session to session.

举个例子来说明，想象一下如果学生是这样学萨克斯的。

Just to give an illustrative example, imagine if this is the way that students learn to play the saxophone.

有一个学生，他以前从未吹过萨克斯。

So you have one student, he's never played the saxophone before.

他走进音乐厅，试着吹。

He goes into the music hall, he tries to play it.

当然，这是他第一次，所以他失败了。

Of course, this is his first time, so he fails.

他写下了一堆关于哪里出错的笔记。

And he writes down a bunch of notes about what went wrong.

还有一个学生在音乐厅外面等着。

And there's a next student who's waiting outside the music hall.

他进来，读了所有这些笔记。

He comes in, he reads all these notes.

他也从没吹过。

He's also never played.

所以当然他也搞砸了，然后继续往这些笔记上添加内容。

So of course he messes up and he continues to add on to these notes.

然后音乐厅外面有无数学生，他们不断给下一个人写笔记。

And you have an infinity of students who are outside the music hall who keep writing notes to the next person.

我觉得他们之间无论写什么文字，都不可能让后面的学生第一次就完美吹奏萨克斯。

I don't think there's any sequence of texts they could write to each other that would allow the subsequent student to just nail the saxophone from the first try.

在某个时候，你确实得把相关经验积累到自己的大脑里。

At some point, you actually have to accumulate the relevant experience into your brain.

我认为对于很多我们希望AI在不同工作场所中积累的技能，情况也会是这样。

I think the same thing will be true for a lot of skills that we want AIs to actually accumulate from all the different workplaces in which they're deployed.

好，那么一旦我们真正实现了持续学习，会有什么变化呢？

Okay, so what changes once we have actual Continual Learning?

第一，我觉得很多关于AI监管的提案，都是假设你先训练一个模型，然后再部署它。

One, I think that a lot of proposals that have been put forward about regulating AI assume that you train a model and then you deploy it.

所以，在模型部署之前，你可以对它做一堆检查，确保它不会帮助网络攻击或者做出什么疯狂的事情。

And therefore, you run a bunch of checks on the model before it is deployed, we can make sure that it's not going to aid in cyber attacks or do something crazy.

我觉得这个假设在未来不一定成立。

I don't think this assumption necessarily makes sense in the future.

这也是我为什么现在对锁定某种安全监管制度感到担忧的原因之一，因为我们连一年后要面对什么样的技术都不知道，更别说五年或十年后了。

And this is one of the many reasons I'm actually kind of worried about locking in some kind of safety regulatory regime right now because we don't know what kind of technology we're gonna be dealing with even within a year, let alone within five years or ten years.

如果模型每天都在根据当天数百万次的工作会话不断改进呢？

What if the model is improving every single day based on the millions of sessions of work it does in that day?

如果真是这样，我们可能会锁定一种过时甚至适得其反的方法来应对AI带来的威胁。

If that happens, we could potentially be locking in an archaic and potentially counterproductive approach to dealing with the threats from AI.

如果政府想对模型提供商进行某种安全评估，我觉得按月或按季度进行风险检查更有意义，而不是试图找出一个训练完成之后、部署开始之前的特殊时刻，因为将来这个时刻不会是一个有意义的独立阶段。

To the extent the government wants some way to do some kind of safety evaluation on model providers, I think it would make more sense to do monthly or quarterly risk inspections rather than trying to single out some special moment that occurs after training is done but before deployment begins because that will not be a meaningfully distinct category in the future.

第二，实验室做技术对齐的方式可能完全需要改变。

Two, how the labs do technical alignment would probably totally need to change.

目前，很多研究都集中在如何确保一组固定的权重在部署时表现良好。

Right now, a lot of research is focused on the question of how we make sure that a frozen set of weights behaves well during deployment.

但据我所知，关于如何确保即使权重不断更新，AI系统也不会被越狱或变成欺骗性或邪恶的人格，这方面的研究还不多。

But I'm not aware of much research on the question of how we make it so that even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona.

而且，如果AI在用户之间整合学习成果，我们如何防止用户向基础模型注入后门或某种恶意倾向呢？

And if AIs are consolidating learnings between users as well, how do we prevent users from injecting backdoors or some kind of malicious inclination into the base model?

从某种意义上说，这其实就是人类对齐问题，对吧？

In some sense, is actually kind of what the human alignment problem is, right?

人类是以自我导向的方式进步的。

Humans improve in a self directed way.

如果你有孩子，我没有孩子，但我想情况就是这样。

If you have kids, I don't have kids, but I imagine this is what happens.

如果你有孩子，他们出去学习新东西。

If you have kids, they go out, they learn new things.

有时他们会变得疯狂。

Sometimes they go crazy.

他们会被疯狂的意识形态一击即中。

They get one shotted by crazy ideologies.

他们吃错药了。

They take the wrong drug.

他们会变得非常奇怪，但你希望你已经给了他们足够的常识和基本价值观，让他们能够自我提升，而不会最终产生一些奇怪的信仰或反人类的想法。

They become super weird, But you hope that you've given them enough common sense and basic values that they improve as people in a self directed way without ending up with some super weird beliefs or some misanthropic ideas.

第三，AI思维的多样性将会增加。

Three, the diversity of AI minds will increase.

目前，知名的AI思维不到五个，我指的是那些同时服务数百万、数亿甚至数十亿用户的基础模型。

Right now there are less than five prominent AI minds by which I mean the base models which are served to millions or hundreds of millions or billions of users at once.

而且它们彼此之间都很相似，因为它们也都是在大致相同的数据上训练的。

And they're all quite similar to each other by the way because they've also been all trained on roughly the same data.

但如果AI从经验中学习，而这些经验不仅在不同AI公司之间不同，而且在同一AI模型的不同实例之间也不同，我们实际上可能会看到这个世界最终产生很多多样性。

But if AIs are learning from experience and that experience is different between not only different AI companies but also between different instances of the same AI model, we could actually see a lot of diversity come out the other end in this world.

我认为这将是一个净正面的结果。

And this would be I think a net good outcome.

我觉得未来需要担心的一个问题是，可能会出现一个非常无聊的单一模型。

I think one of the things to worry about in the future is just having this monolithic singleton that's quite boring.

如果世界能实现持续学习，那应该会比现在这种各种模型趋同的情况有趣得多。

A world where we have Continual Learning would hopefully be more interesting than the mode collapse of different models we see in the world right now.

第四点，当部署成为训练的一部分时，在AI竞赛中领先的回报会加速增长。

Four, when deployment becomes part of training, the returns to being ahead in the AI race accelerate.

因为如果你拥有最好的模型，并且更多人使用你的AI来处理更复杂、更有用的工作，他们就会给你大量反馈，这些反馈可以整合到会话窗口之外，那么你的模型就会变得更聪明。

Because if you have the best model and more people are using your AI for more complicated and useful work, And as a result, they're giving you lots of feedback that it can integrate beyond the session window, then your model will become even smarter.

第五点，如果模型主要从部署中学习，那么实验室会感到很大的压力，需要更早地部署他们最聪明的模型。

Five, if the model learns mainly from deployment, then labs will feel a lot of pressure to deploy their smartest models earlier.

据报道，Throtpec从二月份开始就在内部使用Methos，但直到六月份才向公众发布。

And Throtpec has reportedly been using Methos internally since February but it only shipped the model to the public in June.

在真正实现持续学习的机制下，这种情况根本不可能发生。

In the regime with actual Continual Learning, this kind of thing would just not be possible.

你不可能在内部和外部部署之间保持四个月的差距还能保持竞争力，因为竞争对手在发布日推出一个较差的模型，但基于真实世界的经验，他们会拥有一个更聪明的模型。

You could not keep a four month gap between internal and external deployment and still be competitive because a competitor who ships the worst model on release date will have a smarter model based on actual real world experience.

第六，持续学习将为领先的AI实验室创造一种他们目前缺乏的清晰模式。

Six, Continual Learning will create a clear mode for the leading AI labs that they currently lack.

很多人一直在问，AI实验室到底怎么赚钱？

Many people have been asking, how will the AI labs actually make money?

我也一直在问这个问题。

I have been asking this.

当我请Dario上播客时，我问了他这个问题，他拿云服务商做了类比，他说，你看，云服务商提供很多无差别的服务，但他们的利润率依然很高。

When I had Dario on the podcast, I asked him this question and he made the analogy to cloud providers and he made the point, look, the cloud providers are offering many undifferentiated services but they're earning high profit margins nonetheless.

如果你看看亚马逊或谷歌的季度财报，你会发现他们过得很好。

You will have noticed this if you look at Amazon or Google's quarterly earnings, they're doing just fine.

但云服务利润率之所以这么高，是因为从一个云切换到另一个云非常耗时且昂贵。

But the reason that the cloud margins are so high is that it's really time consuming and expensive to switch from one cloud to another.

目前，没有什么能阻止我创建一个带codecs的软件仓库，然后用cursor继续开发，最后用quad code完成。

Currently, there's nothing that's stopping me from starting a software repository with codecs and then doing more work on it with cursor and then finishing it up with quad code.

但一旦我们有了真正的持续学习，你使用的模型在每次交互中不断进步，那么切换成本就会变得相当高。

But once we have actual Continual Learning and the model you're working with is actually getting better as it interacts with you from session to session, then there are actually pretty significant switching costs.

如果你想换掉正在使用的AI，基本上就等于解雇一个已经积累了你组织几个月背景信息的员工，然后换上一个非常新、毫无经验的新实习生，还得从头开始培训。

If you want to change the AI that you're using, you basically had to fire an employee that has accumulated months of context on your organization and you replace them with a very fresh, very unexperienced new intern that you had to retrain from scratch.

一旦有了这种锁定效应，模型提供商就可以要求相当高的利润率。

And once you have this kind of lock in, model providers can demand pretty hefty margins.

很难刷新它。

It's hard to refresh it.

真正强调的是热情。

Really emphasis on the passion.

第七点。

Seven.

当然，他们会意识到这种动态。

Of course, will be wise to this kind of dynamic.

他们会尽量避免这种锁定。

They will try to avoid this kind of lock in.

但如果面临的选择是：要么被模型提供商锁定，要么失去那个超级有价值的功能——你的模型会随着每次会话为你不断改进，那该怎么办？

But what if the choice is that you either get locked into a model provider or you lose out on the super valuable feature where your model improves for you from session to session?

如果实际使用最终成为模型改进的主要方式，那么AI实验室可能会补贴那些允许模型在其会话上进行训练的企业用户。

If real usage ends up being the main way the models improve, then the AI labs may subsidize users in enterprises which allow the model to train on their sessions.

如果你看看提供给编程产品新用户的那些优惠，就会发现这已经在发生了。

This is already happening if you look at the kinds of deals that are offered to new users of coding products.

这和谷歌为什么免费提供搜索服务非常相似。

This is very similar to why Google gives away search.

反过来，实验室可能会说，任何拒绝让他们在会话上训练的企业，都无法使用最好的模型。

And conversely, the labs may say that any enterprise that refuses to let them train on the sessions can't have access to the very best models.

通过胡萝卜和大棒，实验室可以采取很多措施来让用户允许AI从经验中学习。

With both Keras and Sticks, the labs can do a lot to get users to allow AIs to learn from experience.

当然，我忽略了一个事实，即更新单个用户的权重与将这些不同的权重分支合并回主模型之间是有区别的。

Now, of course, I'm glossing over the fact that there's a difference between updating one user set of weights and pulling all these different weight forks back into the main model.

后者可能在技术上更具挑战性，但假以时日，这个问题也会得到解决。

And the latter may be more technically challenging, but in due time, this too will be solved.

第八，AI训练已经具有很大的规模经济效应。

Eight, AI training already has large economies of scale.

你可以把所有这些昂贵的训练成本分摊到更多用户身上。

You get to amortize all this expensive training across more users.

而且你可以从实验室收入增长远快于其算力增长这一事实中看到证据。

And you see the evidence for this in the fact that the lab revenues are increasing far faster than their compute.

但持续学习也可能在推理端为用户带来规模经济，也就是通过批处理实现。

But continual learning may also lead to economies of scale in inference for end users, namely from batching.

你可能看过我和莱德·波普的那期节目，我们详细讨论过这个问题。

You might've seen my episode with Ryder Pope where we discussed this in detail.

但如果每家公司的指令都需要完整的权重更新，而不是放在低秩适配器中，那么批处理会带来巨大的优势。

But if per company instructions require full weight updates rather than living in low rank adapters, there's huge advantages from batching.

粗略估算一下，对于像DeepSeek v3这样的稀疏模型，最优的推理批处理大小是同时生成超过2400个并发序列。

Back of the envelope math suggests that the optimal inference batch size for a sparse model like say DeepSeek v3 is more than 2,400 concurrent sequences being generated at once.

如果你不这样做，那你的算力就没有得到充分利用。

If you don't do this, then you're under utilizing your compute.

如果你想了解原因，我强烈推荐那期和赖纳关于推理经济学的节目。

And if you want to understand why, again, highly recommend that episode with Reiner on inference economics.

但不管怎样，重点是，一组权重只有在同时处理数千个序列的解码时才能高效服务。

But anyways, the point here is that a given set of weights is only served efficiently when thousands of sequences are being decoded against it all at once.

一个拥有大量员工和代理、从事各种不同业务的大公司，可以非常高效地服务他们的等待队列。

A large company with lots of employees and agents who are doing lots of different kinds of things can very efficiently serve their wait fork.

而个人用户如果只运行批量大小为1，他们的计算效率可能会差两个数量级以上。

Whereas an individual user who's only running a batch size one may suffer more than two orders of magnitude worse efficiency on their compute.

所以，服务个性化权重的经济性强烈偏向于大型组织。

So the economics of serving personalized weights strongly favor big organizations.

显然，到持续学习真正实现的时候，还会有更多变化。

Obviously plenty more will have changed by the time that Continual Learning actually works.

而最重要的变化可能是那些最难提前预料的。

And the most important changes are probably the ones that are hardest to anticipate in advance.

但上面提到的这些，即使现在看起来也相当明确。

But the ones above seem kind of clear even now.

这是对我网站上发布的一篇博客的叙述。

This was a narration of a blog that I also published on my website.

去 dwarkesh.com 看看吧。

Go check it out at dwarkesh.com.

不然的话，我们下期播客再见。

Otherwise, I will see you on the next podcast.