下一个重大突破将是人工智能在工作中学习。

所以，现在所有实验室都在押注一个重大研究方向。

So here's the big research bet that all the labs are making.

他们认为，只要让AI在成千上万个不同的强化学习环境中完成数百万个可验证的任务，就等于基本造出了通用人工智能。

They think that if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we will have basically built AGI.

因为这种训练方式会催生出一种具备问题解决能力的智能体。

Because this kind of training will have created a kind of problem solving agent.

一种能在面对错误、模糊和不确定性时，连续数周持续推进任务的智能体。

The kind of thing that can make progress on opening the task for weeks on end in the face of errors and mistakes and ambiguity.

那些对这个愿景持乐观态度的人会说，我们目前训练模式里所有被诟病的根本性缺陷——比如数据效率低、缺乏持续学习能力——只要加大训练规模，这些问题都能被强行碾过去。

And the people who are optimistic about this vision will say that all these things that we talk about as the fundamental deficits in the current training paradigm, for example, the data inefficiency of these models or the fact that they lack continual learning, These things can just be steamrolled if we just scale training more.

就像当年自然语言处理领域所有核心难题，一投入足够算力到大语言模型上，就全都被解决了。

And the same way that all the fundamental research problems in natural language processing collapsed when we just threw enough compute into LLMs.

在上一篇文章里，我提到过，这些模型的样本效率只有人类的百万分之一。

So in the previous essay, I talked about how these models are one one millionth the sample efficient as humans.

而支持当前训练范式的人会说，没错，这可能是真的，但那只发生在训练阶段。

And the people who are in favor of the current training paradigm will say, look, that might be true, but this is only true during training.

而训练是一次性投入，成本可以分摊到模型经历的数十亿次会话中。

And training is this one time cost that is amortized across billions of sessions that a model will experience.

真正关键的是，模型在单次会话中有多聪明、多通用、多高效。

And what really matters is how smart and general and sample efficient the model is during a session.

随着我们不断加大强化学习的训练，这一点显然在持续提升。

And this has clearly been improving as we've been doing more RL training.

AI代理现在能解决越来越复杂的问题，而且持续时间也越来越长。

AI agents are able to solve more and more ambitious problems over longer and longer time spends.

任何用过这些模型写代码的人都能体会到这一点。

Anybody who has used these models for coding knows that.

同样，有人会说，至于持续学习——

Similarly, people would say, look, continue learning.

我反复提到的这种能力，也就是模型根据实际使用中的学习来更新权重，可能根本没必要。

Capability I keep harping about where the model's weights get updated based on what it's learning from deployment may simply not be necessary.

因为如果上下文学习在更长的时间跨度上变得足够强大，我们就无需再把模型在工作中学到的一切都回传到权重里了。

Because if in context learning gets so good across longer and longer time horizons, that you don't need to distill back everything the model is learning on the job into the weights.

人们常说，新员工要工作六个月以上才能真正产生效益。

People often say that their employees are not net productive until six months or more of them working on the job.

所以显然，在线学习对提升能力是必要的。

So clearly online learning is necessary for competence.

但如果你能把这六个月的经验直接塞进上下文窗口里呢？

But what if you could just fit those six months into the context window?

近年来出现了大量架构上的创新，大幅提升了Transformer能处理的信息量和上下文长度。

There's been tons of architectural innovations that dramatically increase the amount of information or the amount of context that a transformer can store.

再过几年，说不定我们就能拥有感觉上无限长的上下文窗口。

And why not think with a couple more years of progress, we might what feels like infinitely large context windows.

明白了。

Okay.

在深入讨论这项研究之前，我想先退一步，提一个完全不相关但我觉得特别有意思又让人困惑的问题——关于当前AI进展的本质。

So before we discuss this research a bit further, I wanna step back and I wanna ask a completely tangential question, which I find actually very interesting and confusing about the nature of current AI progress.

为什么AI在计算机操作方面的进步，比其他领域慢这么多？

Why has progress on computer use been so much slower than other domains?

电脑操作这事儿是能明确验证的。

Computer use is so clearly verifiable.

你可以问，我订的那个Etsy商品到底送到了没？

You could ask a question like, did the desired Etsy item I ordered get delivered?

我正筹备的那个活动场地，订好了吗？

Is the venue for an event I'm trying to organize booked?

我的税表交上去没？

Have my taxes been submitted?

那为什么电脑操作的进步，比编程和数学这些同样能验证的领域慢这么多呢？

So isn't it weird that computer use has been making so much slower progress than coding and math and these other verifiable domains?

我肯定这背后有好多原因。

I'm sure there's many reasons for this.

其中一个原因，当然是模型在预训练时接触到的高质量多模态数据太少了。

And one of them, of course, is the fact that the models are exposed to far less high quality multimodal data during pre training.

但我觉得还有一个原因，被很多人低估了，它揭示了AI进展这条河正在撞击的峭壁——光有可验证性还不够。

But one reason that I think is actually quite underrated by people and which I think reveals the canyon walls against which this river of AI progress will only slowly chip away at is that it is not enough for a domain to be verifiable.

它还必须非常容易反复试验，也就是说，你得能在一个确定性且可复现的模拟器里，同时运行大量并行的模拟过程。

It also has to be very grindable in the sense that you have to be able to run lots of parallel rollouts against a deterministic and replayable simulator.

而且这些模拟都得从同一个初始状态开始。

And you're run those rollouts from the same starting point.

如果你想让模型更擅长编程，你可以定义一个包含软件仓库的容器，里面有个你让AI去实现的缺失功能。

If you're trying to make a model better at coding, you can define some container that has the software repo, it's a missing feature that you have tasked the AIs with creating.

然后你让上千个并行的智能体同时去解决这个问题，每个智能体都拥有完全相同的容器副本。

And then you have a thousand parallel agents that go at the problem, each of which has an identical copy of the container.

但这种方法用在电脑操作上就不行了，至少没那么简单。

But this doesn't work with computer use, at least not trivially.

你不能让一千个用户同时去亚马逊走一遍同样的结账流程来提升AI使用网站的能力，因为亚马逊的反爬系统会立刻发现你的机器人，然后直接封号。

You can't just have a thousand Asians go try the same checkout flow on Amazon to get better at using websites because Andy Chassis will find your bots and shut your ass down.

你可以通过复制Slack、Gmail和其他常用应用和网站来解决这个问题。

You can solve this by making clones of Slack and Gmail and all the other common applications and websites.

但至少目前来看，这种构建模拟环境的方式非常耗时，而且根本没法规模化。

But at least currently, this is a very labor intensive and unscalable way to build environments.

当然，一旦AI足够强大，能够以极高保真度自行构建这些模拟环境，那么计算机操作方面的进展肯定会比现在快得多。

Of course, once AIs get good enough at coding themselves to build these clones with extremely high fidelity, then I'm sure the computer use will make quicker progress than it is right now.

而且这种做法还一举两得，因为让AI从零开始重建整个应用程序，本身就是一个极好的编程强化学习目标。

And you're also killing two birds with one stone with this kind of procedure because getting AI's to rebuild whole applications from scratch is also a great RL objective for coding.

所以，虽然计算机操作的问题很快可能得到解决，但目前的进展缓慢其实给我们提了个醒。

So while computer use itself may soon be solved, its current lethargy is telling us the following.

除非你能为某个领域建立一个高度可复现的训练目标，否则模型很难取得实质性进展。

That unless you can build a very replayable training target for a domain, the models will struggle to make much progress.

而之所以会这样，原因很明确——模型在训练过程中对样本的利用率实在太低了。

And the reason this is true, of course, the models are incredibly sample inefficient during training.

这一点我在上一个视频论文里已经提到过。

This is a point I was making in my last video essay.

因此，对于计算机操作，你或许可以通过构建这些强大且确定性的模拟器来弥补样本效率的不足。

So for computer use, you might be able to make up for the sample efficiency deficit by building these formidable deterministic simulators.

但对于我们需要AI掌握的许多其他技能，根本没法这么做。

But for so many other different kinds of skills that we need AIs to have, we simply can't do this.

我们该怎么训练AI，让它真正学会从零开始创办一家企业？

How do we train an AI to get really good at building a business from scratch?

那赢得官司、在市场中做一天盈利的交易，或者帮候选人赢得选举呢？

How about winning court cases or having a profitable day of trading in the markets or helping a candidate win an election?

这些事情需要跟真实世界互动，光靠数据中心里的数据是复现不了的。

The rollout here requires interacting with the real world, and you can't recreate it from just within the data center.

要验证这些长期结果，可能得等上几个月甚至几年的真实行动，你没法靠在成千上万次并行模拟里微调模型动作，来精准找出到底哪个举动起了作用。

The outer loop verification here may take months or even years of real world actions to elicit, And you can't re observe it by perturbing the model's actions slightly in thousands of parallel rollouts to isolate exactly what the model did that actually worked.

处理这种无需重置、不断变化的环境，本来就是强化学习领域一个公认的难题。

Now dealing with such reset free non stationary environments is a known open problem in RL.

我这里说的都不是什么新观点。

I'm not pointing out anything new.

但我真想强调一点：正因为世界上大多数领域的数据都独特又稀疏，想要精通，就必须靠高样本效率。

But I really do want to emphasize that because of the idiosyncratic and sparse nature of data in most domains in the world, you need sample efficiency in order to get proficient.

如果AI要掌握人类拥有的所有技能，甚至人类都没有的那些技能，它们就必须能从极少的真实互动中，学习那些结构松散、无法验证、充满模糊性的信息。

If AIs are to develop all the skills that humans have and even skills that humans don't have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real world interaction.

因为在许多领域，相关的训练信息根本不存在于其他任何形式中。

Because in many domains, the relevant training information simply doesn't exist in any other way.

要训练出一个在政治上堪比林登·约翰逊的AI，它的强化学习环境该是什么样？

What is the RL environment to make an AI that is as good at politics as Lyndon Johnson?

或者要让它在创办航天发射业务方面媲美埃隆·马斯克，环境又该怎样设计？

Or as good at building a space launch business as Elon Musk?

各大实验室都押注于RLVR能够泛化。

The labs are betting that RLVR will generalize.

他们的想法是，只要在足够多的可复现、容器化环境中训练，就能培养出一个高度通用的智能体，它能在单次会话中快速制定计划、吸收新信息，甚至掌握全新技能。

That is that if you train on enough containerized reproducible environments, you will develop a very general agent that can make it execute plans and learn rapidly for new information and even pick up new skills all within a single session.

如果你把这种无限泛化的RLVR AI丢进1948年的德克萨斯政坛，它给出的参议员选举策略，可能比约翰逊本人还高明。

If you drop this endlessly RLVR AI into Texas politics in 1948, it could give you better advice than LBJ about winning the senate seat.

如果你在2002年给它一亿美元，让它放手去干，它真能给你打造出SpaceX。

And if you give it a $100,000,000 in 2002 and let it cook, it would build SpaceX for you.

但RLVR究竟能不能达到这种泛化水平，这还是个需要实证检验的问题。

Now whether RLVR can generalize this well is an empirical question.

如果实验室把在强化学习环境上的投入从几十亿美元增加到一万亿美元，你就能在上下文窗口内得到一种完全像人一样的通用智能吗？

If the labs went from spending billions of dollars on RL environments to a trillion dollars, would you get the kind of thing that is a fully human like general intelligence within the context window?

达里奥在我们播客对话中说过一句很有意思的话，我觉得这暗示了RLVR的泛化能力并不是无限的，他当时是在解释为什么模型在长上下文下表现会下降。

Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not infinitely strong, but he was explaining why model performance tends to degrade at long context.

他说，有两件事。

He said There's two things.

一个是训练时使用的上下文长度，另一个是实际部署时使用的上下文长度。

There's the context length you train at, and there's a context length that you serve at.

如果你在短上下文上训练，却指望在长上下文上运行，就可能会出现性能下降。

If you train at a small context length and then try to serve at a long context length, like maybe you get these degradations.

也许我过度解读了，但听起来他的意思是，短周期的强化学习训练，并不能自然地迁移到长周期任务的表现上。

Now maybe I'm reading too much into this, but it seems like he's saying that short horizon RL training doesn't necessarily generalize to long horizon RL performance.

如果连短周期到长周期都无法泛化，那这些智能体又怎么指望从一堆白领任务的训练中，直接跳到被扔进现实世界后，能像山姆·沃尔顿那样从零开始创办一家企业呢？

And if you can't generalize from short horizon to long horizon, then how are agents supposed to generalize from getting trained at a bunch of white collar tasks to say, having the ability to be dropped in the real world and build a business from scratch as well as Sam Walton.

就算经过大量上下文经验后，AI能变得像亨利·福特或罗伯特·爱因斯坦那样，但如果这些学到的东西没法回传到模型权重里，一切终究只是过眼云烟，白白浪费了。

And even if after enough in context experience, the AIs can become like Henry Ford or Robert Einstein, all that would be ephemeral and wasted if you couldn't get those learnings back into the weights.

实验室大约有30%到50%的算力用在了推理上。

Around 30 to 50% of a lab's compute goes to inference.

而这些算力目前根本没在帮助模型提升性能。

And that compute is currently not playing any productive role in helping improve the model.

这简直是巨大的浪费，而且情况比听起来更糟——因为只有在实际部署时，模型真正能从中学习的宝贵信息才会浮现出来。

This is like a huge waste and it's even worse than it sounds because it is only in deployment that the most valuable bits of information which your model could learn from are actually revealed.

比如，我被用在哪些组织里，他们到底拿我做什么，我在真实世界里常犯哪些错误。

Things like what's actually happening in the organizations where I'm being used and what are they using me for and what kinds of mistakes do I tend to make in the real world.

我们有个天才研究生，却从来没机会去企业实习过。

We've got some genius grad student who's never been allowed to take a real internship.

我们只是不断给他塞更多课堂案例，形式就是各种强化学习环境训练。

And we keep giving it more and more classroom case studies in the form of RL training on environments.

真奇怪，AI早就广泛应用于经济各个角落，参与了这么多不同任务，接触了如此多行业和组织特有的知识。

It's so bizarre that we have AIs that are broadly deployed through the economy already and are participating in so many different kinds of tasks and are privy to so much domain and organization specific task and knowledge.

可它们却完全用不上这些信息。

And they're not able to make use of it.

但这种持续学习需要回到权重本身进行调整。

But this kind of continual learning requires going back to the weights.

AI不可能随着用户增多，就不断堆大键值缓存来学习。

AIs can't just keep building up a bigger and bigger k v cache as they learn from more and more users.

这根本没法扩展。

That's just not scalable.

而且人类也不是这么学习的。

And that's also not how humans do it.

我们的大脑里，参数和激活状态根本分不清。

There's no clean separation in our brain between parameters and activations.

你学的东西越多，脑袋也不会像气球一样越胀越大。

And it's not like some part of your skull keeps expanding as you learn more things through your lifetime.

我们学习新东西时，明显会做某种压缩。

When we learn stuff, there's clearly some kind of compression.

这种压缩反而帮我们更好地泛化和真正理解。

And this aids our generalization and grokking.

确实有一些人患有自闭症学者综合征，能多年后依然准确回忆起随机的数字表或无意义音节。

There are, in fact, some humans who have this autistic savant type ability to recall random tables of numbers or nonsense syllables years later.

也就是模型在上下文中所保留的那种高精度信息。

Basically, the kind of fidelity information that models have in context.

但这种惊人的记忆容量反而严重阻碍了他们理解抽象概念和隐喻的能力。

And such sheer volume cripples these humans ability to understand abstractions and metaphors.

人类的持续学习，与其说是把所有经历都记在脑子里，不如说是不断打磨出正确的直觉和宏观知识，并将其固化到大脑的‘权重’中。

Human continual learning is less about having all your observations at the tip of your tongue and more about chiseling the right intuitions and big picture knowledge back into the weights.

但一旦你开始调整这些‘权重’，就不得不放弃基于上下文的学习效率。

But the moment you move into the weights, you have to give up on in context learning sample efficiency.

因为参数更新本身效率极低，所有成功上线的在线学习模型，都不得不让数百万用户重复学习完全相同的内容。

Because grading updates are super sample inefficient, all of the successfully shipped online learning models have had to learn the exact same thing across millions of users.

比如，光是光标补全模型，每天就要对超过四亿次请求重复预测同一个目标。

For example, the cursor tab model online learns by predicting the same exact objective for over 400,000,000 requests a day.

这个目标就是：用户最终接受了哪些编辑操作。

The objective here being which edits actually got accepted by the user.

到目前为止，我们还没见过模型能为不同用户学习不同的东西。

At least so far, we haven't seen models online learn different kinds of things for different users.

因为虽然单次会话产生的数据对人类来说已经足够学习，但对训练更强大的AI来说还远远不够。

Because while a single session may generate more than enough data for a human to learn from, it's not enough to train a more capable AI.

当前的在线学习只适用于少数几种场景，但持续学习的真正意义在于，这个世界实在太复杂了。

Current online learning can work for a very limited number of use cases, but the whole point of continual learning is that the world is very complicated.

每个工作、每家公司、每个问题都不一样。

In each job and company and problem is different.

你需要你的智能系统能学会与特定部署相关的具体信息，而这些信息根本没法塞进一个共享的训练流程里。

And you need your intelligence to be able to learn the specific information related to a particular deployment, which simply can't be stuffed into some shared training run.

我们说的‘在岗学习’，指的就是这些事情。

These are all the things we're talking about when we talk about on the job learning.

比如你公司里各种系统是怎么运作、怎么配合的，怎么跟周围的基础设施和其他人协作推进大项目，常见的失败模式有哪些，还有其他很多类似的问题。

Things like how does everything in your organization work and fit together, and how to cooperate with all the infrastructure and the other people around you to make progress on some larger project, and what the common failure modes are, and many other things like this.

随着播客规模扩大，我不得不处理越来越多的运营事务。

As the podcast has grown, I've had to deal with more and more operational overhead.

比如付账单这件事。

Take paying bills.

以前，承包商都会直接发邮件给我发票。

In the past, contractors would just email me their invoices.

每隔几周，我就得翻一遍邮箱，建个文件夹把所有账单收在一起，然后手动一一支付。

Every few weeks, I'd dig through my inbox, I'd create a folder with all the bills, and I'd manually pay each one.

不过现在，我给每个人一个专门的邮箱地址，直接连到我的银行平台Mercury。

At this point though, I just give everybody an email address that goes straight to Mercury, which is my banking platform.

只要有人把发票发到这个地址，Mercury就会自动下载、扫描，并提取所有关键信息。

Whenever anybody sends an invoice to that address, Mercury automatically downloads it, scans it, and extracts all the relevant information.

比如承包商的名字、地址、金额、发票号和截止日期，然后用这些信息自动生成一笔待付款草稿。

Things like the contractor name, address, payment amount, invoice number, and due date, and then uses all of this to create a draft payment.

Mercury会把所有这些草稿列成清单，供我核对。

Mercury then stores a list of these drafts for me to review.

我只需要过一遍清单，确认账单没有算错就行了。

I just go through this list and double check that I've been billed correctly.

我不用自己追踪任何信息，也不用手动输入数据。

I don't have to track anything or enter any information myself.

Mercury 把企业运营中最基础的功能都做得非常好，而且全都集中在一个地方。

Mercury does all the fundamental things for your business extremely well, and it puts them all in one place.

如果你想了解更多，可以访问 mercury.com。

If you want to learn more, go to mercury.com.

这是一家金融科技公司，并不是一家受 FDIC 保险的银行。

Is a fintech company, not an FDIC insured bank.

银行服务由 Choice Financial Group 和 Column NA 会员银行提供，受 FDIC 保障。

Banking services provided through Choice Financial Group and Column NA Members FDIC.

这样一来，样本效率和持续学习实际上是紧密关联的问题。

In this way, sample efficiency and continual learning are actually deeply connected problems.

模型在实际工作中能接触到的数据非常有限。

Relatively little data is available to the model on the job.

要从这些数据中学习，就需要高样本效率，而模型可以通过注意力机制动态生成的快速权重在上下文中实现这一点，虽然提升了样本效率，但内存开销却大得惊人。

Now to learn from this data requires sample efficiency and models can do that in context, but using the fast weights that are built on the fly by attention, which allows for the sample efficiency, but scales very poorly in terms of memory.

所以我们需要一些架构上的创新，来支持某种中间表示形式。

So we need architectural innovations that allow for some kind of intermediate representation.

我之前提到过，其实我们已经有不少可行的思路了，比如稀疏注意力、KV缓存压缩这些。

I talked before about how we already have many different working ideas for this kind of thing, from sparse attention to k v cache compaction.

每周都有人发布新论文，提出各种新的架构优化方案。

And every week, somebody releases a new paper suggesting some kind of other architectural optimization.

在我看来，架构本身并不是制约持续学习的根本瓶颈。

It doesn't seem to me that architecture is fundamentally what is bottlenecking continual learning.

所以，也许真正的瓶颈在于损失函数。

So perhaps the bottleneck is the loss function.

我们该怎么更新权重？

How do we update the weights?

换句话说，怎么根据某次会话中学到的信息，来提升模型本身？

AKA, how do we improve the model itself based on information that was learned from one particular session?

即便在这一点上，直觉上也觉得有很多方法应该是行得通的。

Even here, naively, it seems like there are many ideas that ought to work.

最近很多人都在讨论一种叫在线自蒸馏的技术。

A lot of people are talking about this technique called on policy self distillation recently.

如果你想深入了解，我几周前和萨沙·拉什用iPhone随手录了个黑板讲解，链接在简介里。

You wanna learn more about it, I recorded a little impromptu Blackboard lecture on my iPhone with Sasha Rush a couple weeks ago, and it's in the link in the description.

简单说，这个方法的核心是让基础模型在解决实际问题时，做出的预测和经过长时间会话后积累了全部上下文的模型一致。

But to summarize the explanation, the idea is that we encourage the base model to make the same predictions when trying to solve some real world problem as the model with all the context accumulated after a long session would have made.

这样做的目的，就是把模型在一次会话中学到的东西，重新压缩回模型的权重里。

The whole point of this procedure is to distill what the model learned in a session back into the weights themselves.

相比RLVR，这种方法有两个优势。

This is better than RLVR for two reasons.

第一，在线自蒸馏不需要外部可验证的奖励机制。

One, OPSD doesn't require us to have some outer loop verifiable reward.

我们只需要一个能在上下文窗口内学会正确内容的模型就行。

We just need a model that can learn the right things within the context window.

只要具备这一点，我们就能训练基础模型，让它模仿那个在会话中积累了丰富经验的‘老手’模型。

And as long as we have that, we can train the base model to match our veteran teacher model, which has built up all this experience during the session.

第二，OPSD提供的监督信号比简单的强化学习要密集得多。

And two, OPSD provides a much denser supervision signal than naive RL.

你不需要把一个单一的奖励信号贯穿整个序列，而是可以根据教师模型和学生模型在每个词上的概率差异来进行训练。

Instead of projecting a single reward through the whole trajectory, you can train on the per token probability discrepancy between the teacher and student.

对于持续学习来说，OPSD也比监督微调更有效。

For continual learning, OPSD is also superior to supervised fine tuning.

最简单粗暴的监督微调方式，就是让基础模型去复述会话中出现过的每一个词。

The most naive version of SFT for this application that you can imagine is just to train the base model to predict all the tokens that are observed during the session.

但如果你仔细想想，这根本不是一个合理的学习目标。

But this makes no sense if you think about it as a learning target.

你提升工作能力，不是靠一字不差地记住每天发生的所有事情。

The way you get better at your job is not by recalling the transcript of every single thing that happened every day with perfect fidelity.

而是通过提炼出那些真正有助于你进步的关键洞见和知识片段。

Rather, it's by consolidating the handful of insights and pieces of knowledge that are actually relevant to you getting better at your job.

强化学习交易法不会出现这个问题。

RL trading doesn't suffer from this failure mode.

强化学习的优势在于，它只聚焦于对达成正确结果真正重要的部分进行更新。

RL is great at concentrating the update to only what is relevant to getting the outcome right.

所以，在一次强化学习的训练步骤中，实际被调整的参数非常少。

That's why actually very few parameters are changed during an RL training step.

这一点对持续学习特别关键，因为在工作中学习时，我们不希望覆盖或遗忘基础模型原本掌握的其他知识。

And this is a very important property for continual learning because as you're learning on the job, we don't want to overwrite and forget all the other things that the base model knows.

几个月前我写过一篇文章，指出强化学习每个样本学到的信息比监督学习少得多。

I wrote a post a few months earlier arguing that RL learns much less information per sample than supervised learning.

但这未必是坏事，反而可能是好事。

But this may be a good thing rather than a bad thing.

你只在绝对必要的范围内调整模型，不多也不少。

You only change the model as much as it is absolutely necessary to achieve the outcome and no more.

OPSD保留了监督学习的这一特性：它不像监督学习那样直接猛扑向教师模型的分布，而是只提取那些让你在真实任务中达到教师同等表现所必需的知识。

OPSD preserves this property of supervised learning where instead of slingshotting towards the teacher distribution as supervised learning would have you do, you only extract the knowledge that is necessary for you to achieve the same results of the teacher on actual real world tasks.

OPSD是解决样本效率问题的一种方法。

OPSD is one way to attack the sample efficiency problem.

你把有限的真实世界经验，提炼成精准而高效的微调。

You take the scarce real world experience and you squeeze all the signal into a tiny well targeted update.

但还有一个更富想象力的思路。

But there's also another much more speculative idea.

我们不妨叫它‘模拟做梦’。

Let's call it dreaming.

如果AI能构建出逼真的现实模拟，用来演练新技能、尝试不同策略，并强化真正有效的做法，那么在同样的实际时间里，它就能体验到成百上千倍的模拟样本。

If the AI can build a good simulation of reality against wish to rehearse new skills or try alternative strategies and reinforce what actually works, then AIs could experience orders of magnitude more simulated samples in the same wall clock time.

我们稍微回溯一下历史。

Let's go back in history a bit.

DeepMind发布AlphaZero几年后，一群研究者训练了一个叫EfficientZero的模型。

A couple of years after DeepMind released AlphaZero, a group of researchers trained a model called EfficientZero.

这个模型的核心目标就是极致高效地利用数据。

And the whole point of this model is to be very efficient with data.

如果这个模型和一个新手都只有两小时时间去玩一个从未见过的Atari游戏模拟器，这个模型很可能会赢过人类新手。

So if this model and a human both got two hours to play against a simulator of an Atari game that they hadn't seen before, this model would actually probably beat the novice human.

这意味着这个模型比人类更节省样本吗？

Does this mean that the model was more sample efficient than the humans?

训练目标确实是这样，但怎么衡量样本效率也很关键。

Well, that was the goal of the training, but it depends on how you measure sample efficiency.

因为在现实游戏的每一步中，EfficientZero都在脑子里模拟了几十场游戏。

Because for each step in the real game, EfficientZero is playing dozens of simulated games in its head.

同样地，未来的大型语言模型或许能用少得多的真实数据，不断在自己构建的环境中练习。

In a similar way, future LLMs might be able to consume far less real world data while practicing endlessly against environments that they build for themselves.

当然，最大的区别在于，模拟整个世界要比模拟围棋难得多。

The big difference, of course, is that it will be much harder to build a simulation of the whole world than it is to emulate the game of Go.

所以我才说，这个想法要更 speculative 一些。

That's why I said this is a much more speculative idea.

如果能实现，它将成为扩展模型能力的第四条路径，和预训练、强化学习、推理算力并列。

If it works, it would become a fourth axis of scaling alongside pre training RL and inference time compute.

你可以管它叫测试时训练，或者叫做梦。

You can call it test time training or dreaming.

模型会消耗计算资源来构建强化学习环境，然后在这些环境中进行训练。

The model spends compute writing up RL environments and then training against them.

它还在反复演练那些将来会真正用于特定用户场景的技能。

And it's rehearsing all the skills that will actually be used in production for a specific user.

所以你不需要像在 Codex、Cursor 或 Claude 里敲斜杠 compact 那样，只用一点算力生成个摘要，假装是持续学习——你可以直接敲斜杠 dream。

So instead of hitting forward slash compact in codex or cursor or clod, which kindles a small amount of compute to write up a summary and which gives you the simulacrum of continual learning, you hit forward slash dream.

而这个操作会消耗海量算力，去构建并训练一个与现实世界所见对应的虚拟游戏版本。

And this incinerates huge amounts of compute to build and train against the video game version of what the model is witnessing in the real world.

那么到 2027 年或 2028 年，我的持续学习会是什么样子？

So what my continual learning look like by 2027 or 2028?

我们又该怎么实现它？

And how do we get there?

这里有一个可能的场景。

Here's one scenario.

所有这些 RLVR 训练，最终会造出一种智能体：面对陌生问题时能迅速定位方向，尝试不同策略，并在遇到障碍时不断调整迭代。

All of this RLVR training is producing an agent that can get its bearings when it's thrown at an unfamiliar problem, and it can try different strategies, and it can iterate when it hits a roadblock.

这才是 RLVR 带给你的关键：一个至少具备基本能力的 AI，如果你打算让它从经验中学习，它就能开始积累真实世界的实战经验。

This is the crucial thing that RLVR has given you, an AI that is at least competent enough to start getting some real world experience if you're gonna learn from it.

一旦有了这个基础，你就可以把它派出去干活，哪怕是那些不在常规分发渠道里的项目，它也能胜任。

And once you have that, you send it out into the world to do real work even on projects that are off the trading distribution.

假设到了这一步，有效上下文长度已经扩展到了让 AI 能和你连续协作整整一周（按实际时钟时间算）。

Now let's say at this point, the effective context lengths have expanded such that AIs can jam and co work with you for a full week of wall clock time.

一周结束后，你给它点个赞或者点踩。

At the end of a week, you give it a thumbs up or a thumbs down.

你给它做一次工作评估。

You give it a work review.

如果你给它点个赞，基础模型就会把这次会话里 AI 学到的东西全部提炼出来。

And if you give it a thumbs up, the base model distills everything that the AI learned during the session.

它可能会用 OPSD，可能会用“做梦”机制，也可能用我们还没意识到的其他技术，或者把这些方法结合起来用。

And it may use OPSD, it may use dreaming, it may use some other technique that we aren't even aware of, or it'll use a combination of all of the above.

一旦做到这一点，AI 就能在与其原本通过 RLVR 训练明确掌握的技能相邻的领域里，表现得越来越好。

And once it does so, the AI starts getting better domains that are adjacent to what it was explicitly trained to beforehand with RLVR training.

在接下来的一轮中，它还能在之前在线学习过的内容基础上，进一步提升相关领域的能力。

And in the next round after that, it can get better at things that are adjacent to what it had previously online learned.

这样一来，人工智能的技能、知识和能力范围，就能远远超越它最初训练时所覆盖的那些可验证领域。

In this way, the gamut of AI skills and knowledge and capabilities can expand far beyond the verifiable domains that the model was originally trained against before it was deployed.

就像预训练打造了一个足够聪明的基础智能，再叠加足够的RLVR训练，就能让它成为称职的代理。

Just as pre training created a base intelligence that was smart enough to become a competent agent with enough RLVR on top.

而RLVR正是创造出了一个足够称职、可以真正广泛投入实际应用的智能体。

So RLVR has created an agent that is competent enough to be actually broadly deployed in the world.

一旦这种持续学习的训练方法成型，AI就能通过广泛部署、在实际工作中不断学习。

And from this broad deployment to learn on the job once the training recipe for continual learning actually arise.

到这个时候，AI提升的主要途径，已经不再是发布前的那些初始训练了。

By this point, the main way that AIs get better is not from the training they have received before they are released to the public.

真正让它变强的，是它在经济中广泛部署后，所积累的大量实践经验，以及处理的各类任务。

Rather, it's from all this experience that they'll be accumulating from being broadly deployed in the economy and engaging in so many different kinds of tasks.

你每次和AI互动，它都会变得更聪明。

Every time that you interact with an AI, it'll be smarter.

不只是因为系统从你之前的对话中学到了东西，还因为它从全球所有其他用户的互动中不断积累经验。

Not only because it's been learning from your previous sessions, but also because it's been learning from all its interactions with all the other users in the world.

这既让人感到不安，又令人兴奋，和现在AI进步的方式完全不同。

And that's very scary and exciting and different from the way that AI improves right now.

这篇内容是我之前在 dwarkesh.com 网站上发布的一篇博客文章的更新版本。

This was an iteration of a blog post that I also released on my website at dwarkesh.com.

如果你想查看所有脚注，或者想订阅以便及时收到我下一篇博客的更新，欢迎去那里看看。

Go there if you wanna read all the footnotes or if you wanna sign up so you can find out when I release the next blog post.

否则，我们下一期再见。

Otherwise, I'll see you on the next episode.