超越更大模型：递归作为人工智能的下一个扩展法则

欢迎回到《解码》的又一期节目。

Welcome back to another episode of Decoded.

今天我请来了YC的访问合伙人弗朗索瓦·肖巴尔，聊聊AI研究里最近一个特别有意思的趋势——递归。

Today, I'm back with YC visiting partner Francois Chaubard to talk about one of the most interesting recent trends in AI research, recursion.

我们重点要说的是，怎么在推理阶段用递归，而不是一味把模型做得越来越大，来提升模型的推理能力。

Specifically, we're gonna talk about how we can improve a model's reasoning performance by using recursion at inference time rather than by just making the model bigger and bigger.

2025年有两篇论文，把这种做法的威力讲得特别清楚。

There were two papers that made the power of this approach really clear in 2025.

一篇是关于层次化推理模型，也就是HRM；另一篇是关于小型递归模型，简称TRM。

One on hierarchical reasoning models or HRM and another on tiny recursive models, TRM.

弗朗索瓦，谢谢你来参加我们的节目。

Francois, thanks for joining us.

你能给我们简单讲讲这两个模型，到底哪儿特别有意思吗？

Can you tell us a little bit about these two models and what was so interesting about them?

当然可以。

Sure.

为了打个底，你之前那期视频讲RNN和语言模型已经讲得特别棒了，我就不重复了。

I guess to set up a little bit of a foundation, you already did an amazing lecture on RNNs and LMs in one of the previous videos, so I won't overdo it.

简单说一下要点，RNN就是不断自己调用自己、循环执行的模型。

But just to give the cliff notes, an RNN is just a model that you recursively call again and again and again on itself.

那时候大家都觉得，要实现通用人工智能，非得靠这种递归结构不可——直到2016年Alex Graves在NeurIPS的演讲，那场演讲太精彩了，他那些自适应计算时间的研究简直神了。

And we were very much in the belief that this was required to get to AGI peak RNN usage probably until 2016 with Alex Graves NeurIPS keynote, which is just fantastic, and all his his his adaptive compute time work.

这都十年前的事了。

So this is about ten years ago.

那时候大家都在搞这些模型。

People were working on these models.

那会儿流行的是LSTM，还有带注意力机制的LSTM。

This was in the era of LSTMs and LSTMs with attention.

是的。

Yeah.

在注意力机制出来之前，你去问不同的教授，他们看法都不一样。

And depending on which professors you talked to before attention was invented.

对。

Yes.

没错。

Yes.

完全对。

Totally.

是的。

Yeah.

我觉得RNN普遍受限的关键在于‘通过时间反向传播’——你得把模型展开，然后为了更新权重，还得去近似梯度，一路倒推回去，不断展开。

And and I think what really was the the the limiting step on RNNs in general was this thing called back prop through time where you have to you roll out the model, and then to update the weights, you need to approximate the gradient and you step back, back, back and you keep rolling out.

随着模型越来越大，展开的步数越来越多，误差不断累积，梯度也越来越噪，最后根本没法训练了。

And as the the model gets bigger and bigger and as you roll out for more and more steps, then you have all these accumulation of errors and the gradient gets noisier and noisier and then it just stops the work.

所以你遇到的就是梯度消失或爆炸的问题，因为如果输入有20步，你就得把矩阵连乘20次，这就导致了……

So you have these vanishing or exploding gradient problems and it's because if you have an input with 20 steps, you're multiplying these matrices 20 times that causes

我们现在说的可是上下文长度达到百万甚至十亿级别。

And we're talking about doing context length of like a million or like a billion.

所以根本不是20步，而是几十亿步。

So it's not even just 20, it's like a billion.

更糟的是，你还得在每一步都保存激活值。

And even worse, you have to retain the activations at every single step.

要是这事儿发生在你脑子里，那你得在每个激活时刻都准备上百万个你的大脑副本，才能做反向传播。

And so like if this were happening in your brain, would need like a million copies of your brain at every single activation so that I can back prop through it.

不过也有一些技巧能缓解这问题，比如梯度检查点之类的，能减少点内存压力。

There's tricks around this that you can you can do and you can do gradient checkpointing and things like that to reduce that issue.

但这么一来，其实就是拿内存换时间和算力。

But then you just like trading off memory for wall clock time and and compute.

对啊。

Right.

那现在你要是拿这个跟大语言模型比，比如现在大家普遍用的那些，表面上看好像差不多，但训练的时候，它们其实是对每个输入做一次性前向传播，对吧？

So now if you contrast that with LLMs, the ones that people are widely using, these while at face value they appear to be similar, at training time they're doing basically this one shot feed forward process for every input, right?

大语言模型的Transformer模块可以并行处理所有输入。

The LLM, the transformer block can take all of the inputs in parallel.

训练的时候它并不是一个一个地迭代处理，所以不会出现要存海量激活值的问题，也不会有严重的梯度消失问题。

It's not actually iteratively going over them one at a time at train time, so you don't have this needing to store tons of activations problem or this giant vanishing gradients problem with them.

对。

Yeah.

没错。

Exactly.

就像是所有步骤在一瞬间同时完成了一样，简直像魔法一样。

Like it's actually like all happening in time in one shot magically.

其实就是那种三角掩码的技巧，也就是那个因果掩码在起作用。

And that was like the the trill or lower triangle trick that kind of happens, this causal mask that occurs.

所以你实际上是一次性完成所有时间步的前向计算，把整个序列丢进前馈网络，再一次性反向传播，训练速度快得惊人。

And so you actually do all all time steps in one shot, and you forward pass a feed forward model on all time steps in one shot, and you backwards in one shot, and it's amazing for train time in terms of like wall clock.

虽然需要大量的浮点运算，内存占用也还是很大。

It requires a lot of flops and it still requires a lot of the memory.

内存确实还得撑着，但至少不用担心梯度消失了。

You still need it there, you don't have the vanishing gradient issue.

但你为此付出的代价是失去了这种潜在的推理能力和时间维度上的压缩。

And that what you actually paid for that you have to give up is this latent reasoning thing and this compression in the time direction.

语言模型里根本不存在压缩。

There is no compression in LMs.

我每次解码，都得把整部《莎士比亚全集》记在脑子里，就为了生成那么一小段。

Every single decode that I do, I still have to retain the entire, you know, Shakespeare novel just to, like, decode a little bit.

而RNN就不需要这样。

And in RNNs, don't have to do that.

所有信息都压缩在那个隐藏状态里，慢慢滚动推进就行。

It's all compressed in this hidden state that you kind of roll out.

明白了。

Okay.

那咱们再详细聊聊这个。

So let's talk about that in a little bit more detail.

你提到的那种内在的推理能力。

Like, you you refer to this inherent reasoning ability.

很多人觉得大语言模型是在进行推理，这个我们待会儿再细聊。

You know, many people think about LLMs as doing reasoning, and we're going to talk about that a little bit later.

但你能帮我理清楚一点吗？在实际的前向计算过程中，你觉得大模型的推理能力最大的瓶颈到底在哪？

But help me understand where you see the biggest limitations in LLMs' reasoning ability is in terms of what the model does in an actual forward pass.

是的。

Yeah.

所以我们还得回过头去看 GPT-2。

And so I guess we go back to chat GPT two.

GPT-2 那个架构在当年可是个里程碑，就靠一个词接一个词地预测，结果还真能用。

GPT two was this landmark architecture in paper that basically was just next token, next token, next token, and it kind of worked.

我们当时就看着验证损失一路下降，困惑度也越来越低，模型表现越来越好，甚至能写出点像那么回事的莎士比亚风格文字。

And like we just watched val loss go down, perplexity goes down, like the model just is more performant, looks better, starts to make some Shakespeare that actually sounds somewhat plausible.

对吧。

Right.

但后来我们得让这些模型真正学会推理，去解决一些特别难的问题。

And then we have to get these things to reason and to actually solve some really hard problems.

关于这个问题我做过大量的实验，就拿排序这个事儿来举例吧。

And and I've done extensive experiments on this, but like if you take, for example, sort.

你给模型喂无穷无尽的未排序列表，还有对应的排好序的列表。

You get you have infinite amounts of unsorted lists and you give it sorted lists.

就算一直这么训练它，它也没法真的把这个任务学好。

You keep feeding it to the model, it should work.

是这么个理儿吧？

Right?

这个模型根本没法直接把一个乱序的列表整理成有序列表。

It's actually impossible for the model to map from unsorted list to sorted list.

你是说一次性完成的话？

If I have In a one shot basis.

对，就是一次性完成的场景。

In a one shot basis.

说白了从理论上讲，我们都知道比较排序法的复杂度下界是n log n步，你不可能用更少的步骤完成排序。

It's like it's like literally that we know a theoretical lower bound that for comparison sort you can't do better than n log n steps.

如果我有一个包含31个字符或元素的列表，而我的Transformer最大只能处理30个，那我根本没足够步骤去做所有比较。

And if I have a list that's 31 characters or elements long and my transformer is 30, I run out of steps to do comparisons.

我根本没法完成所有必需的步骤。

It's not possible for me to like do all the steps that is needed to to be done.

在HRM和TRM里，他们用数独当不可压缩问题的典型例子，迷宫也是同样的道理。

In HRM and TRM, they use sudoku as an incompressible problem similarly and so are mazes.

这些都属于不可压缩的问题。

Those are incompressible problems.

滚动求和也是个不可压缩的问题。

Rolling sum, incompressible problem.

你提到排序算法的时候，让我想起大学学算法那会儿，唯一能突破n log n下限的方法，就是能访问外部缓存。

So when you mentioned the sorting algorithm, when I think back to my algorithms class from college, the one way you could get faster than n log n in a sorting algorithm is if you had some access to an external memory cache.

如果你能用个磁带啥的临时存数据，那确实能比N log N更快——通过有选择地把元素挪到外部存储里。

If you had some tape you could write to, then you actually can do faster than N log N by basically selectively putting things onto this memory.

我觉得这正是当前大模型的硬伤：模型本身没有外部记忆磁带，所以很多提速的可能性就被堵死了。

And I suspect that's a key limitation of these LLMs in that because there's no external memory tape inbuilt into the model, you lose certain performance possibilities in terms of how fast you go.

没错。

That's right.

所以我觉得基数排序应该是最常见的那种。

And so I guess like a radix sort would be like the most common one.

取决于你有多少个桶，就能把时间复杂度从 n log n 降到接近 n。

Had like depending on this the number of buckets that you have, you can kind of get from n log n to order n.

但不可能比 n 还快。

You can't get less than n.

你总得把每个元素都碰一遍。

You have to touch all the the elements.

不好意思。

Sorry.

这一步是躲不掉的。

You have to do that.

如果你的神经网络里 Transformer 层用完了，就没机会再做这些操作了。

And if you run out of layers and transformer layers in your neural network, then you ran out of chances to do that.

是的。

Yeah.

这简直就是回到艾伦·图灵时代，搞图灵机那一套，对吧？

This is just like a this is like going back to like Alan Turing now and like a Turing machine, right?

那到底该怎么把图灵机和大语言模型类比呢？我觉得这种类比有点说不太通。

Like what's the analogy there exactly that we should think about in terms of LLMs, I guess, not quite satisfying how you think about a Turing machine?

对。

Yeah.

那咱们就直接聊ChatGPT-2吧。

So if we let's just talk about like ChatGPT two.

就是最原始的那个GPT-2，没加任何花里胡哨的东西。

GPT two, the the original, like no bells and whistles.

它就是一个前馈网络。

It's just a feed forward model.

就是一步一步地前向传递，然后取

And so it's just forward passing one step and to Taking

输入一个东西，然后生成一堆输出。

an input, creating a bunch of outputs.

对。

Mhmm.

以数独为例，如果我有50个不同的A格子。

In the Sudoku case, if I have 50 different A squares.

如果它——

And if it's

只要根据现有信息能确定的就只有一种解法，那我有多少层，也就只能做到这一步。

provable that I can only do one given this information, then and I have this many layers, then that's all I can do.

而所谓的捷径，就是思维链。

And the cheat this the cheat is the the chain of thought.

所以确实，在测试时，它们是图灵完备的，能模拟所有图灵可计算的函数。

And so it's completely true that at test time, they are, turn complete and you can simulate all Turing computable functions at test time.

但问题是，你怎么让它学会这个？

But how do you get it to learn it?

你得训练它。

You need to train it.

但问题在于，除非你用人类标注的推理过程来训练，可像千禧年难题这种，我们根本就没有现成的推理轨迹。

And that's where unless you're training it on human labeled traces for which there's a lot of problems like the millennial prize problem, we don't have the trace for.

所以我们特别希望能有

So we'd love to have

这些推理轨迹，

the trace for

但根本不存在。

it, it doesn't really exist.

没错。

Totally.

说得通。

Makes sense.

明白了。

Okay.

明白了这个背景，咱们来聊聊这两篇论文吧，我觉得这能很好地突出它们和大家平时熟悉的模型之间的差异。

So with that context in mind now, let's talk about these two papers because I think that sets up a lot of the contrast we're gonna draw between these papers and the models that people are maybe more used to.

先说说HRM吧。

So let's talk about HRMs first.

你能不能给我讲讲这个模型是怎么工作的，背后有什么直观的思路？

Walk me through a little bit about how this model works and some of the intuition behind it.

当然可以。

Sure.

这个模型其实是从RNN直接发展过来的。

So this is directly in the lineage of RNNs.

从RNN的角度来看，我觉得它并没有太多真正新颖的东西。

There's not that much novel from like the RNN standpoint, at least in my opinion.

他们有个想法，说是受大脑启发——大脑里不同区域是以不同频率运作的。

They do have this idea of, you know, from a inspired by the brain where I have like, there's different parts of the brain that operate on different frequencies.

有些区域频率特别高，属于层级结构里比较底层的部分。

There's some that operate at really high frequency which is on the low level of the hierarchy.

有些则运行在很低的频率，属于更高层的结构，这两者之间的互动特别有意思。

Some that operate in a really low frequency which is the higher level of the hierarchy and the interplay between those things is really interesting.

这确实是直接借鉴了人脑的机制，就像大脑不同区域会以不同频率产生不同的脑波，对吧？

So this is like literally in the human brain, there's some like bio inspiration here, which is that like you have like different waves running at different frequencies at different parts of the brain or

差不多就是这个意思。

something like that.

太棒了。

Cool.

我觉得他们对这种频率层级的解释，至少是一种合理的理解方式。

And and I guess that interpret that's one interpretation of it, of the way that they they're talking about, you know, classifying these these hierarchies of frequencies.

而最让我感兴趣的部分，是他们训练神经网络的方法。

And the way the most interesting part, at least for me, is the way that they train the neural network.

你输入一个x，不管是不完整的数独、迷宫，还是ARC挑战题，先让底层模块跑t_l次，然后切换到高层模块。

You take in some x, some input, whether it's a incomplete Sudoku puzzle, a maze, or an ARC Prize challenge, you do TL steps with the l the the lower level module, then you do go to go to h.

高层模块跑t_h次，最后再做n次外部精调。

You do that, t h times, and then you have n sup outer refinement steps.

对。

Yeah.

所以你其实就是用同一个矩阵、同一个变换，反复处理输入数据。

So you basically are like running through the input with a given matrix, with a given transformation repeatedly on it.

你在这两个层级上做精炼，然后把这个过程重复好几遍。

You're doing that through two levels of refinement and then basically running that process several times.

没错。

Yes.

所以这里实际上有三层递归结构。

So there's exactly three levels of recursion occurring here.

有底层、高层，还有外层的精炼步骤。

There's the low level, there's the high level, and then there's the outer refinement steps.

我们叫它递归，是因为每次都用的是同一组权重。

And we're calling it recursion because it's the same weights that are being applied repeatedly.

这些步骤之间，权重是不变的。

We're not changing the weights in between these steps.

没错，就是这样。

Exactly right.

你可以对 L 网络递归调用 TL 次。

You get to recurse on on the L net, TL times.

对 TH 和 TL 进行循环递归，各 TH 次，然后再做 N sup。

You recurse on the TH and the TL, this looped recursion, TH times, and then you do N sup.

整个外层的精炼步骤，重复 N sup 次。

You do this whole outer refinement step, n sup times.

太棒了。

Cool.

那为什么这样设计能奏效呢？背后的直觉是什么？

And so what's the basic intuition for why that works?

这种做法到底为啥能出好结果？这篇论文实际展示了哪些效果？

Like, why does that produce an effective paper result, and what even were the results that this paper showed?

对。

Yeah.

所以呢，这个模型在ARC奖第一和第二轮都拿了state of the art。

And so, I mean, this got state of the art on ARC Prize one and two.

这模型只有两千七百万参数，而且只在ARC奖数据上训练的。

This was a only a 27,000,000 parameter model that was only trained on ARC Prize.

大概就一千个输入吧，其实就是一些谜题之类的。

So it's like a thousand inputs or something like that, like puzzles basically Literally a

一千个任务，量特别特别小。

thousand tasks, which is extremely small.

根本没做任何预训练。

There is no pre training at all.

它从头开始，完全是白板权重，但当时已经能超越O3了——你要是还记得那时候的事的话。

This starts from like literally Tabula Rasa weights, and it can outperform at that time if we go back, you know, we had, o three, if you remember back way back when.

而O3呢？直接得零分，真的一分都没有。

And it and it o o three gets zero, Literally zero.

但这个模型在ARC奖第一轮至少拿到了70%，那会儿简直是重大突破。

And this got like something like 70% on ARC Prize one at least, at the time, which was just a huge breakthrough.

所以你可以这么理解，这有点像变量的作用域问题。

And so kind of the the way you can kind of think this is like variable scoping.

比如说，我有三个嵌套的函数，最内层的那个函数里会有一些作用域内的变量，他们叫 zl，就是那个会输出零的进位变量。

And so like if I have like, you know, three nested functions, guess the first the lowest level function has like scoped variables which they'll call z l, which is the carry that emits the zero.

一个潜变量。

A latent variable.

潜变量。

Latent variable.

在传统的 RNN 文献里，这个东西叫隐藏状态。

And and like traditional, RNN literature, they would call this the hidden state.

对。

Yeah.

底层的隐藏状态。

The low level hidden state.

我可以一层层递归下去，最后再把 zl 传回外层函数，也就是更高层的那个。

And I get to recurse, recurse, recurse, and then I pass back that ZL back to the the outer scoped function, the higher level one.

我让这一层跑一次迭代，它又会回去调用更底层的函数。

I let that one do one iter, it goes back and calls the lower level again.

这个过程是在第三层外循环里完成的，他们管这叫外层精化步骤。

It does this whole thing in a third outer loop which is called the outer refinement step.

但你这么一说，听起来它跟RNN一样，也会遇到反向传播时的时间依赖问题。

But when you describe it like that, it seems like it would have the same back prop through time problem that you would have at RNNs.

我觉得他们想了个挺聪明的办法来绕开这个问题。

And I think they came up with a clever trick to basically get around that.

对，没错。

So like Yeah.

他们到底用了什么妙招？

What was that trick that they figured out?

这其实是这篇论文最核心的地方，我觉得正是它跟以往研究的根本区别：他们没像Alex Graves在神经图灵机、自适应计算时间、微分神经计算机那些论文里做的那样，把所有递归步骤都反向传播一遍。

And this is really the the crux of the paper that like differentiates it in my opinion in the literature is they instead of doing what Alex Graves did in all of his papers from neural Turing machines to, adaptive compute time, to a differential neural computers is he always backpropped through all of the recursion steps.

而Alex之所以受限，就是因为反向传播要贯穿整个时间序列，模型根本没法做大。

And he was limited by backprop through time, so you can only make the model so big.

你面临各种问题，比如梯度消失之类的。

You have all these issues, vanishing gradients, etcetera, etcetera.

他们用的是一个叫DEQ的固定点迭代方法。

And what they do is they they kind of have this DEQ of method of doing fixed point iteration.

听起来像是深度均衡模型。

Sounds like deep equilibrium

均衡学习——这做法对搞计算机视觉的人来说完全反直觉，因为你根本不会这么干，但其实特别有道理，我来跟你解释为啥。

equilibrium learning where if I take a batch and this this is completely counterintuitive as a computer vision person because you'd never do this, but it actually does make sense and I'll explain why.

如果我拿一批ImageNet或者CIFAR-10的数据，前向跑一遍，算出损失，反向更新权重，下一波我肯定换一批新数据。

If I take a batch of like ImageNet or CIFAR 10 and I forward pass through the model and I get some loss on a back prop and I update the weights, I would go get a different batch for the next one.

但他们不是，他们就用这一批数据反复跑16遍。

But what they do instead is they actually do that 16 times.

这么反复搞下来，你会发现残差越来越小，越来越稳定。

And so and as you do that, you actually can see the change in your residuals get less and less and less.

这其实说得通，因为在RNN里，z_l和z_h是负责状态传递的——

And why it actually makes sense is because when in the RNN case, the z l and the z h, which are the carry, the task carry

隐藏状态。

Hidden states.

对。

Yeah.

隐藏状态。

Hidden states.

一开始全是零。

Start off at zeros.

就是零。

Those are zeros.

然后我们在这整个循环递归里走一遍，至少是那两个循环：TL 和 TH 这两步。

Then we go through this whole loopy recursion three the the the the at least the two loops, the two lower loops, t the the TL and TH steps.

接着我只反向传播一次，就经过那两个模块。

And then I back prop just through the two modules, just once.

我不用一直往回递归。

And I don't recurse all the way back.

做一下 stop grad。

Do a stop grad.

我就停在这儿。

I stop right there.

然后有个很大的残差，我就不重置 z l 和 z h 了。

And then there's a huge residual and then I don't reset z l and z h.

我会在携带状态或隐藏状态的另一个位置再做一遍。

I do it again at a different point in the carry or hidden Right.

变量空间。

Variable space.

所以实际上，每次都可以看作是不同的 batch，尽管输入的 x 完全一样。

And so one can actually look at it as like a different batch every time even though it's the same exact x's.

是的。

Yeah.

我理解的方式是，你递归的那16个步骤，其实是在构建一个 mini batch，只不过不是来自不同的输入，而是来自不同的记忆状态。

Like the way I kind of think about it is like the the 16 or whatever that you're recursing over, it's like constructing a mini batch not from different inputs, but from like different memory states basically.

其实就是跨越这个隐藏层或者传递状态的内存访问。

It's like across this hidden or carry memory access basically.

而且这个数学关系是成立的，确实管用。

And and that math holds and it works.

当 ZL 和 ZL 的变化量、ZH 的变化量都趋近于零的时候，这完全符合 DEQ 的设定，对。

It follows DEQ directly in the event that the ZL and the delta in ZL and the delta in ZH go to zero Mhmm.

但实际情况根本不是这样。

Which it actually just doesn't do.

我们待会儿会讲 TRM，但 Alexia 已经证明了，这根本站不住脚，这数学方法根本用不上。

And so we'll get to TRM, but Alexia basically shows that it's just not the case and you can't actually apply this math.

所以它能跑起来，原因就在这儿。

And that's why it's working.

但这还远远不够解释它为啥能跑起来。

That's not sufficient support for why it's working.

我们其实根本不知道它到底为啥能跑起来。

We actually don't know why it's really working.

她发现其实可以一直反向传播到深层递归里，这个我们待会儿讲TRM的时候再细说，而且这么做真的能大幅提升性能。

And she figures out that you actually can back prop through all the way to the deep recursion, which we're going get into TRM in a and that actually improves performance much, much more.

挺有意思的。

Interesting.

明白了。

Okay.

在聊TRM之前，关于这篇论文，我觉得大家对这个问题其实有好几种理解方式，对吧？就是他们怎么想到这个方法的，以及为什么这个方法有效或者无效。

So before we get into TRM, yeah, on this paper, I think there's a bunch of different ways people have looked at this, right, in terms of how they came up with it and then why this may or may not be working.

一种说法是生物合理性。

One, it's a sort of bio plausibility argument.

你知道的，我一般不太看重这种说法。

As you know, I'm usually not super keen on these.

机器学习历史上老有人拿生物合理性当起点，结果后来发现，真正效果更好的反而是那些明显违背生物合理性的变体。

Think machine learning tends to have a long history of people starting with bio plausible arguments and then realizing that there's some variant of them that seems highly bio implausible that actually works better.

你举个例子，对吧。

I think you have example Yeah.

对

A

经典啊，最早掀起这波深度学习热潮的论文就是AlexNet。

classic, the first deep learning paper that started this whole craziness is AlexNet.

AlexNet里头有个挺有意思的小设计，叫什么局部响应归一化或者抑制之类的，就是说一旦某个神经元激活了，它周围就进入一个‘不应期’之类的。

In AlexNet, there's actually this funny little thing called like local receptive activation or depression or something like that, where like once this activation fires then like I have this like you know a refractory region or something like that.

结果根本没用。

It actually doesn't work at all.

根本用不着这玩意儿，后来VGG一出来，直接说：全删了。

And like it didn't work and you didn't need that and then VGG came out and said get rid of all that.

就往深了做。

Just go deeper.

三层。

Three.

用三个三乘三的卷积堆起来，效果直接炸了。

And three by three comp and it actually just like outperforms dramatically.

对。

Right.

所以啊，你得这么搞，才能让论文被NeurIPS接收。

And so like this is like always the you need to do it to get accepted into NeurIPS

没错。

and Yeah.

完全对。

Totally.

就是这么回事。

Totally.

是啊。

Yeah.

你才是这方面的专家，那你觉得什么是可生物降解的，什么不是？

You're definitely the expert here, but what do you consider to be bioplasable and what's not?

我觉得很多机器学习的研究其实和神经科学领域的人干的活儿重合得挺多的。

Well, think that a lot of machine learning literature has overlapped a lot with people working in neuroscience.

我觉得我们天然就会好奇大脑是怎么工作的，毕竟大脑就像一台超级厉害的计算机，干的活儿多得惊人，效率还高得离谱。

I think it is very natural for us to ask questions about how does our brain work because our brain is like an incredible instrument that does a ton of computing obviously and does it in a very shockingly efficient manner it seems like.

所以长期以来，很多机器学习的研究都试图从人类思维中找灵感，想搞明白大脑是怎么运作的，然后把这些机制用到各种机器学习系统里。

And so a lot of machine learning resources for a long time sought analog from how we think to understand our brain to work and try to encode that in various machine learning systems.

比如神经网络这个概念，名字就这么起的，因为我们觉得它大概是模拟了神经元的基本运作方式；还有那些激活函数，也是想模仿生物神经的一些特性。

So from the very basic concept of what a neural network is, it's called a neural network because we think it's some basic model for what a neuron is, How certain activation functions work are meant to be inspired by certain biological

你觉得呢？

Do think

这个问题问得好。

that's good question?

其实吧，我们经常用生物可能性来激发灵感，但最后做出来的玩意儿，往往偏离了生物现实，跑到了一个更接近但其实不太可能在生物体内实现的方向——可偏偏效果还更好。

The thing about them is that often we use biopossibility to inspire us to come up with ideas, but we end up veering away from the biopossible to something adjacent to them that is likely bioimplausible, but that seems to work better.

就是那种在GPU上跑得飞快的。

Something that runs better on a GPU.

没错。

Exactly.

在GPU上跑起来更快。

It runs better on a GPU.

它在某些方面更高效，而这些效率恰恰跟我们实际在计算系统里实现它的方式有关。

It's more efficient in some capacity that is relevant to how we actually encode it in a computational system.

我觉得琢磨生物可能性挺有意思，确实能激发我们去想一些新点子。

So I find thinking about bioplasibility fun and interesting and it's definitely a great way to inspire us to think about new things.

但我自己在选机器学习方向的时候，一般不会被生物可能性捆住手脚——除非它只是个引人入胜的科学起点，能带我们深入探索。

But I tend to not be bounded by biopossibility when I think about what machine learning systems we should prioritize working on or think are as particularly exciting other than as an interesting scientific launching point for a deeper exploration.

我觉得更让我着迷的，其实是之前咱们聊过的自动机理论那一块。

I think the version of this that I find more compelling is actually that original discussion we were having around automata theory basically.

说白了，就是基础的数据结构和算法理论：当你跑一个复杂的算法时，有个内存缓存真的特别有用，能让你跑得更高效。

And honestly just actually like fundamental data structures and algorithms theory which is that if you're running a complex algorithm having access to sort of a memory cache is actually very useful for being able to run that algorithm efficiently.

我总觉得这些隐藏状态或者记忆机制，就像图灵机的纸带，或者基数排序里的内存池，能让模型在一次前向传播里，聪明地使用这个缓存，从而实现原本需要更复杂操作才能达成的高效计算。

And I kind of think of this set of hidden states or carry as akin to a Turing machine tape or akin to the radix sort memory bank where you can basically train a model to use this memory cache in an intelligent way in a single forward pass so that you can get a more efficient time operation that would otherwise require some sort of more complicated

推理。

reasoning.

是的。

Yeah.

我想之前想说的是，我们搞COT和工具使用，其实是为了突破GPT-2的局限。

Think the a point I wanted to make earlier is that like we did this COT stuff and this tool use thing as ways to get beyond the the the the of GPT two.

所以我们是怎么做到的呢？我其实做过这个实验。

And so the way that we get We You can actually I've done this experiment.

如果你给我无限多的无序列表和有序列表，只要我能做链式思考，能教它每一步怎么走，那它在测试时真的就能实现排序，变成一台图灵机。

You can actually If you give me infinite amounts of unsorted list and sorted list, if I can do chain of thought and I can do every single step and teach it to do every single step, then I can actually get it to do do sort and become a Turing machine at test time.

同样，还有一个更简单、更容易实现的办法：你告诉它，嘿，Python里有个叫sort的函数。

And similarly, an even cheaper one that is much easier to do is you teach it and you say, hey, there's this Python function called sort.

直接调用这个函数就行。

Just call the function.

直接调用这个函数就行。

Just call the function.

这可是最省事的办法，根本不需要反向传播。

And like, that's the easiest thing to do and you don't need back prop at all.

所以这两种就是窍门。

And so those are the two hacks.

现在，弗朗索瓦，这问题解决了。

Now, well Francois, this is solved.

咱们完事儿了。

Like, we're done.

对吧？

Right?

不对。

No.

因为我得知道什么是排序。

Because I needed to know what sort was.

如果我们根本不知道归并排序该怎么办？

What happens if we didn't know what The merge

链式思考不可能从零开始自己发现排序算法。

chain of thought is not going to inherently discover sorting from first principles.

它是从模型训练时所学的所有历史知识中找出来的。

It's finding it from our historical knowledge of everything it's trained on.

对。

Yeah.

我的意思是，那些演示里头都提到过，终极考验其实是‘爱因斯坦测试’。

Mean, this is like the the demos had this whole thing about like the ultimate test is the Einstein test.

就是回到1911年，看它能不能从头把现代物理重新推导一遍。

Like go back to 1911 and then like have it rebuild all the physics up until now.

同样，咱们就假设我们只知道冒泡排序。

Similarly, let's just pretend that we only had bubble sort.

别的排序方法一概不知道。

We knew other no other sort system.

如果你只用冒泡排序的输入输出来训练它，它就只会冒泡排序。

If you chain of thought it on all the bubble sort input and output, it will only do bubble sort.

而且连冒泡排序都未必能做好。

In fact, won't even do bubble sort that well.

所以这是最理想的情况，工具只能知道冒泡排序。

Like so this is the best situation and then the tool use, of course, it can only know bubble sort.

我想达到归并排序的水平。

I want to get to merge sort.

那我该怎么发现归并排序呢？

How do I discover merge sort?

我想强调一下，这点可能之前没说清楚：其实LLM里早就有一种人们熟悉的递归方式，就是我们之前说的思维链。

And I think the interesting thing just to emphasize here because it may not have been extremely clear is there already exists some type of recursion that people are used to in LLMs which is chain of thought we mentioned earlier.

但这种递归发生在模型输出的词元层面，并不是模型本身固有的能力。

But that is a recursion that's happening in the token space of the model's outputs, not inherent to the model itself.

这正是根本的限制——模型只能做前馈式的单次输出，我们只是靠一个‘作弊’手段：不断让它输出，再让它读自己的输出，假装能做出点聪明的事。

And that's sort of the fundamental limitation is that the model can only do a feed forward one shot output and then we basically just have this hack that if you keep letting it output things then it can read its outputs and do somewhat intelligent seeming things with it.

但它的能力似乎被我们喂给它的数据上限卡死了，现在各大实验室都在疯狂抢购数据，对吧。

But it seems to sort of be upper bounded by the data that we feed it that, you know, the labs are very hungrily buying right now Yeah.

而不是那种真正内在的、递归式的推理能力。

And not this sort of like inherent underlying recursive reasoning.

是的。

Yeah.

所以不管是用思维链还是工具调用，这两种方法都受限于人类已有的知识范围。

So in both cases, both hacks to solve this in COT and tool use, you're bounded by the bounds of human knowledge.

一旦问题超出了人类知识的边界，那你就真没辙了。

In the event it's outside the set of human knowledge, then like you're kinda SOL.

所以这是一点。

And so that's that's one.

你提到离散空间和潜在空间的区别，这点特别到位。

The other you make a great point about discrete versus latent space.

在离散空间里做推理，LLM 只能输出一个个具体的词，就像进位一样，必须强行映射回离散的词元空间。

Reasoning in a discrete it can only output the carry in the case of LLMs has to be snapped back to some discrete token space.

而像 RNN 这类模型，它们一直保持在连续的潜在空间里，维度要高得多。

And in the case of RNNs in general, they remain in this continuous latent space which is much higher dimensional.

假如给我一条这么长的带子，你把它切成十个格子，跟让它自由发挥所有可能的值相比——

If you give me like a tape that's this long and you cut it up into 10 buckets like versus all the possible values

对。

Right.

没错。

Exactly.

是啊。

Yeah.

连续空间的表现力强得多。

It's much more expressive to being continuous space.

但我们没法这么训练，主要是因为反向传播在时间上的限制。

But we can't train it that way because we actually, you know, because you're inhibited by back prop through time largely.

所以这篇论文才这么让人兴奋。

And this is why this paper is so exciting.

明白了。

Okay.

那在我们转到TRM论文之前，先总结一下。

So before we then go over to the TRM paper, let's just summarize here.

在我们转到TRM论文之前，从HRM论文里最值得我们记住的关键点是什么？

What matters most from the HRM paper that we should take away before we transition and contrast them with the TRM paper?

对。

Yeah.

我觉得最重要的一点就是这个外层优化循环。

I think that the the number one piece to take away is this outer refinement loop.

这个外层优化循环是可以扩展的，而且有非常清晰的分析。

The outer refinement loop scales and there's a a great breakdown.

basically，Sapient团队——得给这篇论文点个赞，因为里面创新太多了——其实并没有对每个输入都做详细的规模消融实验。

Basically, the the Sapient authors, which is huge kudos for this paper because there's so many innovations in this paper, didn't really do like scaling ablations on every single one of the inputs.

但Constantine，他在François Chaubard公司那边，就在印度，真的做了这项工作。

But this guy Constantine at Francois Chaubard's company, India, actually did.

好。

Mhmm.

他做了一个特别棒的分析，发在YouTube上，你去看看就知道了。

And it's this amazing breakdown that he put on posted on YouTube that you can go check out.

但说到底，最关键的一点是，外部精炼循环才是这些东西效果这么好的核心原因。Alexia 其实也独立发现了这一点，她把它放大了，证明你可以把其他那些乱七八糟的东西全砍掉。

But basically, main takeaway is that the outer refinement loops is the main beneficiary is the main reason why these things work so well, which Alexia basically takes the she found, I think, in parallel and scales up and shows that you can get rid of a lot of all those other stuff.

明白了。

Okay.

就像很多机器学习领域那样，后续的论文基本上就是把第一篇论文删掉75%，就像我们以前视频里常做的那样，只留下最核心的魔法。

Like a lot of machine learning, the follow on paper is basically delete 75% of the first paper as we've often done in videos here and keep the magic basically.

对。

Yeah.

所以，好吧。

So okay.

那这个魔法到底是什么？

So what's the magic then?

在TRM论文里，真正关键的是哪一部分？

Like what's the part that actually matters in terms of what stays in the TRM paper?

咱们现在来对比一下这两篇论文在架构上的核心差异。

And let's now contrast the core architectural differences between these two papers.

是的。

Yeah.

我觉得如果拆成两个主要部分，这个外部精炼循环确实非常有效，效果特别好。

So I think that I guess if I break it down into two major things, this outer refinement loop thing is really great and works really well.

还有这个截断的随时间反向传播，其实就是普通的随时间反向传播，只不过我在某个较早的时间点就截断了。

And that this like truncated back prop through time, which is back prop through time except I truncate at some time Some earlier point.

较早的时间点，对。

Earlier point Yeah.

叫T-back。

Called t, t back.

T等于1就已经完全够用了。

T equals one is actually is completely sufficient.

所以截断到T等于1的随时间反向传播，完全够用。

And so truncated back wrap to time t equals one, completely sufficient.

这听起来特别反直觉。

And that's very counterintuitive.

这正是HRM发现的。

Which is what HRM found.

HRM是这么做的，而TRM更进一步，不是只走一次h网和l网的调用，

Which way HRM found and TRM does a little bit further rather than going through just one call to the h net and the l net.

它实际上完成了一整个递归循环。

It actually goes through one full recursion loop.

比如我做16次，其实只要回溯一次就够了，效果就差不多了。

So if I do it 16 times, I just go back through one time and that is is is kind of sufficient.

如果你用这种伪不动点迭代的方法，每一步都用梯度去修正，奇怪的是它真能跑通，而且在携带空间上批量处理也没问题。

And if you do it with this like fixed point iteration thing, pseudo fixed point iteration thing where you keep hitting it with a gradient at every single step, It like weirdly works and this batch size across the carry space Right.

真的能行。

Like actually works.

所以这部分，这两个模型都保留了。

So that part is also kept between these two models.

对。

Yeah.

另一个变化好像是，以前有这种高低层次思维的双层结构，现在没了。

It seemed like another thing that changed was having these this sort of double layer of like you know higher order thinking and lower order thinking.

看起来现在被简化成单一层次了。

It seems like it collapsed that down into just a single one.

这背后的直觉是什么？TRM论文里是怎么实现的？

What's the intuition there and how does that actually work in the TRM paper?

对。

Yeah.

挺有意思的。

So it's interesting.

她其实做了个消融实验，对比了用两个独立网络和只用一个网络的效果。

She actually ablates having two separate networks versus just having one.

我觉得更关键的是变量作用域：低层特征和高层特征都该有，但用同一个网络就够了。

I guess the more important space is the variable scope is that you should have low level features and high level features but the same network.

所以表现最好的是

And so the best performing