Eric Jang – 从零构建AlphaGo

如果你有一台超强的电脑，根本不需要概率。

If you had a very powerful computer, there is no probabilities.

你直接就能算出每个动作的真实平均价值。

You just you can just compute the true average of what the the mean action value is.

那概率是从哪儿来的呢？

So where does the probability come in?

其实，在AlphaGo之前的计算机围棋里，我们一直用的是蒙特卡洛方法——就是从随机选取的博弈树里取期望Q值的平均。

Well, it turns out that as in, you know, Computer Go before AlphaGo, we've always done some sort of Monte Carlo method where we have some we we take the expected q value averaged over a randomly selected tree.

而这个随机选取的博弈树，就是概率的来源。

And that randomly selected tree is where probabilities come in.

所以，Q值的含义就是：在某种随机搜索过程所形成的随机分布下，某个动作的期望价值是多少？

So, the interpretation of Q is what is the expected action value under the random distribution induced by some random search process?

这样理解就对了。

Makes sense.

那这个随机搜索过程又是从哪儿来的呢？

And so where does the random search process come in?

这时候，动作的概率就派上用场了。

That's where, you know, p of action comes in.

对。

Yeah.

如果我们假设一个非常简单的算法，每个合法动作被选中的概率都一样，那每个动作的概率就是合法步数的倒数。

So if we assume a very naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over the number of valid moves in this setup.

然后你就相当于在这样一个非常分散的搜索树上做平均。

And you would be kind of taking this average over this very diffuse tree.

对吧？

Right?

这种做法虽然可行，但效率很低，因为你得评估大量价值很低的分支。

And this is a valid interval you can take, but it's very slow because you're going to consider a lot of trees that have very low value.

这其实有点像重要性采样问题——真正能带来高价值的，只有少数几个动作和路径，其他几乎全是低价值的。

And it's essentially almost like a importance sampling problem where you want to there's only a few actions and and and sort of paths that can contribute, you know, high value, and almost everything else is low value.

所以这个问题确实挺棘手的。

So so this is sort of a tricky problem here.

好的。

Okay.

这就是选择下一步行动的依据，决定你往哪个方向深入搜索。

So this is the action selection criteria for how you decide which moves to move down.

当你在树搜索中不断往下走时，总会遇到一个节点，胜负已经非常明显了。

Now, as you move down in in Tree Search, you will eventually run into a node where it's quite clear you've won or lost.

对吧？

Right?

到了游戏的最后阶段，当所有合法走法都用完了，按照特朗普-泰勒计分规则，你就能判断自己是赢了还是输了。

At the at the very, very end of the game when when there are no valid moves to play left under under Trump Taylor scoring, you can decide whether you, like, you know, won or lost.

对吧？

Right?

所以你不是赢就是输。

So you you either win or you lost.

这其实就是整场游戏最终的得分结果。

And so this is basically, you know, the the final return of the whole game.

对吧？

Right?

所以这里的问题是，我们可以给树的终端叶子节点赋予一个值 u，

And so the the question here is, like, we we can assign a a value, u, to a terminal leaf node of the tree.

但那些更上层的父节点，它们的值该怎么定呢？

But how do we assign the values for nodes prior to that, the parents?

其实很简单，你只需要把动作值取平均就行了，

And it turns out, you know, what you simply do is you just take the your mean action value is essentially your average.

假设这些全是叶子节点，

So let's suppose these were leaf nodes.

抱歉，

Sorry.

这些全都是叶子节点，

These were all leaf nodes.

那么这个节点——也就是这个动作——的平均动作值，就是它下面所有叶子节点胜败结果的平均值。

The the mean action value of this node, you know, this action here, is just the average of whether you won or lost at the leaf nodes.

相应地，你可以沿着这条链往上推，比如这个节点的平均动作值，我们叫它 q_b，对应动作 b，其实就是这些子节点的加权平均值。

And correspondingly, you can kind of walk up the chain and say, like, well, the mean action value of this node, let's call this like q b, and this is action b, is just the average of a weighted average of these ones here.

对。

Yep.

没错。

Right.

这个加权平均可能取决于你使用的采样分布是否不同，但基本思路是：先解决那些结果确定的输赢局面，然后反过来推算。

And and the weighted average is it could be dependent on if you have a different sampling distribution or not, but the the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can kind of go backwards.

这叫回溯步骤，就是根据最终终端节点的平均值，给这些节点或动作赋值。

This is called the backup step and assign values to these these these these nodes or actions corresponding to the averaged over over the final terminal.

是的。

Yeah.

明白了。

Okay.

所以，如果你不用神经网络，光靠这种方法，计算量还是大得无法处理。

So if you were to do this without neural networks, it would still be intractable.

你很难确定该采样哪些动作。

You would you would have a trouble finding, you know, which actions to sample.

很多动作带来的价值都很低，尤其是当你处于劣势，想翻盘的时候，只有少数几个动作才有高价值。

A lot of the actions will contribute very low value, especially if you're like, you know, trying to fight your way out of a losing position and only a few actions give you high value.

所以实际搜索的开销依然非常大。

So the search in practice is still very, very expensive.

但关键是，因为围棋具有树状结构，只要你搜索得足够深，并且下游节点的估值都准确，你其实可以对当前节点的价值做出非常精准的估计。

But but the the idea is that like if you can because Go follows a tree structure, you can actually, you know, inform a very good estimate of the value of this node based on the values of downstream, assuming they're all correct and assuming you've searched deep enough.

对。

Mhmm.

你之前提到的那种局面——人类一眼就能看出谁占优势，但算法却看不出来。

Your explanation earlier about the the sorts of states where it's obvious to a human who's gonna win, but it's not obvious to Right.

或者即使胜负已定，你还是得把棋下完。

Or, like, you deterministically, you still had to play it out Mhmm.

这真正让我明白了，为什么价值函数不仅可训练，而且是有效学会这个游戏的关键。

Actually drove home the intuition of why the value function both is trainable and two, why it's necessary in order to actually be able to learn this game effectively.

也许我们首先该明确一下什么是价值。

And maybe it's worth defining value in the first place.

听起来不错。

But Sounds good.

对。

Yeah.

是的。

Yeah.

我们之前聊过，这个 u 值其实就是你最终判断输赢的结果。

So we we talked about, you know, this u value being, you know, your final resolution of whether you won or lost.

这是一种终端叶子节点的情况。

And this is a terminal leaf node condition.

但人类下棋并不会一直走到树的最末端、走到叶子节点。

Now humans don't play all the way to the sort of edges of the the tree the leaves of the tree.

对吧？

Right?

他们通常会在几十步之前就停下来了，高水平对局中甚至可能在一百步之前就结束了。

They kind of stop, you know, some dozens of moves before, maybe maybe even 100 moves before in in sort of high level play.

那他们是怎么知道的呢？

So how do they know?

对吧？

Right?

你可以把人类想象成自带一个隐式的神经网络，也就是价值函数，它接收棋盘状态，然后大致判断出输赢前景。

Like, you can think about humans as implicitly having a neural network called a value function that basically, you know, takes in a board state and then it kind of evaluates, you know, key win.

所以人类扫一眼棋盘，就能感觉到：我这局大概要输了。

And so the human glances at the board and they know like, I'm probably going to lose.

对吧？

Right?

他们其实是在运行一个神经网络，快速评估棋盘局面，隐式地模拟了大量可能的对局路径，算出平均胜率，再决定这盘棋有没有胜算，该认输还是继续下。

And and they're essentially running a neural network that looks at a board and implicitly, they are amortizing a huge number of possible game playouts and and taking that average and then deciding whether the board is winnable or not and then whether they should concede or or, you know, keep playing or not.

这真的很了不起。

And this is remarkable.

如果你想想这种事的美妙之处，人类的大脑就像一个神经网络，能一眼看下去，几秒钟内就凭积累的经验和直觉，把所有可能的走法都算明白了，根本不用一步一歩地逻辑推演。

If you think about, like, the the beauty of something like this, it's like a a neural network in a in a human can somehow do all of this simulation at a glance and then just know like within a few seconds without actually playing every single game logically based on just kind of like crystallized knowledge and experience that, like, they can do this.

这就给我们一个提示：像围棋这样的游戏，其实有办法大幅加快搜索过程。

And so this gives us a hint that, like, in games like Go, there are ways to basically radically speed up the search process.

这也是AlphaGo之所以能成功的核心直觉之一——你可以训练一个价值函数，让它一看棋盘，就能快速判断胜负，而不用把每一条分支都深挖到底。

And this is one of the fundamental intuitions behind why AlphaGo works, is that you can train a value function to look at a board and quickly resolve the game without playing out all of these trees into the you know, into a very deep search depth.

是的。

Yep.

说得通。

Makes sense.

我想跟观众说一下，我之前准备节目时，想解释AlphaGo是怎么工作的，发现特别难讲，特别让人糊涂。

I will say for the audience, I sort of found for previous episodes when I was prepping and it would seem somewhat relevant to understand how AlphaGo works, I would find it very, very confusing.

但一旦你从这个角度理解了问题，再一步步把后面的内容补上，就会发现其实没那么难，反而特别清晰。

And but it's the kind of thing where once you understand the problem in this way and then you'll build the next few pieces, it is actually much more understandable and it will make a lot of sense.

你现在觉得困惑很正常，但等听完这节课，你会发现比你想象的要容易懂得多。

And it's okay to be confused right now, but it's it's probably simpler to understand by the end of this lecture than you anticipate.

是的。

So Mhmm.

对。

Yeah.

我给听众做个备注。

I'll just make that note for the audience.

对。

Yeah.

从宏观角度来看，关键的直觉是，过去下围棋这类游戏，人们会构建搜索树，但我们的计算机还不足以支撑这种做法。

The important intuition at a high level just to, you know, step back about where we're going with all this is that classically, for games like Go, you could build a tree, but we don't have computers powerful enough for that.

对。

Yeah.

而且，估算每一个可能动作的价值也很困难，因为不到游戏结束，你根本不知道结果如何。

And estimating the value of every action that you could possibly take is also hard because you don't know until the end of the game.

是的。

Yep.

你可以通过下到终局来取平均值，但这也很难，因为你不知道该选哪些动作来采样这些平均值。

You could take averages by playing them to the end, but that's also hard because you don't know which actions to take to sample these averages.

所以从概念上讲，这其实有两个问题。

So conceptually, there's kind of two problems.

一个是树的宽度，另一个是树的深度。

There's the breadth of the tree, and then there's the depth of the tree.

而AlphaGo提供了一种方法，能把这两者都大幅缩小，变得容易处理。

And AlphaGo gives us a way to basically shrink both of those to be very attractive.

是的。

Yeah.

这基本上就是它的核心思路。

That's that's essentially the kind of core idea behind it.

好的。

Okay.

所以我们现在借鉴一个想法：人类看一眼棋盘，就能立刻判断出谁会赢。

So we we take this idea that like, know, humans can glance at a board and instantly predict whether we win.

这或许让我们有机会大幅减少搜索的深度。

And maybe that gives us the opportunity to really truncate the how deep we we search.

对。

Yeah.

而且我们都知道，人类看一眼棋盘，就能直觉地判断出哪些落子位置可能比较好。

And then, know, we also know that humans can look at a board and and decide, you know, what what boards, you know, like intuitively at a glance, what moves might be good on the board.

没错。

Right?

所以，这两点正是我们可以用深度神经网络来加速搜索的关键。

So so these are kind of two things that we can use deep neural networks for to accelerate this search process.

我们回退一下。

Let's go back.

在谈神经网络之前，先回头看看这个对弈过程是怎么运作的。

Before we talk about neural nets, let's just go back to how this play out works.

到目前为止，我们只讨论了走一步棋。

So we've only talked about making one move.

对吧？

Right?

所以AI会看这个编码后的围棋棋盘。

So so the AI looks at this encoded Go board.

它有一个搜索树。

It has a tree.

它会深入这棵树里，找出哪些走法可能是最好的。

It searches for you know, deeply into the tree to find out which of its actions might be the best.

然后它就下出那一步。

And then it takes that action.

接着，它又轮到人类了。

And then now, you know, it goes back to the human.

这时候，人类看到的棋盘可能就像这样。

So maybe now the human sees a Go board that looks like, you know, like this.

然后他们就下自己的那一步。

And and then they they make their move.

所以也许他们把棋子放在这里。

So maybe they put they put their stone here.

然后AI再次查看这个新的编码棋盘。

And then now we we go back to the AI, which now looks at a new encoded board.

我用2表示AI执白，1表示人类执黑，0表示空位。

So I've used two to denote the AIs playing as white and one to denote the human playing as black and zero as empty.

轮到AI时，它会从头开始重新进行MCTS树搜索。

And then now on the AI's turn, it does the MCTS tree search all over again from scratch.

对吧？

Right?

它会丢掉上一轮搜索过的旧树，现在有了一个新的根节点，然后重新开始搜索。

So so it throws away this old tree that it searched last round, and now there's a new root node, and it begins to search anew.

如此反复。

And then so and so forth.

MCTS本质上是一种搜索算法，借助神经网络来决定最佳落子位置。

So, MCTS is basically a you can think about it like a search algorithm that is deciding what moves to play best, aided by neural networks.

每走一步都会重新执行一次。

And and it's it's done on every every move.

明白了。

Okay.

很好。

Great.

那我们来聊聊神经网络这部分。

So let's talk about the neural network part of this.

在你下棋的时候，我还有一件事需要搞清楚，就是这个MCTS的数据结构，包含节点和子节点这些部分。

And while while you're racing, another sort of thing that was important for me to understand was this MCTS data structure with nodes and childrens of nodes and whatever.

每走一步都会重新构建一次，一旦有人落子，就会重新初始化。

This is done per move and reinstantiated once a move is made.

人类下完一步后，AI就会分析当前局面，通过大量模拟来判断下一步该走哪里。

So a human makes a move, then the AI looks at this and is trying to basically run a bunch of simulations to figure out, okay, what should make move should I make next?

这些模拟其实就是探索MCTS树中的下一个节点。

And those simulations just a simulation is basically like exploring one more node in this MCTS three.

最后，当你跑完这一千次模拟后，这些数据就会告诉你下一步该走哪一步的概率，这部分内容你会保存下来。

And at the end, once all these once all this you know, you run a thousand simulations that informs then this, I guess, you'll explain this probability of what move to make next, that's what you store.

你会根据这些概率来选择最优的走法。

You you sort of choose the best move given those probabilities.

然后把所有这些数据都丢掉，等对方走完下一步，你再从头开始这个过程，每走一步都重新来过。

You discard all of that, then the next player makes a move, and you restart this process at the beginning of every move.

对。

Correct.

不过有一点要补充，不是所有东西都丢掉。

The one small addendum, you don't discard all of that.

你会保留其中一项，后面还会用到。

You keep one thing behind that we'll use later.

是的。

Yeah.

没错。

Yeah.

就像我之前为莱纳做的那样，我想为这一期内容制作一些闪卡，帮助大家更好地记住这些概念。

Just like I did for Rainer, I wanted to make flashcards for this episode so that people could retain these concepts.

理想情况下，大模型能帮我生成一些初步的卡片，供我后续优化。

And ideally, an LLM could generate some candidates for me to then refine.

但要获得高质量的建议，我得设计一套完整的流程：让AI能接收黑板在特定时间点的截图，生成SVG图表（如果视觉辅助有帮助），再让一个评审模块对文字和绘图进行评估，最后根据反馈修改卡片内容。

But to actually get high quality suggestions, I needed to design a whole pipeline where the AI could take and ingest screenshots of the Blackboard at the right timestamps and then make SVG diagrams in case visuals were helpful and then run their writing and drawing through a critic and then revise the card in response to this feedback.

光靠反复调用大模型，根本没法实现这种效果。

It's very hard to accomplish this just by sacking LLM calls.

如果你有一个能贯穿整个流程、持续参与每个环节的智能代理，这种分步骤的方法效果要好得多。

This sort of step by step recipe works much better if you have a durable agent that's been engaging with the task across all the previous stages.

所以我用Cursor SDK为每张卡片都创建了一个独立的代理。

So I used the Cursor SDK to spin up an agent for each card.

Cursor的框架省了我不少功夫，不用自己设计上下文模板，也不用琢磨怎么调用工具来截屏或制作动画。

The Cursor hardness saved me a bunch of work in designing some custom context scaffold or figuring out how to design tool calls for taking screenshots or making animations.

这些代理都在云端运行，我不用担心得一直开着笔记本电脑。

These agents all run-in the cloud, so I don't have to worry about leaving my laptop open.

我收到邮件时，就意味着有候选卡片可以审阅了。

I just get an email when I have candidates to review.

你可以去 flashcards.thwarkash.com 看看我的卡片。

You can check out my cards at flashcards.thwarkash.com.

想用代理 SDK 开始构建的话，访问 cursor.com/thwarkash 就行。

You can start building with the agents SDK at cursor.com/thwarkash.

明白了。

Okay.

现在我们对搜索如何生成走法有了基本的理解。

So now we have a basic intuition of how moves are made with search.

接下来我们要聊一聊，神经网络怎么通过模拟人类直觉来加速这个过程。

We're going to talk about how neural networks can speed this up by providing an analog to like the human intuition.

这里涉及两个网络。

So there's two networks.

一个是价值网络，它接收当前局面，然后预测：我最终是赢还是输？

There is the value network, which takes in a state and it predicts, you know, am I going to win or lose?

这是一个二分类问题。

It's a binary classification problem.

然后我们会有一个策略网络，用来生成可行动作的概率分布。

Then we're going to have a policy network which induces a distribution over good actions to take.

对。

Mhmm.

我这里画的是一个一维的扁平化动作分布，但实际上它更像是一个方格网格。

So I'm gonna draw a one dimensional flattened move distribution, but this is really like, you know, a square kind of grid.

对吧？

Right?

所以它可能会认为，这些才是更有潜力的动作，对应的概率分布也在这里。

So so maybe, like, it thinks actions are like these are the kind of probability distribution over good actions.

这两个都是分类问题。

And both of these are categorical classification problems.

对吧？

Right?

所以你可以像训练普通分类器一样用深度学习来训练它，比如用交叉熵损失之类的。

So you can train this like any classifier in with deep learning, you know, cross entropy loss, that kind of stuff.

具体的架构其实没那么重要。

So the the specific architecture does not actually matter too much.

我试过几种不同的架构。

I I tried a few different architectures.

Transformer 是有效的。

Transformers work.

ResNets 也有效。

Resonets work.

在数据量小的情况下，我的经验是 ResNets 通常比 Transformer 表现更好，性价比更高，不过这也不一定对。

For small data regimes, my experience is that Resonets still kind of outperform transformers and and kind of give you more bang for the buck at at lower budgets, but this may not be true.

为什么会这样？

Why is that?

因为它们自带局部卷积的归纳偏置。

They they provide the inductive bias of, like, local convolutions.

是的

Yeah.

一般来说，当你需要更全局的上下文时，Transformer的效果就会超过残差卷积网络。

And generally, transformers start to outperform residual convolutional networks when you want more global context.

明白了。

I see.

好的。

Okay.

《Katago》论文中有一个有趣的发现：他们发现，把全局特征在整条网络中不断融合和聚合，能帮助网络更好地理解棋盘两端之间的价值关联。

So one interesting finding from the Katago paper was that they found it actually quite useful to pool together global features together and aggregate global features like throughout the network to kind of give the network a global sense of how to like connect value from one side of the board to another side of the board.

那‘聚合全局特征’具体是什么意思？

Well, what does it mean to aggregate global features?

是的

Yeah.

比如，你面对一个19乘19的围棋棋盘。

So if you have a go a very large 19 by 19 Go board.

是的。

Yeah.

你看，这边在打仗，那边也在打仗。

And you you know, you've got some some sort of battles going on here, and you got some battles going on here.

当你把这些输入到卷积神经网络时，没错。

When you pass this through a convolutional neural network Yep.

卷积网络的感受野擅长处理局部信息，并保持其不变性。

The receptive fields of the convolutional network are going to be good at computing local things and making that invariant.

但它们很难把这两个特征联系起来。

But they won't be able to kind of connect these two features easily.

对吧？

Right?

它们需要被汇总起来，相互关注。

They need to sort of be pooled together and attend to each other somehow.

所以，人们认为Transformer在计算机视觉任务中表现好，比如视觉Transformer，就是因为它们能全局关注整个图像，更容易建立这些关联。

So, the argument about why transformers are good for computer vision tasks like with vision transformers and so forth is that because they have sort of global attention across the whole thing, they can more easily draw these connections.

但你需要更多的数据，才能通过数据学习到那些局部不变的特征。

But you do need more data there so that you can kind of learn through data the the sort of invariant local local features.

明白了。

Got it.

我曾竭尽全力想让Transformer在这类问题上发挥作用，因为我很好奇，它是否能在围棋领域带来突破，从而淘汰掉那些繁琐的技巧。

I've tried very hard to make transformers work for this problem because I was kind of curious if transformers would present some sort of breakthrough in Go and just remove a lot of those tricks.

但不管我怎么尝试，目前还是没能找到让Transformer比ResNet表现更好的方法。

But to try as I might, I actually haven't figured out a way to make transformers better than ResNet for for now.

所以，抱歉，

So one sorry.

再问一个相关的问题。

One more tangential question.

能理解为什么Transformer凭借全局信息整合，在需要考虑非空间邻近信息时更有优势。

It it makes sense why transport enmers with their, like, global pooling of information would be better if you need to consider information that is not just spatially yeah.

CNN则天然倾向于认为邻近的元素更重要。

CNNs give you a sort of bias that the things that are next to you are especially relevant.

然后它们会被逐步汇总起来。

And then they're sort of aggregated up.

是的。

Yeah.

没错。

Exactly.

对。

Yes.

但假设一下，

But suppose okay.

对于那些局部情况没那么重要的游戏，你得从整体上来考虑。

So for games where it it isn't that relevant, what is happening locally, you just kind of have to consider the whole thing.

对。

Mhmm.

你是说Transformer会更有效。

You're saying transformers would work better.

那像这种特别涉及空间维度的游戏呢？

How about games where so in particular, talk about the spatial dimension.

再来看时间维度，我们现在只考虑上一步，因为这是确定性、完全信息的游戏；但如果是扑克或者《外交》这类游戏，对方很久以前的一个虚张声势，其实对理解当前局势、决定下一步行动至关重要，这时候你就得把所有历史状态都考虑进去。

How about the temporal dimension where right now we're only considering the previous move because it is a deterministic full information game where but what if it was something like poker or diplomacy where really a bluff they made a while back is sort of relevant to understanding now and isolating to decide to make your next movements, you need to consider all those previous states.

这会不会改变我们对哪种归纳偏置和哪种架构更合适的看法？

Would that that change the consideration of what inductive bias is most relevant and what architecture is most relevant?

没错。

Right.

好问题。

Great question.

围棋就是一种完全信息游戏。

So Go is a perfect information game.

是的。

Yeah.

在完全信息游戏中，确实存在一个纳什均衡策略，采用它就不会比其他策略更差。

And in perfect information games, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.

如果你知道对手有某种特定倾向，比如喜欢 aggressive 下法，理论上你完全可以针对这种策略做出更有效的应对，而不是单纯依赖纳什均衡策略。

So, if you know that your opponent has a particular bias, like they love to play aggressively, you can actually, in principle, counter that specific strategy better than a Nash Equilibrium policy.

但要应对任何一种具体策略，确实存在一个仅凭当前棋局状态就能确定的纳什均衡。

But to counter any given strategy, there does exist a single Nash equilibrium that can be decided solely using the current state.

所以，像 AlphaGo 这样的围棋程序选择的设计方案，事后证明非常成功，因为纳什均衡的水平已经超出了人类能力。

So, is a design choice that most Go agents AlphaGo chose to do, which in hindsight turned out to work very well because the Nash equilibrium seems to be superhuman.

根本没有任何人类的下法能真正赢过它。

Like like no human strategy seems to be able to beat it.

当然，也有一些变体情况，这时候你就必须考虑历史时间维度的信息了。

Now, there are variations of this where you would actually need to consider temporal history.

这其实是一个非常有前景的研究方向，我鼓励大家去 fork 我的代码库，亲自试试看——比如说，玩双人对双人的围棋时，你就得建模你队友的行为。

So and and this is a very exciting research area that I I would encourage people to kind of fork my repo and try these things out, which is if you were to play, let's say, two v two Go, then you actually need to model your partner's behavior.

你可能根本不清楚他平时怎么下棋，那就得从过往行为中汇总信息，才能做出合适的回应。

And you like, you may not have information on how they play, so you need to aggregate some information on, like, how they play so that you can respond accordingly.

对。

Yeah.

对吧？

Right?

比如有些情况就不再是完全信息博弈了。

Like, are situations where it's no longer a perfect information game.

是的。

Yeah.

在这些不完全信息或部分可观测的游戏中，确实需要借助上下文来建立模型。

And then in those cases in in games of imperfect information or partial observability, then you do need some context to build a model.

是的。

Yeah.

没错。

Yeah.

我觉得在自我对弈或类似外交策略这类领域，这里特别有意思。

And and I think that's a place where things get very, very exciting in terms of, like, self play or diplomacy style.

是的。

Yeah.

有意思。

Interesting.

好的。

Okay.

我们再回到神经网络的架构，其实它没那么关键。

So returning back to the neural network, the architecture, again, is not super important.

用Transformer也能跑起来。

You can get it to work with transformers.

用ResNets也行。

You can get it to work with ResNets.

我发现，在预算有限的实验里，ResNets表现稍微好一点。

I found that for low budget experiments, ResNets work a little better.

你也可以用类似Carpathi那种自动搜索超参数的方法，把架构调得挺不错。

You can also use kind of a Carpathi style auto research hyperparameter tuning to make make your architecture pretty good.

所以你没必要在这上面太纠结。

And so so you don't have to worry too much about that.

你只需要把问题设置好，让它有一个明确的目标优化方向。

You just need to sort of set up the problem so that you have a a sort of target optimization.

对。

Yeah.

好。

Okay.

所以我们就选一个稍微随意点的架构，这个架构在我之前做的实验里效果还不错。

So we're going to pick just a somewhat arbitrary architecture that worked for for, you know, what I did.

不过再说一次，这部分其实没那么重要。

But again, this part is not super important.

你手里会有编码后的棋盘状态。

You have your encoded board state.

咱们就定个三吧，就像 RGB 那样，搞三个通道。

And we're going to just choose to, let's say, do three three like, you know, similar to an RGB, we're gonna have three kind of channels.

一个通道用来编码黑子，一个通道编码白子，剩下的那个通道嘛，可以编码空位，或者如果你打算训练不同尺寸的棋盘，也可以用来编码被掩码的区域。

One channel to encode black, one channel to encode white, and then and then one channel maybe to encode like empties or maybe like a masked region if you want to train on multiple board sizes.

我现在先不讨论多尺寸棋盘的问题。

I'm actually not going to talk about multiple board sizes for now.

那个有点太复杂了。

That's a little bit too complicated.

我们就假设有一个类似RGB的两到三通道图像，然后输入一个ResNet。

So we'll just say like, you know, we've got this two or three channel RGB like image, and then we go into a, you know, a ResNet.

接着分成两个分支头。

And then we have two branching heads.

一个分支预测价值函数，就是一个单独的逻辑值。

One head predicts the the value function, and this is like a single logit.

所以这个就是r1。

So this is like r one.

另一个是策略分支，也就是r361。

And then we have the policy, which is, you know, r three sixty one.

这就是整个架构。

So this is the architecture.

我们要训练这个模型，根据棋盘状态来预测对局结果。

And we're going to basically train this to predict the outcomes of games given the board state.

同时也要训练它判断哪些步法是好棋。

And we're also gonna train this to predict what are good moves.

没错。

Yeah.

对。

Right.

最初的AlphaGo论文，也就是AlphaGo Lee，是用专业棋手的对局数据来初始化这个网络的。

So the OG AlphaGo paper or called AlphaGo Lee initialized this network with a supervised learning dataset of expert human play.

后来，他们取消了这个限制，让模型自己学会如何下得好。

Later, they removed this restriction by having the model teach itself how to play well.

但从实际操作的角度来说，我觉得对听众而言，一开始就用简单的方式启动实验特别好，先把问题跑通，再慢慢挑战更复杂的部分，别一上来就想啃下整个难题。

But I find it actually from a matter of like implementation for your audience, super super nice to always kind of initialize your your experiments to something that's easy and then like, you know, get the problem working before, you know, trying to bite off the whole thing and learn a Tableau recipe.

一般来说，就像深度学习里常说的，初始化至关重要。

You you generally want to kind of initialize just as in deep learning, initial initialization is everything.

对吧？

Right?

做研究项目时，最好从最接近成功的起点开始，尤其是当你在尝试从未做过的新东西时。

You always wanna initialize your research project to something as close to success as possible, especially if you're doing something new that you haven't done before.

永远先选一个能跑通的方案，再让它变得更好，而不是从一个完全行不通的东西开始，硬要把它改好。

Always pick something that works and then get it to do something better rather than start from something that doesn't work at all and then try to make it work.

基于这个思路，从一个有良好初始设定的方案入手是个很棒的选择。

So under that philosophy, it's a great idea to start from something that has a good initialization.

所以我们打算用人类专家的对局数据，训练这个模型来预测好的走法。

So we're gonna take human expert plays and train this model to predict good actions.

对吧？

Right?

我们会把所有人类专家获胜的对局都挑出来，提取那些获胜时下的步法，然后让模型学习预测这些走法。

So we're gonna take all of the winning games, all the moves in which human won and sorry, an expert won and then predict those actions.

然后不管棋盘状态如何——不管你赢了还是输了，模型都要预测最终的胜负结果。

And then regardless of board state, like, you know, whether you won or lost, you're gonna predict the outcome.

是的。

Yeah.

你可能在想，好吧。

So you might be wondering like, okay.

有些早期的棋盘，就放了一颗子而已，你怎么可能知道谁会赢呢？

Well, some of the early boards, you know, where basically only one stone has been put down, how could you possibly know whether who who the winner of this game is?

对吧？

Right?

但如果你有几十万盘棋局，平均来看，像这样的开局，从这一步分支出去的棋局，大概一半会赢，一半会输。

Well, if you have, you know, hundreds of thousands of games, then in on average, you'll probably see that boards that start like this have a sort of half of the games that branch off from this will win, and half of the games that branch off branch off of this will lose.

所以这其实没问题。

So that'll actually be fine.

当你训练模型去预测这些时，输出的对数几率会慢慢趋近于0.5。

When you train this model to predict those, the logit will sort of converge to, you know, 0.5.

因此，训练完模型后，开局局面的预测值接近0.5，这是很正常的。

And and so so for these for these things, it's it's sort of expected that once you train the model, a starting board state will look like point five.

随着游戏接近尾声，胜率就会变得明显——比如，如果一开始是百分之五十，那它要么往这边走，要么往那边走。

And then as you progress towards the end of the game, it'll actually look something like you know, if this is point five, the the win probability will sort of either go like this or it'll it'll go like this.

对吧？

Right?

这代表的是你的步数。

And and this is sort of your move number.

没错。

Yep.

当你推进到几百步之后，根据你的专家数据分布，谁更可能赢、谁更可能输，就清晰多了。

And so as you, you know, get hundreds of steps into the game, it becomes much more clear, like, who's more likely to win or who's more more likely to lose under your expert data distribution.

我不太明白，为什么这种对胜率的思考方式特别跟专家数据有关。

I I didn't understand the significance of why the this way of thinking about value is especially relevant to the expert data.

这跟专家数据没关系。

It is not relevant to the expert data.

任何你用来训练的数据都是如此。

It's true for any data that you trade it on.

是的

Yeah.

所以如果你从零开始学习，也会预期出现同样的情况。

So if you were to learn a Tableau Rasa, you would also expect this to follow.

是的

Yeah.

所以你就想象一下，你在用 vibe 编程开发 AlphaGo，你从网上收集了一些围棋高手的对局数据，或者你有一组人类玩家的对局记录，然后用这些数据训练模型。

So if you just do this, like so imagine, you know, you're vibe coding AlphaGo and you you you gather some expert datasets from like how to go online or you you know, you have a dataset of human players and you train this model.

结果发现，这个模型已经是个相当不错的围棋选手了。

Actually, it turns out this model is already a pretty good Go player.

它大概率能赢过大多数人类玩家。

It'll most likely beat most human players.

对吧？

Right?

所以如果你直接用这个策略的推荐，对它的概率分布取最大值，也就是选概率最高的那一步作为你的落子，那它就会成为一个非常非常快的围棋选手，完全不靠一步步推理。

So if you just take this policy recommendation and take the argmax over its if this is the, you know, probabilities, if you take the arg max and you just take this action as your Go play, it'll be a very, very fast Go player that doesn't think in terms of, like, reasoning steps.

它就是凭直觉出招，但已经能成为很强的围棋选手了，想想看，才十层神经网络，参数还不到三百万，就能做到这种程度，简直不可思议。

It just kind of shoots from the hip, and it'll be a very strong Go player, which is already quite miraculous if you think about like, you know, 10 neural network layers, maybe under like 3,000,000 parameters can already do something that impressive.

是的。

Yeah.

所以你可以从这种方式开始。

And so you can start this way.

在实现的时候，最好验证一下这确实成立。

And it's important when implementing this to kind of just verify that this is probably true.

确认你的围棋规则写对了很重要，确保这些模拟能跑得足够快。

It's good to verify that your Go rules are implemented correctly, that, like, you know, you can run these simulations relatively quickly.

这就像一个检查点，你得先确保能顺利完成这一步，再往上叠加更复杂的比如搜索这类功能。

And just as almost like a sort of checkpoint that, like, you wanna make sure that you can actually do this basic step before you try to layer on more complex things like search.

是的。

Yeah.

不过，我们当然能做得比直接用神经网络来下棋更好。

So but, yeah, we can do a lot better than taking the raw neural network and playing the moves.

我们就可以用这种方法来应用到蒙特卡洛树搜索上。

And this is how we can apply it to Monte Carlo Tree Search.

接下来，我们用神经网络来优化蒙特卡洛树搜索。

So let's apply the neural network to improve Monte Carlo Tree Search.

首先，从根节点开始。

So we start with our root node.

现在我们有了一个四步的迭代流程来执行MCTS。

And we now have a four step iterative process to do MCTS.

我刚开始读这篇论文、想弄懂它的时候，这里可把我难住了。

So this tripped me up when I was first reading the paper and trying to understand it.

但简单来说，我们要做的就是设定一定数量的模拟次数。

But, essentially, what we're going to do is we're going to choose a number of simulations.

比如说，模拟次数这个参数。

So like, you know, num simulations.

这个数值并不是固定的。

And this number varies.

这个数字大概在200到2048之间吧。

This can be, you know, somewhere between 200 to twenty forty eight.

我记得在AlphaGo对李世石的那场比赛中，每步都用了上万次模拟，因为他们想把模型的强度推到极致。

I believe in in the AlphaGo Lee match, they use tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible.

是的。

Yeah.

但在训练时，其实不需要那么多模拟次数。

But in training, you don't actually need too many.

CutGo我猜用的也是差不多的数量级。

And CutGo, I think, uses something on this order as well.

你知道吗？看纪录片的时候，他们比赛时好像还拿出了笔记本电脑？

Do you know if they used if you watch a documentary, they had a laptop out during the game?

对。

Yeah.

但他们其实并没有真的用那台笔记本电脑。

They didn't use a laptop itself.

好像是在某个设备上

It was like on some

我觉得是用的TPU集群。

It was on some TPU pod, I think.

是的。

Yeah.

说实话，这有点不公平。

Now Honestly, it's kind of unfair.

嗯，

Well,

李世石下一手棋可没用上亿亿次浮点运算，对吧？

like, Lee is not using like one e 22 flops to do a move, you know?

这话有道理。

Fair enough.

有趣的是，现代的围棋AI在实际对弈时并不需要那么多计算资源。

Interestingly enough, modern Go bots don't need that much compute at test time.

是的

Yeah.

有意思。

Interesting.

当我们讨论蒙特卡洛树搜索如何改进策略时，会发现随着时间推移，原始网络其实把整个TPU集群的负担都扛了下来，全压到了自己身上。

And what we'll actually find out as we talk about how the MCTS policy improvement works is that over time, the raw network actually takes all of the burden of that big TPU pod and just push pushes it into the network.

你只需要一个神经网络的前向计算，就能完成所有这些工作。

And and you can do all of that work with one, you know, neural network bypass.

但TPU集群始终能再给它添一把劲。

But but the TPU pod will always add the extra oomph on top.

所以比赛时他们就是想要这种配置。

And so that's what they wanted for the match.

所以我们得选这个模拟次数的参数。

So so we're gonna pick this kind of like num simulations thing.

每次模拟，我们都会同时做几件事情。

And for every simulation, we're going to basically do several things simultaneously.

我们要看看当前树中哪些走法是最优的。

We're going to see which which moves are the best in the current tree.

如果需要扩展叶子节点，我们就加新的叶子，同时更新树中各个动作的估值。

We're going to add extra leaves to the tree if we get to a point where we need to add a leaf, and we're gonna update the action values for for the tree.

所以每次模拟其实都包含这四个步骤。

So that's that's what every every simulation involves these kind of like four step process.

这四个步骤就是：选择、扩展、评估和回溯。

So so the four step process is basically selection, expansion, evaluation, and backup.

在蒙特卡洛树搜索刚开始的时候，我们的树结构非常简单。

So so at the beginning of our Monte Carlo Tree Search, our tree is very basic.

它只有一个根节点，也就是AI当前要落子的棋盘状态。

It only has the the root node or our current board that our AI wants to play at.

所以我们首先要为这个状态选出最佳的走法。

And so we're going to basically select the best action for this.

当根节点创建时，我们也能用神经网络来评估它，得到价值vθ和各个动作的概率分布。

So when this root node is created, we also know that we can evaluate this under our neural network and get the quantities, you know, v theta as well as our probability over actions.

我就叫它根节点吧。

And I'm going to say root.

所以对于这些动作，我们都可以生成一堆子节点。

So for all of the actions here, we can create a bunch of children.

对吧？

Right?

这里我画的是一个三乘三的棋盘，缺了一格。

So so this one has well, in in this case, I'm drawing a three by three board with one one board missing.

所以总的来说，这个根节点一共有八个可能的子节点。

So basically, there are, you know, eight possible children associated with this root node.

每个子节点都对应着执行该动作的概率。

So, like and each of these has an associated probability of taking that action.

对吧？

Right?

比如 p8、p1、p2 这些。

So so there's p eight, p one, p two, etcetera.

好的。

Okay.

在蒙特卡洛树搜索刚开始的时候，我们有一个根节点，可以为它初始化一些子节点。

So at the beginning of our Monte Carlo Tree Search, we have our root node and we can initialize it with some children.

对吧？

Right?

因为根据策略网络对根节点的评估，在已经放了一颗子的三三棋盘上，AI一共有八种可能的走法。

Because we know it's the the policy network evaluate on the root node gives us on a three by three board with one existing stone placed eight possible children that this AI could take.

每个子节点对应的策略网络也会给出选择该走法的概率。

So with each of the children, their policy network also gives us the probability of selecting that child.

所以第一步是进行树的选取。

So the, first step is to do the selection of the tree.

而且这个树其实非常浅。

And again, this is a very shallow tree.

到目前为止，我们拥有的只是一个深度为一的树。

All we have so far is a tree of depth one, essentially.

对吧？

Right?

所以我们第一步是通过最大化PUCT公式来选择，也就是Q值加上C乘以P(a)除以(1+N(a))。

So our our first move is to select by maximizing or argmaxing the pucked criteria, which is basically, you know, q q s a plus, you know, c pucked times p of a divided by n over one plus n a.

一开始，所有动作的N(a)都是零。

So for each of these, we're going to you know, n a is zero for for all of the actions initially.

N也是零。

N is zero.

所以我们就根据这个公式来选动作。

And and so we're going to basically just, you know, pick according to this.

一开始，被选中的动作很可能会偏向概率最高的那个动作。

Initially, what is going to be the, you know, chosen action here is most likely going to be biased towards, you know, the highest likelihood action here.

对吧？

Right?

因为每个节点的这些值都差不多。

Because these are sort of uniform for every node.

我们假设 p1 是概率最高的那个节点。

So let's suppose p one was the highest probability node.

所以你选择了这个节点。

So you you you selected this one here.

现在你到了这个节点，发现它不是叶子节点。

Now, you've got to this node and you realize that it's not a leaf node.

对吧？

Right?

这不是一个终局，所以无法直接得出结果。

There are more it's not a terminal game, so you cannot resolve Yeah.

最终的判断还得继续。

The the final resolution.

接下来你要做的是扩展。

So the next step that you do is expansion.

然后你会把这个节点，也就是这个棋盘状态，输入到策略网络中。

So you will then run this node, this board state, through the policy network.

注意，这是AI的走法。

Note that this is the AI's move.

对吧？

Right?

也就是说，是AI在下这一步。

Like, AI is making this move.

所以当我们扩展这棵树时，就要考虑人类玩家或者对手可能怎么走了。

And so when we expand this tree, we're now thinking about what the human might do or any opponent might do.

对吧？

Right?

这就好比你的对手。

So this is like, you know, your your your opponent.

树的扩展过程其实完全是从这个玩家的角度来评估当前节点的。

The tree expansion process actually is completely so so so when we evaluate the the node here, we're gonna now evaluate the the node from the perspective of this player.

没错。

Yep.

所以这一层有我们可以采取的行动，我们会把这里的叶子节点全部展开。

So then this one has possible actions that we could take, and we we expand basically the the the leaf nodes here.

对于每一个可能到达的节点，我们都要评估它们的价值如何。

So for each of these nodes that we could, you know, arrive at, we're gonna now check how good those nodes are.

对吧？

Right?

比如从这里，人类可能会下在这里，或者这里，又或者这里。

So so maybe from here, like, the human could play here, the human could play here, or human could play here.

我们会为每一个这样的节点存储对应的 v theta 值。

And we're gonna store essentially the v theta for each of these things.

比如节点一的 v theta，或者节点一_prime 的 v theta。

So v theta of, you know, node one or, like, node one prime, v theta node one prime.

我们其实是用神经网络来直观地判断，这个棋盘局面对当前玩家来说有多好。

And so we're we're basically using our neural network to make an intuitive guess of how good is this, board from the perspective of this player.

是的。

Yeah.

幸运的是，因为这是个零和博弈，所以很容易推断出，当前玩家在这个步骤的值，其实就是一减去对手视角的值。

And fortunately, because the it's a zero sum game, it's easy to deduce that, you know, the value for this player at this this step is just one minus the value for you know, from this perspective.

所以根据当前是哪一方走棋，翻转搜索过程就很简单。

So it's easy to flip the search process depending on which player you're at.

这一步就是扩展阶段。

And so so this is the expansion step.

你把一个非叶子节点展开，然后评估了它的价值。

You've taken a a a non leaf node and expanded it and evaluated the value.

这本质上是个快速估算：如果我直接下到终局，会不会赢？

And this is essentially a quick guess as to, like, if I were to play to the end, am I gonna win or not?

对吧？

Right?

你可以把 v theta 想象成一种捷径，省去在每次模拟中从头搜索到树的末端的麻烦。

So you can almost think about the v theta as a shortcut for searching to the end of the tree for for any given simulation.

然后这一步，就是评估阶段。

And then we're and this is this is essentially the evaluation step.

我们正在评估这些棋盘局面的质量。

We're we're evaluating the quality of each of these boards.

在原始的AlphaGo中，他们做了一件挺有意思的事：把这个估值和实际走完一盘围棋的结果取了个平均。

In original AlphaGo lead, they actually did something, kind of interesting, which is that they took this value and they averaged it with the value of a real Go playout.

所以他们真的从这里开始，把棋局完整地走下去，直到结束。

So they actually played a real game from here all the way to the end.

我就画条波浪线，代表一条可能的走法路径。

So so, like, I'm just gonna draw this squiggly line to indicate some path.

然后他们把这条路一直走下去，直到棋局彻底分出胜负。

And they kind of, like, play this all the way to res Trump Taylor resolution of a full board.

所以这个结果要么是0，要么是1。

And so this is like a zero or one.

对吧？

Right?

于是他们就把这个估值和刚才那个实际走出来的结果取了个平均。

And so they took this value and they just averaged it with with this one here.

所以他们用的公式是，阿尔法乘以某个节点的vθ，再加上一减阿尔法乘以一次真实的随机模拟结果。

So the the formula they did was like, you know, alpha times v theta of of, like, you know, some some node plus sort of, like, one minus alpha of a of a true randomly sampled playout.

你可能在想，嗯，这具体是怎么操作的呢？

And you might be wondering, like, okay.

他们是怎么完成这个模拟的？

Well, how do they play this out?

对吧？

Right?

要是再对这次模拟做一次搜索，成本会高得离谱，简直像在树里再建一棵树。

Like, it would be very, very costly to do another search on on this playout, like, almost like a tree within a tree.

但他们并没有这么做。

So they don't do this.

相反，他们直接用策略网络让双方对弈。

Instead, they just take the policy network and play it against itself.

也就是说，同一个网络同时扮演双方，一直下到终局。

So they just take this as both players, they just play it all the way to the end.

这有助于让这里的估算更贴近现实，因为你能直接得到一个样本，知道最终是赢还是输。

And and this is something that helps ground the the estimates here in in reality because you can get a single sample estimate of, like, whether you win or not.

在终局阶段，棋盘局势基本明朗时，这种方法特别有用，因为按照策略随机走子，大概率能给出一个相当合理的胜负判断。

You can think about in the endgame where the board is almost resolved that this one actually becomes quite useful because the random the the the play according to the policy will most likely decide a pretty reasonable guess of the game.

所以你就不会遇到这种问题——估算完全脱离了现实。

And so you're not, you know, facing a problem where this one kind of becomes untethered from from reality.

结果发现，这一步根本没必要。

It turns out this is totally unnecessary.

所以从《AlphaGo Lee》之后的所有论文里，他们都直接删掉了这一步。

So in all subsequent papers after AlphaGo Lee, they just got rid of this.

对。

Yeah.

在我的实现里，我也做了同样的处理，这样速度提升很多，因为不用在每次模拟中都完整模拟一盘棋。

And so in my implementation, also did the same, and it speeds things up a lot because you don't have to roll these games out on every single simulation.

对。

Yeah.

好的。

Okay.

我再重复一遍，主要是为了巩固自己的理解，也方便重新解释一下。

So again, just to reinforce my own understanding and just to re explain it.

顺便说一下，如果还不清楚的话，select里的p指的是这个情况下网络输出的概率。

For the items, by the way, in case it's not obvious, the p there in the select, that is the probability coming from the network in this case.

没错。

Correct.

就是策略网络。

The policy network here.

对的。

Yeah.

好的。

Okay.

本质上，一次模拟就是在搜索过程中再展开一个节点。

So fundamentally, a simulation, just think of it as like rolling out one more node in the search process.

差不多了。

Almost.

当整棵树已经存在时，模拟就很容易理解了。

So a simulation is easy to think about when the whole tree already exists.

对吧？

Right?

你只需要根据UCB选择标准一路往下走，然后继续下去。

You just walk down the tree using the puck selection criteria, and you you and then and then you keep going.

是的。

Yeah.

在AlphaGo里，数据结构的特点是，一开始树几乎只有第一层，也就是只有根节点的直接子节点。

In AlphaGo, the the data structure is such that we begin with a tree that has no, like, basically only depth one, which is its only children.

然后你需要逐步扩展这棵树，没错。

And you want to iteratively build out the tree Yep.

同时你也在沿着树选择动作。

As you're also selecting actions down the tree.

所以这里的核心在于，围棋的组合复杂度太高了，你根本没法提前把整棵树建好再搜索。

So that's the kind of core thing here is that because Go is such combinatorially complex game, you cannot afford to build the tree in advance and then search it.

你必须边构建树，边进行搜索。

You must search while building the tree.

对。

Right.

明白了。

Okay.

那我最后再讲一下最后一个步骤，就是反向传播。

So let me just finish up with actually the last step, which is the backup.

对吧？

Right?

一旦你给这些节点打完分，就把这个动作对应的 Q 值更新为所有评估结果的平均值。

So once you've scored these things, you basically take the mean the the value the the q value assigned to the node here for taking this action is now just the average across your evaluated values.

没错。

Yep.

你需要对所有模拟结果取一个动态平均值，把子节点的值都算进去。

It's you take a running mean over over all of the simulations that you've taken, and they average the values of the children nodes.

对。

Yep.

这就是所谓的回传步骤。

So so that's what is known as the backup step.

一旦算出这个值，就可以逐级往回推了。

And once you evaluate this, you can actually kind of recursively go back.

如果你知道了这个节点的动作价值，就能继续算它的父节点，再往上推，依此类推。

So if you know the, you know, the action value of this node, you can then take the average on its parent and so on and so forth.

所以整个过程大致分为四步：先选目前看来最优的动作。

So so you have this kind of four step process where you are choosing the best action that you know of so far.

然后你可能会遇到一个从未访问过的节点，这时候就得扩展树的结构。

Then you may run into a node where you you you haven't been to before, so you need to grow the tree a bit.

接着用神经网络预测一下，看看你赢的概率有多大。

And then you run it through the network to guess whether you're gonna win or not.

然后你一路回溯到根节点，更新关于最佳走法的数值。

And then you walk all the way back up to the to the root node to update your values on what the best moves are.

随着你不断迭代，这个选择标准会让你优先访问那些你认为当前分支下最好的动作。

So as you do this iteratively, this selection criteria will cause you to visit the because you're always selecting according to this criteria, you're always gonna be selecting the best action you think at any given branch.

对吧？

Right?

所以最终的访问次数——也就是你选择每个动作的频率——会反映出通过这个搜索过程所形成的正确策略分布。

So so the final visit counts of, like, how often you chose these things will reflect your correct policy distribution as induced through this search process.

因此，我们之前存在节点里的访问次数，其实就相当于一种投票，决定最终该选哪个动作。

And so the visit count that we store in the node earlier actually becomes the sort of vote for like which way we should finally select an action here.

没错。

Yep.

作为检验理解的一个方式，你可以想想看，我们能不能把这个过程再简化一点。

So, you know, as a sort of test of understanding, it's worth thinking a little bit about whether we could make this even simpler.

对吧？

Right?

我们是不是干脆把这个也去掉，还能让整个系统正常运行？

Like, could we actually maybe even get rid of this one and still make the thing work?

回想一下，当你进行扩展并评估某个节点时，比如这个节点，你会计算每个子节点的胜率。

So recall that, you know, when you do an expansion and then an evaluation at, let's say, this node, you you are checking the sort of win probability of each of the child nodes.

对吧？

Right?

如果这个子节点胜率是1，其他都是0，那你其实已经能大致判断出哪个动作更好了。

And so if this one is, you know, like one and these are zero, you do kind of know something about which action might be better to take.

那为什么还需要这个机制呢？

And so why would you need still need this?

对吧？

Right?

为什么不直接把胜率归一化，当成策略分布呢？

Like, why not just, normalize this one into some distribution and call that your your policy distribution?

这样也行。

This is fine.

你可以这么做。

You can do this.

而且这很可能真的有效。

And this probably does work.

但实际操作中，通过一次前向计算就能给出不错的预估，这才是真正压缩搜索广度的方式。

But in practice, having a single forward pass that gives you a pretty good guess is is how the the breadth is is is pruned out.

这里其实存在一种对偶关系。

The there is a sort of duality here.

比如说，如果策略推荐的动作和价值评估结果相矛盾，那就有点奇怪了。

Like, it would be weird if, let's say, the policy recommended an action that disagreed with the value.

对吧？

Right?

如果策略认为这个动作概率极高，但价值评估却说它价值很低，那说明你的策略网络和价值网络之间肯定出了根本性问题。

If if, let's say, policy said this was very high probability, but this one said it was a, you know, low value, then there's actually something kind of fundamentally wrong between your policy head and your value head.

所以它们是相互关联的，如果你能找到别的方法，只靠价值评估就还原出这个信息，那这个部分其实可以去掉。

So they are linked and you probably could get rid of this if you came up with a different way to recover this from just the value evaluations.

对。

Right.

但为了确认我理解对了，你不这么做的原因是为了避免做360次独立的前向计算，来一一查看。

But just to make sure I understand, the reason you don't do that is so that you don't have to do 360 independent forward passes to, hey.

把所有东西的值都算一遍。

Here's the value of everything.

然后再取最大值。

Let's talk max over it.

对吧？

Right?

相反，你只需要一次前向计算，就能直接得到所有动作的概率。

Instead, you can just do one forward pass and get, like, the probabilities of all of

这些计算其实通常可以批量处理，效率还不错。

You can usually batch these somewhat efficiently.

所以实际操作中，计算负担并不会特别大。

So it probably is not a huge computational burden in practice.

但没错，你要想评估所有这些值，然后做归一化，就得把最多361个棋盘状态一次性输入到一个最小批次里。

But, yes, you would have to pass 361 board like, up to 361 boards into a single mini batch update to evaluate all the values here, then normalize them.

不过，我们之所以还这么做，其实还有个更重要的原因，就是蒙特卡洛树搜索如何通过自我反馈，逐步递归地提升它的预测和搜索能力。

Now, there's actually a more important reason why we still do this, which is how Monte Carlo Tree Search is used to feedback on itself and and sort of recursively improve its own predictions and search capabilities.

正因如此，把这一点作为一个明确建模的实体，而不是简单地对价值做隐式归一化，反而更合理。

And that's where this this one, having this as an explicit entity you're modeling rather than an implicit normalize normalization over your value is is a is a good idea.

有道理。

Makes sense.

明白了。

Okay.

好的。

Okay.

我们刚才聊了模拟的过程。

So so we we talked about the simulations.

简单说，随着模拟次数增加，你最终会得到一棵树，样子大概就像我这儿画的这个，虽然维度低得离谱。

And, basically, you know, what you end up with as you roll out the number of simulations is a tree that kind of looks like I'm I'm I'm drawing a very low dimensional version of this.