雷纳·波普——大型语言模型训练与服务背后的数学原理

这个是40。

This is 40.

你可以把这想象成一列火车的发车时刻表。

You can think of this as a schedule for the train.

每二十毫秒就有一趟新火车发车。

A new train departs every twenty milliseconds.

所有准备好的乘客都能上车。

Any passengers who are ready board the train.

如果火车坐满了，那就得等下一趟。

If the train is full, then they wait till the next train.

就算火车没坐满，它也会照常发车。

If the train is not full, the train's gonna go anyway.

这么说吧，关于排队延迟，最坏的情况是：请求刚错过火车，一来就发现火车刚走。

And so in terms of what that means for queuing latency, it means that the worst case is that you like, a request arrives just after the train departed.

那它就得等下一趟，最多等二十毫秒，还得再等这趟火车跑完。

It has to wait for the next train, so that's up to twenty milliseconds, and then it has to wait for that train to to complete.

是的。

Yeah.

所以最坏情况下的延迟是四十毫秒。

And so the worst case latency is forty milliseconds.

那这个‘列车毫秒’是怎么算出来的？

So how is the train milliseconds derived?

说白了，这只是一个经验法则，具体来源还没完全讲清楚，不过到目前为止，我们主要关注的是内存带宽和计算时间。

I mean, rule of thumb, but where it comes from is not fully explained yet, but so far we've focused on memory bandwidth and compute time.

当我们看内存时，另一个要考虑的是，我们希望充分利用现有的所有内存容量。

When we look at memory, the other consideration is that we want to use all of the memory capacity we have.

所以一般来说，我们会用这些内存来存储权重或者kV值。

And so, generally, we're going to use all of that memory capacity to store the weights or the kVs.

因此，我们希望在一次前向传播的时间里，能把所有内存容量都读进芯片里。

And so we just want to read, like, in the time of doing a forward pass, maybe we want to read all of the memory capacity into into the chip.

所以这就等于容量除以带宽。

And so that is capacity divided by bandwidth.

在很多代HBM上，这个值通常是二十毫秒。

That tends to be twenty milliseconds on on many gen different generations of HBM.

单位是对的。

The units make sense.

你就是用字节除以每秒字节数嘛。

You would have a byte divided by bytes per second.

对。

Yeah.

比如说，在Reuben这一代，大概是288GB除以20TB每秒。

So, for example, I mean, on on, I think, the Reuben generation, it is something like 288 gigabytes, divided by 20 terabytes per second.

算下来大约是十五毫秒。

And this looks like it comes out to about fifteen milliseconds.

我再确认一下，我理解得对不对。

Just to make sure I understand what it's saying.

我的意思是，我知道单位为什么这么算，没错。

Mean, I understand how why the units can't Yeah.

对的

Yeah.

就是这个单位分析的逻辑嘛

The the sort of unit analysis.

没错

Mhmm.

不过这句话到底想表达什么？是说我们能清空、替换什么东西吗

But what is it saying is we can evacuate and replace

没错，就是这样。

That's right.

HBM得在这么短的时间内完成读写。

The HBM in this amount of time.

我们可不想碰到两种麻烦：要么是HBM容量不够大，没法把要存的东西都写进去，或者要取的数据都读出来；要么就是我们的双向传输能力太弱，远远跟不上需求。

And so we don't wanna be in a situation where the HBM is not big enough that we're not gonna actually able to keep write everything you want to it or take everything out of it, or we don't wanna be in a situation where our ability to write back and forth is so big or sorry, so small compared.

对的。

Yeah.

这话其实有道理。

There's sort of truth there.

咱们为啥不选个超过十五毫秒的延迟呢？

Why don't we pick a latency that is bigger than fifteen milliseconds?

如果这么想的话，就意味着我有足够时间把HBM读两遍。

And if I think what that means, it means I actually have time to read the HBM, like, twice.

没错。

Yep.

顺便说一句，HBM的访问大部分都是读，不是写。

By the way, most of HBM accesses is reads, not writes.

几乎是全读，因为权重矩阵是只读的，而KV缓存的访问也差不多，所以在大概三十毫秒内，我能把HBM读两遍。

It's like almost all reads because the weight matrices are read only, and then almost all of the KV cache accesses are So in like, let's say around thirty milliseconds, I can read all of HPM twice.

但这么做的意义在哪？

But what's the point of that?

我是说，我根本没必要把权重矩阵读两遍。

Like, I I don't wanna read the weight matrices twice.

我不想把KB读两遍。

Don't wanna read the KBs twice.

对。

Yeah.

有道理。

Makes sense.

这话说得

Makes a

很有道理。

ton of sense.

好的。

Okay.

我有几个简单的问题。

So a couple of actually quick questions.

第一，如果最优批量大小真的是2000左右，那这完全取决于稀疏性。

One, if it is the case that the optimal batch size is something like 2,000, and that I actually true it's totally dependent on the sparsity.

这跟模型大小没关系，什么都没关系。

It's not dependent on the model size or anything.

我是说，稀疏性确实跟模型大小有关，但除此之外，就只看稀疏性，跟规模没啥关系。

I mean, sparsity shows up in model size, but beyond that, it only depends on sparsity, not on scale.

这结果挺有意思的。

But that's a very interesting result.

这似乎引出一个问题：这种推理和批处理带来的规模经济，到底会多大程度推动集中化呢？

And that seems to imply that you can one question is how much of a push towards centralization is it that you would have these economies of scale from inference, from batching?

是啊。

Yeah.

但好像也没那么重要。

But it seems like it's not that big a deal.

说实话，我也说不准。

Like, I don't know.

同时有两千个用户，算多吗？

Two is 2,000 users at the same time a lot?

看起来也不算多吧？

It doesn't seem like a lot?

我们可以稍微分析一下，与其看用户数量，不如换个角度，想想每秒处理多少个词元更实际。

So we can do a bit of analysis on this, which would be actually it's like you can think of it in terms of number of users, but maybe a more productive way to think of it is in terms of number of tokens per second.

对。

Mhmm.

那这个批量大小，换算成每秒多少词元的话，意味着什么？

So what does this batch size mean in terms of tokens per second of this of the system?

每秒词元数。

So tokens per second.

每秒词元数就等于批量大小。

Tokens per second is going to be equal to the batch size.

每次处理一批词元，然后每隔一段时间重复一次。

We run a batch many tokens, and then we do that every t.

每隔一段时间，比如十五毫秒、二十毫秒这样的间隔。

So So every time intervals, which is, let's say, which is which is this thing is equal to the fifteen milliseconds, twenty milliseconds number.

所以最终就是批量大小乘以60左右。

So this ends up being batch size itself times about 60.

比如64乘以b。

So, like, 64 times b.

那这就大约是2000乘以64了。

And so this ends up being around 2,000 times 64.

也就是大概128万tokens每秒。

So, like, a hundred and twenty eight hundred and twenty eight k tokens per second.

这样换算成更容易理解的单位了。

So this is sort of in more digestible units.

光想并发用户数很难搞清楚，但一个系统的全局流量到底有多大呢？

Like, it's hard to reason about concurrent users, but what is the track global traffic for for a system?

你看一些厂商的公告，有时候他们会吹嘘自己有多少流量。

When you look at some of the announcements, sometimes the API providers will will brag about how much traffic they have.

我记得去年Gemini发布时，他们说全球每秒处理的tokens高达数亿。

The the the numbers that I've remembered from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide.

是的

Yeah.

所以，这大概是那个量的千分之一。

So so about a like, this is one thousandth of that printer.

Gemini。

Gemini.

是的

Yeah.

但Gemini体量这么大，所以它的千分之一已经相当可观了。

But, I mean, Gemini is big, so I think that's actually one thousandth of Gemini is a lot.

对。

Yeah.

要想在规模上具备竞争力，你得能

To to actually become, like to be competitive at scale, you need to be

提供至少Gemini千分之一的服务能力。

able to serve at least one thousandth of Gemini.

是的。

Yeah.

这挺有意思的。

That's interesting.

不错。

Cool.

好的。

Okay.

稀疏度越高，所需的计算量就越少。

So the more sparsity you have, the less compute you need.

而且看起来，随着批量变大，计算能力确实成了瓶颈，对。

And it does seem that as batch sizes get bigger, compute ends up being the bottleneck Mhmm.

根据这项分析。

According to this analysis.

那问题来了，稀疏度能推进到什么程度？

So then the question is, how far can you take sparsity?

也就是说，随着稀疏率上升，活跃参数占总参数的比例越来越低，模型的性能会下降多少？

That is to say, as the sparsity ratio increases, as you have fewer and fewer active parameters relative to total parameters, how much is performance of the model degrading?

而且这种性能下降的速度，会不会比你靠提高稀疏度节省的计算资源还要快？

And is it degrading faster than your saving compute by increasing the sparsity factor?

是的。

Yeah.

你是说模型的性能，而不是运行速度吧？

So perform equality of the of the model, you you mean, rather than speed of the model.

对。

Yeah.

没错。

Yeah.

可惜我们没法从理论上给出确切答案。

So, unfortunately, we're not able to answer that analytically.

这其实是个得靠实验来验证的模型质量的问题。

That's that is an empirical question of model quality.

我所能做的就是找篇论文，用实证数据来回答这个问题。

Best I can do is pull up a paper and answer that empirically.

对。

Yeah.

好的。

Okay.

我们现在就看这篇论文吗？

Should we follow the paper now?

还是说

Or is that

有道理，没错。

makes sense yeah.

这篇论文叫《路由语言模型的统一定律》。

So this paper, this is Unified Laws for Routed Language Models.

虽然这篇论文有点老了，但他们做了一件很重要的事：研究了当稀疏度不断提高时，模型质量会受到什么影响。

It's a somewhat old paper by this stage, but one of the things that they did is looked at, if I keep increasing sparsity, what is the model quality impact?

这个结果对专家混合的具体配置特别敏感。

This answer is very sensitive to the actual choice of mixture of experts.

专家混合这个方法早就有了，历史很长。

Mixture of experts has been around for a really long time.

我记得可能早在2017年就有了。

I think it was maybe even back in 2017.

但后面的技术手段变化太大了。

But the tech techniques have changed a lot.

DeepSeq的专家混合方式，彻底改变了它的运作机制。

DeepSeq mixture of experts was was a big change in how it worked.

之前还有些老论文，比如G-Shard和Switch Transformer。

There have been older papers which are G Shard switch transformer.

所以实际的实验结果，全看用的是哪一套方法。

So the actual empirical results are gonna depend on all of that.

但你看这里展示的早期方法，如果保持活跃参数数量不变，只增加稀疏度——他们这儿叫专家数量——模型效果反而越来越好。

But on one of the older techniques that is shown here, you can see if I hold constant the number of active parameters at a certain size, and then I increase the sparsity, which they call expert count here, The quality keeps increasing.

如果你想象从13亿密集模型那里画一条水平线过去，就会发现，比如在这个情况下，64个专家、3.7亿激活参数的模型，表现和13亿的密集模型一样好。

And then if you imagine drawing a horizontal line from 1.3 b dense across, you end up seeing that, for example, in this case, the 64 expert, three seventy million activated parameters model is as good as a dense 1,300,000,000 model.

所以说，某种程度上，你根本不需要把总参数量增加一百倍，就能达到激活参数增加十倍的效果，这回报其实没那么夸张。

So, some sense, it's actually not amazing returns where you need to increase total parameters a hundredfold to get to the equivalent of 10 x as many active parameters.

是的。

Yeah.

实际上甚至更明显。

Mean, actually, even more so.

是的。

Yeah.

参数量增加这么多，但性能提升却只有一点点。

It's a huge increase in parameter count for a modest increase in

对。

Yeah.

那在这种情况下，具体是多少来着？

So in this case, it's what what is it?

四倍？

Four x?

六十四倍换四倍。

64 x for four x.

对啊。

Yeah.

虽然说，增加稀疏度确实能节省计算时间，乍一看这买卖挺划算的。

So while it is while it is true, I guess, that the you get this benefit of being able to economize on your compute time if you increase sparsity, naively, would seem like, oh, that's a trade off worth making.

但你每把稀疏度翻一倍，这个数值反而降两倍，那个却涨八倍。

But if this this you're decreasing this by two x and then having this go up by eight x every time you double sparsity.

这到底是好是坏啊？

So is that good or bad, actually?

从内存角度看，别忘了，你每次都在把内存读取量翻倍，而这部分本来是靠批量摊薄的。

Even from a memory point of view, keep in mind, you are doubling this portion of the memory fetches, which is amortized by batch.

所以干脆直接加大批量就行。

And so just just keep running a larger batch size.

从我们这里做的分析来看，这完全是赚到了。

From the point of view of the analysis we've done here, this is pure win.

继续这么做。

Keep doing it.

直到你用光了所有可用用户为止，就一直做下去。

Keep doing it until you run out of available users, basically.

对。

Mhmm.

所以实际上，如果你想用稀疏模型，或者你用户很多，就可以用更稀疏的模型。

So there's actually this equivalence between if I want to go sparse, or if I have a lot of users, I can go to a much sparser model.

从这个角度看，这个权衡是合理的。

So from that point of view, it's it's a reasonable trade off.

另一个问题是它也会占用内存容量，我们刚才只讨论了内存带宽，但它确实也会消耗内存。

The other trade off that shows up here is that it also consumes memory capacity, which we we we've only reasoned about memory bandwidth here, but it also So consumes memory

让我确认一下我有没有理解对。

let me just make sure I understood.

你是说我们希望减少计算时间，对吧？

You're saying we want bigger we we want to spend less time computing.

所以我们要增加稀疏性。

Therefore, we do more sparsity.

为了实现这个，我们需要更大的批量，对的。

To make that work, we need bigger batch sizes Mhmm.

这就意味着我们需要更多的内存容量。

Which means we need more memory capacity.

没错。

Yeah.

所以为了实现更高的稀疏性。

So To have more sparsity.

没错。

Yeah.

那咱们不妨趁这个机会聊聊，专家混合层在实际的GPU集群上一般是怎么部署的。

So, I mean, maybe this would be a good point to actually talk about how a Mixture of Experts layer is typically laid out on on a, like, on a rack of GPUs or something.

是的

Yeah.

对

Yeah.

有道理

Makes sense.

嗯

Yeah.

我们刚才说到哪儿了？

Where were we?

稀疏专家混合模型

Sparse Mixture of Experts.

没错

Yes.

也许可以聊聊怎么在GPU上部署这个结构

Maybe how we lay that on out on a GPU.

对。

Yep.

咱们先聚焦在专家混合层上，画个图看看它长什么样。

So let's zoom in on the Mixture of Experts layer first and and and sort of draw what that looks like.

通常我们会有一个路由层，这个没错。

So we typically will have a some kind of a router layer Mhmm.

它负责决定把每个词元分发给哪个专家。

Which is making the decision of where we route the experts the the tokens to.

所以会有词元输入进来。

So we get tokens coming in here.

它们经过路由层，然后被送到一堆不同的专家那里。

They go through a router layer, and then we have a bunch of different experts.

我多画几个，好对齐一下。

I'll draw draw a few more to line some up.

路由层会决定把词元派给哪些专家，但只会选其中一小部分，比如三十二个里选一个。

And then the router will make a decision on which experts am I gonna route to, and it'll be a small fraction of them, maybe one in 32.

所以它可能会决定把任务交给这个专家，也可能交给这个，还有这个。

So maybe it'll make a decision to route to this one, maybe this one, and maybe this one.

这些专家本身呢，其实就是普通的MLP。

These experts so these each expert itself is a normal MLP.

它有个上层投影，接着是个非线性激活，再下来是下层投影。

It has a up projection and then a down projection and a nonlinearity in between.

最后呢，我们再做一遍反过来的操作。

And then finally, we sort of do the inverse operation.

之前是把数据分发出去，现在就要收回来，加总在一起。

So where we were broadcasting things out here, we're gonna bring them back in and sum them up.

就像这样把它们收回来。

So bringing them in like this.

最后，还有残差连接，原始的token也会直接走这条路，然后加到MoE层的输出上。

And then finally, we have our residual connections that the token is also passed through here, and it gets added to the result of the MoE layer.

所以，这就是一个标准的MoE层。

So so this is a normal MoE layer.

我想聊聊这个结构是怎么映射到GPU机架上的，以及这对通信意味着什么。

What I want to talk through is how this is mapped to a, like, a GPU rack and what this means for communication.

因为我觉得这会开始暴露出我们速度上的瓶颈。

Because I think this will will start to show some of the the limits of how fast we can go.

没错。

Yeah.

所以目前的标准做法——也是最优解——是用专家并行。

So the standard practice here, and it it is the best solution, is to use expert parallelism.

也就是说，不同的专家分配到不同的GPU上。

So that means different experts go on different GPUs.

比如DeepSeq模型，它有256个专家。

So if we take something like a DeepSeq model, they have 256 experts.

假设我们要在Rockwell机架上跑这个模型。

Let's say we want to run that on a Rockwell rack.

这个机架一共有72块GPU。

So there are 72 GPUs.

我们遇到一个整除问题。

We have a divisibility problem.

这不是2的幂次。

This is not a power of two.

所以咱们干脆点，就用64个好了。

So we'll just, like, simplify and say we're only gonna use 64 of them.

剩下的八个直接忽略掉。

Just ignore the other eight.

没啥大不了的。

It's not a big deal.

这样每块GPU上分配四个专家。

And so we we have four experts per GP.

很简单吧。

Very simple.

为了图示清楚，咱们就假设每块GPU放两个专家。

For the sake of the diagram, I'll actually just say, let's let's say we have two experts per GPU.

所以我们最后就把这些作为GPU的分界线。

So we we end up just putting these are the GPU boundaries.

每两个专家都放在单独的GPU上。

Every pair of experts is on its own GPU.

然后我们来看看通信开销。

And then we can look at the communication cost.

有些专家模型存在这里，还有一些token集中存放在中央。

We had some experts stored there's some tokens stored centrally here.

这些token会被分发到所有的专家那里。

They get routed to all of these experts.

所以这里会产生一些通信开销。

And so there's some communication cost paid here.

输出端也有同样的通信开销。

There's the same communication costs paid on the output.

我们希望这种开销不会成为瓶颈。

And then the hope is that this does not become communication limited.

那这里的流量模式是怎样的呢？

Now, what is the traffic pattern here?

这里的流量模式是，任何一块GPU都可能跟其他任意一块GPU通信，具体取决于模型的决策。

The traffic pattern here is that any GPU, in fact, will be talking to any other GPU depending on the decisions made by the model.

所以这是一种全对全的通信模式。

So this is an all to all traffic pattern.

你刚才说任何GPU，意思是……对，没错。

So when you say any GPU in the pretense, Yeah.

这个路由组件不止一块GPU吧？

The router is more than one GPU?

是的。

Yeah.

我画的时候只画了一个路由模块。

The route so I I drew this as one router.

但实际上，你会有好多份这样的路由模块。

In reality, you would actually have many copies of the router.

实际上，有多少个GPU，就有多少个路由器。

And so you would have as as many routers as as GPUs, in fact.

就跟进入的流量一样。

As the incoming traffic.

对。

Yeah.

这里是64个GPU。

So, these are 64 GPUs.

这里是64个GPU。

These are 64 GPUs.

其实都是同一组GPU。

It's actually the same GPUs.

我们只是画成分开的，因为它们承担的职责不一样。

We just draw them as separate because they're serving different purposes.

所以这时候，任何一个GPU都可以跟其他任意一个GPU通信。

So at this point, any GPU can be sending to any any other GPU.

所以这种全互联的通信模式，正好契合了MoE模型实际需要的通信方式，而Blackwell机架的配置正是为此设计的。

So this all to all pattern of communication that shows up and how how the Blackwell racks are configured is a is a perfect fit for the the communication pattern that the MoE actually wants to do.

不过，如果你想着，也许一个机架太慢了，想用两个机架，那就会遇到一个问题——比如你在外头划了一道机架边界。

However, if you think, maybe I want to do like, maybe one rack is too slow and I want to do two racks, then I have this challenge that, like, maybe I've got some sort of rack boundary drawn outside here like this.

这样一来，两个机架里的所有GPU之间，就不再是全互联的通信了。

And I no longer, in fact, have all to all communication between all the GPUs in two racks.

于是，机架之间的通信就成了一个很大的瓶颈。

And so the rack to rack communication ends up being a substantial bottleneck.

所以核心问题在于：一个机架的规模，实际上就决定了你能部署的专家层的大小。

So this sort of like, the fundamental thing here is that one rack is actually the bounds the size of an expert layer you can do.

这也正是推动我们不断扩展互联范围的原因之一。

And so this has been part of what's been driving towards larger and larger interconnect domains.

是的。

Yeah.

但在我们继续之前，也许你该先解释一下，到底什么叫做‘机架’。

But before we it may be worth you explaining what exactly a rack is.

对的。

Mhmm.

机架内和机架间的带宽差异

The differences in bandwidth between a rack and within a rack

没错。

Yeah.

机架内是全互联，机架外就不是了，这个区别挺关键的。

And the all to all versus not all to all nature of communication within versus outside.

对。

Yeah.

这一点上，NVIDIA、谷歌，还有我们自己，做法其实差别挺大的。

And and this is a place where it starts to be very different, in fact, between NVIDIA, for example, and Google, and then others, including us.

一般来说，机架就是一个物理结构。

The so generally, a rack is a it is a physical structure.

大概几米高，一两米宽，具体看配置，里面通常能放64个GPU或者XPUs。

It it's a few meters tall, a meter or two wide, depends on configuration, and it stores some number of GPUs or XPUs, which is typically about 64.

这种尺寸的限制主要来自供电、重量和散热能力。

The the cons what constraints it being a certain size is power delivery, weight, and cooling ability.

很多时候，机架就是做成这么大，因为物理条件就这么摆着。

It ends up being about this size in in many cases because of these physical constraints.

所以当我部署一个数据中心时，比如，一个数据中心可能有成千上万个这样的机架。

So then when I deploy a data center, like, I've got a data center may have thousands of these racks.

我这儿有一个高高的机架。

So I've got one of these tall racks.

里面装了一堆GPU之类的设备。

It's got a bunch of GPUs in it and so on.

然后我再把另一个机架挨着放上去。

And then I put another rack next up.

你这话听着也太简单了吧。

You make it sound so easy.

对。

Yeah.

我就直接把它们塞进去了。

I I just, like, dropped them in.

在英伟达的方案里，他们把显卡放在机架外侧，而把交换机放在机架内侧。

In NVIDIA's case, the the communication topology is actually they put the GPUs on on the outside of the rack, and then they put these switches on the inside of the rack.

所以实际上，这里面有一组交换机。

So what this ends up being is that there's a set of switches in here.

这些就是NV交换机。

These are the NV switches.

对。

Mhmm.

然后他们拉了一大堆线缆。

And then they run a bunch of cables.

每一块显卡都有一根线连到中间的交换机。

Every single GPU has cables going going to the switches in the middle.

每块显卡都连到中间的交换机，而交换机又连着所有显卡。

So every GPU goes to switches in the middle, and then the switches have connections to all the GPUs.

所以所有的GPU之间通信，最多只需要两跳：先到交换机，再跳到目标GPU。

So all of the GPUs can talk to all the other GPUs in in just, like, two hops, going to switch, going to the other GPU.

但当我需要把数据传出机架时，就得走另一条路了。

Now when I want to leave the rack, I end up going via a different path.

GPU还有一条慢得多的连接通道，速度大概是前者的八分之一，我在这儿画的绿色线，就是NVLink。

The GPUs have also a much slower connectivity, which is typically about eight times slower, which is so so the green that I drew here in GPU cases is the NVLink.

更广义地说，这叫‘扩展上行网络’。

More generally, it's called the scale up network.

这就是扩展上行网络。

This is the scale up network.

通常你还会有一个‘扩展出域网络’，用来连到数据中心的交换机。

You will typically also have a scale out network, which allows you to connect to, like, some data center switch.

就是数据中心交换机。

So data center switch.

然后所有GPU都会通过某种方式连到某个数据中心交换机上。

And then all of the GPUs will have some connectivity up to some data center switch somewhere.

对。

Mhmm.

但这个就是所谓的扩展网络，速度大概慢八倍，说白了就是挺慢的。

But this is this is about times like, this is the scale out, and it tends to be about about eight times slower in in bad words.

所以如果你要把专家混合层跨两个机架部署，问题就来了——这边一半的GPU得跟那边的GPU通信。

So the challenge, if you want to, for example, lay out a mixture of expert layer across two racks, is that half of the GPUs here are gonna be wanting talk to the GPUs here.

平均来看，当我看这些GPU上的token要往哪儿走时，有一半的token其实只想在机架内部走。

And so, like, half of the like, just on average, like, when I look at where the tokens on on these GPUs want to go, half of the tokens are gonna wanna go inside the rack.

这没问题。

That's great.

它们可以用高速的机架内互联网络。

They can use the the fast scale up network.

但另一半的token得跨机架去另一个机架，这就麻烦了。

But half the tokens are gonna want to leave the rack and go to the other rack, and that's not as good.

它们只能用慢得多的网络。

They're gonna need to use a much slower network.

所以这成了全连接模式的瓶颈。

And so that becomes the bottleneck on on the all to all pattern.

另一种做法是，干脆在这儿放个大交换机，把两个机架全都连到一个更大的交换机上。

A different choice would be, well, why don't I, like, have a big switch here and sort of, like and connect everything to some big switching like, much bigger switch that actually combines the two racks together.

这方面的想法不少，但一般来说，之所以用分层交换结构而不是一个大交换机，就是为了避免线缆太拥挤。

There are many ideas in this direction, but in general, it becomes the reason you have this sort of hierarchy of switches rather than one big switch is to manage the cabling congestion.

你得布大量的线。

You just need to run a large number of cables.

不好意思。

Sorry.

你刚才问的，是不是就是为什么不用更大的高速互联？

Is this is that question you just asked basically, why isn't it a bigger scale up?

是的。

Yeah.

没错。

Exactly.

那为啥不直接搞上百万个芯片，直接往上堆呢？

Why not why not just, like, have, like, a million chips and scale up or something like that?

到底是什么变了，让英伟达从Hopper的8个，到Blackwell的72个，现在Reuben据说要到……我不太确定。

What has changed that has allowed NVIDIA to go from Hopper was eight, then Blackwell is 72, and now Reuben will be I don't know.

五百个？

Is it 500?

五百多吧。

500 and something.

对。

Yeah.

到底是什么让这一切成为可能？

What what has allowed that to happen?

从Hopper到Blackwell，最主要的就是决定把封装形式从托盘改成机架，就是把原来那种托盘换成机架。

So from Hopper to to Blackwell is is mostly just the decision to switch from trays as the form factor, one of these as a tray, just just switching to racks as the form factor.

这是个产品决策。

That's a product decision.

其实那里并没有什么重大的技术障碍。

It's there wasn't a substantial technical barrier there.

从64个直接跳到500个左右，中间确实有点詹森式算法的成分，但至少真有四倍的提升，这来自更复杂、更难搞的机架设计。

Switching from from the, like, 64 to to 500 or so, there's a bit of Jensen math there, but there is at least a a genuine four x increase, which is coming from a much more complicated and difficult rack design.

所以这其实是全新的物理设计，要塞进更多线缆。

And so that that is actually, like, new new physical design to run more cables.

线缆的复杂性就在于得搞清楚哪根线连到哪，哪个信号走哪条路。

And the cable complication is just the the the the cost figuring out which cable hops to which or, like, which signal

对。

Yeah.

咱们不妨仔细看看这线缆密度的问题。

I mean, it's let's sort of zoom in on this and look at the the wire density.

我再画一次这个图，这次画个更清楚、更大的版本，方便咱们分析。

I'll draw this diagram just once more so we have a bit of a cleaner version to work with, I mean, a larger version.

假设中间放一些交换机。

Let's say have some switches in the middle.

对。

Yep.

我先在每边放两个GPU，或者每边放两块GPU托盘。

And let's say I'm gonna have initially, I'm gonna start with just two GPUs on each side or two two trays of GPUs on each side.

假设每块托盘要引出两条线。

And let's say maybe each tray wants to have two cables coming out of it.

所以我得布一些垂直的线，像这样连到交换机上。

So I get some kind of I I physically run vertical cables that look like this running onto the switches.

如果我想把机架里的GPU数量翻倍，那线缆密度也得翻一倍。

Now if I wanna double the number of GPUs in a rack, I need to run, like, literally twice the density of cables.

所以这些线也得布上去。

So I need to run, yeah, these as well.

这个问题可能有点傻。

Extremely naive question.

是的。

Yeah.

但如果你看看实际的数据中心

But if you look at a physical data center

对的

Mhmm.

感觉机架里面空间挺大的

It seems like there's a lot of space within a rack.

我不太确定

I don't know.

那些线缆看起来特别粗，而且

Just like the cables are, like, really big and

没错

Yeah.

机架外面其实是有空间的

So there is space outside the rack.

但机架里面呢，这些机架越来越紧凑，空间几乎被塞满了

Inside the rack like, these racks are, like I mean, as they become more optimized, these racks are very tight.

所以说吧。

So Yeah.

从托盘到机架，再到机架背板，连接器的密度越来越高，背板本身的密度也特别高。

There's connector density going from from from the tray into the rack and the rack's backplane, and then the backplane itself has a has a really high density.

明白了。

Alright.

还有其他一些物理限制，比如电缆的弯曲半径。

There are other physical constraints, including, like, bend radius of cables.

你可不想把它弄断，等等。

Like, you don't wanna snap them and so on.

对。

Yep.

好的。

Okay.

所以实际上就是得留出足够的物理空间来布线

So it's literally the physical space to put a cable

是的

Yeah.

这确实是个限制

That's constraining

没错

it.

是的

Yeah.

我完全没想到

I had no idea.

挺有意思的

Interesting.

真没想到，机架这么大，怎么就不能多塞点线呢？

That seems surprising that, like, how about the rack is so big, and they're just, like we can't just stuff more cables in there.

是啊

Yeah.

所以啊，机架设计真不是我的专长。

So, I mean, rack design is not my expertise.

但当我跟人聊的时候，问他们现在面临哪些限制，其实都是在权衡几个关键的物理因素：比如，你最想优化的是什么？

But, like, when I talk to to folks and what are the constraints they're up against, it's it's a combination of so what are the big physical things you're optimizing for?

空间、机架的重量。

Space, weight of the rack.

机架其实特别重，所以得有足够的顶部金属支撑，不然会变形塌下来。

Like, it's actually really heavy, and so, like, you need enough metal top to not sag and fall.

但你要是加更多金属，重量又上去了，接着就是功耗和散热的问题。

But then you add more metal and it's heavier, and then power and cooling.

所以这些因素全在互相拉扯，现代机架的设计已经把所有这些物理极限都推到极致了。

And so all of those are competing for like, modern racks are pushing all of those to very extreme physical limits.

深度工作本质上就是让人不舒服的。

Deep work is, by its nature, quite aversive.

所以像 Slack 和邮件这种看起来像工作的事，反而成了逃避的借口。

So even things which seem like work, like Slack and email, can be easy ways to distract yourself.

我经常真想把互联网直接关了。

So I often wish that I could just turn the Internet off.

但要是我准备面试，就算手头有论文和书，还是能靠大模型来回讨论，把概念理清楚，再顺藤摸瓜查后续内容，特别管用。

But if I'm prepping for an interview, even if I have the papers and books on hand, it's still super useful to be able to do a back and forth on the LLMs so I can break down concepts and research follow ups.

谷歌新出的Gemma 4，是第一个让我能完全离线、专心做事的开源模型。

Google's new Gemma four is the first open model that allows me to have this kind of fully disconnected focus machine.

它够小，能在我的笔记本上跑，但又够强，真能派上用场。

It's small enough to run on my laptop, but good enough to actually be useful.

所以为了准备这期节目，我下载了Reiner的《Scaling》这本书，然后断了网。

So to prep for this episode, I downloaded Reiner's scaling book and shut off the Internet.

我靠Gemma帮我理解内容、回答问题，特别顺手。

I was able to have Gemma help me understand and answer my questions.

如果你想找个能跑在笔记本甚至手机上的本地大模型，真该试试Gemma 4。

If you want an LLM that you can run locally on your laptop or even your phone, you should check out Gemma four.

GPT-4是啥时候发布的来着？

When was GPT four released again?

是2022年还是2023年？

It was 2022 or 2023?

2023年。

'3.

对，就是2023年。

'3.

明白了。

Okay.

据说它的参数超过了万亿级别。

And it was rumored to be over 1,000,000,000,000 parameters.

但直到最近这半年，才陆续有模型的参数量明显超过三年前发布的产品。

And it seems like only now and within the last six months have models been getting released that are significantly more parameters than a model released three years ago.

是的。

Yeah.

按理说这中间早就该有规模上的突破了，可我们其实是在等足够多内存的服务器，好装下五万亿参数的模型，外加它的KV缓存——得撑得住大量用户、长序列的请求，要是做强化学习，情况也差不多，得把所有待处理任务的缓存都存下来。

When supposedly there should have been this scaling in the meantime, is the reason that we were just waiting for racks with enough memory to hold the 5,000,000,000,000 parameter model along with its k b cache for enough, you know, users for a full for a lot of sequences or RL if you're doing RL, kind of a similar consideration of actually holding the KBChash for all the the the the batch of problems you're trying to solve.

你看Hopper架构，当时用八张Hopper，到2022年总共也就640GB显存。

So if you look at, like, Hopper, you had eight Hoppers, and I think the that's 640 gigabytes as of 2022.

对。

Yeah.

直到Blackwell终于上线了，那是啥时候？2020年？

With Blackwall finally, which was deployed, what, 2020

最近才出的。

Very recently.

可能是去年吧。

Maybe last year.

去年？

Last year?

对。

Yeah.

这才终于有了能撑起十到二十TB显存的规模。

You finally have a scale a up with on the order of, like, ten, twenty terabytes Mhmm.

这足够支撑一个五万亿参数的模型加上KBCh了。

Which is enough for, like, a five t model plus KBCh.

对。

Yeah.

在更大规模的互联域里部署，简直是重大突破。

Deploying in in larger scale up domains is a huge unlock.

没错。

Yeah.

我这儿画的是英伟达Blackwell的部署架构。

I mean, I've drawn here the sort of NVIDIA Blackwell deployment.

谷歌的部署其实早就有了超大规模的互联域，

The Google deployment has actually had very large scale up domains for

这也解释了为什么Gemini看起来领先了。

And that that also explains why Gemini was seemed to be ahead.

比如Gemini 2.5是真成功了，还是只是它的预训练比其他实验室持续得更久？

Like, was Gemini 2.5 was a successful or it just seems like Gemini has that successful pre train for longer than some of the other labs.

我没在现场，不太确定有多少优势是来自成功提升了稀疏率，这确实有可能。

I not having been there at the time, I'm not sure how much is coming from, like, successfully deploying higher sparsity ratios, which which could be.

也可能是，毕竟在模型设计上还有很多细节，比如专家混合到底该怎么搞。

It could also be I mean, there's a whole bunch of actual modeling things of, like, specifically, how do you do the mixture of experts?

我们看到DeepSeek的专家混合模型就提到，真正突破点在于激活更多专家，但每个专家更精细。

We've seen the DeepSeek like, the DeepSeek mixture of experts has said, actually, activate more experts, but finer grained experts was a big innovation.

我确信模型架构和训练数据上还有很多其他创新。

I'm sure that there are many other innovations on the model architecture as well as on the training data.

这些因素混在一起，真的很难分得清谁贡献更大。

It's kind of hard to disentangle all of them.

但就目前来看，能激活的参数量受限于算力成本，而总参数量则受限于扩展规模。

But what shows up in terms of the limits of what you can do, The the active parameters, as we saw, is limited by the compute cost, and then the total parameters is limited by the scale up size.

对。

Yep.

当你在单个扩展域内运行时，这个限制是特别影响前向传播、反向传播，还是预填充和解码阶段？

When you're operating within a single scale up domain, is that a consideration specifically for either forward or backward or specifically for pre fill versus decode?

还是说，无论什么情况，最好都保持在同一个规模内？

Or is it is it preferred to always be within a scale up?

对。

Yeah.

不管你是做预训练、RL生成，还是给用户做推理，都一样？

Whatever kind of workload you have, whether you're doing a pre training run or whether you're doing RL generation or whether you're doing inference for users?

是的。

Yeah.

这问题真有意思。

Really interesting.

那这么说吧。

So okay.

要回答这个问题，我们得聊聊通信模式——之前我们谈过专家混合的通信模式。

So to answer that question, we're gonna need to talk about the communication patterns that so we've talked about the mixture of expert communication pattern.

那就是全连接通信。

That is this all to all.

就是全连接。

There's all to all.

全连接。

All to all.

全连接特别依赖完全互联，也就是我们刚才说的，最好都在同一个机架里。

All to all very strongly favors full connectivity, which what we've kind of just shown here, and it favors being within one rack.

除了专家并行，还有别的并行方式。

There are other kinds of parallelism besides expert parallelism, which which we just showed here.

文献里常见的是张量并行。

In the literature is tensor parallelism.

现在趋势是专家模块越来越小，这种模式已经没那么重要了，可以忽略。

This is with the trend towards smaller experts, this has become much less relevant, so we can ignore that.

不过我们还有另外两种方式：数据并行和流水线并行。

But the other two things that we have available are data parallelism and pipeline parallelism.

它们反而更适合跨多个机架来部署。

And they are actually much they can be a much better fit for using multiple racks.

那我们先专门聊聊流水线并行。

So let's focus on pipeline parallelism specifically.

这只是一个 MoE 层。

This is one layer of MoE.

上面我还要再加一百多层。

I'm gonna have, like, a 100 more layers up above.

比如在这里，我完全可以换到另一个机架上。

I could decide at this point, for example, to move to a different rack change rack.

那这样会不会变成通信瓶颈？

Now, is that gonna become a communication bottleneck?

我们其实可以算出什么时候会变成瓶颈。

So we can actually just solve for when this becomes a communication bottleneck.

不过在算之前，咱们先画个图，理一理数据流的路径。

But before we do that algebraically, like, let's just sort of visualize it out and and sketch the path.

我们会有一堆这样的 MoE 层，这边一个，那边一个，就这么继续下去。

So we're gonna have a bunch this is another MoE layer, and we're gonna have another MoE layer here, and so on.

所以假设我在这儿换机架，然后过几层之后，我又在这儿换一次机架。

So let's say I change rack here, and then some number of layers later, I change rack here as well.

我们判断这种换机架的地方会不会成为通信瓶颈的方法，就是把横向扩展的带宽需求和纵向扩展的带宽需求做个对比。

So our our our methodology that we're gonna use to determine whether we have a communication bottleneck in this, like, in this point where we change rack is we're gonna compare the this this is the scale out scale out bandwidth requirements to the scale up bandwidth requirements.

对。

Mhmm.

咱们来试试看。

So let's try this.

关键点在于，这儿的跨机架传输量大得多。

And and, I mean, the the hint is gonna be that there's a lot more transcends here.

这边要传好多东西，而那边就只传一个。

Like, we're sending many things here, whereas we're only sending one thing here.

而且可能还要反复这么干。

And then we're also maybe doing it many times.

这才是真正造成区别的地方。

That's so that's going to be the what makes the difference.

我能猜一下吗？

Can I try to guess?

对。

Yeah.

就是好奇看看我有没有真懂。

Just out of curiosity to see if I'm actually understanding.

你好像把批量大小发到机架里了？

It seems like you're sending, like, batch size into the rack.

是这里吗？

In here?

没错。

Yes.

但机架内部的通信量大概是批量大小乘以GPU数量。

But the communication within a rack is sort of batch size times number of GPUs.

对。

Yeah.

所以是实际激活的GPU数量。

So number of activated GPUs.

对吧？

Right?

我根本就没往这个GPU发数据。

So, like, I I don't send to this GPU at all.

对吧？

Right?

所以在这个图里，数据量从一变成了三倍，直接爆炸了。

So there's an explosion from one to, like, three times larger here in in this diagram.

是的。

Yeah.

关键是，我根本没必要往这个GPU发数据，这就省了不少东西。

The key thing is that I I didn't even need to send to this GPU at all, and so that's a big saving.

明白了。

I see.

对。

Yeah.

好的。

Okay.

那我们来聊聊，扩展性能的瓶颈到底有多大？是向上扩展比横向扩展慢多少？

So we're gonna talk through sort of how much more what is the slowdown of to what extent is scale up a bottleneck over scale over scale out?

我们直接来看向上扩展所花时间与横向扩展所花时间的比值。

So we will directly jump to the ratio of the time spent on scale up, time on scale up, over the time spent on scale out.

这就是我们今天要讨论的指标。

This is this is the quantity we're talking about.

首先，一般来说，向上扩展的速度是横向扩展的八倍左右。

And the first consideration is that the scale up is, like scale up is is eight times faster than scale out, generally.

所以，如果带宽一样的话，理论上这个比值应该是八分之一，这完全是带宽带来的差异。

And so at a baseline, if the bandwidths were the same, we would have this one one over eight, which is coming from bandwidth bandwidth.

但实际中，我们传输的数据量还会进一步扩大。

But then we have some amount of expansion in in in how much data we're sending.

所以如果有一个token进来，是这样的。

So if one token comes in here, Mhmm.

那么在这个深度稀疏模型里，这个token会被分发给大约32个专家，或者16个专家。

Then this one token gets routed to in the deep sea case, it'll get routed to maybe 32 experts or or 16 experts.

它会被分配到一定数量的专家那里。

It gets routed to some number of experts.

这就是被激活的专家数量，被激活的专家数量。

So this is the number of activated experts, number of activated experts.

而且这个情况在多个不同层上也会重复发生。

And then it also this this same thing applies on multiple different layers.

比如我可能要跑两层。

So maybe I'm gonna run two layers.

所以每个阶段的层数也会乘上一个倍数。

So there's also multiple times number of layers per stage.

然后整个计算还得再乘以二，因为

And just need to multiply the whole thing by two for the

对于最优情况来说。

For the the optimal.

对。

Yeah.

没错。

Yes.

是的。

Yes.

还有一个两倍的因子。

And there's a factor of two.

谢谢。

Thank you.

我们希望扩容所需的时间比扩展所需的时间更长，因为扩容才是更关键、更宝贵的资源。

So what we would like is the for the scale up time to be greater than the scale out time, because, like, the scale up time is the more important and precious resource.

所以我们希望这个数值至少大于或等于一。

And so we just we want this one we would like this number to be greater than or equal to one.

这其实没那么难。

And this really doesn't seem hard.

咱们只需要克服一个8倍的差距，也就是说，这三个因素的乘积得大于8。

Like, we've we've there's just a factor of eight that we need to overcome, so we need the product of these three things to be bigger than eight.

通常情况下，我们会有相当多的激活专家。

Typically, we have a fairly large number of activated experts.

光是这个数字本身就能到8。

It could be eight by itself.

然后我们可以大幅增加每阶段的层数，直到满足这个条件。

And then we can increase the number of layers per stage a lot until until we satisfy this.

明白了。

I see.

所以最终的效果就是，我可以整个流水线都用机架来搭，一个机架负责一层，接着换下一个机架做下一层，再换下一个。

So what this ends up looking like is that I can, in fact, have an entire pipeline of racks where one rack does one layer, and then I move on to the next rack, and I do another layer, and then I move on to the next rack.

我还能再加一层。

I can do another layer.

我觉得特别有意思的是，实际效果最好的并行策略，居然跟硬件架构长得一模一样。

It's interesting to me that the best parallelism strategy in practice ends up being one which physically resembles the actual architecture.

根本不是什么天马行空的脑洞设计。

It's not some galaxy brain thing.

你懂的吧？

You know?

就是，咱们有专家模块，

It's like, oh, we have experts.

那就把它们分别放到不同的GPU上。

We're gonna put them on g different GPUs.

咱们有不同层，

Oh, we have different layers.

那就把它们分到不同的机架上。

We're gonna put them on different racks.

我觉得这挺有意思的，物理结构和

Isn't that I feel like that's interesting that the physical and

模型架构和实际的划分方式完全对得上。

The the the model architecture matches like, the the the cutting matches the model architecture.

是的。

Yeah.

没错。

Exactly.

对。

Yeah.

本来搞成张量并行那一套乱七八糟的也行，但没那么干。

I mean, it could have been something wackier with tensor parallelism and whatever.

对。

Yeah.

我觉得可以这么理解，就是说——

So, I mean, I think a way to think of it is I mean, okay.

从那种‘银河脑洞’的角度来看，模型扩展到底有哪些不同的维度？

The Galaxy Brain way to think of it is, like, what are all the different dimensions in which a model is scaled up?

所以它是通过增加层数来扩展的。

And so there is it is scaled up by layers.

它是通过模型的维度来扩展的。

It is scaled up by the, like, demodel dimension.

它是通过DFF维度来扩展的。

It is scaled up by the DFF dimension.

它是通过专家数量来扩展的。

It is scaled up by the number of experts.

这些参数中的每一个，你都可以选择沿着它们做切分。

Every single one of those numbers, you can choose to cut along.

只要这些数值足够大，沿着它们切分就会变得划算。

And if those numbers are big enough, it eventually becomes profitable to cut along there.

是的。

Yeah.

我们已经选了其中两个。

And we have selected two of them.

另外两个，按常规模型的规模来看，根本赚不到钱。

The other two, in the way typical models are typically sized, are not profitable.

有人提到伊利亚说过，现在别搞流水线并行。

So there's talk about Ilya where he says, today, know not to do pipeline parallelism.

霍拉斯给我们这群人做过一场讲座，我真讨厌这种像医生随口说的话。

And Horace gave my friends and me, I hate that it's something like a doctor's quote.

但他跟我们讲了各种并行方式，说流水线并行的问题在于，除了那些空泡，还会带来一堆架构上的限制。

But he gave us a lecture on these different kinds of parallelisms, and he said, the problem with pipeline parallelism is that it other than the bubbles, constrain it creates these architectural constraints.

对。

Yes.

比如Kimmy，它的残差连接会回头关注好几层之前的输出什么的。

On like, Kimmy, for example, has these residuals where attention attends to the A fewer back or something.

没错。

Yeah.

所以这种结构在实际实现上就特别麻烦。

Layers a few back and so that it becomes hard to implement in this way.

是的。

Yeah.

所以我们其实都没说清楚，流水线到底能带来什么实际好处。

So and I guess we didn't really fully articulate even what is the benefit that we're getting from pipelining.

对。

Yeah.

这些复杂性确实是实实在在的。

And so these complexities are real.

流水线太折腾人了。

Pipelining is a massive hassle.

但它确实有些好处。

It's but it does give you some benefits.

你可以自己权衡一下，这些好处值不值得付出这些代价。

And then you can then decide whether those benefits are worth the costs.

最明显的好处是在推理阶段，但训练阶段可能收益更大。

Biggest benefit that shows up so it has some benefits in inference, maybe bigger benefits in training.