雷纳·波普——自下而上的芯片设计

这样做的目的，是为了让这个优势项随着 y 增长。

And this is sort of the intention so that we get this advantage term going as y.

我们已经把乘法操作安排好了。

So we've laid down the multiplications.

接下来我们要引入一个大小为二的向量，这正好符合我们对通信量的要求。

We're going to want to bring in a vector of size two, so that sort of already is in line with our comms target.

这样没问题。

That's fine.

但问题是，这个矩阵的通信开销还是超出了我们的 x 预算。

However, we need to somehow manage the communication of this matrix, which which which does which exceeds our budget of of x.

所以在人工智能的场景下，这个矩阵其实会长时间保持不变。

And so the idea is that in an AI context, this matrix is actually going to stay fixed for a long period of time.

因此，我们没必要每次都从外部加载它——这里有个寄存器文件，我们不希望从这里频繁输出大量数据。

And so instead of like bringing it in from the outside, so we've got some register file sitting over here, We don't want to have the amount of stuff coming out of this register file.

这个部分，我们希望它的规模能控制在 x 的水平上。

This is the term that we want to go sort of as x in some sense.

我们不想每个周期都从寄存器文件中加载整个矩阵，因为资源不够。

We don't want to bring this full matrix in from the register file every cycle because we don't have enough.

那样的话，寄存器文件到处理单元之间的布线开销会太大。

That would cost too much in terms of wiring from the register file.

所以我们用的关键方法是，把这个矩阵直接存在脉动阵列本地。

And so instead we're gonna store our key trick is that this matrix can be stored locally to the systolic array.

我们会把数字零、一、二、三存在一个类似寄存器的硬件单元里，反复复用它们来处理大量不同的向量。

And so we'll store these numbers zero, one, two, and three just like a gate called a register that like physically stores these numbers and we're gonna reuse these numbers over and over again for a for a large number of different different vectors.

这里的优化在于，矩阵乘法的特性允许我们把这种平方形式的计算直接放在运算逻辑旁边。

And so the optimization here is that like the nature of matrix multiplication is you can store this like square quadratic thing directly where the logic is happening Mhmm.

这比输入向量的维度更高，或者说多出一个维度，而输入向量是不断切换的。

And which is like higher dimension than the or has an extra dimension compared to the inputs which you keep swapping in and out.

没错。

That's

对。

right.

我的意思是，矩阵乘法的本质就是，要算很多次乘法才能得到一个结果。

I mean, is the nature of what a matrix multiplication is, is that you do, a lot of multiplication to get one value out.

点积其实就是很多次乘法相加的结果。

Like a dot product is the result of a lot of multiplications.

所以这个优化就是，你可以在输出一个值之前，先堆积一大堆乘法运算。

And so that optimization means that you you're just like, can stuff a lot of multiplication in before you get some value out of it.

没错。

That's right.

没错。

That's right.

为了更清楚地说明这一点，我把这里的3和2对调了。

So just to complete the picture here of concretely how that looks, I swapped the three and the two here.

3和2。

Three and two.

就像这样，0和3会分别乘以3和7，然后我们在列的方向上形成一个点积。

So just like this zero and three is going to multiply by the three and the seven, and so we're going to form a dot product sort of along columns here.

所以我们得把3和7输入到这里面。

So somehow we're going to feed a three and a seven in here.

这些数会参与乘法运算，既影响这边的乘法，也影响那边的乘法。

These participate in sort of this feeds into this multiplication and also feeds into this multiplication.

同样，那个3也会同时参与这边和那边的运算。

Likewise the three feeds into here and also into here.

然后我们会沿着这一列做累加，从顶部开始输入0，最后从底部输出结果。

And then we're gonna sum sort of along here with like starting at the top of a column we feed in zeros and then coming out the bottom we get results coming out.

简单来说，矩阵中每一列都在做点积运算，这正好对应了这里脉动阵列的空间处理方式。

So sort of just to visually see what we've got, there's a dot product that is performed along columns in a matrix and that sort of maps exactly to what is done spatially in the systolic array here.

所以这是一个垂直方向的点积，那一个是另一个也垂直累加的点积。

So this is one dot product summed vertically and this is a second dot product also summed vertically.

那需要输入和输出寄存器文件的数据是什么呢？

And then what is the data that needs to go into and out of the register file?

这里输出端会输出一定量的数据，输入端也会输入同样数量的数据。

We have x amount of data that's coming out here on the output, and we also have sort of this data coming from the input, x amount of data from the input.

所以，至少从输入和输出向量的角度来看，我们已经实现了目标——进入和离开寄存器文件的数据量只有原来的 x 倍。

And so with respect to the input and output vectors at least, we've sort of met our goal of having only x as much data going in and out of the register file.

但这留下了一个问题：我说过权重矩阵是存在 systolic 数组内部的，那它最初是怎么进去的呢？

This leaves open the question of, like, I said that the weight matrix is stored locally in the systolic array, how did it get there in the first place?

说白了，芯片启动的时候总得把数据加载进去，那这些数据到底从哪儿来的？

It's sort of like at some point you need to boot your chip and populate this data, and so where did that come from?

诀窍就是，我们慢慢来。

The trick is just we just do it very slowly.

所以我们一点点地，把数据慢慢流入 systolic 数组。

So we we very slowly trickle feed it in into the systolic array.

最简单的方法就是用一条菊花链：把一个数输进去，下一个时钟周期，它就会自动往下传到数组的下一个位置。

The sort of the simplest strategy is that we we sort of run this daisy chain that says like feed a number into here, and then and then on the next clock cycle, it'll move down to the next entry with the systolic array.

我们可以同时在每一列都这么做，这样一来，数据也会从这里进来，带宽需求又增加了大约 x 倍。

And so we can do that in every in every column in parallel, and that gives us sort of this is also going to come from here, and this is going be another factor of approximately x units of bandwidth coming in.

你能再重复一遍刚才那句话吗？

Can you just would you mind repeating that sentence one more time?

所以我们知道，往矩阵里输入数据的频率其实很低。

So we're sort of like, we know that we're going to be bringing in numbers only rarely into the matrix.

所以我们只需要设计一种方案，让真正穿过这个 systolic 数组边界、进入内部的连线数量保持在 x 以内，别超过它。一个特别简单的办法就是，把数据先送到 systolic 数组的最上排。

And so we just want to come up with any construction at all such that the amount of wiring that actually feeds into sort of crosses this boundary of this systolic array, like this boundary right here, just we want to keep that bounded to x and not go And as x so a particularly simple strategy is that we sort of bring in a number into the top row of the systolic array.

这样一次时钟周期就能完成。

That's what we can do in one clock cycle.

接下来的 y 个时钟周期里，每次我们都把最上排的数据再送一遍，同时把其他所有行往下移一位。

And then for y consecutive clock cycles, we're gonna be bringing in the top row every time and then sort of shift all of the other rows down by one.

这样一来，从那个昂贵的寄存器文件里出来的连线就只用 x 倍的带宽，而不是 x 的平方倍。

And that keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than x one.

明白了。

I see.

好的。

Okay.

关于通信，现在有两个问题。

So there's two questions in terms of communication.

通信方面

There's like communication

时间

time

还有通信带宽。

and then there's communication bandwidth.

对。

Yes.

你说得对，因为我们只加载一次，那就尽量减少带宽。

And you're saying since we're only gonna be loading this in once, let's maximize let's minimize bandwidth.

没错。

That's right.

带宽越小，芯片面积就越小。

Because minimum bandwidth equals die area.

是的。

Mhmm.

咱们可以慢慢通过更窄的通道加载，因为这个值我们会一直保留着

And let's just, like, load it in slowly over, like, smaller lanes because we're just gonna keep this value in there for

挺长时间的。

a while.

没错。

Exactly.

这挺有意思的。

Interesting.

我记得上次我们聊多芯片推理时，核心目标其实是提升单位通信带宽下的计算量，也就是单位通信量能做的运算更多。

So it's interesting to me that when we were talking last time about inference across many chips, the big high level thing we're trying to optimize for is increase the amount of compute per memory bandwidth, that is to say per communication.

现在这里也是，我们想提高实际乘法或加法操作的数量，相对于在寄存器和逻辑单元之间传输数据的开销。

And here also we're trying to increase the amount of like actual multiplies or actual additions relative to transporting information from registers to the logic.

所以这两种情况，本质上都是在最大化计算与通信的比例。

So in both cases, you're trying to maximize compute relative to communication.

对。

Yeah.

这种现象在整个系统栈中随处可见。

This this shows up sort of all the way up and down the stack.

这算是比较底层的了，差不多接近晶体管层面。

This is sort of close to the bottom, sort of like to to the gates.

甚至还有更接近晶体管层面的版本，比如你选择的数值精度格式。

There's sort of a a version that's maybe even closer to the gates of just like even the precision of number format that you choose to use.

我们之前也观察到同样的效果。

We saw that same effect.

无论是这个ALU的精度，还是矩阵的规模，都存在一种平方与立方的关系，或者平方项对线性项的对比。

There's like a square cube law or a like squared versus linear term going on both in just purely the precision of this ALU, but then also in terms of the size of the matrix.

是的。

Yeah.

非常有意思。

Very interesting.

这个模块是我们上一层更大的单元，之前是乘法电路，再往上就是规模很大的脉动阵列。

So this unit is sort of the next bigger unit we had, like the multiplication circuit, and then on top of that, we have a, like, a pretty large systolic array.

我画的是一个2乘2的结构，但比如在早期的TPU中，这种电路单元被描述为128乘128的规模。

I drew it as two by two, but in, like, for example, older TPUs, they were described as a 128 by a 128 of of this circuit shown here.

这种电路结构是目前所知实现矩阵乘法最高效的方案。

And this circuit ends up being this is the the most efficient known mechanism for circuit for implementing a matrix multiply.

我明白了。

I see.

我们之前讨论过，看起来很明显，应该尽量提高计算量相对于通信量的比例。

So we we've talked about sort of it seems obvious that you should try to maximize compute relative to communication.

但有哪些不太明显的权衡，才是真正让人夜不能寐的问题？比如到底该选X还是选Y？

What are non obvious trade offs that actually are, know, keep you up at night about what should we do x or should we do y?

答案其实并不明显。

And it's not obvious what the answer is.

是的。

Yeah.

我觉得芯片设计里的大多数决策，本质上都是关于尺寸的取舍。

So I mean, I think most of the decisions in chip design are sizing decisions.

所以，就我们目前画出的这些来看，所有的AI芯片都包含这种电路。

And so already in what we've drawn so far, so AI chips all have this circuit in it.

它们都有一个脉动阵列，旁边还配有寄存器文件，用来提供输入和输出数据。

They have a systolic array, and then somewhere near it, a register file providing inputs and outputs.

在这个范围内，有两个关键的设计问题：脉动阵列该做多大？寄存器文件又该多大？

The two sort of even within this scope sizing questions that you have are how big should I make my systolic array and how big should I make the register file?

而脉动阵列的尺寸选择，其实和这个问题紧密相关——你可以这么想：我得先定一个预算，芯片总面积里有多少比例要分配给数据搬运。

And then the trade off for the size of the systolic array, actually, these two questions are coupled, is one way to think of it is to say, I'm gonna, like, have a budget for how much what percentage of my chip area I want to spend on data movement.

比如我决定让数据搬运占10%，脉动阵列占90%。

So maybe I just say that I want this to be 10% and the systolic array to be 90%.

然后我就能根据这个比例来确定寄存器文件的大小。

And then sort of I can size my register file.

寄存器文件越大，灵活性就越高。

Bigger register files are more flexible.

它能让我运行更多样化的任务，整体应用性能也能提升更多。

They allow me to run sort of more I can get more application level performance out.

但这样一来，用于系统本身的面积就被压缩了。

But then they sort of take away from this area spent on the system.

对吧？

Right?

是的。

Yeah.

有道理。

Makes sense.

我最近办了一个征文比赛，让参与者写一写我认为当前AI领域最重要的几个未解问题。

I recently ran an essay contest where I asked people to write about what I consider to be some of the biggest open questions about AI.

投稿窗口上周刚关闭，我用Cursor做了几个不同的界面来帮忙审阅这些文章。

The submission window closed last week, so I used Cursor to create a couple of different interfaces to help me review the entries.

其中一个界面会匿名处理投稿，隐藏掉不必要的信息。

One interface anonymizes submissions and hides unnecessary information.

它让我能按问题分类整理回复，添加批注，还能记录评分。

It lets me group responses by question, add notes, and record my scores.

另一个界面帮我筛选那些适合研究员岗位的申请者。

The other interface helps me review entrants who be want considered the researcher role that I'm hiring for.

这个界面把申请人的文章和简历、个人网站并排显示，让我能一眼看完所有信息。

The UI puts the applicant's essay right next to their resume and their personal website so that I can see everything at once.

Crusoe 的工具在帮助这些模型观察和优化界面方面真的很出色。

Crusoe's harness is really good at helping these models see and improve their UIs.

我亲眼看着它在内置浏览器里渲染界面、截屏、点击各个板块，不断迭代改进。

I watched it render these interfaces in the built in browser, take screenshots, click through sections, and keep iterating.

现在，我大部分工作都在 Cursor 里完成了。

At this point, Cursor is where I do most of my work.

不管是读论文、做可视化，还是开发申请评审界面，或者为黑板课做闪卡，Cursor 都能让 AI 轻松看清我在看什么，帮我理解内容，和我一起协作。

Whether I'm reading and visualizing a bunch of research papers or coding up an interface to review applications or making flashcards for my Blackboard lectures, Cursor just makes it very easy for an AI to look at whatever I'm looking at and help me understand it and work with me on it.

所以，不管你正在做什么，都建议你用 Cursor 来做。

So whatever you're working on, you should do it in Cursor.

去 cursor.com/dwarkesh 看看吧。

Go to cursor.com/dwarkesh.

芯片的时钟周期是怎么回事？

Where does the clock cycle of a chip come in?

那这个时钟周期是由什么决定的？

What determines what that is?

对。

Yeah.

芯片的时钟周期到底是什么？

And what is the clock cycle of a chip?

首先得说，芯片的并行度高得惊人。

So I guess at baseline it's sort of worth observing that chips are incredibly, incredibly parallel.

对吧？

Right?

你想想，一颗芯片里有上百亿个晶体管，要处理这么大规模的并行任务，就必须让各个单元同步运作。

You've got 100,000,000,000 transistors in a A key thing that you need to do whenever you have massive parallelism is you need to synchronize between the different parallel units.

在软件里，我们通常用像互斥锁这样开销很大的同步机制。

In software, typically, have these very expensive synchronization methods like a mutex.

所以一个线程做完自己的任务后，会去获取内存里某个锁，然后通知另一个线程它已经完成了。

So one thread will finish what it's doing, it will ink, like it will grab a lock somewhere stored in memory and then notify the other thread that it's done.

而芯片的做法完全不同，它大约每纳秒就会让整个芯片的电路暂停一下，同步一次，也就是说，几乎每纳秒都在同步。

On chips, take a very different approach and say that every nanosecond or so, all circuitry in the chip will kind of pause for a moment and then synchronize every so it actually synchronizes every single nanosecond or so.

这就是时钟周期。

And so that is the clock cycle.

整个芯片通常会一次性、齐步地进入下一个操作阶段。

The entire chip, typically all in sort of one fell swoop, goes in lockstep to the next operation that happens.

在电路设计中，这通常表现为时钟信号通过寄存器来控制，这些寄存器就是我们之前画过的存储单元。

And so what this looks like in circuitry is that you will have it's typically drawn so the clock is sort of mediated by registers, which are these storage devices that we've drawn elsewhere.

你可以这样理解：我有一个存储单元，里面存着一个比特，可能是0或者1。

And the way to think of it is that I have I have some storage which is storing like a bit, which might be zero or one.

然后我有一大块逻辑电路，比如一个脉动阵列、乘法器之类的。

And then I have some sort of cloud of logic which maybe is like this systolic array or this multiplier or something like that.

这些电路会产生输出结果。

And then I've got some and that's gonna produce some output.

所以我的输入端，有一堆数据传入这个逻辑模块。

So my inputs, I've got a bunch of inputs feeding to this cloud of logic.

然后过一会儿，会有一个输出寄存器接收这些结果。

And then eventually later there's gonna be some output register that this writes to.

有一个全局时钟信号驱动所有这些寄存器，它会在特定时刻触发：当钟脉冲到来时，此刻线路上的任何值都会被存进去。问题在于，我希望时钟频率尽可能高，因为如果我能跑到两GHz，每秒能完成的操作就是一GHz的两倍。

There is a global clock signal which drives all of these registers, and it says, at a certain instance in time when the clock strikes, whatever value happens to be on this wire at that instant, that's what's going to get stored in so the challenge here is I would like to have my clock speed run as fast as possible because if I can run at two GHz I can get twice as many operations done per second than if I run at one GHz.

但这样一来，我对这个逻辑模块的延迟就变得非常敏感，因为所有在这里进行的计算都必须在下一个时钟周期到来之前完成。

But what that ends up meaning is that I'm very sensitive to the delay through this cloud of logic, because any computation that is gonna happen in here needs to sort of finish before the next clock cycle hits.

对。

Right.

所以芯片设计中的一个关键优化目标，就是尽量缩短这段路径的延迟，

So a major point of optimization on any chip then is to sort of make this delay, delay from here,

越短越好。

as short as possible.

有意思。

Interesting.

而且，如果逻辑电路太复杂，可能会赶不上时钟周期，这种情况会不会发生？

And is there ever because the constraint here seems to be that if you add too much logic, then you might risk missing the clock cycle.

但要是逻辑太简单，又等于浪费了计算能力。

But if you don't add enough, then you're, you know, leaving potential compute on the table.

有没有可能你干脆赌一把，看计算能不能在时钟周期内完成？还是说，必须确保百分之百能完成？

Is there ever a situation where you're like, you'd take a probabilistic chance that a compute computation finishes and or is it just like, no, either it's gonna finish by clock cycle or not?

是的。

Yeah.

在常规芯片设计里，工程师会留足余量，虽然理论上存在概率，但已经远远超出好几个标准差了，实际使用中完全不用担心出问题。

In standard chip design, you you margin it such that I mean, there is a probability, but it's like many, many standard deviations, like way standard deviations out such that for all intents and purposes, is a reliable part.

它一定会在时钟到来前完成。

It it will always meet the clock.

当然也有少数特殊情况。

There are some weird exceptions to that.

比如跨时钟域的时候，信号从一个时钟域切换到另一个时钟域，这时候就真的得仔细考虑这个问题了。

There are clock domain crossings where you go from one clock to the other clock, and then you actually do have to reason about this Interesting.

主路径上，你会留出余量，确保能提前25%的时钟周期到达。

The main path, you just like you margin it such that you'll get there like 25% of the clock cycle in advance.

所以几乎不可能出现延迟的情况。

So it's very unlikely that

时钟同步点，也就是寄存器所在的位置，对吧？

And this the clock, where the clock synchronize, I guess, where the registers are.

这并不是芯片设计师自己决定的。

This is not something you determined as a chip designer.

这更像是一个副产品——我只想实现某段逻辑流程。

This is sort of just like an artifact of, hey, I want whatever sequence of logic.

而你用的那些工具，把Verilog代码转成交给台积电的版图时，会自动判断：为了保证电路正常工作，必须在这些地方加寄存器，避免任何一段逻辑路径过长，导致整个芯片的时钟周期被迫拉长。

And then the software you use to convert your Verilog into the thing you send to TSMC, that just determines like, in order to make this work, you gotta kinda you gotta put a register here, here, and here to make sure that there's no one step that is like too long such that it makes the whole clock cycle of the entire chip longer than it has to be.

是的。

Yeah.

所以，插入这些寄存器，其实是芯片设计中非常关键的一环。

So this is actually a huge part of the work of designing a chip actually is inserting them.

所以这是人工和自动方式结合完成的。

So it is done in a combination of manually and automatically.

我来给你举个最简单的例子，你可以把这段逻辑平均分成两半。

So I mean, just to show the very sort of dumb version of what you can do here, you can take this logic and split it in half.

也就是说，不是只有一大块逻辑，而是分成两小块，功能一样，中间加一个寄存器隔开。

And so, say, actually instead of just one cloud of logic, I'm gonna have two smaller clouds of logic, which do the same thing, but split them up by a register.

对。

Right.

信号就这样传进来。

Feeding in like this.

如果你正好在中间切开，时钟频率就能翻倍。

And this is, like like if you split it like in the middle, you can hit twice the clock frequency.

这太棒了。

That's great.

性能翻倍了，代价是多了一个寄存器，还有额外的存储开销。

You get twice the performance at the cost of this extra register and so at the cost of some more storage.

所以退一步说，我们为什么需要让整个芯片同步呢？

And so stepping back, why do we need to synchronize the whole chip?

比如你玩《工厂大亨》这类游戏的时候，根本就没有一个全局的时钟周期。

Like, if you have like, if you imagine playing factorio or something, there's no, like, global clock cycle.

事情办完了自然就完成了。

It's shit is done when it's done.

铁矿石已经放在传送带上。

There's iron on the plate.

你想拿走随时都可以。

You can take it if you want.

是的。

Yeah.

拿这个打比方，你得注意的是，如果我有两条不同的逻辑路径。

So taking that analogy, the the the the thing that you need to be mindful of is if I've got two different paths through some logic.

比如在这边做运算 f，在那边做运算 g，然后它们最终要在某个地方汇合，一起做运算 h。

So I have to do a computation like f here, and then computation g here, and then they're gonna come and meet for computation h somewhere here.

所以这里会存在制造上的差异。

And so there's gonna be manufacturing variance here.

有些芯片里，F 的运算会慢一点，有些芯片里 G 的运算可能会慢一点。

In some chips, F will take a little longer, maybe in some chips G will take a little bit longer.

如果有一个信号在这里传递，F 和 G 的结果需要在 H 处汇合，那可能出问题的地方就是 F 提前到了，和 G 的上一个值或者下一个值撞上了。

And so if I've got some signal that's propagating through here, and the result from f and g have to sort of meet up at h, what can the the thing that can go wrong is that f can get there early and it meets like the previous value of g or the next value of g

或者类似的情况。

or something like that.

H 需要知道什么时候该启动。

And h needs to know when to start.

没错。

Exactly.

对。

Yes.

比如，为什么用同样的工艺节点、同样的台积电技术制造的芯片，时钟周期却会不一样，这就解释了原因。

Like when when has this next iteration of and so this explains why different chips made at the same process node, the same like TSMC technology can have different clock cycles.

比如，两颗同样用三纳米工艺制造的芯片，时钟频率可能不同，这取决于它们是否成功优化了关键路径，避免某条路径过长拖慢整个芯片的时钟速度。

Like, two chips made at three nanometer might have different clock cycles based on whether they were able to optimize making sure that, like, there's no one critical path that is so long that it it slows down the whole ship's clock cycle.

没错。

That's right.

我刚才展示的这种优化，叫做流水线寄存器插入。

This optimization that I that I showed here, this is just the this is sort of pipeline register insertion, it's called.

对。

Yeah.

我们在流水线中间插入了寄存器。

We've inserted in the middle of the pipeline register here.

这本质上是在时钟速度和芯片面积之间做权衡。

This is a sort of pure trade off between clock speed and area.

这种情况还算简单的。

This is the easy case.

还有一种更复杂的情况，虽然我这里画的是线性流水线，但有时候某些计算结果会反馈回来，形成环路。

There is a harder case too, which is sort of drawn it as a pipeline of logic here, but in other cases you may have some calculation which actually feeds back in on itself.

所以它会运行某个函数 f，然后把结果写回自身，就像这样。

So it runs some function f and then writes back to itself like this.

举个例子，这可能是一个加法操作，比如每个时钟周期你都在往一个数上加上另一个值，这里就是不断累加某个固定数字。

So, for example, this might be this addition, like you've got some number that you're adding into every clock cycle, and so this could be like a plus where adding in some number every clock cycle.

这个小电路本质上就是在把每个时钟周期输入的数字全都加起来。

So this little circuit, essentially, it's just gonna sum all of the numbers that are presented on on different clock cycles.

问题是，如果这个加法运算耗时太长，我该怎么办？如果我在中间插入一个流水线寄存器，比如就放在这儿，就会改变原本的计算结果。

And the challenge is if this plus takes too long, what can I do if I, like, split it in if I try and put a pipeline register, like, right in the middle of it, like like here in the middle of it, this will end up changing the computation that's done?

这样一来，就不再是把所有输入累加成一个总和，而是会变成两个独立的累加序列。

Instead of forming a running sum of everything that comes here, I will actually have two different running sums.

一个累加的是偶数周期的输入，另一个累加的是奇数周期的输入。

I'll end up having a running sum of the even numbers and a running sum of the odd numbers.

这种逻辑环路——所有芯片里都存在这样的结构——才是真正最难处理的问题，也直接决定了芯片的时钟频率。

So sort of this constraint where I have a loop in my logic, which all chips have somewhere, this is actually the thing that is the hardest thing to address and sets the clock cycle.

我不明白，为什么这样会成问题。

I don't understand why it'd be a problem to have that.

或者我都不确定在那儿放个寄存器到底意味着什么。

Or I'm not sure even what it would mean to have a register there.

这不像是一个原子操作吗？

Like, is sort of atomic operation, right?

其实加法并不是原子操作。

Well, so plus is not really atomic.

你刚才已经演示过了。

As you just demonstrated.

对。

Yeah.

做一次加法要花不少步骤。

It took a whole lot of work to do a summation.

可以把前面的部分先做完，中间插个寄存器，再处理后面的部分。

So, can take the early parts of that work and then stick a register in middle and then take the late parts of that work.

明白了。

Got it.

好的。

Okay.

对。

Yeah.

我想，TSMC 提供的 PDK 会列出芯片中可用的基本逻辑单元，而他们得确保每个基本单元的延迟都不超过目标工艺节点的时钟周期。

And I guess it's then up to so TSMC offers a PDK which says that I say, here's the primitives of logic that we can grant you in the in the chip and it's up to them to determine that no no primitive is bigger than, like, the clock cycle they're hoping a process node targets.

但除此之外，还有没有其他优化方式？你不能直接说：

But other than that, is there, like, what further optimize can't you just say, like, hey.

把 TSMC 给的所有基本单元列出来，然后在它们之间不断插入寄存器，直到满足目标时钟周期就行了吗？

Here's all the primitives from TSMC, and keep adding registers in between the primitives as much as is needed until you get to your desired clock cycle?

对。

Yeah.

作为逻辑设计师，芯片架构师会决定时钟周期的长度。

As a logic designer, the chip architect set the clock cycle.

举个例子，TSMC 提供的基本单元通常是与门、全加器这类东西。

So just for for one example, the primitives you get from TSMC are on the order of like AND gates or full adders.

这些参数很大程度上取决于电压、频率，还有你选用的工艺库等等。

They depends a lot on voltage and frequent and which library you choose and so on.

但一般来说，一个时钟周期内通常可以串接大约十个、二十个，甚至三十个这样的逻辑单元。

But generally, you can typically have about like 10 or 20 or 30 of these in a clock cycle sequentially.

所以这些参数的速度非常快，大概只有十皮秒左右。

So these parameters are very, very fast, like ten picoseconds or something like that.

因此，作为逻辑设计师，理论上如果你只是让寄存器和与门这样循环连接，确实能实现极高的时钟频率，比如四五十甚至六吉赫兹都有可能。

And so as a logic designer, in principle if you literally just had register and then AND gate in a loop like that, you could get an insanely fast clock speed, like more than four or five, six gigahertz, something like that.

但如果你看这个简单电路所占用的面积，一个逻辑门算作单位一，而这个寄存器的面积可能是单位八左右。

But if you take this really simple circuit and you look at the area you're spending here, this is called one gate equivalent in size, so unit of one in area, and this thing is unit of eight in area or something like that.

所以，这里绝大部分的成本其实都花在了同步和通信上，而不是实际的逻辑运算。

And so this is just, again, almost all of your cost has been this synchronization or communication cost compared to the actual logic.

这种情况就是你过度优化了。

And so this would be a case where you've gone too far.

你虽然把时钟频率推得极高，但几乎把所有芯片面积都用在了流水线寄存器上。

You've made your clock speed really, really fast at the cost of spending almost all of your area on pipeline registers.

有意思。

Interesting.

所以你的意思是，虽然时钟频率可以非常高，但实际完成的工作量却不多。

So what you're hinting at is a dynamic where you can have really fast clock speed but you're not getting that much work done.

对。

Yeah.

所以你可能会有低延迟，但带宽或吞吐量却很低。

And so you can have low latency but low bandwidth or throughput rather.

是的。

Yeah.

实际上，这会拖累吞吐量，因为芯片的吞吐量等于每个时钟周期能完成的工作量（也就是面积效率）乘以每秒的时钟次数。

The it hurts your throughput in fact because like like the the throughput of your chip you can think of as the product of how much I can get done per clock cycle, which is based on this area efficiency thing times how many clocks I get per second.

这其实跟我们上次讨论的批量大小问题很像：如果批量太小，单个用户确实能很快拿到下一个词，但一小时内总共处理的词数反而会比用大批量时少。

This is this is actually so similar to the thing we're discussing last time about like batch size where if you have a low batch size, then you can any one user can receive their next token really fast, but the total number of tokens that are processed in, say, an hour will be kind of lower than it could otherwise be.

对。

Yeah.

没错。

Exactly.

如果你一味提高时钟频率，并行度反而会下降。

You get you get less parallelism out if you if you drive your clock speed

这很明显。

up clearly.

语言模型已经开始和顶尖的人类预测者竞争了。

Language models are starting to compete against the best human forecasters.

我找来了两位简街的资深员工，罗恩·明斯基和丹·庞特科沃，问他们：有没有那么一天，AI会完全取代简街的工作？

I sat down with two senior Jane Streeters, Ron Minsky and Dan Pontecorvo, and asked, at some point, does AI just do what Jane Street does?

我们得认真考虑一种可能性：未来我们会造出比地球上所有人类都更聪明、在所有认知任务上都更强大的大型语言模型或其他AI系统。

There's a world that we should take seriously, where know, we're gonna build large language models or some other AI systems that are, like, strictly smarter than all humans on the planet and more capable at all cognitive tasks.

在我看来，交易尤其像是一个‘通用人工智能完备’的问题，就像NP完全问题一样，归根结底，交易就是要判断事物的价值，也就是预测未来。

Trading in particular feels to me as, like, kind of AGI complete, sort of like NP complete, because at the end of the day, trading involves figuring out what things are worth, which means making predictions about the future.

简街并没有在跟AI对赌。

Jane Street isn't betting against AI.

他们刚签了一个六百万美元的算力合同。

They just signed a $6,000,000 compute deal.

但罗恩认为，技术前沿一直在变化。

But Ron's view is that the edge keeps moving.

我从未像现在这样迫切地想招更多工程师和交易员。

I have never been more desperate to hire more engineers and more traders than I am today.

你还是会遇到那些我们目前还不知道怎么自动化的难题。

You you have the usual thing of, like, the other hard parts that we don't yet know how to automate.

而这些恰恰就是竞争优势所在。

Well, that ends up being where the competitive edge lies.

你可以在 Jane Street 官网 Thorcash 页面上查看这些职位和完整访谈。

You can find these open positions and watch the full interview at Jane Street dot com Thorcash.

明白了。

Okay.

我记得以前跟 Jane Street 的一位 FPGA 工程师克拉克聊过，他帮我准备了我们之前那次访谈。

So I remember talking to an FPGA engineer at Jane Street, Clark, who actually helped me prep for your the the previous interview we did together.

他当时解释了他们为什么选用FPGA。

And he was explaining why they use FPGAs.

我觉得对于高频交易来说，吞吐量没那么关键，延迟才是核心。

And I I imagine that for high frequency trading, the throughput is less important than latency.

所以能精确、稳定地控制每个时钟周期，才是最重要的。

And so having very specific control over the clock cycle in a deterministic way is the most important thing.

也许我们可以聊聊，为什么不用ASIC，而偏偏选FPGA呢？

I, maybe it'd be interesting to talk about how you why you can't just achieve that with an ASIC or why FPGA is the yeah.

为什么在高频交易里，要用FPGA来保证时钟周期的确定性？

Why you might use an FPGA to have deterministic clock cycles for like high high frequency trading?

对。

Yeah.

首先，咱们得从商业角度看看FPGA和ASIC各自的优劣。

So I mean, firstly, like, let's consider the sort of the business case for an FPGA versus an ASIC.

FPGA和ASIC在概念上其实很相似，都是由与门、或门这些基本逻辑单元组成，通过固定的时钟周期和布线连接在一起。

FPGAs and ASICs use largely the same conceptual model, which is that I have a series of gates built from ANDs, ORs, six ORs, those very small primitives, connected together with a fixed clock cycle, and connected together with wires that are running in a fixed clock cycle.

任何你能用FPGA实现的功能，ASIC也能实现，而且ASIC的成本低一个数量级，能效也高得多。

So anything you can express in an FPGA, you can express in an ASIC too, and it'll be about an order of magnitude cheaper and better energy efficiency on an ASIC than an FPGA.

但代价是，第一个FPGA只要一万美元，而第一个ASIC的制作成本高达三千万美元，因为需要完整的光刻掩模流程。

The trade off is that the first FPGA costs you $10,000 whereas the first ASIC you make costs you $30,000,000 because it requires an entire tape out.

所以FPGA的商业应用场景是：你需要极低且稳定的延迟、快速的运行速度和高并行性，但你又会频繁改动设计。

So the business use case for an FPGA would be that I want something that has this very deterministic latency fast runtime and high parallelism, but, you know, I I'm gonna change it very frequently.

可能每个月都要调整一次功能，那每次都要付光刻掩模的费用就太不划算了。

Can change what I do every month or something like that, and so then I don't wanna pay the tape tape out cost every time.

那FPGA到底是怎么实现的呢？它用一块固定的硬件来模拟ASIC的编程模型，这背后是怎么运作的？

Now how does an FPGA actually implement it sort of like it emulates the ASIC programming model, but in fixed piece of hardware.

那具体是怎么做到的呢？

And so how does that work actually?

它的底层其实就包含我们刚才说的两个部分。

So what it has at the base is it's got the two components we just talked about.

一个是寄存器，用来存储数据；另一个是LUT，也就是查找表，负责实现所有的逻辑门。

It's got these registers as storage devices, and then it's got these, these are called LUTs, lookup tables, which actually provide all of the gates.

接下来我们会看到第三个部分，其实就是一堆寄存器和查找表组成的网络。

And then we're gonna see even the sort of the third component, we then have like sort of a swarm of these registers and lots.

所有这些组件都可以连接起来，通过一大组多路选择器来实现互联。

And all of these are available and then they are connected by sort of this big set of sort of Muxes.

每个组件前面都接了一个多路选择器，可以从所有其他组件中选一个信号输入进来。

So in front of every single one of these, we've got something like one of these Muxes, which selects one input from everywhere else, sort of selecting from all of these different things.

有大量不同的信号源同时输入到这些选择器里。

We've got a whole bunch of different options feeding into all of these things.

所以当我对FPGA编程时，我可以指定把这些组件按特定方式连起来——让信号从这一组流到另一组，再进入某个寄存器，然后继续流向下一组，诸如此类。

So so what what what this allows is essentially a when I program my FPGA, I can say that I'm gonna take all of these components and I'm gonna sort of superimpose on top of this a particular wiring which, like, goes through this lot and then feeds into this lot and then goes to this register and then feeds into this lot or something like that.

我用橙色标出来的部分，就是FPGA里‘可编程’的那部分。

So what I've drawn in orange is you FPGA means Field Programmable Gate Array.

橙色代表在实际使用中被配置的线路，而白色部分则是FPGA芯片本身必须内置的固定布线。

The orange is what has been programmed in the field, whereas the white is all of the wires that must exist in the FPGA in order to make the device in the first place.

‘在现场编程’是什么意思？

What does it mean to be programmed into a field?

就是现场编程的意思。

Like programmed in the field.

比如设备已经部署在数据中心里，放在现场，然后你再过去进行编程。

So like the device is being deployed in a data center, it's sitting in the field and then you can come and program.

不是指电场的场。

Not field as an electric field.

对。

No.

是说在现实世界中的那个‘现场’。

Field as in like out there in the world field.

明白了。

Okay.

如果我看到现场编程的信号从第一个查找表出来，再进入第二个查找表，

And so if I see look at the how the the field programming comes out of the first lookup table and goes in a second one.

这个过程到底是怎么实现的呢？

How is it how yeah.

怎么做到的？

How?

那些让这发生的具体线路在哪？

Like like, where where are the wires that made that happen?

我猜是这样。

Guess this yeah.

我画这些的时候有点偷懒了。

So I I I got a little bit, like, lazy in drawing all of these.

这里每个设备前面都有一个多路复用器，可以从附近所有可用的电路中选择信号。

Every single device here has a Mux sitting in front of it, which can select from all of the, like, nearby, circuits that are available.

所以FPGA的实际配置本质上就是这些多路复用器的控制信号。

And so the actual configuration of the FPGA is like amounts to it, it is the Mux control.

在这个多路复用器里，有数据输入端，还有用来选择的控制端。

So in this Mux here we have the data inputs, and then we have the control that selects.

每个多路复用器旁边都有一个小存储单元，指定它该从哪里获取输入信号。

And so there's a little storage device sitting next to every single one of these Muxes saying this is where you're gonna source your input from.

明白了。

Got it.

所以编程的过程就是配置每一个这样的多路复用器。

And so programming it consists of configuring every single one of these Muxes.

这下就说得通了。

So that makes sense.

那查找表里面到底发生了什么？

So what is happening inside of the lookup table?

对。

Yeah.

查找表的作用呢，它也会接收一些控制信号，告诉它该怎么运作。

So the purpose of the lookup table so it's gonna also have a little bit of control feeding, telling it what to do as well.

查找表的设计目的是能灵活地充当与门、或门、异或门，或者其他任何逻辑功能。

The purpose of the lookup table is to function to be able to configurably take the role of an AND gate, OR gate, XOR, any of those different things.

实现这一点其实有好几种方法。

So there's many ways you could consider doing that.

传统FPGA的做法是，这个查找表有四个输入位和一个输出位。

The way it is done in sort of traditional FPGAs is to say it will support will be a lookup table will be it will have four bits of input, one bit of output.

从四个比特到一个比特，一共有多少种不同的函数？

How many different functions are there from four bits to one bit?

一共有16种不同的函数。

There are 16 different functions.

所以你可以直接把这16种情况列成一组数字。

And so you can actually just tabulate this as 16 different numbers.

你有一个表格，里面是0、1、1、1、0、0、1，一共16行。

You've got a table of zero one one one zero zero one, 16 entries.

这个表格会存放在蓝色的配置位里，系统把那四个输入位当作二进制数，查到对应的行，然后输出那个比特值。

And so what it does is this table is stored in this blue configuration bit, and then it views these four bits as binary, looks up the relevant row of the table, and emits that bit.

这其实就是把查找表看作一个真值表。

So this is a truth table view of lookup tables essentially.

明白了。

Okay.

所以你看查找表，比如与门、或门、非或门、异或门，这些都是一些输入组合。

So the lookup table, if you think about an AND gate, OR gate, NOR gate, XOR gate, these are all take as input.

这些都属于双输入的函数。

Those are, like, two input functions.

对。

Yeah.

有时候我们会遇到更复杂的，比如三输入的函数，就是三路异或，对吧。

So sometimes we have, like, more complicated like, a three input function would be a three way XOR Right.

或者四路异或。

Or a four way XOR.

这种情况下，具体要看它有多大。

And in this case, how many it it depends how how big it is.

查找表通常用四输入的，这在各方面算是个平衡点，有个权衡在里面。

Typical size for LUTs is four input, which is sort of just a sweet spot between there's another computer communication trade off like here.

如果输入太少，就得用更多的查找表。

Like, if it if it has too few inputs, then you need to use more LUTs.

对。

Yeah.

如果输入太多的话。

If it if it has too many.

但本质上，查找表就是一个真值表。

But basically, the lookup table is like a truth table.

没错，就是一个真值表。

It's a truth table.

是的。

Yeah.

有了真值表，就能实现任何你想要的逻辑门。

And with a truth table, can program in any gate you want.

没错。

That's right.

对。

Yeah.

所以这个查找表就像是一个可编程的逻辑门。

And so it's a lookup table that just thinks like a programmable gate.

没错。

That's right.

所以，你可以从这里看出为什么会有这么一种说法：FPGA的成本大约是ASIC的十倍——只要算算这个查找表里到底包含多少个逻辑门就知道了。

And so, I mean, one of the things you can do here is you can see why where the rule of thumb that an FPGA is like an order of magnitude more expensive than an ASIC comes from, is to count how many gates would be inside this lookup table.

对。

Yeah.

因此，我们可以把这个查找表看作是一个多路选择器。

So we can view this lookup table essentially as one of these Muxes.

它需要从16个不同的值中进行选择。

And so it is a Mux with, has to select between 16 different values.

也就是说，这是一个有16个输入、1位输出的多路选择器。

And so it is a MUX with sort of n equals 16 options, p equals one bits.

我们之前已经看到，这种电路的成本大约是 n 乘以 p 个逻辑门。

And so what we saw way earlier is that this circuit costs like n times p many gates.

所以成本大约是 n 乘以 p，也就是 16 个与门，再加上 16 个或门。

And so it costs like n times p equals 16 and gates and also 16 oars.

这个电路就是多路复用器。

This circuit being the Mux.

对。

Yeah.

没错。

Exactly.

多路复用器。

The Mux.

多路复用器是核心部件。

The Mux is the core

进入查找表的就是这个多路复用器。

of The Mux that goes into the lookup table.

所以这个查找表本质上就是一个大型多路复用器，从 16 行中选一条输出。

So the lookup table itself, can think of as being actually a big Mux that selects from all 16 rows down to one output.

是的

Yeah.

好的

Okay.

明白了

See.

这就是查找表。

That is the lookup table.

但你这里画的，是一个多路复用器后面接着一个查找表。

But the way you've drawn it here, there's a Mux and then a lookup table.

其实底层全是多路复用器。

It's it's Muxes all the way down.

我的意思是，这里头还有一个第二级的多路复用器。

So, I mean, there's a there is a second Mux that is inside here.

这个多路复用器就是这个。

This Mux is is this Mux.

明白了。

Got it.

好的。

Okay.

然后另一个多路复用器只是表示

And then the the other Mux is just saying

这个信号在这一堆增益里是从哪儿来的。

Where it came from in this sort of mess of of of gains.

对。

Right.

然后是第二个多路复用器，明白了。

And then the second Muxes, okay.

现在你有一个值，但这个值仍然是一个四位的值。

Now you have one value, but that value is still still a four bit value.

是的。

Yeah.

所以我从这一堆数据里选了四位。

So I've selected four bits from the soup.

对。

Right.

然后用这四位来查查找表里的对应条目。

And then and then I use those four bits to select which entry in the lookup table.

是的。

Yeah.

我要用的就是这个。

I am gonna use.

明白。

Right.

好的。

Okay.

为了确认一下，比如第一个多路复用器，你是从八个相邻的寄存器里取数据作为输入。

And just so make sure, so like, suppose in the first Mux there's like eight nearby eight you're you're pulling from eight nearby registers as input.

对。

Mhmm.

所以总共是32位输入。

And so that's like a total of like 32 bits going in.

对。

Mhmm.

然后从中输出4位。

And then out of that, four bits come out.

这4位会进入第二个多路复用器，也就是查找表内部。

Those four bits go into the second Muxes, which is inside the lookup table.

所以实际上，这么说吧。

So actually, would say in yeah.

在这种情况下，这些寄存器都是单比特的。

In this case, these registers are single bit registers.

如果有八个相邻的寄存器和查找表，那就会有总共八位输入。

So if there are eight, like, eight nearby registers and lookup tables, then I have eight bits total coming in nearby.

我从八个输入中选出四个不同的值，所以实际上有四个多路复用器，每个都对应一个输入位，也就是这些小标记。

I select from eight down to four different values, so there's actually like four different Muxes, one associated with each of these input, like little marks associated with each of these input bits.

每个复用器都是从八个选项里选一个。

Each of them is selecting one out of eight.

那这八个输入是从哪儿来的？

And where are those eight coming from?

来自附近的寄存器和其他模块。

Nearby registers and other lots.

每个寄存器都是一位的吗？

And each register is one bit?

对。

Yes.

明白了。

Yeah.

所以我觉得，像AMD这样的FPGA制造商，还是得对哪些寄存器和哪些寄存器相连做出明确的设计选择。

And so I guess you AMD or whoever makes these FPGAs still has to be opinionated about what registers are connected with registers.

然后你可以编程实现具体的逻辑门，包括布线和连接方式，也就是通信拓扑结构。

And then you can program in the actual gates, they had a wire and the connection, like the communication topology.

对吧？

Right?

是的。

Yeah.

所以在局部范围内，你有一定的灵活性。

So there's there's sort of like you get flexibility in a local grain thing.

在附近的小范围内你可以自由选择连接，但更远距离的连接，系统本身会做出固定安排。

There's a sort of nearby neighborhood where you can select from, but then more grossly, like more coarsely, longer distant connections, they they form an opinion on that.

对。

Right.

明白。

Yeah.

那为什么慢了十倍呢？

And the reason it's 10 x slower is why?

所以如果你看一下

So if you look at

搭建这个查找表的成本，大概就是32个逻辑门。

the cost of, like, building this lookup table, it's, like, 32 gates.

对。

Yep.

然后它能让我实现一个挺有意思的功能，是什么呢？

And then it can give me the equivalent of, like, what's one an interesting thing I can do here?

我可以做一个四输入与门。

I can do a four way AND gate.

也就是说，我用32个逻辑门的查找表，来模拟一个四输入与门——那什么叫四输入与门呢？

And so that's like, I am using 32 gates of lookup table to sort of implement, like, a four way AND means, like, what is a four way AND?

通常我会用三个与门，一个接一个地连起来：与、与，再与一次。

I would do, like, AND, AND, and then AND of AND.

所以，这种结构其实可以直接用三个与门在ASIC里实现，根本不需要查表。

So to implement like, this is a a socket that I could implement in an ASIC directly using these three and gates.

但要是用这个方法，虽然也能实现，却得用上这32个门电路，而不是三个。

But using a lot, I can also implement it, but it's gonna take, like, these 32 gates instead of three.

而且

And

所以额外开销主要来自查找表里的多路复用器——其实用逻辑门直接描述真值表，比列全所有输入组合要简洁得多。

so the overhead is really coming from the, like, the fact that the lookup table the Mux in the lookup table is there's a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to like write out the gate.

对。

Yeah.

就像实际布线时，得铺多晶硅、布线这些。

Like to to like place down the polysilicon and the the wires and so on.

是的。

Yeah.

没错。

Yeah.

有意思。

Interesting.

他跟我提过一个关键点，他们更喜欢FPGA而不是CPU，是因为FPGA能提供确定的时钟周期。

One important point he made to me is that the reason they prefer FPGAs to CPUs is because they get deterministic clock cycles.

他们能准确知道数据包什么时候进来、什么时候出去。

They know when a packet will come in and go out.

那为什么CPU做不到这一点呢？

Why is it not a guarantee in CPUs?

其实你也可以设计出具有确定延迟的CPU。

So you can actually design a CPU that has deterministic latency as well.

事实上，很多AI芯片里用的处理器，延迟也是确定的。

And in fact, like, the the processors that are inside a lot of AI chips actually also have deterministic latency too.

厂商们已经把这一点作为卖点宣传了。

Has advertised this.

TPU的核心也具备这个特性。

TPUs have that in the core as well.

难点在于如何同时实现确定的延迟和高速度。

The challenge is getting sort of deterministic latency and high speed at the same time.

那延迟的非确定性到底从哪儿来的呢？

And so where does the nondeterminism in latency come from?

CPU的非确定性延迟源于一些特定的设计选择。

Nondeterministic latency comes from specific design choices in a CPU.

其实完全可以去掉这些设计，做出一个具有确定性延迟的CPU。

It's actually possible to remove those design choices and make a CPU that has deterministic latency.

但这类CPU在市场上没什么吸引力，所以现在没人再生产了。

Those are not very attractive in the market, and so people don't make those CPUs anymore.

不过从某种角度说，确定性延迟反而是更简单的设计起点，后来才有一些芯片设计师加了东西，让系统变得非确定。

But actually, in some sense, deterministic latency is maybe a sort of a simpler designing starting point, and then and then, like, some chip designers have added things in to to be nondeterministic.

举个具体的例子，最重要的就是CPU的缓存本身。

To take a concrete example of that, probably the most important example is on a CPU just like the CPU cache itself.

在CPU里，这是芯片本身，然后旁边还连着一块内存。

So in a CPU, have the the CPU this is this is the CPU die itself, and then there's a memory off on the side.

这边是外接的DDR内存。

This is the DDR memory off on the side.

然后这里还有一个缓存系统。

And then you have a cache system here.

缓存就在里面。

Inside it is the cache.

它会记住最近对 DDR 的访问，并把这些数据存起来。

That sort of remembers recent accesses to DDR and stores them.

所以当我执行 CPU 指令时，每次要访问内存，都会先查缓存里有没有需要的数据，如果没有，才去 DDR 里取。

And so I'm running through my CPU instructions, every time I have an instruction that accesses memory, it first checks in cache, was was the data stored in cache, and then if not, it goes it fetches out to DDR.

是的。

Yep.

这可是个巨大的优化。

This is a huge optimization.

缓存的速度比 DDR 快差不多一百倍。

The cache is like two orders of magnitude faster than DDR.

如果你完全不用缓存，几乎所有程序都会慢上一百倍。

If you never if you never use the cache, basically all programs would run a 100 times slower.

所以，缓存的存在对CPU以合理速度运行至关重要，但能否命中缓存，却取决于CPU的运行环境，比如有哪些其他程序在运行、最近执行过什么操作，以及缓存系统内部的随机数生成器在做什么。

So the presence of a cache is absolutely necessary for a CPU to to run at reasonable speed, but whether or not you get a cache hit is dependent on the ambient environment of the CPU, like what other programs are running, what has run recently, what is the random number generator inside the cache system doing.

因此，这也是CPU运行时间产生不确定性的一个主要来源。

And so that is a big source of nondeterminism the runtime of a CPU.

这基本上就是CPU的内存系统结构。

So this is sort of the memory system for a CPU.

你可以采取的一种不同做法是，不把决定权交给硬件去判断该从缓存还是主存读取数据，而是把这种决策直接交给软件来处理。

The big thing that you can do differently is instead of having the hardware say, I'm gonna read memory and then decide the hardware decides whether or not it comes from cache or not, you can actually bake this decision into software.

另一种设计思路是，你可以在TPU这类处理器上看到这种做法。

So a different design philosophy is to, and you see this in maybe for example TPUs.

TPU的做法是，我画的图虽然和之前一样，但不叫缓存，而叫暂存区。

The the TPU instead has I mean, I'll draw the same diagram, but I'll call it a scratch pad.

主要区别在于，这是一颗TPU，搭配的是HBM而不是DDR，不过它仍然是芯片外的内存。

And so the the main difference is so this would be like a TPU and then like HBM, in this case, rather than DDR, but it's still an off chip memory.

在这种架构里，软件不是先访问内存再让硬件决定，而是会发出专门的指令来控制数据的存取。

And instead of the software saying first access memory and then the hardware decides, you've got some instructions that go here.

这是一种指令，还有一种完全不同的指令是直接访问HBM的。

This is like one kind of instruction, And then a totally different kind of instruction that goes to HBM.

这种设计通常被称为暂存器，而不是缓存。

And so this style is generically known as Scratchpad instead of cache.

关键区别在于，有一种指令专门用于读写暂存器，另一种指令则用于读写HBM。

The key distinction being that you have one kind of instruction that says read or write Scratchpad and a totally different instruction that says read or write HBM.

那么暂存器就是缓存吗？

So is is Scratchpad being the cache?

是的。

Yeah.

这里这个就是暂存器。

This this thing here is the is the Scratchpad.

我再明确一下。

Just to be clear.

如果从更宏观的角度看，人们常说计算机采用的是所谓的冯·诺依曼架构，也就是信息按顺序处理。

So stepping way back, people say computers have the, quote, unquote, John Von Neumann architecture where there's this serial processing of information.

我们刚才一直在聊并行加速器，但我觉得FPGA其实非常并行。

And maybe just because we've been talking about parallel accelerators, but I just don't like, the FPGA is super parallel.

对。

Yeah.

现在这些AI芯片，

The the kinds of AI.

发现TPU也是高度并行的。

Saw that TPUs are super parallel.

就连CPU，想想它那么多核心，其实也非常并行。

Even CPUs are super parallel if you think about all the cores they have.

那现代硬件到底在什么意义上还算是冯·诺依曼架构呢？

And so is it actually like, in what sense is modern hardware actually the Von Neumann architecture?

这样描述现代硬件真的合适吗？

Is it actually a fair way to describe modern hardware?

我觉得这

I think it's

用这个说法来形容CPU还挺合适的。

a fair way to describe CPUs.

主要是并行度的问题。

Like, just the amount of parallelism.

以CPU为例，它的并行度大概是上百个核心，每个核心再配上大约16路的向量单元。

Like, on a CPU, the amount of parallelism you get is about a 100 cores times maybe, like, 16 way vector unit.

所以CPU的并行度大概能达到上千路。

So 106 about a thousand way parallelism on a CPU.

没错。

Yeah.

有个问题是，CPU所用的那块芯片上，

Is it one question is, what is the there is a die that is being used for the CPU.

对。

Mhmm.

如果线程数少，从晶体管电压切换的角度看，是不是只有芯片的一小部分在实际工作，而且只有一条控制流在切换电压？

And if there's fewer threads, just as a matter of, like, transistor voltages are, like, switching on and off, Is it just that there's like literally one control flow, like a small part of the die where like voltages are switching on and off?

或者，如果核心数量这么少，那CPU芯片上的空间到底是怎么被利用的呢？

Or like in what how how do you actually occupy the die area of a CPU if there's as opposed

如果核心这么少，那其余的面积岂不是都浪费了？

to If there's so few cores, am I spending all the there.

是的。

Yeah.

核心本身要大得多，结构也复杂得多。

The cores are just much bigger and more complicated.

所以，我觉得我们应该比较一下：一个CPU核心占芯片面积的百分之一，而一个GPU单元可能只占很小一部分。

So, I so I guess we should compare a CPU core which takes up one hundredth of a die I mean, to a lot.

比如，一个GPU单元可能就只有这16个逻辑门那么大。

Like, a lot is just only these 16 gates.

对。

Right.

所以，为什么FPGA里的单元数量远多于CPU的核心数，这就很清楚了。

So, like, it's clear why there are so many more lots in an FPGA than cores in a CPU.

但为什么CUDA核心的数量会比CPU核心多得多呢？

But then sort of maybe the the like, why are there more CUDA cores, for example, than than than CPU cores?

这其实涉及到CPU和GPU之间的根本区别，比如它们的设计思路就很不一样。

Think would be like, why what's the difference between a CPU and a GPU or something like that would would would be a a big difference.

在CPU内部，占用面积最大的部分之一其实是缓存。

Inside the CPU, you have so one big use of the top uses of area inside a CPU are the cache.

真正用于算术逻辑单元（ALU）的空间其实很少，大部分面积都被寄存器文件占了，而不是逻辑电路。

Very little is actually the ALUs, like mostly it's like these register files rather than the logic units.

而这些组件在GPU里也有对应的结构，所以这并不是关键差异。

And then both of these things have equivalents in GPU, and so that's not a big difference.

但有一样东西在GPU里是没有的，那就是分支预测器。

But the thing that does not have an equivalent in a GPU is the sort of this branch predictor.

CPU里有一大块区域专门用来装各种预测器，它们不断在猜下一条分支指令什么时候出现、目标地址在哪。

And so there is a whole big area in the CPU which is sort of just a whole bunch of predictors that are saying when will my next branch be and where's the branch target for that.

去掉这些复杂的预测机制，同时让寄存器文件更紧凑，这才是GPU能大幅增加核心数量的关键原因。

And so stripping a lot of that out as well as sort of making these register files tighter in a sense is driving a lot of where the GPU gains

那分支预测器的作用是同时执行两个分支吗？它到底是干什么的？

are And what is the purpose of the branch predictor to execute both branches at once, or what does it do?

问题是，当我有一连串指令——指令、指令、指令——中间突然有个分支时，处理每条指令本身要花很长时间。

So the issue is that when I've got a series of instructions like like instructions instructions instructions instructions, I if I have a branch like here, if this instruction is branch, The the actual processing step of processing an instruction takes a really long amount of time.

大概要五纳秒左右。

It takes like maybe five nanoseconds or something like that.

从检测到分支，到判断条件真假，再更新程序计数器指向新地址，然后从指令内存里读取下一条指令，这一整套操作可能就要花五纳秒。

So like the time to actually notice that I've got a branch and then like evaluate the boolean whether it's true, and then update the program counter to the new target, and then read from the instruction memory for that.

完成这些步骤真的要花五纳秒。

That could take actually five nanoseconds to finish.

所以实际上，这个分支的处理结果可能要到后面才出来。

And so in reality, this may finish somewhere down here.

但我又不想把时钟频率限制在五纳秒这么慢的速度上。

I don't want to but I want to run a clock speed that is much faster than what five nanoseconds allows.

五纳秒只相当于200兆赫的时钟频率。

Five nanoseconds is 200 megahertz clock speed.

我想让处理器运行在一二吉赫兹左右。

I would like to run at one or two gigahertz or something like that.

所以在分支被计算的时候，我得先继续执行其他指令。

And so I need to run other instructions while the branch is is being evaluated.

我其实就是想不停执行后面那些指令，别停下来。

And so I, like, I really just wanna keep running the following instructions that happened after me.

但这么做可能会出错。

But that might have been wrong.

如果分支真的跳转了，那我就得知道，不该继续算这些指令，而应该直接跳到目标地址，执行那边的指令。

If the branch ended up being taken, then I need to know that instead of evaluating these instructions, actually need to jump to wherever the target is and run these instructions instead.

所以分支预测器的作用，就是提前好几拍，还没到分支指令的时候，就猜它会不会跳转。

And so the purpose of the branch predictor is genuinely to predict based on before you even get to the destruction to be like five cycles earlier to predict there was gonna be a branch that's gonna happen.

如果我把大脑的工作方式和你刚才说的做个对比，从宏观上看，区别可能在于：这些加速器可以利用结构化的稀疏性，省下本来要给逻辑门留的空间。

So if I think about how the brain works versus what you're describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and then save yourself some area that you would have otherwise had to dedicate to these gates.

但大脑用的是非结构化的稀疏性。

In the brain, there's unstructured sparsity.

任何一个神经元都可以连接到任何一个其他神经元，不是那种必须按列对齐之类的固定方式。

Any neuron can connect to any other neuron and not in like ways where they'd have the column aligned or whatever.

对。

Yeah.

还有一个事实是，内存和处理器是放在一起的。

Then there's a fact that memory and computer are co located.

某种程度上可以说，内存和处理器确实就在一起。

I guess you could say in a way the memory and computer are co located on This

这正是在某种意义上内存的紧密集成。

is exactly the co location in some sense of the memory.

没错。

That's right.

没错。

That's right.

是的。

Yeah.