第18集 - 为什么人工智能需要一种新型的超级计算机网络

大家好，我是安德鲁·梅恩，欢迎收听OpenAI播客。

Hello, I'm Andrew Main, and this is the OpenAI Podcast.

今天的节目，我们来聊聊如何让超级计算机更高效地训练模型。

On today's episode, we're discussing how to make supercomputers better at training models.

今天和我一起的有核心网络团队的马克·汉德利，以及工作负载系统团队的格雷格·斯坦布雷彻。

Joining me are Mark Handley from the Core Networking team and Greg Steinbrecher from Workload Systems.

他们会分享一项突破性进展，如何让模型训练变得更高效，从而让所有人更快获得更智能的模型。

They'll discuss how a breakthrough has made training more efficient so everyone gets smarter models faster.

这项进展真正帮我们扫除了进一步扩展的一大障碍。

This has really allowed us to remove one of the key barriers to continuing to scale.

我们讨论的是如何让全球最快的那些GPU协同工作，共同完成一个任务。

We're talking about a lot of the world's fastest GPUs and making them all work together on a single task.

当研究人员不再需要关心某个集群用的是什么网络协议时，我们就知道成功了。

We know we've won when researchers stop needing to know what network protocol this particular cluster is using.

能跟我们说说你的背景吗？

So tell me a bit about your background.

我本科时学的是物理和数学，就是想搞清楚复杂系统是怎么运作的。

I started out doing physics and math in undergrad, wanting to basically understand how complex systems work.

我特别喜欢物理里那种部分——把一个根本没法完全理解的复杂东西，简化成一个虽然完全不真实、但能帮你抓住关键的模型，然后靠着这个建立直觉，再一步步构建更复杂的模型。

I always liked the part of physics that's about how do you take this thing that is unknowably complicated and build a simple model that is a complete modern lie, but tells you something about that system, And then build your intuition on that and kind of build more complex models.

后来我读博士，目标是造量子计算机。

I ended up doing a PhD trying to build quantum computers.

这博士挺有野心的吧？

An ambitious PhD?

就是一些小问题，小问题而已。

You know, little things, little things.

可惜我喜欢的是那种庞大复杂的系统，而你也知道，量子计算机到现在还跑不起来，根本没法扩展。

Unfortunately, what I liked is big complicated systems, and you'll note that quantum computers don't work, and therefore they don't scale yet.

总有一天它们会行，但现在还做不到。

They will someday, but they don't yet work.

于是我看了看我们当时为控制光子而设计的芯片，突然觉得，这不就跟网络交换机很像吗？

So I kind of took a look at the chips we were designing to control light for quantum computers, and I went, that kind of looks like a network switch.

如果我们把这个用作网络交换机呢？

What if we use this as a network switch?

我很快发现，学术界对真实数据中心的工作负载其实知之甚少。

And what I found out pretty quickly was that academia does not know a whole lot about what real, like, data center workloads look like.

你看到的都是一些非常简化的模型，但没什么实际参考价值。

You get a whole bunch of kind of very toy models, but they're not very informative.

于是我就去说服一家科技公司，申请了一个奖学金。

And so I ended up kind of pitching an industry company to get a fellowship.

他们资助了我博士最后两年的费用，我也在那里工作了一段时间，初步搭建网络硬件，就是为了搞清楚：数据中心网络到底需要什么？

They paid for the last two years of my PhD, ended up working there for a while, building out, kind of doing initial network hardware just to try to understand, like, what is it that we actually need from data center networks?

我发现，传统的数据中心网络硬件其实还有很大的优化空间。

What I found out was that, like, there's a huge amount of headroom on just like conventional data center networking hardware, lots of room for optimization.

我们根本不需要我那个小小的光子芯片。

We did not need my little like optical chip.

我们也没搞什么花里胡哨的东西。

We did not do anything fancy like that.

但这里面有很多特别有趣的问题。

But there's all sorts of really fun problems.

就在那段时间，整个AI热潮开始兴起。

And then around that time, the whole AI boom started to kick off.

我们决定需要搭建大型GPU集群，尤其是为这些集群构建网络。

We decided we needed to build big GPU clusters and in particular, we needed to build networks for those GPU clusters.

于是我就被拉去帮忙做这些系统的模拟，好弄清楚到底该建什么。

And so I got roped in on trying to build kind of simulations of those so that we could figure out what to build.

在尝试搭建这些系统模拟的过程中，你就会对它们的工作方式有更深的理解。

And then in the process of trying to build a simulation of these systems, you learn a lot about how they have to work.

后来有一天，我想，干脆直接去把真正的系统做出来算了。

And some point, I said, well, why don't I just go build the thing, the actual thing?

于是我从写模拟软件，转而开始写让GPU之间能互相通信的软件。

And so I transitioned from writing software to build simulations to just writing the software that allows GPUs to communicate with each other.

一年多前，我来到OpenAI，做类似的工作，但能更贴近实际的模型训练。

And then a little over a year ago, ended up coming here to OpenAI to do some of the same stuff, but to get even closer to the actual model training.

我所在的团队主要负责确保GPU得到高效利用。

So the team I'm on is responsible for more or less making sure that we'd use the GPUs efficiently.

也就是说，模型的训练速度是不是已经达到了极限？

So are we, you know, are the models training as quickly as they can be?

网络会不会成为瓶颈？

Are we not bottlenecked on the network?

出了问题该怎么处理？

What do we do when something fails?

重启的时候是不是够高效？

Are we restarting efficiently?

我们怎么应对硬件的各种小毛病？

How do we kind of work around quirks in the hardware?

现在我能接触到世界上最有意思的硬件之一，想办法把它们的性能压到极致。

And, yeah, now I get to play with some of the most fun hardware in the world and, like, try to make it try to kinda squeeze every last ounce of performance out of it.

马克，你在让计算机互相通信并完成实际任务方面，经验非常丰富。

Mark, you've had a considerable amount of experience trying to get computers to talk to each other and do meaningful work.

能多讲讲你的背景吗？

Tell me more about your background.

我在OpenAI之外，还在伦敦大学学院当教授，做网络研究的时间比我想起来的还要久。

So when I'm not at OpenAI, I'm a professor at the University College London, and I've been doing networking research for more decades than I care to think about.

我最早是想让互联网能支持视频会议。

Originally, I started working out on trying to make the internet do video conferencing.

那时候这事儿特别难，因为电脑太慢了。

And back then that was a really difficult thing because the computer was so slow.

后来我们想出了一种方法，结果整个世界突然都感兴趣了。

And then we came up with a way for doing that that suddenly the rest of the world got interested in.

我们当时制定的那些标准，现在你的手机用4G和5G网络通信，靠的都是那些东西。

And so the standards we wrote for doing that are now the things that your phone uses to communicate with four gs and five gs networks.

所以，我人生前半段主要就是推动这些技术标准化，让所有人都能用上。

So that was like the first part of my life was trying to standardise all of that sort of stuff that everybody could actually take advantage of that.

但标准化的麻烦在于，所有人都得达成一致，所以要让大伙儿同意一件事，往往得花上很久。

And the problem with standardising is everybody needs to agree, so it just takes a long time to actually get everybody to agree on anything.

几年前，我对数据中心领域发生的变化产生了兴趣，它的优势在于，你只需要在构建时达成一致，而不需要让全世界都同意。

A while back I got interested in what was happening in the data center world and it had the big advantage that you could actually do something different because you only need to agree with it while building, not the whole world.

于是我就开始思考，数据中心网络这个领域真是大有可为。

And so that was how I got into thinking, well, this data center networking stuff is a really interesting place to be.

这确实是个有意思的问题，因为现在要同时使用这么多GPU，发展速度实在太快了，快得让人措手不及。我们至今还在用原本设计用来处理图形的GPU。

Well, it's an interesting problem because I think that just the idea that would be scaling as many GPUs as we are now, it just, it's happened so fast, happened so quickly and often, you know, we're still using GPUs, which are graphical processing units.

直到现在，我们才刚开始用新一代芯片来做这些事。

You know, we're just now starting to use next generation chips to do this.

所以，要改变我们对这个问题的理解，更新我们所用的工具，这背后付出了多少努力？这正是我们今天要聊的重点——多路径可靠连接。

So how much has been the work required just to update the way we think about it and update the sort of the the tools we're using for this, which is here what we're here to talk about with Multipath Reliable Connection.

从网络的角度来看，我们过去构建的数据中心，很大程度上是沿袭了互联网的做法。

So from a network point of view, the the data centers that we traditionally used to build, they kind of got derived from what we used to do in the Internet.

在互联网上通信时，有成千上万的人在同时交流。

And when you do communication in the internet, you have lots and lots of people communicate.

虽然数据流量很大，但每个人都在进行各自独立的对话。

It's like there's a lot of data moving around, but they're all doing their own separate conversations.

所以一般来说，同一个共享网络上的通信越多，整体就越平稳、越均匀。

And so generally speaking, if the more communications you add onto the same shared network, the more things smooth out and become even.

这当然很好。

And so that's great.

你可以利用大数统计的效应。

You can take advantage of the statistics of large numbers.

但遗憾的是，当我们看训练模型时的情况，恰恰相反。

Unfortunately, when you look at what we're doing when we're training things, it's exactly the opposite from that.

我们用的是全球最快的那些GPU，让它们协同完成同一个任务，

We're talking about a lot of the world's fastest GPUs and making them all work together on a single task,

所以这件事才这么难。

which is why this stuff gets hard.

把海量数据吞进去，让AI模型并行学习这些数据，这才是我们能打造出更智能、更强模型的关键。

That process of basically ingesting all of this data and teaching the allowing the AI model to learn from that data in parallel is what allows us to build smarter, better models.

但假如有一块GPU稍微慢了一点……

But let's say one GPU goes a little slow.

那其他所有显卡都得等着它。

Well, now all the other GPUs have to wait for it.

这完全是浪费时间。

That's all wasted time.

或者某个显卡被宇宙射线击中，数据位出错了，直接停了。

Or one of the GPUs, you know, a cosmic ray hits it and some bits flip and it stops.

好吧，那这一整个步骤可能就白做了，我们得回退，或者停下来好好看看出了什么问题。

Well, Okay, now that whole step is maybe not useful and we have to maybe roll back or kind of stop and take stock of what has happened.

而当我们停下来检查的时候，所有的显卡都在干等着，什么有用的事都没做。

And while we stop and take stock, like all the GPUs are not doing useful work.

是的，这里关键的一点是，显卡之间的通信其实本身就是计算的一部分。

Yeah, A key thing here is that the communication between the GPUs is actually part of the computation.

对。

Mhmm.

它们其实是在协同完成一个巨大的计算任务。

It's like they're doing one big computation across all of them.

它们并不是在做不同的事情。

It's not they're doing different things.

我们必须让它们互相通信，才能确定这一步计算的最终结果。

We have to actually have them communicate with each other in order to agree on what the the result of that step of computation is.

这简直是网络能承受的最糟糕的负载了。

And that's just about the worst possible workload you could think to put onto a network.

过去几十年，行业一直在慢慢改进这种通信方式。

And so the way the industry has evolved over the last couple of decades has been slowly coming up with improvements on how we actually do that.

但直到最近，规模还不够大，问题还不明显。

But until recently, the scale wasn't enough that it really mattered.

以前我们还能照搬互联网的做法，只是放大一点而已。

You could get away with doing the same sort of things we did on the Internet, just bigger.

但现在这种做法行不通了。

But you can't get away with that anymore.

所以我们开始思考新的方法，来应对这种高度同步的计算需求。

And that's why we tried to think of different ways to actually solve these problems to cope with these very synchronized workloads as we scale things up.

人工智能已经彻底改变了世界上的很多东西。

AI has upended a lot of things in the world.

它确实彻底改变了科技公司对数据中心建设的思考方式。

It has definitely upended the way that technology companies have to think about the data centers that they're building.

过去互联网时代的大型云服务商，

So conventional hyperscalers for kind of the web era.

负责搭建这些系统的团队，和具体的计算任务几乎是脱节的。

The teams that built this were very kind of disconnected from the individual workloads.

他们的目标很简单，就是提供一片无穷无尽的计算资源。

Know, the the goal was to basically just provide an ocean of compute.

而人工智能迫使我们从根本上重新思考这个问题。

AI has forced us to think very differently.

特别是OpenAI，很早就意识到，系统架构和设计本身，和模型训练是密不可分的。

And OpenAI in particular has been kind of at the forefront of realizing that the systems and the design of the systems is integral to the training of the models.

你不能让基础设施团队在一栋楼里，只管提供计算资源，而模型团队在另一栋楼里，单靠这些资源去拼命优化模型。

And that you can't just have your infrastructure team sit over here in one building and kind of deliver a notion of compute and your model team sit over here and like try to make the best model that they can on that compute.

你真的需要从整体上进行协同设计。

You really have to do kind of a co design across these whole things.

所以，我们团队就直接坐在研究人员旁边，每天跟他们讨论，他们的任务怎么才能更好地适配现有的服务器。

And so, you know, we, on my team, sit literally next to the researchers and talk with them every day about, you know, how they can best you know, how their workloads fit best onto the existing servers.

通过这种方式，我们能清楚地看到哪些地方最让人头疼。

And in doing that, we learn a lot about where the pain points are.

而且，大型训练任务期间，我们得随时待命。

And, you know, we are on call for the big training runs.

一旦系统出问题又没法立刻修复，半夜被叫醒是家常便饭。

You know, we get woken up in the middle of the night if something is broken and can't be fixed.

在这个过程中，我们就开始想：下一代能不能把这些毛病改掉？

And so through that process, you know, and you start to go, Oh, well, what if in the next generation we fix these problems?

如果我们不照搬过去为网页规模设计的数据中心架构，会怎么样？

What if we didn't build data centres with the same properties as we did for web scale workloads?

如果我们能一个个解决这些环节的问题，会怎样？

What if we instead, you know, fix this piece or this piece or this piece?

我觉得网络一直是我们的一大痛点。

And I think the network has been a real source of pain for us.

是的。

Yeah.

所以当你在搭建数据中心网络时，这些GPU都要同时进行计算和通信，需要极大的带宽。

So when you're building a data centre network, all of these GPUs, because they do this computation and they all need to talk at the same time, you need a lot of bandwidth.

明白。

Mhmm.

但问题是，单靠一个交换机，甚至一层交换机结构，根本无法满足需求。

And the problem with that is you can't build that with with a single switch or even a hierarchy of switches.

你得搭建多层交换机的层级结构。

You have to build hierarchies of hierarchies of switches.

这就意味着，当一个GPU和另一个GPU通信时，数据在网路上可能走的路径非常多。

And so that means that when you communicate from one GPU to a different GPU, there are many different paths your traffic could take through that network.

因为我们用了大量不同的交换机，数据可能有成千上万种路径可以选择。

Thousands of different paths the traffic could take through the network because we build so many different switches in there.

这些大楼里有好几千个交换机。

There's several thousand switches in one of these buildings.

这就带来了一个有趣的问题：从一个地方到另一个地方，到底该选哪条路径？

Now that gives us an interesting problem, which is which path do you take from one place to the other?

如果你的要求是必须让两个GPU之间的通信尽可能快，而你随便选了一条网络路径，要是运气好，没人跟你走同一条路，那就没问题。

And if you basically have the requirement that I won't be able to send from one GPU to another GPU as fast as possible and I choose, say, a random path through the network, if I get lucky and nobody else chooses the same path, then great.

但要是运气差，有两个人选了同一条路，速度就慢了；要是十个人都选了这条路径，那就慢得不行。

But if I get unlucky and two people choose the same path, we go slower and if 10 people choose the same path, we go really slow.

我们过去设计互联网时用的那种统计复用方式，在构建数据中心网络时根本行不通。

And so this statistical multiplexing that we used to have when we designed the internet just doesn't work out very well when we're trying to build these networks for our data centers.

所以我们才介入进来，试着用一种完全不同的方式来设计。

And so that's where we came in to try and design things somewhat differently.

也许说得更具体一点，

Maybe to put a fine point on that,

我们之前提到的这种工作负载的同步特性，正是导致这个问题如此严重的原因。

the synchronous nature of the workload that we talked about earlier is why this becomes such a problem.

重点不是一对GPU平均能多快通信。

It is not about how fast can the average pair of GPUs talk to each other.

关键在于最糟糕的情况会发生什么。

It's always what is the absolute worst case that occurs.

对。

Mhmm.

想象一下，有成千上万块GPU在互相通信。

So if you think about you've got thousands of GPUs that are trying to talk to each other.

这意味着有数万甚至数十万条网络流，通过数以万计的链路传输。

So there's tens of thousands or hundreds of thousands of network flows on tens of thousands of links.

你必须纵观整个网络，找出哪一条链路成了最大的瓶颈。

What you have to do is you have to look at that entire network and go, what is the link that got most bottlenecked here?

正是这条链路决定了所有GPU的运行速度，也决定了数据在这套系统中传输所需的时间，因为所有操作都是同步进行的。

That one link is going to set how fast all of your GPUs are able to work and how much time it takes for you to move data through this because everything is proceeding in lockstep.

过去我们或许依赖或利用过平均统计规律，但现在我们没这个条件了。

And so whereas previously we might have been subject to or kind of taken advantage of average statistics, We don't have that luxury anymore.

我们现在面对的，是尾部中的尾部。

We instead are subject to, like, the tail of the tail.

我们关注的是第100百分位的数据。

We call P 100, the hundredth percentile statistics.

这带来的系统需求，和过去能靠大数定律来兜底的时候完全不同。

And that leads to very different systems requirements than when you can kind of rely on the law of large numbers to take care of you.

另一个问题是，当我们搭建网络时，已经尽最大努力了，选最好的设备厂商，用最优的光模块等等。

The other problem with this is that when you build your networks, we build the best possible networks we can, we go to the best equipment vendors, we use the best optics and so forth.

但一旦规模真正做大，故障总是难免的。

When you really scale things really big, things are always going be failing.

链路会断，交换机也会出错，不得不重启，诸如此类。

Links will fail, switches will get confused and will have to be rebooted and so forth.

这些故障中的任何一个，都会影响网络上的流量。

Any one of these failures is going to affect the traffic running over the network.

所以，如果我们只关心第100百分位的性能，而这时网络里又恰好有一条链路出了问题，那会怎样？

And so if you've got this problem where we only care about the one hundredth percentile and then a link fails in the network, what happens?

东西总会出问题的。

Well, stuff will fail.

在路由重新收敛之前，我们可能还有点时间，可以把流量切换到别的路径上。

We may have time before the routing reconverges and then we move the traffic onto a different path.

那里会出现一次短暂的中断。

We take a glitch there.

这种中断可能持续很久，代价不小。

That can be quite a long glitch that can cost us.

更糟的是，我们甚至可能让某个数据传输直接失败。

Or worse, we can actually cause one of these communication transfers to fail.

只要一次传输失败，整个任务都有可能崩溃。

A single transfer fails and you could end up with the whole job crashing.

所以我们一定要尽量避免这种问题。

And so that's we really want to avoid that kind of problem.

我们要设计一种使用这些网络的方式，不仅要能应对可能出现的临时拥塞，还要在设备出故障时也能继续运行，几乎让人察觉不到异常。

We want to build a way of using these networks that is resilient to not only the potential transient congestion we might build in the network, but also when things fail, we just want to be able to carry on and basically not notice.

但这要求我们从一开始就以不同的方式设计网络协议。

But that requires that we actually design the network protocols differently from the start.

你没法把这套方案硬加到现有的网络协议上。

You can't retrofit this onto existing network protocols.

对。

Yeah.

你描述的这个问题真的很有趣，因为我觉得，比如说，你有上千个GPU的时候，可能只有十分之一的概率会出问题，之类的。

It's an it's a very interesting problem as you describe it because I can see that where you could say, like, oh, well, we have a thousand GPUs, there's only one chance out of 10 that they're gonna fail or whatever.

但现在你有一百万个GPU了。

And now I have a 100,000 GPUs.

那你觉得会怎样？

Well, guess what?

你随时都会遇到故障。

I'm gonna have a failure all the time.

每次你扩大规模，都得解决这个问题。

And that's what you have to sort of solve for each time you scale it.

那这个问题出在哪儿呢？

So where does this break?

无处不在。

Everywhere.

哪儿都是问题。

Everywhere.

没错。

Right.

就是这样。

There we go.

如果你想想

If you think

设备平均故障间隔时间，不管哪类设备，总会在你整个大楼里的某个地方出故障。

about sort of the meantime between failure of the equipment, this for any particular range of equipment, you've got some time between which something will fail somewhere in your building.

当然，设备越多，成本不变的情况下，这个故障间隔时间就会越来越短。

And of course, the bigger we get, with the same cost of equipment, that time comes down further and further.

最终你会达到这样一个地步：设备故障频繁到根本没法完成大型同步任务。

And eventually you get to a point where actually something is failing sufficiently often that you don't get any work done on a large synchronous workload.

这种情况我们绝对不能接受。

And we can't have that happen.

所以我们必须换种方式来解决这个问题。

So we have to do things differently to make that work.

也许吧。

Maybe, yeah.

这里有个简单的数学逻辑：如果故障是独立发生的，那么当你把系统规模翻倍，故障间隔时间就会减半。

So the very simple math here is like you can basically assume that if failures are independent and you double the size of your system, you're going to have half the time between failures.

对吧？

Right?

平均无故障时间直接减半。

Your mean time to goes down by half.

对网络来说，关键在于，每块GPU背后都有几十个，甚至上百个网络组件。

The important thing to think about here for the network is for every GPU, we have tens, if not hundreds of network components.

就拿一个GPU连到一个网络适配器来说吧。

So even just like say you've got one GPU connected to one network adapter.

如果这个适配器里有光收发器，那里面可能就有四颗激光器。

In that network adapter, if it has an optical transceiver in it, maybe you'll have four lasers.

在收发器的另一端，还会再有四颗激光器。

On the other end of that transceiver, you'll have another four lasers.

所以，仅仅把一个GPU连到它的第一级交换机，激光器的数量就已经比GPU多了一个数量级。

And so already, just connecting that one GPU just to its first hop switch, you've had an order of magnitude more lasers than you have GPUs.

再算上多层交换结构，网络里的组件总数就会比边缘设备多出好几个数量级。

Now add in multiple layers of switching and you start to get into, you know, several orders of magnitude more components in the network than you have kind of at the edge of your network.

因为我们需要巨大的带宽，所以必须这么搭建网络，没法简化成只用几个组件，不然GPU的性能根本发挥不出来。

Because we need to have so much bandwidth here, so we have to build these networks, we can't, you know, kind of taper them down and only have a couple of components in the network because then we'd be starving the GPUs.

它们就没法以最快的速度完成计算任务。

They wouldn't be able to actually kind of use their full capability to do math as quickly as possible.

结果就是GPU只能干等着，白白浪费时间、金钱和电力。

Instead, we would kind of we would just be letting them sit idle and wasting time, money, energy.

我们训练模型的速度会慢很多。

We would get trained models more slowly.

这可不行。

It would be bad.

所以我们建了一个非常庞大的网络，现在网络里的组件数量远远超过了边缘设备的数量。

So we built a very big network, but now you have many, many, many more components in that network than you have maybe at the edge of your network.

同一个建筑里，光链路的数量可能高达数百万条。

You could have literally millions of optical links within the same building.

规模真是大得惊人。

So it's huge scale.

你知道的，

You know,

你提到最初的数据中心是用来搜索云端、收邮件之类的。

you mentioned the data centers originally were I go do something to search the cloud, to get my email, whatever.

那时候可能只跟一台服务器打交道。

I may be talking to one server.

那里可能还有个备份。

There might be a backup there.

现在数据中心里互相通信的计算机，比几年前用户连进来的时候还要多得多。

The idea that we would be having more computers inside a data center talking to each other than we did just a few years ago, people trying to connect to it.

那这些协议是怎么演变的呢？

And so how have these protocols evolved?

我当初开始接触网络，是因为有个特别清楚的记忆：2017年参加OFC光通信会议时，有个Facebook的演讲者上了台。

One of the things that got me into networking in the first place, I have a very distinct memory of I was at a conference, OFC, it's an optical communications conference back in 2017, and they had a presenter from Facebook.

他展示了一张堆叠折线图，一边是发给终端用户的流量，另一边是数据中心内部的流量。

And he put a chart up that had a stacked line chart of the traffic that they serve to their end users and the traffic that goes inside of their data centers.

数据中心内部的流量在疯狂增长，而发给终端用户的流量却基本没变。

And the traffic inside of their data centers was exploding, even while the kind of amount of traffic that they were sending to end users was staying constant.

那时候还没出现GPU集群和AI呢。

This is way before GPU clusters and AI.

AI其实就是把过去所有人遇到的系统难题，直接放大了十倍。

So this is what AI does is it takes all of the systems challenges that people were having previously and cranks them up to 11.

为了解决这个问题，你一直在研究一种叫多路径可靠性连接的方法。

So to address this, you've been working on a method, a Multipath Reliability Connection.

核心思路其实是，我们在网络中管理拥塞时，可以综合运用多种手段。

The insight was basically that you have we try to manage congestion in networks.

把这些方法结合起来，就能有效应对。

And so there are several pieces that you can pull together and do this.

第一个关键点是，把数据包分散到多条路径上，这样就能让各条路径的负载分布得非常均匀。

And the first part of the insight was if you spray the packets across many paths, you can low balance those paths through the network really equally.

只要网络拓扑有足够的带宽，就能避免出现网络瓶颈。

And if you do that and you build a network topology that has enough capacity, then you don't cause hotspots in the network.

这样一来，唯一可能出现拥塞的地方，就是当多人同时向同一个目的地发送数据时。

It leaves you with just one place where you have congestion, which is if multiple people try to send to the same destination at the same time.

但这也带来一个问题：数据包走不同的路径，到达时顺序可能会乱掉。

But it also leaves you some problems because the packets can get jumbled up in transit because they're taking different paths.

如果真的发生拥塞或丢包，就很难判断到底是真的丢包了，还是只是包在路上耽搁了，还没到。

And so if you do manage to cause congestion and cause loss, it's a little bit difficult to figure out whether you got loss or whether you should still be waiting for packets because, you know, they got jumbled up in transit.

所以第二部分是一种我们称为数据包裁剪的技术：当网络出现拥塞、队列要溢出时，通常我们会直接丢弃数据包。

And so the second piece of this is a technique we call packet trimming, which is if you're causing congestion in the network and you would overflow a queue, normally we just drop the packet.

这就带来了不确定性。

And then we've got ambiguity.

数据包是丢了么？

Did it get lost?

数据包是丢了么？

Did it got lost?

我该等多久才好？

How long do I need to wait for it?

但我们采取的做法是，把数据包的有效载荷裁掉，只把很小的包头转发到目的地，这样目的地能立刻请求重传，我们就能重新发送这个包。

But what we do instead is we will trim off the payload of the packet and just forward the little tiny packet head to the destination, which can immediately request a retransmission and we can retransmit that packet.

这样一来，就彻底消除了困惑——不知道是拥塞导致丢包，还是包只是 reordered，还得再等等。

And that totally removes the sense of ambiguity as to whether we lost packets due to congestion or whether we should still be waiting for them because they got reordered.

有意思。

Interesting.

所以我要确认一下，那段话是说，

So just making sure that the part saying,

你在吗？

are you there?

能传过去，然后你就能判断了，再发送剩下的内容。

Goes through, and then you can figure that out and then send the rest.

对。

Yes.

你必须清楚，到底是该继续等它，还是不该再等了。

You really need to know that you should still be waiting for it or you should not still

不该再等了。

be waiting for it.

那

What

这对最终用户意味着什么？

does this mean for the end user?

这意味着你们会更快地从OpenAI获得更强大、更智能的模型。

The biggest thing that this means is that you're going to get better models, more intelligent models faster from OpenAI.

所以MRC能让我们加速整个研究和部署流程的每个环节。

So MRC allows us to accelerate every part of our research and deployment pipeline.

它让每个用户都不用担心任务失败，也不用操心任务调度方式，或者因为和别人的任务在同一机架上运行而影响性能。

It allows individual users to not worry about their jobs failing, not worry about how their job has gotten scheduled and whether the performance of it is going to be different because they're, you know, placed on the same rack as someone else's job.

它让我们能更快、更稳定地训练前沿模型，真正大幅提升整个流程的速度和可靠性。

It allows us to train frontier models much faster, more reliably, and really just to turn the entire crank of that pipeline much faster and much more reliably.

所以你们会看到我们不断推出越来越令人期待的新功能。

So you should expect to see an ever increasingly exciting pipeline of releases from us.

情况很不错。

The vibes are good.

情况很不错。

The vibes are good.

情况非常好。

Vibes are very good.

这个想法源于我们过去几十年积累的大量研究工作。

The idea came out of a lot of research work that we've had over over the last few decades.

我们并没有从根本上发明什么新东西。

We're we're not fundamentally inventing anything new.

我们只是把别人已经发明的东西整合起来，形成一套功能。

We're just just doing things that other people have invented, pulling the combination together into a set of features.

于是我们组建了一个团队，大家都对这件事感兴趣，去年终于实现了落地，从第一批硬件到位，到真正跑起来训练模型，整个系统顺利运转，只用了几个月时间。

So we formed this group of people who are all interested in doing this and last year we finally got to the point where we were able to deploy this and we went from in a few months from the first hardware available to actually running and training models and it actually all operating.

这样一来，我们就避免了之前提到的网络拥塞问题。

So this has the result that we don't cause that congestion that we talked about before.

另一个非常棒的特点是，如果网络中某个地方出了故障，虽然所有流量都会受到一点影响，但影响非常有限。

The the second really nice property is that if something fails in the network, every single one of these flows go through there will probably be affected by the failure, but it'll only be affected a little bit.

在经过几次网络往返后，我们就会自动绕开故障的链路。

And within a few round trip times across the network, we stopped using a failed link.

所以，链路故障导致整个网络瘫痪的问题，就彻底解决了。

And so this problem of links failing, bringing down the other network just goes away.

网络一侧的各个数据流在穿越网络时，都会自动绕开这些故障点。

All of the flows themselves from the network interface at one side, network interface are just avoiding those failures as we go through the network.

这就是自我修复。

So self annealing.

没错。

Exactly.

是的。

Yes.

我觉得马克说得还保守了。

So I think Mark has mildly undersold this.

好吧。

Okay.

但也许我们该

But maybe we should

马克，别谦虚了。

Mark, come on, man.

所以

So

传统上，当网络中的某个链路断了，通常是一端的交换机，或者两端都会检测到这个故障。

conventionally, like, when a link goes down a network, what happens is, you know, one side of that link, the switch at the one side of that link, or maybe both sides of that link notices.

但接着它得通知所有相邻的设备，说这条链路断了。

But then it has to tell all of its neighbors that that link went down.

然后这些设备又要各自通知它们的邻居，说这条链路断了。

And then they have to tell all of their neighbors that that link went down.

这就成了一个分布式系统的问题。

And so you have a distributed systems problem.

通常用一种叫BGP——边界网关协议的技术来解决，本质上就是个‘传话’机制，让这边的一条链路，最终能通过五六跳甚至七八跳，把消息传到网络另一头的交换机，告诉你：别走这条路了，去不了那个目的地，得换别的路。

It's conventionally solved with a technology called BGP, Border Gateway Protocol, which is basically just like a gossip protocol that allows, you know, one link over here to eventually tell this switch all the way over here, maybe through five or seven hops through the network that, hey, you can't get to this destination if you take this link, you have to use these other links.

这其实是个有收敛时间的分布式系统问题。

That's a distributed systems problem that has a convergence time.

而MRC的做法是，直接消除了这种协调的必要性。

What MRC has done is it has taken that and it has broken the need to coordinate.

每个终端都能独立快速发现：这条路径不能用了，直接停止使用就行。

Every endpoint independently very quickly detects, hey, that path I shouldn't use that path and just stops using it.

这听起来可能有点反直觉，因为你可能会想，要是有个中央系统告诉我哪条链路断了，再由它统一通知大家，岂不是更简单？

And this is maybe counterintuitive because you would think, oh, it's easier if I just have some central authority that tells me that, you know, this link is down and that central central authority can distribute that information.

任何等过网站更新的人都知道，这根本行不通。

Anybody who's waited for a website update knows that it's not going to work.

没错。

Right.

中央系统通常也被称为单点故障。

Central authorities are, generally speaking, also known as single

点故障。

points of failure.

所以我们现在不用再等整个网络收敛了，那过程可能要几秒，极端情况下甚至要几十秒。

And so instead, what we've done here is we no longer have to wait for this whole convergence process to to occur, which can take seconds or in the tail, tens of seconds.

相反，大家通常在毫秒级内就能察觉到，立刻停止使用这条链路。

Instead, everyone within, generally speaking, milliseconds notices and just stops using that link.

这非常关键，因为以前一旦链路断了，整个任务就得停几秒钟，等网络稳定下来。

This is a very big deal because previously, you know, the link goes down and the whole job stops for a few seconds as we wait for kind of the network to stabilize.

这段时间，GPU 就在干等着，什么正事都没做。

That's time again that the GPUs aren't doing useful work.

而且随着规模扩大，这种零零散散的几秒钟会越来越多。

And as you again scale up, you're going to have more and more and more of those individual little seconds.

我们是在数据中心建设过程中就启用了这个功能。

And here now, what we've observed is that like, you know, we turned this on as we were basically as the data center was being built.

就像马克说的，硬件一到，我们几个月内就跑起了任务，开始训练了。

As Mark said, we were able to, you know, get jobs up and training within months of hardware arriving.

建这样的数据中心，需要大量人工操作。

There's a lot of manual labor that goes into building one of these buildings.

很多地方都是共享节点，比如一个机房的光纤要连到另一个机房，技术人员得现场接线，诸如此类。

There's a lot of kind of shared points where fibers from one data hall are coming in and technicians are trying to assemble another data haul or things like that.

所以我们发现，因为这些频繁的人工操作，链路一直在上下波动，频率远超正常硬件故障的水平。

And so what we saw was that because of all of this kind of manual effort that was going on, links were going up and down all the time, like way even way more often than you would kind of hope in just due to natural failures.

我们完全没在意。

We did not care.

我们甚至都没察觉到。

We didn't even notice.

MRC 自动处理好了。

MRC just took care of it.

它会自动检测，然后说：嘿。

It just kind of would detect, hey.

那条路走不通。

Can't use that path.

换下一条。

Move on to the next one.

根本顾不上。

Didn't care.

简直太不可思议了。

It was incredible.

除此之外，由于MRC能自动处理这些问题，一旦它开始绕开故障路径，我们过去通常还得在网里跑路由协议，来找真正能用的路径。

So the other thing this gives you beyond just that is because once it's handling that by itself, once MRC is already working around the failures, the traditionally, we would probably still have been running a routing protocol in the network to find the path that actually work.

但路由协议本身很复杂，交换机也很复杂，交换机的软件更是复杂。

But routing protocols themselves are complicated and switches are complicated and switch software is complicated.

这些环节都可能出问题。

And these are all things that can fail.

后来我们意识到，其实MRC自己就能判断哪些路径还能用。

And we realized that actually MRC itself was able to figure out which paths were still working.

所以我们干脆决定把路由协议全关了。

And so actually we just decided that we would turn off the routing protocols.

我们在所有交换机上都改用静态路由，而且是最大规模的部署。

We use completely static routing in the switches at the largest possible scale.

有些路径会断掉。

And so some paths are broken.

那又怎样？

Who cares?

MRC会找到那个还在运行的代理，然后继续工作。

MRC will find the broker once it still work and and keep going.

这样一来，我们就把网络管理中一大坨不必要的复杂性彻底去掉了。

And that just removes a whole complexity out of our network management that we just don't need anymore.

我们根本不关心交换机的控制平面有没有收敛，因为它根本不需要收敛。

We don't care about whether the switch control plane is converged because it doesn't need to.

整个系统都是静态配置的。

It's entirely static.

它们在启动时就加载好了一套配置。

They have a configuration that they have at boot time.

启动之后，路由表就再也不变了。

They boot up and they never change their routing tables from then onwards.

这可是个大工程，得跟很多人紧密合作。

So this is a very big effort working with a bunch of different people.

能聊聊你们的合作伙伴吗？

Care to talk about some of the partners?

是的

Yeah.

我们一直与微软合作，他们为我们建造了Fairwater数据中心。

We've been working with Microsoft who build our Fairwater data centers for us.

我们还与英伟达、博通、AMD和英特尔合作，共同制定这套标准，以便为我们的新一代超级计算机打造硬件。

And then we've been working with Nvidia, Broadcom, AMD, and Intel to standardize this specification with all of these guys to actually build our hardware for our new supercomputers.

这挺有意思的，因为作为这项技术的使用者，你听人们讨论时总在问‘下一个模型什么时候出来？’

It's interesting because as a user of this technology and you listen to sort of the way we sort of think about things where people talk about, like, when's the next model coming out?

好像每年都会有一个软件更新，但其实不是这么回事。

Like, it's a software update that comes up every year, but it's not.

每一个模型本质上都是一个研究项目，完全取决于训练过程中的进展。

Every every model is essentially a research project, and it's dependent upon what goes on in the training.

我们会试着估算需要多长时间，也努力预测它最终会达到什么程度。

And you try to have an estimate of how long it's going to take and we try to predict where that's going to be.

但这种可靠性听起来会是一个巨大的优势。

But this kind of reliability sounds like it's going to be an incredible advantage.

天啊，确实如此。

God, yeah.

我

I

听那些更早来这里的人说，简直是一堆噩梦，他们经常被吵醒。

would say, from talking to the people who were here even earlier, I mean, you hear just horror stories about what it was like, how often they were getting woken up.

我记得有一次在食堂看到搞网络的那些人，个个愁眉苦脸，因为他们根本不知道为什么运行突然停了，就只是……

I remember walking in the cafeteria and seeing of the people working on networking sort of sad faces because they didn't know why something had stopped in a run, and it was just

是的。

Yeah.

我们听到的全是正面反馈，说用了MRC之后集群特别稳定，运行得非常好，研究人员根本不用再操心这些事了。

I we have heard nothing but universally positive feedback about how stable the clusters with MRC are, how well they're working, how basically the researchers don't have to think about this anymore.

再看看那些统计数据，你才会意识到平时有多少东西一直在出问题，但他们

And then you look at the statistics and you realise just how much stuff is breaking all the time and they're

根本没察觉。

not noticing.

是的，某种程度上，这确实是最理想的状态，对吧？

Yeah, it's it's I mean, to some degree, this is the ideal, right?

我之前也说过，我们确实在不断突破基础设施的极限。

I mean, I know I said earlier, right, we are pushing the limits of infrastructure.

所以，永远不可能完全不管基础设施的问题。

And so there's never going to be a time that you can just completely ignore infra.

理想的情况当然是研究人员根本不用操心这些，但现实里这永远做不到。

Know, the ideal world, right, is researchers don't have to think about this, but that's never going to be the case.

但每次看到研究人员不再提网络，也不再关心某个集群用的是什么网络协议时，我们就知道又成功了一步。

But every time we we know we've won when researchers stop saying network or stop needing to know what network protocol this particular cluster is using.

而且说实话，能集中精力做别的事真的很好——当然，别的地方还是会出问题，还有很多工作要做。

And yeah, it's been really nice to basically be able to focus all of because there's, don't get me wrong, there's plenty of other things that break, there's plenty of other work to do.

但至少这件事让我们移除了一个阻碍我们继续扩展、无法更快推出更优模型的关键障碍。

But this has really allowed us to remove one of the key barriers to continuing to scale and to being able to deliver newer and better models on a much you

我们知道，我们正在各个层面推进规模扩张，包括提升迭代速度。

know, we are we are trying to scale everywhere, including our velocity.

你们决定让所有人都能免费使用这个系统。

And you have all decided to make this open to everybody to use.

是的。

Yeah.

所以这个规范将会通过OCP发布为开放标准。

So the specification is due out through OCP as an open standard.

正如你所说，我们决定向所有人开放这项技术。

And we've, as you say, decided to open this up for everybody to use.

我们非常支持开放标准和开源。

We're big believers in open standards and open source.

我们的所有网络都基于以太网，而以太网本身就是一个开放标准。

We are building all of our networks on top of Ethernet, which itself is an open standard.

当整个行业能够跟上我们所面临的这些挑战时，我们才能从中受益。

And we benefit when the industry has velocity, when the industry can keep up with the things that we are trying to do on the challenging side of things.

如果我们大家都致力于部署自己认为最优秀的解决方案，那对所有人都是有利的。

And so it's in everybody's interest if we're all trying to actually deploy what we think are the best solutions in this space.

关于AI基础设施建设的规模，媒体报道已经非常多。

There's no shortage of coverage of the scale of the AI build out.

从个人角度来说，如果这条供应链被割裂，那真的太可惜了。

Think on a personal level, I think it would be a real shame if that supply chain was fractured.

有些人因为想占一点小便宜，就投资完全不同的技术和底层硬件。

You have people investing in totally different technologies and underlying hardware just because they're trying to get some small advantage.

我觉得这真的很好，毕竟这会成为开放标准。

Think it is, I mean, I'm really excited that this is going be the open standard.

这不仅对OpenAI有利，对其他所有人也都有好处。

I think it will really benefit other people outside of OpenAI.

如果我们大家都能朝着同一个方向努力，对所有人都有利。

It also benefits all of us if we are kind of all pushing in the same direction.

基础设施其实是整个行业共同的命运。

Infrastructure is kind of this shared fate of the whole industry.

我觉得把这项技术开源，让大家都参与进来，是非常好的决定。

And I think it is a very good thing that we are open sourcing this and kind of bringing everyone along.