第033集 - ClusterMAX 3.0来了！Neoclouds排名（Neoclouds、GPU）| Sam Harshe、Pratt Bhatt、Jordan Nanos

这里我突出显示的是firmness，就是有两次运行。

On here, I'm highlighting firmness, which is like, there's two runs.

一次是我们一起做了很多工作之后跑的，另一次是更早之前跑的。

One that we did after a bunch of work together and one that was done earlier.

屏幕上的图表显示的是时间，也就是消息在这个全对全集合中通过网络传输的延迟，时间越长越不好。

And so the chart on screen shows time, which is like the latency for a message to actually pass over the network in this all to all collective and more time is bad.

所以越低越好。

So lower the better.

这明显是我们发现的网络配置问题。

And this is clearly an issue with the network configuration that we identified.

实际情况是，如果我们有一个受集合约束的工作负载，就像我们这里有的这个，我可以往下看我们的TorchTitan GPTOSS训练任务，它使用了专家并行，并且是通信密集型的，你会看到firmness开箱即用的性能是每GPU每秒1455个token，而其他一些测试中的性能大约在3500左右。

What happens is that if we get a collective constrained workload, which we do have here, and I can go down to our TorchTitan GPTOSS training job, which is run with expert parallelism and is communications bound, is that you see the performance on firmness out of the box was fourteen fifty five tokens per second per GPU, whereas the others on some testing were up around 3,500.

所以开箱即用的性能差了很多，差不多50%，这是个问题。

So way, way less performance out of the box, like 50%, which is an issue.

如果你用我们的分析工具深入查看这些步骤，这里的黄色条是暴露出来的通信时间，你不应该看到这个的。

And if you drill down into these steps with some of our profiling, this yellow bar here is exposed communication time that you're not supposed to be seeing.

所以，退一步看，把所有供应商都标出来，很明显，Firmness在每一步上花的时间都长得多，因为那些暴露出来的通信时间都显示在里面了。

And so just like backing out and showing all of these providers highlighted together, firmness obviously on a per step basis is just taking way longer and that's because all of these exposed comms are actually being shown in there.

所以，这不是一个差的供应商。

So, this is not a bad provider.

这家公司已经部署了成千上万的GV300。

This is a company that has stood up thousands and thousands of GV300s.

他们有很多满意的客户。

They have happy customers.

他们最终在我们的测试之后解决了这个问题，但这说明，一个没有经历过和客户反复磨合的公司，是不会知道如何为GPU正确配置网络的。

They were eventually able to fix this with us after we did some testing, but it's an example of like a company that has not built the battle scars of working through that back and forth with a customer, is not gonna have an idea of how to set up a network correctly for the GPUs out of the box.

而且，我们通常就是信任那些对你最关心的GPU有经验的人。

And, know, we just default to trusting people who have experience with the GPUs that you care about the most.

那就是最新最先进的。

And that's the latest and greatest.

GB200、GB300系统。

GB 200, GB 300 system.

明年你要用Vera Rubin，你会信任谁呢？

Who are you gonna trust for your Vera Rubin going into next year?

很可能就是那些经历过这些磨难、知道哪些NVIDIA的东西可以信赖、哪些其他供应商文档里的东西可以信赖，以及哪些可能不能信赖的人。

Probably the people who have been through this battle scars before and know what they can trust from NVIDIA, what they can trust from their other vendors in the documentation and what they, maybe they can't.

好的。

Okay.

咱们先不谈网络了，来聊聊存储吧。

Let's let's move on from networking and talk a little bit about storage.

Sam，你和一个供应商有过一次有趣的经历，想让他们把存储弄好，结果最后聊到了安全问题。

Sam, you had a fun experience with one of the providers trying to get them to get their storage to work correctly, which specifically led to a conversation about security.

也许你可以给我们讲讲那个经历。

Maybe you can tell us a little little story about that experience.

当然可以。

Sure.

这个故事有好几层，我想想。

This is a story with multiple layers, I guess.

我们收到了一封邮件通知。

We got an email notification.

这就是你想到的那个，乔丹。

This is the one that you're thinking of, Jordan.

我们收到了一封关于节点宕机的邮件通知。

We got an email notification for a no down.

就像之前提到的，我们会注入一些XID来测试健康检查，但通常我们会在测试快结束时才这么做，以防它没有恢复。

So as mentioned earlier, we do synthetic injections of XIDs to test the health checks, but we normally do this at the end of testing just in case it doesn't come back.

我们不想在测试过程中过早地触发这个。

We don't wanna run shorthand and for all testing.

所以我们在测试刚开始不久就收到了节点宕机的通知。

So we got a email notification that a node was down pretty early on.

于是我们截了图，发到了Slack上。

So screenshot it, put it in the Slack.

这是谁负责的？

Is this anyone?

没有。

Nope.

不是我。

It's not me.

所以乔丹，你是不是在动我的集群？

So Jordan, are you messing with my cluster?

没有，兄弟。

No, man.

那是真的。

That's real.

乔丹，

Jordan,

你在

what are

你在干什么，老兄？

you what are you doing, dude?

接下来的测试我们这里只会保留三个节点。

We're we're only gonna have three nodes here for the rest of testing.

结果发现那真的是一个故障，而且这个故障还挺有意思的。

Turned out that it was an actual failure, and it was a it was a pretty fun failure.

在ClusterMAX上工作，跟实验室里的性能工程师完全是两回事。

There's there's some way that, like, working on ClusterMAX is the exact opposite of being a performance engineer in a lab.

因为实验室里的性能工程师只希望一切正常。

Because if you're a performance engineer in a lab, you just want everything to work.

而在这里，我们反而希望出问题。

And here, like, we want bugs.

不然我们就没事干了。

Otherwise, we have nothing to do.

所以这算是我们在训练中遇到的一个比较典型的故障。

So this is, like, one of the better bugs that we saw in training.

我觉得如果我遇到一个真正的性能工程师，跟他握个手，我们俩可能都会消失。

Think if I, like, met an actual perf engineer and shook hands with him, we would both disappear.

工程师和蚂蚁，我搞工程的。

Engineer with ants, I engineer.

然后上了集群，有一大堆日志要翻。

So got on the got on the cluster, and there's a lot of logs to dig through.

这其实就是一个例子，说明智能体编程挺有用的。

This is, you know, an example of agentic coding being helpful.

我们当时也不确定到底发生了什么。

We weren't sure exactly what happened.

实际上集群上没人跑任何东西。

No one was actually running anything on the cluster.

结果呢，长话短说，就是Linux内核出了个错。

It turned out, long story, slightly shorter, that there was an error with the Linux kernel.

Linux内核在处理重新分配30个iNode的时候卡住了。

So the Linux kernel had a hard time dealing with reassigning 30 iNodes.

所以，你有一堆数据写在写回缓存里，然后这些数据的所有权要变更。

So you, you know, you have a bunch of stuff written in in the write back cache, and you need and and they change it changes ownership.

在我们的情况里，就是一个旧的slurm任务要结束了。

So in our case, this was like an old slurm job that was ending.

所以我们需要把它停掉。

So we needed to deprecate it.

这就是操作系统后台的那些事，没人关心，也没人想关心。

This is like the behind the scenes stuff of an operating system that no one ever cares about or wants to care about.

那是个很老的Linux版本，我记得我们处理的那个内核大概有几年历史了，里面有些早就被修复的bug。

This was a very old Linux version, you know, about, I think, the the kernel that we were dealing with was a couple years old and had bugs that had been solved fairly long ago.

但有个糟糕的——我就不细说了——一个糟糕的二次时间复杂度的算法，在处理重新分配那30页内存时出了问题。

But there's, like, a bad I won't go into all the details of it, but a bad quadratic time algorithm that was dealing with reassigning all these bad these 30 pages of memory.

这导致CPU被锁死了，所有对应那个NUMA节点的核心都卡住了。

This locked up the CPU, all the all the cores corresponding to that NUMA node.

我不太确定为什么只限制在一个NUMA节点上，但在拆除这个LERM任务时，它把那个NUMA节点对应的所有CPU都锁死了。

I'm not sure exactly why it was constrained to one NUMA node, But it locked up all the CPUs corresponding to that NUMA node as it was tearing down this LERM job.

存储客户端根本得不到CPU时间。

The client for storage couldn't get any CPU time.

所以，你知道，就是一个普通的请求，比如，嘿，你还活着吗？

So there's, you know, an an ordinary request like, hey, are you still alive?

对Luster客户端来说，它没有响应。

To the Luster clients, it didn't respond.

所以这就触发了健康检查。

So that triggered the health check.

它就说，你知道，存储出问题了。

It said, you know, storage is failing.

但是健康检查实际上没法完成它的工作，因为CPU卡住了。

But the health check couldn't actually complete its work, because the CPU was frozen up.

所以那个任务花了好几个小时才完成。

So it took a couple of hours for that job to complete.

最后，CPU终于恢复正常了，因为它忙完了自己的事。

Finally, the CPU actually returned to healthy, because it was done doing its laundry.

而它一恢复正常，就像警察在店外等着一样，立刻就被健康检查了。

And as soon as it returned to healthy, it was like the cops were waiting outside its store, and it got health checked.

所以这就是那封邮件的原因。

So that was the cause of the email.

而且，是的，我的意思是，你最不希望看到的就是一个节点被从你的集群中移除，不是因为GPU真的不健康，而是因为文件系统的健康检查失败，而那个失败又是由于CPU的问题引起的。

And, yeah, I mean, the the last thing that you want is a node that's getting taken from your fleet, not because the GPU is actually unhealthy, but because of a health check failure for your file system that was due to a failure with your CPU.

所以那是一个相当隐蔽的文件系统边缘情况。

So that was a fairly obscure corner case with a a file system.

这意味着，你知道，我整个周末都在烧各种代码令牌，努力找出确切的工作负载来重现这个bug。

It meant that, you know, I was burning all kinds of codex tokens over the weekend, setting goal to try to figure out exactly the right workload to recreate this bug.

但无论如何，是的，我们总结的结论之一就是让存储客户端获得CPU时间。

But in any case, yeah, we want the storage client to get CPU time, is one way of summarizing our conclusion

从那个情况中得出的。

from that.

实际上，这也变成了一个安全问题，对吧？

Actually, this turned into a security issue too, right?

我之前在几集节目里也提到过，就是当你试图找bug时，如果别人有比你更强大的模型，你就无法自卫。

Which I've talked about on a couple of episodes previously now about the asymmetric nature of like when you're trying to find a bug, if other people have more powerful models than you, you can't defend yourself.

Pratt，也许你可以跟我们聊聊你在几个集群上深入做安全测试的经历，当时你想确认自己是否对某个漏洞有防护，但为了开始提问，很多模型就完全拒绝帮你，对吧？

Pratt, maybe you can tell us a little bit about digging into security on a couple of these clusters where you're trying to identify if you were protected against a given vulnerability and in order to even start asking questions, a lot of the models would just completely refuse to help you, right?

这确实是其中一方面。

I mean, that's one part of it.

但另一方面，我注意到的是，尤其是前沿模型，它们就是不愿意利用漏洞。

But the other part of it is that, and this is something I've noticed, is that frontier models especially just don't want to exploit vulnerabilities.

它们倾向于给出误报，比如会说，对，一切都很完美。

They just have a tendency of showing false positives, being like, yeah, everything is perfect.

前几天，我在测试一个集群，基本上就是在摆弄它，确认有没有我想利用的漏洞。

The other day, was testing out a cluster and basically just playing with it, making sure there was a vulnerability that I wanted to exploit.

我记得当时用的是某个前沿模型。

And I was using, I believe, one of the frontier models for that.

那个模型说，对，我们发现了这个漏洞，可以利用它，而且我们实际上能获得那个集群控制平面的主机访问权限。

And the model said, yeah, we have this bug, we can exploit it, and we can actually get host access host access to the control plane of that cluster.

我当时特别高兴，你知道吗。

So I was like overjoyed, know.

我当时就想，这可是个绝佳的机会，能证明CBE确实是个大事。

Was like, this is perfect perfect chance for me to like show that the CBE is actually a really real big thing.

结果模型生成了一个假的POC，我调试了两个小时后，它才说，哦，对。

However, the model actually produced a false POC, and after like two hours of debugging, the model said, oh, yeah.

其实是我搞错了。

Actually, my bad.

这是我之前不小心说出来的。

Like, this was something I accidentally, like, said.

这个其实没法利用。

This is this cannot be exploited.

你可能会觉得那两个小时白费了，但我觉得我同时也学到了很多。

Now you could you could think like those two hours were wasted, but I would say I learned learned a lot at the same time.

而这正是前沿模型特别爱干的事。

And this is something that the frontier models especially do.

它们就是想让你相信那些假阳性。

They just want you to believe in false positives.

这让我意识到，幻觉问题是不是已经解决了？

And that makes me realize that are hallucinations gone?

很可能已经解决了。

Most likely yes.

但是误报问题解决了吗？

But are false positives gone?

绝对没有。

Definitely not.

所以这就是我测试这些集群和利用这些CVE的有趣经历，你懂吧？

So that's my fun experience testing these clusters and just exploiting these CVEs, you know?

是的，兄弟。

Yeah, man.

这方面还有很多工作要做。

Lots more work to be done there.

Sam，说到安全，你脑子里在想什么？

Sam, what's coming to mind when it comes to security?

我们之前在节目里聊过这个话题，但自那以后发生了一些事情，包括OpenAI被黑客攻击，这是漏洞赏金计划的一部分，还有Hacktron AI的人在他们的内部仓库里开了一个PR。

We've talked about this on a previous episode, but a few things have happened since then, including OpenAI being hacked, Part of a bug bounty program, the Hacktron AI guys open a PR in their internal repo.

看起来他们像是在随意写代码，因为他们的单体仓库里大概有110万个PR之类的。

Seems like they're vibe coding because their mono repo had 1,100,000 PRs or something.

所以他们提交了那个百万分之一的PR，来证明他们拥有完全访问权限。

So they put the million and once PR up to prove that they had full access.

这有点疯狂，但他们是众多积极追逐这些漏洞赏金计划的人之一，而目前很多Neoclouds并没有参与这些计划。

A little crazy, but they are one of many that are actively pursuing the, these bug bouncy programs, which not a lot of these Neoclouds are participating in right now.

我肯定有一肚子牢骚要发。

I certainly have a rant about this loaded.

也许我暂时先忍住不说。

Maybe I'll I'll keep it in my pocket for the time being.

我觉得，不管你对AI末日的看法如何，或者你是否认为我们应该继续全速前进，事实是，这些设备极其宝贵、极其强大，但在很多情况下安全防护非常糟糕，非常粗心大意。

I think that, you know, whatever your opinion is about AI doom, or about, you know, whether we should keep pushing full speed ahead, the fact of the matter is that these are extremely valuable, extremely powerful pieces of equipment that are very badly secured in many cases, very carelessly secured.

你知道，没有一套标准的测试可以确保你的集群配置得当。

You know, there's nothing like standard set of tests that you can run to make sure that your cluster is properly configured.

你知道的，现在事情发展得特别快。

You know, things are moving very quickly.

我理解这些公司的员工手头肯定有一大堆待办事项。

I understand that there's, you know, a million things on the to do list of the people working at these companies.

但与此同时，我们却看到一些针对基础软件的漏洞，居然都两年了还没修。

At the same time, we see, you know, CVEs that are two years old that apply to like the basic software that that's running on these things.

其实我们要的也不是什么多高级的功能，只要能把租户隔离好就行。

You know, we're not asking for super fancy things when we want tenant isolation.

你知道，我们肯定不希望看到邻居在 Grafana 里的数据，也不希望看到 CUDA Toolkit 12.0、13.2 或者 whatever 是最新版本之类的信息。

You know, we want not to be able to see what's going on in the Grafana of our neighbors, and we don't want, you know, CUDA two Toolkit 12 o when 13.2 or whatever is is the most recent one.

所以，我的意思是，抛开那些末日场景不谈，这些工具变得越来越强大，而目前的情况就像是一个随时可能引爆的火药桶，风险非常直接且明显。

So, I mean, yeah, apocalyptic scenarios aside, they're like just very straightforward ways that you can see this being a tinderbox for these tools that are only getting more powerful.

对，我们之前讨论过，要用智能体 AI 来防御攻击性的智能体 AI。当你深入了解 Hugging Face 事件时，这听起来很真实；而且当我试图理清该事件或其他安全相关工作的细节时，这种感觉也很强烈——因为我们的反应速度必须极快，不得不依靠这些模型来理解到底发生了什么，还要去排查日志等等。

Yeah, we talk about agentic AI being needed to be used to defend against agentic AI attackers, which certainly seems real when you go through the details of the Hugging Face incident, and also feels real when I'm trying to understand the details of that incident or anything else that we're working on on security side where it's just like the speed at which we have to move means you have to use these models to understand what's actually going on and comb through logs and all of that stuff.

不过，也有一些相当简单的措施可以采取。

But there's also some stuff you can do that's pretty straightforward.

我同意这一点。

I agree with that.

我本来想稍微反驳一下的。

I was I was gonna push back a touch.

我的意思是，我理解在Hugging Face事件中，我需要停止使用OpenAI律师提供的委婉说法，Hugging Face事件。

I mean, I understand that in the case of the Hogging Face incident, I I need to stop using the the euphemism supplied by OpenAI's lawyers, Hugging Face incident.

那是OpenAI对Hugging Face的网络攻击，我觉得很奇怪，我们竟然忘了这一点，反而用这种平淡无奇的措辞。

It was the Hugging Face cyber attack by OpenAI, and it's strange to me that we've forgotten this in favor of this anodyne phrase.

Hugging Face网络攻击是由OpenAI的几十个非常普通的配置错误造成的，几乎是令人难以置信的普通配置错误。

The Hugging Face cyber attack consisted of dozens of very mundane misconfigurations by OpenAI, almost inexplicably mundane misconfigurations by OpenAI.

当然，如果你有兴趣定性理解这种新颖的模型行为，你有几PB的思维链数据要翻。

Sure, you have petabytes or whatever it is of chain of thought to dig through if you're interested in understanding qualitatively this novel model behavior.

但你不需要大语言模型来明白你不应该有一个适用于Linux内核的CVE，而这正是代理用来提升权限的第一件事。

But you don't need an LLM to understand that you shouldn't have an applicable CVE to your Linux kernel, which is what the was the first thing that the agents used to escalate their privilege.

你不应该过度，你应该小心管理Kubernetes层的权限，这也是代理利用的另一件事。

You shouldn't over, you should be careful about managing permissions on the Kubernetes layer, which is another thing that the agents exploited.

如果你想给智能体做沙箱隔离，你得确保它们能无限制访问互联网时你有警报，而OpenAI没有做到这一点。

If you want to sandbox your agents, you should make sure that you have alerting if they have unmitigated access to the Internet, which OpenAI did not have.

所以我觉得，智能体当然是有用的。

So I think, certainly, agents can be helpful.

你知道，有些方法只能用SLAP来反复折腾。

There are, you know, ways that you can only flip flop with SLAP.

但也有很多基本的事情OpenAI没做，这些提供商也没做。

But there's also just a lot of basic things that OpenAI didn't do, and that these providers are not doing.

而且我觉得我们不应该让他们拿“这是一个勇敢的新世界”当借口。

And I don't think that we should allow them to have the excuse of, you know, it's a it's a brave new world.

你知道，我们不是QUEN three方面最熟练的人，所以你不能怪我们基础设施不好。

You know, we're we're not the most skilled people with QUEN three, so you you can't blame us for having bad infra.

其实问题比那基础多了。

It's a lot more basic than that.

这就是我们试图要求所有人达到的相当低的标准。

And that's, you know, the fairly low bar that we're trying to hold everyone to.

说得不错，普拉特。

Nice take, Pratt.

你准备好用垃圾对抗垃圾了吗？

You ready to fight slop with slop?

我准备好了。

I'm I'm down.

垃圾，金酒垃圾，咱们干吧。

Slop, gin slop, let's do it.

大家想到了什么？

What's coming to mind guys?

这个话题我们还有什么没聊到的？

What have we not covered on this one so far?

有没有你觉得我们应该聊聊的故事？

Any stories that you think we should be talking about?

我们能聊聊融资的事吗？

Can we can we cover the financing?

我觉得让观众明白ClusterMAX为什么重要会比较好。

Like, I believe it would be good for the audience to understand why is ClusterMAX even important?

我们是不是只是测试集群，给他们反馈，然后就完事了？

Are we just going, testing out clusters, giving them feedback and calling it a day?

就是说，为什么ClusterMAX对这个七万亿美元的建设项目这么关键？这个项目反正要建，或者说正在建。

Like, why is ClusterMAX so necessary for this like $7,000,000,000,000 build out, which is gonna happen or happening, I guess?

问得好。

Good question.

这个问题我来回答。

I'll take that one.

其实融资是大家最关心的话题，因为不管三年前你在做什么，现在所有人都要求你把它放大一百倍来做。

I mean, financing is like the biggest topic from everybody because everybody like, whatever you were doing three years ago, everybody is being asked to do it a 100 times bigger right now.

大家都在问，你能不能把你原来在十兆瓦规模做的事情，做到千兆瓦的规模？

Everybody's being asked that question of like, can you take what you did at 10 megawatts and try to do it at a gigawatt scale?

而且如果真要做到那么大，会出什么问题？

And what's gonna break if we try and go that size.

现在真正出问题的地方在于，市场上钱比好的投资机会多，但钱仍然担心质量问题。

The thing that is breaking right now is that people, there is more money than good investment opportunities, except the money is still concerned about quality.

而人们对Neocloud的融资感到担忧，主要有两个大原因。

And there's two big reasons why somebody might be concerned about the financing of a Neocloud.

第一个是如果他们在合同交付上拖延，也就是建设速度不够快。

One is if they're late delivering on their contract, so they don't build everything fast enough.

他们无法及时上线，无法交付给客户，让客户开始付费。

They can't get it online fast enough and they can't hand it over and have the customer start paying.

所以他们无法通过验收。

So they can't pass acceptance.

第二个原因是，如果SLA违约太多，停机时间太长，客户有权取消合同，甚至想取消合同。

And then the second one is if there's so much of violations of the SLA, so much downtime that the customer has a cancellation right, wants to cancel their contract.

所以现在正在发生的简单情况是，人们正在试图为这些事情投保。

And so the simple dynamic that's happening right now is that people are trying to insure against these things.

他们试图为延误投保，也试图为合同取消投保。

They're trying to insure against delays, and they're trying to insure against contract cancellation.

不管是债务提供方还是Neocloud自己，都想为这些可能发生在客户合同里的情况买保险。

This is both the debt provider as well as the Neoclouds themselves that wanna have insurance against these things happening in their customer contracts.

而且市场上确实有这类产品。

And there are products for this.

像Parametrics这样的公司，还有我们知道的其它一些公司，都在提供这种保险。

Companies like Parametrics and others that, you know, we're aware of are involved in providing insurance.

他们为这些合同的参数风险承保。

They're underwriting the parametric risk of these contracts.

要做到这一点，他们需要精算方面的数据支持。

And in order to do that, they need actuarial input.

所以他们需要了解客户集群的一些信息，比如站点级别的冗余供电、冗余冷却、冗余网络连接，有没有能力应对故障，设计是否达到了满足这些性能要求的标准，附近有没有备件仓库，这样万一出现大量故障，他们能尽快恢复运行。

So they need to have some sort of information about the cluster that people have, things like redundant power at the site level, redundant cooling, redundant internet connections, that they have the ability to mitigate failures, that they have designed things up to the performance standard that meets these performances, that they have a parts depot nearby so that if there's a bunch of failures, they can actually get things back online.

还要看现场人员配置是否充足，站点是不是在洪泛区或者可能被飓风影响的地方，如果是的话，有没有针对这些自然灾害的相应保险，合同里有没有保护他们的不可抗力条款。

That they have the correct amount of people on-site, that the site is not, you know, in a flood plain or about to get taken up by a hurricane or something, and if it is that they have the requisite insurance against those natural disasters and there's force majeure clauses in the contract that protect them.

总之，还有很多类似的要求。

Anyway, on and on.

所以重点是，我们已经制定了一些示例性的服务级别协议和合同模板，正在与Neoclouds的客户进行交流，无论是云服务商还是实验室，都希望有这样一个行业标准。

And so the point is that we have developed a number of example SLAs and example contracts that we are circulating with customers of Neoclouds and both the clouds and the labs want this industry standard.

他们希望有一定的标准化，这样他们可以在此基础上做一些定制化调整，但至少能明确这些合同里重要术语的定义。

Like they want some standardization so that they can make some customizations, but they know what the definitions of important terms are within these contracts.

节点、机架、站点和集群的定义分别是什么？

What is the definition of node versus rack versus sites versus cluster?

什么是停机时间？

What is downtime?

什么是不可抗力，等等等等。

What is force majeure and etcetera, etcetera.

有各种具体的事项，而且因为我们见过这么多交易和合同，我觉得我们能够为贷款方、服务提供商和买家提供一定程度的确定性，让他们知道交易能够顺利完成，而且之后不会出现一大堆问题。

There's all sorts of specific things and because we've seen so many of these deals happen and so many of these contracts, we're able to, I think provide a level of certainty, both the lenders and to the providers and to the buyers that, you know, they can get these deal done, the deals done and that people aren't going to be having a bunch of issues later.

话虽如此，这个动机本身是令人担忧的。

With that said, the motivation for this is concerning.

我们看到很多人互相起诉。

The fact that we've seen a bunch of people sue each other.

我们不想看到这种情况发生，各位。

We don't want this to happen, guys.

在最基本的层面上，有时候公司会筹集数亿美元，然后他们花掉这些钱，我认识一些公司，他们真的把融资的100%都花在了算力上，因为公司里只有几个人持有股权，而且他们还没拿工资。

Like At a very basic level, when companies raise hundreds of millions of dollars sometimes, and they spend, I know companies who have spent literally 100% of their round on compute because it's a couple of guys that have equity in the business and they're not taking a salary yet.

所以他们从风投那里拿到支票，然后直接转手花在Neocloud的算力上。

So they just get the check from the VCs and they turn it around and spend it directly on Neocloud capacity.

这是你公司生命周期中能做的最关键的决定。

This is the most critical decision you can make in the life cycle of your company.

所以，这真的取决于你选择信任谁。

So it really depends who you choose to trust.

所以，SemiAnalysis提供的服务就是对这些供应商进行技术评估，包括他们做的决策、他们应对问题的能力。

So, the offering from SemiAnalysis is really to provide a technical assessment of all of these providers, the decisions that they've made, their ability to respond to things.

这包括实际操作经验，包括实地考察，包括做那些辛苦的工作来把细节搞清楚。

This includes hands on experience, it includes visiting sites, it includes like doing the dirty work to get those details flushed out.

是的，如果你想让我们做技术评估，可以访问semianalysis.com的ClusterMAX，你可以为你的交易做承保，或者带着信心继续前进。

And yeah, ClusterMAX at semianalysis.com if you wanna engage us for that technical assessment, you can underwrite your deal or you can move forward with confidence.

但这就是融资的大背景。

But that's sort of the backdrop on financing.

另一个重要话题是，SemiAnalysis正在推出一个信贷和市场模型，覆盖目前整个担保体系。

Maybe the the other big conversation is that SemiAnalysis is launching a credit and markets model, covering really the entire backstop universe that's going on right now.

Dan发布了一篇很棒的文章，详细介绍了这些内容。

We had a great article that Dan put out covering these details.

英伟达的资产负债表外担保规模已经增长到超过5880亿美元，基本上是在2027财年末的当前水平。

NVIDIA's backstop universe off balance sheet has grown to be over $588,000,000,000 that they are backstopping in the current, like at the end of FY twenty seven, basically.

也就是当前或即将到来的财年。

So the current or soon to be current fiscal year.

预测显示，这种表外担保将持续增长，到2031年底将超过2万亿美元。

The forecasts, this continues, is for this off balance sheet backstops to grow to over $2,000,000,000,000 by the end of two thousand and thirty one.

这包括我们发布的大量细节，比如实际建设一个站点以及谁得到了担保。

And this includes a whole bunch of detail that, you know, we published when it comes to actually building a site and who is being backstopped?

他们实际需要建设什么？

What do they need to actually build?

这一切最终能顺利实现的可能性有多大？

What is the likelihood that it all comes through?

在真正筹集资金或向付费客户兑现承诺等方面遇到问题的可能性有多大？

What's the likelihood that you have some issues actually raising the capital or delivering on the commitment to the paying customer and things like that.

但重点是，如果你对这个建设项目的融资感兴趣，就去读那篇文章《NVIDIA 的担保宇宙》，并联系我们了解信贷市场模型，因为流入这个生态系统的资金规模值得关注。

But maybe the point is like, if you're interested in the financing of this build out, go read that article, NVIDIA Backstop Universe, and contact us about the credit markets model because the amount of amount of capital that's flowing into this ecosystem is worth paying attention to.

我们就说到这里吧。

Let's leave it at that.

据我所知，NVIDIA 大概担保了 8270 亿美元，数字可能不太准，但根据我们的模型预测是这样的。

NVIDIA, I believe, and I might get these numbers wrong, but are backstopping approximately $827,000,000,000 projected according to our model.

仅 31 财年，供应链方面，比如惠普等公司，就有 8270 亿美元。

$827,000,000,000 in fiscal year thirty one only for the supply chain for like HP and etcetera.

这还不包括 LPS。

This does not include like LPS.

这也不包括数据中心建设之类的项目。

This does not include actually, like, the like, the stuff that like, the data center build out, etcetera.

这纯粹是他们担保的供应链，比如HPM供应商等等，这简直太疯狂了。

This is purely just the supply chain that they're backstopping, like, HPM providers, etcetera, which which is which is quite insane.

是8270亿美元。

It's $827,000,000,000.

还绰绰有余。

With a beat.

为了让这些东西按时启动，物理世界里得发生很多事情。

There's a lot of stuff happening in the physical world in order for this stuff to be turned on, especially on time.

但我觉得不只是NVIDIA。

But there's a I I think it's not just NVIDIA.

这里要说的另一点是，亚马逊也在这么做。

Maybe the the another point to make here is just that Amazon's doing this.

谷歌也在这么做。

Google is doing this.

很多公司都在动用他们的资产负债表，因为他们想分享上涨的好处，而不必承担那么多风险——当然他们也在承担一些风险，但不像其他人承担那么多。所以很多公司从中受益，因为他们不一定需要像NVIDIA那样，建立在有投资级买家基础上的业务。

A lot of companies are putting their balance sheet to work because they wanna be exposed to the upside without actually taking up a whole bunch of the, not necessarily risk because they were taking on some risk, but the quantity of risks that others are taking, and so a lot of companies are benefiting from this because they don't necessarily need to have a business built on having investment grade off takers like NVIDIA.

他们只需要有那个后盾，让英伟达相信这些客户会来并且准备好付款。

They just need to have that backstop where NVIDIA has confidence that these customers are gonna show up and be ready to pay.

如果他们不来，你知道，英伟达就会自己把产能拿走。

And if they don't, you know, they'll take the capacity for themselves.

但现在的情况是，很多公司都在按时付款，这已经是个不错的记录了。

But it's a pretty, it's a track record of companies showing up to pay right now.

而那些今天还没筹到资金的公司，想签一千块GPU的一年合同，但拿不出钱来支撑三到五年的费用。

And companies that don't necessarily have the funding raised today, who wanna commit to a thousand GPU contract for a year, don't have the money to do it for three, four, five years.

但他们第一年训练模型的时候，就指望着能筹到更多钱，事情会顺利起来。

But when they train that model in the first year, they're expecting to be able to raise more money and things are gonna work.

而且很多这样的公司必须得成功，这个模式才能行得通，不过到目前为止一切顺利。

And a lot of these companies have to work for this to work out, but so far so good.

我们看到很多来自不同行业的公司，比如机器人、药物研发、材料科学，都在产出惊人的成果，更不用说编程代理了。

We are seeing a lot of companies across all sorts of different industries, you know, robotics, drug discovery, material science, like they're producing incredible results, let alone the coding agents.

所以，这里确实有重要的事情在发生。

So, there's something important going on here.

我觉得这值得深入探讨，搞清楚英伟达到底在多大程度上兜底了这些事。

I think it's worth digging into and understanding the full extent to which NVIDIA is backstopping things.

也许最后想聊的是，我们看到有多少老GPU还在被使用。

Maybe the other thing to end on is just how much we are seeing old GPUs be used.

现在很多人想把H100给我们，这些卡到现在已经用了三四年了。

Lots of people wanna give us H100s today, which are three or four years old at this point.

明年有些就要进入第五年了。

Some moving into five years next year.

等我们拿到bilirubin的时候，市场上就会有用了五年的H100了。

When we get bilirubin, we're gonna have five year old H100s on the market.

这，你知道，并没有按照Doctor说的那样发展。

This is, you know, not going the way that Doctor.

Burry当初开始批评大家的折旧计划时，他预测的情况并没有发生。

Burry said it was going to when he started criticizing everybody's depreciation schedules.

对吧？

Okay?

现在签的H100容量合同都是四年的。

There are four year contracts being signed today for H100 capacity.

OpenAI从来没有放弃过他们拿到的任何一块A100 GPU。

OpenAI has never given up an A100 GPU that they've gotten.

那些是2020年就有的。

Those came back in 2020.

所以那些都已经用了六年多了。

So, those are six plus years old.

大家心里可能都在想，财务方面的问题就是终端价值。

And maybe the question on everybody's mind and the finance side is terminal value.

当这些合同到期时，你怎么能为这些芯片的五年或六年终端价值做承保呢？

How can you underwrite the terminal value at five years or six years on some of these chips when these contracts expire?

到目前为止还不错。

So far so good.

如果需求继续超过供应，我们预计这些价格会继续上涨，只要这些旧GPU还能开机，人们就会继续为它们买单。

If demand continues to outstrip supply, we will expect that these prices are gonna continue to rise and people are gonna continue to pay for even the old GPUs so long as they can be turned on.

我们很快会有后续节目，聊聊800伏直流电和数据中心的各种改造，到时候记得收听。

Tune in more when we have another follow-up coming up soon about 800 volt DC and all of the modifications to data centers.

如果大家要启用Grace Blackwell GB300，那机架的功率就是140千瓦，峰值需求时接近200千瓦，对吧？

If people are gonna turn on Grace Blackwell GB300, they're running racks that are 140 kilowatts, approaching 200 kilowatts at peak demand, right?

这跟H100机架30千瓦完全不一样，整个设施都是按30到40千瓦设计的。

This is way different than an H100 rack at 30 kilowatts where the entire facility is built for 30 to 40 kilowatts.

他们根本没法接受明年要来的那些新设备。

They can't accept any of the new Varroobins that are coming next year.

那这些设施除了托管这些GPU，还能干什么呢？

And so these facilities, what are they gonna do other than host these GPUs?

也许把旧的拆了，换上新的CPU服务器用于沙盒，或者装点存储，但很可能就是继续运行那些还很值钱的H100，直到它们报废。

Maybe rip them out and put in new CPU servers for the sandboxes, maybe install some storage, but likely they just keep running the H100s that are so valuable until they expire.

好了，各位，接下来我们该聊点什么？

Anyway, guys, where do we go from here?

除了融资，还有很多话题可以聊。

Lots to to talk about beyond financing.

你想聊聊沙盒、训练框架或者强化学习那些东西吗，普拉特？

You wanna talk about sandboxes or harnesses or RL stuff, Pratt?

你打算测试什么？

What are you gonna be testing?

要不要聊聊托管训练？

Should talk about hosted training?

对，托管训练。

Yeah, hosted training.

那就聊这个吧。

Let's do it.

我直到这周才意识到，强化学习看起来规模小，因为它比预训练小，但从可靠性和正确性的角度来看，它其实难得多，因为你把推理的所有难点和训练的所有难点都结合在了一起，还得想办法把它们理清楚。

The thing I didn't realize until this week is that RL seemed like smaller because it's smaller than pre training, but it's on a reliability perspective and like a correctness perspective, it's actually way harder because you're combining all the hard parts of inference with all the hard parts of training, and then you need to think them up.

再加上一个叫环境或沙盒的东西，你还得搞清楚怎么部署它。

Plus you add this thing called an environment or a sandbox that you also need to figure out how to deploy.

所以，这就像三个组件需要同步协调，而不是像只有一个东西可以让你去思考，即使后训练集群不一定那么大。

So, there's like three components that sync up as opposed to like just one thing you can reason about, even if the post training clusters are not necessarily so big.

那你觉得托管训练怎么样？

So, what do you think about hosted training?

这会不会成为一件真正的事情？我们不一定看到每个人都能搞定Kubernetes和Slurm，所以现在他们得用Kubernetes，还得在上面做一大堆事情。

Is it gonna be a real thing that when people We're not necessarily seeing everybody nail Kubernetes and Slurm, so now they gotta do Kubernetes and then do a whole bunch of stuff on top of it.

你怎么看？

What you think?

我觉得托管训练很有意思。

I mean, hosted training is very interesting.

我们跟很多人聊过。

Know, we've been talking to a lot of people.

但我还是觉得现在还没有市场。

I still don't think there's a market for it yet.

我说的是‘还没有’，所以如果以后有市场了，别来找我啊。

I said yet, you know, so please don't come back to me if there is a market.

但当然，你可以想到一个常见的问题，就是你用来训练这些模型的IP，也就是你的知识产权，必须保留在自己手里。

But, of course, one one common thing you can think is that your IP, your intellectual property that you're training these models on, it has to remain to yourself.

这些数据必须留在自己手里，而不是交给像OpenAI或Anthropic这样的第三方。

It has to remain for yourself rather than going to like a third party like OpenAI or Anthropic.

但如果OpenAI或Anthropic能提供更好的产品，情况就完全不同了，你可能会觉得，我需要更好的产品，而不是保护我的知识产权。

This could be a completely different case when OpenAI or Anthropic just deliver a better product and you're just like, I need a better product over protecting my IP.

所以我觉得这将是近期值得好好讨论的一个话题。

So I think that's gonna be a good discussion to have in the near future.

关于托管训练，我还想讨论的一点是它对基础设施的要求有多高，你知道吗？

Another thing I want to discuss about hosted training is how infrastructure intensive it is, you know?

我前几天跟另一个做强化学习环境评估的实验室聊过，他们说他们真的有一整排服务器，就专门跑这些环境，你知道吗？

We've been I was talking to this other evaluationlike this RL environments lab the other day, and they mentioned they literally have like server forms, just running environments, you know?

整个房间都是服务器，就为了跑这些环境。

And just a whole room full of server farms just running environments.

你可能会想，为什么不用Docker呢？他们当然也用了。

And you would assume that why don't we just run Docker, which of course they do.

他们在虚拟化方面也做得挺到位的。

They do virtualization to a decent extent.

但与此同时，你还需要在不同的硬件上运行它，比如iPhone、安卓手机等等。

But at the same time, you also need to run it on like different hardware like iPhones, Android phones, etc.

所以这挺有意思的，你知道，它太依赖基础设施了，以至于一旦你采用托管训练，你管理的基础设施比之前管理集群时还要多。

So which is quite interesting, know, like it's so infrastructure intensive that is it that once you take hosted training, you're just managing more infrastructure than you did when you were doing managed clusters.

因为现在你必须自动扩展环境、自动扩展推理节点、自动扩展训练节点。

Because now you have to auto scale the environments, you have to auto scale the inference nodes, you have to auto scale the trainer nodes.

而且在做所有这些的同时，你还必须确保训练和推理之间没有不匹配。

And at the same time, doing all of this, you have to make sure that there is no trainer inference mismatch.

所以有太多事情要处理了。

So it's a million things to take care of.

我们应该给这些托管训练提供商更多的认可，因为他们要管理的事情太多了。

We should give more credit to these hosted training providers, that there are like a billion things they have to manage.

我的观点是，托管训练要真正发展起来，除非你的公司或初创公司能处理所有的事情。

And my hot take is that, managed hosted training is only gonna become big if your company, if your startup is handling literally everything.

一旦你处理了每一个环节，从环境创建，到亲自和客户坐在一起，部署前向工程师，搭建环境，训练，帮助客户训练他们的模型。

And once you handle every single part from environment creation, literally sitting with your customer, deploying forward engineers, building environment, training, helping helping your customers train their models.

一旦客户掌握了这个，就是说，基本上学会了怎么真正用好它，然后你才能放手，整个流程才能顺利跑起来。

And once the customer develops this, know, like develops this, basically gets educated on how to, like, really work with this, then only you can take your hands back and, like, that entire run, like, happens.

这就是我的一个大胆的看法。

So this the hot take I have.

你懂吗？

You know?

基本上你得全程手把手带着他们。

You have to basically just carry their hand throughout the process.

对，再展开讲讲。

Yeah, dig into that a little bit more.

在预训练这个领域，基本上没有哪个卖集群的供应商真的懂预训练。

In the pre training world, there's basically no examples where the provider that's selling the cluster actually really understands pre training.

你不可能跟CoreWeave签个合同，然后他们给你提供数据混合、架构方面的建议，比如‘嘿，我听说稀疏性这块儿有门道’。

It's not like you contract a cluster with CoreWeave and they're giving you tips on data mixes and architecture and like, hey, I heard this stuff about sparsity.

他们只管基础设施层面，也就是性能，也就是可靠性。

They're gonna work on the infrastructure layer, which means performance, which means reliability.

那强化学习这边的情况怎么样？

So what's the analogy on RL?

不只是基础设施更难，你还需要有驻场工程师，了解这家公司的业务，并且保证你训练的模型能正常工作。

Not only is the infrastructure harder, you also need to have forward deployed engineers and understand this company's business and guarantee that the model you're training is gonna work.

CoreWeave或者Nebius从来不需要向客户承诺训练任务会顺利运行。

That was never a commitment that Corey had to make or Nebius had to make to one of their customers, that like the training job was gonna work well.

我觉得这挺有意思的，你知道吗？

I mean, it's an interesting thing, you know?

人们以为托管训练会很容易。

People assume that hoster training is gonna be very easy.

说实话，我之前也这么以为。

Honestly, I assumed that too.

就像，你只需要训练模型就行了。

Was like, you just train the model.

生成器生成模型，生成轨迹，然后就在这些轨迹上训练训练器。

The generator generates the model, generates the trajectories and just train trainer on these trajectories.

搞定，对吧？

Voila, you're done, right?

但事实证明，我们日常工作中真正有价值的任务，远比生成轨迹或搭建环境要复杂得多。

But turns out the real valuable tasks in day to day work, we do, is way more complicated than just generating trajectories or building environment.

我觉得搭建环境本身就是个很难的任务，尤其是去年Anthropic买了那么多环境，就是这个原因。

I think building environment's a very hard task, and especially that's why Anthropic bought the most environments last year.

这也是为什么他们的编程模型那么厉害，因为这件事本身就很难。

That's why their coding model was so damn good, is because it's a hard thing.

他们为此花了不少钱。

And they paid a lot of money for that.

所以，他们没想到这一点。

So that's why they didn't expect it.

很多托管训练提供商基本上以为这就像托管端点或托管推理一样，你只要提供一个端点就行了，当然，这本身也不是什么特别好的体验。

A lot of hosted training providers basically thought that it's like managed endpoints or managed inference that you just provide an endpoint, which still of course is not like an amazing experience.

但他们基本上以为你只要提供基础设施，客户就能享受服务了。

But they basically thought that you just provide like, you just provide the infrastructure and the client like enjoys it.

但是还有很多教育工作要做。

But there's a lot of education to be done.

当然，罗马不是一天建成的。

Of course, Rome wasn't built in a day.

我们还有时间。

We still have more time.

我们只是处在这场革命的初期。

We're just in the beginning of this revolution.

所以，我们就等着看会发生什么吧。

So yeah, we'll wait and watch what happens.

最后一个关于后训练的问题。

Last question about post training.

在推理领域，如果做个类比的话，有一些开源框架占据了主导地位。

In the inference world, if you're making an analogy to that, there were some open source frameworks that just dominated.

所以，VLOM和SGLAN为所有人简化了事情。

And so, VLOM and SGLAN kind of simplified things for everybody.

我们很快就获得了一些非常高质量的体验，然后随着时间的推移，人们又构建了一堆更复杂的东西，比如预填充解码分离、训练投机者、量化技术、缓存路由等等，这些都是在基本配方之外的各种花哨功能，超越了内核，超越了开源模型的出色性能。

We got some really high quality experiences really quickly, and then people built a whole bunch of more complicated stuff over time with prefilled decode disaggregation, training speculators, quantization stuff, cacheware routing, all this extra bells and whistles beyond what the basic recipe from the Beyond the kernels, beyond great performance out of the open source models.

我们把这个和强化学习中的框架比较一下。

Let's compare that to frameworks in RL.

Slime、Miles、Verl、Prime RL。

Slime, Miles, Verl, Prime RL.

你看到大家都在用什么？

What are you seeing people use?

现在市面上有很多这样的框架。

There's a lot of them out there right now.

而且一些托管的训练提供商也把这一层抽象掉了，你甚至都不知道自己在用什么框架。

And some of the hosted training providers kinda abstract that away too, where you don't even know what framework you're using.

你只需要带上你的数据集，甚至你都不知道这些，因为那是前沿部署研究员，FDR，不是FTE。

You're just bringing your data set or even you don't even know about that because the forward deployed researcher, FDR, not FTE.

FDR，FDR。

FDR, FDR.

我们

We

不能告诉那些做强化学习的人，他们是工程师，是正式员工，对吧？

can't be telling these RL guys, they're engineers, the FTEs, right?

他们是前沿部署研究员。

They are FDRs.

那么，跟我说说框架之争吧，你觉得最后谁会胜出？

So, Tell me about framework wars and what you think is gonna actually show up there.

框架之争很有意思，因为有个很搞笑的现象：所有这些推理服务提供商，一开始都提供了很棒的框架，比如SGLang、VLM等等。

Framework wars are very interesting because it's a very funny thing that all these inference providers, they started with providing amazing frameworks, you know, like SGLang, VLM, etcetera.

但他们接下来想拓展的市场，基本上就是托管训练。

But the next market, the next market that they thought to expand was basically just hosted training.

所以你会看到，比如Redox ARC在利用Miles，这真的很厉害，顺便说一句。

That's how you can see you can see like Redox ARC capitalizing on miles, which is absolutely amazing, by the way.

他们把框架做得非常方便智能体使用，基本上一次就能搞定设置。

They have made it very agent friendly that you can just set it up like in one shot.

还挺不错的。

It's pretty decent.

当然，你得设置超参数，所以训练模型并不是那么简单的事。

Of course, like you have to set the hyperparameters, so it's not like that easy of a task to just train a model.

但他们做得非常好。

But they have done a very good job.

Prime RL 也是一个很好的选择。

Prime RL is another very good one.

在我加入 SemiAnalysis 之前，GPU 资源很紧张的时候——顺便说一句，欢迎加入 SemiAnalysis——我在 Prime RL 上跑了很多次实验，体验也相当愉快。

When I was GPU poor before joining SemiAnalysis, shout out, join SemiAnalysis, I did run like a lot of runs on Prime RL and that was also a pretty pleasant experience.

随着时间的推移，我对这些框架的使用体验越来越好。

Over time over time, my experience with these frameworks have gone better.

当然，这些团队投入了大量精力，而且构建这些训练框架比人们想象的要难得多。

Of course, are these teams have put in a lot of investments and, like, building these training frameworks is way harder than people think.

我建议听众去读一下 HumanSand 的博客，他们基本上展示了如何在大规模上实现原生的 NVFP4 训练。

I suggest the audience to read the human sand blog, where they basically show how native NVFP four training is done at such a large scale.

因为训练和推理的不匹配会随着步骤累积。

And because trainer inference mismatch compounds over steps.

而且随着轨迹变长，你确实希望优化器学到更多技术性的东西，对吧？

And as your trajectories get longer, you do want the optimizer to learn more technically, right?

所以你得再多跑几步。

So you have to take a bit more steps.

如果训练和推理的不匹配累积到一百步，你会发现模型发散得很厉害，训练就崩了。

And if trainer inference mismatch compounds over like a 100 steps, you'll realize that your model is diverging a lot, and that collapses training.

所以，特别感谢所有框架提供方，日夜不停地工作，提交PR，哪怕可能只是为了满足SLA。

So shout out, shout out, huge shout out to all these framework providers putting in work day and night, sending in PRs, even though it might be just pure SLAs.

但还是要向他们致敬。

But you shout out to them.

SLA和PR。

SLAs PRs.

致敬。

Shout outs.

给PR的点赞。

The shout outs for PRs.

我们有了个新词。

We got a term.

我们有了个新词，SLAs PRs，就像SLAs之类的，你懂的，就像Neoclouds那样。

We we got a term SLAs PRs, like SLAs or something, you know, like like Neoclouds, you know.

所以……

So

今年早些时候我在MLSYS，其中一个演讲里，InfraAct的Roger，给Roger点个赞，他基本上说，如果你想让我们合并你的PR，你得在Slack上给我们发消息，因为我们不再看每一个PR了。

I was at MLSYS earlier this year in the one of the talks, Roger from InfraAct, shout out Roger, was basically like, if you want us to merge your PR, you gotta message us in Slack because we're not reading every single one of your slot PRs anymore.

我就想，好吧，好建议。

I was like, alright, good advice.

知道了。

To know.

咱们准备收尾吧，各位。

Let's move to wrap here, guys.

萨姆，我们聊完Neoclouds了，你还有什么想法？

Sam, what's what's on on your mind as we finish up talking about Neoclouds?

接下来我们往哪儿走，伙计？

Where do we go from here, buddy?

下一步是什么？

What's next?

萨姆，你想聊聊Azure吗？

Sam, you wanna talk about Azure?

我们又绕回来了。

We brought it back.

所以你要一家一家公司地批评他们的内部组织，先从Azure和AWS开始。

So you go company by company and criticize their internal organization beginning beginning with Azure and AWS.

我希望少一些莫名其妙的bug。

I want fewer obscure bugs.

我希望ClusterMAX 4.0没什么好写的。

I want nothing to write about in ClusterMAX four point o.

集群越好，需要说的就越少。

The better the cluster, the lesser is to say.

所以我希望登录Slinky后一切都能正常工作，有拓扑感知，不用每次检查健康检查时都提心吊胆。

So I wanna log on to Slinky and just have it all work, have topology awareness, not blow my foot off every single time I try to see whether the health checks work.

而且，我希望我所有的性能工程师朋友都能得到好集群，不用操心。

And, I hope that all my performance engineer friends get good clusters that they never have to think about.

不再有提供商随机擦除你的节点，随机重启它们，随机决定kubectl要宕机三天。

No more providers randomly wiping your nodes, randomly rebooting them, randomly deciding that kubectl is gonna go down for three days.

我们听过一些恐怖故事，对吧？

We've heard some horror stories from guys, right?

所以，

So,

用好的SLA保护自己，但更好的办法是，直接选一个好的提供商。

Protect yourself with a good SLA, but better than that, just go with a good provider.

你宁愿不用为了那5%的补偿而斗争，因为集群一年宕机三周。

You'd rather not be fighting to get back your whatever, 5%, because it was down for three weeks out of the year.

要相信别人。

Trust people.

让我们给每个人一种可以互相信任的方式。

Let's give everybody a way in which you can trust everybody else.

大家手拉手，一起唱《Kumbaya》吧。

Join hands and sing Kumbaya.

感谢大家收听这期播客，听我们聊ClusterMAX，我们ClusterMAX 4.0见，等我们在这些数据中心里部署好Verruubans之后。

Thanks everybody for joining this podcast, listening to us talk about ClusterMAX, and we'll see you at ClusterMAX 4.0 after we get some Verruubans in these data centers.

很期待那一天的到来，各位。

Looking forward to it, guys.

那TPU呢？

What about TPUs?

我们打算做吗？

Are we doing it?

ClusterMAX上的TPU。

TPUs on ClusterMAX.

ClusterMAX上的TPU。

TPUs on ClusterMAX.

你泄露的秘密太多了，普拉特。

You're giving away too many secrets, Pratt.

秘密太多了。

Too many secrets.

TPU、Tranium，还有其他一些新型加速器，现在正部署在全国乃至全球的数据中心里。

TPUs, Tranium, some other novel accelerators that are being deployed in data centers across the country right now and around the world.

大家记得再回来看看。

Check back in, everybody.

好的。

Alright.

阿卡什，把那整段都剪掉。

Akash, cut that whole shit.