为何AI基础设施必须为代理体验而演进——Modal首席技术官Akshat Bubna

更好还是

Better or

一样。

the same.

没错。

Exactly.

对吧？

Right?

是的。

Yeah.

所以我们一直在研究 DeFlash，这是一种基于块的预测器。

And so we've been working a bunch on DeFlash, which is a block based speculator.

它不是一次预测一个词元，而是预测一个块。

So it's instead of predicting one token at a time, it's predicting a block.

我们已经把相关成果开源了。

And we've been open sourcing our work with it.

我们接下来要做的，是帮助大家训练推测器和自定义模型。

The the next thing for us here is for for helping people train speculators and custom models.

这在过去通常非常依赖人力，得靠支持工程师亲自上门服务。

It's it's something that traditionally is very FTE driven, support deployed engineer driven.

就是跟客户一起合作，帮他们完成这些工作。

Like, you work with customers and help them do that.

我们推出 Auto Endpoints 的初衷，就是想让前沿级别的性能人人都能用上。

And our vision for this is why we launched Auto Endpoints is we want to make frontier level performance available to everyone.

这一点我们在发布时也提到过。

And so we've mentioned this in the announcement.

当时只是稍微提了一下。

We kind of teased it.

接下来我们要重新推出的功能是：当你运行 Auto Endpoint 时，我们会悄悄复制流量并

The next thing we're relaunching is, basically, as you run an Auto Endpoint, we shadow traffic and

你能不能解释一下什么是 Auto Endpoints？

Do do you want to explain what Auto Endpoints are?

是的

Yeah.

真不错

How lovely.

没错

Yeah.

对

Yeah.

所以，这其实又回到你提到的模态了——你确实可以修改代码，但有些人并不想碰代码，他们只想直接用一个性能出色、可扩展性强的端点马上开始工作。

So this is, I guess, going back to your modal is is you touch the code, but sometimes people actually don't wanna touch the code, and they wanna get started with an endpoint that works and has all the great performance and scalability that Modal has.

所以我们通过界面和命令行工具，让你能轻松创建一个端点，里面已经内置了我们之前说的所有优化功能，比如DeFlash之类的，而且全程透明。

So we've made that easier with, basically, a way to create an endpoint from our UI, from the CLI that has all of our optimizations that we talked about, like the DeFlash stuff already baked in, and there's full transparency.

我们会把代码直接提供给你。

So we give you the code.

你可以自己运行看看。

You can go run it yourself.

如果你需要，也可以切换到完整的 Modal 体验，我们发现用户越熟练，就越会这么做。

And if you want, you can eject out into the full Modal experience, which we see as people get sophisticated.

他们确实希望调整模型。

They they do wanna tweak the models.

他们想进行精细调优。

They wanna fine tune stuff.

这些功能你依然都能用。

You you can still do all of that.

这可不是个黑箱。

It's it's not a black box.

另外，就像我们后面提到的，接下来我们要做的是：让你的草稿模型随着数据分布的变化自动进化，完全不需要人工干预。

And, yeah, the next thing, as we teased later in the post, is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again, without having to talk to a person.

而且

And

对。

yeah.

我想直接了解一下具体情况。

I guess just to kind of understand it directly.

我的意思是，你们肯定有 GPU 资源。

I mean, you know, obviously, you you have the GPUs.

还有兼容的端点服务。

You have Endpoint that's compatible.

你们还提供 OpenModel 的部署服务。

You serve OpenModel.

如果用户自己来做这件事，你们提供的核心价值差异在哪里？

If someone was to do this themselves, what's the delta that you guys provide?

你们在高效推理方面做了很多优秀的开源工作。

So you do a lot of open source great work on effective inference.

那跟市面上现有的方案比怎么样？比如我拿同一个模型，GLM 5.2 FP8，配上现成的推理引擎，像 vLLM、SGLang 这些，再配上同等算力和成本的基础设施。

How does it compare to say, I take the same model, GLM 5.2 FP eight, take off the shelf inference engine, vLLM, SGLang, you know, get compute of similar capacity, similar cost.

在这种情况下，接入你们的服务，除了规模扩展方面的优势，还能带来哪些具体的差异化价值？

What's the kind of delta that plugging into something this like this offers outside of the benefit of, you know, scaling?

这挺有意思的，因为我们一直坚持把我们的贡献开源并提交到上游。

It's kind of interesting because we we've taken the approach of open sourcing our contributions and upstreaming them.

我们和SGLang团队合作得很紧密。

We work closely with the SGLang team.

我们真心希望，我们团队做出的改进能被纳入开源项目，让哪怕不用Modal的人也能用上。

We actually want the improvements that our team comes up with to be there in open source for others to use, even outside of Modal.

对我们来说，好处是我们有一支经验丰富的团队，如果你遇到某些功能还没实现，他们能帮你第一时间达到最佳性能。

The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance first.

另外，这些端点的弹性远超其他任何方案，就像你提到的那样。

The other thing is, with these endpoints, we are way more elastic, as you said, than anyone else.

你真正能做到零扩展，也能真正应对突发流量。

And you have true scaling to zero, You have true burstiness.

实际上，对用户来说，这比单纯找块GPU跑模型代码重要得多。

And in practice, that matters a lot more to people than just finding the GPU and running model code on

没错。

the Yeah.

说实话，这事儿没我刚才说的那么简单，道理是这么个道理，做起来可没那么容易。

And I I will say it's actually not that straightforward to just like, what I said is easier said than done.

对吧？

Right?

是的。

Yeah.

对普通人来说，要凭直觉判断该怎么选，依然挺难的，因为可选的组合实在太多了。

It's, I think, still for the average person, still hard to just gut check using different there's there's quite a bit of combinations you can make there.

这些选择背后的权衡，光看表面是看不出来的。

The trade offs aren't really known at face value.

对。

Yeah.

这还不只是个简单的问题。

I mean, it's it's not just that.

我觉得，真正难的是把生产级别的推理系统跑起来。

I think it's it's that running production grade inference is a hard infer problem.

就算把自动扩缩容那些事儿排除掉，比如控制尾部延迟、确保每个请求至少被投递一次等等。

Even if you subtract out the auto scaling is controlling things like tail latency and making sure every request is delivered at least once and whatnot.

这里头

There's a

还有很多创新的空间可以做。

lot of innovation that you can do here.

我觉得挺有意思的，你开始涉足别人的领地了。当你成为一个完整的云平台时，你其实是在侵入其他厂商的地盘。

Think it's very interesting that you're starting to encroach on, like, as you become a full cloud, you're starting to encroach on other people's turf.

对。

Mhmm.

那你们不会做哪些事呢？

What will you not do?

嗯，我们想跟着用户的需求走，确保他们能拿到一个各方面配合得很好的平台。

Well, we wanna follow our users and make sure they they get, like, a platform that has everything that works well together.

所以目前我们主要聚焦在模型的生命周期和智能体的生命周期上。

So right now, we're kind of focused on the model life cycle and the agent life cycle.

所以无论是从数据准备到训练再到推理，还是说我想部署一个后台代理，比如沙箱环境、超持久存储，还有其他一大堆东西。

So both, like, going from data prep to training to inference, and then also, if I wanna deploy a background agent, let's say, you know, sandbox, super persistent storage, a whole bunch of other stuff.

我们跟做了OpenInspect的Cole聊过。

We talked to Cole, who did OpenInspect.

是的。

Yeah.

显然，RealInspect也是跑在Modal上的。

And obviously, RealInspect also is on Modal.

对。

Yeah.

RampInspect就是一个很好的后台代理案例，特别成功，因为他们用了快照和快速扩缩容这些功能，让整个系统响应特别灵敏，运行得也很流畅。

So RampInspect was was a great example of a background agent that was really successful because they were able to use some of parameters like snapshotting and fast scaling to just have something that feels really reactive and and works well.

是的。

Yeah.

那位就是RAMP新上任的CTO。

That's the new CTO of RAMP right there.

是的

Yeah.

拉胡尔

Rahul.

真的特别有意思

It was really, really fun.

对

Yeah.

我的意思是，好吧

I mean, okay.

我觉得大家都非常看好

You know, I I think all very bullish.

我回想一下，当初刚认识你们的时候，确实没怎么关注GPU领域

Like, know, one of my reflections was also I did not originally so, obviously, when I met you guys Mhmm.

但现在你们全身心都投入到推理上了

You weren't that much in the GPU game, and now you're all about inference.

今年GTC大会上，詹森演讲中让我特别关注的一点，就是我们所说的‘推理拐点’。

And one of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling, like, the inference inflection.

对吧？

Right?

以前在AI或机器学习任务里，GPU和CPU的配比大概是8比1，现在却更接近1比1了，这挺有意思的。因为现在智能体经常被阻塞，或者频繁调用CPU密集型任务，导致性能瓶颈在GPU和CPU之间来回切换，不像以前那样基本全靠GPU，偶尔才用一下CPU。

That let's say in AI workloads or machine learning workloads, it used to be, like, let's call it eight to one GPU to CPU, and now it's more like one to one, which is like a interesting like, because of how much agents basically are blocked or call out to to CPU heavy stuff, the actual, like, limiting factor, like, swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU.

没错。

Yep.

GPU，CPU。

GPU, CPU.

现在是不停地在两者之间切换。

And now it's like just constantly.

所以你必须把它们放在一起部署。

And you just have to co locate everything.

是的。

Yeah.

这正是我们觉得 Modal 吸引人的一点，我们构建了一个覆盖17家云服务商的算力池。

And that's one of the things that actually, again, we see as something appealing about Modal, which is we've built this capacity pool that spans 17 cloud providers.

所以我们非常擅长在各种不同的云资源上运行任务，

So we're very good at running on various kinds of cloud capacity across

全球各地都可以。

the world.

我们自己没有数据中心。

You don't have your own data centers.

我们确实没有自己的数据中心。

We don't have our own data centers.

我们只是在很多新兴云平台上运行，是的。

We we just run across a lot of neo clouds and Yeah.

你们是硬件提供商吗？

Are you metal providers.

对。

Yeah.

问号？

Question mark.

是的。

Yeah.

对。

Yeah.

你在算账的时候，就会想，到底在哪个临界点上，你会觉得？

You're you're running the math, and you're like, what what's the cut over point where you're like?

是的，这是个好问题。

Yeah, that's a good question.

一方面，我们的优势在于软件层，轻资产、专注软件让我们能快速推进。

I mean, part of it is we we see our differentiator in in the software layer, and being capital light and focusing on the software helps us move really fast.

到目前为止效果不错，因为有很多人在建数据中心，我们可以和他们高效合作，同时专注于我们独特的地方。

So far, it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again, focus on what makes us special.

十七家能让你接触到一些本地服务商。

Seventeen gets you into like the local providers sometimes.

那最有趣的是哪一个呢？

Like, what was the most interesting one?

实际上，Neo Cloud的数量比你想象的多得多，而且它们的可靠性水平各不相同。

There are actually a lot more Neo Clouds than you expect, and they all have various degrees of various levels of reliability.

所以我们花了很多时间去打造自己的可靠性层，就是为了解决这个问题。

And that's why it's something we've invested a lot of time in, is actually building our own reliability layer on top.

哪怕GPU从总线上掉下来，或者出现其他问题，我们的用户任务也不会受影响。

So if the GPU falls off the bus or something happens, we user workloads are not affected.

这样一来，我们就能使用比普通用户所能用的多得多的资源。

And that that actually lets us use a lot more capacity than Yeah.

作为用户，你根本用不了这么多。

You know, you as a user would be able to.

这确实很有用，因为现在大家都清楚你处在哪一层，你也因此专注于成为最强大的云平台。

It's a useful thing to have because I now everyone knows, like, what layer you are and, like, you you sort of optimize for being the super cloud of Yeah.

所有

All

没错，就是这样。

That's that's the idea.

所以当你提到同地部署时，这其实是个挺有意思的现象——我们发现，有人来找我们，就是需要特定位置的CPU或GPU，比如他们明确要求...

And so I guess when you mentioned collocation, that's that's another interesting thing where one thing we've seen is people come to us when they want very specifically located CPUs or GPUs like they want.

哦，他们是想把计算节点固定在欧洲吗？

Oh, they pin it in, like, EU?

对，就是这样。

Exactly.

或者欧洲，欧洲？是出于数据就近处理的需求，还是性能考虑？

Or EU, EU Like, data retraining the locality thing or performance or what?

要么是数据靠近，要么是延迟低。

It's either data locality or latency.

他们会在Modal里运行沙箱环境。

Want your they're running sandboxes in Modal.

而且希望这些沙箱就部署在...

They want them to be right next to

你的没错。

your Yeah.

这些方面都很重要。

That is important in all those things.

你无意中，我不知道算不算巧合，但确实打造了一个非常适合智能体表达自己的基础工具。

So you've kind of accidentally I don't know if it's accident, but, like, you've built the perfect primitive for agents to to express themselves in it.

说来还挺有意思，每次新增功能都离不开更多的文件系统、更多的CPU。

You know, like, it's almost very funny how every extra development just involves more file system, just involves more CPU Yeah.

就是你已经有的那些东西。

Just like the things that you already have.

我对网络方面的应用了解不多，不过你们在Networking上也做了不少出色的工作。

I don't know much about if there's any, like, networking usages that are interesting, but you've also done some good work on networking.

是的。

Yeah.

没错，就是这样。

I mean, that's exactly right.

我们其实就是基于计算、存储和网络这些基础层，搭建人们真正需要的东西。

Like, we we're sort of just taking compute storage and networking and building stuff on that layer for, again, the stuff people need.

对。

Yeah.

我们看到一些有趣的网络应用正在出现。

We we see a few interesting networking things coming up.

比如，有人真正想要的是联网的沙箱环境。

One is people actually want networked sandboxes.

我们有一个

We have a

这是为了

It's for

类似于Docker集群那样的东西吗？

like a Docker cluster type thing?

不好意思。

Sorry.

对。

Yeah.

Docker Swarm。

Docker Swarm.

那这个叫什么名字？

What is it called?

Compose。

Compose.

类似Compose的东西。

Compose type thing.

是的。

Yeah.

所以如果你需要Docker Compose，我们的沙箱现在支持一种叫边车的功能。

So actually, if you want Docker Compose, our sandboxes now support this thing called sidecars.

一个沙箱实际上是一个容器组，你可以在里面同时运行多个容器。

So you can a sandbox is actually a pod of containers, and you can run multiple containers in the sandbox.

这一点也很有用，因为说到网络，用户希望对沙箱的出站网络有更多控制权。

Also useful because, going back to networking, people want a lot of control over outbound networking from a sandbox.

比如，他们可能想部署中间人代理，用于记录日志、控制对特定域名的外联行为，或者注入凭证。

Like, they might wanna run man in the middle proxy for, like, maybe logging stuff for RL or controlling how egress can happen to a domain, injecting credentials.

所以我们不得不自己开发了大量这类功能。

And, yeah, so we've kind of had to build a lot of that stuff ourselves.

对。

Yeah.

但有时候，用户其实希望沙箱能跨多个节点相互通信，这是我们最近看到的一个新趋势。

But then also sometimes people actually want sandboxes spanning multiple nodes to talk to each other, which is an emerging thing we're seeing.

我们之所以支持这个功能，最初是出于别的原因。

We have support for that for for a different reason.

至于它会不会成为主流，我们还得再看看。

And yeah, we'll see if that becomes the

就像只是

Like just

一个开放的套接字。

an open socket.

这跟MTLS直接相关。

This is directly like MTLS.

我们确实支持这个功能，你可以在沙箱内部建立一个隧道。

We do support that, which is you can expose a tunnel inside the sandbox.

对。

Yeah.

然后你可以把这个隧道暴露给公网，也可以在上面加一层HTTP代理。

And then you can either expose it to public Internet or it can be and you can add like a HTTP Odds layer above it.

但我们还有一个叫IPv6的覆盖网络功能，之前没提到过。

But we have this thing called I six p n, which we haven't talked about, which is this like overlay network using IPv6 addresses.

如果启用了这个功能，同一个工作区里的Modal容器就能用私有的IPv6地址互相访问，其他人完全访问不到。

So if modal containers within the same workspace, when this enabled, can actually address each other using this private IPv6 address, and no one else can.

是的。

Mhmm.

这相当于为容器提供了一种私有网络。

This is like sort of private networking for containers.

我们其实是为自己的争议性训练产品才开发了这个功能，作为基础组件。

We actually built it because we needed it as primitive for our disputed training product.

我们还有另一个功能：你可以在函数上加一个装饰器，就能获得一组 GPU，它们之间通过 RDMA 网络通信。

So we have this other feature, which is you can add a decorator to a function, and you get a cluster of GPUs, and they have RDMA networking.

你可以运行真正无服务器的分布式训练任务，而这正需要这个覆盖网络。

So you can run a distributed training job that's truly serverless, and we need the overlay network for that.

但后来我们发现，人们还在用它做别的事情，我也挺好奇，到底还会有人拿它来干什么。

But then we've seen that people are using it for other reasons, and I'm kinda intrigued to see, yeah, what would people do with it.

先搭好基础组件，让用户自己去探索。

Build primitives and let people figure it out.

对吧？

Right?

是的。

Yeah.

没错。

Exactly.

你这么说挺有意思的。

You put that up pretty interesting.

他们好像真的读过我们的文档。

They're like they read the docs we think.

让我先试试这个。

Let me use that first.

一些你根本没打算

Something that you never intended to

连文档里都没提过的东西。

is literally not even in our docs page.

人们不知怎么就发现了，还真的在用。

People somehow found it, and they're using it.

你刚才用RDMA和TCP做对比，讲得特别清楚，尤其是大规模强化学习场景下传输速度的差异，这功能本来就是内置的。

I mean, the way you portrayed it with, like, RDMA versus TCP, like, very well laid out, but just the transfer speed change at scale for RL, yeah, you have it built in.

我敢打赌，肯定有人在你正式推出这个功能之前，就已经发现它了，而且觉得它效率高得多。

I'm sure someone found it, found it to be a lot more efficient before you actually made a thing out of it.

对吧？

Right?

是的。

Yeah.

而且我不是故意挑刺，但其实这个叠加网络本质上就是基于 TCP 的叠加网络。

And and not to split hairs, I I guess the the overlay network actually is the TCP overlay network.

我们之所以用它，是因为在建立 RDMA 网络之前，必须先通过它完成密钥交换。

The reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that.

但后来有人发现了这个 TCP 层，直接拿来用了。

But then people found the the TCP part.

我能跟你说吗？这对我而言真是个重要时刻。

Can I tell you this is like a big moment for me?

因为确实如此。

Because Yeah.

我一共审了2200份提交材料，都是为世界博览会准备的。

So I, you know, I review 2,200 submissions for for the World's Fair.

没错。

Yep.

然后我收到了约翰·奥斯特哈尔特发来的东西，你认识他吗？我不确定你知不知道这个人。

And then I got this from John Osterhalt, who I don't know if do you know John Osterhalt?

他

The

这个名字听起来有点耳熟。

My name sounds familiar.

他是一位知名教授，写过很多关于软件设计的书，这份投稿就是他选中的演讲内容。

He published a he's a well known professor, published a lot of interesting software design books, and this is the talk he chose to submit.

主题是关于推理过程中的RDMA技术。

It's on t it's on RDMA at inference.

我心想，这位搞操作系统出身的专家，居然会对RDMA感兴趣？

And I'm like, you wouldn't think that this guy who is, like, kind of operating systems guy would care about RDMA.

我的意思是，这对我来说很合理，毕竟这是云计算。

I I mean, it it makes sense to me because This is the cloud.

对吧？

Right?

没错。

That yeah.

比如你怎么在不同设备间传输KV缓存，如何高效地把训练用的GPU上的权重迁移到推理用的GPU上，这里面有太多可调的空间，本质上就是一个内存调度和搬运的系统问题。

Like like, the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is there's a lot of degrees of freedom, and it is basically a systems problem of moving memory around and scheduling.

这让我意识到，我对网络方面的理解有多原始。

This shows you how primitive my understanding of networking stuff is.

这跟WireGuard属于同一领域吗？

Is this like the domain of WireGuard as well?

不完全是。

Not quite.

所以是相关联的？

So It's adjacent?

那你把整个情况说清楚吧。

So explain everything.

当然可以。

Sure.

没问题。

Sure.

我们是怎么在不同GPU之间移动内存的？

How do we move memory around GPUs?

不好意思，我先插一句。

Well, sorry.

是的。

Yeah.

那确实是内存。

That is memory.

抱歉。

Sorry.

我刚才可能说了五分钟，讲的是你设置的私有IPv6地址。

I I was talking more and maybe I was talking, like, five minutes back about the the private IPv6 addressing that you've set up.

这本质上就是一个VPN。

It's basically a VPN.

是的。

Yeah.

有点像VPN。

It's sort of like a VPN.

而且没错，WireGuard确实如此。

And, yeah, WireGuard is is yeah.

你说得对。

You're right.

没错，就是这样。

It is it is Right.

你已经转到下一个话题了。

You already moved on to the topic.

差不多，明白了。

Similar Okay.

在同一类场景里，WireGuard 是加密的，不过这个嘛，

In the same space, WireGuard is encrypted, and and this is Oh,

你不需要加密。

you don't need encrypted.

对。

Yes.

它没有加密。

It's not encrypted.

这才是关键区别。

That's the main difference.

这是 TCP，我们用 eBPF 程序来判断是否允许某个 TCP 连接。

This is TCP, and we have eBPF programs that will reject or allow the TCP connection based on whether you're allowed to do it.

以前得用完整的服务边车，现在是直接在 Linux 内核里做了？

Used to involve a full sidecar, but now you have in the Linux kernel?

对。

Yep.

没错。

Yeah.

我不确定这算不算自然地接上我之前对分布式训练的质疑，嗯。

I don't know if this is a natural follow on to the topic of, like, my skepticism on distributed training Mhmm.

人们花大价钱买电缆来连接GPU，可就连这个都还不够快，这才是真正的瓶颈。

Is that, well, like, people spend a lot of money on, like, cables to hook up GPUs, and even that is not, like, fast enough, and that's the bottleneck.

你的网络够快吗？

Is your networking fast enough?

是的。

Yeah.

所以你是说那种完全分布式的训练，比如Dylocore之类的，跨数据中心的那种，是吧？

So I I guess you're talking about sort of fully distributed training, like a Dylocore or something, which is, like, cross data That would

对。

be yes.

这已经是最极端的情况了。

That's the extreme.

对。

Yeah.

你算是居中吧，其他人则会在实际的数据中心里用上Mellanox这样的线缆。

You're kind of in the middle, and then other people would have, like, the Mellanox cables up in in, like, their actual data center.

在Modal上运行多节点训练时，RDMA——我觉得Mellanox或者InfiniBand用的都是RDMA。

When you run multi node training on on Modal, RDMA, I think Mellanox is or InfiniBand is like a is you also use RDMA.

这本质上是一种绕过TCP网络栈、让节点之间更快传输数据的方法。

But basically, it's a way to bypass the TCP networking stack and transfer stuff much faster between one node to the other.

我们内部的网络带宽大概是每秒三太比特，这已经是行业标准了。

And and we we have, I think, like, three terabit per second internal networking, which is the standard that's needed.

明白了。

Okay.

所以我之前误解了CPU TCP 3.2那部分的意思。

So I misunderstood what part of the CPU TCP 3.2 of Okay.

是的。

Yeah.

这工作确实非常出色。

I I mean, very impressive work.

所以，你是把 Modal 的理念延伸到了交易集群上，对吧？

So is it effectively you're you're extending sort of, like, the modal philosophy to the the trading cluster?

对。

Like, yeah.

对。

Yeah.

我们当然不是为了大规模预训练设计的。

And we're we're not going for, obviously, like, large scale pretraining runs.

我们做多节点训练，主要是应对大量小规模的后训练需求。

The thing that we've built multi node training for is we see a lot of smaller scale post training.

很多人会用中等规模的量化模型做后训练，以提升推理时的精度。

Like, people are post training, like, medium sized quant models so they can get higher quality on inference.

这正好适合这样的场景。

This is a perfect fit for something like that.

没错。

Yeah.

我觉得很多实验室都是这样，先在微调阶段尝试各种方向，最后把发现有效的结果合并起来。

That is my impression of how a lot of these labs explore branches in post training and then eventually merge whatever they find in.

对。

Yeah.

我们看到的另一个多节点训练用途是，即使你有大型集群，研究人员还是在做小规模的训练任务，确实如此。

The the other use case we've seen for multi node training is even if you have a big cluster, your researchers are still doing small runs And Yes.

这样能更好地利用弹性资源，没错。

You're having elasticity there better Yeah.

很多情况都是这样。

A lot

这实际上是当前自动研究的主要瓶颈，因为你得给模型分配一些GPU。

The like, this is actually, like, the current limiting factor for auto research, which is, like, you you basically need to give your model some GPUs.

我们有个关于自动研究的博客，用的是Modal，确实不错。

We have a blog that you run on auto research and Modal is Yeah.

对。

Yeah.

说起来，它确实是个挺合适的平台。

Like, turns out to be a pretty good substrate for that.

我的理解是，自动研究这个词涵盖的内容挺多的。

So my impression is auto research means many things.

比如说，安德烈·卡佩西那类项目。

Like, if anything, the Andre coins.

目前还更像是科学展览阶段。

Right now, it's still science fair.

对吧？

Right?

说到底，我还真不知道该怎么真正实现它。

Like, not not not actually, like I don't know how we can Yeah.

实际上确实要

Actually do

教的也是同样的内容。

taught the same thing.

对。

Yeah.

你当然清楚。

You would know.

我们内部的训练和推理团队其实经常用这种通用架构。

We like, our internal both training and inference teams actually use this sort of the general shape of this quite a bit.

我们有一个内部仓库叫 AutoInference，就是用这个框架自动化的我们的 FD 工作，让智能体自动运行各种参数组合。

Like, we have this one internal repo called AutoInference, which essentially we've automated our own FD efforts using this harness, which is the agent will just spin up a sweep of different things.

它甚至会运行 NVIDIA Insight Profiler，调整配置，最终找到最优解。

It'll even run like NVIDIA Insight Profiler, and it'll, like, tweak configs and it'll arrive at the right thing.

它还会更换你的 GPU。

It'll change your GPUs.

从H200到B200都适用，效果真的很好。

It's both from h 200 to b 200, and it actually works really well.

不错。

Nice.

所以

So

顺便说一句，我很欣赏你的FTE岗位这么技术导向，还得亲自做这些事。

By the way, I enjoy that your FTE is so technical that you have to do these things.

和其他人的FTE相比，这确实很不一样。

It's very different for FTE from other people.

是的。

Yeah.

对。

Yeah.

没错。

Yeah.

我们的FTE团队本质上是做应用型推理或应用型训练研究的。

For our FTE team is essentially, they're, like, applied inference researchers or applied training researchers.

有人跟我说，他们必须得

Someone told me, like, they have to be

能动手搭建，同时也得会推广。

able to build, but they also have to be able to sell.

他们是必须去推销，还是说本来就挺厉害的？

Do they have to sell or are they, like, they're good?

这属于售后阶段的事了。

This is, post sale type of thing.

能跟客户有效沟通，这一点非常重要。

It does being able to talk to a customer and engage effectively with them matters a lot.

他们都在做同一件事。

They're all on the same thing.

你懂的。

You know?

但这其实并不算是销售工作。

But it's it's not really a sort of sales thing.

我们会给他们搭配解决方案架构师，那些人更偏向售前支持。

We we pair them with We have solution architects as well that are more on the presales side.

咱们多花点时间聊聊自动研究吧。

Okay, let's spend a bit more time on auto research.

今年这可是我的重点方向。

This is a big focus for me for this year.

这个方向最终会走向哪里？

Where does this go?

人们已经探索得足够多了吗？

You know, like have people explored enough?

你看，那些图表总是显示进步、进步，然后就 plateau 了，接着又发现新的突破点。

Like, you know, there's all these beautiful charts of, like, improve it, improve, then they sort of level off a bit, and then you find the next thing.

这本质上是不是比普通训练高了一个抽象层次？

Is this basically sort of one abstraction up from normal training?

我们是这么理解的吗？

Is that how we think about it?

还是你有别的看法？

Or do you think about it differently?

比如，是模型层面的训练，还是那种高度依赖AI的超参数搜索？

Like, model level training versus basically a high like, AI driven hyperparameter search.

有些人管这叫神经网络架构搜索之类的，对吧？

Some some people call it, like, neural architecture search or whatever.

没错。

Right?

比如说

Like

是的。

Yeah.

我的意思是，我见过的那些应用，根本没涉及到架构层面。

I mean, I so the the stuff I've seen people do with it is nowhere on the architecture level.

其实就是调参，但它是靠模型的直觉来引导的超参数搜索。

It's pretty much tweaking parameters, but it's it's basically a hyperparameter sweep that's guided by some sort of model intuition.

所以相比其他任何搜索方法，这种方式高效多了。

So it's, like, much more efficient than whatever other sweeper you would

没错。

have.

是的。

Yeah.

说到底，关键看你愿意把算力花在哪一块。

I mean, you just know, it's just a question of where you wanna spend your computer.

你懂的。

You know?

对吧。

Right.

毕竟，你只要砸足够的钱进去，总能撞出点东西来。

Because, yeah, you can just throw infinite amounts of money on this, and somehow you'll bang on Shakespeare.

你知道的。

You know?

对。

Yeah.

无限猴子。

Infinite monkey.

对。

Yeah.

这对Modal来说特别合适。

I mean, so, like, very good for Modal.

而且我觉得，代理能够启动其他代理也非常重要。

And I I think it's also very important that agents can spin up other agents.

它们可以搭建自己的基础设施。

They can spin up their own infrastructure.

这对你来说特别有帮助。

Like, very, like, very good for you.

大语言模型在生成 Modal 代码方面表现如何？

How good are LLMs at generating modal code?

你知道，在大型语言模型出现之前，优势在于你拥有数据。

Like, you know, like, the benefit of existing pre LLMs is that you are in the data?

是的。

Yeah.

它们的表现其实相当不错。

They're they're actually surprisingly good.

我觉得在 Cloud 4 之前，他们还不行，但现在他们能直接开箱即用，一次性搞定。

I think, like, pre Cloud four, they were not, and then now they're able to one shot stuff out of the box.

我们之前试着搞了个 Modal Bench，用来处理那些大模型还搞不定的难题，然后我想问，那个……

We were playing around with releasing, like, a Modal Bench for, like, the harder things that the LMs cannot do yet, and and maybe What what's

举个例子

an example

是啥？

of that?

我觉得，如果没有正确的指导和技能，智能体有时很难掌握如何利用我们的其他可观测性工具。

I think the the things that sometimes agents struggle with without right guidance and a skill is how to use the rest of our observability.

比如，当某个环节出问题时，该怎么去看日志，然后找到该修正的地方？

Like, how to something is failing, like, how do you look at the the logs and then update the right thing?

这其实是一种对这些问题的推理能力。

It's sort of reasoning about that.

但它们现在能一次性完成很多任务，确实如此。

But they're able to one shot, like Yeah.

光靠加一个技能是不够的。

Can't just add a skill to it.

对。

Yeah.

所以我们现在有了一个 Modal 技能，这正是我们开发 Modal Bench 的主要原因。

So so we have a modal skill now that which is kind of actually why we built this Modal Bench.

就是为了发现这类问题，然后在我们的系统里加以改进，确实如此。

It's to find things like that so we can we we can address them in our Yeah.

技能。

Skill.

对。

Yeah.

没错。

Yeah.

不行。

No.

不行。

No.

我的意思是，这还不错。

I mean, it's it's good.

你们现在有资源短缺的问题吗？

Are you facing any shortages?

我们经常聊GPU短缺，但CPU和内存也一样。

You know, like, we talk a lot about GPU shortages, but also CPU, also memory.

是的

Mhmm.

对

Yeah.

我们增长很快，所以必须更主动地做好规划。

We have had a lot of growth, which means that there's a we've had to be much better about proactive Yeah.

容量方面，我们已经

Capacity So we we have Which, by

顺便说一句，这简直像是MBA毕业生的理想工作。

the way, like, it's like an MBA's, like, dream job.

就是专门做这种规划。

It's like just planning this stuff.

我记得上次我和你聊这个的时候，还跟别人讨论过。

I think last time you and I talked with somebody about this.

对

Yeah.

我们有一支非常能干的团队，负责叫计算策略的岗位。

I mean, we have a a really competent team of people that we call in the roles called compute strategy.

所以，有谁对计算策略这个方向感兴趣吗？

So, yeah, anyone listening here who wants to work on Compute strategy?

是的。

Yeah.

普通人一般管这叫财务计划与分析，或者类似的名字。

I think the norm the normies call it FP and A or something.

但实际上这比那更复杂。

Well, it's more

这根本不是财务计划与分析。

it's it's not FP and A.

这里面有很多有意思的财务问题，比如一年期和三年期预留资源该怎么搭配？

It's it's there's a lot of interesting financial questions of, like like, what is the blend between one year and three year reservations?

我们该怎么预测自己的资源容量？

How do we forecast our own capacity?

我们得好好建模，特别是因为我们的算力在不同类型的GPU和不同地区之间可以灵活调配，还得对供应链未来怎么发展有自己的判断，然后据此做出决策。

How do we basically, especially since our capacity is very fungible across different GPU types and different regions, like, you have to model a lot of it, and you also have to have an opinion on how the supply chain is gonna evolve, and then you have to, like, take bets based

基于这些判断。

on that.

原子化交易。

Tokatomics.

这可能不算什么真知灼见，但我一直在想，还有哪些行业也遇到过类似的问题——我们不可能在这些难题上第一个吃螃蟹。

This is, like, probably not a real point, but at the end, I was trying to think about, like, what other industries I was trying to think about, you know, we cannot be first to, like, these kinds of problems.

是的。

Yeah.

还有哪些行业经历过这种事？

And what other industries have had this?

我想到了航空公司，它们处理燃油成本的方式。

And I was like, airlines with with fuel.

它们得对燃油价格进行对冲。

And, like, they have to hedge their fuel.

长期以来，西南航空因为大胆押注燃油价格，成本远低于其他航空公司。

And, like, I think for a long time, Southwest, because they made them like a hero fuel bet, they like were like super low cost because compared to everyone else.

是的。

Yeah.

我之前还真没想过这一点。

I hadn't thought about that.

而我们现在

And then we're

也正处在一个挺有意思的时代，你知道的。

in a fun time too, you know?

对。

Yeah.

对我们来说，整个计算业务的核心之一，就是把资源管理做到极致。

It's a lot of the compute business in general for us is also about being very good about capacity management.

这不仅能让单位经济效益更好，长期来看，也能为客户创造更多价值。

That is how you have great unit economics, but also over time, it's how you can unlock more value for customers.

比如，我们现在正在开发一种功能，允许那些对延迟不敏感的客户享受更优惠的价格。他们可以在大概24小时内收到结果，本质上就是一个批量处理层级。

Like, one of the things we're building now is like a way for customers to get if they don't care about latency, like, get much cheaper pricing and they'll get results back in, like, next twenty four hours or something, like a batch tier, essentially.

对。

Yeah.

这些都是我们可以利用的手段，因为我们掌控着整个技术栈、调度系统等等，从而能为用户提供足够的……

And those are levers we have because they control the whole stack and scheduling and whatnot to give people sufficient

是的。

Yeah.

我觉得它们没那么受欢迎。

I feel like they're not as popular.

比如像 Frontier Labs 提供的那些 API。

Like, like, the the Frontier Labs have all those APIs.

它们的受欢迎程度远不如应有的水平。

They're not as popular as they should be.

我们看到的对这类东西的需求，其实并不是针对大语言模型的。

The demand that we see for something like that is actually not for LLMs.

不过有时候人们确实需要运行评估任务或生成合成数据，这种情况下就挺合适的。

Although sometimes people wanna run evals do and synthetic data prep, and and there it makes sense.

好的。

K.

但很多非LLM公司，比如做计算生物学的，他们从来没跑过大规模批处理任务，也不在乎结果什么时候出来。

But it's from a lot of non LLM companies, like people who are doing computational bio, like they they haven't run really big batch jobs and they don't care about when they get

他们不着急要结果。

it back.

是的。

Yeah.

而且，这其实也跟另一个问题很像——就是任务能不能按时完成。

And, like, they have a reasonable it's it's also, like, a cousin to the stopping problem of, like, will this finish in time?

是的。

Yeah.

你可以给它设定一个时间上限。

You you can you can bound it.

是的

Yeah.

你可以给用户设定服务等级协议。

Like, you can give people SLAs on it.

是的

Yeah.

对

Yeah.

我觉得有趣的是，Modal接下来的发展方向。

I think what's what's interesting is, like, the the next phase of Modal.

对

Mhmm.

现在你们已经站稳脚跟，成了这些顶尖公司中知名的计算平台，大家对你们有什么新的期待？

Like, what, you know, do people expect from you now that you're sort of established and you're, like, a well known compute player among all these leading companies?

你们刚办完一次推理功能的发布周，我们之前也简单聊过这些发布。

You had an inference launch week, and we talked a little bit about the launches.

还有别的吗？

Like, what else?

人们还应该知道些什么？

Like, what else should people know?

我们正在打造一些能让用户生活更轻松的基础功能。

We are building primitives that make our users' lives much easier.

比如在大语言模型推理方面，会有成千上万家公司不再自己训练模型，而是直接部署开源模型来做推理。

So I think, for example, with LLM inference, thousands more companies are going to post training their own models and deploy open source models for inference.

所以我们一直在思考，什么样的产品形态最适合这种情况。

So we're thinking a lot about what is the best product shape for that.

这涉及到从我们的训练平台到能实现前沿性能的端点服务，方方面面。

And that involves everything from our training gym to then endpoints that get frontier level performance.

不过说实话，我还没跟谁深入聊过这个。

You know, again, but I haven't talked to anyone.

在其他行业里，情况看起来会不太一样。

It looks somewhat different on other verticals.

我们还看到很多实时音视频方面的应用，所以我们正在开发区域路由和故障切换这类功能。

Like, we're also seeing a lot of real time audio video stuff in there, which is why, like, we're working on things like regional routing with with fallbacks.

这样用户就能接入离他们最近的GPU。

So you can you can get sort of GPUs that are as close to users as possible.

从而实现视频流媒体等场景的低延迟体验。

So you you get like low latency for video streaming and and whatnot.

在智能代理这一块，我们仍然和客户保持紧密合作，因为他们的需求变化实在太快了。

And then on the agent side, it's we're still sort of working very closely with our customers because stuff is changing so fast in terms of what they need.

除了沙盒和持久化文件系统，当人们构建生产级代理时，还会需要很多其他功能。

And I think beyond sandboxes and persistent file systems, there's a lot of other things people will need from this agent stack as they build production agents.

所以我们也在考虑那些能填补这些需求的其他功能。

So, yeah, we're thinking about those other things that fit in there.

我想了解一下，还有哪些其他功能？

I wanted to ask what what the other things are.

对。

Yeah.

我应该把它分享出来。

I probably should share it.

我觉得可以。

I think okay.

我经常思考云计算的核心组成部分，你们也提到过计算、存储和网络。

So I do think a lot about the principal components of cloud, and you do talk about compute storage networking.

对。

Mhmm.

是的。

Yeah.

到目前为止，对我来说都没问题。

Because so far for me, it's fine.

就前几代云计算而言，确实没问题。

So far for the I mean, the first couple generations of cloud, it's fine.

代理系统有什么本质上的不同，以至于需要新的权限级别？

What's different qualitatively different about agents that you you need some new permission level?

很多人，你知道的，显然没问题。

Like, lot of people you know, obviously okay.

我会不停地抛出一些想法，希望能激起你的灵感。

And I'll just kinda spew tokens at you until until it, like, hopefully sparks something.

对。

Yep.

现在的新情况是，Cloud Code 做的事，会绕过权限，或者通过命令直接放行，诸如此类。

Like, the the new level now is whatever Cloud Code does, which is dangerously skip permissions or, like, allow this by command or, like, whatever.

对吧？

Right?

有时候他们就会说，好吧。

And and sometimes they're like, okay.

我们有个自适应思维模式，简单说就是：相信我就行。

We we have, like, this adaptive thinking mode where, like, just just trust me, bro.

我会替你做决定。

I I will make calls for you.

就这些了吗？

Is that it?

你懂的。

You know?

其实就是由语言模型来管理权限，差不多是这样。

Like, basically, like, sort of LM mediated permissions.

用目标循环驱动，然后疯狂刷请求，老兄。

Looping it with a goal and flooding, bro.

是的。

Yeah.

我对语言模型在沙盒层级上管理权限这事持怀疑态度，毕竟你得有明确的边界。

I mean, I'm I'm skeptical of LM mediated permissions for stuff that is at the sandbox level because you do want hard boundaries.

没错。

Yeah.

不然的话，很明显有人会把数据偷走。

Otherwise, obviously, someone can exfiltrate stuff.

但是呢，就是

But, like like

也许这种想法有点过时了。

like, maybe maybe that's old school thinking.

我们可能才是老古董。

Maybe we're the dinosaurs.

也许所谓的AIOS或者LLMOS，本质上就是个牛逼的LLM内核。

Maybe the AIOS or the LLMOS is really the kernel as a goddamn LLM.

这让人感觉挺不舒服的。

Like, it makes you feel uncomfortable.

对。

Yeah.

明白。

Know.

但这就是信任LLM的代价。

But that's what trusting the LLM is.

想象一下，一个完美的球形奶牛式大模型。

Like, imagine a spherical cow perfect LLM.

对。

Right.

就让它这样吧。

Let it.

也许吧。

Maybe.

也许吧。

Maybe.

我想试试它的极限在哪。

I wanna test the boundaries.

对吧？

Right?

当然，我其实并不真这么认为，但我就是想看看自己错在哪，因为这才是真正的分歧所在。

Like, obviously, I I don't believe that, but I wanna see where I'm wrong because that's that's the non consensus.

是的。

Yeah.

我的意思是，当你想要的时候，总是需要硬性限制，同时也可以搭配一些更灵活的约束。

I mean, I think you always need hard guardrails when you want and you can pair those with softer guardrails.

对吧？

Right?

而且这个请求是通过中介处理的。

And the ask a deal on mediated.

我想请你谈谈 Modal 之外的整个生态系统的看法。

There I'll get you end with a couple of your commentary on, like, the ecosystem outside of Modal.

每个平台都有自己的托管代理。

Managed agents, everyone has one.

Gemini、OpenAI、Cloud，这些对你来说都很有用。

Gemini, OpenAI, Cloud, very useful for you.

但同时，这也是他们逐步渗透进你领域的手段。

But also, like, it is their way of starting to edge into your space.

是的。

Yeah.

怎么回事？

What's going on?

是的。

Yeah.

我们非常高兴能与Anthropic以及其他一些基础实验室合作，不过有些合作伙伴我们暂时不方便透露。

I mean, we're very excited to partner with Anthropic and some of the other foundational labs, we'll not name who we're also working with.

我们认为，对于刚开始构建智能体的人来说，托管智能体是个很好的起点。

The way we see it is the managed agent thing is a great place to start if you're starting out building an agent.

但当你准备打造更接近生产环境的系统时——比如像Ramp这样的公司，他们不仅自己开发智能体，还用我们的平台运行他们的会计代理和对外服务代理——

And but but then when you get to building something more production grade, like you're a company that's like Ramp, that's building their own Ramp also runs their accounting agent on us, their external facing agent.

你就需要对计算基础组件有更多控制权，比如：智能体能访问哪些文件、如何持久化存储、如何进行快照和恢复，

You need a lot more control over your compute primitive on things like what sort of how do you persist different files that the agent has access to and how do you snapshot and restore?

以及如何管理网络访问。

How do you control the networking?

也许你需要GPU。

Maybe you want GPUs.

到了这一步，你就希望有个专门的沙盒服务商能提供这些资源。

When you get to that point, you kind of want a specialized sandbox provider that gives you those things.

这正是我们想扮演的角色。

And that's the role that we are trying to play.

是的。

Yeah.

我们对运行框架本身没什么特别看法，不管是用云托管代理连上Modal沙盒，还是直接在Modal沙盒里运行框架，都行。

We don't really have an opinion on the harness, whether it runs in it's a Cloud Managed Agent and you hook it up to Modal Sandbox, or you run the harness in Modal Sandbox.

我们等着看大家最终会怎么选。

We'll see where people converge with that.

对。

Yeah.

你对元框架有什么看法吗？

Do you have any opinions on, like, the meta harnesses?