开源AI如何成为关键基础设施

一个有趣的思维实验是，如果GPU价格下降99%，那我们是不是就能回到真正的开源时代？

The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source

世界？

world?

如果审核问题永远解决不了，未来人们会默认选择开源方式，因为在那里你才能确定能控制自己的护栏，用于可信的用例。

If moderation is never solved, in the future, people will go to open way by default because that is where you know for sure you can control your guardrail for trusted use cases.

那你能谈谈vLLM在这个架构中处于什么位置吗？

And can you talk about where vLLM sits in that stack?

vLLM是一个推理引擎。

VLLM is a inference engine.

它有点像数据库、操作系统以及其他关键软件，为AGI提供动力。

It is kind of like databases and operating system and other critical software to power AGI.

NVIDIA、AMD、谷歌，它们最新的芯片都会确保VEON能在上面运行。

NVIDIA, AMD, Google, their newest chip will make sure VEON can run on them.

而且很多情况下，它们会用VEON作为基准测试。

And then a lot of cases, they use VEON as a benchmark.

我们正在弥合近10倍的差距。

We're bridging almost a 10 x gap.

对于专有模型，有常规模式和快速模式。

For proprietary models, there is a regular mode and fast mode.

但对于开放权重模型，每个提供商都可能提供多达10种不同的速度级别。

But for open-weight, every provider can offer potentially even 10 different levels of speed.

五年后，开源AI模型会缩小与前沿模型的差距吗？

Five years from now, open source AI models, have they closed the gap with frontier models?

从能力上看，我真的没看到大的差距，

Capability wise, I don't really see a big gap,

甚至今天也没有，因为开源AI已经成为塑造行业的最重要力量之一。

not even today, because Open source AI has become one of the most important forces shaping the industry.

但让前沿模型惠及全球，需要的远不止发布模型权重。

But making frontier models available to the world requires far more than releasing model weights.

它需要一个全新的基础设施层。

It requires an entirely new infrastructure layer.

在本期节目中，Elena Burger和Matt Bornstein邀请了Inferact的联合创始人兼CEO Simon Mo，一起讨论开源推理如何成为AI的关键基础设施，为什么企业越来越多地转向开放权重模型，以及下一代AI系统需要什么。

In this episode, Elena Burger and Matt Bornstein are joined by Simon Mo, cofounder and CEO of Inferact, to discuss how open source inference became critical infrastructure for AI, why enterprises are increasingly turning to open weight models, and what the next generation of AI systems will require.

他们还探讨了模型许可、推理经济学以及OpenAI的未来。

They also explore model licensing, inference economics, and the future of OpenAI.

今天，我们请到了Inferact的联合创始人Simon Mo，他也是vLLM的主要维护者，这个开源推理引擎目前在任何时刻都运行在五十万块GPU上。

Today, we're here with Simon Mo, cofounder of Inferact and a lead maintainer of vLLM, the open source inference engine now running on half a million GPUs at any moment.

我们还请到了Matt Bornstein，他是a16z的普通合伙人。

We're also joined by Matt Bornstein, an a sixteen z general partner.

Simon，Matt，非常感谢你们今天能来。

Simon, Matt, thank you so much for joining us.

我想我们首先应该从开源AI以及开源AI的近期历史开始聊起。

I think first we should start with open source AI and kind of the more recent history of open source AI.

vLLM实际上起源于2022年，在ChatGPT出现之前。

So vLLM actually has its origins kind of back in 2022 pre chat GPT.

当时你们的团队想加快一个慢吞吞的开源演示，结果却遇到了一堆未解决的问题。

And your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems.

那你能否谈谈，为什么服务大语言模型与大家已经知道如何运行的机器学习工作负载有如此根本的不同？

So can you talk about what made serving an LLM so fundamentally different from the ML workloads everyone already knew how to run?

是的。

Yep.

很高兴来到这里。

Good to be here.

服务大语言模型是一个根本不同的问题，因为服务它需要在GPU或TPU等加速器上运行，这是一个计算密集型的过程，需要大量的工程工作，并确保对于每个请求，用户都能快速高效地看到语言模型的响应。

So serving large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs and it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LM's response quickly and efficiently.

所以这通常意味着我们需要处理输入分布、每个请求的长度、输出分布（这是不确定的）以及批处理和调度等方面的差异，这些在推理引擎的核心中更为重要。

So this typically means we need to handle differences in input distribution, how long each request is, output distribution, which is nondeterministic, and batching and scheduling a lot more in the at the core of the inference engine.

对。

Yeah.

这个项目已经存在大约四年了，但公司成立的时间稍晚一些。

And so the team, the project has been around for about four years now, but the company is a little bit more recent.

所以我想把这个问题抛给你们两位，马特。

So I wanna throw this out to either of you, Matt.

我知道你认识这个团队很久了，也一直在观察他们。

I know you've known the team and observed the team for a very long time.

那你是在什么时候看到它从一个备受喜爱的开源项目，转变为关键基础设施，然后成立公司的？

So at what point did you see this sort of transitioning from being a much beloved open source project to critical infrastructure and then a company?

对。

Yeah.

我觉得得往回看一点。

I mean, I think you have to go back a little bit.

对吧？

Right?

早期，开源是AI模型的常态。

Open source was the norm for AI models early on.

对吧？

Right?

我们甚至有一家公司叫OpenAI，这现在有点成笑话了。

I mean, we literally had this company called OpenAI, which, you know, it's become a little bit of a joke.

它已经不像以前那么开放了，或者说远不如以前开放了。

It's not as open as it once was or not nearly as open as it once was.

但在早期，所有前沿AI工作都是开源的，或者至少发布了开放权重，这跟真正的开源有点不同。

But early on, all all the frontier AI work was being open source or at least released into open weights, which is a little bit different than true open source.

人们大多可以在自己已有的硬件或电脑上运行这些模型。

And people could mostly run these models sort of on hardware or computers that they already had.

西蒙，我很好奇，你还记得吗？第一个需要你专门去找特殊软件和一套特殊电脑才能运行的模型是哪个？

I'm curious, Simon, like maybe you remember, like, was like the first model that, like, you actually needed to go out and get special software and, a special set of computers to be able to run?

可能是伯特。

Probably Burt.

在那之前，像ResNet这样的，用于图像计算、计算机视觉分类的模型。

And before that it was like ResNet for computation like images, computer vision classification.

所以ResNet已经需要在NVIDIA K80上运行了，那是最早的SPU之一，在AWS和其他地方。

So ResNet already need to run on NVIDIA K80, which is kind of one of the first SPU AWS and other places.

但即使在那时，ResNet仍然可以在普通设备上运行，甚至CPU设备，只是非常慢。

But even at this point, ResNet, you can still kind of run on a commodity, even CPU devices, it's just very slow.

但对于BERT来说，运行它就像，哇，你必须在GPU上运行它，才能让它对任何翻译或任何任务来说更快更高效。

But for BERT, where running at it is like wow you have to run it on GPU to make it anything faster and efficient for anything translation or any task.

那甚至是在2020年之前。

So that was like before 2020 even.

想想这个真有趣。

It's so funny thinking about this.

我觉得所有的记忆都涌上来了。

I'm like all the memories are flooding.

Hugging Face上有一千种BERT变体。

Hugging Face had a thousand Burke variants on it.

对。

Yeah.

你得去找适合你特定任务的BERT变体。

You have to go find, like, the right Burke variant for your particular task.

对。

Yeah.

对。

Yeah.

而且，是的，你说得对。

And, yeah, and you're right.

有些人，我想，有自己的GPU，可以运行它。

You had to some people, I guess, had their own GPUs and could run it.

但很多人不得不去云上配置资源之类的。

But, a lot of people had to go provision cloud and like that.

那真的很有趣。

That's really funny.

所以，是的，你看，我的意思是，BERT是一个早期的模型。

And so, yeah, so so look, I mean, BERT was an early Yeah.

后来的模型更大、更复杂，占用更多内存和计算资源。

Language model that newer models are much bigger, much more sophisticated, take up a lot more memory, a lot more compute.

所以LLM从一开始，对吧，就是关于运行这些更强大的模型，你不能只是随便搞搞就能在本地解决的。

And so bLLM really from the early days, right, was about running these more powerful models that you couldn't just sort of do it, figure it out on your Yeah.

对。

Yeah.

我觉得要聊到现在的情况，最好先谈谈它是在什么时候真正成为这些模型的关键基础设施的。

I think to get us to this present day, I think it would be good to talk about kind of at what point it really became critical infrastructure for these models?

还有，我们是什么时候开始看到这些更大的开源模型进入这个领域的？

And when did we even start to see these larger open source models kind of come into the field?

嗯，我们确实看到了软件栈的关键性。

Well, we really see the criticality of the software stack.

这总是关于带来开放前沿智能。

It's always about bringing open frontier intelligence.

开放前沿智能是这次K3模型发布的一个口号，也关乎vLLM在推理基础设施方面的使命。

This is open frontier intelligence is a catchphrase for the K3 model release this time, and it's also about vLLM's mission about inference infrastructure.

所以如果我们看的话，它们什么时候开始变得关键，就是当人们开始依赖它们来做应用、日常生产力的时候。

So if we look at it, when do they start being critical is when people are relying on it for their application, for their day to day productivity.

所以可能是2023年，那时GitHub Copilot和ChatGPT已经成为人们离不开的东西了。

So maybe 2023 when GitHub Copilot and ChargeGPT are something that people just cannot live out with anymore.

那时候，权重模型已经成为人们日常生活的基石。

And at that point, weight model is already backing and become a cornerstone of how people are living every day.

那时候，我们需要加速器来运行开放权重模型和开放权重软件，以及开源软件，确保一切运行良好。

And at that point, we will need the accelerator running open weight model and open weight software to open source software to make sure it works well.

是的。

Yeah.

并且要对其拥有完全的灵活性和控制权。

And have all flexibility and control over it.

对。

Yeah.

从初创公司的角度来看，这挺有意思的。

It's sort of interesting from a startup standpoint.

就像你提到的，GPT-3或者早期的ChatGPT，那些闭源解决方案当时开始对一小部分人变得至关重要。

Like you mentioned sort of GPT-three or like early ChatGPT, those closed source solutions were starting to become critical to like a small group of people in that time.

开源也存在，但当时更像是一种新奇事物，或者是爱好者的玩物。

And open source existed, but it was a little bit of a curiosity or sort of an enthusiast thing.

随着前沿的扩展，尤其是闭源模型的兴起，越来越多的开源被拖进来，成为其背后的关键，如果你明白我的意思的话。

As the frontier has expanded, particularly with closed source models, more and more open source has been dragged in as critical behind it, if that makes sense.

比如，在任何时候，包括现在，我认为OpenAI和Anthropic的模型总体上比开源模型使用更广泛，也更关键。

Like, at any given point in time, including now, I think models from open AI and Anthropic are kind of more widely used and more critical kind of in general than open source models.

但我确实认为我们在大约一年前跨过了一个门槛，当时很多小公司或新的应用公司在试图弄清楚：我怎么能真正构建一个AI，而不只是OpenAI之上的一个包装？

But I do think we passed a threshold in, I wanna say about a year ago, where a bunch of smaller companies or new application companies, as they were trying to figure out, how do I really build an AI without just being a wrapper on top of open AI?

这个问题的答案最终是开源。

The answer to that question turned out to be open source.

Cursor就是这么做的。

This is what Cursor did.

Decagon和Harvey现在也正在这样做，还有其他一些非常强大的应用层初创公司也做出了决定：我们不能只建立在闭源之上。

This is what sort of Decagon and Harvey are in the process of doing now, and a bunch of other, like, really, really strong application level startups sort of made the determination we can't build just on closed source.

我们需要自己做中期训练、后期训练，以及我们自己的推理和部署技巧，所有这些都意味着必须建立在开源之上。

We need to do our own mid training, our own post training, our own sort of inference and deployment tricks, and all of that means it must be built on top of open source.

闭源供应商不会给你这样做的权限。

The closed source vendors won't give you the access to do this.

所以我的看法是，大概一年前吧。

So my read is like kind of a year ish ago.

对。

Mhmm.

开源变得非常核心，虽然这不一定总是显而易见，因为它深深嵌入在一些产品中，但一些最具创新性的产品和应用现在真的非常依赖它。

Open source became really central in a way that's not always visible because it's deeply embedded in some of these products, but some of the most innovative products and applications now really depend on this very deeply.

是的。

Yeah.

是的。

Yeah.

那你能谈谈vLLM在这个技术栈中的位置吗？特别是当一些大型企业选择使用开源模型的时候。

And can you talk about where vLLM sits in that stack where we do have these larger enterprise companies that are choosing to use open source models?

对他们来说，vLLM在技术栈中处于什么位置？

Like, where does vLLM sit in the stack for them?

好的。

Yeah.

我是说，你应该——我想说，基本上每个人都在用vLLM。

Mean, you should I'd like to just about everybody uses vLLM.

你应该介绍一下它。

You should describe it.

基本上每个人都在用vLLM。

Just about everybody use vLLM.

vLLM是一个推理引擎。

VLLM is a inference engine.

也就是说，它的工作就是把可用的GPU变成一个运行中的智能端点。

That means its job is to turn available GPUs into a running endpoint for intelligence.

所以，它就像数据库、操作系统以及其他关键软件一样，为这个经济或AGI提供动力，大家现在都在用它来确保成本效益、效率、可靠性，并且始终保持在最前沿，因为对于vLLM来说，我们目前支持超过1000种模型架构，其中很多是专有的，但也有很多是开放权重的，对吧？

So that means it is kind of like databases and operating system and other critical software to power this economy or power the AGI that everybody really uses today to ensure they can have cost effectiveness, efficiency, reliability, and also always staying on the frontier because for vLLM, we support more than 1,000 model architecture up to today, and a lot of those are proprietary, but also a lot of those are open weight, right?

而且很多模型架构，当它们从研究原型转变为全球可访问的开放权重模型架构时，它们会立即在vLLM上运行。

And a lot of those model architecture, when they're becoming transitioning from a research prototype to world accessible open weight model architecture, they are live on vLLM immediately.

这就是我们所说的“零日模型发布”流程。

So that's a process we call day zero model release.

另外，vLLM 还和所有硬件厂商紧密合作。

And additionally, vLLM also work closely with all the hardware vendors.

也就是说，无论是 NVIDIA、AMD、谷歌、亚马逊、英特尔还是其他很多厂商，他们最新的芯片都会确保 vLLM 能跑在上面，而且很多时候他们会用 vLLM 作为基准来测试芯片性能。

So that means across like NVIDIA, AMD, Google, and Amazon, Intel, a lot more, their newest chip will make sure vLLM can run on them and then a lot of cases they use vLLM as a benchmark to make sure it runs well on them.

所以，这种模型运行和硬件结合的融合点，就是魔法发生的地方，也是 vLLM 的所在。

So this kind of fusion of where models run and where it gets to meet the hardware is where the magic happens and that's where vLLM is.

你之前跟我讲过一些幕后故事，说现在其实不容易。

And you've told me some of the behind the scenes stories, like it's actually not easy these days.

模型发布不仅仅是技术工作，还有很多人的戏剧性。

There are model releases, it's like a lot of human drama in addition to like technical work.

我想知道有没有什么故事你觉得可以分享的？

I guess are there any stories there that you think are okay to share?

哦，这其实是一个非常有趣的协同设计过程，因为从模型实验室的角度来看，这些是构建了模型的天才研究员。

Oh it's actually a very fun co design process because from model lab's point of view, these are brilliant researchers who have built this model.

现在他们最大的问题是，怎么把这个模型推向世界，确保每个人都能用，而且跑得好。

Now their biggest question becomes, how do we get this out of the world and make sure everybody's able to use it and run it well?

我们和一些模型实验室合作过，因为他们已经在生产或研究过程中使用了vLLM，所以他们会为你把所有事情都处理好。

And we have worked with model labs that are very just because they just use vON already in production or in their research process, they will just down everything for you.

这时候我们去找他们，就像说，嗨，我们是vLLM团队，我们想支持你们的开源模型，我们想以开源的方式提供这种广泛的玻璃服务，让你们的模型在我们的架构上运行良好。

This is a moment when we go to them, it's like, hi, we're the vLM team, we would like to support your open source model, we would like to offer in a way this kind of open source, but wide glass service to get your model running well on our architecture.

然后你回来，我们得到模型实验室的回复，比如，哦，我们已经让它运行起来了，因为我们正在为RO流程运行它。

And then you return and we get the model labs like, oh, we'll get it working already because we're running it for the RO process.

给你，只要审查我们的代码并合并我们的拉取请求。

Here you go, just review our code and merge our pull request.

另一方面，我们确实有一些模型实验室不知道这怎么能行。

And then on the other end, we really have model labs that just don't know how this can work.

他们

They

因为系统不是他们的核心？

Because systems is not like their core?

对。

Yeah.

因为系统不是他们的核心，而且他们一直在训练，或者他们有自己的内部推理引擎，就是不知道如何适应开放生态。

Because system is not their core, and they have been training or maybe they have their internal inference engine that just don't know how it will adapt to the open way ecosystem.

顺便说一句，这其实是一个非常多方参与的过程。

And by the way, this is also a very much a multi party kind of involvement process.

每次模型发布通常都涉及模型实验室、主要或次要硬件供应商、我们、Hugging Face，他们是模型格式和模型中心的供应商。

Every model race typically involves the model lab, involves primary or secondary hardware vendors, involves us, involves Hugging Face, who are the model format and model hub vendors.

然后根据需求，这个模型实验室会涉及10到20个不同的发布合作伙伴。

And then depending on the appetite, this model lab involves 10 or 20 different kinds of release partners.

这些可能是推理云，可能是公共超大规模云，任何运行这个模型的人，你都希望他们确保模型能成功运行。

These could be inference clouds, these could be public hyperscalers, whoever is going to run this model and you want them to ensure the model are running successfully.

所以即使到今天，如果你看K3模型的发布，那是一个完整的合作伙伴关系和努力，确保模型一旦发布，因为那只是互联网上的几TB文件，人们真的会很好地使用它。

So even up to today, if you look at the K3 model release, it's a whole partnership and a drive to make sure that the model is once the model is released, because it's just a few terabytes of files sitting on the Internet, that people are actually going to be using it really, really well.

甚至从2022年初，2024年，如果你还记得Mistral发布他们的第一个模型时，他们只是放了一个PTP的种子链接，然后每个人都挣扎着试图让它跑起来。

Even from the beginning of 2022, 2024, if you remember when Mistral dropped their first model, they just dropped a torrent link for PTP, and then everybody's, like, struggling and trying to get it up and running.

然后我们在幕后与Mistral团队合作，试图让推理引擎在vLLM中支持它。

And then we're working behind the scene with the Mistral team trying to get the inference engine support working in vLLM.

这是最早期令人兴奋的周末之一，能花时间在这上面。

This is one of the most early on exciting weekend that are able to spend on this.

然后周末过后，当大家都在努力分析发生了什么，周一、周二，嗯，

And then after the weekend, when everybody's trying to really analyze what's going on, and Monday, Tuesday, well, Mr.

O和我们刚宣布，你可以在这里成功运行vLLM，大家都能轻松复用很多工作，并在此基础上继续构建。

O and us just announced, here, you can run it on vLLM successfully here, and everybody will able to easily reuse a lot of the work and start building on top of it.

那段时间挺有趣的，像我这样的爱好者可以手忙脚乱地下载模型，然后在某个地方跑起来。

That was sort of a fun time where, like, enthusiasts like me could just, like, scramble to, like, download the model and, like, get it running somewhere.

对。

Yeah.

对。

Yeah.

我很高兴专业人士没有接管它，因为它从来都不是很好用。

I'm glad the professionals didn't take it over because it never worked, like, very well.

挺有意思的。

Was fun.

那真是一段有趣的时光。

Was like a fun moment in time.

对。

Yeah.

那么，把话题拉回现在，我觉得开源模型以及蒸馏技术最近经常上新闻。

So so to bring things, you know, forward to the present, I think open source models and also, you know, distillation have have been in the news recently.

Inferact 签署了 NVIDIA 的开放权重和美国 AI 领导力信函，a16z、Meta、亚马逊等几十家公司也签署了。

Inferact signed the NVIDIA open weights and American AI leadership letter that that was signed by also a16z, Meta, Amazon, dozens of other companies.

你能谈谈你决定签署这封信的原因吗？你当时在市场上和新闻中真正回应的是什么？

Can you just talk about, you know, your decision to sign that and and sort of what what you were really kind of responding to in the market and and kind of in the news?

是的。

Yeah.

对我们来说，我真正想支持的是开放权重在生态系统中绝对重要。

So for us, what I really want to stand behind is open way absolutely matters in the ecosystem.

世界不能只被专有 API 控制，也不能让这些模型的开放权重、开放开发和研究的权利被封锁或禁止，对吧？

The world cannot just be controlled by proprietary APIs and where open way, open development and research of these models are blocked or banned, right?

Inferact Greet签署的承诺是，我们希望帮助并促进这个生态系统的发展。

The pledge that Inferact Greet signed out for is we want to help and foster this ecosystem.

我们通常处于这个生态系统的下游，对吧？

Where we are typically in a little bit downstream of this ecosystem, right?

推理引擎不参与预训练或强化学习过程，但它们是模型真正与世界接触的地方。

Inference engine are not part of the pre training process nor the RL process, but where the model actually meets the world.

从我们所见，人们正在充分发挥他们的想象力，并将这种对开放权重模型的想象付诸实践。

And from what we're seeing, people are just really using their imagination and ability to materialize this imagination of open way model.

他们能够如此高效地利用这些开放权重模型。

They're able to leverage this open way model so much effectively.

这里

There's

几乎有两个方面，对吧？

almost two pieces to this, right?

一个是成本问题，即封闭模型太贵了。

There's like the cost thing where it's like the closed models are too expensive.

然后还有一种控制方面的考虑，就是我想掌控我的基础设施，并且控制模型。

And then there's sort of the control thing where I I I wanna sort of be in control of my infrastructure and and and control the model.

对吧？

Right?

如果我想扩展它，或者加上自己的防护措施之类的。

If I need to extend it or or put on my own guardrails or anything.

我只是好奇。

I'm I'm just curious.

你有没有从客户那里听说过？

Have you heard from from customers?

比如，这两件事对他们都重要吗，还是说只要他们有控制权，就愿意付钱？

Like, are are both those things important to them, or or, like, are they kind of willing to pay as long as they have the control?

或者也许有不同的使用场景。

Or or maybe there are different use cases.

我觉得，就像，你会随着时间变化。

I think, like, you fluctuate over time.

所以过去几年控制权很重要，但最近几个月成本开始变得重要。

So control matters a lot over the last few years and then costs just start to matter over the last few months.

所以成本真的很重要，从人们试图摆脱他们昂贵的编程计划，以及像飙升的token最大支出开始。

So cost really matters starting from people trying to migrate off their expensive coding plan and like every skyrocketing token maximum spend.

但控制权一直是核心，他们甚至想控制成本，对吧？

But control has always been in the backbone of this, is they want to, even in a way, to control the cost, right?

但这也是关于控制系统性能，相对于他们支付的费用。

But also it's about controlling the system performance against what they're paying for.

所以例如，对于语音代理公司，他们想控制自己的模型，以确保模型能在要求的时间内响应，这样客户在打电话时，可以确保代理按照SLA响应。

So for example, for a voice agent company, they want to control their own model so that they can make sure the model actually responds by their required time so the customer, when they're on the phone, they can ensure the agent is responding according to a SLA.

这有时只能通过你控制的智能来实现，因为你了解你运行的整个硬件和你监控的整个系统，而不是依赖专有API的关键基础设施，那可能随时宕机或违反合同。

And this sometimes is only you can do with your controlled intelligence because you know the whole hardware you're running and the whole system you're monitoring versus signing up for relying on your critical infrastructure with a proprietary API where they might go down anytime or have violation of the contract anytime.

是的。

Yep.

西蒙，你回到成本点，你在最近写的关于Kimi K3发布的文章中确实指出，经济性其实不是重点。

Simon, you also to to go back to the cost point, you actually make the point in an essay you recently wrote about the release of Kimi K3 that actually the economics is besides the point.

而且实际上，你知道，对于这些真正非常优秀、处于前沿的开源权重模型，它们是由非常杰出的研究人员设计的。

And it's actually, you know, in in the case of these, you know, just really, really great open weight models that are on the frontier that were designed by really brilliant researchers.

比如，这些模型在某些情况下，和闭源模型一样昂贵。

Like like, these models are in some cases just as expensive as, you know, the closed source model.

所以在那些情况下，运行它们有什么意义呢？在运行过程中，我们又能从架构上学到什么呢？

So in those cases, kind of what is the point of running them and kind of what do we learn architecturally in course of running them?

对。

Yeah.

所以首先关于成本，它们不一定和专有模型一样贵，而是，首先，成本这个话题已经被反复讨论过，甚至几个月前GLM 5.2发布时也是这样。

So first on cost, it's not necessarily they are as expensive as a proprietary model, but rather, first, the cost discourse has been discussed over and over again with even GLM 5.2 a few months back.

所以开源模型有时确实便宜很多。

So open model are sometimes definitely a lot cheaper.

但对于这个模型，有一个很大的阶段性变化，我们正在弥合近10倍的差距，但最终落在中间的某个位置，Kimi K3不像Claude或GPT Soul那么贵，但比G1 5.2贵很多。

But for this model, there's a big sort of step change where we're bridging almost a 10x gap but strike somewhere in the middle, where Kimi K3 is not as expensive as Claude or GPT Soul, but it is a lot more expensive than G1 5.2.

这是为什么呢？

Why is that?

而且我确实认为，这就是正确把握市场定价、理解市场定位的关键所在。

And I do believe this is the point of where pricing intelligence with the market correctly and understanding where it is.

但讨论的重点应该更多地放在，哇，这个模型把OPUS 4.8级别的能力带到了我们自己的基础设施上，我可以使用它、运行它、微调它，还能确切知道需要多少token，了解具体的性能表现。

But then the majority part of the discourse should be focusing on, wow, this model is bringing a OPUS 4.8 level model to our own infrastructure that I can use, I can run, I can fine tune, I can be able to understand exactly how many tokens do I need, understand the exact performance profile.

比如说，对于专有模型，只有常规模式和快速模式两种选择。

The reason here for example is for proprietary models, there is regular mode and fast mode, and that's only the two switch here.

但对于开放权重模型，当你运行它时，每个提供商可能提供多达10种不同的速度级别，从最慢的模式（可能便宜很多）到每秒400个token，在很多工作负载下甚至接近500。

But for open weight, when you're running it, every provider can offer potentially even 10 different levels of speed going from the slowest mode, which can be a lot cheaper, to 400 tokens per second, almost up to 500 in many cases for some workloads.

这通常比现有的快速模式快2到3倍。

And this is typically 2x or 3x faster than the fast mode out there today.

所以这种在性能上的精细控制，更不用说对客户如何与模型交互的控制，以及对数据保留的控制了。

So this kind of level control even in terms of performance and then let alone control over how customers are interacting with the model, control over data retention.

请记住，Fable没有零数据保留政策，至少很多数据需要留在那里，这样能更好地控制安全性和合规性。

Keep in mind Fable doesn't have zero data retention policy and at least a lot of the data need to be staying there and let it control security and compliance a lot more.

是的，这就是我特别看好K3的原因，不仅仅是从成本角度，更多的是因为它把这种水平的智能带给了人们可以拥有的东西。

Yeah, this is why I'm particularly excited about K3, not just from a cost perspective, but a lot more on bringing this level of intelligence to something people can own.

在调整速度和其他参数方面，后端需要做些什么？你看到用户是怎么做的？谁在这方面特别聪明？

In terms of calibrating things like speed, calibrating other things, just sort of on the back end, what needs to happen and and kind of what are you seeing your users do and, like like, who who is being really clever about this?

我们看到，当用户启用快速模式时，他们能从这个模型中获得最大的收益。

So we do see users are able to get the maximum benefit out of this model when they enable fast mode.

我这里说的当然是vLLM的快速模式，能达到每秒400到500个token，这真的是一个巨大的飞跃，尤其是当开发者与模型交互时，他们会发现，哦，我真的可以更快地完成任务了。

Like what I'm talking about here, of course, is vLLM's own fast mode getting up to four hundred and five hundred tokens per second because it is really a big step change from like, especially when developer interacting with the model, they can see, oh, I can really just get my task done faster here.

模型不再卡在同步上，而是直接执行、执行，与环境交互。

And the model are no longer stuck in syncing, rather it is just executing, executing, interacting with the environment.

所以对于高级开发者来说，专注于任务时，我们发现这非常有益。

So for premium developer blocking focus task, we're seeing it's very benefiting.

而且K3还允许人们修改和微调它，让他们能够针对自己的工作负载进行优化。

But also K3 are just be able to have the ability for people to modify it and fine tune on top of it, allow them to make it better for their own workload.

这确实已经在发生了。

And this is definitely happening today as well.

你能解释一下最近的开源模型与过去的相比，许可条款是什么吗？还有，你觉得他们为什么这么做？

Can can you just explain what the licensing term is for the most recent open source models compared to the past and and and, like, why you think they're doing that?

哦，对。

Oh, yeah.

所以从历史上看，开源模型就像是我们送给世界的礼物，你可以拿去用、修改，想怎么处理都行。

So historically, the open way model are just, like, a pass you to, like, our software, which is like take it, modify it, do whatever you want with it, here is a gift to the world.

但最近，模型实验室在想办法找到一种经济上可持续的方式来资助他们自己的模型开发。

And then recently the model lab are trying to understand a way to economically fund their own model development.

毕竟，训练、研究和数据都非常非常昂贵。

After all, training and research and the data are very, very expensive.

所以我们开始看到一些条款，甚至追溯到Llama时代，当时Meta发布Llama时，他们就有规定：如果你的日活跃用户或年度经常性收入超过某个阈值，就需要与Meta签订商业协议。

So we have been starting to see terms even to the LAMA days for when Meta was releasing LAMA, they do have a term of if your daily active user or annual recurring revenue exceeds some threshold, please enter into a commercial agreement with Meta specifically.

对吧？

Right?

我记得当时那些数字是特意选定的，你还能找到。

I do remember that the numbers were, like, specifically chosen at that time that you could go find.

当时全世界大概只有两家公司符合他们许可证中排除的定义。

It was, like, two companies in the world that, like, fit the definition that they had excluded from their licenses.

对，没错。

Yeah exactly.

但大家已经从中得到了启发，尤其是现在各个实验室都在想办法让开源模型在经济上可持续。特别是当他们开源一个模型，每个人都可以直接拿去自己跑，很多时候就没人再用他们的API了，而他们的API目前还在发展初期，对吧？

But like people have taken a hint from that, especially now the labs are trying to figure out a way to economically fund it, especially when they open source a model, everybody can just take it and run it themselves, whereas nobody will use their API anymore in many cases, while their API currently still taking up shape, right?

现在我们看到生态系统发展得很健康，从最近的Minimax发布M2.7模型开始，他们就有一个专门针对使用量的条款。

And now we're seeing a very healthy ecosystem development starting from even Minimax recently when they're releasing the M2.7 model, they have a term specifically focusing on usage.

Kimi最初也有类似的规定，比如如果你有衍生作品，就像当时Fireworks和Cursor基于Kimi模型构建那样，这在当时是个大新闻。

And Kimi initially also has like if you have derivative derivative works, like this is kind of big news back then with Fireworks and Cursor about how they build on top of Kimi model.

而且，如果我可以稍微补充一下，我觉得这并非贪婪，至少从我看到的开源模型实验室的情况来看。

And and it's if I I I if I can just expand on that a little, like, I I I don't think it's greed, at least what I've seen from open source model labs.

对吧？

Right?

是的。

Yeah.

开源模型，实际上我们说的是开放权重。

Open source models, really what we're talking about are open weights.

对。

Yeah.

对吧？

Right?

而且这不仅仅是软件。

And and it's just not software.

AI模型不是软件。所以，开源软件过去是靠人们贡献时间，或者大公司让员工贡献时间来支持的。

An AI model is not software at And the end of the so open source software used to be supported by people donating their time, or big companies authorizing their employees to donate their time.

所以，这有点像人们以实物形式捐赠时间。

So, it was sort of like a bulk, in kind donation of people's time.

这在AI领域真的行不通。

That really doesn't work at AI.

是的。

Yep.

对吧？

Right?

我总不能晚上回家，跟朋友一起随便训练一个前沿的开源模型吧。

I can't just go home at night and train a frontier open source model with friends for fun.

我们需要数百万甚至数十亿美元的计算资源。所以，为了做到这一点，我认为这确实支持了你的观点，显然需要有经济激励和资金机制。

We need millions or billions of dollars of computing resources enabled So, in order to do I think it does support your point that, obviously, there need to be economic incentives, and there need to be funding mechanisms in place.

坦白说，我认为中国模型比国内生产的模型更需要这些。

Frankly, I think even more so with Chinese models than with domestically produced models.

如果月之暗面没有资金来源继续训练模型。

If there's no source of funding for Moonshot to continue to train models.

我们知道资金会从哪里来，那不是我们能控制的。

We know where the funding will come from instead, and it's not something we write.

是政府，而且我认为那些对我们来说更糟糕。

It's government and things that are actually worse for us, I think.

所以我觉得你提出了一个有趣的观点，这是一个重要的经济结构，我认为这意味着未来我们会看到更多这样的情况。

So I think you raised sort of an interesting point that this is an important economic structure, and I think this means we'll see more of this in the future.

你同意吗？

Would you agree with that?

是的，归根结底还是可持续性的问题。

Yeah, it's really about sustainability in the end.

关键在于，如何确保所有这些初始的资本投入——几乎就是反复训练模型、失败再训练——最终能够收回成本，并且让所有相关方都有足够的信心和资金去进行下一轮。

It's about how do you make sure that all this initial capex almost to train the model fail again and again and train the model again, how do you really pay it back and how do you make sure that there's enough confidence and funding from everybody involved to go to do the next one?

我最近听到有人拿这个和制药行业做类比。

I recently heard heard someone someone making an analogy to this, to the pharmaceutical industry.

就像如何确保新药研发过程得到充分资助，并且有一种可持续的方式，让人们愿意承担巨大风险、下大赌注去研究新药，因为他们知道最终会有经济回报——当新药上市后，除了分销渠道之外，一部分收入会回流，继续资助下一轮的研发。

It's almost like how do you make sure that the R and D process of new drugs are properly funded and is proper sustainable method to making sure that people are willing to take big risk, big bet, to go to do research for new drugs and then later because they know there's an economic incentive in the end when the new drug released to the market, a portion of those, of course, like besides just distribution channels, a portion of those revenue will flow back to continue to fund the next R and D effort.

我们现在在模型开发上看到的，也差不多是这个情况。

And this is where we're kinda seeing similar to the model development now.

对。

Yeah.

这真是

That's a

一个很有意思的类比，因为就像一旦一种药物、一个分子上市，你就有了最强的控制力，那就是其他任何人都无法生产它。

really interesting analogy because it's like once a once a drug, a molecule is released, you have the strongest possible control, which is nobody else can manufacture it at all.

对吧？

Right?

这就像是最封闭的源代码。

It's like the most closed possible source.

对吧？

Right?

就像是个秘密。

Like, it's like a secret.

但是，是的，对于模型，尤其是开源模型，你知道，一旦发布出去，任何人都可以使用、扩展它等等。

But but, yeah, like, in the case of models, especially open source models, you know, once it's out there, anybody can take it, use it, extend it, etcetera.

所以，是的。

So so having yeah.

所以，给它附加一些经济机制可能确实有意义。

So having some economics attached to it probably does make sense.

对。

Yeah.

对。

Yeah.

嗯，实际上，这也让我想到一个问题，马特，你之前提到过开源模型和开源软件维护的动态很不一样。

Well, actually, that that raises a a question for me too, which, Matt, you were alluding to this earlier about how different open source models are from the dynamics of open source software maintenance.

对于开源AI来说，到底需要维护什么？

When it comes to open source AI, what actually needs to be maintained?

是它周围的基础设施吗？

Is it the infrastructure around it?

模型本身需要维护吗？

Do the models themselves need maintenance at all?

这些动态到底是怎样的？

Just kind of what what are those dynamics?

因为我觉得即使是开发者围绕它的行为也很不一样。

Because I think even the the developer behavior around it is is pretty different.

对。

Yeah.

我是说，西蒙已经说过这个了，但也许我再稍微展开一下，就是，你知道，你看到一次大规模训练的结果，现在训练指的是预训练，然后，你知道，强化学习，在训练中或训练后，或者预训练、SFT、RL这些。

I mean, Simon sort of said this already, but maybe maybe I'll just expand a bit, which is, you know, you you see the results of of a big training run, you know, where training now means it's pre training and then and then, you know, RL kind of, you know, mid training or sort of post training on these things, or, you know, pre training SFTRL.

对吧？

Right?

就像，这是一个复杂的流程。

Like, it's for a complicated pipeline.

我们只看到最终的结果。

We see only the result Yeah.

就是最后的结果。

Of this at the end.

而且数字很大。

And the numbers are big.

你会想，哦，这是一次一亿美元的训练。

You're like, oh, you know, this was a $100,000,000 training run.

但你常常忘记的是，可能之前已经有五次失败的大规模训练，才走到现在这一步。

But what you often forget is, like, there may have been five failed training run, you know, large scale failed training runs before you even get to now.

是的。

Yeah.

真是血汗和泪水啊。

The blood, sweat, and tears.

对。

Yeah.

对。

Yeah.

我最喜欢的一个例子，也许我们可以找到链接，就是早期的LAMA模型之一。

One one of my favorite artifacts, maybe we could even track down the link, is one of the early LAMA models.

对。

Yeah.

他们发布了整个对话记录，就是那些照看训练集群的人之间的对话，在模型训练的时候。

They published the whole, like, conversation log between the or between the people who are babysitting the training clusters, like, while the models were training.

是的。

Yep.

而且真的很好笑。

And it's it's so funny.

就像，哦不。

It's just like, oh, no.

一切都出问题了。

Everything's gone wrong.

一片混乱。

Like, chaos.

大家都很慌张。

Like, panic.

对。

Yeah.

然后，下一条评论就是，好的。

Then, like, then the next comment's like, okay.

我们解决了。

We solved it.

一切都没问题。

Everything's okay.

你知道，集群起来了，损失在下降。

You know, clusters up, losses, going down.

所以这些模型发布之前，幕后有很多很多很多工作要做。

So there's there's a lot, a lot, a lot that goes in behind the scenes before these models get released.

一旦它们发布出去，我想那就更接近你的领域了，你要确保它能够实际运行起来。

Know, once they're out there, I guess it's a little bit more in your in your zone, you know, to to kinda make sure it's, like, operationalized.

哦，对。

Oh, yeah.

但这也是一个非常有趣的点，一旦模型发布，整个社区都要努力去运行它，因为模型是在特定的硬件和特定的架构上训练的。

But but this this is also a very interesting point once it's out there is a whole community effort trying to operate this model because the model is trained on a given type of hardware and given type of architecture.

但当它真正投入使用，每个人的集群拓扑和用例都不一样。

But when it's out in the wild, everybody has different cluster topology and use cases.

关键是如何把一个用例变成几乎无限的用例。

And it's about how do you turn like a use case of one now to a use case of almost infinity.

现在有人试图把它适配到边缘设备上，也有人试图以最大规模运行它，适配它，确保它既能用于语音代理，也能用于编码代理，这些是完全不同的使用场景。

Now you have people trying to adapt it to the edge devices and people trying to run at largest scale ever, adapt it, making sure it runs for voice agent but also for coding agent which are entirely different kind of use cases.

所以这是一个整个社区的努力，试图进一步优化、专业化，并确保其运行可靠，并且能够持续优化。

So this is a whole community effort trying to further optimize specialized and making sure the running of it is reliable and continue to be able to optimize against it.

所以这需要整个开源社区的努力来确保它不断改进。

So that's a whole village later throughout the open source to make sure it's improved.

对。

Yeah.

这很酷，因为这真的就是开源软件。

And that's cool because that really is like open source software.

我的意思是，这就是你做的。

Mean, and this is what you do.

但是，你知道，就像这样，任何人都可以贡献。对。

But, you know, like, this is like anybody can contribute Yeah.

对。

Yeah.

而且，就像，让它们变得更好。

And and, like, make these better.

我有个有趣的思想实验，如果GPU价格下降99%，你知道的。

My the the fun thought experiment is if GPUs, you know, dropped in price by 99%.

对吧？

Right?

如果基于GPU的计算真的变得便宜又普及，那我们是不是就能回到一个真正的开源世界？

Like like, if if if GPU based compute actually became, you know, kind of cheap and widely available, like, then do we get back to a kind of a real open source world Mhmm.

在那里，一个人坐在家里，或者一百个人利用空闲时间，就能想出新的东西，尝试那些大公司排队等着做的模型训练路径，然后真正地共同推动领域发展。

Where, you know, one person sitting in their basement or a 100 people working their free time can, like, come up with something new, try many of these sort of model training paths that, you know, that, like, are in the queue somewhere at one of the big companies and, you know, kinda kinda see you know, really expand and advance the field collectively.

对。

Yeah.

嗯，这跟你一开始说的有关。

Well, I mean, this relates to what you were talking about at the beginning.

就像，一开始，你需要的计算量在所谓的“前沿”是微不足道的，现在却变得巨大无比。

It's like, you know, at the beginning, the amount of compute you needed to be at the, quote, unquote, frontier was negligible, and now it's just like it's it's enormous.

那你怎么才能再次达到消费者级别的同等水平呢？

And how do you how do you get that, you know, back to consumer parity again?

我之前在播客里说过，但我会继续说。

I've I've said this on the podcast before, but I'll keep saying it.

AlexNet，就是那个我们关心的在GPU上运行的神经网络，当时是在两块GPU上运行的。

AlexNet, you know, kind of, like, neural network to run on on GPUs that we care about ran on two GPUs.

这不是说少了小数点或逗号，就是实实在在的两块。

And that's not that's not like there are no missing decimal points or commas in there.

就是两块。

Literally two.

对。

Yeah.

现在两块GPU根本什么都做不了。

Now that would get you literally nowhere.

对。

Yeah.

是的。

Yep.

所以我想这跟我们收到的另一个问题有关，那就是过去十八个月里，推理变得越来越难，因为规模、多样性、我们拥有的模型种类，当然还有那些执行越来越长时间任务的智能体，这些因素加在一起。

So I guess this this relates to to another question that we've had, which is which is inference has gotten harder over the past eighteen months because of this combination of scale and diversity and the kinds of models that we have and, of course, like agents that are doing increasingly long running tasks.

那么，在这种情况下，开源为什么不仅仅是锦上添花，而是绝对必要的呢？

So what makes open source in this world and in this scenario not just like nice, a nice thing to have, but absolutely necessary?

对。

Yeah.

规模来自几个方面。

So scale comes from a few points.

一方面取决于你能不能把这种巨型模型放在数据中心的一个机架上运行。

It comes from whether or not you can run this gigantic model on a data center rack.

但另一方面也取决于你能不能把它优化到极致，优化到光速，从而从中获得最大的价值。

And but also it comes from, can you optimize it to the extreme point, to the speed of light, so that you are getting the most value out of it?

但还需要整个社区、合作和伙伴们的共同努力来验证它，确保没有那些只有0.0001概率出现的最后残留的bug。

But also you need a whole community and collaboration and partners of effort to validate it and making sure that there is no last remaining bugs that appear like 0.0001 of the time.

所以，扩大部署规模，让更多人以最大的规模运行它，这样才能确保每个人运行这个模型的体验都更好。

So this is where scaling up the deployment and then making sure more people are running it at largest footprint possible we'll ensure everybody's experience of running this model is better.

这就是为什么开源推理目前是主流方式，而不是闭源的推理引擎。

And this why open source inference is the current leading way right now instead of a closed source inference engine.

坦白说，抱歉，

And frankly, a sorry,

很多

a lot

现在的推理云和API服务，其实底层都在使用和利用开源推理引擎。

of the inference cloud and API as a service today do use and leverage open source inference engine under the hood.

之所以这么做，是因为有太多经过实战检验的配方和经验，他们可以直接在此基础上构建。

And the reason to do that is just there's just so much battle tested and learned recipes and things that they can build on top of.

这就是为什么开源是目前运行模型的主流方式。

And this is where open source is the current leading way to run models.

我觉得有点好笑的是，回顾一下不只是vLLM和Inferact的历史，还有像OpenRouter甚至Ollama这样的公司，这些不同的团队差不多都是在2022年和2023年起步的。

Something that I find kind of funny in, like, looking back at not just the history of vLLM and Inferact, but also company like OpenRouter or even Olama, all of these different teams kind of got started around 2022 and 2023.