为什么扩散模型将在AI推理中胜出：与Inception联合创始人兼首席执行官Stefano Ermon

嗨，听众朋友们。

Hi, listeners.

欢迎回到《No Priors》。

Welcome back to No Priors.

今天，我请到了斯坦福大学的资深教授Stefano Ermon，他现在也是Inception公司的联合创始人兼CEO。

Today, I'm here with Stefano Ermon, who is a longtime Stanford professor and now co founder and CEO of Inception.

Stefano在生成模型领域的研究非常广泛，但尤其以扩散模型的奠基人之一而闻名。

Stefano has a extraordinarily broad body of work around generative modeling, but is especially well known as one of the fathers of diffusion.

我们聊到了他的公司如何挑战那些大型实验室，以及为什么未来几年速度和效率会成为AI领域的关键。

We talk about his company challenging the large labs and why speed and efficiency are going to be the name of the game in AI over the next few years.

欢迎你，Stefano。

Welcome Stefano.

Stefano，非常感谢你来做客。

Stefano, thanks so much for being here.

很高兴来到这里。

Great to be here.

我想先聊聊你的研究背景，以及你是怎么开始创办公司的。

I would love for us to just start with a little bit of your research background and how you ended up starting your company.

当然可以，好的。

For sure, yeah.

我基本上整个职业生涯都在做生成模型的研究。

I've been doing research in generative models for like basically my entire career.

我2014年以助理教授的身份加入斯坦福，当时就在研究构建生成模型。

I started at Stanford in 2014 as an assistant professor, I was working on building generative models.

那时候，这个研究领域并不是特别热门。

Back then, the the research area was not particularly hot.

你知道，模型效果也不太好。

You know, the the the models were not quite working well.

我们还在用MNIST数据集构建一些小的生成模型，如果能生成那些模糊的数字图像，就算很大的成功了。

We were still, like, building little generative models over MNIST, and it was, like, a big success if you could generate these grainy images of digits.

而且那时候在这个领域发表论文都很难，你得证明训练生成模型是为了从无标签数据中学习特征，然后可能帮助你在监督学习上做得更好，因为那是当时大家最关心的事情。

And it was even hard to publish papers back then on that topic, and you had to justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because that was the thing that everybody cared about.

但后来，事情当然就发生了翻天覆地的变化。

But then things took over, of course.

所以，我算是天时地利人和，正好在做对的事情。

And so it was I was at the right place at the right time working on the right thing.

所以基本上，我从一开始就在这个领域做研究。

And so I've been doing research in that space since the beginning, basically.

除了对这个领域的好奇心之外，你在2014年、2015年的时候，有没有个人对模型能做什么的期望？

Did you have like a, besides a curiosity in the area, a personal hope for what the models would do back in 2014 and 'fifteen?

是的，我一直觉得那是从无标签数据中学习的正确方式，构建生成模型确实是确保你理解数据结构的正确方法。

Yeah, I mean, I always felt like that was going to be the that was the right way to think about learning from unlabeled data, that building a generative model is really the right way to make sure you understand the structure in the data.

那是我一直以来的思路。

That was kind of like the way I was getting at.

我甚至做梦都没想到今天这些大语言模型能有这样的能力。

I was not even dreaming about the kind of capabilities that these LLMs that we have today could do that.

但我更多是从世界模型的角度来思考的。

But I was thinking more from a, I think, world models perspective.

比如，我当时花了很多时间在图像上，所以就在想，好吧。

Like, I was working a lot on images and so thinking about, okay.

就像，我有一个世界模型。

Like, I have a world model.

我能想象如果我站起来走出门会发生什么。

I can imagine what's gonna happen if I were to stand up and walk out the door.

我可以在脑海里大致描绘出那个画面，而这对做决策和模型预测控制很重要，因为拥有这种世界模型需要一定的生成能力。

Like, I can kinda, like, picture that in my mind, and that's important to make decisions and kinda model predictive control when having this kind of model of the world requires some generative capabilities.

所以我一直觉得，对，这就是该研究的方向。

And so I always felt like, okay, that's the right direction to work on.

我觉得这个问题会非常难。

I felt like this is going to be very hard as a problem.

所以这会让我忙活一辈子，是个值得研究的好问题。

So this is going to keep me busy for my whole career, and so it's a good problem to work on.

结果当然是我大错特错，事情发展得比我预想的快多了。

And then, of course, I was very wrong and things evolved much faster than I was expecting.

对，我觉得这其实是个普遍现象。

Yeah, I think that's kind of universally true, though.

那你给我讲讲你的研究进展，以及它是怎么促使你创办这家公司的。

And sort of walk me through the state of your research and how that led you to start the company.

好的，我当时在研究图像生成模型，最开始做自回归模型，那些模型速度很慢，而且生成出来的图像很模糊，后来VAE和GAN出现了。

Yeah, so I was working on generative models of images, initially working on autoregressive models, which were very slow and kind of like very blurry, and then VAEs and then GANs took over.

是的。

Yes.

那时候，我们对图像生成模型的状态非常不满意。

And back then, we were very unhappy with the state of generative models for images.

比如GAN虽然能用，但训练起来非常不稳定，结果也很难复现。

Like, the GANs were they they they worked, but they were very unstable to train, very hard to reproduce results.

所以我们想看看，有没有办法做出一个和GAN效果一样好，但更有原理性的东西。

So And we were trying to see, is there a way to build something that is as good as a GAN, but it's more principled?

于是我们从2019年开始研究基于分数的生成模型，这基本上就是后来扩散模型的前身，当时是和我的博士生杨松一起做的。

And so we started working on score based generative models, which are basically what eventually became Diffusion Models back in 2019 with my PhD student, Yang Song.

所以我们想出了一个主意，就是训练一个神经网络来给图像去噪。

And so we came up with this idea of, let's train a neural network to denoise images.

如果你能成功给图像去噪，那就说明你对图像的结构理解得足够透彻，基于这些去噪器构建一个生成流程是可行的。

And if you can denoise an image, then you really are understanding enough about the structure of the image that it should be possible to build a generative procedure based on these denoisers.

这基本上就成了现代扩散模型的核心技术。它不是从左到右一个像素一个像素地生成图像，而是从纯噪声开始，逐步细化，直到最后得到一张清晰的图片。

And that basically became the underlying technology of modern diffusion models, where instead of generating images left to right, one pixel at a time, you start from pure noise and then you gradually refine the object until you get a clean picture at the end.

这个想法是2019年在我实验室里开始的。

And that started back in 2019 in my lab.

后来它就在这个领域占据了主导地位。

And then it took over the space.

即使到今天，最好的图像生成、视频生成、某种程度上音乐生成，还有不少蛋白质相关的模型，都是基于扩散模型的。

And even today, the best models for image generation, video generation, music to some extent, a lot of the protein stuff, they are based on diffusion.

所以我的团队在多种扩散模型上做了很多工作，包括加速生成样本的技术，以及提升这些模型的质量。

And so my group has worked a lot on various kinds of diffusion models, technique for accelerating them to generate samples very quickly, to improve the quality of these models.

既然我们能让扩散模型在图像上奏效，我就开始思考，怎么让扩散模型也适用于文本、代码生成以及离散对象呢？

And so since we were able to get them to work on images, I started thinking about, okay, how do we get Diffusion models to work on text or code generation and discrete objects?

有没有办法超越自回归模型，实现更并行、自带纠错能力的模型呢？

Is there a way to move beyond autoregressive models to something that it's more parallel, with built in error correction?

所以我在斯坦福做了很多研究，想让扩散模型也能用于文本和代码生成。

And so I've been doing a bunch of research at Stanford on getting Diffusion Models to work on text and code generation.

我们在2024年取得了突破。

We had a breakthrough in 2024.

我们发表了一篇论文，基本上首次证明了在GPT-2规模上，扩散模型可以达到自回归模型的质量。

We published a paper basically showing that for the first time, it was possible to match the quality of an autoregressive model at the GPT-two scale.

也就是不到十亿参数，还算是学术层面的。

So less than a billion parameters, still fairly academic.

但我们基本上还是训练了一个Transformer模型。

But we were able to train, basically, still a transformer model.

作为扩散模型，在同样的数据上，我们能够匹配质量，也就是同样的困惑度。

As a diffusion model on the same data, we were able to match the quality, like the same perplexity.

而且你拟合数据的效果，和相同参数量的自回归模型一样好。

And you were fitting the data just as well as an autoregressive model with the same number of parameters.

但扩散模型的速度要快得多。

But the Diffusion model was significantly faster.

因为它是扩散模型，因为你可以同时输出很多词元，我们生成文本的速度比自回归模型快了大约10倍。

Because it's Diffusion, because you're outputting many tokens at the same time, we were able to generate text like 10x faster compared to the Autoregressive model.

所以这让人感到非常非常兴奋。

And so that felt very, very exciting.

我真的很想看看，如果扩大规模，训练更大的模型，会发生什么。

And I really wanted to see what happens if you scale up, if you train bigger models.

于是，我创办了Inception公司，目的就是扩大这项技术的规模，尝试构建商业规模的基于扩散的语言模型。

And so I started Inception, a company to basically scale up the technology and try to build commercial scale diffusion based language models.

现在大家都见过扩散模型的输出，尤其是图像。

Everyone has now seen the outputs of diffusion models in particular images.

而且我认为，扩散模型生成的短视频正日益成为一种主导形式，在世界其他地方，它已经是一种主要的娱乐形式，在这里很可能也会如此。

And I would argue that it's increasingly a dominant form of like generated short form video from diffusion models is like a dominant form of entertainment in other parts of the world and it will likely become so here.

这简直令人难以置信，至少对我来说是这样，即使我关注这个领域已经超过十年，今天能达到的质量水平依然让我惊叹。

It's kind of unbelievable, at least to me, even having followed the field for, the last decade plus the quality that is possible today.

所以我觉得这挺明显的，对吧？

So I think that is kind of obvious, right?

而且图像和视频生成这个应用场景太大了，以至于大家甚至都在开发专门的硬件来提升性能。

And it's such a big use case in images and video generation that folks are even creating, you know, hardware to support better performance here.

但要说这种方法在其他领域也能行得通，或者说这是对当前那些专注于Transformer架构的AGI实验室的一个有竞争力的方向，这就不那么直观了。

It's not intuitive that would work for other fields or that, you know, this is a interesting competitive direction to the current, you know, full transformer focus like AGI Labs.

你能解释一下这其中的原理吗？

Can you offer some intuition on that?

从研究者的角度来看，现在的世界状态非常有趣，因为主要有两种范式，两种构建生成模型的方法：一种是自回归，就是模型预测下一个token或像素，然后从左到右逐个生成。

So it's a very interesting kind of like state of the world right now from a researcher perspective, because like there is like two main paradigms, two ways of building generative models that is autoregressive, where you you kind of like have a model that predicts the next token or the next pixel, and then you generate left to right, one token at a time.

另一种是扩散模型，也就是从粗到细的生成，通过迭代去噪来生成。

And then there's Diffusion, which is coarse to fine generation, iterative denoising generation.

正如你所说，在连续模态中，扩散模型占主导地位。

And as you said, we have Continuous Modalities where Diffusion dominate.

而在离散模态，比如文本和代码，所有大型实验室主要都押注在同一个架构上，也就是自回归模型。

There is Discrete Modalities, Text and Code, where primarily all the big labs are betting on the same architecture, autoregressive models.

而且随着我们越来越走向多模态模型，有一种想法是，也许我们会有一个能处理所有模态的模型，并且对世界无所不知。

And as we move towards more and more multimodal models and there is this idea that maybe we'll have a model that can handle all modalities and will know everything about the world.

那这个模型会是什么架构呢？

What architecture will that be?

会是自回归模型吗？

Like, will it be an autoregressive model?

还是扩散模型？

Will it be a diffusion model?

没人知道。

Nobody knows.

我觉得现在下结论还为时过早。

I think that I think the jury is still out there.

在Inception，我们押注扩散模型，因为我们相信最终重要的是推理时的扩展能力。

At Inception, we're betting on diffusion models because we believe that what matters eventually will be inference time scaling.

而且从根本原因上讲，扩散模型在推理时确实比自回归模型更有优势。

And there are fundamental reasons for why diffusion models are better than autoregressive models at inference time.

所以即使你回顾自回归模型的发展历程，2017年是一个转折点，那时人们从RNN转向了Transformer。

So even if you think about the story of autoregressive models, there was an inflection point in 2017 when people switched from RNNs to transformers.

那是什么原因呢？

And was that?

问题在于RNN本质上必须逐个处理token，一次只能处理一个。

The problem was that RNNs had to essentially process tokens sequentially, one at a time.

训练速度非常慢。

And training was very slow.

所以人们想出了这个主意，设计一种架构，让你能同时并行处理多个token。

And so people came up with this idea of, let's have an architecture that allows you to process many tokens at the same time in parallel.

那就是Transformer。

And that was the transformer.

它在训练方面扩展性更好，也为LLM的许多成功奠定了基础。

And that was the thing that scaled better for training and that enabled a lot of the successes behind LLMs.

但如果你考虑推理，不是训练，而是推理生成，自回归模型仍然是顺序处理的。

But if you think about inference, now not training inference generation, autoregressive models are still sequential.

计算是从左到右，一次只处理一个词。

The computation is one left to right, one token at a time.

你没法生成第十个词，除非你先生成它前面的所有词。

You cannot generate the tenth token until you've generated everything that comes before it.

这种工作负载在GPU上跑起来效率不高。

That kind of workload does not map well to GPUs.

这种工作负载非常受内存限制。

That kind of workload is extremely memory bound.

你大部分时间都在内存层级里搬运权重，而真正做的算术运算很少。

You're spending most of your time moving around weights across the memory hierarchy, and you're doing very little arithmetic.

这是自回归模型的一个根本性问题。

And that's a fundamental problem of autoregressive models.

那么，对应的替代方案是什么呢？

And so what's the equivalent?

如果你想想RNN、Transformer、自回归模型，在推理阶段对应的就是扩散模型。

If you think RNNs, transformers, autoregressive models, the equivalent at inference time is a diffusion based is a diffusion model.

因为扩散模型的设计初衷就是，在推理时，可以同时处理很多token。

Because a diffusion model is built to have, at inference time, a workload where you process many tokens at the same time.

所以扩散模型在推理时的工作负载，其实和训练时非常非常相似，都是并行处理大量token。

And so the workload that we have at inference time in a Diffusion model, it's basically very, very similar to the workload you have for training, where you're processing many tokens at the same time in parallel.

所以它的架构本质上就是为了让推理负载能很好地映射到GPU上，也就是GPU特别擅长的那种计算方式。

And so it's built to essentially have an inference workload that maps really, really well to map models, but the kind of things GPUs do really, really well.

所以我们押注于构建那种在推理时扩展性最好的架构和模型，因为经济性主要取决于模型每瓦特能带来的智能，以及每美元能带来的智能。

And so we bet on trying to build the architecture and trying to build the kind of models that will scale best at inference time because economics are dominated by the kind of intelligence per watt, the intelligence per dollar that you're able to get from the models.

想想推理模型取得的很多进步，很大一部分在于扩展测试时的计算量。

If you think about a lot of the advances with the reasoning models, a lot of it is scaling test and compute.

所以如果能在这个维度上更好地扩展，也会很重要。

And so being able to scale better along that axis will also matter.

即使考虑强化学习后训练，很多瓶颈也在于生成rollouts，也就是让模型探索，然后给轨迹打分，再根据发现的东西改进模型。

And even if you think about RL post training, a lot of the bottleneck is generating rollouts, like letting the model explore and then scoring the trajectories and then improving the model based on the kind of things it finds.

所以推理再次成为我们后训练的关键瓶颈。

And so inference is, again, a key bottleneck for our outpost training.

所以，如果你的模型在推理时扩展性更好，那么在后训练阶段自然也会获得更好的扩展效果。

And so if you have a model that scales better at inference time, then automatically you're gonna get better scaling during our output training.

这就是我们决定押注基于扩散的LLM的原因，因为它本质上更并行。

And so that's why we decided to bet on a diffusion based LLM because it's inherently more parallel.

而苦涩的教训是，更并行的解决方案最终会胜出。

And the bitter lesson is that the more parallel solution is the one that is eventually going to win.

你是怎么考虑适用性的，或者你做了哪些实验来解决离散和连续模态之间的难题？

How did you think about applicability or what experiments did you run-in terms of cracking the nut on discrete versus continuous modalities?

因为我觉得人们也通过新的分词方法或让视频和语音工作的方式，塑造了现有的主导范式。

Because I think people have also shaped the existing dominant paradigm through new tokenization efforts or methods to make video and voice work, for example.

你并不在同样的基于标记的范式里。

You're not in the same token oriented paradigm.

你在这里是怎么让它工作的？

How do you make it work here?

是的，所以有很多研究致力于弄清楚如何应用一种本质上与连续数据结构紧密相关的技术。

Yeah, so there was a lot of research that went into figuring out how to apply a technology that was inherently very tied to continuous structuring the data.

所以如果你想想扩散模型，它是在学习如何给图像去噪。

So if you think about a diffusion model, it's learning how to denoise images.

这对连续数据来说是合理的。

And it makes sense for continuous data.

因为即使你考虑两个像素的颜色，你也可以在它们之间进行插值，而且仍然有意义。

Because if you think about even two pixel colors, you can interpolate between them and it will still make sense.

但如果你考虑两个词，它们之间不一定有中间的东西。

But if you think about two words, there is not necessarily something in between them.

它们都是离散的。

It's all discrete.

所以这需要大量的研发和新的科学，必须发展出来，才能把这些想法扩展到离散空间。

And so it required a lot of R and D and a new science that had to be developed to figure out how to extend those kind of ideas to discrete spaces.

你今天能说它效果有多好吗？

What can you claim about how well it works today?

我们认为它效果非常好。

We think it works really well.

我们已经成功训练出基于扩散模型的大语言模型，在质量上与前沿实验室的速度优化模型不相上下。

So we've been able to train diffusion based LLMs that are comparable in quality with the speed optimized models for Frontier Labs.

我们的Mercury模型在基准测试中与OpenAI的HICU模型、flash模型、mini nano模型相当，但速度要快得多。

So our Mercury models are on par with the HICU models, flash models, mini nano models from OpenAI, if you look at benchmarks, while being significantly faster.

所以我觉得我们已经从纯粹的研究原型跨越到了实际应用阶段。

So we've crossed, I think, the the we went from pure research prototypes to things that are actually used.

我们现在已经在生产环境中部署这些模型了。

We are serving these models in production today.

我们完成了所有工作，包括如何构建一个服务引擎。

We did all the work of figuring out how to even just build a serving engine.

你不能在LLM或SGLAN上运行这种基于扩散模型的大语言模型。

You cannot run this diffusion based LLMs on the LLM or SGLAN.

我们必须自己构建服务引擎，并且能够处理真实生产工作负载的复杂性。

We have to build our own serving engine, and we can handle a lot of the complexity of real production workloads.

我们已经解决了所有这些挑战，今天就能端到端地为真实客户提供这种新体验。

And we've solved all these challenges, and we can deliver this new experience end to end to real customers today.

其实，现在正好可以聊聊Inception这家公司目前的情况。

Actually, great time to just talk about where Inception is as a company.

比如，你们有多少人？

Like how many people?

你们实际在提供哪些服务？

What are you guys actually serving?

研究进展怎么样？

Sort of state of research?

我们公司成立大概两年了，大约50人，现在还是花很多时间在研发上，比如摸索怎么训练这些模型，怎么加速推理。

So we are about two years old, around 50 people, spending a lot of time still on R and D, kind of like figuring out what's the right way to train these models, how to accelerate inference.

说实话，这并不容易，就算你想想自回归模型，思路其实挺清楚的。

Like, it's not obvious how you even if you think about an autoregressive model, it's pretty clear.

那种模型能做的事情不多，对吧。

There's not a lot of things you can do there in terms of like, okay.

就是一次生成一个token，就这样。

You generate one token at a time, and that's it.

在基于扩散的模型里，我们知道推理时有很多不同的方式可以用计算量来换取质量。

In a diffusion based model, we know that there is a lot of different possibilities for trading compute for quality at inference time.

比如，就算你想想图像扩散模型或视频扩散模型，也有很多技术可以用来加速采样，比如蒸馏或者花哨的微分方程求解技巧，让你能非常非常快地从这些模型里采样。

Like, even if you think about image diffusion models or video diffusion models, there is a lot of techniques that you can use to accelerate sampling of distillation or fancy differential equation solving techniques that allow you to sample very, very quickly from these models.

所以，在训练、推理以及工程方面有很多研究要做，比如考虑数据混合、评估、强化学习后训练的基础设施。

And so there is a lot of research on the training, on the inference, and then engineering, just thinking about data mixes, evals, RL post training infrastructure.

要搞清楚怎么为这种新模型构建有效的配方，还有很多工作要做。

There is a lot of work that needs to happen to figure out how to build recipes that work for this new model.

我们尽量多利用现有的东西。

And we try to leverage existing things as much as possible.

比如说，它仍然是一个基于Transformer的模型。

For example, it's still a transformer based model.

所以你不必抛弃很多在好架构上已经完成的工作。

So you don't have to throw away a lot of the work that has been done on good architectures.

我们仍然使用注意力机制。

We still use attention.

我们仍然使用很多公开的数据集，还有人们创建的评估和基准。

We still use a lot of the public data sets that people have created and evils and benchmarks.

所以我们是个初创公司。

So we're start up.

我们尽量保持灵活。

We try to be scrappy.

我们尽量利用现有的东西，把精力集中在能产生最大影响、最能体现差异化的方面。

We try to use existing things as much as possible and focus the aspects where we can have the highest impact and then where we can be the most differentiated.

而现在，就是速度。

And right now it's speed.

未来嘛，谁知道呢？

In the future, who knows?

比如，基于扩散的语言模型可能会比自回归模型聪明得多。

Like, it's possible that a diffusion based language model will be maybe significantly more intelligent than an autoregressive one.

没人知道。

Nobody knows.

所以我觉得这非常令人兴奋，因为我们在开发这些非常强大的AI系统，但这一切都还很新鲜。

That's why I think this is very exciting because we're developing these really powerful AI systems, but it's all very fresh.

一切都还很新。

It's all very new.

我怀疑我们还没有发现构建这些系统的最佳方式。

I doubt we've discovered the best way of building the systems.

肯定还有别的选择。

There's got to be alternative.

肯定还有其他方法来构建这些模型。

There's got to be other ways of building these models.

而且最终，是的，效率会变得非常重要。

And eventually, yeah, efficiency will be very important.

如果你想想AI工厂，那会怎么运作呢？

If you think about the AI factory, how is that going to work?

我觉得没人真正知道。

I think nobody really knows.

而且能够在这个领域里探索，思考创造智能的其他方式，我觉得很令人兴奋。

And just being able to play in that space and thinking about alternative ways of creating intelligence, I think is exciting.

确实。

Absolutely.

而且我也觉得，在一个越来越从根本上受计算和供应限制的环境中。

And I also think that in a increasingly like fundamentally compute and supply constrained environment.

我不会说之前没有对模型效率的专注研究，但对很多工业研究来说，这更多是次要考虑，而不是纯粹在现有基础上追求能力扩展，对吧？

I wouldn't say that there wasn't very focused research on efficiency of models, but it was a second order sort of consideration for many of the industrial research efforts versus like pure capability scaling on what we've got, right?

以及在这方面的新方法。

And new methods in that.

但我只是觉得，如果你从根本上相信我们可以用掉所有拥有的算力，而且有经济上的理由这么做，那么对效率的关注度会大幅提升。你知道，现在很多人认为，AI领域的公司最重要的决策之一就是，我们如何在训练和最有价值的应用场景之间分配算力，对吧？

But I just think if you fundamentally believe that we can use all of the power we have and there will be economic reasons to do so, then I think the focus on efficiency is gonna go dramatically up, You know, I think many people now argue like one of the most important decisions for a company in the AI space is like, how do we use the power we have between, you know, training and the most valuable use cases for any what?

说到这个话题，比如在2026年9月，速度在哪些地方能胜出？

On that topic, like here in, you know, September '26, where does speed win?

就是说，现在哪些地方的人已经足够重视这个了？

Like where do people care enough about this already?

对，基本上就是那些对延迟敏感的应用。

Yeah, it is basically applications where latency matters.

我是说，一般来说，我觉得大家都关心速度，因为如果你能给我同样的质量但更快，人们总会选更快的方案。

I mean, generally speaking, I think everyone cares about speed in the sense that if you can give me the same quality but faster, people will always pick the faster solution.

我们也能看到，即使是前沿实验室的模型，推出的更快版本也是这样。

And we're seeing it with this faster versions of even the models from Frontier Labs.

人们愿意花更多的钱来获得更快的模型。

People are willing to pay more to get access to faster models.

而且我觉得一旦你习惯了快的模型，就很难再回去了。

And I think once you get used to a fast model, it's hard to go back.

就像宽带一样，对吧？

It's kind of like broadband, right?

然后它变得越来越快。

And then it gets faster and faster.

如果你能试一下，即使那些已经习惯快模型的人，也没法再回去了。

Then if you were able to try, even the people cannot go back once you get the fast model.

有没有哪些客户是你现在可以公开谈论的，他们确实很在意这个？

Are there customers that you can talk about publicly that care about this today?

有几个我们可以提一下，比如在语音领域，Open Call 就是一个例子。

There are a few that we can mention, like in the voice space, for example, Open Call is an example.

他们正在做语音代理这类产品。

They they are building like voice agents.

他们当然很看重速度。

They you know, speed, of course.

整个流程大概是这样的：先有 ASR 模型，然后有一个 LLM 负责调用工具、决定下一步说什么。

The the pipeline is like, you have an ASR model, you have an LLM that it's kinda like doing all the tool calls and figuring out what to say next.

通常需要推理型的 LLM 才能达到最高质量。

It has to be a reasoning LLM typically to to have the highest level of quality.

最后还有一个文本转语音的组件。

And then there is a text to speech component at the end.

速度对他们来说非常重要。

Speed matters a lot to them.

他们之前是用Cerebras来跑他们的LLM的。

They were previously using serving their LLMs on Cerebras.

所以他们用定制芯片来达到所需的速度，以便给客户提供最好的体验。

So they were using custom chips to get to the kind of speed that they need to deliver the best experience to their customers.

后来他们换成了我们的基于扩散的LLM，因为基本上能获得和在定制硬件上跑自回归模型一样的速度。

And then they switched over to our diffusion based LLMs, because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware.

如果有一个基于扩散的LLM，它是为并行而设计的，在软件层面加速，那你就能在NVIDIA GPU上获得同样的速度，这意味着可用性大大增加。

If you have a diffusion based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on NVIDIA GPUs, which means much more availability.

虽然GPU也很紧缺，但总比定制芯片多，而且成本更低，质量更高。

I mean, GPUs are scarce, but there's more of that than custom chips and lower costs, higher quality.

这就是语音领域的一个例子。

So that's an example in voice space.

对，我其实想问你，你怎么看这和硬件格局的互动？毕竟我们已经看到很多用例的需求，比如，是的，我想要一个大的、贵的、带大量SRAM的芯片，人们愿意为它的输出买单。

Yeah, I was actually gonna ask you how you think this interacts with the hardware landscape as well, given we've seen enough now demand from use cases where like, yes, I want a big expensive chip with a lot of SRAM and people will pay for the outputs of that.

在编程和其他用例中。

In coding and other use cases.

对，当然。

Yeah, for sure.

当然。

For sure.

而且我觉得硬件是加速的一种方式。

And I think hardware is one way to accelerate things.

好的一点是——

And the good thing

关于——软件可能更好，如果我们能……没错。

about- And software might be better if we can Exactly.

利用现有的——

Use the existing

而且它们其实是互补的。

And especially they're complementary.

最让人兴奋的是，从某种程度上说，软件带来的提升和硬件带来的提升是相乘的。

That's the exciting piece is that to some extent, the gains that you get from the software, they are multiplicative with the gains you get from the hardware.

也许有一天人们会开发出更适合你们正在构建的模型的硬件系统。

And maybe someday people develop a system, you know, hardware that fits even better the models that you're building.

当然。

For sure.

当然。

For sure.

如果我们把视野放宽到整个行业，我觉得有一种担忧，如果我理解错了请纠正我，那就是今天很难投资于新的架构，因为如果架构或方法上有进步，那些拥有资源扩大算力的大公司会直接吸收这些进步。

If we just like zoom out to the, you know, Inception in the broader industry, I think there is a vein of concern, and correct me if I'm wrong, that it's very hard to invest in new architectures today because some, if there are advances in architecture or methods, it will simply be absorbed by large players with the resources to scale compute.

跟我聊聊，在这种情况下，你作为大卫，是怎么考虑去竞争的？

Talk to me about how you think about going and competing as David in this situation.

是的，这是个非常合理的问题，也是我们一直在思考的事情。

Yeah, it's a very valid point and something that it's also top of mind for us.

我觉得对我们来说，一开始的护城河肯定是知识产权、商业秘密，以及我们和研究人员用来构建和优化这些模型的想法。

I think initially for us, for sure, the moat is sort of like the IP, the trade secrets, the ideas that we and our researchers have build these models and make them better.

随着我们公司逐渐成熟，我们不只是做纯研究，还开发了产品，有真实客户，从现实世界获得模型反馈，这样做的原因之一是，我们也在开发部署这些模型非常重要的组件。

As we mature as a company, and one of the reasons we are not just doing pure research, we're also like, we've developed a product and we have real customers and we are getting feedback on the models from the real world, is that by doing that, we are also developing components that are also very important to deploy these models.

比如说，像是一个推理引擎。

For example, like a serving engine.

你知道，如果没有推理引擎，你就没法在生产环境中真正运行这些模型。

You know, if you don't have the serving engine, you can't really serve these models in production.

所以从一开始就逼着自己去端到端地部署一些东西，我们学到了很多关于如何运行这些模型，以及如何构建运行这些模型所需的软件。

And so by forcing ourselves from the very beginning to go out and and deploy something end to end, we're learning a lot about how to serve these models and how to build software that it's needed to run these models.

而这些，又变成了知识产权。

And that, again, becomes IP.

即使你训练了基于扩散的大语言模型，如果没有推理引擎，没有像VLLM那样的东西来运行它，你还是卡住了，还是没法用。

Even if you train the Diffusion Based LLM, if you don't have the serving engine, if you don't have the VLLM equivalent to serve it, you're still stuck and you still cannot use it.

同样地，我们和真实客户合作，从他们那里得到关于模型的反馈。

Along the same lines, we are working with real customers and we're getting feedback on the models.

我们弄清楚什么有效，什么无效。

We figure out what works, what doesn't.

有时候我们也会从他们那里收集数据。

We collect data sometimes from them.

我们会根据他们看到的内容来创建评估。

We create evals based on what they're seeing.

所以这又成了技术壁垒的一部分，因为这些东西确实比较难复制。

And so that again becomes part of technical mode because of course those things are a little bit harder to replicate.

如果这个问题在技术上不太合理，你可以直接告诉我。但扩散模型在图像或视频生成上有个结构上的优势，就是你在复现一个应该有某种一致结构的东西，语音也是一样，对吧？

You can tell me if this doesn't make sense as a question to ask technically, but one of the things that diffusion models benefit from structurally in images or video generation is, you know, you're replicating something where there should be some consistent structure in the world, Voice as well, right?

它只是真正可能且最有可能的结果。

It is whatever is really possible and most likely.

这些是AI迄今为止最有价值的领域之一。

There are like some of the fields where AI has been most valuable to date.

比如说，很多用于训练的数据，比如代码数据，其实非常杂乱，对吧？

I'd say like, you know, a lot of the input data you use to train like code data, for example, it's very messy, right?

而且，可以说其中很多数据并没有你想要的真正正确的结构。

And, you know, one could argue that a lot of it doesn't actually have the like correct real structure you're looking for.

那你觉得，对于人类生成的数据，和图像、视频、语音这些相比，你是怎么看的？

How do you think about that when it's like a human generated input data versus images, video, voice?

对，这是个好问题。

Yeah, it's a good question.

从根本上说，如果你想想训练生成模型时你在做什么，不管是自回归模型还是某种程度上扩散模型，你其实是在通过构建一种压缩方案来识别数据中的结构。

And fundamentally, if you think about whenever you train a generative model, what you're doing is whether it's an autoregressive model or, to some extent, even a diffusion model is you are trying to identify structure in the data by essentially building a compression scheme.

这可能不太明显，但训练这些模型时，你实际上是在试图通过找到一种高效的数据压缩方式来识别共同结构。

That might not be obvious, but whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data.

所以，你能压缩的数据越多，你识别出的结构和模式就越多，这就是这些模型的工作原理。

And so the more you can compress the data, the more structure, the more patterns you're identifying, and that's how these models work.

神奇之处就在这里，仅仅通过预测下一个词，你就能学到关于数据结构的一些东西。

Which is the amazing thing, it's just like by predicting the next word, you are learning something about the structure of the data.

不管是扩散模型还是自回归模型，都是一样的，这两种方法本质上都是在学习一种压缩方案。

And that's the same whether you're using a diffusion model or you're using an autoregressive model, both methods are essentially trying to learn a compression scheme.

当我提到2024年的那篇原始论文，我们展示了与自回归模型达到同等水平时，我们使用的指标基本上是困惑度，这衡量的是你在数据中识别出了多少结构。

And when I mentioned the original 2024 paper when we showed that we are achieving parity with autoregressive models, the metric that we're using is basically perplexity, which is a notion of how much structure have you identified in the data.

所以，尽管可能不太明显，我们实际上在GPT-2规模上识别出了与自回归模型相同数量的结构。

And so even though it might not seem obvious, we were actually able to identify at the GPT-two scale, the same amount of structure as an autoregressive model.

是的。

Yes.

我觉得实证结果确实是这样，但直觉上可能会觉得，代码这个数据集并不是基于物理规律的，对吧？

I think like that empirical result is there, but the intuition would be like, well, code is not, the dataset that you are working on is not like grounded in physics, right?

里面有很多噪声，听起来你觉得这个问题是可以解决的。

There's a lot of noise in there and it sounds like that is, you believe that's a manageable problem.

对，确实有噪声。但数字不会说谎，只要你能把困惑度降下来，就说明你确实能构建一个达到那种压缩水平的压缩方案。

Yeah, there is noise in And so to the extent, you know, the numbers don't lie, to the extent that you are able to drive the perplexity down, then it means that you can actually build a compression scheme that will get you that level of compression.

所以结构一定存在，模型也一定能够发现它。

And so the structure must be there and the model must have been able to uncover it.

接下来更多是归纳偏置的问题。

And then it's more a question about inductive bias.

比如，Transformer是不是识别这些模式的更好方式，还是其他方法更好？

Like, is transformer a better way of identifying those patterns or something else?

下一个词预测是不是正确的建模框架，还是说去噪更合适？

Is next token prediction the right modeling framework, or is it more like denoising?

这其实是一个非常实证性的问题，我觉得目前我们甚至还没有工具去理解它。

And that's very much an empirical question that I think at the moment we don't have tools even to understand.

我能问个问题吗？因为你刚才用语音客户作为例子。

Can I ask a question just because you used a voice customer as the example here?

一些构建这些AI产品的人发现，在语音管道中间放一个LLM的好处之一是，他们能更好地理解如何在那里进行对齐或可控性。

One of the benefits that some people building these AI products have identified of having an LLM in the middle of this voice pipeline is they understand how to do alignment a little bit better there or controllability.

我想对于基于扩散的模型来说，这肯定得有所不同。

I imagine that has to look different for a diffusion based model.

你能谈谈这个吗？

Can you talk about that?

是的，这是一个关键的价值主张。

Yes, that's a key value proposition.

他们总是会研究的一件事是，他们提供的很多价值就像是一个控制框架，确保模型确实在做正确的事情。

And one of the things that they always look into is, yeah, to what extent a lot of the value they provide is like the harness and making sure that the models indeed are doing the right thing.

关于扩散基础模型，有趣的是——

And the interesting thing about a DiffusionBase.

LLM 的好处在于我们把一切都设计成向后兼容的。

LLM is that we've built everything to be backwards compatible.

所以 API 还是一样的。

So it's still like the API is the same.

仍然兼容 OpenAI，文本进，文本出。

It's still OpenAI compatible, text in, text out.

而且我们训练的模型恰好很擅长遵循指令。

And it so happens that the models we've trained are good at following instructions.

它们很擅长输出，如果你用 JSON 的话，结构化输出这些都能处理。

They're good at outputting, if you're using JSONs, like structure outputs, they can handle all of those things.

而且效果已经足够好了。

And it was good enough.

实际上比他们之前用的模型还要好。

It was better in fact than the models they were using before.

所以他们用 Mercury 依然能为客户提供同等水平的服务。

And so they are still able to provide the level of service to their customers by using Mercury.

嗯，很简单，如果接口一样，你就可以直接用你原来的那套技术栈。

Well, very simple if the interfaces are the same and you could just use your same stack.

我觉得这其实是个很有意思的点，就是我们知道扩散模型通常比自回归模型更容易控制。

It could be that I think that's actually a very interesting point is that we know that diffusion models are typically easier to control compared to autoregressive models.

原因在于，如果你想想自回归模型，你得等它生成完整的对象，才能知道它是否满足某个约束，或者是不是一条线，或者是不是符合品牌风格，或者任何你关心的目标函数。

And the reason is that if you think about an autoregressive model, you kind of have to wait until you've generated the full object to know whether or not it satisfies, let's say, a constraint or whether or not it's a line or whether or not whatever, it's on brand, whatever it is that the objective function that you care about.

比如你在生成一个分子，你关心它的溶解性。

Maybe you're generating a molecule and you care about solubility.

那你得等拿到完整的对象，才能用某个奖励函数给它打分。

And then you kind of have to wait until you have the full object to be able to score it with some reward function.

但扩散模型是那种从粗到细的生成方式。

But a diffusion model, it's more coarse to fine generation.

从一开始就……

From the very So

逐步细化。

progressively do it.

从一开始，你就会想，这个东西是不是我想要的？

From the very beginning, you know, is this object the kind of thing I want or not?

然后你可以根据外部奖励函数或某些约束条件来引导生成过程。

And you can steer the generation in the direction provided by an external reward function or some set of constraints.

所以至少在学术文献中有很多证据表明，扩散模型更容易控制，而且有多种引导方式，这些方式在自回归模型中根本不可能实现。

And so there is a lot of evidence in the academic literature, at least, that diffusion models are easier to control and there are different ways of steering them that are just not possible with autoregressive models.

所以这可能是模型的一个不同接口，也许自回归模型根本不会有这种接口。

So that would be a different interface for the model that maybe might not be even available for autoregressive models.

我觉得这是我们一直在思考的问题，比如我们应该围绕这些新能力打造什么样的产品体验，而这些能力是自回归模型无法提供的。

I think that's something that we've been thinking a lot about, like what would be the right what's the right product experience that we can build around new capabilities that are just not provided by autoregressive models.

除了性能之外，你觉得Inception的模型在规模化之后，会有哪些今天模型还不具备的能力？

Are there capabilities that you imagine Inception's models having at scale that today's models don't have beyond performance?

对，这就是关键。

Yeah, that's the thing.

不知道。

Don't know.

这就是新兴的，它是涌现出来的。

That's the emerging It's emergent.

是的，是的。

Yes, yes.

比如现在，楔形速度，我们知道它们快得多。

Like right now the wedges speed, we know they are much faster.

那是最初的赌注，因为那更容易测试。

That was the initial bet because that was easier to test.

而且它也很容易衡量，显然很有价值。

It's also like easy to measure and it's obviously valuable.

这就是为什么我觉得它如此令人兴奋，因为随着我们更多了解如何训练这些模型，我们不知道会发现什么。

That's why I find it so exciting is that as we learn more about how to train these models, we don't know what we're going to find.

而且学术文献中有相当多的证据，例如，基于扩散的模型比自回归模型更数据高效。

And there is a decent amount of evidence in the academic literature, for example, that diffusion based models are more data efficient compared to autoregressive models.

直觉上，就像，如果你考虑训练一个扩散模型，你是通过去噪来学习的。

And the intuition is just like, if you think about training a diffusion model, you're learning by denoising.

你从一张图片开始，加上噪声，然后学习如何去除噪声。

You start with an image, you add noise, and then you learn how to remove the noise.

所以这实际上是在做数据增强，因为同一张图片被多个带噪声的版本增强了。

And so it's effectively doing data augmentation in the sense that the same image is augmented by many noisy views.

好的。

Okay.

所以它们往往更节省数据。

And so they tend to be a little bit more data efficient.

如果在大规模下这一点成立，而且你相信的话，那么也许我们会进入那些数据较少的任务。

And so if that holds up at scale and then you believe, then maybe we'll get into Tasks

数据较少的情况。

where we have less data.

对，当数据成为瓶颈时，那就更有意思了，对吧？

Yeah, where data becomes the bottleneck, then it becomes more interesting, right?

所以，我们拭目以待。

And so, we'll see.

这就是为什么它这么令人兴奋，因为事情在变化，而且这项技术如此重要、如此有价值，拥有差异化优势，我觉得就能创造价值。

That's why it's so exciting because things are changing and then this technology is so important and so valuable that having something differentiated, I think will create value.

如果我们展望五年，那在AI领域其实太长了。

If we project out five years, that's actually way too long in AI world.

如果我们展望两年，你觉得扩散模型和传统模型的工作负载会怎么分配？

If we project out two years, oh, what do you think is the workload split between Diffusion and traditional models?

我觉得我们还没达到智能的前沿水平。

I think we're still, we're not at the frontier level of intelligence.

而且我认为很多工作负载确实需要前沿水平的智能。

And I think a lot of the workloads do require frontier level intelligence.

但根据我的估计，即使你只是去OpenRouter看看，它有一个很好的方式展示所有不同的用例，你可以看到研究、对话、编程、软件工程以及日志处理等。

But on my estimates, so even if you just go to OpenRouter has this very nice way of looking at all the different use cases, and you can see the research and conversational and coding and software engineering and the log processing log.

他们基本上有一个很好的任务分类体系。

They have a nice hard taxonomy basically of tasks.

我做过一些估算，我觉得大约有20%到30%的任务，延迟真的非常重要。

And I was doing some estimates, and I think that is between 2030% where latency is really, really important.

所以至少，作为一个下限，我认为这是可以覆盖的。

And so at the very least, as a lower bound, I think it could be Addressable.

在给定的延迟预算内，模型可以覆盖这些任务，同时提供最高质量的结果。

Addressable by models that are within a given latency budget, they would give you the highest possible quality.

而且，所有技术方法都有权衡取舍。

And then just, all technology approaches have trade offs.

使用扩散模型有哪些挑战？

What are the challenges of working with Diffusion models?

是的，这是一个不同的技术栈。

Yeah, it's a different stack.

其中一个挑战是，我们很多组件都需要自己开发。

And so one of the challenges is that we have to build a lot of things in house.

目前还没有成熟的生态系统，比如服务引擎或内核，很多东西都得自己动手做。

And there is not a mature sort of like ecosystem of, if anything about the serving engine or like kernels, like a lot of the things had to be developed in house.

而且开源的东西不多，虽然有一些开源模型，但质量都不太好。

And so there is not really anything open source or there are some open source models, but they're not particularly good.

所以这让部署变得更困难，让客户尝试新东西也更难，因为他们不习惯。

And so that makes it a little bit more difficult to deploy, to get customers to try things, they're not used to it.

所以这是挑战之一。

So that's been one of the challenges.

我想这也会反映到外部，对吧？

I imagine that also reflects externally, right?

你知道，在一个客户比较成熟、会关心成本和性能的市场里。

You know, in a landscape where folks at some sophistication where they would care about cost and performance.

而且对于某些用例，你一开始就会关注成本和性能。

And then you might, for certain use cases, you will care about cost and performance the beginning.

现在大家对后训练的兴趣越来越大了，对吧？

There is an increasing amount of interest in post training, right?

所以我想，在新的架构上，这会更难。

And so I imagine in a new architecture that's even more challenging.

所以我们得自己搭建做SFT、RLHF和RL的整套流程。

So we have to build our own stock for doing SFT, for doing RLHF, doing RL.

我是说，这在某种程度上就成了知识产权了。

I mean, that becomes IP to some extent.

这也是我们决定不完全开源的原因之一，主要是想把知识产权保留在自己手里，不公开。

It's one of the reasons we decided not to open source everything was really to keep the IP a little bit closer to us and not opening it.

但这样做也有弊端。

But then there are downsides.

社区贡献的机会就少了。

There is less an opportunity for the community to contribute.

采用起来也更难。

It's harder to adopt.

本地部署之类的操作会很麻烦。

It's hard to do on prem kind of like deployments.

所以这两种选择各有利弊。

And so there are pros and cons with the two choices.

你能谈谈你们自己的训练规模吗？不管是现在的还是期望达到的。另外，你们现在有50人，肯定还在继续招人，那为什么研究人员或工程师应该考虑这个方向，或者来Inception工作呢？

Can you talk about, you know, scale of your own training and then like current or aspirational, and then at 50 people, I'm sure you're continuing to hire like why researchers or engineers or others should consider investing in this direction or working at Inception.