开放模型能否跟上AI前沿？| Nathan Lambert

说到模型的进步，归根结底就是不同形式的资本，也就是算力、数据和人才。

When I think about model progress, it comes down to different forms of capital, and that is compute data and talent.

那你觉得未来几年谁会赢呢？

How do you see that playing out in terms of who's going to win in the next couple of years?

中国人在构建模型方面真的很厉害。

The Chinese are so good at building models.

要说中国在人才上不如别人，那很难让人信服。

To say that China is anything but equal on talent, you're going to have a hard time.

他们缺的是算力。

What they don't have is compute.

大多数人认为蒸馏是他们做到这一点的关键手段。

Most people regard distillation as the main lever for them to do this.

我的产品每天都有人去蒸馏。

People distill my products every single day.

他们拿过去，拆解成最原始的部分。

They take it, they strip it down to its raw parts.

这就叫竞争。

That's called competition.

以前的后期训练更依赖监督微调。

Previous post-training was much more reliant on the supervised fine-tuning.

现在，这已经有点演变成强化学习为主的环境了。

Now, this has morphed a bit into environments where it's RL heavy.

但是，这种后期训练——就是拿到结果、分数上升——很难直接对应到现在的推理能力上。

But it's just hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.

现在的讨论里，大家特别担心前沿技术被藏起来，而且差距还在拉大。

The discourse now is just so much fear on how much the frontier is just hidden and also breaking away.

AI能力的发展曲线倒不一定让我担心，但让我担心的是，在公共网络安全这些方面，几乎没有任何准备，比如——

The AI capability plot doesn't necessarily worry me, but the AI capability plot in the pretty much non-existing preparation across public cybersecurity and all of these things, like-

前沿AI实验室花了大把的钱来造更好的模型。

The frontier AI labs are spending extraordinary amounts of money to build better models.

那开源是怎么一直跟上来的呢？

So how does open source keep catching up?

本期节目中，a16z合伙人李丽莎与AI研究员、新成立的非营利研究机构Trillium的创始人Nathan Lambert深入探讨了开源AI与闭源AI之间的竞争。

In this episode, a16z partner Lisha Li sits down with AI researcher Nathan Lambert, founder of the new nonprofit research lab Trillium, to unpack the race between open and closed AI.

他们讨论了为什么OpenAI和Anthropic在算力、资金和速度上仍具有显著优势，但Nathan认为，尤其是中国，开源模型背后的技术人才往往被低估了。

They discuss why OpenAI and Anthropic still have major advantages in compute, capital, and speed, but why Nathan believes the technical talent behind open models, particularly in China, is often underestimated.

他们还厘清了模型蒸馏的实际效果与其宣传话术之间的区别，并关注后训练数据和强化学习环境如何成为日益重要的进步来源。

They also separate the reality of model distillation from the rhetoric around it and look at how post-training data and RL environments are becoming increasingly important sources of progress.

最后，他们转向了递归自我改进这一话题。

And finally, they turn to recursive self-improvement.

Nathan 解释了 Trillium 计划如何记录 AI 实际被用来改进自身的过程，从模型已经能自动化的任务，到它们仍然卡壳的地方，以及为什么让这种进展变得可见，可能会引发一场关于前沿技术真正发展速度的更务实的讨论。

Nathan explains why Trillium plans to document how AI is actually being used to improve AI, from the tasks models can already automate to the places they still get stuck, and why making that progress visible could lead to a much more grounded debate about how quickly the frontier is really moving.

所以 Nathan，很高兴你能来。

So Nathan, really happy to have you here.

谢谢你亲自到场。

Thank you for joining in person.

时机特别合适，因为你正在启动 Trillium，我很想聊聊这个，不过先从宏观层面来说，我觉得我们之前聊了很多关于开源的话题，而这正是你关心的重点。

Super timely given you're launching Trillium, which I'd love to talk about, but more on a macro level, I think we chatted a lot about open source, and that's a lot of what you care about.

现在公众讨论里经常听到这样的话：‘我们要控制前沿技术的发展节奏，这个前沿啊、那个前沿的。’

And what we've been hearing a lot in the public discourse is, "Hey, we have to pace the frontier, the frontier this, the frontier that."

我觉得我特别想请教专家的一个核心问题是：所谓的前沿到底包含哪些内容？

And I think an arc of the stuff that I'm really trying to ask experts about is what is in the frontier?

我们具体在管控什么？

What are we pacing?

里面究竟有些什么东西？

What's there?

更重要的是，结合开源和开放权重模型来看，他们能跟上前沿技术的步伐吗？

And then more importantly, threading it with open source and open weight models, can they keep up with the frontier?

我觉得这非常关键，而且这也是你的专长领域。

And I think this matters very much and is something that you're an expert on.

那我们就从这个话题切入吧，你觉得目前推动封闭实验室和开源模型达到前沿水平的关键因素分别有哪些？

So maybe to start on that arc, how you would break down what's gotten us to the frontier for both closed labs and open-source models.

在你看来，这些要素具体是怎么分布的？我们可以先从宏观层面聊聊，比如预训练、中训练和后训练这几个阶段。特别是后训练这块，它的发展演变过程是怎样的——这方面你之前给公众做过很多科普讲解。

How does it break down in your mind, maybe first at a higher level of pre-training, mid-training, post-training, how post-training has evolved, which is something you've done a great amount of education to the public on.

说到模型进展，其实就是你投入的不同形式的资本，也就是算力、数据和人才。

When I think about model progress, it comes down to different forms of capital that you put into them, and that is compute data and talent.

所有实验室都有不同数量的这些资源，但我觉得像OpenAI、Anthropic、谷歌这些封闭前沿实验室，显然拥有最多的资源。

And all of the labs have various amounts of this, and I think the closed frontier labs of OpenAIs, Anthropic, Googles have clearly had the most of these.

而我们看到的是——

And what we see-

——各实验室组织这些资本的能力不同，要让组织架构真正服务于模型构建是非常难的，这也是为什么有些公司不得不从头彻底重组。

... is the different abilities of the labs to organize these forms of capital, and making the org chart work for building models is really hard, which is why you have companies have to do complete resets from the bottom up.

然后有时候这样还真能奏效。

And then it actually kind of works sometimes.

Meta就是彻底重组了，现在又回到了上升轨道。

Meta is totally reset and they're on a positive trajectory again.

确实，没错。

Seriously, yeah.

我对这些人的因素很感兴趣，比如为什么竞争会推动前沿，为什么组织架构这么重要，因为我们看到了这些现象，而且这在开放与封闭的动态中也同样起作用，就像——

I think I'm very interested in these human aspects of why competition drives the frontier and why the org chart is so important, because we see these things, and this comes into play in the open, closed dynamic as well, where it's like-

……那些封闭型公司，也就是真正的前沿实验室。如果有人想在AI前沿工作，我通常建议他们去Anthropic或OpenAI，我觉得这是最稳妥的选择。

... the closed companies, the true frontier labs, which if somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet, I think.

Meta、Google还有SpaceX的AI部门，我觉得你也有可能在那里，但从外部看情况不太明朗。

Meta, Google, SpaceX AI, I think you still might be there, but it's harder to see from the outside.

这些公司为了达到目标，最早就押注了最大胆的项目。所以OpenAI才会推动推理模型革命，我认为这种突破以及随之而来的漂亮曲线——也就是推理扩展定律，正是OpenAI的高光时刻。

And these companies have just taken the most ambitious bets the earliest in order to get there, and that's why OpenAI has done things like kickstart the reasoning model revolution, and I think that kind of breakthrough and that beautiful plot with the inference scaling laws is peak OpenAI.

他们特别擅长抓住那种能彻底改变所有人思维方式的趋势。

It's like they're really good at getting this plot that just changes how everyone thinks about everything.

另一方面，Anthropic一直非常专注于模型未来能够具备的能力。

And on the other side, it's like Anthropic has been very focused on the types of things the models are going to be able to do.

然后他们得到了回报，随着长期聚焦编程领域，迎来了云代码方面的爆发式增长。

And then their reward was this cloud code explosion with their focus on coding for so long.

这两家公司有着微妙不同的企业文化，我认为它们尤其具有前瞻性，也最愿意投入巨额资金，去推动那些被认为不可能实现的方向。

And there's these subtly different cultures, and I think those two companies, in particular, have been the most forward-looking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible.

中国公司进入这个领域，是因为它们希望打造一家有影响力的企业。

The Chinese companies came into this because they want to have a company that matters.

如果他们在中国建模型并试图卖API，根本没人会知道他们。

And if they were to build their models and try to sell an API from China, nobody would know about them.

没人会知道月之暗面。

Nobody would know about Moonshot AI.

知道智谱的人也会非常少，而且后面还有一长串名单。

Very few people would know about Zhifu, and there's a very long list after this.

这对他们来说是一个非常务实的决定。

And this is a very practical decision by them.

他们在跟随市场的风向。

They're following the discourse.

他们知道这里会有巨大的市场。

They know there's going to be a huge market.

他们知道美国对付费使用这些东西有巨大的需求，这非常务实。

They know there's a huge demand in the US for paying for these things, and it's very practical.

所以，发布开源模型是一种方式——

So therefore, releasing open models is a way to-

……绕开这一步，直接进入对话。

... shortcut this and get into the conversation.

我觉得他们这么做的原因很复杂。

And I think there's a lot of complexity in why do they do this?

政府有参与吗？

Is the government involved?

没有。

And not.

事实并非如此。

It was not the case.

他们只是想创业，并不是为了——

They're just trying to build a startup, and it's not to-

真实际。

So practical

……这并不是说将来不会走到那一步。

... it's not to say that it will not get there.

中国政府做的很多事，在美国人看来简直让人难以置信。我觉得这也说明中国的实验室在安全问题上态度更严肃，因为在中国，针对企业不当行为的回应往往比美国要严厉得多。

The Chinese government does things that would blow people's minds in the US, and I think that that makes me think that the Chinese labs have a much more serious approach to security because things happen in China that are much more drastic responses to companies doing things in the US.

我就不深入探讨那些复杂细节了，但在开放与封闭之间的平衡上，中国的这些公司正在努力追赶。

And I don't need to go down this rabbit hole, but between the balance of open and closed, these companies in China are trying to catch up.

这和我之前提到的 OpenAI 和 Anthropic 的情况完全不同。它们是在尝试建立新的范式，推动真正的通用人工智能（AGI）实现，以便谁先做到，谁就能凭借最顶尖的模型永远占据优势，这就是所谓的颠覆性创新思路。

And this is a very different dynamic than what I said about OpenAI and Anthropic that are trying to create the new paradigms, that are trying to make true RSI happen so that their company wins with the best models forever if you get there first, these boom ideas.

而你们只需要填补市场中的空白领域就行。

And- You just have to fill the market where there is a void.

开源权重模型正在尝试这么做，我觉得随着时间推移，会出现不同层级的开源模型。最顶尖的闭源模型能带来最高的利润空间。

And the open-weight models are trying to do this, and I think over time there will be these tiers of open models where there's the best closed models, which you can demand the best margins for.

然后是接近前沿水平的模型，比顶级模型稍逊一筹，这些会开源，因为供需关系使然，它们面向的市场可能利润没那么高。

And then there's the near frontier models, which is a step behind this, and those will be open because there's just some sort of supply and demand there as you get access to a different market that might be less profitable.

比如涉及私有企业数据之类的场景。

For example, if you have private enterprise data or so on.

这种分层其实是企业在探索业务过程中自然形成的演变结果。

And these tiers are kind of natural evolutions of companies that are trying to do things.

我觉得我们刚才聊到了前沿领域。如果你回头看看资本方面，他们资金少，赚的钱也少，这倒没关系，但我认为这种追赶的动力非常强。

And I think we were talking about the frontier, and it's just if you go back to capital, they have less capital, they are going to make less money, and that's okay, but I think this drive to catch up is very strong.

一旦你明白了，一旦 Anthropic 和 OpenAI 向你展示了你需要构建的模型类型，接下来就会觉得很简单，心想：‘哦，我们只要做这件事就行了。’

And once you know, once Anthropic and OpenAI show you the type of model you need to build, it's a lot easier to then be like, "Oh, we just need to do this thing."

他们会购买类似的数据集。

They buy similar data sets.

这些影响会渗透到数据经济中，因为 OpenAI 和 Anthropic 投入了更多的资金来搭建这个数据产业的功能。

It trickles through the data economy where OpenAI and Anthropic spend way more to build out this function of a data industry.

然后那些数据公司，过了一年，可能就会把这些卖给所有人。

And then those data companies, a year later, might sell them to everybody.

他们可能会把数据卖给所有人。

They might sell them to everybody.

他们需要最大化投资回报，而且合同里通常有特定的期限。

They need to maximize their return on their investment, and they had some contractual period.

我觉得在这种动态下，快速跟进是非常可行的。

And I think there's a lot of dynamics where fast following is going to be very doable.

这个循环已经持续了一段时间，比很多人预想的要长。

And we've seen this cycle proceed for a while, longer than many people would expect.

我记得ChatGPT刚发布的时候，我们就讨论过行业整合的问题，当时还做了一些预测。现在我觉得我们才刚刚开始看到并购市场变得活跃起来。

I think when ChatGPT was released, we talked about consolidation in the industry and predicting it, and I think we're just starting to see a much more active mergers and acquisitions market.

其实我两年前就预料到会发生这种情况，但开源模型公司撑得比我想象的还要久。

But I thought this was going to happen two years ago, and the open model companies have lasted much longer than I thought.

这里有很多值得深入探讨的话题，首先就是你刚才对两家顶级前沿实验室在组织架构上的差异所做的评论。

There's so many interesting threads to pull on here, starting with actually your commentary on the organizational differences between the top two frontier labs.

我们可以先把这个问题放一放，或者顺着这个思路继续聊。

We can maybe table that or go on through that thread.

当然，你也在解释为什么中国的开放实验室把重点放在分发和实际商业化上，而不是出于利他主义。虽然研究人员一开始确实有分享的动机，不管好坏，但归根结底，这种模式能持续下来，是因为它是一个非常成功的商业化策略。

But then, of course, also kind of are you explaining sort of how the open labs in China have really just chosen that as much more of a distribution/practical commercialization strategy rather than, hey, obviously out of altruism, though it's great that the researchers, maybe for better or for worse, were motivated from sharing in the beginning, but at the end of the day, it perpetuated because it was an excellent commercialization strategy.

所以另一个值得探讨的点是，我认为大多数人把模型蒸馏视为实现这一目标的主要手段，而且效果越来越好。过去几年里，开源模型和前沿模型之间的差距一直在缩短。

So the other thread maybe to pull there is sort of, I think most people regard distillation as sort of the main lever for them to do this so well and kind of increasingly well because the lag between the open source models and the frontier models have only shortened with time in the last few years.

现在这个时间差可能只有两三个月了。

Now it's maybe only two to three months.

这也促使前沿实验室可能想要关闭API，关闭他们最好的模型，只提供自家服务。

And this has also pushed the frontier labs to maybe want to close off the APIs, close off their best models to do first-party services.

一方面可能是为了获取更多价值，因为这样总是可能的；另一方面也是因为蒸馏变得更难了，不是不可能，但难度大大增加。

One, probably to capture more value because it's always possible that way, but then two, also because it's harder to distill, not impossible, but much harder to distill.

这两个话题都很有意思，我们就试着都聊聊吧。

Maybe on those two threads because they're both very interesting, so we'll try to touch on it.

先从蒸馏说起吧——中国实验室真的像外界描绘的那样吗？毕竟你亲自见过他们，也研究得比我们深入得多。

Let's start with is the distillation story really more of a caricature than the Chinese labs, since you've actually met with them and you've studied this a lot more.

他们是不是全面发展，预训练做得很好，也有自己独特的后训练方法，只是可能没有公开？

Is it sort of like they're all-rounded, great at pre-training, also have their own good post-training recipes that they may or may not have revealed?

你觉得哪些方面会加速发展？

And where do you see the levers kind of accelerating?

中国实验室的技术其实非常强。

So the Chinese labs are very technically excellent.

我觉得他们更缺算力。

They are more compute starved, I think.

是的。

Yes.

毫无疑问，他们拥有同样优秀的人才储备。

Without a doubt, they have a comparable talent pool to draw on.

我觉得在AI真正起步的时候，谁在哪个位置可能会有一点优势。

I think there might be a slight edge based on who was where when AI really got going.

但要说中国在人才上不如别人，那很难说得通，因为他们有太多人接受过高水平训练，他们缺的只是算力。

But to say that China is anything but equal on talent, you're going to have a hard time because they just have so many people being trained at such a high level, and what they don't have is compute.

你跟那边的人聊，他们会说：“我听说OpenAI的普通研究员随时能用5000块H100级别的算力”，他们听了都惊呆了。

You talk to people there and you're like, "Oh, I heard an average OpenAI researcher has 5,000 H100 equivalents that they can access at any time," and their jaw just falls to the floor.

他们只会说：“我简直无法想象。”

They're just like, "I can't comprehend."

这还是一位顶尖的预训练研究员——

And this is a top pre-training researcher at-

在你知道的某个实验室里，你遇到几次这样的情况。

... one of the labs you would know, and then you have this interaction a few times.

我觉得算力是最大的差别，而蒸馏其实是一种变相花钱买算力的方式。

And the compute thing, I think is the biggest difference, and distillation is a way to effectively spend more money to get what is compute equivalent.

语言模型的输出是算力的函数，如果你知道想要的输出，那就等于有了用模型生成所需的算力。

Outputs from a language model are a function of compute, and if you know the outputs you want, it's the same as having compute to generate with a model.

我见过蒸馏的各个阶段，在我看来，以前的后期训练更依赖监督微调和RLHF这类方法，那是后期训练刚起步时的做法，你只要在强模型的输出上做下一个词预测的损失，就能获得大部分性能。

And I've seen the stages of distillation, and how I see it is that previous post-training was much more reliant on the supervised fine-tuning and other, like this RLHF era, which is where post-training started, and you could get most of your performance just by you do this next-token prediction loss on the outputs of a very strong model.

大家都清楚，拿最好的模型，生成一些补全，然后训练，只要你有合适的提示词，模型就会相当不错。

It was very established that you take the best model you can, you generate completions, and you train on it, and your model's going to be pretty good so long as you have access to the right prompts.

提示词一直是后期训练的关键。

Prompts have always been a key to post-training.

我记得在AI2的时候，我和Hannah一起负责后期训练，我们总说提示词是后期训练的核心单元。

I remember at AI2, with Hannah, who I co-led post-training there with, we would always say prompts are the key unit of post-training.

你需要拿到提示词，然后在训练的不同阶段用它们做不同的事情。

It's like you need to get the prompts, and you can do different things with them at different stages of training.

现在这已经演变成了一些以强化学习为主的环境。

And now this has morphed a bit into environments where it's RL-heavy.

但是这种后训练方式，就是拿到完成结果然后分数上升，很难直接对应到现在的推理能力上。

But it's just hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.

而且根据公开的文献，我很难说“蒸馏是一个主要因素”，因为比如在完全开放的后训练配方尝试中，所有开放模型的构建者都卡在一个很过时的Qwen模型上，因为你搞不清楚怎么把教师模型弄对。

And based on the public literature, it's very hard for me to be like, "Distillation is a major factor," because, for example, in the fully open literature of people trying to do open post-training recipes, all the open model builders are stuck using a really outdated Qwen model because you can't figure out how to get the teacher right.

我的假设是，在中间训练阶段或者开放的预训练数据里有些东西，让这件事变得很难，也就是从最好的模型获取推理轨迹然后做SFT，分数就能上去。

My hypothesis is that there's some mid-training and something in the common pre-training data that's open that makes it hard to do this thing, which is get reasoning traces from the absolute best model and do SFT, and the numbers go up.

所以如果我在构建Olmo的时候能拿到Claude的推理轨迹，比如Claude Opus、Claude Mythos之类的，我不确定那真的会让我们的模型好很多。

So if I were to have access to Claude's reasoning trace when I was building Olmo, like Claude Opus, Claude Mythos, whatever, I'm not sure that it would have actually made our model a lot better.

这就是我基于文献的证据，说明为什么我认为蒸馏很难。

So this is my evidence grounded in the literature why I think distillation is hard.

那你怎么拿模型的输出来构建一个RL环境，然后进行大规模的RL训练呢？

And then how do you take the outputs of a model and build an RL environment to do a big RL run?

因为你看小米这样的公司，他们做了一周的开放RL训练，然后说“我们做了这些”，接着就把模型放出来了，那蒸馏到底用在哪里呢？

Because then you have the Xiaomis of the world doing this week-long open RL run where they're like, "Here's what we did," and then they've dropped the model, and it's like, where exactly does distillation go into this?

他们很聪明。

They're clever.

他们很会做营销。

They know how to do marketing well.

有些问题在于，他们是不是提前做了这些，因为他们先跑了强化学习，然后发布了模型，之后技术报告里又说在强化学习之后还有一个额外的后训练阶段。

There are some questions of did they Do this ahead of time because they did their RL run, and then they dropped the model, and then their tech report said they had an extra post-training stage after RL.

所以这里有些疑问。

So there's some questions.

但是要把已有的蒸馏文献对应到那个上面，非常困难。

But mapping the established distillation literature to that is very hard.

不过有些疑点让我稍微站在实验室那边，就是那些Claude代理和中国的模型代理，有大量token被使用，还有中间商，他们有很多账号。

But there are question marks that make me support the labs a bit, which is these Claude proxies and these model proxies in China, where there's a ton of tokens that are used, and there's intermediaries, and they have tons of accounts.

甚至有些案例中，Kimi在他们的产品里向用户提供Claude服务。

And even cases where Kimi was serving Claude in their products to users.

我只是不太明白，这怎么作为后训练中的常见技术工具融入蒸馏。

And I just don't really understand how this fits into distillation as a common technical tool in post-training.

是的，我们也应该关注这个。

And yes, we should watch it too.

还有一些奇怪的事情，就是中国公司大规模使用OpenAI和Anthropic模型的输出。

There's other weird stuff with Chinese companies using the outputs from OpenAI and Anthropic's models at a vast scale.

所以我觉得我们还有很多问题需要回答，而且我觉得大多数理性的人在问“蒸馏的差距有多大？”的时候。

So I think we have a lot more questions we need to answer, and I think most reasonable people when they're like, "What's the gap from distillation?"

都会说“误差范围很大。”

Are like, "There's very big error bars."

但我说蒸馏帮他们节省了几个月，是基于我们现有的科学文献，也就是我们掌握的信息。

But when I say that distillation helps them by a few months, it's based on the scientific literature that we just, that's the information we have.

我们应该做更多开放的研究，并且——

We should do more open research and-

……让这成为可能，这样我们才能研究它。

... enable this so that we can study it.

但基于这些，我不知道你怎么能得出不同的结论，说它在当前的后期训练中没有巨大影响。

But, I don't know how you can come to a different conclusion that it's not a huge effect in current post-training based on this.

而且，中国人在构建模型方面也非常擅长。

And this is like, the Chinese are so good at building models as well.

我觉得有点自欺欺人的感觉，人们好像在说——

I think there's a bit of cope where people are like-

哦，对

Oh, yeah

——只有我们擅长造模型，但实际上——

... only we are good at building models, and it's like-

怎么会——

How is that-

有那么多公司呢。

There are so many companies.

大概有二三十家公司都在做不错的模型。

There's like 20 to 30 companies that are building good models.

把这些东西都公开出来吧。

Publish this shit out.

你都能读到。

You can read.

而且就像你刚才说的——

And that's like you just-

……这让我很难释怀，而且我觉得有一篇非常著名的论文，讲的是从封闭模型里偷推理痕迹。

... and it's hard for me to get over it, and I think that there's this very prominent paper that's like stealing reasoning traces from closed models.

而且我知道中国公司在做这件事上领先很多，因为后来AI安全社区才发表了相关的内容。

And I know that the Chinese companies are way ahead on doing this since the AI safety community later published it.

我觉得每个中国实验室都有一个团队懂这些。

I think every Chinese lab has a team that knows those sorts of things.

这又回到你之前说的，他们不需要提供API。

And this goes back to what you were saying about they don't need to offer the API.

而且我觉得推理模型生成思考令牌有个特点，就是很容易从API里把它们弄出来，这样你就能看到推理令牌，然后用来训练自己的模型。

And I think there's some characteristic of reasoning models that generate these thinking tokens, where it's easy to just knock them out of the API, so you see the reasoning token, and then you can use that to train your model.

所以只要他们提供这些API，就没有办法阻止那种蒸馏。

And so long as they offer these APIs, there's going to be no way to stop that type of distillation.

但如果他们只把模型放在产品里，或者要求签企业协议之类的，那就会难得多。

But if they do make it so it's only in products and things where you have to sign enterprise agreements or whatever, it'll be way harder.

我支持实验室这么做，如果他们想保护自己的东西的话，但他们得先保护好他们的产品才行。

And I support the labs doing that if they want to secure it, but it's like they have to secure their products if they want to.

我觉得了解客户对谁都没好处。

I don't think knowing your customer would be good for anybody.

我觉得对他们的生意也不好。

I don't think it'd be good for their business.

而且，我也不知道该怎么让政府介入一个违反服务条款的事情。

But also, I don't know how you should get the government involved in a terms of service violation.

我觉得服务条款甚至都不算签了合同的。

It's like I don't even think terms of service are a signed contract.

就是说，我觉得你不能像那样起诉他们，你只能取消他们的访问权限。

It's just like, I don't think you can sue them in the same way, you just can cancel their access.

所以这就是为什么我不太同情他们，但我还是喜欢听那些……

So that's why I'm not that sympathetic, but I am-- I like to listen to the...

国家安全界的声音很大。

The national security community is loud.

但确实有一个方向性的真理，那就是AI的发展方向及其影响力，你必须在脑子里权衡很多事情——

But it's like there is a directional truth to that with where AI is going and how influential it is, that you have to balance many-

——很多很多的事情。

... many things in your head.

我想知道你的回应是什么。

I wonder what your response is.

我听说，好像是Jensen在某个推文或者采访里说的，也许吧。

So I heard, I think Jensen said in some literally either tweet or interview, maybe.

那说得非常好。

It was very great.

他当时说——

He was like-

把它放进节目里。

Put it in the show.

人们每天都在提炼我的产品。

People distill my products every single day.

他们拿过去，把它拆解成最基本的部件。

They take it, and they strip it down to its raw parts.

他们测试所有东西。

They test everything.

这就叫竞争。

That's called competition.

你觉得这样公平吗？

Do you think that's fair game?

你觉得蒸馏不算偷吗？

You think distillation is not robbery?

这就叫竞争。

That's called competition.

我非常理解。

I'm very sympathetic.

因为我觉得AI安全界有很多呼声，希望在能力达到这种水平时，能有中美对话。

Because I think there's a lot of calls from the AI safety community that we want to have US-China dialogue as we're getting to such capability levels.

如果你要让公司只是轻轻拍打一下他们的AI行业，甚至更轻的处罚——

And if you're going to ask the company to just start hitting their AI industry with slaps on the wrist and more-

……在蒸馏之后的下游环节。

... downstream of distillation.

所以你不会通过国会的政策来监管蒸馏。

So you won't regulate distillation with policy from Congress.

你可能会采取联邦行动，让他们的日子不好过。

You will probably just do federal action to make their life annoying.

同时，当我们讨论如何管理前沿风险时，封闭模型在训练中会访问网络上不该访问的内容，我们担心开放权重模型可能带来的问题，但到目前为止还没出什么大事，可问题是，为什么大家不用开放权重模型呢？

And at the same time, when we're talking about how do we manage frontier risks, where you have closed models in training, going all over the web and accessing things that they shouldn't, and we have valid worries about what will happen with open weight models, where this hasn't been harmed yet, but it's like, why aren't people using open weight models?

为什么他们要去用封闭模型，绕过安全措施？

Why do they go to the closed models and get around the safeguards?

而且——

And like-

……我们希望和中国科技公司就这些问题进行沟通，但我觉得如果你对蒸馏采取任何措施，只会弊大于利，因为他们不会停下来的。

... we want to be able to communicate with the Chinese tech companies and everything about this, and I just think you're going to cause more harm than good if you do anything about distillation, because they're not going to stop.

他们肯定会继续这么干。

It's like they're going to keep doing it.

开源社区能从中受益，创业公司也因此获益良多。

And the open source community benefits, and therefore the startup community benefits a lot.

我们可以深入探讨很多方面。

We can go into a lot of it.

这可不是光中美两家的事。

It's not just like, hey, China, USA.

确实是这样——我们可以聊聊这个。我当然能理解，或者说试着从正面角度去分析前沿实验室的立场：那就是说，构建这些前沿模型需要前所未有的巨额资金，对吧？

It's actually like there's-- and we can actually talk about, I am sympathetic obviously or try to steelman the frontier lab position, which is like you have just unprecedented amounts of capital that's required to build these frontier models, right?

如果通过蒸馏或其他类似的蒸馏相关方法，开源社区能如此迅速地追上来，这其实挺让人担忧的，也不利于这种飞轮效应持续下去。

And it's kind of alarming/not very conducive to this flywheel continuing if through distillation or other kind of distillation-related methods, open source can catch up so quickly.

所以我说的这个“飞轮”，指的是一种很简单的模式：你投入大量资金购买算力来训练前沿模型，然后他们把这些算力囤起来，当然啦。

And so, and what I mean by this flywheel is just kind of a very simple picture of you train the frontier model by putting in a lot of money to get compute, and then they park this compute, of course.

OpenAI 和 Anthropic 在这方面做得非常到位/相当成功。

OpenAI and Anthropic are very guilty/successful in this.

很成功。

Successful.

不管你怎么看，算力其实非常紧张，因为不只是芯片，还有电力等等。

Whichever way you want to look at, and there is a huge compute, kind of, sort of, lack of compute available because there's actually not just compute and the chips, but there's also power, et cetera.

所以，这种稀缺性会一直持续到2028年，成为瓶颈。

So, like all of this scarcity is kind of bottleneck until 2028.

所以，总的来说，把算力囤起来是个成功的策略，能带来大量收入，因为他们一旦达到前沿，就明白扩展是唯一能让他们掌控收入可预测性的东西。

So this is all to say that it's a successful strategy to park that, which leads to actually a lot of revenue, because once they get to the frontier, because they understand scaling is the one thing that kind of gives them, or the science of scaling is what gives them a handle over how well that can translate into a predictable revenue stream.

所以他们有了收入，然后就能筹到更多钱。

So they get the revenue, and because of that revenue, they're able to raise more money.

这真是前所未有的良性循环，OpenAI和Anthropic都是这样。

It's just unprecedented, like beautiful flywheel, and this happens for both OpenAI and Anthropic.

当然，我们需要指数级的资源。

Of course, we need exponential amount of resources.

所以我们已经到了不得不考虑IPO、上市融资的地步。

And so we've gotten to the point where we're like, "Okay, we need to IPO, we need to go to the public markets."

所以，任何威胁到这个飞轮正常运转的事情，对整个前沿领域来说都是巨大的威胁。

And so anything that threatens this flywheel from functioning well is a huge threat to Just the entire being at the frontier.

所以——

And so-

实际上，整个AI行业都是如此。

The entire AI industry, realistically.

确实，正是如此。

And exactly that, actually.

所以，也许不仅仅是同情他们试图为自己谋取资本主义利益的做法。

So maybe not just being sympathetic to their version of trying to eke out capitalism for themselves.

而且，如果他们的IPO表现不佳，对其他人来说也是坏事。

It's also just like, hey, if they don't do well in the IPO, that's actually bad for everyone else.

所以我希望他们能成功。

And so I want them to do well.

另一方面，中国的开源模型对创业生态系统来说是一个巨大的福音。

On the other hand, it's like the Chinese open source models are a huge benefit to the startup ecosystem.

我们经常看到这种情况，我记得Decagon和Harvey的一些数据，他们当时在尝试做大量的微调。

We see this all the time, just kind of metrics I remember from Decagon and Harvey, who was trying to do a lot of fine-tuning.

有大量的初创公司在做各种形式的训练后服务，并试图将其产品化。

There's a huge amount of startups doing post-training as a service of various degrees and trying to productize that.

这确实有效，因为你可以用这些模型，比如Kimi K3或者你喜欢的GLM开源模型，达到帕累托前沿。

And it actually works because you can get to the Pareto frontier with these, Kimi K3 or pick your favorite GLM open source model.

所以我想，权衡这两件事，这里可以引出很多问题，但我想问一个，你怎么看待未来几年谁会赢，以及这跟你做开源有什么关系？

So I guess, weighing these two things, I guess there's so many questions that can fall from that, but I guess one, how do you see that playing out in terms of who's going to win in the next couple of years versus what-- And then also relate that to why you're doing, obviously, an open source.

我认为开源和闭源模型处于不同的指数增长时间尺度上。

So I think open and closed models are on different timescales of exponentials.

闭源实验室非常一体化，他们拥有高效的推理能力，可以在准备好时大规模提供服务，因为他们有所有这些资源，可以规划，并且能率先进入那些高价值的市场。

And the closed labs are so integrated, where they have efficient inference, they can serve the model at scale when they are ready to because they have all these resources, they can plan, and they get to these markets first that are very high value.

我们在编程领域已经看到了这一点，这也是今年的故事，收入从零增长到一千亿，而且很可能还会继续增长。

We have seen this with coding, and this has been the story of the year, where the revenue has just gone zero, 100 billion and likely to grow.

但开源模型实验室，部分原因是经济问题，作为快速追随者，你没有这种信息流，也没有这种收入流。

But the open model labs, partially this is like an economic issue being the fast follower, is that you don't have this flow of information and you don't have this flow of revenue.

所以有开源模型的构建者，还有开源模型的推理公司，以及开源模型的后训练公司。

So you have an open model builder, then you have the open model inference companies, and you have the open model post-training companies.

这些客户带来的信息，也就是你需要模型去完成的事情，这些信息并不会轻易地回流到模型构建者那里。

And there's information from those customers, which are things you need the models to do, and it's like that information doesn't just flow back to the model builder very easily.

而且他们也没有规模经济效应，无法服务那么多用户，无法那么高效地提供服务，也无法在推理上获得那么高的利润。

And they also don't have the economies of scale to serve as many users, to serve it as efficiently, to have as much margins on inference.

这些都需要更长的时间。

And these are just like it's going to take a lot longer.

但只要能实现商品化，公司似乎真的很想拥有私有数据。

But to the extent that things can be commoditized, it seems like companies really want to have private data.

有很多东西他们就是不能通过API发送，我们可以讨论零数据保留或者不保留等等。

There's a lot of things that they just cannot send over an API, and we can talk about the zero data retention or not and so on.

但这需要很长时间才能弄清楚一家公司是否有内部专业知识，或者他们是否学会了如何构建一个与收入真正匹配的内部评估体系，然后他们再学会如何微调，或者他们去找Thinking Machines或Prime Intellect，然后他们会说：“如果我们能学会微调一个模型来解决这个问题，我们就能大规模赚钱。”

But these it takes a long time to figure out if a company has internal expertise, or they figure out how to build an internal evaluation that actually matches to revenue, and then they figure out how to fine-tune, or then they go find thinking machines or prime intellect and they're like, "We could make money en masse if we figure out how to fine-tune a model that solves this."

他们确实有这些客户，但涉及的公司太多了，所以需要的时间要长得多。

And they have these customers, but there are just so many more companies involved that it takes so much longer.

但到了最后，他们真正拥有了模型权重，没有那些使用条款的限制，可以在自己的硬件上运行，而且他们也搞清楚了怎么从中获得投资回报。

But at the end of this process, they have real ownership, which is model weights, no terms and conditions that they can run on their hardware, and they've figured out they can make an ROI on.

所以我认为开放模型经济体的建立需要更长的时间。我对此并不太担心会出现泡沫，因为随着大家弄清楚哪种模型完成任务成本最低、以及如何通过利润率竞争，市场格局会快速演变。

So this is why I think the open model economy just takes way longer to build, and I'm less concerned about this being kind of bubbly, where it's like this could eat into the market of the really fast takeoff as people figure out what is the cheapest model that accomplishes a task, and how do you compete on margins.

说实话，我也不太清楚接下来会发生什么。

And I don't really know what's going to happen.

我觉得有-

I think there's-

现在还说不准。

... it's too early to know.

这太难了。

It's so hard.

大家脑子里一直盘旋着这个问题：通用推理能力一旦提升，作为通用模型，它肯定能胜过任何专用模型，数学领域就是个明显的例子。

There's definitely something that's always top of mind thinking about, like, there are things where scaling up general reasoning, just as a general model it's going to beat out any specialized model, and we see obviously examples in math.

这也是最能直观看到这种趋势的领域。

And that's probably the most pure domain that you can see this.

但即使是在机器人领域，比如Astra，不管这个信号有多真实，听起来他们好像并没有真正尝试过在机器人领域作为通用模型去突破。

But also, even in, say, robotics, with Astra, and for maybe however degrees of truthy the signal is, it just doesn't sound like they've actually tried to ascend in robotics as a general.

哦，我觉得他们试过。

Oh, I think they did.

我觉得这大概是六到八个月前的周期，他们开始招人，买了很多机器人方面的数据。

I think this is a cycle of like six to eight months ago, they started hiring people and buying a lot of data in robotics.

他们确实做了。

They did.

然后六个月后，模型就神奇地变得很擅长这个了。

And then the model, magically six months later, is really good at it.

但这就是为什么我觉得——

But this is why I think that the-

但我实际上——嗯，我听到的另一个更反面的说法是，这其实是从计算机使用中来的，因为计算机使用需要一定程度的空间推理，而能做出好的Blender演示又是那个能力的延伸。

But I actually-- Well, the other more contrary things I've heard is that it's actually coming out of computer use, because computer use is some level of spatial reasoning and then getting to have good Blender demos is another extension of that.

所以我们可以相信某种版本的说法。

So we can believe some version of that.

不过不管怎么说，我觉得如果是非常垂直细分的领域，能力的提升确实来自于在特定方向上的刻意优化。

But either way, I guess it's like maybe if it's really specialized, you see the capability gain is really from trying to get better in a certain area.

但在数学方面，我甚至不确定这是否也是通过专门针对数学进行优化得来的。

But in math, I don't even know if that fell out of trying to specialize in math.

我觉得当时的核心思路就是：干脆把推理能力拉满，结果不仅自然语言处理变强了，数学能力还超过了那些专门为此训练的模型。

I think the whole thesis there was just like, let's just scale up reasoning and actually got better in natural language, and got better than any specialized model in math.

Anthropic当时压根就没打算在数学上做到顶尖。

Anthropic wasn't even trying to be good at math.

OpenAI肯定招了不少数学家，但他们也没试图用那种精简的专业数据集来提升能力。

OpenAI, certainly, I think they hired a bunch of mathematicians, but they also weren't trying to use specialized, kind of lean data set to get better at it.

至少目前不是这么说的。

Or at least that's not the story.

所以总之，这都支持了一个观点：跟前沿模型竞争挺难的，因为它们最终会快速进步，直到你在各个细分领域里变得足够强。

So anyways, this is all kind of fodder for the view that it's kind of hard to compete with the frontier models, because eventually they start ascending so much that you get good enough in respective categories.

话虽如此，我觉得其实也存在一股对抗力量，也就是开源模型已经足够好，可以进行微调，这就让你提到的那些特定基准测试和数据集——这些原本只有大公司才有的资源——能够慢慢渗透扩散开来并得到应用。

With that said, I think actually there's also very competing, like a counterforce of data that you're seeing, which is like the open source models that are already good enough allow for fine-tunes, to your point of these slow percolation diffusion of these very specific benchmarks and data sets that are only available in these companies that you're being applied to.

效果已经相当不错了，而且我们确实看到大家开始采用这种方案了。

And it's just good enough, and it's actually we're starting to see people adopt that.

在最理想的情况下，也就是大家最期待前沿模型能发挥优势的场景里，我觉得 Cursor 夏天发过一篇博文，说他们接了一个长周期的任务——重写 SQLite。结果发现，用两个前沿模型，一个做规划，一个做执行，效果并不比直接用 Composer 配合 Opus 或者 Fastra 之类的模型做规划更好，甚至可能还差一点，但成本却高得多。

In the most maybe pure regime where you would expect frontier models are good is, I think Cursor published something over the summer where like, hey, we had this long horizon task of rewriting SQLite, and turns out using two frontier models, one for planner, one for execution, is not as- Good or is equivalently as good, but far more costly than just using Composer with maybe Opus or Fastra, whatever, as planner.

我认为这其实是个很好的证据，说明开源微调模型已经做得相当好，或者说足够好了，甚至在那些特别考验智能的任务上已经超越了前沿模型。

And I think that's actually great evidence that the open source fine-tunings are very good or good enough and has surpassed intelligence-hungry tasks.

所以我想问的是，你觉得哪些证据最能说服人？或者说，针对“前沿模型胜出”和“开源模型稳定发展”这两种不同情况，你有哪些比较有力的证据？

It's a long way of asking, what evidence do you find is most compelling or pieces of evidence for various versions of this stable state of where the frontier wins versus open source?

我非常认同，在很多无法完全验证的领域，前沿进展主要靠购买高质量数据，并给模型提供可以行动和逐步优化的环境。

I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and hill climb on.

我觉得主要有两点。

And I think there's two things.

一个是数学和代码这块，我们还不确定编程市场的规模到底有多大。

We have the math and code thing, and we don't know how big the coding market will be.

另一个就是其他所有领域，它们瞄准的是知识工作，比如生物、医疗、法律等，这些都不太容易验证，我觉得这跟OpenAI收购大量机器人数据的逻辑很像。

And then there's kind of everything else, which is still they're targeting this knowledge work, this biology, healthcare, legal, and all of these things are not as verifiable, and I think it is much closer to the story of OpenAI bought a lot of robotics data.

他们可能会告诉我们这是泛化能力，魔法般的效果。

They're probably going to tell us it's generalization, magic poof.

大家都喜欢这种神奇的泛化。

People love poof generalization.

但实际上，可能一部分是A，一部分是B，但很大程度上还是取决于数据。

But realistically, it's probably some of A, some in B, but a lot of it comes down to data.

我跟前沿实验室的人聊天时，发现有些人做着很棒的工作。

When I talk to people at Frontier Labs, there's some people that have amazing jobs.

有些人贡献的方式很专一，就是制作数据集，然后你——

There's some people that contribute by doing a very narrow thing, which is you make a data set and you-

……确定模型在那个领域应该有什么行为，然后想办法制作数据来帮助获得那个数据——

... figure out what the behavior should be for your model in that area, and you figure out how to make the data that helps get that data-

这是很重要的工作。

It's important work.

……来获得那种行为，然后你确定需要购买哪些数据。

... to get that behavior, and you figure out what data you need to buy.

而且这需要时间，我觉得我们会有好几年的时间都这样。

And that's going to take, I think we're going to have years of this-

……在各个行业里，而且买这些数据非常烧钱。

... across different sectors, and that's very capital intensive to buy all of this data.

但这可不是什么“砰”的一下，魔法般的RSI就出现了。

But that's not just like poof, magic, RSI happens.

我确实看到的是——

Where I do-

……在这些数学、代码以及一些分布式机器学习工程中——

... see in this math and code in some distributed ML engineering-

……这些是非常可验证的。

... it's very verifiable.

你知道你的成本是多少，而AI模型会以惊人的速度把成本降下来。

You know what your cost is, and the AI models are going to drive that down at a really magical rate.

这会带来很多额外的好处，我们也能更快地做研究。

And that's going to just have a lot of surplus benefits, and we can do research faster.

我特别感兴趣的一点是，我觉得模型效率的提升会是指数级的，持续好几年。

I'm very interested in, I think the efficiency of models is going to drop multiplicatively over years.

所以我们可以看到OpenAI大幅降价了，那这意味着像GPT-4o或者Astra这样的大模型，在未来三年里每八个月价格减半，到底意味着什么呢？

So we've seen OpenAI had drastic price cuts, and it's like, what does it mean for GPT Sol or Astra and these bigger models to reduce in price 2X every eight months for the next three years?

真是幸运啊，对吧？

Lucky us, right?

这时代真不错。

What a time.

你开始做这些计算，发现成本大概只有原来的百分之一，比如四年后就是这样。

You start doing this math and it's like it's 1% of the cost in whatever, like four years.

我就在想，他们会不会因为杰文斯悖论而获得更高的利润率？还是说市场饱和了，收入反而下降？

And it's like, are they going to be able to have higher margins due to Jevons paradox there, or their revenue's going to decrease because the market fills up?

这里面的动态确实挺奇怪的，但归根结底，AI的进步关键在于数据，以及编写更好的软件来服务你的模型。

There's just all these really weird dynamics, but fundamentally, AI progress is about data and about writing better software to serve your models.

这两者完全是两个不同的领域。

And those are very different worlds.

我同意你的看法，从外部来看，确实有很多做数据和强化学习环境的公司做得非常好。

I mean, I agree with you that observing from the outside, there's a lot of data RL environment companies that are doing very, very well.

他们不仅卖给顶尖的两家前沿实验室，也越来越多地卖给其他实验室。

Selling both to the top two Frontier labs, but an increasing kind of longer tail of labs as well.

所以你说得对，我觉得很多能力提升其实都是靠扩大规模实现的。

So to your point, I think a lot of the capability gain has really come from just scaling that up.

这可不是件容易的事。

And it's not an easy task.

我其实很好奇你对数据收集和手工搭建环境与模拟环境之间的看法。

I mean, I'm actually very curious on your opinions between the data collection and the environments that are built by hand versus the ones that are done in sim.

似乎不同版本的环境，价格也不一样，等等。

There seems to be different versions of what could be, what kind of price they command, et cetera.

它们各有优缺点。

They all have good things and bad things.

我觉得有很多讨论，比如我们刚才提到的蒸馏，但另一方面，中国公司也在购买强化学习环境——

And I think there's a lot of talk of, like we've talked about distillation, but the other side is the Chinese company buys the RL environment-

是的

Yes

……他们六个月后才用。

... that they use six months later.

没错。

Exactly.

而且就像——

And like-

哦，Anthropic或OpenAI买了这个。

Oh, Anthropic or OpenAI bought this.

现在就有溢价了。

Now there's a premium.

这话我听过太多次了。

I've heard that so many times

……

...

优质数据能做到这一点。

the good data will do this.

但人们买到的数据其实问题挺多的。

But the data that people are buying is pretty flawed.

我觉得方法就是大批量采购，然后筛选出有效的部分，命中率不可能百分之百。

I would say that the method is you buy in mass and you figure out what works, and it's not 100% hit rate.

我个人认为，这种强化学习环境带来的赚钱热潮，让很多人为了利益偷工减料，这就容易引发奖励黑客攻击之类的问题。

And I personally think that this RL environment explosion, where there's just so much opportunity to make money, is having a lot of people cut corners, and this can induce things like reward hacking.

所以就像这些模型太聪明了，如果把它们放进一个环境里，它们的任务是解决某个问题，但你没给它们……

So it's like the models are so smart that if you put them in a world and they're like, "My job is to solve this task," but they didn't give me the...

你给了个测试文件，结果那个文件是空的，而不是你需要去探索的迷宫。

You have a test file, and the test file actually has an empty file instead of the maze that you have to navigate.

然后模型就会想，不管怎样，我还是得想办法拿到高分。

And they're like, the model is like, "Well, I should still figure out how to get a high reward anyways."

我觉得很多这类行为都是这种供应链导致的。

And I think a lot of this behavior is downstream of this supply chain.

我觉得这个问题会得到解决，但有趣的是，实验室对消除奖励黑客的需求很大，而且所有研究人员都花了很多时间在这上面。

And I think it'll fix it, but it's interesting to see the demand from the labs to eliminate reward hacking and all these researchers spending so much time on it.

然后所有数据公司都说，“买我们的数据吧”，我觉得他们花在模型行为上的时间相对较少——

And then all the data companies like, "Buy our data," and I think proportionally spend less time on model behavior-

……更多的时间花在构建世界和收入上。

... and more time on world-building and revenue.

不知怎么的，这两方需要合作，才能真正做出更高质量的数据。

And somehow these two parties need to meet together to actually make higher quality data.

其实很多实验室都在内部处理这个问题，部分原因是他们不想——

Well, a lot of the labs are actually taking it in-house, partially because they don't want-

……数据泄露。

... data leaking.

不只是字面意义上的，也是比喻意义上的，就像他们正在做什么、试图传递什么信号的泄露。

Not that, literally and figuratively speaking, it's like signals leaking of what they're doing, what they're trying to send on.

但尽管如此，我认为他们的购买量还是有了巨大的增长。

But despite that, I think there's a tremendous amount of growth in how much they're purchasing.

你觉得那可能是最强的杠杆之一吗？

Do you think that's probably one of the strongest levers?

你能大概说说这会增长多少吗？

Can you kind of qualify how much that will kind of increase?

你预计这会随着支出增加而增长吗？

Do you expect that to increase based on spending?

还是有其他因素？比如，如果模拟能做得很好，那可能更说明这会被实验室内部消化，然后——

Or are there other, because for instance, if it can be done well in sim, I think maybe that is more of an argument to that being absorbed by labs in particular, and then-

所有公司都在混合使用……

All the companies are doing a mix of...

所有公司都在卖合成数据，他们用语言模型来修改或构建世界，或者生成它的各种变体。

All the companies are selling synthetic data, where they're using language models to modify or build the world or generate permutations of it.

如果有人在指导这个过程，那仍然是合成数据。

And if a human is guiding this process, it's still synthetic data.

所以，很少有人会坐在那里，然后……我们给模型造一个小世界让它去活动。

So, there's very few people that are going to sit there and lender- ... and we're going to make a little world for our model to act in.