#490 – 2026年人工智能现状：大语言模型、编程、扩展法则、中国、智能体、GPU、通用人工智能

我在这儿都快疯了，你居然用路由器和非思考模型。

I'm like saying over here losing my mind that you use the router and the non thinking model.

我是说，你怎么能忍受这样？你怎么能忍受这个？

I'm like, how do you how do you live with how do you live with that?

是的。

Yeah.

这就像我的反应。

It's like my reaction.

我最近一直陷在悲剧里。

I'm been heavily on tragedy for a while.

我从来没碰过那个非思考模型。

Never touched five non thinking.

我发现它的语气和错误倾向。

I find it it's tone and then its propensity of errors.

它就是更容易出错。

It's just like it has a higher likelihood of errors.

这部分内容源于OpenAI发布o3的时候，那是第一个能够进行深度搜索、查找多个来源并为你整合信息的模型。

Some of this is from back when OpenAI released o three, which was the first model to do this deep search and find many sources and integrate them for you.

所以我逐渐习惯了这种方式。

So I became habituated with that.

因此，只要是我需要查找任何工作相关的资料，无论是论文还是代码参考，我都只使用GPT 5.2的思考模式或专业模式。

So I will only use GPT 5.2 thinking or pro when I'm finding any sort of information query for work, whether that's a paper or some code reference that I found.

我经常同时运行五个专业模式的查询，每个查询都在寻找某一篇特定的论文，或者对某个公式进行反馈。

And it's just like, I I will regularly have, like, five pro queries going simultaneously, each looking for one specific paper or feedback on an equation or something.

我有个有趣的例子，当时我需要尽快给出一个答案。

I have a fun example where I just needed to answer as fast as possible.

在去旅行前录制这个播客时，我家里有一台本地GPU，我想运行一个长时间的强化学习实验。

For this podcast before I was going on the trip, I have a local GPU running at home, and I wanted to run a long RL experiment.

通常我也会拔掉电源，因为你永远不知道自己不在家时会发生什么。

And usually, I also unplug things because you never know if you're not at home.

我不想让设备一直插着电，结果我不小心把GPU的电源拔掉了。

Don't wanna have things plugged in, And I accidentally unplugged the GP.

我妻子已经上车了，我当时就想：‘糟了。’

It was like my wife was already in the car, and it's like, oh, dang.

然后，我急需一个能快速运行我不同实验和评估的 Bash 脚本。

And then, basically, I wanted as fast as possible a Bash script that runs my different experiments and the evaluation.

我确实知道怎么用 Bash 界面或 Bash 终端。

And did something I know I learned how to use the Bash interface or Bash terminal.

但那一刻，我只需要十秒钟内给我

But in that moment, I just needed, like, ten seconds to give me

那个命令。

the command.

这真是个搞笑的情况，不过确实如此。

This is a hilarious situation, but yeah.

那你用了什么？

So what did you use?

于是我用了那个不经过思考的最快模型。

So I did the nonthinking fastest model.

它给了我一个Bash命令，可以把不同的脚本串联起来。

It gave me the Bash command I to chain different scripts to each other.

然后问题是，你得把输出重定向到日志文件。

And then the thing is, like, you have the t thing where you want to route this to a log file.

当时我脑子里一片空白，只是赶时间。

Top of my head, I was just, like, in a hurry.

我本可以自己想出来的。

I could have thought about it myself.

顺便说一下，我不确定有没有一个典型的例子。

By the way, I don't know if there's a representative case.

老婆在车里等着，你得赶紧跑。

Wife waiting in the car, have to run

你知道的，插上GPU。

you know, plug the GPU.

你得

You have

生成一个bash脚本。

to generate a bash script.

这听起来像一部电影。

This sounds like a movie.

我是说《碟中谍》。

I I Mission Impossible.

我用Gemini来做这个。

I use Gemini for that.

所以我用思考功能处理所有信息类任务，而Gemini则用于快速处理或那些我本可以去谷歌搜索的事情——它擅长解释内容，我信任它具备这样的知识背景，而且操作简单。

So I use thinking for all the information stuff, and then Gemini for fast things or stuff that I've could come time to Google, which is like it's good at explaining things, and I trust that it has this kind of background of knowledge, and it's simple.

Gemini应用现在进步了很多，很适合做这类事情。

And the Gemini app has gotten a lot better, and it's good for that sort of things.

至于代码和任何哲学类讨论，我使用Claude Opus 4.5。

And then for code and any sort of philosophical discussion, I use Claude Opus 4.5.

而且，我总是开启深度思考模式。

Also, always with extended thinking.

扩展思考和推理时间缩放只是让模型稍微更聪明的一种方式。

Extended thinking and inference time scaling is just a way to make the models marginally smarter.

当进展非常显著时，我总会倾向于选择这一侧，因为你不知道什么时候会解锁新的使用场景。

And I will always edge on that side when the progress is very high because you don't know when that'll unlock a new use case.

然后我会偶尔用Grok来获取实时信息，或者找我在AI推特上看到过但需要挖出来、并且一直惦记着的内容。

And then sometimes use Grock for real time information or finding something on AI Twitter that I knew I saw and I need to dig up and I just fixated on.

尽管Grok 4发布时，Grok 4非常臃肿，但它的专业版实际上表现非常好。

Although when Grok four came out, the Grok four was a super heavy, which was like their pro variant was actually very good.

我对它印象深刻。

And I was pretty impressed with it.

我当时只是出于肌肉记忆，因为开着Chat2bt应用，就忽略了它。

And I was just kind of like muscle memory lost track of it with having the Chat2bt app open.

所以我用很多不同的工具。

So I use many different things.

是的。

Yeah.

我确实会用Grunt四号重型版来调试。

I actually do use Grunt four heavy for debugging.

对于那种极端的调试任务，其他工具都解决不了的时候，我发现它是最出色的，这很有趣，因为你说ChatGPT对你来说也是因为同样的原因最好，但这可能只是惯性使然。

For, like, hardcore debugging and the other ones can't solve it, I find that it's the best at and I it it's interesting because you say ChatGPT is the best interface for me for that same reason, but this could be just Momentum.

嗯。

Mhmm.

是的。

Yeah.

对我而言，Gemini是更好的界面。

Gemini is the better interface for me.

我觉得是因为我爱上了他们最擅长的‘大海捞针’功能。

I think because I fell in love with their best needle in the haystack.

如果我输入一些内容量很大的信息，但只想找到非常具体的细节，确保它能追踪到所有内容，我发现至少对我来说，Gemini是最出色的。

If I ever put something that has a lot of context, but I'm looking for very specific kinds of information to make sure it tracks all of it, I find at least the Gemini for me has been the best.

所以这些模型中，有些如果赢得了你的心，就挺有意思的。

So it's funny with some of these models, if they win your heart over

嗯。

Mhmm.

对于某个特定功能、某一天、某个特定查询或提示，你会觉得这个模型更好。

For one particular feature at one on a one particular day for that particular query, that prompt, you're like, this model is better.

于是你会暂时坚持用它，直到它做出一些非常愚蠢的事情。

And so you'll just stick with it for a bit until it does something really dumb.

这就像有一种阈值效应。

There's like a threshold effect.

它做了一件聪明的事，让你爱上它，然后它又做了一件蠢事，你就想：你知道吗？

Some smart thing, and then you fall in love with it, and then it does some dumb thing, and you're like, you know what?

我要换掉它，试试Clawd、试试GPT之类的。

I'm gonna switch and try Clawd and try GPT and all that kind of stuff.

这就像你一直用它，直到它出问题，然后你才换掉语言模型。

This is exactly like you use it until it breaks, until you have a problem, and then then you change the LM.

我觉得这和我们使用任何东西都一样，比如最喜爱的文本编辑器、操作系统或浏览器。

And I think it's the same how we use anything, like our favorite text editor operating systems or the browser.

我的意思是，浏览器的选择太多了，Safari、Firefox、Chrome，它们都差不多，但有时候你会因为某些特殊的扩展功能而切换。

I mean, there are so many browser options, Safari, Firefox, Chrome, all the relatively similar, but then there are ex edge cases, maybe extensions you wanna use, and then you switch.

但我不认为有人会把同样的网址输入到不同浏览器里去比较它们。

But I don't think there is anyone who types the same thing, like the website, into different browsers and compares them.

我想，你只有在网站无法正常显示、出问题的时候才会这么做。

You only do that when the website doesn't render if something breaks, I think.

这确实是个不错的观点。

So that's that's a good point.

我觉得你是用到它出问题了，才会去尝试其他选择，我觉得是这样。

I think you use it until it breaks, and then you explore other options, I think.

关于长上下文这件事，我之前也是Gemini的用户。

On the long context thing, I was also a Gemini user for this.

但GPT 5.2的发布博客里，长上下文的评分高得离谱，很多人都在问：他们是不是发现了什么算法上的突破？

But the GPT 5.2 release blog had, like, crazy long context scores where a lot of people were like, did they just figure out some algorithmic change?

在一次小版本更新中，它的分数从大约30%直接飙升到了70%左右。

It went from, like, 30% to, like, 70 or something in this minor model update.

所以要跟踪所有这些事情也非常困难。

So it's also very hard to keep track of all of these things.

但现在我对GPT五点二的长上下文有了更积极的看法。

But now I'm look more favorably at GPT five point two's long context.

我只是在想，我到底该怎么去测试它呢？

It's just kinda like, how do I actually get to testing this?

永无止境的战斗。

Never ending battle.

有趣的是，我们都没从用户使用角度讨论过中国模型。

It's interesting that none of us talked about the Chinese models from a user usage perspective.

这说明了什么？

What does that say?

这意味着中国模型不够好，还是说我们只是非常有偏见，过于关注美国？

Does that mean the Chinese models are not as good, or does that mean we're just very biased and US focused?

我认为这目前反映了模型与平台之间的差异。

I do think that that's currently the discrepancy between just the model and the platform.

所以我认为，开源模型目前更出名的是它们的开源权重，而不是它们的平台。

So I I think the open models, they are more known for the open weights, not their platform yet.

还有很多公司愿意以非常低的成本向你提供开源模型的推理服务。

There are also a lot of companies that are willing to sell you the open model inference at a very low cost.

我认为，像 OpenRouter 这样的平台，很容易实现多模型的查看和使用。

I think, like, OpenRouter, it's easy to do the look at multi model things.

你可以在 Perplexity 上运行 DeepSeek。

You could run DeepSeek on Perplexity.

我想我们所有人坐在这里，都一直在持续使用 OpenAI 的 GPT-4 Pro。

Think I all of us sitting here are like, we use OpenAI GPT five Pro very consistently.

我们都愿意为那一点点智能提升付费。

We're all willing to pay for the marginal intelligence gain.

那些认为美国这些模型在输出质量上更优的人。

And anyone that's like, the these models from The US are better in in terms of the outputs.

我想问题是，它们在未来这一年乃至多年里，还能继续保持优势吗？

I think that the question is, will they stay better for this year and for years going?

但只要它们更好，我就愿意付费使用。

But it's like, so long as they're better, I'm gonna pay for it to use them.

我认为还有一些分析表明，中国模型的部署方式——无论这是否源于专家控制——导致它们在复制时使用的GPU更少，因此速度更慢，错误也不同。

I think there's also analysis that shows that, like, the way that the Chinese models are served, this you could argue due to expert controls or not, is that they use fewer GPUs for replica, which makes them slower and have different errors.

这关乎速度和智能。

And it's like the speed and intelligence.

如果这些优势对你作为用户有利，我认为在美国，很多用户都会选择它们。

If these things are in your favor as a user, I think in The US, a lot of users will go for this.

我认为这会促使中国公司以其他方式竞争，比如提供免费服务或大幅降低成本，或者激发服务创新，这对整个生态系统是有益的。

And I think that that is something that will spur these Chinese companies to want to compete in other ways, whether it's, like, free or substantially lower costs or it'll breed creativity in terms of offerings, which is good for the ecosystem.

但我只是觉得，简单来说，美国的模型目前更好，我们就在用它们。

But I just think the simple thing is that US models are currently better, and we use them.

我试过中国模型和其他开源模型，觉得挺有趣，但不会回头用。

And I try Chinese I try these other open models, and I'm like, fun, but not gonna I don't go back to it.

我们其实没怎么提到编程。

We didn't really mention programming.

这是很多人非常关心的另一个使用场景。

That's another use case that a lot of people deeply care about.

所以我基本上一半用Cursor，一半用Codium，因为我觉得它们的体验根本不同，而且都挺有用。

So I use basically half and half cursor and clogged code, because there I find them to be, like, fundamentally different experience and both useful.

你们经常编程吗？你们平时用什么？

What do you guys you program quite a bit, so what what do you use?

现在的趋势怎么样？

What's the current vibe?

我用的是Versus Code的Codex插件。

So I use the Codex plug in for Versus Code.

你知道，这非常方便。

You know, it's very convenient.

它就是一个插件，然后提供一个可以访问你代码仓库的聊天界面。

It's just like a plug in, and then it's a chat interface that has access to your repository.

我知道Cloud Code可能有点不一样。

I know that Cloud Code is, I think, a bit different.

它更具主动性。

It's a bit more agentic.

它会涉及更多内容。

It touches more things.

它能为你完成整个项目。

It does a whole project for you.

我还没到能完全放心使用那一步，也许我是个控制狂，但我还是想稍微看看发生了什么。

I'm not quite there yet where I'm comfortable with that because maybe I'm a control freak, but I still would like to see a bit what's going on.

对我而言，Codex 目前正是最佳平衡点：它在帮助我，但并没有完全接管一切。

And Codex is kind of like right now for me, like, the sweet spot where it is helping me, but it is not taking completely over.

我应该提一下，我使用 Claude Code 的一个原因是培养用英语编程的能力。

I should mention one of the reasons I do use Claude code is to build the skill of programming with English.

我的意思是，这种体验本质上是不同的。

I mean, the experience is fundamentally different.

你不是在微观管理代码生成过程的细节，也不是查看差异（如果你用的是 Cursor，可以这么做），而是在不断修改、调整、阅读和深入理解代码；相比之下，你只是在设计空间里大致思考一下。

You're, as opposed to micromanaging the details of the process of the generation of the code and looking at the diff, which you can in cursor, if that's the ID you use, and and in changing, altering, looking and reading the code and understanding the code deeply as you progress versus just kinda like thinking in this design space Mhmm.

而且在这个宏观层面上进行引导，我认为这是另一种思考编程过程的方式。

And just guiding it at this macro level, which I think is another way of thinking about the programming process.

另外，我们应该提到，Cloud Code 似乎在某种程度上更好地利用了 Cloud Opus 四五。

Also, we should say that Cloud Code, it just seems to be somehow a better utilization of Cloud Opus four five.

这对人们来说是一个很好的对比方式。

It's a good side by side for people to do.

你可以同时打开 Cloud Code、Cursor 和 VS Code，让它们使用相同的模型，提出相同的问题，这非常有趣。

So you can have Cloud Code open, you can have Cursor open, and you can have Versus Code open, and you can select the same models on all of them and ask questions, and it's very interesting.

比如，Cloud Code 在这个领域要好得多。

Like, the like, Cloud Code is way better in that domain.

这很了不起。

It's remarkable.

好的。

Alright.

我们应该说，你们两位在多个方面都是真正的专家：研究人员、程序员、教育者、推特用户，甚至在出书方面也是如此。

We should say that both of you are legit on multiple fronts, researchers, programmers, educators, tweeterers, and on the book front too.

所以，内森，很快，希望如此，他将出版一本关于RLHF的书。

So Nathan, at some point soon, hopefully, has an RLHF book coming out.

这本书已经可以预购了，而且有一个完整的数字预印本，正在为实体版做美化和更有序的整理，这正是我这么做的主要原因——因为当我们的生活大部分都是数字化的时候，创造一些在实体形式上非常出色的东西是很有趣的。

It's available for preorder, and there's a full digital preprint just making it pretty and better organized for the physical thing, which is a lot of why I do it because it's fun to create things that you think are excellent in the physical form when so much of our life is digital.

我得提一下，去Perplexity看看，塞巴斯蒂安·罗什卡是一位机器学习研究员和作者，以多本有影响力的书籍而闻名。

I should say, going to perplexity here, Sebastian Roshka is a machine learning researcher and author known for several influential books.

我想提到其中几本，我强烈推荐《从零开始构建大语言模型》和新书《从零开始构建推理模型》。

A couple of them that I wanted to mention, which is a book I highly recommend, build a large language model from scratch, and the new one, build a reasoning model from scratch.

所以我对这本书非常期待。

So I'm really excited about that.

从零开始构建东西是学习最有效的方式之一。

Building stuff from scratch is one of the most powerful ways of learning.

老实说，从零开始构建一个元素非常有趣。

Honestly, building an element from scratch is a lot of fun.

同时，这也是学习很多东西的好方法。

It's also a lot of to learn.

就像你所说的，这可能是了解某事物真正工作原理的最佳方式，因为你虽然可以看图示，但图示也可能有错误。

And like you said, it's probably the best way to learn how something really works because you can look at figures, but figures can have mistakes.

你可以看概念和解释，但你可能会误解它们。

You can look of con concepts, explanations, but you might misunderstand them.

但如果你看到代码始终存在，而且代码能运行，你就知道它是正确的。

But if you see the always there is code and the code works, you know it's correct.

我的意思是，这里不存在误解。

I mean, there's no misunderstanding.

它是精确的。

It's like it's precise.

否则，它根本无法运行。

Otherwise, it wouldn't work.

我认为这正是编码的魅力所在。

And I think that's, like, kind of like the beauty behind coding.

它确实像从不撒谎一样。

It is kind of like it doesn't lie.

这基本上就是数学。

It's math, basically.

所以即使在数学中，我认为你在看书时也可能遇到错误，却永远发现不了。

So even though with math, I think, you can have mistakes in a book you would never notice.

因为你读书的时候并没有运行数学，无法验证它。

Because you're not running the math when you are reading the book, you can't verify this.

而代码的好处在于你可以验证它。

And with code, what's what's nice is you can verify it.

是的。

Yeah.

我同意你关于从零开始学习语言模型这本书的看法。

I agree with you about the LM from scratch book.

把其他一切——互联网等等——都屏蔽掉，专心看书，这很好。

It's nice to tune out everything else, the Internet and so on, and just focus on the book.

但我读过好几本历史类的书。

But, you know, I've read several, like, you know, history books.

不知怎么的，感觉没那么孤单了。

It's just less lonely somehow.

真的更有趣了。

It's really more fun.

比如在编程方面，我觉得用大语言模型编程确实更有趣。

Like, for example, on the programming front, I think it's genuinely more fun to program with an LLM.

嗯哼。

Mhmm.

我觉得用大语言模型阅读也确实更有趣。

And I think it's genuinely more fun to read with an LLM.

嗯哼。

Mhmm.

但你说得对。

But you're right.

这种干扰确实应该尽量减少。

Like, this distraction should be minimized.

所以你是用LLM来丰富体验，也许增加更多背景信息，或者对我来说，小规模内与LLM互动的高光时刻确实很多。

So it's you use the LLM to basically enrich the experience, maybe add more context, maybe the I just the rate of moments for me in a small scale is really high with LLMs.

完全同意。

100%.

我也想纠正一下自己。

I would I also want to correct myself.

我不是建议不要使用LLM。

I'm not suggesting not to use LLMs.

我建议分多次进行，比如第一次用离线专注模式。

I suggest doing it in multiple passes, like one pass just offline focus mode.

之后，我会做笔记，但我会尽量克制立刻去查资料的冲动。

And then after that, I mean, I also take notes, but I I try to resist the urge to immediately look things up.

我会进行第二次阅读。

I I do a second pass.

对我来说，这样更有条理，有时候答案就在章节里，但有时候也让信息沉淀下来，好好思考一下。

It's just, like, for me, more structured this way, and I get let I mean, sometimes things are answered in the chapter, but sometimes also it just helps to let it sink in and think about it.

其他人有不同的偏好。

Other people have different preferences.

我强烈推荐在读书时使用大语言模型。

I would highly recommend using LLMs when reading books.

对我来说，这并不是首先要做的事。

For me, it's just it's not the first thing to do.

这更像是第二遍。

It's like the second pass.

作为建议，我要说，我恰恰相反。

By way of recommendation is to say, I do the opposite.

我喜欢在一开始使用大语言模型。

I like to use the LLM at the beginning Mhmm.

来全面了解我即将进入的这个世界是什么样的？

To lay out the full context of, like, what is this world that I'm now stepping into?

但我尽量避免从大语言模型跳转到Twitter和博客这类外部世界，因为那样你会陷入信息的漩涡。

But I try to avoid clicking out of the LLM into the world of, like, Twitter and blogs, and because then you're now down this rabbit hole.

你正在阅读别人的观点。

You're reading somebody's opinion.

某个话题引发了激烈争论，突然间，你不再是你自己，而是进入了互联网和Reddit等世界。

There's a flame war about a particular topic, and all of sudden, you're no longer you're now in the in the realm of the Internet and Reddit and so on.

但如果你只是让LLM告诉你为什么这很重要、有哪些宏观理念，有时书籍本身也能做到这一点，但并不总是如此。

But if you're purely letting the LLM give you the context of why this matters, what are the big picture ideas, but sometimes books themselves are good at doing that, but not always.

所以这就是我喜欢ChatGPT应用的原因，因为它把AI变成了你电脑上的一个专属空间，让你可以专注于它，而不是让它只是我众多网络选项中的一个标签页。

So That's why I like the ChatGPT app, because it gives the AI a home in your computer when you are folk you can focus on it rather than just being another tab in my mess of Internet options.

我认为Claude Code这类工具特别擅长让这个过程变得愉快，它看起来像是一个精心设计的界面，你的AI可以从这里出发去探索世界。

And I think Claude Code and these, in particular, does a good job of making that a joy, where it seems very engaging as a product designed to be an interface that your AI will then go out into the world.

在它和Codex之间有一种难以言喻的差异，那就是它给人的感觉温暖而吸引人，而Codex虽然同样优秀，但总显得有点粗糙。

And it's something that is very kind of intangible between it and Codex is that it just feels kind of warm and engaging, where Codex can often be as good from OpenAI, but it just kind of, like, feels a little bit rougher on the edges.

相比之下，Cloud Code让从零开始构建东西变得有趣，你根本不需要操心，但你相信它能创造出一些东西。

Whereas, like, Cloud Code is makes it fun to build things, particularly from scratch where you just don't like, you don't have to care, but you trust that it'll make something.

显然，这对网站开发和更新工具之类的事情很有帮助，而我用它来做数据分析。

Like, obviously, this is good for websites and kind of refreshing tooling and stuff like this, which I use it for data analysis.

所以我的博客会抓取 Hugging Face 的数据。

So I my my blog, we scrape Hugging Face.

我们现在持续追踪每个数据集和模型的下载量变化，因此我们有了这些数据。

We keep the download numbers for every dataset and model over time now, so we have them.

当时 Cloud 就说，是的。

And it's like, Cloud was just like, yeah.

我确实用过这些数据。

I've made use of that data.

没问题。

No problem.

我当时想，这本来得花我好几天时间。

And I was like, that would have taken me days.

然后我意识到，自己有足够的背景认知，心想：好吧。

And I was like and then I have enough situational awareness to be like, okay.

这些趋势显然说得通，你可以去核实一下。

These trends obviously make sense and you can check things.

因为这种界面非常出色，你可以通过一个中间层来操作，而无需去做那些繁琐的底层工作，比如维护不同的网页项目并完成这些任务。

Because that's just the kind of wonderful interface where you can have an intermediary and not have to do the kind of awful low level work that you would have to do to maintain different web projects and do this stuff.

好的。

Alright.

所以我们刚刚讨论了很多闭源权重模型。

So we just talked about a bunch of the closed weight models.

现在来聊聊开源的模型吧。

Let's talk about the open ones.

给我讲讲 OpenLM 模型的现状吧。

So tell me about the landscape of OpenLM models.

有哪些有趣的模型？

Which are interesting ones?

哪些模型让你印象深刻？为什么？

Which stand out to you, and why?

我们已经提到过 DeepSeq 了。

We already mentioned DeepSeq.

你想看看我们能一口气说出多少个吗？

Do you wanna see how many we can name off the top of our head?

是的。

Yeah.

对。

Yeah.

不用看笔记。

Without looking at notes.

DeepSeq、Kimmy、MiniMax、Z dot a I、Antlang。

DeepSeq, Kimmy, MiniMax, Z dot a I, Antlang.

我们这纯粹是在列中国公司的名字。

We're just going Chinese.

我们再加上Mistral AI Gemma吧。

Let's throw in Mistral AI Gemma.

好。

Yeah.

GPT OSS，由Jet GPT开发的开源模型。

GPT OSS, the open source model by Jet GPT.

实际上，英伟达的Nemotron系列有一个非常酷的模型，叫Nematron three。

Actually, NVIDIA Nemotron had a or NVIDIA had a really cool one, Nematron three.

有很多东西，尤其是在年底的时候。

There there's a lot of stuff, especially at the end of the year.

Quen one可能是其中一个。

Quen one may be the one

哦，是的。

Oh, yeah.

Quen是我当时在听的明显名字。

Quen was the the obvious name I was listening.

我试图至少找出10个中国模型和10个西方模型。

I was trying to get to the you can get at least 10 Chinese and at least 10 Western.

我认为，OpenAI自GPT-2以来发布了他们的第一个开源模型。

I think that I mean, OpenAI released their first open model since GPT two.

当我提到我写关于OpenAI开源模型发布时，他们都说别忘了GPT-2，我觉得这特别有趣，因为那已经是完全不同的时代了。

That when I when I meant talk when I was writing about OpenAI's open model release, they're all like, don't forget about GPT two, which I thought was really funny because it's just such a different time.

但GPT OSS实际上是一个非常强大的模型，能完成其他模型不太擅长的一些任务。

But GPT OSS is actually a very strong model and does some things that the other models don't do very well.

而且，自私地说，我会大力推广一些西方公司。

And I think that selfishly, I'll promote a bunch of, like, Western companies.

所以美国和欧洲都有这些完全开源的模型。

So both in The US and Europe have these, like, fully open models.

我在艾伦人工智能研究所工作，我们一直在开发OMO，发布数据、代码等所有内容。

So I work at Allen Institute for AI where we've been building OMO, which releases data and code and all of this.

现在，我们有了真正的竞争对手，他们也努力发布一切，以便其他人能训练这些模型。

And now we have actual competition for people that are trying to release everything so that other people can train these models.

还有基础模型研究所或LM360，他们发布了各种类型的K2模型。

So there's the Institute for Foundation Models or slash LM three sixty, which is, like, had their k two models of various types.

Apertus是一个瑞士研究联盟。

Apertus is a Swiss research consortium.

Hugging Face 的小语言模型非常流行。

Hugging Face has small lm, which is very popular.

NVIDIA 的 NeMo 已经开始发布数据了。

NVIDIA's Neematron has started releasing data as well.

斯坦福的 Marin 社区项目正在建立一种流程，让人们可以提交 GitHub 问题、实现新想法，并将其集成到稳定的语言建模框架中。

And then Stanford's Marin community project, which is kind of making it so there's a pipeline for people to open a GitHub issue and implement a new idea and then have it run-in a stable language modeling stack.

所以这个领域在 2024 年时，这个列表要小得多。

So this space, that list was way smaller in 2024.

所以那时候可能就只有 AI2。

So I think it was like just AI two.

这让更多人能够参与进来并理解语言模型，而中国公司目前还没有类似的对应项目。

So that's a great thing for more people to get involved and to understand language models, which doesn't really have a, like, a Chinese company that is has an analog.

在我说这些的时候，我想指出，中国的开源语言模型通常规模更大，这带来了更高的峰值性能，比如 MoE 模型；而我们很喜欢的很多模型，比如 Gemma 和 NeMo，大多是美国的小型模型，不过现在美国和欧洲的情况也开始发生变化。

While I'm talking, I'll say that the Chinese open language models tend to be much bigger, and that gives some of this higher peak performance as MOEs, where a lot of these things that we like a lot, whether it was Gemma and Nematron, have tended to be smaller models from The US, which is which is starting to change from US US and Europe.

Mistral Large 3 在十二月发布了，这是一个巨大的 MoE 模型，架构与 DeepSeek 非常相似。

Mistral Large three came out, which was a giant MOE model, very similar to DeepSeek architecture in December.

然后一家初创公司RCAI以及Nematron和NVIDIA都透露了参数规模超过一万亿的MOE模型，预计在2026年第一季度左右会推出高达四万亿参数的模型。

And then a startup RCAI and both Nematron have Nematron and NVIDIA have teased MOE models of this way bigger than a 100,000,000,000 parameters, like this 400,000,000,000 parameter range coming in this like q one twenty twenty six timeline.

所以我认为今年这种平衡将发生变化，人们使用中国和美国开源模型的用途将有所不同，我个人对此非常期待。

So I think this kind of balance is set to change this year in terms of what people are using the Chinese versus US open models for, which will be which I'm personally, so I can just be very excited to watch.

首先，能说出这么多模型的名字，真是非常厉害。

First of all, huge props for being able to name so many of these.

你真的提到Lama了吗？

Did you actually name Lama?

没有。

No.

我觉得Lama完了。

I feel like RIP.

这并不是故意的。

This was not on purpose.

Lama完了？

RIP Lama?

是的

Mhmm.

好的

Alright.

你能说说有哪些值得注意的模型吗？

Can you mention what are some interesting models that stand out?

你提到了Qwen 3，这显然是一个突出的模型。

So you mentioned QUEN three is is is obviously a standout.

我觉得这一年几乎被DeepSeek V3和R1以及12月的DeepSeek V3.2所贯穿，我喜欢这些模型的原因是它们总有一些其他人没有的有趣架构调整。

So I would say the year is almost bookended by both DeepSeek version three and r one, and then on the other hand, in December, DeepSeek version 3.2 because what I like about those is they always have an interesting architecture tweak that others don't have.

但除此之外，如果你想选一些熟悉但性能非常出色的模型，那就是Qwen 3，就像内梅特说的，还有GPD OSS。

But otherwise, if you wanna go with, you know, like, familiar but really good performance, QN three, and like Nathan said, also GPD OSS.

我觉得GPD OSS有趣的地方在于，它是第一个真正以工具使用为设计目标的公开或开源权重模型，我认为这确实是一种小小的范式转变，因为当时的生态系统还没准备好迎接它。

And I think GPD OSS, what's interesting about it is kind of like the first public or, like, open weight model that was really trained with tool use in mind, which I do think is kind of a little bit of a paradigm shift where the ecosystem was not quite ready for it.

所谓工具使用，指的是大语言模型能够进行网页搜索或调用Python解释器。

So with tool use, I mean that the LLM is able to do a web search to call a Python interpreter.

我认为这确实很突出，因为它带来了巨大的突破，因为人们对大语言模型最常见的抱怨之一就是幻觉问题。

And I do think this it's a standout because I think it's a huge unlock because one of the most common complaints about LLMs are, for example, hallucinations.

对吧？

Right?

嗯。

Mhmm.

所以在我看来，解决幻觉问题最好的方法之一就是不要试图总是记住信息或凭空捏造。

And so in my opinion, one of the best ways to solve hallucinations is to not try to always remember information or make things up.

做数学运算时，为什么不直接用计算器或Python呢？

For math, why not use a calculator app or Python?

嗯。

Mhmm.

如果我问大语言模型1998年足球世界杯的冠军是谁，它不必死记硬背，而是可以直接去搜索。

If I ask the LLM who won the, I don't know, soccer World Cup in 1998, instead of just trying to memorize, it could go do a search.

我认为，大多数情况下，这通常还是通过谷歌搜索来实现的。

I think mostly, it's usually still a Google search.

所以JGPD和GPTOSS会调用工具去搜索Google，可能找到FIFA官网，然后获取信息。

So JGPD and GPTOSS, they would do a tool call to Google, maybe find the FIFA website, find okay.

是法国赢了。

It was France.

这样能可靠地为你提供信息，而不是试图死记硬背。

It would get you that information reliably instead of just trying to memorize it.

所以我认为这是一个巨大的突破，但目前开源和开放权重生态体系还没有充分运用它。

So I I think it's a huge unlock, which I think right now is not fully utilized yet by the open source, open weight ecosystem.

很多人不使用工具调用模式，因为首先这是一个信任问题。

A lot of people don't use tool call modes because I think it's first, it's a trust thing.

你不想在自己的电脑上运行这种可以访问工具、可能删除你硬盘数据的程序。

You don't wanna run this on your computer where it has access to tools, could wipe your hard drive or whatever.

所以你可能希望将它容器化隔离起来。

So you wanna maybe contain containerize that.

但我确实认为，未来几年拥有这种能力是非常重要的一步。

But I do think, you know, that that is, like, a really important step for the upcoming years to have this ability.

是的

Yep.

所以有几件快速的事情。

So a few quick things.

首先，感谢你定义了你所说的工具使用是什么意思。

First of all, thank you for defining what you mean by tool use.

我认为，对于我们讨论的概念，这样做非常好。

I think that's a great thing to do in general for the concepts we're talking about.

即使是像MOEs这样已经相当成熟的概念，嗯。

Even things that sort of well established as MOEs Mhmm.

你必须说明这意味着专家混合，而且你得帮助人们建立直觉，理解它意味着什么，如何实际应用，以及有哪些不同的类型。

You have to say that means mixture of experts, and you kinda have to build up an intuition for people what that means, how it's actually utilized, what are the different flavors.

那么，开源模型如此激增意味着什么？

So what does it mean that there's just such explosion of open models?

你的直觉是什么？

What's your intuition?

如果你发布了一个开源模型，首要目标就是希望人们使用它。

If you're releasing an open model, you want people to use it is the first and foremost thing.

其次才是透明度和信任等问题。

And then after that comes things like transparency and trust.

我认为，当你看中国时，最主要的原因是他们希望全球各地的人都能使用这些模型，但我认为很多人并不会使用。

I think when you look at China, the biggest reason is that they want people around the world to use these models, and I think a lot of people will not.

如果你看美国以外的地区，很多人不愿意为软件付费，但他们可能拥有计算资源，可以在上面部署并运行模型。

If you look outside of The US, a lot of people will not pay for software, but they might have computing resources where you can put a model on it and run it.

我认为也可能存在一些不想上传到云端的数据。

I think there can also be data that you don't want to send to the cloud.

所以，最核心的一点是让那些如果没有模型访问权限就无法使用AI的人能够使用模型、使用AI，或者使用你的AI。

So this the the number one thing is getting people to use models, use AI, or use your AI that might not be able to do it without having access to the model.

我想我们应该明确说明一下。

I guess we should state explicitly.

我们之前一直在讨论这些中国模型和开源权重模型。

So we've been talking about these Chinese models and open weight models.

通常，这些模型是在本地运行的。

Oftentimes, the way they're run is locally.

所以你并不是把数据发送到中国或任何开发模型的人——无论是硅谷的谁。

So it's not like you're sending your data to China or to whoever developed to Silicon Valley, whoever developed the model.

许多美国初创公司通过托管这些模型来赚钱。

A lot of American startups make money by hosting

是的。

Mhmm.

它们从中国购买这些模型并出售，这被称为出售令牌，意思是有人调用模型来完成某项任务。

These models from China and selling them, selling to it's called, like, selling tokens, which means somebody will call the model to do some some piece of work.

是的。

Mhmm.

我认为另一个原因是，对于美国公司来说，比如OpenAI，它们严重缺乏GPU。

I think the other reason is for US companies, like, to OpenAI is so GPU deprived.

它们已经达到了GPU的极限。

Like, they're so they're at the limits of the GPUs.

每次他们发布新版本时，总在说：‘我们的GPU快撑不住了。’

Whenever they make a release, they're always talking about, oh, like, our GPUs are hurting.

我想在一次GPTOSS的发布活动中，萨姆·阿尔特曼曾说：‘我们发布这个是因为你们的GPU可以帮上忙。’

And I think there's, like, like, one of these, like, GPTOSS release sessions, Sam Altman said, like, oh, we're releasing this because we can use your GPUs.

我们不必使用自己的GPU，OpenAI依然能从中获得分发优势，这确实是个非常现实的问题，因为这对他们来说几乎不花任何成本。

We don't have to we don't have to use our GPUs, and OpenAI can still get distribution out of this, which is another very real thing because it doesn't cost them, though, anything.

对用户而言，我认为也有一些用户像使用Chekipudi一样本地使用模型，但对企业来说，拥有这些模型是一个巨大的突破，因为你可以对它们进行定制。

And for the user, I think also I mean, there are users who just use the model locally how they would use Chekipudi, but also for companies, I think it's a huge unlock to have these models because you can customize them.

你可以训练它们。

You can train them.

你可以进行后训练，加入更多数据，比如将它们专门化为法律、医疗等领域的模型，随你所需。

You can add post training, add more data, like specialize them into, let's say, law, medical models, whatever you have.

你提到的Llama，它的吸引力在于。

And the appeal you mentioned Lama.

中国开源权重模型的吸引力在于，这些开源模型的许可协议也更加友好。

The appeal of the open weight models from China is that the open weight models are also the licenses are even friendlier.

我认为它们是完全无限制的开源许可证，如果我们使用像Llama或Gemma这样的模型，就会有一些附加条件。

I think they are just unrestricted open source licenses where if we use something like Lama or Gemma, there are some strings attached.

我认为在用户数量上存在一个上限。

I think it's like a upper limit in terms of how many users you have.

如果你超过了，比如说几百万用户，你就得向Meta之类的公司报告你的财务状况。

And then if you exceed, I don't know, so so many million users, you have to report your finance situation to, let's say, Meta or something like that.

我认为这虽然是一个免费模型，但附带了一些条件，而人们更喜欢没有任何附加条件的东西。

And I think, well, it is a free model, but there are strings attached, and people do like things where strings are not attached.

所以我认为，除了性能之外，这也是中国开源权重模型如此受欢迎的原因之一，因为你可以直接使用它们。

So I think that's also one of the reasons besides performance why the open weight models from China are so popular because you you can just use them.

从这个意义上说，没有任何陷阱。

There's no there's no catch in that sense.

是的。

Yeah.

在这方面，生态系统已经有所改善，但主要是得益于这些新提供商提供如此开放的许可证。

The ecosystem has gotten better on that front, but mostly downstream of these new providers providing such open licenses.

你调出Perplexity的时候真有趣。

That was funny when you pulled up Perplexity.

它显示Kimmy k two Thinking托管在美国，这正是我们所讨论的敏感点的完美例子，我以前从没见过这种情况。

It said Kimmy k two Thinking hosted in The US, which is just like an exact I've never seen this, but it's an exact example of what we're talking about where people are sensitive to this.

比如Kimmy K two thinking，而Kimmy K two是一个非常受欢迎的模型。

Like Kimmy K two thinking and Kimmy K two is a model that is very popular.

人们说它在创意写作和一些软件任务方面表现非常出色。

People say that it has very good, like, creative writing and also in doing some software things.

所以，人们会注意到不同模型这些细微的特性，并因此偏好某些模型。

So it's just these little quirks that people pick up on with different models that they like.

这些模型中有哪些有趣的想法是你能谈一谈的，特别是让你觉得特别有意思的？

What are some interesting ideas that some of these models have explored that you can speak to, like, that particular interesting to you?

也许你可以按时间顺序讲一讲。

Maybe you can go chronologically.

我的意思是，如果我们只关注2025年，当然有DeepSeq DeepSeq r one在1月份发布了。

I mean, there was, of course, DeepSeq DeepSeq r one that came out in January if we just focus on 2025.

然而，这是基于DeepSeq第三个版本，该版本于2024年12月发布。

However, this was based on DeepSeq version three, which came out the year before in December 2024.

在架构方面有多个创新点。

There are multiple things on the architecture side.

有趣的是，你仍然可以——我的意思是，这正是我在从零开始的编码项目中所做的。

What is fascinating is you you can still I mean, that's what I do in my from scratch coding projects.

你仍然可以从GPT-2开始，然后在这个模型基础上添加功能，将其变成另一个模型。

You can still start with GPT two, and you get can add things to that model to make it into this other model.

所以它们本质上仍属于同一条技术脉络，彼此关系非常紧密。

So it's all still kinda like the same lineage, the same it is a very close relationship between those.

但就我目前所想，DeepSeq的独特之处在于混合专家机制，或者更准确地说，他们并没有发明混合专家机制。

But top of my head, DeepSeq, what was unique there is the mixture of ex or not I mean, they were not inventing mixture of experts.

我们或许可以再多解释一下什么是混合专家机制。

We can maybe talk a bit more what mixture of experts means.

不过在深入细节之前，先列出这些要点：混合专家机制，此外他们还采用了多头潜在注意力机制，这是一种对注意力机制的改进，我认为在2025年，这是这些开源模型之间最主要的区别，即各种针对推理效率或KV缓存大小的优化调整。

But just to list these things first before we dive into detail, mixture of experts, but then they also had a multi head latent attention, which is a tweak to the attention mechanism where this was, I would say, 2,025, the main distinguishing factor between these open weight models, different tweaks to make inference or KV cache size.

我们也可以在稍后定义KV缓存。

We can also define KV cache in a few moments.

但为了更经济地支持长上下文，需要缩小KV缓存的大小，那么我们可以做哪些调整呢？

But to kind of make it more economical to have long context to shrink the KV cache size, so what are tweaks that we can do?

其中大多数都聚焦在注意力机制上。

And most of them focused on the attention mechanism.

DeepSeq中使用了多头潜在注意力。

Is multi head latent attention in in DeepSeq.

分组查询注意力也非常流行。

There is group query attention, which is still very popular.

它并不是由这些模型中的任何一个发明的。

It's not invented by any of those models.

它早在几年前就已存在，但那会是另一种选择。

It goes back a few years, but that that would be the other option.

滑动窗口注意力，我认为几乎是在复用它，如果我没记错的话。

Sliding window attention, I think, almost reuses it, if I remember correctly.

所以，这些不同的调整让模型之间产生了差异。

So there are, like like, these different tweaks that make the models different.

否则，我曾经写过一篇文章，把它们全都放在一起进行比较。

Otherwise, I put them all together in an article once where I just compared them.

它们出人意料地相似。

They are very surprisingly similar.

其实只是在Transformer块的重复次数、以及一些细微的调节参数上有所不同。

It's just different numbers in terms of how many repetitions of the transformer block you have in the center and, like, just little little knobs that people tune.

但真正棒的是，无论怎么改，它都能正常工作。

But but what's so nice about it is it's it it works no matter what.

你可以调整各种参数。

You can tweak things.

你可以移动归一化层的位置。

You can move the normalization layers around.

这样可以获得一些性能提升。

You get some performance gains.

而且我在消融研究中几乎总是表现得很好，能够清楚地展示如果移动某些部分，模型会受到什么影响。

And I almost always very good in ablation studies showing what actually what it does to the model if you move something around.

消融研究并不会让模型变得更好或更差。

Ablation studies doesn't make it better or worse.

但有太多方式可以实现Transformer，同时仍能使其正常工作。

But there are so many, let's say, ways you can implement a transformer and make it still work.

至今仍广泛使用的一些重要想法包括专家混合、多头潜在注意力、滑动窗口注意力、分组核注意力。

Big ideas that are still prevalent is mixture of experts, multi ad latent attention, sliding window attention, group core attention.

到了年底，我们看到研究重点转向了让注意力机制在推理时的标记预测中实现线性扩展。

And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction.

比如QN3就引入了门控Delta网络。

So there were QN three next, for example, which added a gated delta net.

这有点受到状态空间模型的启发，即你保持一个固定状态并不断更新，但它让注意力计算变得更便宜，或者用更廉价的操作替代了注意力。

It's it's like kind of like inspired by state space models where you have a fixed state that you keep updating, but it makes essentially this attention cheaper, or it replaces attention with a cheaper operation.

也许我们该退一步，谈谈Transformer架构的整体情况。

And it maybe is it useful to step back and talk about transform architecture in general.

是的

Yeah.

所以我们或许应该从GPT-2架构开始说起。

So maybe we should start with the GPT two architecture.

这个Transformer架构源自《Attention Is All You Need》论文。

The transformer that was derived from the attention is all you need paper.

嗯哼。

Mhmm.

《Attention Is All You Need》论文中的Transformer架构包含两个部分：编码器和解码器。

So the attention is all you need paper had a transformer architecture that had two parts, an encoder and a decoder.

而GPT只专注于解码器部分。

And GPT went just focusing in on the decoder part.

它本质上仍然是一个神经网络，内部包含这种注意力机制。

It is essentially still a neural network, and it has this attention mechanism inside.

并且你一次预测一个标记。

And you predict one token at a time.

你将它通过一个嵌入层。

You pass it through an embedding layer.

这是Transformer模块。

There's the transformer block.

Transformer模块包含注意力模块和全连接层，中间还有一些归一化层。

The transformer block has attention modules and a fully connected layer, and there are some normalization layers in between.

但本质上它还是带有注意力机制的神经网络层。

But it's essentially neural network layers with this attention mechanism.

因此，从GPT-2到GPT-4，比如引入了专家混合层。

So coming from GPT two, when we move on to GPT OSS, there is, for example, the mixture of experts layer.

这并不是GPT-4发明的。

It's not invented by GPT OSS.

它已经有几年历史了，但本质上是一种调整，可以在不增加每次前向计算开销的情况下扩大模型规模。

It's a few years old, but it is essentially a a tweak to make the model larger without consuming more compute in each forward pass.

所以这里有一个全连接层。

So there is this fully connected layer.

如果听众熟悉多层感知机，你可以把Transformer内部的全连接神经网络层看作是一个小型的多层感知机。

And if listeners are familiar with multilayer perceptrons, you can think of a mini multilayer perceptron, a fully connected neural network layer inside the transformer.

这非常昂贵，因为它是全连接的。

And it's very expensive because it's fully connected.

如果你有上千个输入和上千个输出，那就相当于一百万个连接。

If you have thousand inputs, thousand outputs, it's like a 1,000,000 connections.

这是Transformer中非常耗资源的一部分。

And it's a very expensive part in this transformer.

这个想法是将它扩展为多个前馈网络，比如不再只有一个，而是有256个，但这会让计算量大幅增加，因为你现在有了256个。

And the idea is to kind of expand that into multiple feed forward net So instead of having one, let's say you have 256, but it would make it way more expensive because now you have 256.

但你并不会同时使用所有这些网络，而是引入一个路由机制，决定：好的。

But you don't use all of them at the same time, so you now have a router that says, okay.

根据这个输入词元，使用这个全连接网络会更有帮助。

Based on this input token, it would be useful to use this fully connected network.

在这种情况下，它被称为一个专家。

And in that context, called an expert.

所以，专家混合模型意味着你拥有多个专家。

So a mixture of experts means you have multiple experts.

根据你的输入内容，比如如果是数学密集型的，就会使用与将英文翻译成西班牙文不同的专家。

And depending on what your input is, let's say it's more math heavy, it would use different experts compared to, let's say, translating input text from English to Spanish.

它可能会咨询不同的专家。

It would maybe consult different experts.

这并不是很清楚，我的意思是，很难明确地说：好吧。

It's not quite clear I mean, not as clear cut to say, okay.

这个专家只负责数学，而西班牙语则更模糊一些。

This is only an expert for math, and for Spanish is a bit more fuzzy.

但核心思想是，你将更多知识压缩进网络中，但并非所有知识都会被同时使用。

But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time.

是的。

Mhmm.

那会非常浪费。

That would be very wasteful.

所以在生成令牌时，你更像是有选择性地进行处理。

So you are kind of like during the token generation, you are more selective.

有一个路由器来决定哪些令牌应该发送给哪个专家。

There's a router that selects which tokens should go to which expert.

这增加了复杂性。

It's more complexity.

训练起来更困难。

It's harder to train.

有很多可能出错的地方，比如崩溃之类的。

There's a lot of, you know, that can go wrong, like collapse and everything.

所以我认为这就是为什么目前大多数模型仍然使用密集型架构。

So I think that's why almost three still uses dense.

我的意思是，你拥有的所有混合专家模型，但密集型模型——这里的‘密集’也是行话。

I mean, you have, I think, all the models with mixture of experts, but dense models where dense means so also it's jargon.

密集型和稀疏型之间是有区别的。

There's a distinction between dense and sparse.

因此，专家混合模型被认为是稀疏的，因为我们有很多专家，但只有少数几个会被激活。

So mixture of experts is considered sparse because we have a lot of experts, but only few of them are active.

所以这被称为稀疏。

So that's called sparse.

而密集模型则恰恰相反，你只有一个全连接模块，并且它始终处于使用状态。

And then dense would be the opposite where you only have, like, one fully connected module and it's always, you know, utilized.

所以也许现在是个不错的时机来谈谈KV缓存，但在此之前，先从更宏观的角度看，从GPT-2到现在，究竟引入了多少新想法？

So may maybe this is a good place to also talk about KV cache, but actually before that, even zooming out, like, fundamentally, how many new ideas have been implemented from from GPT two to today?

是的。

Mhmm.

这些架构到底有多大不同呢？

Like, how different really are these architectures?

想象一下，专家混合模型。

Picture, like, the mixture of experts.

GPT-3中的注意力机制，就是分组查询注意力机制。

The attention mechanism in GPT OSS, that would be the group query attention mechanism.

所以这是对多头注意力机制的一个小调整，改为分组查询注意力，这样我们就有了两种。

So it's a slight tweak for multi head attention to group query attention so that we have two.

我认为他们用RMS归一化替换了层归一化，但这只是另一种归一化方式。

I think they replaced layer norm by RMS norm, but it's just like a different normalization there.

这不是很大的改动。

Not a big change.

这只是一个小小的调整。

It's just like a a tweak.

关于非线性激活函数，熟悉深度神经网络的人会知道，这就像把Sigmoid换成ReLU。

The nonlinear activation function, people familiar in with deep new networks, I mean, it's the same as changing sigmoid with ReLU.

这并没有从根本上改变网络结构。

It's it's not changing the network fundamentally.

这只是一个小小的调整。

It's just like a tweak.

你只是做了一点点调整。

You take a little little tweak.

我想就这些了。

And and that's about it, I would say.

它并没有本质上那么不同。

It's not really fundamentally that different.

它仍然是相同的架构。

It's still the same same architecture.

所以你可以从一个转换到另一个。

So you can convert one from one.

你基本上只需添加这些改动，就能从一个转向另一个。

You can go from one into the other by just adding these these changes, basically.

它本质上仍然是相同的架构。

It's fundamentally is still the same architecture.

是的。

Yep.

比如，你之前提到过我的书。

So for example, you mentioned my book earlier.

书中的那个是GPT-2模型，因为它简单而且体积很小。

That's a GPT two model in the book because it's simple and it's very small.

大约有1亿2400万到1亿2000万个参数。

So 124, 120,000,000 parameters approximately.

但在附加材料中，我确实从零开始构建了几乎三个模型，比如Gemma从零开始，以及其他类型的从零构建模型。

But in the bonus materials, I do have almost three from scratch, Gemma three from scratch and other types of from scratch models.

我总是从我的GPT-2模型开始，然后调整——嗯，添加不同的组件，就能从一个模型过渡到另一个。

And I always started with my GPT two model and just, you know, tweaked the well, added different components, and you get from one to the other.

这在某种程度上就像一种谱系关系。

It's like it's kind of like a lineage in a sense.

是的。

Yeah.

你能为人们建立一种直观的理解吗？

Can you build up an intuition for people?

因为当你宏观来看时，会发现人工智能领域的发展速度如此之快。

Because sort of when you zoom out, you look at it, there's so much rapid advancement in the AI world.

同时，从本质上讲，这些架构并没有改变。

And at the same time, fundamentally, the architectures have not changed.

嗯。

Mhmm.

那么，所有这些快速进步和动荡究竟发生在哪儿呢？

So where is all the turbulence, the turmoil of the advancement happening?

进步的潜力在哪里？

Where where is the gains to be had?

在开发或训练网络的过程中，有不同的阶段。

So there are the different stages where you develop the network or train the network.

你有预训练阶段。

You have the pretraining.

那时候，他们只是用GPT-2进行预训练。

Now back then, they it was just pretraining with GPT two.

现在你有预训练、中训练和后训练。

Now you have pretraining, midtraining, and posttraining.

所以我认为，目前我们正处于后训练关注阶段。

So and I think right now, we are in the post training focus stage.

我的意思是，预训练仍然能带来优势，特别是当你使用更高质量的数据并扩大规模时。

I mean, pretraining still gives you advantages if you scale it up to better, higher quality data.

但随后我们获得了GPT-2时代所没有的能力突破。

But then we have capability unlocks that were not there with GPT two.

例如，ChatGPT本质上是一个GPT-3模型，而GPT-3在架构上与GPT-2是相同的。

For example, ChatGPT, it is basically a GPT three model, and and GPT three is the same as GPT two in terms of architecture.

新的突破在于引入了监督微调和基于人类反馈的强化学习。

What was new was adding the supervised fine tuning and the reinforcement learning with human feedback.

因此，这些进步更多体现在算法层面，而非架构层面。

So it's more on the algorithmic side rather than the architecture.

我认为系统本身也发生了很大变化。

I would say that the systems also change a lot.

如果你听NVIDIA的发布内容，他们会提到像现在使用FP8这样的技术。

I think if you listen to NVIDIA's announcements, they talk about these things like, you now do FP eight.

现在你可以使用FP4了。

You can now do FP four.

目前这些实验室正在研究如何利用更多计算资源将它们整合到一个模型中，从而加快训练速度。

And what is happening is these labs are figuring out how to utilize more compute to put it into one model, which lets them train faster.

这让他们能够加入更多数据。

And that lets them put more data in.

通过这种方式，你可以更快地找到更优的配置。

And then you can find better configurations faster by doing this.

因此，在进行大规模训练时，你可以关注每块GPU每秒处理的token数量这一指标。

So you can look at like the essentially the tokens per second per GPU is a metric that you look at when you're doing large scale training.

通过开启FP8训练，你可以将性能从大约10k提升到13k，这意味着模型中每个参数占用的内存更少了。

And you could get you can go from like 10 k to 13 k by turning on FPA training, which means you're using less memory per parameter in the model.

通过减少保存的信息量，你减少了通信开销。

And by saving less information, you do less communication.

你可以更快地进行训练。

You can train faster.

所以所有这些系统层面的改进，都支撑着数据和算法上更快的实验循环，这种循环一直在持续，虽然从架构上看它们几乎完全相同，但训练这些模型所用的代码库却大不相同。

So all of these, like, system things underpin way faster experimentation on data and algorithms that is kind of like it's this it's this kind of loop that keeps going, where where it's kinda hard to describe when you look at the architecture and they're exactly the same, but the code base used to train these models is gonna be vastly different.

嗯。

Mhmm.

你可能也发现，虽然GPU不同，但用现在的技术训练GPT-2 20B，实际耗时肯定比当年训练GPT-2快得多。

And you could probably like, I don't the GPUs are different, but you probably train GPT OSS 20 b way faster in wall clock time than GPT two Mhmm.

那时候的训练方式。

Was trained at the time.

是的。

Yep.

就像你说的，比如在专家混合模型中，他们采用了BF16和FP4优化，从而获得了更高的吞吐量。

Like you said, they had, for example, in the mixture of experts is n b f p four optimization, for example, where you get more throughput.

但我确实认为，虽然这在速度上是成立的，但它并没有在本质上赋予模型新的能力。

But I I do think this is for the speed, this is true, but it it doesn't give the model new capabilities in a sense.

它只是让我们能在不显著降低模型性能的前提下，把计算粒度粗化到什么程度。

It's just how much can we make make the computation coarser without suffering in terms of model performance degradation.

但我认为，确实出现了替代Transformer的其他方案。

But I do think mean, there are alternatives popping up to the transformer.

比如文本扩散模型，这是一种完全不同的范式。

There's text diffusion models, completely different paradigm.

而且文本扩散模型可能也会使用Transformer架构，但它们不是自回归的Transformer。

And there's also I mean, the text diffusion models might use transformer architectures, but it's not an auto autoregressive transformer.

还有MAMBA模型，这是一种状态空间模型，但它们也有各自的权衡。

And also MAMBA models, it's a state space model, but they do have trade offs.

目前来看，还没有任何模型能取代自回归Transformer作为最先进的模型。

And what's right is there's nothing that has replaced the autoregressive transformer as state of the art model.

所以，要追求最先进的性能，你还是会选这个方案。

So, like, for state of the art, you would still do that go with that thing.

但现在出现了更多低成本的替代方案，这些方案做了一些妥协，但已不再只有一种架构了。

But there are now alternatives for the cheaper and, like, alternatives that are kind of making compromises, but it's not just one architecture anymore.

新的小众架构正在不断涌现。

There are little ones coming up.

但如果我们讨论最先进的技术，目前基本上还是源自GPT-2的自回归Transformer架构。

But if we talk about the state of the art, it's pretty much still the the transformer architecture autoregressive derived from GPT two, essentially.

我想这里的关键问题是，我们已经讨论了很多关于预训练背后架构的内容。

I guess the big question here is we talked quite a bit here on the architecture behind the the pretraining.

在预训练、后训练、推理、上下文长度、数据和合成数据方面，扩展定律是否依然有效？

Are the scaling laws holding strong across pre training, post training, inference, contact size, data, synthetic data?

我想先从扩展定律的技术定义说起，这个定义会影响我们对所有这些的理解。

I like to start with the technical definition of scaling law, which kind of informs all of this.

扩展定律是一种幂律关系，横轴代表你所扩展的变量，通常是计算量和数据的组合，这两者是相似的。

The scaling law is a power law relationship between you think of the x axis, so kind of what you are scaling is a combination of compute and data, which are kind of similar.

纵轴则是模型在预测下一个token时的保留集准确率。

And then the y axis is like the held out prediction accuracy over our next tokens.

我们之前讨论过模型是自回归的。

We talked about models being auto regressive.

也就是说，如果你有一组模型未曾见过的文本，在训练后它能有多准确？

It's like if you keep a set of text that the model has not seen, how accurate will it get when you train?

而缩放定律的概念源于人们发现这是一种非常可预测的关系。

And the idea of scaling laws came when people figured out that that was a very predictable relationship.

我认为这个技术术语仍在延续，接下来的问题是：用户能从中获得什么？

And I think that that technical term is continuing, and then the question is like, what do users get out of it?

此外，还有更多类型的缩放方式，比如OpenAI的o1以引入推理时缩放而闻名，而不太为人所知的是，它还展示了可以通过扩展强化学习训练，获得对数形式的x轴和线性增长的y轴性能。

And then there are more types of scaling where OpenAI's o one was famous for introducing inference time scaling, and I think less famously for also showing that you can scale reinforcement learning training and get kind of this log x axis and then a linear increase in performance on y axis.

因此，现在有这三大维度：传统缩放定律主要讨论预训练阶段，即模型规模和数据集大小。

So there's kind of these three axes now where the traditional scaling laws are talk talked about for pre training, which is how big your model is and how big your dataset is.

然后是强化学习的缩放，也就是我们接下来要讨论的这种试错学习能持续多久。

And then scaling reinforcement learning, which is like how long can you do this trial and error learning that we will talk about.

我们会进一步定义这些内容。

We'll define more of this.

还有推理时的计算资源，即让模型在特定问题上生成更多令牌。

And then this inference time compute, which is just letting the model generate more tokens on a specific problem.

我对这三者都依然持乐观态度，但低垂的果实大多已被摘取，尤其是在过去一年里，强化学习方面有了可验证奖励（RLVR），而推理时缩放也让这些模型的使用体验截然不同——过去你几乎能立即获得第一个令牌。

So I'm kind of bullish where they're they're all really still working, but the low hanging fruit has mostly been taken, especially in the last year on reinforcement learning with verifiable rewards, which is this RLVR, and then inference time scaling, which is just why these models feel so different to use where previously you would get that first token immediately.

现在它们会花费数秒、数分钟，甚至数小时来生成这些隐藏的思考过程，然后才给出你答案的第一个词。

And now they'll go off for seconds, minutes, or even hours generating these hidden thoughts before giving you the first word of your answer.

这一切都关乎推理时间扩展，这种扩展在模型能力的提升上带来了显著的跃迁。

And that's all about this inference time scaling, which is such a wonderful kind of step function in terms of how the models change abilities.

它们使得工具使用成为可能，并推动了我们之前讨论的更出色的软件工程能力。

They've kind of enabled this tool use stuff and enabled this much better software engineering that we were talking about.

当我们说‘使能’时，这几乎完全是由于使用可验证奖励的强化学习训练，让模型能够轻松地掌握这些技能。

And this is when we say enabled, almost entirely downstream of the fact that this reinforced learning with verifiable rewards training just kind of let the models pick up these skills very easily.

让模型自己去学习。

So let the models learn.

所以当你观察模型在生成大量令牌时的推理过程，它通常会尝试一个工具，然后观察返回的结果。

So if you look at the reasoning process when the models are generating a lot of tokens, what it'll be often doing is it tries a tool, it looks at what it gets back.

它再尝试另一个API，查看返回内容，直到解决问题为止。

It tries another API, it sees what it gets back and if it solves the problem.

因此，在训练过程中，模型会迅速学会这样做。

So the models when you're training them very quickly learn to do this.

到了一天结束时，这就形成了一个通用的基础，模型可以很好地在你的代码库中使用命令行指令，帮你处理Git，移动和整理文件，或搜索更多信息——而如果我们一年前坐在这里，根本不会想到模型能做这些事。

And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo and handle git for you and move things around and organize things or search to find more information, which if we're sitting in these chairs a year ago, it's something that we didn't really think of the models being doing.

所以，这仅仅是今年发生的事情，彻底改变了我们对使用AI的看法，我认为这非常神奇。

So this is just kinda something that has happened this year and is totally transformed how we think of using AI, which I think is very magical.

这是一个非常有趣的演进，极大地释放了价值。

It's such an interesting evolution and just so unlocks so much value.

但目前还不清楚下一步解锁类似能力的方向会是什么。

But it's it's like, it's not clear what the next avenue will be in terms of unlocking stuff like this.

我想我们稍后会谈到持续学习的问题。

I think that there's there's we'll get to continual learning later.

但目前AI领域有很多关于某些方向的热议，却没人知道下一个跃迁何时真正到来。

But there's a lot of buzz around certain areas of AI, but no one knows when the step function will will really come.

你刚才说了很多内容，而且迅速道出了深刻的见解。

So you've you've actually said quite a lot of things there and said profound things quickly.

如果能稍微深入探讨一下这些观点就好了。

It would be nice to unpack them a little bit.