记忆与持续学习：Engram 的 Dan Biderman 和 Jessy Lin

在预训练或者后训练中，是什么让模型能够以这些神奇的新兴方式进行泛化，并且控制这个过程，让公司拥有私有数据？

What about pre training or even post training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that a company has a set of private data?

我们如何让模型像知道法国首都或者怎么写Python那样，同样出色地学习这些内容？

How do we make the models learn that just as well as the models know like the capital of France or you know like how to write Python?

所以我觉得这是一个非常有趣的问题，值得思考。

So I think it's a really fun problem to think about.

欢迎来到训练数据。

Welcome to training data.

今天我们很高兴邀请到Engram的联合创始人Dan Biderman和Jessy Lin。

We are delighted to have Dan Biderman and Jessy Lin, cofounders of Engram today.

Engram是一个专注于记忆和持续学习的新实验室，这是当今AI研究中最热门的两个话题。

Engram is a neolab focused on memory and continual learning and two of the hottest topics in all of AI research today.

我和Shaun今天很高兴能和你们深入探讨这些话题。

And Shaun and I are delighted to dig in on those topics with you today.

太棒了。

Awesome.

太好了。

Happy Great.

是的。

To be

那么，也许我们先从Engram的网站说起，上面写着：“我们不是通过预训练或后训练的视角来看世界。”

So maybe to kick off, the Engram website says, we don't see the world through the lens of pretraining or post training.

我们的模型一直在训练。

Our models are always training.

这是什么意思？

What does that mean?

我觉得现在的模型显然知道很多东西。

I think models today obviously know a lot of things.

它们非常聪明，但我们认为，如今让这些模型更有用的瓶颈并不是原始智能，而是理解新的和不断变化的背景。

They're incredibly smart, but we think the bottleneck for making these models more useful these days is not really raw intelligence, but understanding new and evolving context.

无论是你正在做的新任务，还是某个特定的工作背景，如何将其深入嵌入模型权重，就像预训练和后训练那样？

Whether it's a new task that you're doing or a particular context for a job or something like this, how do you bake that into the model weights the same way that pre training and post training bakes that into the model weights very deeply?

所以我们认为自己在解决记忆和持续学习这两个根本性问题，它们其实是一体两面。

This is why we think of ourselves as working on these fundamental problems of memory and continual learning, which are really two sides of the same coin.

怎么让模型学到新东西，并且把这些新知识深深嵌入到模型的权重里呢？

How do you make the models learn new things and bake them deeply into the weights of the model?

那么你的观点是，把记忆当作一个独立的数据库，或者塞进上下文窗口里的独立模块，并不是真正的记忆，也不是真正的持续学习吗？

And is your premise then that memory as a separate database or a separate thing that you've shoved into the context window is not true memory and is not true continual learning?

我觉得这些工具最终会融合到一起。

I think all of these tools will kind of come together.

所以现在大家解决这类问题的方法叫作上下文工程。

So these days, the way that people are solving these problems is with context engineering.

比如你写一个很长的提示词，或者跟模型聊上很多轮、好几个小时，然后不断调整上下文，让它更清楚你到底想干什么。

So you take a huge prompt, maybe you keep talking to the model over many, many turns and hours and reorganize the context to better understand what you're trying to do.

我们认为像工具调用、上下文工程这类手段确实会发挥作用，但我觉得目前还没被充分利用的一个方法是：把前沿实验室用来让模型在数学或编程等顶尖领域表现出色的那套训练流程或框架，应用到公司里的各种业务场景和上下文中去。

We think these kinds of things, like tool use, context engineering will play a part, but I think an under leveraged tool these days is using the same training pipeline or framework or kind of workflow that the frontier labs are using to make these models really good at frontier math or code, but applying that to every kind of domain, every kind of context that you have, let's say in a company.

对。

Yeah.

对我来说，就像个人做笔记、贴便签一样，这是非常有价值的事情。

To me, it's like as an individual, taking notes and having sticky notes is a very valuable thing.

我们不应该丢弃这个。

We should never discard this.

但是当我们第二天回到工作时，我们的大脑里总会留下一些记忆的痕迹，一些关于事情应该怎么做、应该往哪里看的新直觉。

But whenever we get back to business the next day, we always have some sort of trace of memory in our brain, some new intuition about how things should be and where should we look.

所以这两样东西应该结合起来。

So these two things should come together.

而目前的解决方案更多是外部化的记忆。

And current solutions are more kind of externalized memory.

这有两个问题。

And this has two issues.

一个是，我们所有人每天集体或个人生成的token数量很快就会达到每天数千万个。

One is that the amount of tokens we will all collectively individually generate is going be in the tens of millions of tokens per day soon.

所以仅仅保存和搜索它们，以及重新阅读它们，将会非常昂贵，而且对于模型来说也会非常困难、非常混乱，除非我们在……方面取得重大突破。

So just keeping it and searching through it is going to be and rereading it, it's going to be pretty expensive, but it's going to also be pretty hard, pretty confusing for the models, unless we have major, major breakthroughs in

这取决于肖恩那边有多少亿个令牌。

Depends how on billions of tokens for Shaun.

那很好。

That's good.

看情况，每天都不一样。

Depends on the day.

你能给我们讲讲恩格拉姆的架构或者产品，以及它是如何工作的吗？

Could you maybe tell us a little bit about the Engram architecture or the Engram product and how it works?

是的，我的意思是，从高层次来看，我们想做的是获取任何上下文。

Yeah, mean, at a high level, what we're trying to do is take any context.

比如说，有各种各样的工作空间。

There's all these different workspaces, let's say.

我们正在与Notion、微软和Harvey等合作伙伴合作，这些地方是人们长期进行大量工作的地方。

We're working with partners like Notion, Microsoft, and Harvey that have these places where people are doing a lot of work over a long period of time.

这里有所有的上下文，既包括团队已经写好的文档，也包括现在人们在这些产品中越来越多地与这些代理互动，进行对话、给予反馈，以及弄清楚如何让模型深入理解这些上下文。

There's all this context, both in terms of documents that you've already written as a team, as well as now people are interacting with these agents more and more in these products, or having conversations, giving them feedback, and figuring out how to have a model that deeply understands that context.

所以不仅仅是测试时读取文件，而是真正像在公司工作多年的员工那样理解它。

So not just reading the files at test time, but really understanding it the way that an employee that's worked at your company for years has.

所以你在高层面上理解，哦，这些是公司范围内的举措。

So you understand at a high level, Oh, these are the initiatives across the company.

这是我们做事的方式。

This is the way that we do things.

你研究了如何运行招聘流程，或者如何在公司内部做这类事情，并且能像公司里其他人一样出色地运作。

You've studied how to run the hiring pipeline or how to do this kind of thing within the company and can operate just as well as anybody else can in the company.

我们正在做的是在这些工作空间内训练每个团队的模型，这些模型深入理解那些背景，并能随着时间在人们关心的事情上不断改进。

What we're doing is training per team models within these workspaces that deeply understand those contexts and can improve with time on the things that people care about.

我们在技术层面上的做法可能是将这些训练到权重中。

The way that we do this at a technical level maybe is training these into weights.

我们做了很多适配器微调。

We do a lot of adapter fine tuning.

多种类型的适配器。

Adapters of many types.

我觉得大家研究这个问题已经几十年了，不管是LoRA、前缀微调还是稀疏架构。

I think people have looked into this for decades at this point, whether it's LORAs or prefixes or sparse architectures.

我认为这些工具我们都能用得上。

I think all of these tools are at our disposal.

接下来就是确定什么样的数据最合适。

Then figuring out what the right data is.

你怎么把各种原始文档或交互记录转化成对模型有用的训练信号呢？

How do you turn any kind of raw document or interaction into useful training signal for the model?

再说一遍，我们现在有了各种各样的工具，比如监督微调、强化学习、在线策略蒸馏等等，这些都是这个领域已经开发出来的技术。我们要做的就是把这些模块整合起来，训练出一个能持续学习大家关心的内容的模型。

Again, we have a variety of tools now like supervised fine tuning, RL, on policy distillation, all of these things that the field has developed and trying to fit these pieces together into a model that learns continuously on the things that people care about.

对，而且这也不是赌说没有现成的工具可用。

Yeah, and it's not a bet that tools are not there.

我们的模型始终基于一个假设：有些知识是外部的，有些工具总是可用的。

Like our models always work under the assumption that some knowledge is externalized, some tools are always there.

但你需要做的是搞清楚——这也是最难的地方——哪些东西需要内化，哪些可以保持外部化。

But what you need to do is you need to figure out, and that's the hard task is what needs to be internalized and what can be externalized.

而且即使是外部化的东西，很多个人和公司也有自己定制的工具和做事方式。

And even for stuff that's externalized, many individuals and companies have their own bespoke tools and ways of doing things.

不是每个人都有那些前沿模型训练时用的相同的 bash 命令行工具，如何让模型更好地理解你的定制环境，我觉得这本身就是一件有趣的事情。

Not everyone has the same, you know, bash CLI tools that, you know, the frontier models are training on and how to get the models to better understand your bespoke setup, I think is its own interesting thing.

那么，前提是不是我的 Notion 代理会是一个定制代理，通过 LoRA 微调或者某种适配器调优，这样它就能不断学习我 Notion 工作区里新增的内容？

And so is the premise then that my Notion agent will be a custom agent that is LoRa fine tuned or some way with an adapter tuned so that it's constantly learning on new content that's added into my Notion workspace.

是的，那个……

Is that Yeah, the

而且他们和很多模型合作，是所有前沿模型的早期用户，他们可能还会继续这样做。

and they're working with many models, they're the early users of all the frontier models, and they're probably gonna keep doing that.

这种方法在前沿模型上有效吗，还是说对封闭的前沿模型无效？

Does this approach work on the frontier models, are the closed frontier models or not?

我们需要对权重的白盒访问，对吧？

We need white box access to the weights, right?

所以，我们可以和拥有闭源权重的公司合作，和他们一起做这件事。

So, we can partner with companies that have closed source weights and do this with them.

不过对我们来说，用开源模型来做是最方便的。

But it's easiest for us to do it with open source models.

只要是基于 Transformer 架构的模型，我们都能对它进行相应的处理。

But any model that's a transformer model, we can do our thing to it.

那大家在使用你们的服务前后对比时，主要的权衡点是什么呢？

And what's the trade off then when people are comparing the before and after using you?

是不是因为他们不再需要发送那么长的上下文了？

Is it that they're no longer sending so much context?

所以权衡在于，你前期投入更多算力把公司的做事方式学进模型权重里，之后每次推理时发给模型的上下文就少了。

And so the trade off is like you burn more compute upfront to learn your company's way of doing things into the weights, and then you're sending less context to the model on every inference pass.

大概就是这么个权衡对吧？对，

Is that the rough trade Yeah,

这是其中一点。

that's one thing.

你不用再去查资料、反复读材料，也不用写那种超级长的系统提示词，这能让你的推理 token 消耗量降低两个数量级。

The fact that you don't have to research things and reread things and the fact that you don't have to write like monstrous system prompts, can give you, you know, two orders of magnitude reduction in token inference consumption.

不是那种，你知道，50%的减少，而是可能减少100倍的token，因为很多事情，尤其是与人、团队、组织和优先级相关的事情，这些不是你在一个文档里就能找到的，除非你真的把一切都整理得井井有条，并且记录所有东西。

It's not like, you know, 50% or it can be 100x fewer tokens because many things, especially things that relate to people and teams and organization and priorities, these are things that you can't really find in one document unless like you really have it really regimented and document everything.

而且这类东西，模型可以通过在一些数据上训练来隐式学习，然后用100个token回答，而最好的前沿模型要消耗10万个token才能做到。

And these kinds of things, the model can kind of implicitly learn by training on some of the data and answer within 100 tokens, the best frontier models would consume 100,000 tokens doing.

所以这类例子很有意思。

So these kinds of examples are interesting.

还有质量方面，有些任务对当前这一代模型来说并不是超自然的。

And also the quality, there are tasks that are not supernatural for the current generation of the models.

我们觉得会一直存在这种大约三到六个月的领先差距，有些定制化的事情人们正在探索，模型还不能完全胜任，但模型最终会变得擅长，如果你能以非常轻量的方式自主学习和提升能力，那在这段时间里就能带来价值。

And we kind of think there's going to be consistently this gap of like three to six months ahead, where there's certain things that are bespoke that people are just exploring, the models are not fully great for them, the models will at some point be great for them, but if you can autonomously learn in a very lightweight way, it will give value in that time and cameras capabilities.

为什么要在工作区层面训练，而不是在个人层面，比如？

Why train on the workspace level versus the individual level, for example?

对我们来说都可以。

Either is fine for us.

只是从团队开始更容易，你知道，团队的人在收集上下文方面更有纪律性，而且他们多年积累的上下文量也更多。

It's just easier to start with, you know, teams of people have, you know, are more, you know, disciplined in how they collect context and in the amount of context they have over years.

对我们来说，从那里开始比较容易。

And it's easy for us to start there.

但总有一天，每个人的电脑和手机都会成为我们技术的有用目标。

But every person's computer and every person's phone one day is a useful target for our technologies.

实际上，进入那个领域会非常有趣。

And in fact, it will be very interesting to go there.

我们只是认为，目前大量的信息都集中在团队协作的知识工作中。

We just think the big deposits of information are now in teams of people collaborating in knowledge work.

大型语言模型内置了这么多事实记忆，这算是一个特性还是一个缺陷？

Is it a feature or a bug that there is so much fact memorization built into large language models.

有一种观点认为，模型仅仅记住“法国的首都是巴黎”这类事实其实是不好的。

And there's a school of thoughts that the models just wrote memorizing the fact that the capital of France is Paris is actually a bad thing.

我们更希望模型能够抽象地学习国家和首都的概念，而不是把所有事实都记在权重里。

And what we would prefer for the models to do is abstractly learn the concepts of countries and capital cities, but not to memorize all these facts in the weights.

所以我想听听你对区分记忆和学习的看法，以及目前模型是如何处理的，还有你打算怎么应对这个问题。

And so I'm curious what you think about disentangling memorization versus learning, how it's done in the models today, and then how you're thinking of approaching it.

是的，我觉得这是个非常有趣的问题。

Yeah, I think it's a really interesting question.

在某种程度上，你需要记住一些东西，才能把它们组合成更复杂的概念。

To some extent, you need to remember stuff in order to compose them into more complex concepts.

我觉得缺少的是弄清楚什么才是重要的、值得记住的。

I think the thing that's missing is figuring out what's important to remember.

我觉得即使现在，当你考虑学习新知识时，看看很多学术基准测试，它们关注的是如何学习非常具体的事实，比如某个非洲国家一座桥的长度。

I think even now when you think about learning new knowledge, if you look at a lot of these academic benchmarks, it's like, how can we learn very specific facts, like the length of a bridge in this African country?

这其实不是我们希望模型投入能力去记住的东西。

That's not something that you really want the models to devote capacity for.

我们自己也不会花精力去记这些。

It's not something that we devote capacity to.

我觉得如果你看看人类的记忆，你可以说得更多，但它是会有损耗的，因为智能的一个特点就是压缩重要的信息，把不重要的区分开来。

I think if you look at human memory, you can say a lot more about this, but it's lossy because part of the feature of intelligence is compressing what's important and separating that from what's not important.

我觉得你不能像有些人想的那样，把事实学习和非事实学习或技能学习完全分开。

I think you can't really separate fact learning from non fact learning or skill learning as some people would like to think.

如果你拿一个模型，有些人确实这么做过，把模型里所有的事实都去掉，只留下纯粹的核心或者类似的东西，那这个模型就非常不自然了。

If you take a model and some people have done this with models where you strip out all the facts and just have it the pure core or something like this, it's very unnatural as a model.

它连基本的东西都不知道，而你需要这些基本知识。

It doesn't know basic things and you need that.

为什么你需要这些？

Why do you need that?

为什么你不能查一下事实，然后直接有……

Why can't you look up facts and then just have

我觉得如果你看看事实的话……

I think if those you look at facts?

模型是怎么思考的，如果你需要回忆基本事实才能进行下一步思考，那你就走不了多远。

How the models think, if you need to recall basic facts in order to take the next step in your thinking, you can't get very far.

也许这是一个高层次的想法，但这也是我们认为训练非常重要的原因之一。

Maybe that's a high level intuition, but it's part of the reason why we think training is really important.

为了能对事物进行越来越复杂和深入的思考，你需要在某种程度上内化一些东西，这样才能把它们组合成更抽象的概念。

In order to think more and more complex and deep thoughts about things, you kind of need to internalize something so that you can compose them into more abstract concepts.

是的，而且

Yeah, and

之前有一些努力很难规模化，试图把这两者分开，并以一种允许模型检索和搜索而不是内化它们的方式来预训练模型。

there have been efforts before that were hard to scale to try and disentangle the two and pre train the models in a way that allows it to retrieve and search for things and not internalize them.

我们现在集体知道如何爬山的方法就是这种事实预训练步骤。

The recipe we know to hill climb on collectively right now is this fact pre training step.

我觉得这种方法的神奇之处，或者说神秘之处在于，你知道，传统上在计算机科学中，我们会有数据库作为自己的课程，我们会有算法，而数据库就像是关于世界的知识，比如各国的首都什么的，存储起来或者查询它们。

And I think the magic of, or the mystery of this approach is that, you know, traditionally in CS, we would have, you know, databases as its own curriculum, and we would have algorithms and the databases is like facts about the world and capitals of whatever, store them or query them.

还有算法，如何高效地处理信息并以样本高效的方式得到答案。

There's also algorithms of how do you efficiently manipulate information and get some answers in a sample efficient way.

我认为深度学习的魔力在于这两者现在被混在一起了，我们需要所有那些聪明的人和慈善性的可解释性研究来试图把它们分开。

And I think the magic of deep learning is that these two things are now mushed together, and we need all these smart people and philanthropic interpretability to try and break them apart.

我认为我们现在在AI被经济采纳中看到的很多现象是，这些东西又在逐渐分离，公司有自己的上下文，他们真的小心处理并精心设计。

And I think a lot of what we're seeing now in the adoption of AI into the economy is that these things are gradually separating again, where companies have their own context, and they really handle them with care and engineer them with care.

有一个通用的模型，对这些上下文完全陌生。

There's a generic model that's completely a stranger to these contexts.

模型在处理这些上下文。

And the model is operating on them.

对，而且

But for us, it's clear that there needs to be a certain convergence, at least with some cadence, where the facts and the stories and the details are getting mixed into the model.

但对我们来说，很明显需要某种融合，至少要有一定的节奏，把事实、故事和细节，混合到模型。

It has disadvantages as well, because if you have to, you know, capitals of countries are, you know, they can change, but it's not very frequent, but there's many other facts that are changing all the time.

这也有缺点，因为你知道，国家的首都，你知道，它们可以改变，但这不是很频繁，但很多其他事实都在变化，很多其他事实在变化，所有时间都在变化。

Just imprinting them into weights is a challenging thing to do.

把它们印到权重中。

I see.

我明白了。

So you're saying it's a false dichotomy that's trying to separate algorithms from databases here.

所以你的意思是，这里试图将算法和数据库分开，是一个错误的二分法。

What really matters is how to distinguish what's important to remember versus what's not important.

是的。

Exactly.

这就像是

It's an it's

我们做梦的一部分。

open part of how we dream.

你们在排序方面有从中汲取灵感吗？

Are you guys taking any inspiration from that in terms of ranking?

我觉得非常非常粗略地有。

Very, very loosely, I think.

只是这个想法，可能缺少一个阶段，就是你拿一个上下文，然后深入内化它。

Just the idea that that's kind of a phase that's missing maybe where you take a context and you deeply internalize it.

现在的情况是，一切都在测试时发生。

Right now it's like everything happens at test time.

你看用户给你的上下文，然后临时做一些思考。

You look at the context that the user gives you and you do some thinking on the fly.

但同样，你可能走不了太远，或者也许能走那么远，但一路上会犯错误。

But again, you can't get very far or you can get so far maybe, and you make mistakes along the way.

你怎么把这些消化回模型里，这样下次做的时候就能做对，还能取得更大的进步？

How do you digest that back into the model so that next time you do it, you do it the right way and make even more progress?

对。

Yeah.

那梦是什么呢？

And what are dreams?

梦是很疯狂的东西，说我们要造一个像梦一样的人工智能，听起来有点疯狂。

Dreams are pretty crazy things to say we want to build an AI that's like our dreams sounds a little bit like a nut thing to do.

那里面的连贯性不多。

There's not a lot of coherence there.

但有趣的是，在梦里我们会看到东西，自言自语，还会尝试各种可能性，比如在世界上、在社交场合里能做什么、不能做什么，而且特别偏向社交方面的内容，对吧？

But what's interesting there is like, what happens in our dreams, we see things, we talk to ourselves, and we experiment with the affordances of what can we do and can't we do in the world and social situations and in, you know, any, it's heavily biased towards social stuff, right?

所以对我们来说，在我们构建的东西里，我们给模型时间去回顾，从实际互动中抽身出来，去尝试它的各种可能性。

So for us too, with things we're building is, you know, we give the models the time to then go back, retreat from the actual interaction and experiment with its affordances.

它在环境里能做什么？

What can it do in an environment?

它知道什么？

What does it know?

它能多快地处理这些极端的边缘情况，就像我们晚上做梦时遇到的那些？

How fast can it handle these kind of tail extreme things, the same ones that we dream about at night?

你们是学术背景出身。

You guys come from academic backgrounds.

有没有一个典型的例子能说明这个问题，或者

What's a canonical example that motivates this problem or

到目前为止有什么成功案例？

that's a win so far?

对。

Yeah.

我有一个例子，杰西可以再举一个。

I have one example, and Jessy can give another one.

举个假设的例子，比如想象一个AI实验室，比如OpenAI，必须在一周后赢得某个数学奥林匹克竞赛。

A hypothetical one, for example, imagine one of the AI labs, say OpenAI has to win some math Olympiad in a week time from now.

他们会把所有数学教科书整理出来，让人标注该看哪些章节、哪些图表吗？

Would they construct a catalog of all the math textbooks and really have people annotate which chapters to get and which graphs to see?

还是说他们会实际收集这些数据，合成一些训练数据，启动训练任务，看看五六天后结果如何，然后开始评估之类的？

Or will they actually collect this, synthesize some training data, launch a training job, see where it lands in five, six days, start evaluating it and stuff like that.

所以对于任何训练过模型的人来说，很明显有一种更优越的方式来整合各种想法和能力，这涉及到训练的那种神奇之处。

So it's obvious for anyone who's trained models that there's a superior way to integrate across the ideas and capabilities and involves this kind of magic of training.

我们很清楚这必须在数学、编程、网络安全等高风险的领域发生。

And we are clear that this has to happen in those high stake domains of math and coding and cyber and stuff.

我们只是认为，这种神奇的能力实际上可以以有趣的方式掌握在更多人手中。

We just think much of this magic can actually end up in the hands of many more people in interesting ways.

为什么最终产品不是由基础模型实验室独占呢？

Why isn't it just the Foundation Model Labs that own the end product here?

你们这些巨头之间是怎么合作的？

How do you go between giants?

对。

Yeah.

所以我觉得我们的世界观和前沿实验室的不太一样，他们想要一个越来越大、越来越聪明的模型，能处理各种领域的问题。

So I think the worldview that we have is a bit different from the Frontier Lab worldview, where it's like we want one model that's bigger and bigger, that's more and more intelligent across a variety of domains.

相反，我们设想的是每个人都有自己的模型。

Instead, how we see it, we imagine this world where everybody has their own model.

很多人们想学的东西要么是私密的，永远不会出现在训练数据里，要么甚至是相互冲突的，比如，我想做任务的方式和另一家公司或另一个人想要的方式不一样。

A lot of the things that people want to learn are either private, things that will never see the light of day in a post training dataset, or even conflicting, like, Oh, the way that I want to do the task is different from how another company or another individual wants to.

我觉得我们已经看到很多这样的情况，用我们几十年来在机器学习中使用的工具很难把这些东西训练进模型里，那些工具需要有清晰的监督、有真实奖励信号，还要创造一个良好的环境，然后训练模型使用工具更好地完成编码任务。

I think a lot of these things we're already seeing are hard to train into the models with the same tools that we have used for decades in machine learning, which is you have really clean supervision, you have ground truth reward signals, and you create a nice environment and you train the model to use the tools to better accomplish this coding task.

相反，现实中发生的很多事情都非常模糊，或者很难说清楚什么才算好。

Instead, a lot of the things that actually happen out in the world are very ambiguous or it's hard to say what makes something good.

我觉得很多这些东西都是非常个人化的，而且我认为它们与前沿实验室对整体训练流程以及长期会存在什么样的模型的看法非常不一致，或者说不太一致。

I think a lot of these things are very specific to individuals and I think very misaligned or not very aligned with how the Frontier Labs think about the whole training pipeline and what kind of models will exist in the longer term.

对，而且我想补充一点，前沿实验室的P0是什么？

Yeah, and to add to it, I think, what is the P0 for the Frontier Labs?

在座的有些人跟他们关系很近。

Some of you here are pretty close with them.

他们的目标是实现AGI，打造一个在编程和数学上极其强大的通用模型，然后用它来自动化经济，或者解决密码学、国防等领域的长期难题，长期来看就是这样。

It's getting to AGI, getting this one generic model that's extremely capable in coding and math, and then using it to automate the economy or to solve really hard, you know, long term problems in cryptography and defense or whatever.

而且很清楚需要做什么来推动这个目标，你知道的，更多的预训练、更大的模型、更多的数据、更多的强化学习、更多的推理计算，诸如此类。

And it's pretty clear what needs to happen to push this, you know, more pre training, bigger models, more data, more RL, more inference time compute, that kind of stuff.

那是第一优先级。

That's P0.

大部分的资金和人才都投在那里。

That's where the majority of expenditure and talent goes.

当然，他们都在考虑记忆和持续学习。

And definitely all of them are thinking about memory and all of them are thinking about continual learning.

目前这更像是一个产品层面的努力。

It's just more of a product kind of effort right now.

我们认为这值得单独关注。

We think it deserves its own attention.

我们认为这方面需要突破。

We think breakthroughs need to happen there.

丹尼斯大约一个月前在红杉的活动上说得非常清楚，我们需要在这些话题上取得新的突破。

And Dennis and the Sequoia event about a month ago said pretty clearly that we need new breakthroughs around these topics.

显然，他们也在考虑这些。

And obviously they're thinking about them.

我们只是专注于这件事。

We're just focusing exclusively on this.

而且我们认为，关于数据在哪里以及谁拥有模型的激励机制，有些地方很有意思。

And we think certain things around incentives of where the data is, and who owns the model are pretty interesting.

所以，如果你能大规模地从许多人或组织那里学习，而不一定需要派人去和他们肩并肩地工作，那将是一个很大的突破。

So if you could learn from many humans or organizations at scale, without necessarily sending someone to work with them shoulder to shoulder, that would be a pretty big unlock.

关于这一点，也许还有一点是，我觉得世界上很多事情都需要改变。

And maybe another point on that is like, I think a lot of things need to look different in the world.

第一，需要新的研究突破。

So one is there needs to be new research breakthroughs.

第二，需要新的基础设施，让每个人都能训练小模型，而不是只有一个大模型、一次大训练。

Two is new infrastructure for training small models for everybody rather than one big model, one big run.

第三点我觉得是，研究和产品结合的方式需要改变。

Then the third I think is a different way of combining research and product.

现在我觉得前沿实验室里的研究人员们，

Right now I think there's researchers in these frontier labs.

他们训练模型。

They train the model.

然后把模型扔给产品团队，产品团队再通过提示词或上下文工程，在核心模型之上开发新的产品界面。

They throw it over the fence to the product team who then prompts or context engineers new product surfaces on top of the core models.

但在模型持续训练的世界里，我认为用户提供的输入与模型学习的内容、训练信号紧密相连。

But in this world where the models are always training, I think the inputs that users provide are very intricately tied to what the models learn from, what the training signal is.

研究和产品之间需要更紧密的循环。

There needs to be a lot more of a integrated loop between research and product.

虽然我们专注于解决很多核心研究挑战，这也是我们的背景，但我觉得我们也很注重如何尽快部署，以便从真实世界的反馈中学习。

While we're focused on tackling a lot of the core research challenges, and that's our background, I think we're also very focused on how to deploy this as quickly as possible to learn from actual feedback in the real world.

是什么激励你研究这个问题的？

What motivated you to work on this problem?

我觉得这显然是最大的挑战之一，现在大家都在讨论这个，因为模型太聪明了，还有什么可做的呢？

I think it's obviously one of the grand challenges in I think everybody's talking about it these days because the models are so smart, what else is left?

我觉得是在边缘学习，学习那些让模型有用的剩余部分。

I think learning at the edges, learning the remainders of what makes these models useful.

现在不仅仅是原始智能的问题，而是学习新东西。

It's not just about raw intelligence anymore, it's about learning new things.

我觉得这也感觉很根本，因为它回到了真正理解是什么让模型如此优秀。

I think it also feels very fundamental because it goes back to really understanding what makes the model so good.

现在模型从预训练中偶然学到了很多东西，我们并不真正理解为什么。

Right now the models incidentally know a lot of things from pre training and we don't really understand why.

就像互联网是送给我们的礼物，里面有各种各样的数据，包含了编程、写作等所有不同的例子。

It's like the internet was just this gift granted to us where there's a diverse set of data that contains all of these different examples of coding and writing and all of these other things.

事情就是这样发生的。

It just happened that way.

现在要解决持续学习这个问题，就要弄清楚预训练甚至后训练中的什么让模型能够以这些神奇的新兴方式泛化，并控制这个过程，以便公司拥有私有数据。

Now to figure out how to crack this problem of continual learning, it's about figuring out what about pre training or even post training makes it possible for the models to generalize in these magical emergent ways and controlling that process so that a company has a set of private data.

我们怎么让模型学得跟它们知道法国首都或怎么写Python一样好呢？

How do we make the models learn that just as well as the models know the capital of France or how to write Python?

所以我觉得这个问题挺有意思，值得好好想想。

So I think it's a really fun problem to think about.

丹，你之前是搞神经科学的，对吗？

And Dan, you came from the neuroscience world, is that right?

是的。

Yes.

我最初对意识、人类境况这类问题感兴趣。

So I was initially interested in questions around consciousness and the human condition and things like that.

模型有意识吗？

Are the models conscious?

我对此没有比你能读到的那些想法，我没什么高见。

I don't have any advanced thoughts on this more than you would read.

我觉得不觉得有，但重要的是有聪明人在思考这个问题。

Don't think so, but it's important that smart people are thinking about it.

我想说，我感兴趣的是人类如何思考、如何感知，就像以色列心理学家阿莫斯·特沃斯基常说的，他对人工智能不感兴趣，他感兴趣的是自然愚蠢。

I would say like, I was interested in how humans think, how humans perceive, and as Amos Forsky, the Israeli psychologist used to say, like, he's not interested in artificial intelligence, he's interested in natural stupidity.

所以，我一开始也是类似地，想看看人和动物是如何体验世界的。

So, would say like, I started kind of similarly trying to see how people and animals experience the world.

渐渐地，我的兴趣把我带到了统计和人工智能领域。

Gradually, you know, my inclinations took me to the stats and AI domains.

在那里我发现，记忆和持续学习中的许多相同问题都非常紧迫。

And there I figured that so many of the same problems of memory and continual learning are really, really urgent.

而我们当前系统中的解决方案与生物学中的相比，差距还很大。

And the kind of solutions we have in the current systems are pretty far from what we have in biology.

我不是那种会说机器应该像动物或人脑一样的人。

And I'm not one of these people who would say that the machine should be like, you know, like the animal or the human brain.

我不这么认为。

I don't think so.

有很多事情计算机比我们做得更好。

There's many things computers can do better than us.

但是人类的记忆里包含了很多非常不同的东西。

But human memory has these like very different things in it.

你知道，如果你想存储整个代码库，你可以用电脑，甚至不需要AI就能无损地存储一切并随时取用。

It's, you know, if you want to store a whole code base, or you can use computer, you don't even need AI on the computer to store everything losslessly and just get it.

但是人类大脑是在信息容量等限制下进化出来的，它拥有这些模糊的表征，然后可以抽象化、形成连接，并在第二天形成新的东西。

But the human brain evolved to work in these constraints of, know, information capacity and to have these fuzzy representations that can then, you know, be abstracted and form connections and form the next day.

当前系统除了通用的预训练步骤外，并没有真正具备这些。

Current systems don't really have that beyond the generic pre training step.

我真的很感兴趣的是，有什么方法可以把它构建进去？

And I was really interested in, you know, what are ways to build that in?

有什么方法可以从中学习？

What are ways to learn from that?

这更像是一个哲学问题。

This is more of a philosophical question.

你提到大脑里有不同的区域，不同的协同处理单元之类的。

You mentioned in the brain, there's a bunch of different real estate, different co processing units, whatever.

现代计算机架构里有CPU、GPU、内存，还有各种协处理器。

Modern computer architecture, there's CPUs, GPUs, memory, there's different co processors.

说到苦涩的教训，你觉得现在的情况是不是大语言模型已经收敛到，比如说，一个完全占主导地位的协处理器上了？

With the bitter lesson, do you think that what's happening is that LLMs are converged to, say, one coprocessor that's just totally dominant?

是不是所有计算都会发生在相当于GPU的语言模型里？

It's like everything, all compute is gonna happen in the GPU equivalent of a language model?

还是说这些模型是在内部自发地构建一堆协处理器？

Or do you think that these models are kind of building a bunch of coprocessors emergently inside the model?

拿记忆来说，你觉得模型自己会构建出大脑中擅长记忆的那部分吗？

Take with memory, do you think that the models themselves will just build whatever part of the brain equivalent would be that's good at memory?

还是说需要另一个独立的架构？

Or do you think there needs to be another standalone architecture?

对。

Yeah.

记忆是几乎涌现出来的属性，还是说它是独立的？

Is memory an emergent property almost versus distinct

过程？

process?

对，几乎所有的东西。

Yeah, and almost everything.

我们智能所需的一切，是不是只要训练数据更好、计算规模更大，就会自然涌现出来？

Is everything that we need in intelligence will just be emergent with better training data and more scaled compute?

对。

Yeah.

我想从当前AI部署的更表面角度来看，这远不止GPU，我们看到各种沙盒在爆发，模型在其他计算机上运行并尝试各种事情。

I would say just on a more superficial perspective on the current deployment of AI, it's way more than just GPUs, and we're seeing all these sandboxes exploding and models operating on other computers trying things.

我更多是指

I more mean on the

模型架构层面，而不是说，对，那个

model architecture level rather than on Yeah, the

所以其他实验，要么我们之前参与过很多不同架构的实验，比如状态空间系列和其他，试图更高效地处理非常长的上下文。

so other experiment, either there have been many previous experiments on different architectures that we contributed to, like the state space family and others to try and handle very, very long context more efficiently.

这些方法的问题在于，最终都是个权衡，通常是内存和准确性之间的权衡，而且这里的内存不是行为认知意义上的，而是计算机意义上的，对吧？

The thing with all these methods, it ends up being a trade off, usually a trade off between memory and accuracy and memory not in the behavioral cognitive sense, memory in the computer sense, right?

与其像Transformer注意力那样，内存占用随序列长度呈二次方增长，这些模型是——

Instead of having the memory footprint of the transformer attention, which is quadratic in the sequence length, these models are-

有些人声称它们具有次二次方的复杂度，是的，

Some are claiming they have sub quadratic Yeah,

有些人声称有，有些确实有，对吧？

some are claiming, some do have it, right?

而且一些最好的中国模型有受这些状态空间架构启发的层，成本不是二次方的。

And some of the best Chinese model have layers that are inspired by those state space architectures and are not quadratic in cost.

问题是，在我们手里，我们发现你总是为了内存而牺牲准确性。

Thing is that in our hands, we find that you always compromise accuracy for this memory.

天下没有免费的午餐。

There's no free lunch.

我们想说的是，如果你真的追求极致效率，你要做的是思考如何消耗更多的计算资源。

And what we're saying is like, look, if you're really bitter less and pilled, do you want to do is you want to think how can I burn more compute?

我怎么才能把它用在那些我从未见过的新内容上呢？

And how can I burn it on new context that I have not seen before?

所以我们也一样，我们也想追求极致。

So we're as bitter less and pilled as anyone else.

我们也不认为AGI的整体发展方向会很快结束。

And we are not betting that the overall direction of AGI is gonna end anywhere soon.

只是觉得还有更多的计算资源可以扩展。

Just think there's more compute to scale.

如果我真的想理解肖恩，理解肖恩的工作和肖恩的背景，光靠重读文件是行不通的，尤其是像你这样的人，尤其是像你这样的特别人物。

If I truly want to understand Shaun and Shaun's work and Shaun's context, just rereading files is not going to make it, especially for a special person like you.

得训练一个特别的模型。

Gotta train a Special derogatory.

我们得训练一个一千亿亿参数的模型。

We gotta train a 100,000,000,000,000

就为了这家伙。

parameters for this guy.

余弦。

Cosine.

余弦。

Cosine.

是的。

Yep.

你觉得人们最关心他们的模型学习什么？

What are you finding that people care most about their models learning?

比如，是记住组织里的事实吗？

Like, is it memorizing facts about the organization?

还是记住像，啊，不，我们这样做的CI？

Is it remembering like, ah, no, we do CI this way?

还是说，人们到底希望什么？然后这可能会影响到你如何对记忆槽进行排序等等。

Is it like, what are people actually hoping to And then maybe this feeds into how you do the ranking of memory slots and all that.

对，想想看，如果你看看人们现在在应用层花时间做什么，很多都只是试图让模型为你的用例工作得好。

Yeah, well, think if you look at what people are spending their time in the app layer doing these days, it's a lot of just trying to make the model work well for your use case.

哦，我希望模型能根据我的品牌风格来设计我的网站。

Oh, I want the model to, let's say, design my website with my brand style.

这是现在很常见的例子。

That's a very common example these days.

但人们用智能体做的事情有很多种，比如学习如何运行工作流程，或者你特定的写作方式。

But there's many kinds of different tasks that people do with agents, like learning how to run a workflow or your particular way of writing, let's say.

所以有很多种不同的东西。

So there's many kinds of things.

说实话，我觉得当我们考虑这些方法时，回到事实和技能的区别，其实并没有区别。

And honestly, I think when we think about these methods, kind of going back to this distinction between facts and skills, there really is none.

我觉得这些方法对此并不区分。

I think the methods are kind of agnostic to that.

对，对我来说，这是很自然的事情。

Yeah, to me, it's like the natural thing.

几乎所有的应用层基本上都是前沿模型加上搜索工具之类的循环包装。

Almost all the app layers are basically a frontier model wrapped in a loop with search tools and stuff.

他们跟我们合作，主要就是想找到一种方式，能更快、更高效地处理他们的数据，而且还能更贴合具体场景。

And what they're all interested in doing with us is finding ways to kind of interface with their data in a way that's, you know, faster, more efficient, and also is more contextual.

所以几乎所有人都希望，把公司里的知识编码成更高效的形式，这样我就不用再去查了。

So almost all of them, it's like, we want to have our, you know, our firm knowledge, you know, be encoded in something that's more efficient that I don't have to research.

我们希望模型能精准地知道，该把某件事分派给谁。

We want to have the model know in a targeted way, who's the person I should triage a thing to.

我们只是在向他们展示，通过相对轻量的训练，这些能力模型就能本能掌握。

And we're just showing them that with pretty lightweight training, these things can be instinctual to the models.

他们不需要那些复杂的、长时间的REPL循环来解决这些问题。

They don't have to have these very involved long REPL loops to solve them.

所以在某种意义上，这就像是个“RAG杀手”之类的东西。

So it's, in a sense, it's like, you know, it's a rag killer kind of thing.

再说一遍，我们总是可以做RAG，总是可以检索，但人们感兴趣的是用这种方式对接非常庞大的数据层，并自动化处理那些非常重复的事情。

Again, we can always do a rag and we can always retrieve, but that's the thing that people are interested in interfacing with very large data planes and automating very repetitive things this way.

对，我想深入聊聊这个“RAG杀手”的说法，抱歉老调重弹，但我还没完全理解。

Yeah, and I want to double click on this RAG killer thing, and I'm sorry to beat a dead horse, I just don't fully grok it yet.

这个前提是不是说，做RAG和更新模型权重之间存在某种权衡？

Is the premise that there's some trade off between doing RAG versus updating your model weights?

是不是说你应该两者都做？

Is it the idea that you should be doing both?

比如，哪些类型的事情应该放在权重里，哪些应该外部化到RAG？

Like what types of things should be done in the weights versus what types of things should be external ized to RAG?

我觉得这是个未解决的问题。

I think it's an unsolved problem.

我觉得没人有答案。

I don't think anyone has answer to it.

我们都在努力解决。

We're all working on it.

这也是关于生物记忆的根本问题，什么应该内化，什么不应该。

It's also the fundamental question of like biological memory, what should be internalized versus what not.

我确实认为，比如，你需要记住一年前住过的酒店房间号吗？

I do think that things that are like, you know, do you need to internalize the room number in a hotel that you were in like a year ago?

可能不需要，不会存在你的神经组织里。

Probably no, not in your neural tissue.

可能写下来更好，但你现在需要记住家里的密码吗？

Probably that's good to write down, but do you need to internalize maybe the password to your home right now?

可能在未来几年内，把它记在某个地方是有用的。

Probably it's useful for the next few years to have that imprinted somewhere.

所以，这如何应用到知识工作和产品中呢？

So yeah, how does this translate into like knowledge work and products?

这仍然是我们正在探索的问题，我们尽量采用尽可能少的启发式方法。

This is still something we figure out and we try to take the approach that we try to use as few heuristics as possible.

对数据运行过滤器很容易，比如我会保留这个，丢弃那个，用这个训练，用那个训练。

It's easy to run filters on the data and say like, I'm going to keep this, discard that, train on this, train on that.

但作为人类，我们刷抖音，接触大量垃圾信息，大脑仍然能够学习，不会完全失控，我们认为模型也应该如此。

But as humans, you know, we watch TikTok and we, get exposed to a lot of garbage and still the brain is able to learn and not completely go off the rails, and we think models should be the same as well.

是的，也许具体到短期，我觉得现在很多人担心的是，让这些智能体连续运行数天会产生巨大的推理成本。

Yeah, maybe concretely in the short term, I think a lot of what people are worried about these days is the huge inference costs of running these agents for days on end.

高推理成本是件好事。

High inference costs a good thing.

意思是，Sonya 和 fireworks 合作的代币。

Mean, tokens for Sonya works with fireworks.

她真的很喜欢 XIA。

She really loves XIA.

喜欢推理。

Love inference.

我们也喜欢推理。

We love inference too.

对。

Yeah.

短期内，我认为这是最直接的痛点。

In the short term, I think that's the immediate pain point.

为什么你一遍又一遍地读同样的文件，即使是在同一个查询中？

Why are you reading the same files over and over again, even in the same query?

但确实，同一公司里的人，会一遍又一遍地对相同的文档运行相同的查询。

But definitely across people in the same company, they're running the same queries on the same documents over and over again.

这应该是模型本来就知道的事情。

That should be something the model just knows.

就像你问一个员工，他们不会在搜索框里输入‘我昨天在做什么？’

In the same way you ask an employee, they don't type into the search box, What was I working on yesterday?

他们就是知道。

They just know.

但是缓存不能解决这个问题吗？

But doesn't caching solve that?

我觉得在某种程度上，是的。

I think to some extent, yeah.

但我觉得回到这个问题：哪些应该内化，哪些是在测试时检索的。

But I think going back to this question of what should be internalized versus what's something you retrieve at test time.

我觉得，再说一次，很多是关于建立你的知识。

I think, again, a lot of it is about building on your knowledge.

如果你总是用RAG，你就没法做出像“哦，我看到团队里有人在研究这个”这样的联想。

If you are always doing RAG, you can't make associations like, Oh, I see somebody on the team is doing this research.

我隐约记得一个大概，哦，有件相关的事情你可能想知道。

I recall at an abstract level, Oh, there's this related thing that you might want to know about.

你甚至都没问过这个。

You didn't even ask about it.

但我觉得这类联想只能发生在模型参数里，因为它们并不是“你让我搜这个，我就去搜这个。

But I think these kinds of associations can only happen in weights because they're not really about, you asked me to search for this, I'm going to search for this.

而且我觉得，我觉得检索系统，主要限制，主要问题不在于存什么、放哪里。

And also, I think the main limitation with retrieval systems in general and in AI specifically is the problem is not so much what to store and where to put it.

问题在于怎么去寻址，怎么去查询这个东西。

It's the problem is like how to address it, like how to query the thing.

你甚至知道要找什么吗？

Do you know what to look for even?

这涉及到某种直觉，有意思的是，有趣的是，有时候模型有时候没有，它们不知道往哪儿看。

And this involves some sort of intuition that sometimes the models don't have interestingly enough, they don't know where to look.

而且特别是如果你局限于目前的做法，也就是关键词搜索，那在强化学习里扩展起来更容易，而且在嵌入和基础设施方面涉及的也最少。

And especially if you're limited to the current way of doing things, which is keyword search, it is just easier to scale in RL and least involved in terms of like infra for embeddings and stuff.

所以，是的，知道要搜索什么是一种直觉，可以以这种方式发生，而且这也和缓存及推理有关。

So yeah, knowing what to search is something that's intuitive and can happen in the way it's and also about caching and inference.

比如，这家公司一开始就是让我们深入研究KV缓存和缓存技术。

Like, of this company started with us taking it like a deep dive into like KV caches and caching.

这真是个迷人的东西，对吧？

And this is a fascinating thing, right?

KV缓存是当前做法中的一个庞然大物，你想想看，一个维基百科文章的KV缓存，比如泰勒·斯威夫特之类的，就会占用GPU上80GB的HBM内存，而整个Llama 70B模型，它的全部权重也就大约100GB。

KV cache is a monstrosity of the current way of doing things that, you know, think about it, a KV cache for a single like Wikipedia article for some, you know, Taylor Swift or something like this, it will be like 80 gigabytes of HBM memory on the GPU and an entire Lama, it's for say a 70B Lama model.

整个模型的权重大约100GB。

And the entire weights of the model would be about 100 gigabytes.

而且有些扭曲的是，它们能记住整个互联网。

And with some distortion, they remember the entire internet.

为什么这个东西这么高效，一个东西就这么比特高效。

And how come this thing is so, one thing is so bit efficient.