如何构建长周期 AI 智能体 — Mitch Troyanovsky, Basis

这并不意味着代理被告知要去国税局网站。

That doesn't mean the agent is told to go to IRS website.

对吧？

Right?

可能去，也可能不去，这取决于情况，但关键是，我认为作为代理系统工程师，你要决定对具体情况要具体到什么程度，显然你更倾向于不那么具体，对吧？

It might be, it might not be, it sort of depends, like, but the point is that the way I think about it is the level of as the kind of agent, you know, systems engineer you are making the decision as to how, specific do you want to be with the situation obviously you prefer to be less specific, right?

我认为擅长构建代理的人都知道，在边缘情况下，给出原则、原因和更多背景信息，让代理自己解决问题，效果要好得多。

I think as as people who are good at building agents know like it's much better at the margins to be able to give principles and the whys and more context and let them figure it out.

所以对于行为，你实际上不想定义所有可能发生的事情，而是要说，不不不，我们知道，比如说，你在制作PowerPoint时，在把幻灯片交给用户之前拍张照片，就能发现问题。

And so with behaviors you want to actually not define every possible thing that can happen but just say no no no, We know that for example, let's say you're making PowerPoints, taking a picture of the PowerPoint before you give it to the user is gonna catch issues.

我们确信这一点。

We know that for a fact.

对吧？

Right?

所以我们作为代理工程师，要表明立场，说，在代理要做的所有不同事情中，这是我要对你进行评分的那一项。

And so we as the agent engineers are going to take a stand and say out of all the different things you as an agent are going to do, this is the thing I'm gonna grade you on.

然后你可能在那方面稍微具体一点。

And then maybe you're being a little bit more specific there.

如果你愿意，你可以更具体，比如指定用这个确切的工具，但理想情况下你不需要这么做。

If you want, you can be more specific, like use this exact tool, but ideally you don't have to.

这取决于你的系统是如何构建的。

It depends on kind of how your your system is is built.

所以我认为，具体的程度取决于你想要达成的特定目标，以及这个目标在多大程度上能适用于各种普遍情况。

So I think it I think your level of specificity depends on the maybe, what specific outcome you're sort of trying to drive and how that much that outcome generalizes to the universal situations.

比如你在做PPT的时候，拍张照肯定能让效果更好。

Like if you're producing PowerPoints, of course taking a picture, like, makes it better.

不过，假设你的智能体速度特别快呢？

But, let's say instead your agent actually is a super fast agent.

那拍照就得花时间了。

Well, making a picture takes time.

所以对于这种极速运行的智能体，你可能就不想让它去拍PPT的照片了，不然时间就从三十秒变成一分钟了。

So maybe a super fast agent you don't want for it to take a picture of the PowerPoint because now you went from it taking thirty seconds to taking one minute.

但如果你有一个ASIC代理，需要二十分钟，那从二十分钟变成二十分钟三十秒，每次都值得做。

But if you have an ASIC agent that's taking twenty minutes, you went from twenty minutes to twenty minutes thirty seconds, do it every time.

所以这又回到了我关于产品方面的观点，这不仅仅是智能的问题。

And so this goes back to my point on the product aspect where it's not just an intelligence thing.

这是一个主观的练习，关于你希望代理在生产环境中如何表现，实际上我认为这就是为什么它对产品构建如此关键。

It is a subjective exercise about how do you want the agents to behave in production broadly and actually I think it's why it's so critical to the product building.

我们几分钟前谈到OpenAI在2023年的一项工作，需要80万个人工标注。

So we were talking a few minutes ago about that 2023 effort by OpenAI that required 800,000 human labels.

你们在2026年做的事情有什么不同？

How is what you're doing in 2026 different?

对，好问题。

So, yeah, good question.

所以，至少目前，我们并没有主动奖励底层模型。

So we are, today at least, not actively rewarding the underlying model.

我们目前没有通过奖励这个过程来对我们的模型进行后训练。

So we're not currently post training our own models by rewarding them on this process.

我确实觉得那是一个非常有意思的研究方向，我们以后可以聊聊，但那不是我们现在正在做的事情。

I do think that's a very interesting area of research, we can talk about that later, but that is not actively what we're doing right now.

那是我们另外在研究的，不过那更多是未来的事情。

It is something we're actually researching, separately, but that's, more in the future.

所以如果你退一步看，把智能体构建和上下文工程的工作类比成直接训练模型，是很有帮助的。

And so if you kind of if you maybe take a step back, it's useful to sort of analogize the work of agent building and context engineering to the work of training a model directly.

我觉得当人们想到上下文的时候，你知道，人们会说提示词、上下文，我觉得大家通常的心理模型是，哦，我写了一些英文告诉模型该做什么。

I think when people think about context, they, you know, people say prompts, context, I think the mental model people have usually is like, oh, I wrote some English on the model what to do.

我觉得这个心理框架是错的。

I think it's the right the wrong mental framework.

我喜欢的框架是把它看作训练数据。

I think the framework I like is thinking about it as training data.

只不过你是在运行时训练模型。

Except you're just training the model at runtime.

它就是训练数据。

It is training data.

因为模型是在推理阶段进行学习的，所以总的训练数据量要小得多。

And because you the model's learning at inference time, the total amount of training data is far lower.

对吧？

Right?

比如，整个系统中智能体在其运行过程中逐步学习或发现的所有上下文信息，显然比用于模型后训练的数据量低好几个数量级。

Like, the whole amount of context in your system that the agent would progressively learn or discover throughout its trajectory obviously orders a magnitude lower than the amount of data that you're training, like post training a model on.

所以，你实际上是在处理这些数据，并努力确保它们的质量达到最高，以便让智能体展现出你期望的行为。

And so, what you're doing is you're essentially taking this this data and you're trying to ensure that it is of the highest quality to get the agent to exhibit the behaviors that you want it to exhibit.

显然，数据只是其中的一部分。

And obviously the the data is only one part of it.

还有那个框架，也就是工具、能力、环境等等这些部分，当然还有你用的是哪个模型之类的。

There's obvious there's the harness, what are the tools, what are the capabilities, the environment, all that part of stuff, obviously which model you're using, things like that.

所以这里的不同之处在于，我们利用这个信号来改进整个智能体系统，这比试图把这个信号直接作为强化学习的奖励反馈给底层模型要少得多数据量。

And so what's different here is that we're taking this signal and using it to improve the entire agent system, which requires far less data scale than if we were trying to take the signal and more literally reward it in an RL capacity to the underlying model.

你几分钟前提到了一个评判者，能不能给我们讲讲你是怎么训练这个评判者的，或者怎么指导它去执行任务？我猜大家最想知道的肯定是：谁来评判这个评判者呢？

You mentioned a judge a few minutes ago, maybe walk us through how you train that judge to do what or how you instruct that judge to do what, and I guess the obvious question of like who judges the judge.

对。

Yeah.

是的。

Yeah.

我觉得这不仅适用于行为评估，也适用于一般的结果导向评估。

I think that's true not just for, like behavior evals but also for, you know, outcome based evals in general.

这个问题问得很好。

It's a great question.

真希望我们有更多时间深入探讨这个话题。

I wish we had more time, to go deeper on this.

现实情况是，我们既没时间也没资源去花大量精力把每个裁判都打磨得完美无缺，不过我可以给你一个大致的介绍。

The reality is we just don't have the time or resources to like, spend a huge amount of effort like, you know, perfecting every judge but I'll give you kind of that high level.

一般来说，你需要建立起一种直觉，来判断裁判的品味是否准确。

Generally, what you do is you need to build an intuition for if the judge's taste is correct.

我觉得，关于裁判及其品味的研究，是一个很有潜力的后训练研究方向。

And you I do think an interesting area of potential post training research is on judges and potentially, taste there.

今天，你需要设置好评判系统，让它掌握做决定所需的信息，并且有合适的框架来做这件事。

Today, you need to set up the judge so that it has the information it needs to make the decision and it has the right kind of framework to do that.

同时，它还得有足够的数据和思维方式来完成这项工作。

And that it has the, data and mentality to do that.

我给你举个例子，有些行为你可以想象会相当复杂。

So I'll give you an example with behaviors that you can imagine being pretty complicated.

目前，行为评判的成本相对较高，因为它是一个相当高级的评判系统，实际上也是一个智能体。

So right now at least the behavior judging is relatively expensive because it's a pretty advanced judge in that it is also an agent.

它不是传统意义上的评判者，它确实是一个智能体，因为它需要查看整个轨迹。

It's not a judge in the traditional sense, it's literally an agent because it has to look at the trajectory.

所以这相当昂贵。

So it's quite expensive.

你可以想象，在未来，像Braintrust这样的团队已经开始思考，如何更好地标注轨迹，这样你就能更容易地筛选出可能相关的部分，交给评判系统，而不是让它以某种方式查看整个轨迹。

You can imagine in the future and you know I think folks like Braintrust and others are starting to think about this is like how can you, label trajectories better so you can potentially more easily, filter the trajectory to only the potentially relevant parts to give to a judge instead of having it like look at the whole trajectory in some ways.

但如果你有一个轨迹，特别是那种时间跨度长、可能包含很多子智能体的轨迹，作为评判者，你需要思考：在这个轨迹中，我应该关注哪里？

But if you have a trajectory especially if you have a long horizon one that might have a lot of sub agents, you as the judge need to think about well where do I go in this trajectory?

我该怎么理解它呢？

How do I understand it?

你可能需要以某种形式对它有个整体把握。

You probably want to have a map of it in some form.

我到底在评判谁呢？

Who am I even judging?

假设你有一个子代理系统，层级深度大概是七层左右。

Imagine you have like a sub, an agent system with like depth of you know seven.

你可能会有整整七层子代理，比如我到底是在判断根节点的行为是否得当？

You could have literally seven layers of sub agents like am I judging whether the root behaves properly?

还是在判断这些其他代理中的某一个？

Am I judging like one of these other agents?

所以你需要给裁判设计好提示词，甚至可能还要微调一下，让它清楚该去评判哪里，理解行为本身是什么、在什么条件下发生，以及知道怎么判断这个行为有没有出现。

And so you need to actually properly prompt and potentially tune the judge so it has a good understanding of knowing where to go judge, understanding the behavior itself, like what is the condition and understanding how to like, you know, judge whether the behavior occurred.

这种方法有什么权衡或者代价吗？

Are there any trade offs with that approach?

所以我想说的是，是的，有一个人工验证的流程确实能保证，或者至少有助于确保一个严谨的方法，这样不太容易出错。

So what comes to mind is, yes, having a human validated process guarantees or at least help secure a rigorous approach that's less likely to fail.

但同时，你也不会得到像“第37步”那样的结果，就是AI实际上能做得更好、更高效，通过绕过这一步，直接跳过那三个步骤。

At the same time, you are not going to get a Move 37 kind of result where actually the AI would do much better work, much more efficient work by sidestepping, you know, this part and fast forwarding through those three steps.

那么，这种方法的优缺点和权衡是什么呢？

What are the sort of pros and cons and trade offs?

这是个好问题。

It's a good question.

我觉得首先，我想确保每个听众都明白一件事，那就是你把一个行为写下来是有成本的，因为这意味着你现在要维护一个状态。

I think for starters, and I get one thing that should be I wanna make sure it's clear I think to to everyone listening is that you writing a behavior down is expensive because it is something where you are now keeping state.

对吧？

Right?

你需要让它保持最新。

You need to keep it up to date.

如果你让模型或智能体表现出这种行为的方式是通过上下文，那么现在你可能有重复的状态，这取决于你怎么看。而且因为这种维护负担，你最好尽可能少地定义行为，所以不是说你看看需要做什么，然后说“嘿，我们把所有最佳实践都写下来，看看它有没有做到”，而是你挑出几个你认为最重要的、对生产环境泛化能力最强、影响最大的，然后关注这些，而不是所有东西。

If the way you are, making the model or the agent exhibit this behavior is through context, now you have potentially duplicate states depending how you're thinking about it, and because of that maintenance burden you ideally want have as little behaviours as possible so it's not that you look at what it takes to do texture and you say hey let's write all the best practices down and see if it's doing it it's that you take the couple that you think are most important that have the largest amount of generalization to production and are the most impactful and you care about those, not everything.

所以我觉得这是很重要的一点。

So I think that's one important part.

不过理想情况下，如果你做得对，理论上还是给那些“Move 37”留了空间的。

So ideally though, if you do it properly, you still have rooms for, you know, the Move 37s in theory.

但我觉得还有另一个方面，就是你到底在向别人推销什么？

But I think there's another part here, which is that what are you selling to someone?

比如说，如果你要雇一个工程师，你在公司里有一套流程，比如你先画个简单的架构图，跟CTO聊一下，得到批准，然后提交一个PR，再把那个PR拆成十个小PR，按顺序合并，确保每个测试都通过，然后部署，这就是流程。

If you are trying to just like if you were to hire, you know, an engineer, and you know, you know, at your job, at the company you have a process and the process is like, you, I'm just making this up but like you write a quick architecture diagram and you like chat with the CTO and you get it approved and you make a PR and you split up the PR in like tens 10, different smaller PRs into a stack and then you merge it in and you make sure every n10 tests you deploy it like that's the process.

想象一下，一个工程师跑过来跟你说，嘿，这是我的一千行PR，我现在就要合并到生产环境。

Imagine an engineer came to you and they're like, hey, here's my thousand line PR, I'm gonna merge in production right now.

如果它比你的方案更好呢？如果它是“Move 37”呢？

What if it's better than what you what if it's a Move37?

它可能比CTO想出来的方案更好，但这并不意味着它就是好的。

It could be better than what the CTO would have came up with But that doesn't mean it's good.

就是说，这并不意味着CTO或者公司会对这个结果感到满意。

Like, that doesn't good in the sense that doesn't mean the CTO or the company is happy about that result.

仅仅因为它更好。

Just because it's better.

对吧？

Right?

因为归根结底，你工作的原因并不是因为某一项单独的工作有多出色，而是因为它能规模化地应用到公司、系统或整个组织中。所以，客户从我们这里买到的，并不是说这会是史上最好的报税方案。

Because at the end of the day, the reason you perform work is not because any individual unit of work is incredible, but because you can scale it to a company, to a system, whatever it is, to an organization and so the thing that, you know someone is buying from us is not this will be the best ever tax return.

他们买的是那种信心，也就是

They're buying it that, you know, the confidence that

看起来最漂亮的那个。

The best looking one.

对，就是那个外观最好的，或者像你说的，它把那些数据搬过来之类的。

Yeah, the best looking one or like it, you know, it moved 37 the TBs over here or something.

对吧？

Right?

他们买的就是这种一致性和可靠性，以及一种可以信赖的东西。就像对待真人一样，随着时间的推移，他们会越来越信任你，然后逐渐赋予你更多的自主权。

They're buying that it's going to be consistent and reliable and something that they can trust that actually will, they will you know just like with a human they can learn to you know trust more and more and then granted more agency over time.

对吧？

Right?

而且我觉得，那种信任和可靠性，才是真正需要部署到现实世界里的东西。

And and and I think that that level of trust and reliability, like that's what you need to deploy into the real world.

你不需要那些Move 37级别的操作。

You don't need the Move 37s.

那种东西可能在国际数学奥林匹克竞赛或者数学比赛里需要，但在现实经济中干活就不需要了。

You need that maybe at the, you know, Olympiad, like math competitions but not like doing work in the real economy.

对。

Yeah.

听你这么说，我觉得特别有意思的一点是，你做这些事的时候，其实对底层模型的工作原理并没有真正的了解，对吧？

As I listened to you, of the many things I find fascinating is that, you're doing all of this without having, actual insight about how the underlying model works, right?

你其实和我们所有人一样，对吧？

Like you sort of like all of us, right?

你也是在通过模型的行为表现，比如那些痕迹，还有工具调用来猜测和推断，那这到底是怎么运作的呢？

Like you're sort of, guessing and inferring from how the model behaves through like artifacts and judging from tool calls and how does that work?

那能不能跟我们讲讲，每次模型更新或者新版本发布的时候，你是不是得重新审视一下你之前做的所有事情？

And then maybe walk us through each time a model changes or the next version of the model, you know, gets released, like do you have to then look at everything that you've been doing in the light

根据那个新模型来重新审视？

of that new model?

这也许又回到了那个，你知道，Opus 3.0.1.0.03 的问题，因为我确实觉得，在构建智能体的时候，尤其是围绕它建立公司的时候，有一点很重要，就是你不应该感到太意外。

Is that maybe goes back to the like, you know, Opus 3.0.1.0.03 because I do think one thing that's really important when you're building agents but definitely building a company around it is you should you shouldn't be that surprised.

你应该有一个对世界的认知模型，当事情发生变化时，你应该更新你的模型。

You should have a model of the world and as things change you should update your model.

但是，要想成功，你知道，你不能一直更新你的模型，你需要有一定的先见之明。

But, to be successful, you know, you can't just update your model all the time, you need to be right a little bit.

而且我确实认为，如果你真正内化了这些概念，对吧，先别管内部机制，先别管这个NLM，你有一个魔法盒子。

And I I do think that if you really internalize some concepts about this, right, that like you now have this forgetting about the internals for a second, forget about this NLM, you have this magic box.

你有一个魔法盒子，或者我有时喜欢称之为外星人，你可以把海量数据输入到这个外星人里，它能在那个魔法盒子里进行推理和学习，然后给你输出结果，现在工具调用也显然能用了，你可以把它接入到系统的其他部分。

And you have this magic box or this alien, I like to call it sometimes, and you could send in huge amounts of data into this alien and it will be able to, reason and learn at inference time within that magic box and then come back to you with output that, you know, now that tool calling obviously works and whatnot, you can plug into the rest of the system.

这基本上就是你需要知道的一切，而且我觉得一旦你真正理解了这意味着什么，并把它推到逻辑的极致，很多事情就会自然而然地浮现出来，因为你开始明白，就像，等等，如果我有这个魔法盒子，我能做这个，那是不是意味着它也能决定去调用另一个魔法盒子呢？

That's kind of all you really need to know and I think once you really appreciate what that means and you take it to its logical conclusion, a lot of stuff starts to fall out of that because you start to understand, it's like, well wait, like, if I have this magic box I can do this does that mean that it could decide to call another magic box?

你知道，这是否意味着它有可能把多个这样的盒子串联起来，对吧？

You know, does that mean that it could potentially, string together like multiple of them in a row, right?

这是否意味着它能够利用——显然在那一刻它有一些状态，虽然它是个魔法盒子，但它有激活状态，我们知道这一点，缓存就是这样工作的。

Does that mean it could, you know, leverage, Obviously at that moment it has some, it's a magic box but it has some state, you know, we know this it has an activation state, that's how the caches work.

所以存在某种激活状态，它必然会偏向当前的轨迹。那么也许在审查时，你想要一个不相关的轨迹，对吧？

So there's some activation state that by definition is going to be biased to that current trajectory And so maybe for review you want an uncorrelated trajectory, right?

就像一个全新的盒子，它只是很聪明。

Where it's like a new box, and it's just a smart.

我认为如果你建立了这些关于LLM的直觉，并将它们与组织设计和管理的基本原则结合起来，你就能开始触及智能体构建的前沿。

And I think if you build these kind of LLM intuitions and you combine them with you know maybe basic principles of like organizational design and management, I think you start to get to maybe what is like the frontier of agent building.

太有意思了。

Fascinating.

实际上，你是怎么建立这种LLM直觉的？

Practically how do you build that LLM intuition?

是通过一直读论文，还是和研究人员交流，或者了解前沿技术的走向？

Is that, by, just, reading papers all the time or talking to researchers or like getting a sense for where the state of the art is going?

对，我觉得其实都不是这些。我觉得关键就在于，我是说，读读推特什么的来了解情况是好的，但我觉得很多人太看重这个了。很多人觉得，哦，我看到了这条推文，看到了这个墨西哥的那个东西。我觉得问题在于，如果没有对事物运作方式和可能性的基本理解，很容易觉得事情变化很大，但实际上并没有。

Yeah I think it's none of that actually I think it is all about I mean I think reading like Twitter and whatnot just to understand is good but I actually think a lot of people over index on that like I think a lot of people they think like oh yeah I saw this tweet, saw this like this in Mexico thing I think the problem is without like a fundamental like rounding in how things work and what is possible it's easy to feel like things are moving around a lot when they're actually not.

自从o3以来，事情真的没什么大变化。我觉得o3之后几乎所有东西都相对在趋势上，不是说我知道这个确切的趋势，但我觉得都在同一个范式里，自从o3之后没有什么范式转变。

Things have really not changed since o3 I would say almost everything since o3 has been relatively on I don't know what to say on trend and not like I knew this exact trend but I would say it's all within the same paradigm like nothing paradigm shifting has changed since o3.

我觉得理解它和学习它的最好方法就是在自己的工作中大量使用它们。

And I think the best way to understand it and learn about it is to just use them in your own work a lot.

尤其是在编程中。

I think especially in coding.

然后试着理解一些东西，一个好的思维模型是，比如说我试着让一个代理帮我实现一个功能，但它没有按我想要的方式做。

And just trying to understand things like, a good mental model is, let's say I try to, you know, have an agent, you know, influence a feature for me and it didn't do the way I want it to.

那是为什么？

Like why?

真正限制的因素是什么？

What is actually the limiting factor?

你知道，这有点像那个著名的马斯克思维模式。

You know, it's kind of like the the famous, like Elon mindset.

就是说，你得找到主要的限制因素，然后去解决它。

It's like, okay, like you go to like the main limiting factor and you figure out.

我觉得如果你用类似的思维方式去对待智能体，去理解为什么它不能自动化这个任务。

Like, I think if if you apply a similar mindset to to agents and you understand, like, why could it not automate this?

是不是它真的不够聪明？

Was it actually not smart enough?

可能不是。

Probably no.

它们挺聪明的。

They're pretty smart.

它们已经聪明了好一阵子了。

They've been pretty smart for a while.

所以，如果你把这种思维方式应用到自己的工作中，我觉得这对建立直觉很有帮助，而且我在面试别人的时候也看到了这一点。

And so once you if you, like, apply that mindset to your own work, I find that is quite useful for building intuition and I see that actually with you know people I interview.

很多直觉最好的人，确实很多有机器学习背景，但那些没有背景的人，很多都是特别擅长自动化自己工作的人，特别擅长思考，特别擅长理解要构建什么样的系统。比如，我在Data Driven做过一个关于本体的演讲，大概一年半前吧，我觉得有很多人在自己的数据和代码库中思考本体，那些积极思考的人，不只是想着怎么提示模型，而是想着怎么构建系统，他们就会开始建立起很好的直觉。

A lot of the people with the best Asian intuition actually, yes a lot of people come from ML backgrounds but people who don't, a lot of them are ones who are just really good at automating their own work, really good at thinking about it, really good at understanding like what is the system to build like, you know, I did a talk at Data Driven about ontologies whatever year year and a half ago or something and I think you know there are a lot of people who think about ontologies in their own data in their own repo and like those people who are like actively thinking not just how do I prompt the model but how do I build a system they start to build really good intuitions.

所以我们确保在对话结束前把它讲清楚。

So we make sure to cover it before the end of the conversation.

你具体在Braintrust上开源了什么？能带我们看看这个项目吗？比如人们在哪里找到它、它的起源是什么，以及你为什么特别选择跟Braintrust合作？

What is it exactly that you're open source with Braintrust like walk us through the project where people find it, the genesis of it, why are you partnering with Braintrust specifically on this?

回到行为这个概念，其实你可以用Markdown把行为写下来，它既是一个规格说明，也是一个评分标准。

If we go back to the idea of behaviors, the idea is that you can actually write down in markdown what is a it is actually both a spec and a rubric.

我们管这个叫规格，有些人会问，这不就是评分标准吗？

We call this specs and there are some people who ask, isn't this a rubric?

对，确实是。

And it is.

它两者都是。

It's both.

之所以两者都是，是因为它不只是用来给智能体打分或者奖励智能体，还用来统一人的认知。

The reason it's both is because it is not just used to grade or potentially reward the agent, it's also used to align the humans.

我觉得这一点被低估了。

I think that is an underrated point.

就像我们之前聊到的，你希望智能体表现出什么样的行为，这其实是个主观的问题。

In that, how you want the agent to behave as we talked about earlier is actually a subjective question.

所以，内部需要建立一套流程，让所有人都达成共识，比如确认这就是最终的产品标准。

And so, internally it's it's you need to build processes to all agree on hey, like this is the product, right?

你希望智能体具体怎么表现？还是回到刚才那个快速验证PPT的例子。

How do you want the agent to behave going back to the example about you know the fast PowerPoint verification?

因此，你需要制定一个标准并把它们写下来。这个项目就是这么来的，当时我跟Braintrust Encore的CEO喝咖啡，说实话我记不清起因了，但我当时就在跟他讲这个概念。因为我们在内部也在用这套方法，我觉得挺酷的，就跟他分享了。

And so, you want a standard to kind of write that down and the the project kind of came about because I was, actually having coffee with the CEO of Braintrust Encore and I forget why honestly but I was like, I still tell you, I was like, I was explaining this concept to him, I was talking about this because we were doing this internally and I thought it was very cool.

他对此非常兴奋。你知道，我内部一直在思考的一个问题是，我觉得我们做了很多前沿的工作，但很少对外提及，因为说实话，我们一直都在忙。

And he got pretty excited about it and, you know, one thing that I had internally that we've trying to think about is we, I think, do a lot of really cutting edge work, but it's not something we talk about much because, to be honest, we're working all the time.

对。

Yeah.

就在我们开始录制之前，你正在展示你和联合创始人Matt之间的一些Slack聊天记录。

Right before we started recording, you were showing some internal slags between your co founder, Matt, and yourself.

如果我没猜错的话，Matt在凌晨四点给你发了Slack，内容是关于提示词自由度的因素。

And if I may disclose them, like Matt was sending you a Slack at 4AM and It was prompt free factors.

对。

Yeah.

而且那是昨晚的事。

And that was last night.

所以，我们录制的时候是周日晚上，而你早上6点就回复了那条Slack消息。

So it was a Sunday night as we were recording this and and you're and you should have you were replying it to to that, Slack at, 6AM.

所以，是的。

So, yes.

对。

Yeah.

不。

No.

Basis的联合创始人之间真是996工作制啊。

996 in full action amongst the co founders of Basis.

是的。

Yes.

对。

Yes.

对。

Yes.

我们其实没有996。

We we are, it's this there's not there's no 996.

对我和Matt来说，是全天候的。

It's for for for for Matt and I, it's it's 247.

对于公司其他人来说，大家工作很努力，但绝对不是996。

For the rest of the company, it's it's people work hard but it's not a it's definitely not a not a 996.

所以我们想多聊聊这个，分享我们在做的事情。我们资源有限，没法大规模宣传，所以我就说，我觉得这对Braintrust，甚至整个行业都很有用。因为如果你有一个标准，人们可以定义它，然后自动被吸收到可观测性平台、监控平台里。对于可能不太先进的人来说，也可以有现成的评判器，或者定义行为的方式，你不需要自己配置评判器，可以直接让它帮你评判，然后看结果。

And so, know, we we wanted to talk about it more and just share what we're doing and we, you know, we don't have a lot of resources to to like blast out, to people and so we were talking I was like well I actually think this could be really good for you know useful for Braintrust and honestly the whole industry because if you have a standard, that could be something that people define and can get automatically, you know slurped up into observability platforms, monitoring platforms and you know for people maybe who are less advanced it could also, you know, have out of the box judges or, you know, ways to define hey, here are the behaviors and you don't have to configure your own judge you can actually get it get it to judge it for you and see the results.

他听了之后挺兴奋的，所以这就是合作的由来。

And so he got pretty excited about that and so that's where, the collaboration came from.

这就是那个开源仓库里有的东西。

So that's what the open source repo has.

里面有一些小例子，还有一个你可以直接用的示例评判器，以及一些写好的行为案例，你可以拿来参考并在此基础上自己构建。

It has, you know, a couple small examples, it has, an example judge that you can use, it has examples of actually written behaviors that you can leverage and sort of build your own.

我觉得思考如何采纳这个标准很有用，但更重要的是要转变思维模式，别再觉得运行十个小时的代理就是个黑箱了。

And I think it is useful to think about how to adopt the standard but I think it is also maybe more importantly thinking about how to adopt the mindset of you know not thinking that an agent operating over ten hours is a black box.

它根本不是黑箱。

It's not.

它产生了海量的数据，如果你不了解它具体是怎么干活的，那对你们的客户其实是不负责任的。

It has a lot a lot a lot of data and you're probably doing a disservice to your customers if you don't understand like how it's going about the work.

你希望人们怎么使用这个开源项目呢？

And what would you want people to do with this open source project?

大概是贡献代码，或者在自己的项目里用它吧？

Presumably contribute to it, use it for their own purposes?

那它怎么才能成为行业标准呢？

Like how does this become an industry standard?

对，确实是这样。

Yeah, it's

这个问题问得好。

a great question.

其实我也不太清楚。

I don't actually know.

我觉得最酷的事情就是大家能往里面贡献想法。

I think the coolest thing would be for people to contribute ideas to it.

我觉得还有很多工作要做。

I think there's a lot of work left to do.

这还只是个开始。

Think it's just the beginning.

我之前提到过几件事，但还有很多事情要做，比如怎么做出好的评判标准，怎么正确地标注和分析轨迹，让评判标准更容易理解，因为按照目前的成本，你没法在所有生产环境中运行它，因为它要对每条轨迹都进行评判。

I mentioned a couple things earlier but there's so much to do around one how you make good judges, two like how do you properly label and dissect you know trajectories to make it easier for judges to understand because at the current level of expense you couldn't run this in all of production for example because it's running judges on every single trajectory.

但我觉得在行为标准之上还有很多工作可以展开。

But there's a lot that can be done I think with building out the work that sits on top of the behaviors.

而且我觉得，我们故意把标准做得比较灵活，就像技能一样，只是一个Markdown文件。

And I think also just seeing, you know, we purposely tried to make the standard relatively flexible, similar to skills, where it is just a markdown.

就像没有过度拟合，嘿，你必须要有这五个确切的词，你可以让它非常宽泛，也可以让它非常具体，只要它仍然是自包含的，让评判者能够看到行为并知道其展示的条件是否满足，如果满足，它是否被展示或没有被展示，只要它有这些，就有很多回旋余地，所以我认为我们想要的是灵活性。

Like there's not an overfit hey and you need to have these exact five words like you can make it very broad and you can make it you know very specific, as long as you're as long as it is still self contained to the point that a judge could look at the behavior and actually know like was the condition for it to be exhibited met and if so like did it get exhibited or did not get exhibited and as long as it has that like there's a lot of leeway there and so I think that's we wanted that to be flexible.

AI 构建者在构建自主的长期任务智能体时，还应该考虑什么？

What else should AI builders think about as they build autonomous, long horizon agents?

我们谈到了评判者，谈到了行为。

So we talked about judges, we talked about behavior.

你刚才提到了本体论，你在 Data Driven NYC 的演讲中也提到了，作为智能体生活的世界。

You just mentioned ontology, which in your talk at Data Driven NYC, you had mentioned as well as a world for agents to live in.

这适合放在哪里？

Where does that fit

在整体图景中？

in the picture?

是的，它们非常重要。

Yeah, they're super important.

很多人看到这个词，听到或想到智能体时，他们的心智模型总是会想到编程智能体，因为那是每个人的经验，至少是可能收听这个播客的人的经验，人们会想到编程智能体。

A lot of people when they see the word, when they hear when they think about agents, their mental model always goes to coding agents because that's everyone's experience with at least the people who probably listen to this podcast, like people think about coding agents.

编码代理很有意思，因为它们显然有一个框架，被部署进去，有特定的工具，还有在上下文中编码的特定行为，对吧？

Coding agents are interesting because they obviously have a harness, get shipped in, they have certain tools, they have certain behaviors encoded in them in the context, right?

顺便说一下，Codecs 是开源的，我强烈建议大家去看看那个开源仓库。

Can Codecs by the way is open source, I highly recommend people go look at the look at the open source repo.

但它们不控制自己的运行时训练数据。

But they don't control their run time training data.

对吧？

Right?

因为它们的运行时训练数据，回到我之前打的比方，就是它们的上下文，实际上是它们正在处理的代码库。

Because their run time training data, which goes back to my analogy earlier, which is your context, is actually the repo they're working on.

所以你可以让同一个代理，所谓的同一个代理，在一个代码库上表现还行，在另一个代码库上却表现非常出色，因为那个代码库有更好的运行时训练数据。

And so you could have codecs, the same agent quote unquote, on one codebase perform, you know, somewhat well and then another codebase performs spectacularly because that codebase is has much better one time training data.

对吧？

Right?

我。

I.

嗯。

E.

它可能有很好的技能，或者对代码库的操作有很好的了解，因为那里没有矛盾、混乱之类的东西，所以代码库的本体论对工程师来说一直很重要。

It has potentially, you know, good skills or good like, you know, context about you know how to operate in the code base because, there isn't like contradictions and confusions whatever it is and so, the ontology of your code base, it always mattered for engineers.

对于真正优秀的智能体来说，这一点同样重要，甚至更重要。

It matters just as much, if not more, for for really good agents over time.

实际上，如果你考虑非编码智能体，我认为这又上升了一个层次。

That actually I think goes up another level if you're thinking about non coding agents.

因为在编码中，你不拥有运行时训练数据。

Because in coding, you don't own the runtime training data.

而在非编码中，你拥有。

In non coding, you do.

对吧？

Right?

比如，智能体在运行时看到的数据，如果是Basis智能体，那么Basis拥有这些数据。

Like, of the data, quote unquote, that an agent sees when it's when it's a Basis agent, Basis owns.

对吧？

Right?

这是我们必须确保运行良好的训练数据。

It's training data that we have to ensure works really well.

再说一次，我说训练数据，其实指的是那种手写的上下文，对吧？

And again, when I say training data, I mean, like, effectively handwritten context, right?

或者是你更广泛的渐进式披露中的一部分，有些可能是手写的，有些不是，反正就是那些东西。

Or things that are part of your broader progressive disclosure, some, you know, could be handwritten, some could not be, whatever it is.

而且因为代理总是从零开始，设计一个对代理友好的本体论，对于构建长期运行的东西来说非常关键。

And the because the agent is always starting from scratch, designing that ontology in a way that is ergonomic for the agent is super key to building something that's long running.

这既包括那些静态的数据，比如所有代理都有的技能之类的，但一旦你面对真正长的时间跨度，比如一个状态化代理可能运行几个月，从几天到几个月，那么突然你就有了一个本体，回到Memento的例子，就是代理留给自己的信息，如果你只运行几个小时，可能只是几条笔记，但如果你运行几个月，那就是文件夹了，对吧？

That's both like the data that is like maybe static like skills and whatnot that all agents have but also once you get the really long horizons if you're talking about you know a stateful agent that's maybe operating over months, days to months, now suddenly you have an ontology of going back to the Memento example of information the agents left for itself which if you're operating for maybe a couple hours could be a couple notes, if you're operating for months you're talking folders, right?

而且你有大量的知识和上下文，描述了代理的亲身经历，所以这个新代理，虽然说是新的，但它在启动时只有非常压缩的上下文，它必须回到整个经历的状态中，所以设计本体让它容易做到这一点，以及你编码的行为，确保它做得好，这些就是让它真正运行得非常好的关键。

And you have so much knowledge and context that describes like the lived experiences of the agent that like suddenly this new agent, well new, that is standing up with you know effectively very compacted context has to get it back into the state of mind of its entire lived experience and so the ontology designed to make it easy for it to do that and the behaviors you encode that you know properly ensures it's doing that well are sort of the key to to making it work really really well.

设计本体到底是什么意思？

What does that even mean designing an ontology?

本体论实际上是什么？

An ontology practically is what is this?

是图数据库吗？

It's a graph database?

是一系列关系吗？

It's a series of relationships?

对，可以有很多不同的形式。

Yeah, could be a lot of different it could be there are different formats.

我觉得最简单的理解方式就是把它看作一个文件系统，里面有一定的结构。

I think the the simplest way to think about it is honestly just a file system, in which you you have like some structure.

显然，大多数人用的是虚拟文件系统，所以灵活性很大，文件和文件夹上还可以关联其他类型的元数据，对吧？

Obviously most people you do virtual file systems and so you have a lot of flexibility there so there could be other types of metadata associated with the files and the folders, right?

如果你愿意，也可以有节点和连接器，就像某种图数据库一样。

You could there could be you know connectors in nodes and like some graph DB if you wanted.

当然，也可以有嵌入向量。

Obviously there could also be embeddings.

有这么多不同的数据来源，你可以用来帮助丰富内容。

There's so many different, you know, sources of of data that you can get to, like, help enrich.

而且现在，顺便说一句，随着模型越来越便宜，越来越多的这些工作实际上可以通过推理来完成，而不是依赖像图或嵌入这样的确定性的东西。

And now, by the way, as models are getting cheaper and cheaper, more and more of that actually can just be done using inference instead of using determined like things like graphs or things like embeddings.

你知道，如果Luna的成本是免费的，那么突然你可以在整个本体上运行Luna，为代理总结内容之类的。

You know, you have if Luna costs is free, then suddenly you could run Luna's across your entire ontology and summarize stuff for the agent up or things like that.

所以这就是我大概的想法。

And so that's kind of how I would think about it.

也许还有一点，本体不仅仅指文件夹的结构。

And maybe also one more piece, it's not just the the ontology doesn't just mean the structure of the folders.

传统上，本体还涉及语言，对吧？

In ontology traditionally it's the it's also like the language, right?

比如对象和概念是什么？

Like what are the objects and the concepts?

因为归根结底，如果你在运行时进行训练，你需要确保不会混淆概念，而且事情大体上要合理，所以这就是我说的定义世界的意思，对吧？

Because at the end of day, if you're being trained at runtime, you need to ensure you're not confusing concepts together and that like things kind of generally make sense and so you're that's what I mean by defining the world, right?

你在定义智能体在探索周围世界时可能会看到什么。

You're defining, what the agent can expect to see as it goes and explores the kind of world around it.

你之前提到过，智能体的内部文档必须像代码库一样对待。

You mentioned somewhere that internal documentation for agents has to be treated like a code base.

删掉一个关键段落，智能体就坏了，就像删掉一行代码一样。

Delete a crucial paragraph and you break the agent just like deleting a line of code.

这些文档是本体之外的东西，智能体会像调用工具一样去搜索吗？

Is that documentation something outside the ontology that the agent goes search like a tool call?

这具体是怎么运作的，最佳实践是什么？

How does that all work and what are the best practices?

对，这是个好问题。

Yeah, that's a good question.

先快速区分一下，我刚才说的本体和历史，更多是指基础产品内部和生产产品里的情况。

So just to quickly separate the just now when I was talking about the ontologies and like historical, I kind of was referring more to, you know, inside of like the basis product and like the kind of like production product.

至于内部使用智能体，比如编码智能体，或者我们可能做的其他内部智能体。

In terms of like internal use of agents, let's say coding agents, or maybe other internal agents that we might make.

这里有点意思，因为根据你的设置，你的环境可能不太受你的本体论控制。

There, it's kind of interesting because you're in depending on what you've set up, your environment might be less controlled by your ontology.

因为正如你所说，在现实世界中，你必须去访问像Linear、Gong、Pylon这些不同的工具。

Because to your point, you know, in the real world you have to go and access like linear and Gong and Pylon and all these different things.

所以，我认为对于内部代理来说，关键之一就是要非常清楚什么是权威的，什么不是权威的。

And so, I think one of the keys for internal agents is having a very keen understanding of what is canonical versus what is not canonical.

就像如果我是一个刚加入组织的人，我可以去读所有的Gong记录。

And so, just like if I'm a human who joins an organization, I could go and read all the Gongs.

但我们今天的销售策略是什么？

But what is our sales strategy today?

如果你看两年前的Gong记录，你会得到很多背景信息。

If you watch the Gongs from two years, you'll get a lot of context.

但你不会知道当前的销售策略是什么。

But you won't know what your current sales strategy is.

肯定有一些权威的文档。

There must be some canonical, piece of documentation.

实际上，很多时候人类是通过跟别人聊天来学习的，你慢慢就懂了。

In practice, a lot of times humans learn this by just like talking to people and you kind of learn stuff.

但对智能体来说，这很难做到。

But with agents, it's hard to get that.

更重要的是，如果你想要真正的组织智能，你不会希望智能体从一个人那里听到一种说法，从另一个人那里听到另一种说法。

And more importantly, if you want real organizational intelligence, you don't want an agent hearing one thing from one person and another thing from another person.

或者让他们有不同的书面记录，比如当前的销售话术是什么，我们怎么做演示文稿，怎么写邮件。

Or having them have kind of different written records of like what is the current, you know, sales pitch how do we make our decks, do we make our emails.

你需要一个唯一的权威来源，这就是为什么我认为对于真正的原生AI公司，尤其是在未来——今天我觉得还太早——但尤其是在未来，清楚理解你公司的核心规范是什么，并将其组织成一个合理的本体论，确保它像代码一样保持更新，我认为这最终会成为公司内部人类最重要的职责之一。

You need one canonical source and that's why I think for true Asian native companies, especially in the future, today I think it's still quite early, but especially in the future, having a clear understanding of what your company canon is and organizing that in an ontology that makes sense and ensuring that that is kept up to date just like code in some form I think ends up becoming one of the most important parts of a human inside of a company.

而且在Basis团队里，你招的人做的是一些两年前根本不存在的工作，比如语言架构师或智能体经理。

And within the Basis teams you're hiring for jobs that literally did not exist, two years ago, like language architects or agent managers.

这些人是谁？

Who are those people?

他们是做什么的？

What do they do?

那他们需要什么样的背景呢？

And what's a good background for them?

对。

Yeah.

我们需要很多这样的人。

We need a lot of them.

所以如果你在听，并且想加入的话，请一定要联系我。

So if you're listening and you wanna join, please please, like, hit me up.

好问题。

Great question.

我们还在摸索中。

We're still figuring that out.

这并不容易。

It's not it's not easy.

以下是我目前知道的。

Here's what I know.

我觉得这个最近可能挺火的，但我觉得真正重要的一个技能就是好的系统思维。

You need I think this is probably gotten someone in vogue recently, but I think one thing one skill that really matters is, good systems thinking.

那好的系统思维从哪来呢？

And where does good systems thinking come from?

它来自于那些不得不去思考某种抽象概念、某个系统，并且设计得能在各种情况下都能运作的人。

It comes from people who have had to, think about some abstraction, some system, something and design in such a way that it performs in a plethora of situations.

所以很明显，如果你是个特别好的工程师，工程就是系统思维。

So, obviously, if you're a really good engineer, like, engineering is systems thinking.

不过我觉得历史上大部分工程其实不完全是基于系统思维的，你知道，更多是偏向执行层面的。

Now, I think the majority of engineering historically has not been really systems thinking based, you know, it's been a little bit more execution oriented.

但如果你想想最难的工程，比如，我在设计系统会是什么样子，我在努力创造正确的抽象。

But if you think about, like, the hardest engineering, like, hey, I'm trying to design, like, what the systems are gonna look like, I'm trying to, like, create the right abstraction.

那真的非常依赖系统思维。

That really is high on systems thinking.

但并不是只有这个职业是这样的。

But it's not the only profession that's like that.

我觉得法律其实也挺像这样的。

I, I think law actually is kinda like that.

你知道吗，在很多方面，我觉得开国元勋们可能会是非常好的上下文工程师或代理管理者，因为你得写一段英文，这段英文会被律师、法官等人在运行时解读无数次。

You know, in many ways I think about I think maybe the founding fathers would have been really good, context engineers or agent managers because you had to write, you know, a piece of English that was gonna be, you know, interpreted at run time literally millions of times by lawyers and judges and whatnot.

所以如果你在写法律，我不是说某个政客，如果你真的在尝试写一部法律，并且试图写好它，你就是在尝试用英文写出某种东西，能在各种情况下以恰到好处的抽象层次进行概括。

And so if you're writing a law, and I don't mean like, you know, some politician, if you're actually trying to write a law and trying to write a well, you're trying to like somehow write something in English that will abstract at just the right level across the universe of situations.

而且你还得对司法系统有心理理论，去思考他们会怎么解读它。

And you have to have like theory of mind over the judicial system to think about like how they'll interpret it.

你知道吗，有趣的是，你在某些机场会看到一些标志，上面写着不要带枪，不要带剑，不要带这个那个，列出一大堆东西，就像你列出一堆僵硬的规则一样。所以同样，你可以列出50条僵硬的规则，或者你可以写出一个恰到好处的抽象，完美地覆盖所有情况。

You know, it's funny you'll look in certain airports and you know sometimes they'll have these signs it's like you know don't bring a gun, don't bring a you know don't bring a sword, don't bring blah blah blah and it's like you list out 30 things to your point on brutal rules And so same thing, can list out like 50 brittle rules or you can like write the right abstraction that somehow covers it just perfectly.

所以我觉得，任何需要你以那种方式思考抽象的事情，都是成为优秀系统工程师的好练习。

So so I think anything where you need to think in abstractions in that way, I think is good practice, for being a good systems engineer.

想想看，那些管理过世界上最复杂Excel模型的人，说实话跟这也差不了太多。

Think, know, people who've had to manage like the most complex Excel models in the world is honestly not that dissimilar either.

所以我觉得，有很多潜在背景都可以培养这种能力。

So I think there's a lot of potential potential backgrounds for it.

你刚才提到数据驱动的时候，说的是你们部署的智能团队。你说在公司里部署智能体，就像招了三百个才华横溢但毫无背景知识的外星员工，所以才叫部署智能团队。

I think data driven when you spoke you were talking about your deployed intelligence team, and, you were saying deploying agents, at a firm, was like onboarding 300 brilliant alien employees who have no context, hence the deployed intelligence team.

那这些人具体是做什么的？

So what do those people do?

他们现在还在吗？还是说这个概念已经演变了？

Are they still around or is that concept evolved?

没有没有，当然还在。

No no no, of course.

DI团队真的很棒。

The DI team is awesome.

我觉得直到今天，我都不该再去搜这个词了，但我们确实提出了“部署智能”这个说法。

I think to this day, I shouldn't need to Google it, but, I we definitely came up with the term deployed intelligence.

我知道是因为你搜一下，排第一的公司就是我们。

I know because if you Google it, are the first company that that comes up.

这名字挺酷的。

It's a cool name.

我不知道有没有其他人用这个名字，我觉得它其实还没流行起来，但它的意思不是FDE。

I don't know if other people have taken on the name, I don't think it's actually caught on yet, but the idea is it's actually it's not FDEs.

所以不是那种进来帮你定制东西的工程师。

So it's not like engineers who are coming in and building something custom for you.

也不是现在有些公司里那种所谓的“代理项目经理”，我就不点名了，他们就是拿着代理构建器来帮你搭个代理。

It's also not, this kind of like agent PMs you see now at some companies that I won't name, where it's like, you know, these like PMs quote unquote are kind of coming in and like building an agent sort of for you using an agent builder.

其实两者都不是。

It's actually neither of those.

而是那些对职业有深刻理解和共鸣的人，知道怎么建立好的流程，也知道当你突然可以把某些事情交给代理时，怎么才能成功，对吧？

It's it's people who have a deep empathy and understanding of the profession, of like what it takes to put in place good process, and what it takes to be like successful when suddenly you can start to offload certain things to to agents, right?

所以DI团队的工作就是和我们的会计事务所合作，帮他们过渡到这个新时代，对吧？

And so TI team's job is to come work with our accounting firms to help them transition into this new era, right?

我们给了他们魔法，但如果不教他们怎么利用这个魔法，不只是日常使用，而是它如何改变事务所的本质，会不会改变他们能接什么业务、雇什么人、怎么扩展成大家都想成为的未来事务所？

We're giving them magic, but if we don't teach them how to leverage the magic, not just in the day to day, but how it changes the nature of the firm, does it change, what kind of business they can take on, who they hire, how can they scale to be that firm of the future that that everyone wants to be?

这就是DI团队和人们紧密合作要做的事情，因为让他们自己去学或者自己做是不公平的。

That's what the DI team works really closely with people, to do because it's it's not it wouldn't be fair to ask them, to like go and, you know, learn that themselves or do that themselves.

相反，他们带来了大量关于自己公司的知识，了解过往的运作方式，当然还有他们的团队。我们可以把这些和他们结合起来，再加上我们部署智能体的专业知识，最终打造出真正处于前沿地位的会计师事务所。

Instead they bring a lot of the knowledge about their firm, about how things have worked, obviously their people, and we can combine that with our knowledge and how to deploy agents and I think together get to something where it can be a really really frontier accounting firm.

说到前沿领域，既然我们快结束这次对话了，我很想花点时间深入聊聊这个话题。

Speaking of frontier, maybe as we start getting to the end of this conversation, would love to spend a little bit of time on that.

你知道的，2026年的一个大热点就是自我提升这个概念。

You know, obviously a big topic in 2026 is the concept of self improvement.

在你的整体规划中，特别是在基础模型和自主智能体方面，这占据着怎样的位置？

Where does that fit in your picture Basis and with Autonomous Agent?

我觉得你提到过，智能体会对其他智能体发展出一种心智理论。

I think you talked about agents developing a theory of mind about other agents.

所以从系统层面来看，能给我们描绘一下自我提升具体是什么样的吗？

So at a system level, paint that picture for us, what does self improvement look like?

对。

Yeah.

其实我们内部在讨论我之前提到的那些研究方向时……

We actually internally, one of our, in the thread I talked about some of our research directions.

我们一个重要的研究方向就是如何真正实现闭环，我们喜欢这么叫它。

One of our big research directions is how do you actually close the loop as we like to call it.

从“嘿，这个代理犯了错”或者“表现不佳”之类的，到我们改进系统去解决这个问题。

Going from Hey, an agent made a mistake or not performing or whatever to you know, we've gone and improved the system to do that.

我觉得这个闭环会很快实现。

And I think that closing the loop is gonna happen pretty fast.

我不知道整个闭环能不能完全闭合，但我觉得到年底应该会相当接近。

Think you'll have like I don't know about the entire loop being closed, but I think you'll be relatively close by end of year.

我觉得随着代理对自身以及对其他代理的心智理论越来越完善，会有两件事发生。

I think that So as agents are getting better theory of mind over themselves and therefore other agents, you have two things happening.

第一，在运行时，它们能更好地协调子代理，也能更好地调节自己的环境；同时，它们也变成了更好的上下文工程师。

One is, at runtime they're being better at orchestrating sub agents and also regulating their own environment for themselves, but it also means they're becoming better context engineers.

对吧？

Right?

它们变成了更好的工具链工程师。

They're becoming better harness engineers.

现在，他们在构建智能体系统方面的能力，比他们在开发大多数软件方面的能力要差得多得多。

Right now, they are far far far worse at engineering agent systems than they are at engineering most software.

差远了。

Far worse.

因为，说白了，这种工作太新颖了，在他们的训练数据里根本没多少。

Because, by definition, like, that kind of work, which is so novel, has not seen a large amount in their in their training data.

所以他们的直觉非常差。

And so they have very bad intuition.

其实，我觉得很多做智能体的人犯的错误是，他们有一种奇怪的直觉，觉得上下文里有点乱码或者智能体有点乱，好像比代码里有点乱更能接受。你会看到有人因为代码文件没抽象好就抓狂，但他们的上下文却是一团糟，这太搞笑了，因为上下文实际上影响运行时的表现，而代码的组织方式并不影响运行时的表现。

Actually, think a lot of the mistakes a lot of agent builders make is, you know, they, they have this weird intuition that like slop in your context or your agent is somehow more acceptable than slop in your code and you'll see people like freaking out over, you know a a code file that isn't abstracted properly and yet their context is like total shit, which is hilarious because the context actually affects the performance at runtime, the organization of code does not affect the performance at runtime.

所以，我觉得，很明显，因为长期养成的习惯，很多工程师把代码看得比英文（提示词）更宝贵。

And so, I think a lot of, you know, obviously because of, built up, beha- you know, behaviors, a lot of engineers, they they they treat the the the code as more precious than the English.

但实际上英文（提示词）更宝贵，因为英文（提示词）影响表现，代码不影响表现，对吧？

When actually the English is more precious because the English affects the performance, the code does not affect the performance, right?

如果逻辑一样的话，代码不影响表现。所以，我觉得随着智能体变得更好，他们也会更擅长这种工程，就是怎么构建智能体系统，到那时你才能真正开始闭环。

If the logic assuming the logic is the same it does not affect the performance and so, I think as agents get better and they get better this type of engineering of like how do you build agent systems, you'll start to actually be able to close the loop.

因为要形成闭环，你得收集反馈信号，而且需要大量的信号。

Because in order to close the loop, right, you need to take the signal that you are getting, and you need a lot of signal.

行为层面的数据就是获取更多信号的一种方式。

One of the pieces of behavior, one of the points of behaviors is a way to get more signal.

如果你只靠最终结果来评估，那得到的信号就太少了。

Whereas if you only have outcome based evals, your signal is pretty sparse.

所以，怎么把这些信号都利用起来，真正去优化你的系统呢？

So how can you take all of that signal and actually now use it to, improve your system, right?

你可以把它推广到整个系统中，这可以通过一个智能体来完成，比如更新上下文、改变工具的性质、更新测试框架等等。

You proliferate it throughout the system and that could be done by an agent that is like updating the context, like changing the nature of the tool, updating the harness, etc.

我觉得这种方式应该是可泛化的，但不会过拟合。

And I think you'll prob in a way that is generalizable, but it doesn't overfit.

如果你现在就想针对某个特定信号做过拟合，那今天就能做到。

Can you can do it today if you wanted to overfit to like some signal.

但如果你想让它具备泛化能力，智能体还得再进步一些，这方面还有不少工作要做。

But if you wanted to do it in a way that generalizes, you you need the agents need to get a bit better still and there's some more work to do there.

那在模型层面上的自我改进呢？

What about self improvement at, the model level?

我记得你之前提到过，你们目前还没有在模型本身上做太多强化学习。

So I think you mentioned earlier that you guys don't do yet, much reinforcement learning on the model itself.

你们主要是在做框架方面的工作，如果我没理解错的话。

You've done you're doing mostly harness work, if that's correct.

但这是不是意味着，下一步的大方向就是开始真正对模型做强化学习，来提升行为遵循度？

But that's the is that the next big bet to start doing actual reinforcement learning on

行为遵循度？

behavior adherence?

对，这是个好问题。

Yeah, it's a great question.

我觉得强化学习的难点在于，首先要确定奖励函数是什么，然后要决定如何把奖励分配到整个过程中，对吧？

The way I think about it is that the hard part of reinforcement learning is deciding what is your reward function and then deciding how you're going to allocate that reward over whatever occurred, right?

所以这其实就是我们正在进行的核心研究。

So we are that is like the active research, that we are doing.

因为那就是行为，那是信号的来源，对吧，你可以用它来构建奖励函数。

Because that's what behavior that's a source of signal, right, that you can use to craft into a reward function.

同样，像其他一些生产监控工作和你们构建的一些评估也是如此。

Same with, like some of the other kind of production monitoring work and some of the like evals that you build.

所以，决定如何构建好的信号，做研究，理解这意味着什么，尤其是在一个不完全可验证的领域，这就是我们在做的工作。

And so deciding how to build good signal and doing the research and understanding what does it mean especially in a non perfectly verifiable domain, that's the work we're doing.

无论你是把那个信号通过权重传播，你知道，通过正式的强化学习，还是通过框架传播，不管你怎么称呼它，比如非正式的强化学习或者框架工程，我认为这跟信号的发展是两回事。

Whether you take that signal and then like proliferate it through the weights, you know, through, formal RL or you proliferate it through the harness, through whatever you want to call it, like informal RL or like harness engineering, I think is like a separate question from the development of the signal.

但我认为信号的发展才是真正困难的部分，也是我们真正关注的重点。

But I think the development of the signal is really the hard part and that's the part that we're really focused on.

而今天，我们不会直接进入权重。

And today we don't go directly into the weights.

我们主要不这么做的原因是，模型在自我编排方面取得的许多进步，带来的性能提升远远超过直接更新模型所带来的好处。

And primarily the reason we don't do that is because a lot of the advancements that models are having when it comes to orchestrating themselves yield far more performance gains than, benefits you'd have of updating models directly.

我认为这可能会达到一个极限，我们拭目以待，但我认为这就是为什么你还不直接进入权重的原因之一。

I think that might asymptote, we'll see, but I think that's one of the reasons that you don't go to the weights yet.

我也觉得这是一个非常有趣的研究方向，我们很快会开始探索。所以对我们这个层面来说，也就是应用层面，我们正在做的就是这类研究。

I do think that's a very interesting research direction as well that we'll start pursuing, and so, for us at our level, at the kind of like applied application level, that's the kind of research that we're doing.

但我怀疑，要让模型真正在经济中做实际工作，这是唯一的途径。

But I suspect that in order for the models to actually do real work in the economy, That's the only way you can get there.

我不认为如果你训练一个模型，增加预训练的计算量，增加基于完全可验证奖励的后训练量，就能突然得到一个能可靠报税的模型。

I don't believe that if you were to train a model, you know, and you scale up the amount of pre training computing, you scale up the amount of like post training from like perfectly verifiable rewards, that suddenly will output a model that, will do a tax return, reliably.

它可能会做得很好，但问题不在于它做得好不好，而在于它是否达到了与它一起工作的人所期望的质量、可靠性和可扩展性水平。

It might do one that's really good but the question is not does it do really good, it's that does it do it at the level of quality, reliability, scalability that would be expected from somebody operating with it.

不过，你担不担心你之前提到的那个“苦涩的教训”呢？

Do you worry about the bitter lesson though that, you mentioned earlier in this conversation?

你觉得你们和其他人在框架层面做的所有这些出色工作，最终会不会都被模型本身吞并掉？

Do you do you think, that, all this work that you guys brilliantly and and others are doing at the harness, level are going to be eventually swallowed up by the model?

哦，我觉得肯定会被吞并。

Oh, I assume it will be swallowed up.

是的，我不担心这个。

Yeah, I'm not worried about it.

那绝对是未来。

That is absolutely the future.

毫无疑问——事实上，如果你看看我的帖子，作为线索，我想我说过，这些行为还没有展示给智能体。

No qua- and in fact, if you look at my thread, as a hint to this, I think I said, the behaviors don't get shown to the agents yet.

原因是，如果你真正遵循苦涩的教训，那么将来这种试图进行上下文工程的想法就会消失，你只需要指定，嘿，我想要这些行为，它就会自动实现。

And the reason is because, if you are truly bitter lesson filled, then in the future this whole idea of like trying to context engineer, it will just go away and you'll just specify, hey, I want these behaviors, it'll just work.

所以我绝对认为它会被吞并，毫无疑问。

And so I definitely think it'll get swallowed up, no question.

至于需要多长时间才会被吞并，我不太确定，你知道，我觉得可能不到五年。

I think that how long it'll take to get swallowed up, I don't know exactly, you know, I think it's probably sub five years.

我觉得不会少于两年。

I don't think it's sub two years.

我觉得可能不到五年，所以对我们来说，我们正处于超大规模模式，我们不能等苦涩的教训到来才去准确处理税务申报，所以我认为这将是关键之一，你知道，去实现它。

I think it's probably sub five years and so for us, you know, we're in hyper scale mode like we we we can't we can't wait for the bitter lesson to arrive to like perform tax returns accurately and so that's why I think that'll be one of the keys, you know, to, to doing it.

我确实认为，我不知道，这完全是猜测，但我确实怀疑，也许某种类型的过程奖励对于这类工作最终会变得非常重要，你知道，用于优化实验室甚至随时间使用的计算资源。

I do think that I don't know, this is now total speculation, but I do suspect that maybe, some types of process rewarding for this type of work might end up being pretty important, you know, for optimizing the compute that the labs even use over time.

因为我不知道你是不是想把经济里的每一个职业都像Move37那样处理。

Because I don't know if you want to Move37 like every single profession in the economy.

我们已经有很多经验了，而且我觉得没什么理由要那样做。

Like we have a lot of learnings already and like there's kind of no reason to do that, would be my guess.

那你觉得，自己做强化学习是不是某种护城河，能防止被模型性能的提升给吞掉？

So do you think of, doing your own RL as some kind of moat against being swallowed up by model performance?

我的意思是，你觉得未来的应用型AI公司，是不是都会变成某种强化学习实验室？

I mean as you think about applied AI companies of the future, will they all be RLabs of some sort?

总的来说，如果有人现在想创业，我建议这么想：技术护城河不是真正的护城河。

So generally, and if anyone's currently trying to found a company, I recommend thinking this way, Technical moats are not real moats.

比如，Basis的长期价值里，没有任何一部分是来自我们发现的什么秘密强化学习技巧，别人都没发现的那种。

Like, I I there's no portion of Basis' long term terminal value that stems from some, you know, secret RL trick we found that nobody else found.

所以那不重要。

So that doesn't matter.

重要的是，我们现在显然非常擅长构建长周期智能体，它们能可靠地部署到生产环境中，而且我们会继续保持这个优势。

What matters is that right now we are obviously very good at building long horizon agents, that can be reliable and deployed in production and we'll continue to be the best at that.

这样我们就能赢得市场份额，并深度融入其中。

And that allows us to win market share and get deeply embedded.

所以我们行动非常快，因为我们想做的就是在通用人工智能到来之前，尽可能多地推广开来，不管你怎么称呼它。

And that's why we move really fast because the work we're trying to do is to, go and proliferate maybe before, like, you get to AGI, whatever you want to call that.

所以我认为，未来存在的护城河大多是商业上的护城河。

So I think that most of the moats that will exist will be business moats.

在通用人工智能时代也是如此。

That is true in the AGI era.

我认为在通用人工智能之前的时代也是这样。

I would argue that's also been true in the pre AGI era.

比如，我不觉得Salesforce能写出比我更好的SQL查询。

Like I don't think that, you know, Salesforce can write a better SQL query than I can.

Salesforce的护城河跟他们的技术无关。

Like the moat that Salesforce has is not related to their technology.

对吧？

Right?

这跟他们的业务地位有关，你知道，就是他们拥有的权力、工作流程，还有所有这些随着深入嵌入而来的东西，这才是关键。

It's related to their business position, know, it's the it's the powers, it's the it's the it's the workflows that they own, it's all these so many of these different things that come with being you know embedded in and that's what matters.

不是技术本身。

Not the technology.

技术只是一个暂时的突破口，让我们这种三年前还不存在的公司，现在突然能做所有这些工作。

The technology is a temporary, like, dislodgement that allows someone like us who obviously didn't exist, you know, three and a half years ago to now suddenly be able to do all this kind of work.

不过我也觉得，从长远来看，就像你说的，它不会完全变成一种商品。

I do think though for a long time, maybe to your point, that it's not like it will be a total commodity.

因为，就像你可以有一群天才，但并不意味着所有天才都能同样出色地完成报税工作。

Because, just like you could have a bunch of genius humans, doesn't mean that all the genius humans are equivalent in being able to do a tax return.

因为归根结底，有些东西是你关心的，有些是主观的。

Because at the end of the day, there are certain things that you care about or for that, that are subjective.

而且我确实认为，提升我们工作的专业能力，对我们能在可预见的未来提供高质量的服务，真的很重要。

And I do think that building up the competency of, you know, the work we're trying to do is actually really important for us to be able to deliver really good quality for, I think, foreseeable future.

好了，Mitch，今天聊得非常愉快。

Alright Mitch, it's been absolutely brilliant.

最后，对于正在构建 AI 的人，或者今天在做 Agent 开发的朋友，除了刚才聊到的那些该做和不该做的事、经验教训之外，还有什么想分享的建议吗？

To close, any advice for AI builders, anybody like building agents today in addition to everything that you've talked about like the do's and don'ts and lessons learned and anything that comes to mind?

我觉得最大的教训是，看着世界变化这么快，很容易产生一种无力感。比如看到推特上各种消息刷屏，感觉像 ADHD 一样混乱；中国的实验室每天都在发布新东西。这种时候，人很容易觉得无从下手，难以做出基于第一性原理的决策。

I think maybe the biggest lesson I would say is it's easy to look at the world and how fast things are changing and say like oh things are just you know one thing here one thing here is like you know ADHD on Twitter it's crazy you know the Chinese labs are releasing something every day and feel powerless to, make first principle decisions.

但我认为，如果你把新世界看作是一种底层的范式转移——就像当年转向云计算那样——然后试着去推演：如果智能提升了 X 倍，或者没提升，这意味着什么？这样你就能构建出更连贯的系统，在技术和商业层面做出更明智的战略选择。

Whereas I actually think if you treat the new world as like an underlying paradigm shift, you know, the same way that like a shift to cloud or something is, you know, some underlying paradigm shift and you try to extrapolate, okay, you have these things, what does it mean, you know, if, you know, the intelligence became x better or if it didn't, I think you'll build a lot more coherent systems and make a lot better strategic bets both at like a technical and at a business level.

事情确实在变，但底层范式并没有变。

Things are changing, but it's not like the paradigm is changing.

所以我认为，真正理解这一点对于在这个领域发展至关重要。

And so I think really understanding it is very very key for being able to build in this world.

Mitch，这次交流太棒了。

Mitch, it was absolutely fantastic.

非常感谢你分享这么多内容。

Thank you so much for sharing all of this.

真的很感激。

Really appreciate it.

当然。

Of course.

谢谢你

Thank you

这么热情地邀请我。

so much for having me.

嗨，我是马特·特克，又见面了。

Hi, it's Matt Turk again.

感谢收听本期MAD播客。

Thanks for listening to this episode of the MAD podcast.

如果你喜欢这期节目，我们非常感激你能订阅（如果还没订阅的话），或者在你观看或收听本期节目的平台上留下好评或评论。

If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from.

这真的能帮助我们发展播客，并邀请到优秀的嘉宾。

This really helps us build a podcast and get great guests.

谢谢，我们下期再见。

Thanks and see you on the next episode.