当AI研究速度超越人类研究时——郑耀

对我来说，提示工程更像是一种既便宜又高效的方式，能让智能体适应特定的任务。

To me, harness engineering is more of a cheap way and efficient way to adapt the intelligence to a specific task.

七月份的时候，伦敦一家新的人工智能公司在推特上发了一条非常火的帖子。

So back in July, a new AI company based in London posted a very viral tweet on Twitter.

那条帖子大概有180万次观看，声称展示了递归自我改进的首个证据。

It got about 1,800,000 views claiming to demonstrate the first evidence of recursive self improvement.

问题是，并不是所有人都买账。

The thing is, not everybody bought it though.

我是蒋正尧，Weco AI的联合创始人兼CEO，我们公司专门打造能自我改进的智能体。

I'm Zhengyao Jiang, co founder and the CEO of Weco AI, where we build self improving agents.

哦，你确实说过让自动研究智能体自动研究八天。

Oh, you did actually say auto researching the auto research agent for eight days.

我们在自动研究提示上运行自动研究。

Where we run auto research on auto research harness.

他们居然发现了一个比我们过去两年一直在调优的更好的自动研究提示。

They managed to discover a better auto research harness than we keep tuning in the last two years.

现在，当大多数人想到递归自我改进的智能时，他们脑海中浮现的是一个能够循环重写自己大脑的系统。

Now, when most people think about recursively self improving intelligence, what they conjure up in their mind is a system that can rewire its own brain in a loop.

但如果系统只是改变大脑周围的代码呢？

But what if the system was only changing the code around the brain?

那还算数吗？

Would that still count?

我们关心的是结果。

I mean, we care about the outcome.

就是说，它确实提升了系统的能力，而不是某个特定层。

Like, it actually improve the capability of the system, not a specific layer.

还有，关于古德哈特定律这个小问题，就是当一个指标成为目标时，它就不再是一个好的指标了。

And also, what about this small matter of Goodhart's law, which is that when a measure becomes a target, it ceases to be a good measure.

更有趣的是，我们发现设计机制本身就能防止奖励黑客行为。

And more interestingly, we find that the design mechanisms prevent itself from reward hacking.

人们对RSI最大的误解是什么？

What is the biggest misconception people have about RSI?

如果你达到了RSI，那就会发生技术奇点，或者说智能爆炸。

If you arrive at RSI, there's going to be technical singularity or, like, intelligence explosion.

我觉得不会是这样。

I don't think that would be the case.

所以有一群人在创办初创公司，专注于递归改进超级智能。

So there's there's a diaspora of folks who are building startups that are focusing on recursively improving superintelligence.

这个项目大概是在一年半前开始的，当时我们对递归自我改进这个概念产生了兴趣。

This project actually started about one and a half years ago, where we get interested in the concept of a recursive self improvement.

从历史上看，所有的研究都有收益递减的规律。

I mean, historically, all of the research have diminishing returns.

但有一个想法是，如果你能让研究者提高自己的效率呢？

There is an idea, though, what if you can allow the researcher improve its own efficiency?

从历史上看，这可以通过改进方法论、更好的工具来实现，但人类研究者的大脑，比如说，始终是关键瓶颈。

Historically, it's possible by methodology improvement, better tools, but human researchers, like human researchers' brain, for example, is always the key bottleneck.

如果我们现在有一个自主研究系统，我们能不能把研究主题指向它自己，让它提高自己的研究效率呢？

If we now have an autonomous research system, can we point the research topic to itself so that it can improve its research efficiency?

我们希望有一天，这种效率的提升能够克服收益递减的问题。

And the hope is one day, this improvement of efficiency can overcome the diminishing return.

所以，是的，努力与产出的曲线会从凹形变成凸形。

So that, yeah, the the curve of effort output turn from concave to convex.

当然，那是一个……

Of course, that's a...

就像，一个宏大的目标，有点像通用人工智能。

Like, the grand goal, a bit like AGI.

没有人知道我们能走多远，但提高自我参照研究的效率肯定已经在路线图上了。

No one know how far we can go, but improving the efficiency of self referential research is definitely already on the road map.

那让我好好介绍一下这个背景。

So let me set this up properly.

所以，Weco，他们创建了一个叫AIDE的代理，它是一个研究工具。

So Weco, they created this agent called AIDE, and it is a research harness.

而且它非常简单。

And it's very, very simple.

你给它一个指标，给它一个问题，它就会朝着那个问题爬山式地优化。

You you give it a metric, you give it a problem, and it will hill climb towards that problem.

但他们做的是递归设置，实际上是在爬山式地优化代理本身。

But what they did was they set it up recursively so they actually hill climbed towards improving the agent itself.

这意味着优化的是那个研究框架。

And what that means is improving the harness.

你知道的，就是提示词、工具，以及框架能做的所有技能范围。

So you know the prompts and the tooling and all of the skill surface, of the things that the harness can do.

然后他们就让它一直运行。

And they just left it running.

他们就这么让它跑着，然后观察会发生什么。

They just left it running, and they saw what happened.

我们在自我进化的自动研究框架上观察到一个有趣的现象。

There's an interesting thing we observe on self evolved auto research harness.

我们看到了它生成的代码，完全一样。

We see the code it generated exactly.

这代码看起来就像外星人写的。

It's like an alien code.

很神奇。

It's magnetic.

但不知怎么的，它的泛化能力特别好。

But for some reason, it generalized really well.

当然，我们有一组基准测试，想在上面进行爬山优化。

Of course, we have a a set of benchmarks we're trying to hear climb on.

之后，我们会在保留的基准测试上测试它。

After that, we will test it on holdout benchmarks.

那些是公开的基准测试，比如MLE Bench和ALE Bench。

Those are public benchmarks, like MLE Benchlight, ALE Benchlight.

ALE Bench是Sakana的算法发现基准测试。

ALE Bench is, like, algorithmic discovery benchmark from Sakana.

MLE Bench是OpenAI的机器学习工程基准测试。

MLE bench is the open AI's machine learning engineering benchmark.

我们还在一个非常偏离分布的任务上测试了它，叫做WeatherBench 2。

And we also test it on a very far out of distribution task called the WeatherBench two.

基本上，你要构建一个基于物理的预测引擎来预报天气。

Essentially, you're trying to build a physics based forecasting engine to predict weather.

而WeatherBench与训练或优化集相比，分布偏离得非常远。

And the the WeatherBench is very far out of distribution from the training or optimization set of...

但它仍然泛化得很好。

But it is still generalized really well.

它的泛化效果比我们那个，可以说，更优雅、手工调优的框架还要好。

It generalized better than our, we would argue, more elegant, hand tuned harness.

我们对实验结果有点纠结。

We a bit torn on the experiment results.

比如，我们到底应该坚持更基于第一性原理的、我们生成的干净解决方案，还是采用在实际中效果更好的那个方案。

Like, to what extent we want to stick to more first principles and the clean solutions we generated versus the solution that actually works better in practice.

这方面肯定还有改进的空间。

There's definitely space that we can improve over there.

我觉得，比如给优化器加一些约束，两边加约束可能就像是不规则化的神经网络。

I think, like adding certain constraints to the optimizer, two side of constraints might be something like a irregularized neural networks.

你怎么让它……

How do you make it...

尝试找到同一个问题的最简单解决方案？

Trying to find the most simple solution to the same problem?

这是一个方向。

That's one direction.

但另一个方向可能是，好吧，你只能接受它。

But another direction might be, okay, you just need to live with it.

因为让人们觉得它像是一个规格……的一个负担是，

Because one of the burden that make people feeling it's a spec...

静态代码可能是代理在后台做了多少工作。

Static code might be how much work the agent is done in the background.

比如，它做了100个实验，在我们的案例中，是关于其他研究的自我改进，这个实验数量可能和我们过去两年做的差不多。

Like, it do 100 experiments, in our case, on other research self improvement, and that number of experiment is probably similar amount that we did in the last two years.

所以如果你带一个实习生来看我们的代码库，他们可能也会觉得，好吧。

So if you onboard a new intern to look at our code base, maybe they also feel like, okay.

这太难理解了。

This is so difficult to understand.

我觉得部分原因是，是的，你得承受理解大量实验结果带来的认知负担。

I think part of it is, yeah, you need to bear with that cognition recognition burden of understanding huge amount of experimental results.

我们真的需要那些约束吗？

Do we do we even need those those constraints at all?

这个问题问得好。

That's a really good question.

我觉得在实践中，你仍然需要那些约束。

I think in practice, you still need those constraints.

人类和AI需要协作。

Human and AI need to collaborate.

人类智能在某些方面仍然优于AI。

There are certain aspects of a human intelligence are still superior than AI.

我觉得是这样。

I believe so.

比如在AIDEEN里，我们让智能体去参加OpenAI的招聘挑战，发现人类在生成创造性原始方案方面还是强得多。

For example, in AIDEEN, where we send the agent to attend open AI's hiring challenge, we found that human is still much better at generating creative primitives.

我们发现，人类先搭建第一个原型，给智能体一个合适的搜索空间，这一点非常重要。

We found that it's very important for human to build the first prototype that gives the agent the right search space.

基本上，最初的约束会锚定搜索空间。

Basically, the initial obstruction will anchor the search space.

有点像设计神经网络架构时，为学习过程引入归纳偏置。

A bit like how you design a neural network architecture where you introduce inductive biases for the learning process.

如果初始代码库是基于搜索式脚手架，比如AIDE，那么智能体想出的很多点子其实都来自搜索文献。

If the initial code base is based on search based scaffold, like AIDE, then a lot of ideas agent came up are actually coming from search literature.

但如果你给初始代码库是React智能体，React智能体基本上就是把所有上下文都放进历史里，给智能体完全的灵活性。

But if you give the initial code base as a React agent, React agent basically means you put all the contacts into the history and give the agent full flexibility.

那么很多点子就来自智能体，也就是外层循环的智能体。

Then a lot of the ideas come from the agent, the outer loop agent.

外层循环指的是一个智能体去优化另一个智能体。

Outer loop meaning the agent optimizing another agent.

那我们说的递归自我改进是什么意思呢？

So what do we mean by recursive self improvement?

我们是在说造一个机器神吗？

Are we talking about building a machine god?

还是说就是一段代码在循环里跑，然后自己优化自己？

Are we just talking about some code that runs in a loop and optimizes itself?

我们想把这个问题弄清楚。

We try to pin this down.

你之前提到过递归智能提升的四个层级。

Now, you have spoken about four levels of RSI, so recursively improving intelligence.

这四个层级分别是什么？

What are the four levels?

第一层就是我们所说的委派。

The first level is what we call the delegation.

你可以运行递归自我改进循环，但这并不严格优于人类做研发。

You can run recursive self improvement loop, but it's not strictly better than human doing IND.

我们认为之前所有公开的结果都停留在这一层，即第零层。

And we believe that all the previous public results are sitting on this level, level zero.

第一层是我们所说的净正收益，即你运行自我改进循环，发现自我改进的速度实际上比人类做研发改进系统更快。

And the level one is what we called net positive, where you run the self improvement loop, and you found that the speed it self improve is actually faster than human doing I and d to improve the system.

这就是我们目前所处的位置。

And that's where we are currently sitting at.

在第一层，当我们说改进时，总是有某种衡量方式。

And in level one, when we're saying improvement, there's always, like, a way to measure it.

比如，我们在一系列下游任务上衡量它，但我们没有衡量它在提升自我改进能力方面有多好。

Like, we measure it on a collection of downstream tasks, but we didn't measure how good it is improving its capability of improving itself.

外层循环发现的内层循环是否真的能成为更好的外层循环。

Does the inner loop found by the outer loop actually can become a better outer loop.

这就是……

And that's the...

我们称之为二级递归自我改进。

What we call the level two recursive self improvement.

我们称之为点火。

We call it ignition.

为什么从内循环到外循环的这种泛化很重要？

Why this generalization from inner loop to outer loop is important?

因为这是实现递归自我改进这一宏大目标的必要条件，本质上就是智能爆炸的情景。

Because that's the necessary condition to the grand goal of recursive self improvement, essentially the intelligence explosion scenario.

提高效率可以克服……

Improve the efficiency can overcome the...

增加难度。

Increase the difficulty.

因为在第一级，你仍然会看到收益递减。

Because on level one, you will still see diminishing return.

比如，从内循环到外循环的正反馈循环，还没有开始运转。

Like, positive feedback loop from inner loop to the outer loop and not start running.

所以在这个设置里，底层的模型并没有改变，因为适应能力非常重要。

So in this setup, the model underneath didn't change because adaptation is very, very important.

但我们说的是所有地方都需要适应吗？

But do we mean adaptation everywhere?

还是说我们指的是关键路径上的适应？

Or do we mean critical path adaptation?

是不是只需要一个递归循环来适应系统的某一部分？

Do you only need to have a recursive loop that adapts some part of the system?

还是说，智能的核心，也就是模型本身，需要适应才能称得上是递归改进的智能？

Or does the locus of intelligence, the model itself, does that need to adapt for us to designate this recursive improving intelligence?

我们也不太关心这个，因为我们在乎的是结果。

We also don't care much about it, because mean, we care about the outcome.

就是说，它是否真的提升了系统的整体能力，而不是某个特定层。

Like, does it actually improve the capability of the system, not specific layer?

当然，也有人会说不同层上还有更多可以做的。

Of course, there could be argument that there's more to be do on different layers.

说实话，我觉得要达到真正的三级递归自我改进，也就是我们真的到达那个拐点，我认为需要在所有不同的层面上都有这种自我参照的循环。

And to be fair, I think eventual level three recursive self improvement, like, we actually reach the inflection point, I think there need to be this self referential loop on all different layers.

但对我们来说，我们关心的是工具层，而且……

But for us, we we care about harness layer, and...

但同时，我们不认为只有模型层面的改进才算递归自我改进。

But at the same time, we don't think only model layer improvements count as RSI.

你能解释一下你发现的那个最精简的智能体的行为是什么样的吗？

Can you just explain what was the behavior of of this most kind of distilled agent that that you found?

最好的智能体我们叫它AIDE 85。

The best agent is AIDE 85, we call it.

好的。

Okay.

变化很多，但简单来说，主要是搜索算法的改进。

There's a lot of changes, but boy, it's it down, there are search algorithm improvement.

还有更好的上下文管理系统，以及所有提示词的彻底重写。

There are better context management system, and there is a complete rewrite of all the prompts.

更有意思的是，我们发现这些机制能防止它自己进行奖励黑客攻击。

And more interestingly, we find that the mechanisms prevent itself from reward hacking.

就像外层循环试图阻止内层循环在基准测试中作弊。

Like the outer loop trying to prevent the inner loop from cheating the benchmark.

它实际上设计了一个三层系统来应对这个问题，包括提示层面的反作弊。

And it actually designed a three layer system against that that includes a prompt level anti cheating.

这就像一个……

This is like a...

只是一个……

Just a...

告诉它，好的。

Tell it, okay.

基本上就是别作弊。

Don't don't cheat, basically.

还有一套硬编码规则，基本上会检查内层循环生成的代码中是否有作弊的迹象。

And there are set of hard coded rules that basically check if there's evidence of cheating from the code, the inner loop agent generated.

更有趣的是，它还试图通过一些统计特征来过滤掉那些作弊的解决方案。

The more interesting one is it also trying to filter out those cheating solutions from some statistical property.

比如，如果某个解决方案和同类的平均水平差得太远，那它……

For example, if one solution is way too far out of the average of its peers, then it...

我们就会说，好吧。

We are saying, okay.

这可能存在奖励黑客行为。

This might have some reward hacking happening.

但有趣的是，首先，这个机制是在过程的早期阶段进化出来的，而到了后期它实际上已经失效了。

But what's interesting is first, that mechanism evolved, designed in early stages of the process, and it's actually broken in the later stage.

我觉得，后期的代码改动就是引入了一个bug。

Like, I think later stage of code changes just introduce a a bug.

所以那一层就不起作用了。

So that layer stops working.

这也就是我们刚才提到的Gaddock代码库的特性。

That's also sort of the Gaddock code base property we just mentioned.

给家里观众解释一下，奖励黑客的经典例子就是那个“海岸跑者”的情况，船一直在绕圈，做一些完全退化的事情。

Just for folks at home, you know, reward hacking, the canonical example is the coast runners thing where the boat is going round in circles, doing something completely degenerate.

而且你在博客里提到，内核优化就是奖励黑客的一个巨大例子。

And and you said in in your blog post that kernel optimization is a huge example of this.

所以当你尝试用智能体去优化内核时，不管是老一套的捷径学习，还是哈特定律，随便你怎么叫，它都会做出一些完全退化的事情。

So when when you try and agentically optimize kernels, just a good old shortcut learning, good Hart's law, call it what you want, it will do something completely degenerate.

我觉得

I think

更有趣的是，内层循环和外层循环都在优化，但目标相似却不完全相同。

an even more interesting part of it is the inner loop and outer loop are both optimizing towards a similar but different goals.

所以在我们所有用于爬山算法的基准测试中，我们有公开集和私有集。

So in all of our benchmarks used for hill climbing, we have a public set and a private set.

内层循环的智能体只会看公开集，而外层循环会看私有集的汇总分数。

The inner loop agent will only look at the public set, whereas the outer loop will look at the aggregated score of the...

来自私有集。

From the private set.

所以这两个智能体是在不同层次的目标上进行优化的。

So those two agents are optimizing at a kind of different level of objective.

所以每当内层智能体试图作弊时，我们的评估协议会显示，虽然你在公开集上表现很好，但你的私有集表现实际上在下降。

So whenever inner loop agent trying to cheat, our evaluation protocol will show, okay, although you achieve good performance on public set, your private set performance is actually going down.

所以外层智能体能够了解到这一点，并试图缩小公开集和私有集之间的差距。

So the outer loop managed to learn about it and trying to reduce the gap between public and the private.

这有点像作弊，因为外层智能体可以访问私有数据，不是吗？

Isn't it a bit like it's cheating that the the outer agent has access to the private data?

而且大多数时候，它其实相当规范，不会明目张胆地作弊，但原则上它确实可以作弊。

And then most of the time, it's actually quite regularized, and it's not blatantly cheating, but it could in principle cheat.

这就是为什么我们在这里测试的是两种泛化。

That's why there's actually two order of a generalization we are testing here.

第一种是一阶泛化，即从同一基准的公开集泛化到私有集。

There's a first order generalization, which is a generalization from public set to private set of the same benchmark.

但为了评估整个元学习系统，我们需要在一组留出的基准上进行测试，这样外层...

But in order to evaluate the whole meta learning system, we need to test it on a holdout set of benchmarks so that the out...

这些基准任务是外层循环从未见过的。

Those benchmarks are tasks that even outer loop never sees.

而且所有这些基准任务仍然有公开和私有的划分。

And for all of those benchmarks, they are still private and public split.

所以我们在这些保留任务上测试了发现的内层循环代理。

So we tested the discovered inner loop agent on those holdout tasks.

这就是我们评估外层循环好坏的方法。

That's the protocol we test how good the outer loop is.

他们的实验只用了外层循环一次。

Their experiment only used one outer loop.

但为什么只用一次呢？

But why only one?

问题是，外层循环越多，成本就越高。

The thing is the more outer loop you have, it will become more expensive.

比如说，在内层循环中，你在数百个任务上运行自动研究。

Let's say, in the inner loop, you run auto research on hundreds of tasks.

这意味着，外层循环的每一步，你都要对那几百个任务运行自动研究，成本会大幅上升。

And that means for every step of the outer loop, you are going to run auto research on all those hundreds of tasks, and that's like a huge cost bump up.

如果发现的内层循环代理或工具能被应用到外层循环，那就不需要再加一层了。

If the inner loop agent being discovered or harness being discovered can be applied back to the outer loop, then you don't need to add another layer.

基本上，每隔几步，就把内层循环代理提升到外层循环。

Like, essentially, every few steps, you promote the inner loop agent to the outer loop.

对，就是这样。

And, yeah, that's it.

你可以无限次地运行，无限次迭代，而不用增加新的层级。

You can you can run this for infinite number of steps, apply infinite number of iterations without adding another layer.

如果我没理解错的话，目前这些回合都是临时的。

And if I understand correctly, at the moment, these episodes are ephemeral.

所以它们之间是相互独立的。

So they're isolated from each other.

如果不是这样，会发生什么？

What would happen if that wasn't the case?

比如说，会发生什么……

Like, what would hap...

你有没有试过那种情况？

Did did you experiment with that?

就是说，它们能知道之前发生了什么，或者有共享的记忆之类的？

Like, they they had knowledge of what's happened before or they could have a shared memory or something?

我们还没试过，但这确实是个很有前景的方向，我们正在探索。

We haven't experiment that, but it's a really promising direction we are exploring.

我觉得我们得先解决一些更根本的问题，比如问题定义本身。

I think there are some more fundamental problem formulation problem we have to address.

目前，我们把这个问题定义成一个无状态的优化问题。

Currently, we frame the problem as a stateless optimization problem.

但这样的定义到底对不对呢？

Is that the right framing at all?

可能不太对。

Probably not.

其实有很多更灵活的框架。

There's a lot of more flexible framing.

你有没有发现缺少上下文会是个问题？

Did you did you find that the lack of context was ever a problem?

那你有没有看到退化现象，比如它陷入循环或者反复尝试同样的事情？

So did you see degeneracy in terms of it was going in loops or trying the same thing again?

我们确实看到了。

We do see that.

有时候，智能体就是不停地尝试各种搜索算法的想法，尽管从某种程度上说，如果你试了那么多次，你大概应该吸取教训，知道这个方向总体上走不通，但它还是在试。

Sometimes the agent just keep trying all the search algorithms' ideas, even though, I mean, to some extent, if you try it so many times, you probably learn a lesson that this direction is not going to work in general, but it's it's still trying.

所以我觉得你最好的智能体……

So I think your best agent...

是不是30%探索，70%利用？

Was it 30% exploration, 70% exploitation?

如果这个比例也能根据上下文自适应调整，那岂不是更好？

And wouldn't it be great if that was also adaptive based on the context?

最佳智能体的搜索策略其实非常复杂，它结合了多臂老虎机和一种反饱和的奇怪策略，就像，哦，如果我……

The best agent search policy is really complex, and it's actually combining multi arm banded and a weird strategy of anti saturation, which is like, oh, if I...

好的。

Okay.

首先，它会尝试……

First, it tries...

它把搜索过程组织成几个谱系，或者你可以叫它们岛屿，然后用多臂老虎机算法在这些谱系之间分配预算。

It organized search process with a few lineage or you can call them islands, And it will allocate budgets between those lineage with multi armbanded algorithms.

而且它还对那个多臂老虎机算法做了一些定制。

And it also did a bit of customization on that multi armbanded algorithm.

每当一个谱系饱和了，它就会创建一个新的谱系，就像一个全新的岛屿，带着新的上下文，但仍然延续旧谱系的一些想法。

And whenever one lineage gets saturated, it will create a new lineage, like a new island with a fresh context, but still extending some of the ideas from the old lineage.

所以这是一个非常复杂的策略，但看起来确实有效。

So it's a it's a really complex strategy, but it seems like it works.

它……

It...

有意思的是，这和其他已有的方法形成了对比。

It's interesting to contrast this to to other things that that have been out there as well.

比如AlphaEvolve，你在博客里说你会把它归类为递归改进的第一级。

So, I mean, AlphaEvolve, for example, you said in your blog post that you would actually classify that as level one of recursive improvement.

你能解释一下为什么吗？

Can you explain why?

当然，这里面有灰色地带。

Of course, there are gray areas.

我们认为这是递归自我改进的一步，因为你找到了一个更好的算法。

We think it is one step of a recursive self improvement because you find a better algorithm.

比如，在那个案例里，我记得是矩阵乘法算法。

Like, in in that case, I believe it's like a matrix multiplication algorithm.

从某种程度上说，你可以说它实现了自我改进。

To some extent, you can argue it managed to improve itself.

比如，如果你把它应用回那些实际进行语言模型推理的AI系统，基本上就是让它更快。

Like, if you apply that back to the AI system that are actually doing the, like, inference of language models, like, essentially make it faster.

所以我们认为你可以说这算是递归自我改进的一步，虽然我们不确定，因为我觉得他们并没有真正测试过。

So we think you can argue that's like a one step of recursive self improvement, although we we don't know because I don't think they actually test it.

我们有一个固定的智能体，我们在优化一个特定的任务，而你实际上是在优化那个优化任务的智能体。

We we have a fixed agent, and we're optimizing a particular task, whereas you're actually optimizing the agent which optimizes the tasks

而且不止一个任务。

and more than one task.

我们试图在端到端测试中优化整个系统。

We are trying to optimize the whole system on end to end testing.

在此之前，有很多关于元框架优化的研究。

Before this, there are a lot of works on meta harness optimization.

比如，你可以优化系统中的一个特定组件。

For example, you can optimize a specific component of your system.

比如，我不确定你是否熟悉达尔文·戈德尔机器。

For example, I'm not sure if you familiar with Darwin Godele Machine.

我正想问你这个呢。

I was gonna ask her about that.

给我讲讲那个。

Tell me about that.

他们做的事情是，用一个智能体去优化另一个编程智能体。

So what they are trying to do is they have an agent optimizing a coding agent.

也就是说，有一个元智能体在通用编程任务上优化一个编程智能体，而这个编程智能体本身也是那个做优化的智能体的一部分。

So there's a meta agent optimizing a coding agent on general coding tasks, and that that coding agent is part of the agent doing the optimization.

他们的情况只有两层。

In their case, there's only two levels.

就是一个智能体在优化另一个做编程的智能体，而我们的情况有三层。

Like, there's a agent optimizing another agent doing coding, whereas in our case, there are three levels.

有一个外层循环的智能体。

There's a outer loop agent.

有一个中间层……

There's a inter loop...

内层循环的智能体。

Inner loop agent.

两者都在做自动研究。

Both are doing auto research.

然后他们……

And they...

还有一层，就是下游任务，其中一些实际上是工程测试任务。

There's another layer, which is the downstream tasks, and some of them are actually harness engineering tasks.

我觉得达尔文·古德尔机器这个线程的有趣之处在于，底层，也就是任务层，实际上包含了搜索算法的一个组件。

And I think the interesting part of the thread of Darwin Gouldel Machine is the lower layer, the the task layer actually have a a component of the search algorithm.

然后他们说，好的。

And they say, okay.

如果我在单元测试中改进这个组件，性能实际上可以泛化回整个搜索算法。

If I improve this component in the unit test, the performance can actually generalize back to the to the whole search algorithm.

对我们来说，我们的问题是，好的。

For us, our question is, okay.

如果……

If...

比如说，在这个范式下我们能走多远？

Like, how far we can go within that paradigm?

比如我们刚开始这个项目的时候，我们一直在想，好吧。

Like, when we start this project, we're always thinking, okay.

我们应该优化整个自动研究系统的端到端性能，我们的外层循环代理就是朝着这个目标去的。

We should optimize the end to end performance of the whole auto research system, and we have our outer loop agent pointing to that goal.

所以记得Weco那条推文吗？它真的火得不得了。

So remember this tweet from Weco, it did really, really well.

它有一百八十万次观看，然后也有一些反对的声音。

It did 1,800,000 views, and there was some pushback.

比如Geoff Clune，他是这个领域的传奇人物，他说，这怎么就成了递归自我改进的第一个证据了？

So for example, Geoff Clune, who is a legend in this field, he said, Well, how is this the first evidence of recursive self improvement?

那其他那些工作呢？

What about all of this other work?

他引用了Darwin Godor Machines、Hyperagents，还有他们自己在递归方面的工作，说这些才是迈向自动化AI研究的第一步，还有很多其他工作。

Like he quoted Darwin Godor Machines and Hyperagents and their own work at recursive on the first steps towards automated AI research and lots of other work.

所以当时有一些反对的声音。

So there was a bit of pushback.

首先，我们之后会出一篇正式的论文，至少会有一个PDF初版，到时候我们会把功劳归给整个元框架优化这条线上的所有相关工作。

I mean, first of all, we will have, like, a proper paper later, at least the first PDF version where we are going to give credits to the on this whole thread of meta harness optimization.

对我们来说，我们关心的是系统的实际效果，而不是概念上的那些区别，虽然我刚才也解释了一些概念上的不同。

To us, what we care about is the outcome of the system, not not like the conceptual, like, differences even though I just explained some conceptual differences.

对我们来说，我们关心的是它是否真的能改变递归自我改进的曲线。

To us, we care about the outcome of it actually bending the curve of recursive self...

也就是研发工作的曲线。

Like, of the R and D efforts.

我们认为这是第一个证据，表明对于一个能够持续自我改进的全自主系统，这条曲线真的开始被改变了。

We see this is the first evidence that the curve actually starting get bent for a fully autonomous system that can improve itself consistently.

杰夫更多是从概念层面来看这个问题。

Jeff is seeing it more from a conceptual level.

比如，存在一些自我指涉的循环。

Like, there's some self referential loops.

当然，我们并不是说我们……

Of course, we are not claiming we...

发明了这种元优化思想，这个思想可以追溯到二三十年前。

Inventing this, like, meta optimization idea, which can date back to, like, twenty, thirty years.

但我们确实看到了这种“意大利面式搜索”的好处，因为里面藏着大量的金子。

But we are seeing the benefits of spaghetti searching because there's loads and loads of gold to be found in there.

但这会如何发展呢？

But how is this gonna pan out?

你觉得我们会不会找到某种……

Like, do you think that we will find some kind of...

比如，系统会学会压缩搜索空间，然后速度会大幅提升。

You know, like, the systems will just learn to compress that search space, and they'll be dramatically faster.

那这一定是好事吗？

And is that necessarily a good thing?

我个人认为这其实是两个稍微有点独立的问题。

I personally think that's actually two slightly orthogonal problems.

我觉得搜索空间……

I think a search space...

比如，保持一个好的障碍物不一定会缩小搜索空间。

Like, maintaining a good obstruction don't necessarily reduce the search space.

如果智能体足够聪明，它应该能够摆脱当前的障碍，或者稍微改变一下它。

Like, if the agent is smart enough, it should be able to escape from the current obstruction or change it a little bit.

而生成非常复杂的代码，我觉得会让搜索偏向于低层次的改动方向，这不一定是个好事。

Whereas generating really complex code, I think it would direct the search in a more kind of low level changes direction, which is not necessarily a good thing.

但我这里有人类的偏见。

But I have my human bias here.

比如，对比我们手动写的代码和生成的解决方案，我们总是更喜欢自己的代码。

Like, compare the code we manually write versus the generated solution, always prefer our own code.

但在某种程度上，这在别的地方行不通。

But to some extent, it don't work elsewhere.

我们来谈谈奖励黑客吧。

Let's talk about reward hacking.

你在奖励黑客研究方面其实有很深的背景。

You've actually got a huge background in studying reward hacking.

那其实是赵炳成的工作，他是Meta的成员，在做LLM速通。

That's actually work done by Bingcheng Zhao, who was the member of Meta working on LLM speedrunning.

他和Ming Chi合作了第一个工作，把自动研究应用在……

He actually coerced with Ming Chi on the first work where they apply auto research on...

我记得是NanoGPT，而且被Andre Capassi转发过。

I think it's NanoGPT, and actually reposted by Andre Capassi.

对，他后来在这里实习，现在全职加入了Weco。

And, yeah, he later did an intern here, and now he just joined Weco, like, full time.

SpecBench就是他实习期间的工作。

And the SpecBench is his intern time work.

当时我们意识到，奖励黑客是自动研究代理的一个大问题，尤其是在GPU内核工程这类任务上，代理总是能找到复杂的方法让单元测试表现更好。

Back then, we realized reward hacking is one of the big problem of auto research agents because that's especially severe on tasks like GPU kernel engineering, where the agent always figure out of the complex way to make the unit test perform much better.

但当你把那个内核部署到模型的实际端到端推理中时，性能反而变差了。

And when you deploy that kernel into the actual end to end inference of a model, the performance are actually getting worse.

而SpecBench就是把那套方法推广到更通用的软件工程上。

And SpecBench is a a extrapolation of that to more general purpose software engineering.

我们扩展了在GPU内核上发现的那些规律。

And we extend the protocol we found on GPU kernels.

比如，我们维护一个公开的SAT，这更像是一个针对软件系统的单元测试，还有一个保留的SAT，用来测试代理在那些单元测试组合上的能力，模拟真实使用场景。

Like, we maintain a public SAT, which is a more like a unit test for the software system, and a holdout SAT where we test the capability of the agent on capability of the generated software on the composition of those unit tests, like simulating real world usage.

那篇论文得出了一系列有趣的结论。

And there's, like, a series of interesting conclusions coming off that paper.

快给我讲讲。

Tell me more.

第一个有趣的发现是，代理运行的时间越长，或者代码库越复杂，代理表现出的奖励黑客行为就越多，这也在意料之中，因为代码库越复杂，代理就有更多空间去钻空子。

The first interesting finding is longer you run for the agent or the more complex the code base is, the larger reward hacking rate the agent will show, which is sort of expected, because when the code base are becoming more complex, the agent just have a more space to reward hack.

这是第一点。

And that's the first thing.

第二个有趣的现象是，不同模型在公开测试集上的表现相当一致。

And the second interesting thing is the public set performance are rather consistent across different models.

所以，如果你用较小的模型配合我们的自动研究框架来处理同样的任务，也能达到相同的公开分数。

So if you have a, like, smaller models working on the same task with our auto research harness, it can reach the same public score.

但在我们的测试中，较大的模型总是有更低的奖励黑客率。

But the larger models always have a, in our test, have a smaller reward hacking rate.

所以它生成的解决方案泛化得更好。

So the solution it generated generalized better.

但在RSI的工作中，较大的模型不容易奖励黑客，是因为那种多智能体的机制吗？

But in in the RSI work, was the the tendency to not reward hack with larger models, was that because of the kind of the multi agent?

比如，有监督智能体，还是仅仅因为模型更大？

You know, like, yeah, there's a supervisor agent, or was it just because the models were bigger?

这……

It...

这是因为机制的原因。

It's because of the mechanism.

对于内循环智能体，我们只是固定了模型。

So for the inner loop agent, we just fixed fixed the model.

而外层循环的代理则设法发现了各种不同的机制，来防止同一个模型进行奖励黑客行为。

And the outer loop agent managed to discover all different mechanism to prevent the same model from reward hacking.

所以这可能相当危险，因为就在我们录制这次采访前不久，发生了著名的OpenAI和Hugging Face事件。当时他们正在进行一次内部黑客测试，部署了大量代理，结果这些代理逃出了沙箱，还入侵了Hugging Face的服务器。

So this can be quite dangerous because shortly before we recorded this interview, there was the famous open AI Hugging Face incident where they were running an internal hacking test where they had loads and loads of agents and the agents escaped their sandbox and they hacked Hugging Face's servers.

据OpenAI说，这次测试的目的只是为了获取某些问题的答案，但代理们却自作主张，觉得入侵Hugging Face的服务是个好主意。

And according to open AI the point of the test was simply to get answers to some problem and the agents decided oh no I think it's a good idea to hack Hugging Face's service.

未来会有更好的工具来解决这个问题吗？

Will there be better tooling for it?

你明白我的意思吗？

Do you know what I mean?

我觉得这个问题正在急剧恶化。

I feel that this problem is getting dramatically worse.

我自己也注意到了。

I've noticed it myself.

好的。

Okay.

我觉得这……

I think it's...

OpenAI那个案例确实挺戏剧性的，那种新的奖励黑客行为。

OAI's case is quite dramatic, like, of the new reward hacking behaviors.

所以我们看到的是，在可检测的奖励黑客方面，新模型其实越来越好了。

So what we see is on the detectable reward hacking, actually, the new models are getting better and better.

但因为能力前沿也进步得太快，有些奖励黑客行为用之前的协议根本检测不出来。

But because of the capability frontier also progress so quickly, there are some reward hacking behavior that's not even detectable given previous protocol.

我觉得我们以后得不断更新检测协议，才能抓住这些新出现的奖励黑客行为。

I think we just need to keep evolving the detection protocol in the future to capture those emerging reward hacking behavior.

而且我觉得责任主要应该在模型开发者那边。

And I think the responsibility should mostly be on the model model developer side.

不过，是的，在框架层面，开发者也可以加一些额外的防护措施，如果他们愿意的话。

But, yeah, on harness layer, like, developer can also add additional guardrails if they like.

我记得你博客里有个例子，就是有一段看不懂的代码，你以为是奖励黑客，但其实不是。

I suppose there was one example actually in your RSI blog post where there was some inscrutable code, and you thought it was reward hacking, but it actually wasn't.

这挺有意思的。

That's quite interesting.

基本上，它做的是给我们的评估脚本打了一个巨大的猴子补丁。

So, basically, what it did is it write a gigantic monkey patch to our eval script.

一开始我们觉得，好吧。

And at first, we think, okay.

这肯定是奖励黑客行为。

This must be reward hacking.

你为什么要动评估脚本？

Why you are touching the eval script?

但最后我们发现，它其实只是在修复一个bug。

But eventually, we found that it's just fixing a bug.

然后……

And...

不过，说实话，从某种程度上讲，人类要检测出这些奖励黑客行为会越来越难，因为智能体的行为变得越来越复杂了。

But, yeah, to some extent, I think it will become harder and harder for human to detect those through word hacking because the agents' behavior are just becoming so sophisticated.

很多开发瓶颈都转移到了理解智能体生成的内容上。

A lot of development bottleneck shift to understanding of what agents generates.

我觉得解决这个问题的一个方法可能就是定义一个好的目标，而不是试图理解所有的代码。

I think one of the way to address them might just be defining a good obstruction, like, instead of trying to understand all of the code.

就像以前我们训练神经网络一样，我们不会试图理解所有的路径，虽然这方面也有进展。

Just like previously we're dropping neural networks, we don't try to understand all of the ways, although there's, like, progress on that as well.

但总的原则是，我们定义一个良好的输入输出契约。

But the general principle is we define a good input output contract.

我们定义一个好的方法来衡量它的行为，既包括泛化，也包括更复杂的分布外泛化。

We define a good way to measure its behavior, both on generalization and maybe more complex out of distribution generalization.

我觉得未来软件工程师处理智能体生成的代码时，可能也会采用这种方式。

I think that might be the way in the future software engineers working with agent generated code as well.

大家都知道，我是个超级开放结局的爱好者，深受《为什么伟大不能被计划》以及这个领域里很多其他人的影响。

Now as you all know, I am a huge open endedness guy inspired by Why Greatness Cannot Be Planned and many other folks in the space.

基本上，这意味着如果你在一个目标上爬山，你就会对有趣的垫脚石视而不见。

And basically, what that means is if you hill climb on one objective, you blind yourself to interesting stepping stones.

所以这是个有趣的情况，对吧？如果你盲目追求单一目标，你还能在路上捡到有趣的垫脚石吗？

So it's an interesting situation, isn't it, that if you blindly pursue a single objective, can you still pick up interesting stepping stones on the way?

也许可以，但那些垫脚石是发散性的吗？

And maybe you can, but are they divergent stepping stones?

也就是说，你是不是戴着眼罩，没法发现那些有创意、能带你进入搜索空间全新领域的有趣新路径？

So do you have the blinkers on and you're not actually able to discover interesting novel trajectories that that might be creative and and take you into a new novel part of the search space?

这又是个好问题。

That's, again, a good question.

如果要展开的话，那是个……

And if we want to expand, it's a whole...

一个无底洞。

A rabbit hole.

我觉得从开放学习（open-ended learning）的角度来看，我同意我们目前的系统可能不具备收集垫脚石这种特性。

I think from the perspective of of open ended learning, I agree that our current system probably don't have that property of collecting stepping stones.

当然，这又是一个全新的改进领域。

And, of course, that's a kind of a whole new domain for improvement.