安德烈·卡帕西谈代码代理、自动研究与AI的循环时代

如果我们能更好地进行这种实验，我们在扩展时也会有更明确的方向。

Like, we're gonna have better direction for when we scale as well if we can do this experimentation better.

是的。

Yeah.

我认为最有趣的项目，也是前沿实验室正在做的，是在较小的模型上进行实验。

Would say that, like, the most interesting project and probably what the Frontier Labs are working on is, you you experiment on the smaller models.

你要尽可能让它自主化，把研究人员从循环中移除。

You try to make it as autonomous as possible, remove researchers from the loop.

他们有太多相反的东西了？

They have way too much what is the opposite?

赚到了，是的。

Earned Yeah.

是的。

Yeah.

他们也不知道。

They don't know.

他们真的不应该接触这些。

They shouldn't be touching any of this, really.

所以，你必须重写整个系统，因为现在他们突然可以提出想法，但好吧，他们实际上不应该执行这些想法。

So, And you have to rewrite the whole thing, because right now, suddenly they can contribute ideas, but, okay, they shouldn't actually be enacting those ideas.

有一个想法队列，可能有一个自动化科学家根据所有存档论文和GitHub仓库提出想法，并将这些想法输入队列，研究人员也可以贡献想法，但只有一个队列，有工作人员从中取出任务并进行尝试，任何有效的结果都会被放入功能分支，有时会有人监控功能分支并将其合并到主分支。

There's a queue of ideas, and there's maybe an automated scientist that comes up with ideas based on all the archive papers and GitHub repos, and it funnels ideas in, or researchers can contribute ideas, but it's a single queue, and there's workers that pull items, and they try them out, and whatever works just gets put on the feature branch, and maybe some people monitor the feature branch and merge to the main branch sometimes.

所以，是的，就是把人类从所有流程中移除，尽可能自动化，并实现高令牌每秒吞吐量。

So, yeah, just removing humans from all the processes and automating as much as possible and getting high tokens per second throughputs.

这需要重新思考所有的抽象概念，一切都必须重新调整。

And it does require rethinking of all the abstractions, and everything has to be reshuffled.

所以，是的，我觉得这非常令人兴奋。

So, yeah, I think it's very exciting.

如果我们再进一步递归一步，模型什么时候能写出比你更好的ProgramMD？

If we take one more recursive step here, when is the model gonna write a better ProgramMD than you?

是的。

Yeah.

所以ProgramMD是

So ProgramMD is

我们并没有脱离循环。

We're not not in the loop.

是的。

Yeah.

没错。

Exactly.

是的。

Yeah.

所以ProgramMD是我对AutoResearch应该如何运作的一种粗糙描述。

So ProgramMD is my crappy attempt at describing, like, how the AutoResearch should work.

比如，先做这个，然后做那个，再做那个，接着尝试这些类型的想法。

Like, oh, do this, then do that, and that, and then try these kinds of ideas.

然后这里可能有一些想法，看看架构、优化器等等。

Then here's maybe some ideas, look at architecture, look at optimizer, etcetera.

但我只是用 Markdown 写出了这个东西。

But I just came up with this in markdown.

对吧？

Right?

所以，没错，就是这样。

And so, yeah, exactly.

你或许想要一种 AutoResearch 循环，你可以想象不同的程序。

You want some kind of an AutoResearch loop maybe that looks for you can imagine that different program.

Mds 会给你不同的进展。

Mds would give you different progress.

基本上，每个研究机构都可以用 ProgramMD 来描述。

So, basically, every research organization is described by ProgramMD.

是的。

Yeah.

一个研究机构就是一组 Markdown 文件，描述了所有角色以及整个系统的连接方式。

A research organization is a set of markdown files that describe all the roles and how the whole thing connects.

你可以想象有一个更高效的研究组织。

And you can imagine having a better research organization.

所以，也许他们早上开的站会更少，因为这些站会没什么用。

So, maybe they do fewer stand ups in the morning because they're useless.

而这所有的内容都只是代码。

And this is all just code.

对吧？

Right?

因此，一个组织可以减少站会次数。

And so you can So, one organization can have fewer stand ups.

一个组织可以增加站会次数。

One organization can have more.

一个组织可以非常敢于冒险，另一个组织则可以更保守。

One organization can be very risk taking, one organization can be less.

正如你完全可以想象的那样，你会有多个研究组织，而它们各自都有代码。

As you can definitely imagine that you have multiple research orgs, and then they all have code.

一旦你有了代码，就可以想象去优化它。

And once you have code, then you can imagine tuning the code.

所以，这中间确实存在一个元层次。

So, 100% there's like the meta layer of it.

你看到我发的关于我的竞赛想法的消息了吗？

Did you see my text about my contest idea?

我的竞赛想法是，让人们编写不同的程序MD。

My contest idea was, like, let people write different program MDs.

对吧？

Right?

那么，在相同的硬件上，哪里能获得最大的改进？

And so for same hardware, where do you get most improvement?

哦，我明白了。

Oh, I see.

然后你可以把所有这些数据交给模型，让它写出更好的ProgramMD。

And then you can take all that data and then give it to the model and say, write a better ProgramMD.

是的。

Yes.

是的。

Yes.

对，正是如此。

Yeah, exactly.

我们会得到更好的结果。

We're gonna get something better.

反正我们肯定能行。

Like, there's no way we don't.

对吧？

Right?

你可以百分百地分析这些改进来自哪里，然后想想，我能不能修改ProgramMD，让这类情况更多地发生？

You can 100% look at where the improvements came from, and like, can I change ProgramMD such that more of these kinds of things would be done?

或者那些没奏效的做法。

Or like things that didn't work.

这是元优化。

It's meta optimization.

是的。

Yeah.

你完全可以想象这样做。

You can 100% imagine doing that.

所以，我觉得这是个很棒的主意。

So, I think this is a great idea.

但你知道，我觉得你可以一步一个脚印，先有一个流程，然后第二个流程，再下一个流程，这些都像洋葱的层层结构。

But it's like, you know, I think like you could sort of go one step at a time where you sort of have one process, and then second process, and then the next process, and these are all layers of an onion.

像，LLM 这部分现在已经被视为理所当然了。

Like, the LLM sort of part is now taken for granted.

代理部分现在也被视为理所当然了。

The agent part is now taken for granted.

现在，像爪子这样的实体也被视为理所当然了，你可以有多个这样的实体，可以给它们下达指令，可以优化这些指令，但这就有点太多了，你知道吗？

Now the claw like entities are taken for granted, and now you can have multiple of them, and now you can have instructions to them, and now you can have optimization of the instructions, and it's just like, it's a little too much, you know?

但我的意思是，这正是它走向精神错乱的原因，因为这几乎是无限的，一切都被归结为技能问题。

But, I mean, this is why it gets to the psychosis is that this is, like, infinite, and everything is skill issue.

所以我觉得，是的，这又回到了为什么这一切如此疯狂。

And that's why I feel like, yeah, that's just coming back to this is why it's so insane.

好的。

Okay.

如果我们只是想诊断当下这个时刻，以及现在哪些技能是相关的呢？

Well, if we're we're just trying to, like, diagnose the current moment and what is a relevant skill right now.

你认为这意味着我们应该在不同领域努力实现这样的循环，并且它确实有效吗？

What do you think is the implication that this is the loop we should be trying to achieve in different areas and that it works?

比如，移除人工干预，创建衡量标准，或者赋予代理持续工作的能力，而无需你亲自参与。

Like, you know, remove, create the metric or create the ability for agents to continue working on it without you.

是的。

Yeah.

我们还剩下性能工程吗？

Do we still have performance engineering?

比如，那个，是的。

Like, what Yeah.

我的意思是，对于LM精神病这个说法，我有几个前提要补充。

I mean, so there's a few caveats that I would put on top of the LM psychosis.

第一，这非常适合那些具有易于评估的客观指标的领域。

Number one, this is extremely well suited to anything that has objective metrics that are easy to evaluate.

例如，为模型的各个部分编写更高效的CUDA内核等，简直是完美匹配。

So, for example, like writing kernels for more efficient CUDA code for various parts of a model, etcetera, the perfect fit.

因为你有低效的代码，然后希望得到行为完全相同但快得多的高效代码。

Because you have inefficient code, and then you want efficient code that has the exact same behavior, but it's much faster.

完美匹配。

Perfect fit.

所以，很多领域都非常适合自动研究，但很多领域并不适用。

So, a lot of things are perfect fit for AutoResearch, but many things will not be.

所以，如果你无法评估它，就无法用自动研究来处理它。

And so they It's just if you can't evaluate it, then you can't AutoResearch it.

对吧？

Right?

所以，这是第一个注意事项。

So, that's like caveat number one.

然后第二个注意事项，我想说的是，我们其实是在讨论下一步，也大致看到了下一步的方向，但从根本上说，整个系统仍然有点撑不住了，存在一些裂缝，还没有完全运作起来。

And then maybe caveat number two, I would say, is, you know, we're kinda talking about next steps, and we kinda see what the next steps are, but fundamentally, the whole thing still doesn't it's still kinda like bursting at the seams a little bit, and there's cracks, and it doesn't fully work.

如果你试图走得太快，整个系统实际上反而没有用处，你明白我的意思吗？

And if you kinda try to go too far ahead, the whole thing is actually net not useful, if that makes sense.

因为这些模型仍然不是……它们确实进步了很多，但依然有很多不完善的地方，这是我能想到的最贴切的描述。

Because these models still are not you know, they've improved a lot, but they're still rough around the edges, is maybe the way I would describe it.

我同时感觉像是在和一位极其聪明的博士生对话，这位博士生一生都在做系统编程，却又像个十岁孩子。

I simultaneously feel like I'm talking to an extremely brilliant PhD student who's been like a systems programmer for their entire life, and a 10 year old.

这太奇怪了，因为人类通常不会出现这种组合，你懂的，你拥有所有

And it's so weird, because humans, I feel like they're lot more coupled, like you have, you know, everything

在现实中不会遇到这种组合。

in the wouldn't encounter that combination.

这种不连贯性非常奇怪，而人类身上这种不连贯性要少得多，尽管他们确实也有一些。

This jaggedness is really strange, and humans have a lot less of that kind of jaggedness, although they definitely have some.

但人类的不连贯性要多得多。

But humans have a lot more jaggedness.

抱歉，应该是智能体的不连贯性更严重，有时候我要求某个功能，它却返回完全错误的东西，然后我们就陷入完全错误的循环中。

Sorry, the agents have a lot more jaggedness, where sometimes, like, you know, I ask for functionality, and it like comes back with something that's just like totally wrong, and then we get into loops that are totally wrong.

我仍然经常对智能体感到非常沮丧，因为你能感受到它的强大，但与此同时，它偶尔还是会做出一些毫无意义的事情。

And then I'm just I get so frustrated with the agents all the time still because you feel the power of it, but you also There's still like it does nonsensical things once in a while for me still as well.

当我觉得智能体浪费了大量计算资源去处理一个本应一眼就能识别出的明显问题时，我会非常恼火。

I get very annoyed when I feel like the agent wasted a lot of compute on something it should have recognized was an obvious problem.

是的。

Yeah.

我认为一些更大的问题可能在于，如果让我推测的话，这些模型本质上是通过强化学习训练的。

I think like some of the bigger things is like, maybe what's underneath it, if I could hypothesize, is fundamentally these models are trained via reinforcement learning.

所以，它们实际上正面临我们刚才讨论的同样问题：实验室可以改进那些可验证且有明确奖励的方面。

So, they're actually struggling with the exact same thing we just talked about, which is the labs can improve the models in anything that is verifiable, but that has rewards.

所以，你把程序写对了吗？它能通过单元测试吗？

So, did you write the program correctly, and does it do the unit test checkout?

是或否。

Yes or no.

但它们在某些方面确实有困难，比如，我认为它们很难把握我心中所想或我的真实意图，以及何时该提出澄清性问题。

But some of the things where they're struggling is, like, example, I think they have a tough time with, like, nuance of maybe what I had in mind or what I intended and when to ask clarifying questions.

就像，我 yeah。

Like, what I yeah.

任何感觉不够明确的事情，都会更糟。

It's just anything that feels softer is, like, worse.

所以，你要么在轨道上，成为超级智能系统的一部分，要么不在轨道上，脱离了可验证的领域，然后一切就开始随意飘荡。

And so you're kind of, like, you're either on rails and you're part of the superintelligence circuits, or you're not on rails and you're outside of the verifiable domains, and suddenly everything kind of just, like, meanders.

也许换种说法是，如果你今天去使用最先进的模型，比如ChachiPPT，问它讲个笑话。

Like, maybe another way to put it is if you go to if today, if you go to, like, state of the art model, ChachiPPT, and you ask it, tell me a joke.

你能猜到你会听到什么笑话吗？

Do you know what joke you're gonna get?

这就是笑话。

There's the joke.

笑话？

The joke?

我说不好标准版本是什么，但我感觉ChatGPT有三个笑话。

I do feel I I can't tell you, like, the, you know, standard form of it, but I do feel like ChatGPT has, three jokes.

是的。

Yeah.

那个 apparently 所有OM最爱的笑话是：为什么科学家不信任原子？

The the joke that apparently all the OMs like love the most is why do scientists not trust atoms?

好吧。

Okay.

因为它们编造一切。

Because they make everything up.

好吧。

Okay.

它们编造了一切。

They make everything up.

好吧。

Okay.

这就是那个至今都还

This is Why still

会出现这个笑点吗？

would that emerge?

所以，三、四年前你会听到这个笑话，放到今天你还是会听到同一个笑话。

So, this is the joke you would get three or four years ago, and this is the joke you still get today.

好吧。

Okay.

所以，即便这些模型的能力已经有了飞跃式的提升，而且要是你给它们布置一个带执行属性的任务，它们能连续运作好几个小时，帮你完成各种不可能的任务。

So, even though the models have improved tremendously, and if you give them an agentic task, they will just go for hours and move mountains for you.

是啊。

Mhmm.

然后你要求讲个笑话，它却给你一个五年前的愚蠢、糟糕的笑话。

And then you ask for, like, a joke, it has a stupid joke, a crappy joke from five years ago.

嗯。

Mhmm.

这是因为这超出了强化学习的范围。

And it's because it's outside of the RL.

这超出了强化学习的范畴。

It's outside of the reinforcement learning.

这超出了正在改进的部分。

It's outside of what's being improved.

难道你不觉得，随着模型变得更好，它们也应该讲出更好的笑话或更多样化的笑话吗？这正是问题的扭曲之处。

And it's part of the jaggedness of shouldn't you expect models as they get better to also have better jokes or more diversity of them?

它没有被优化，所以卡住了。

It's not being optimized, and it's stuck.

你认为这是否意味着，我们并没有看到一种泛化现象，即幽默感的智能与代码能力的智能是相互关联的？

Do you think that that implies that we are not seeing, like, generalization in the sense of, like, broader intelligence of joke smartness being attached to code smartness?

是的

Yeah.

我认为有些方面是可以验证的，有些则不行，有些方面则根据实验室输入的数据被任意地优化，而有些则没有。

I think there's some decoupling where some things are verifiable and some things are not, and some things are optimized for arbitrarily by the labs depending on like what data went in, and some things are not.

但我的意思是，一些研究团队的前提是，如果你在代码生成或这些可验证领域更聪明，你就应该在所有方面都更出色。

But I mean, the premise, there's a premise from some research groups that if you are smarter at code generation or in these verifiable fields, you should be better at everything.

而笑话的情况表明，这种现象并没有发生。

And, like, the joke situation suggests that that's not happening in

我认为并不是所有方面都这样。

all of I don't think that's happening.

好的。

Okay.

是的

Yeah.

我认为并不是这样。

I don't think that's happening.

我觉得我们可能确实看到了一点点这种现象，但还不足以让人满意。

I think maybe I we're seeing like a little bit of that, but not like a satisfying amount.

是的。

Yeah.

人类身上也存在黑色的锯齿状现象。

Black jaggedness exists in humans.

你可以在数学上非常非常出色，却依然讲不出一个好笑的笑话。

You can be very, very good at math and still tell a really bad joke.

是的。

Yeah.

没错。

That's true.

是的。

Yeah.

但这就意味着，我们并没有真正获得那种说法所描述的成果——随着模型越来越强大，我们就能免费获得社会各个领域中的大量智能与能力，而实际情况并非如此根本性地发生，仍存在一些盲点，有些方面并未被优化，而这一切都集中在这类神经网络的黑箱模型中。

But it just it still means that we're not getting, like, the story is that we're getting a lot of the intelligence and capabilities in all the domains of society for free as we get better and better models, and it's not exactly fundamentally what's going on, and there's some blind spots, sometimes some things are not being optimized for, and this is all clustered up in these neural net opaque models.

所以，你要么完全遵循训练时设定的路径，一切像光速一样运行，要么就不是。

So, you're either on rails of what it was trained for, and everything is like you're going at speed of light, or you're not.

因此，这就是那种不连贯性。

And so, it's jaggedness.

所以，我认为尽管进步是显而易见的，也确实应该发生，但你不能完全放任它发展，因为目前它还不能完全奏效，或者这只是一个使用技巧的问题，我们还没真正弄清楚该如何使用它。

So, that's why I think even though progression is obvious, which should happen, you can't let it fully go there yet because it doesn't fully work, or it's a skill issue, and we just haven't, like, figured out how to use it.

所以，很难说清楚。

So, you know, it's hard to tell.

我能问一个有点大逆不道的问题吗？如果这种不连贯性持续存在，而所有这些都被整合在一个最单一的界面里——也就是单一模型，这合理吗？

Can I ask kind of a blasphemous question, which is like if this jagginess is persisting and it's all rolled up in a least monolithic interface, right, but, you know, single model, does that make sense?

还是说，它应该被拆分成各自针对不同智能领域进行优化和改进的独立模块？

Or do you should should it be unbundled into things that are can be optimized and improved against different domains of intelligence.

比如，把模型拆分成多个不同领域的专家？

Like, unbundling the models into multiple experts in different areas, etcetera?

更直接一点。

More directly.

对。

Yeah.

而不是像我们现在接触不到核心的混合专家模型那样。

Instead of just MOE that we have no exposure to.

因为作为普通用户，这会让人摸不着头脑，对吧。

Because that can be, like, confusing as a user from the outside Uh-huh.

也就是大家会疑惑，为什么它在这件事上表现这么好，在另一件事上却不行？

Which is like, why is it so good at this but not at this other thing?

对。

Yeah.

就我目前的观察来看，业内的实验室都在试图打造单一的“单一化”模型，想让它在所有不同领域都能具备通用智能，然后把所有能力都塞进模型参数里。

I think currently, my impression is the labs are trying to have a single sort of, like, monoculture of a model that is arbitrarily intelligent in all these different domains, and they just stuff into the parameters.

但我认为我们应该看到智能形态出现更多的分化。

I do we should expect more speciation in the intelligences.

就像动物王国里的大脑形态极其多样，自然界也存在无数不同的生态位，有些动物的视觉皮层或者其他脑区会特别发达；我觉得人工智能也应该出现更多分化，我们根本不需要一个无所不知的“全能先知”模型。

Like, you know, the animal kingdom is extremely diverse in the brains that exist, and there's lots of different niches of nature, and some animals have overdeveloped visual cortex or other kind of parts, and I think we should be able to see more speciation, and you don't need like this oracle that knows everything.

你可以把它进行分化，然后针对特定任务进行部署，我们应该能看到这种现象，因为你能够拥有更小的模型，它们仍保留着核心认知能力——依然具备胜任力，但会进行专业化，从而在你真正关心的特定任务上实现更低的延迟或更高的吞吐量。

You kinda speciate it, and then you put it on a specific task, and we should be seeing some of that because you should be able to have, like, much smaller models that still have the cognitive core, like, they're still competent, but then they specialize, and then and then they can become more efficient in terms of latency or throughput on specific tasks that you really care about.

比如，如果你是一位使用Lean的数学家，我就看到过一些发布版本专门针对这个领域进行了优化。

Like, if you're a mathematician working in lean, I saw, for example, there's a few releases that really, like, target that as a domain.

因此，可能会出现一些类似的情况，这种解耦确实有其合理性。

So there's probably gonna be a few examples like that where the unbundling kind of makes sense.

我有一个问题：可用计算基础设施的限制是否会推动这种趋势，因为效率实际上变得更加重要？

One question I have is whether or constraint on available compute infrastructure drives more of this because efficiency actually matters more.

对吧？

Right?

比如，抛开资金问题不谈，资金确实贯穿了这一切。

Like, financing aside, the financing is involved in all of this.

如果你能无限制地使用计算资源，那自然可以只保留一个单一模型，对吧？

If you have access to full compute for anything you do, like leaving one single model, right?

但如果你真的面临压力，比如无法为每个应用场景都部署一个超大规模模型，你认为这会促使模型出现分化吗？

But if you actually feel pressure where you're like, I can't serve a model of massive size for every use case, do you think that leads to any speciation?

这个问题对你来说有意义吗？

Does that question make sense to you?

这个问题是有道理的。

The question makes sense.

我想，我正在纠结的是，到目前为止我们还没看到太多分化。

And I guess, like, what I'm what I'm what I what I'm struggling with is I don't think we've seen too much speciation just yet.

对吧？

Right?

不对。

No.

我们看到的是一种模型的单一文化。

We're seeing a monoculture of models.

是的。

Yeah.

而且显然存在一种压力，就是要做一个好的代码模型，然后把它重新合并回主干。

And there's like clearly pressure for like make a good code model, put it back in the main, merge again.

是的，是的。

Yeah, yeah.

尽管模型已经面临压力。

Even though there already is pressure on the models.

我想也许我感觉现在资源极度紧缺，这可能会促使更多分化出现。

I guess perhaps I feel like there's a lot of very short supply crunch, and, like, maybe that causes more speciation now.

是的。

Yeah.

从根本上说，这些实验室是在为模型服务，但他们并不真正知道最终用户会问些什么。

Think fundamentally, like, the the the labs are serving a model, and they don't really know what the end user is going to be asking about.

所以这可能是其中一部分原因，因为他们必须应对所有可能被问到的问题。

So maybe that's, like, some part of it because they kind of have to multitask over all the possible things that could be asked.

但我觉得，如果你是面向企业，或者与他们合作解决你关心的具体问题，那么你可能会看到这种分化。

But I think if you're coming to a business and maybe partnering on some specific problems you care about, then maybe you would see that there.

或者会有一些非常高价值的、更小众的应用。

Or there will be some very high value applications that are, like, more niche.

但我认为，目前它们基本上是在追求所有可用资源的总和。

But I think right now, they're kinda like going after the totality of what's available.

我认为，操控大脑的科学还远未完全成熟，部分原因是这样。

I don't think that the science of manipulating the brains is, like, fully developed yet, partly.

你所说的操控是指什么？

What do mean manipulating?

比如，在不丧失原有能力的前提下进行微调。

So, like, so fine tuning without losing capabilities, as an example.

我们还没有真正用于以context窗口以外的方式与智能体互动的基本工具。

And we don't have these primitives for actually, like, working with the intelligences in ways other than just context windows.

context窗口确实有效，而且操作起来非常便宜，我们目前的定制化很大程度上就是靠这种方式实现的。

Context windows kind of just work, and it's very cheap to manipulate, etcetera, and this is how we're getting some of the customization, etcetera.

但我认为，如何更深入地调整模型、实现持续学习、在特定领域进行微调、提升特定能力，或者直接修改权重而非仅限于context窗口，这门科学还处于发展中。

But I think if it was I think it's a bit more of a developing science of how you more deeply adjust the models, how you have continual learning maybe, or how you you fine tune in a certain area, how you get better in a certain area, or, like, how you actually touch the weights, not just the context windows.

因此，我认为直接修改权重比仅调整context窗口要困难得多，因为你实际上是在从根本上改变整个模型，甚至可能改变它的智能本质。

And so it's a lot more tricky, I would say, to touch the weights than just the context windows, because you're actually fundamentally changing the full model and potentially its intelligence.

所以也许这根本就不是一个完全成熟的物种分化规模，这样说你能理解吗？

And so so maybe it's just like not a fully developed size, if that makes sense, of speciation.

而且它还必须足够便宜

And it also has to be like cheap enough

是的

Yeah.

只有这样，这种物种分化才值得去做，对吧。

For that speciation to be worthwhile Yeah.

在这些特定情境下。

In these given contexts.

我可以问一个关于你所描述的AutoResearch扩展到开放领域的问题吗？

Can I ask a question about an extension to AutoResearch that you described in terms of open ground?

你说，好吧，我们有这个东西。

You said, okay, well, you know, we have this thing.

我们需要为它创造更多的协作空间，以便人们能够共同推动研究。

We need more collaboration surface around it essentially for people to contribute to research overall.

你能谈谈这个吗？

Can you talk about that?

是的。

Yeah.

我们之前谈到过，AutoResearch 的运作方式是单线程的，比如我不断尝试各种方法循环进行。

So we talked about AutoResearch has a single thread of, like, I'm gonna try stuff in loop.

嗯。

Mhmm.

但从根本上说，这种并行化才是有趣的核心部分。

But fundamentally, the parallelization of this is, like, the interesting component.

我其实一直在尝试一些想法，但还没有找到像我期望的那样简单直接的方案，目前还没有特别满意的东西，不过我闲暇时，比如不忙于我的爪子项目时，会继续琢磨这个。

And I guess I was trying to, like, play around with a few ideas, but I don't have anything that, like, clicks as simply as, like I don't have something that I'm super happy with just yet, but it's something I'm like working on the side when I'm not working on my claw.

所以，我认为一个问题在于，如果你能使用大量并行节点，那就很容易让多个 AutoResearch 实例通过一个共同系统进行交流之类的。

So, I think like one issue is if you have a bunch of nodes of paralyzation available to you, then it's very easy to just have multiple auto researchers talking through a common system or something like that.

而我更感兴趣的是，如何让互联网上一群不可信的工作者参与进来。

What I was more interested in is how you can have an untrusted pool of workers out there on the Internet.

比如在AutoResearch中，你只是在寻找能够将模型训练到极低验证损失的那部分代码。

So, example, in AutoResearch, you're just trying to find the piece of code that trains a model to a very low validation loss.

如果有人给你一个候选提交，很容易验证这个提交是否正确、是否优秀。

If anyone gives you a candidate commit, it's very easy to verify that that commit is correct is good.

比如，有人可能从互联网上声称这段代码能带来更好的优化和更高的性能。

Like, they someone could claim from the Internet that this piece of code will optimize much better and give you much better performance.

你可以直接检查一下。

You could just check.

非常简单。

Very easy.

但可能这个检查过程需要大量工作。

But probably a lot of work goes into that checking.

但从根本上说，他们可能会撒谎等等。

But fundamentally, they could lie and etcetera.

所以你实际上处理的是类似的情况——我的那些包含不可信工作者池的设计，看起来有点像区块链，因为这里没有区块，而是提交，这些提交可以相互叠加，并包含你在改进过程中对代码所做的更改。

So you're basically dealing with a similar kind of it's almost actually, like, looks a little bit like my my designs that incorporate an untrusted pool of workers actually look a little bit more like a blockchain a little bit, because instead of blocks, you have commits, and these commits can build on each other, and they contain, like, changes to the code as you're improving it.

而工作量证明本质上就是进行大量实验来找到有效的提交，这很困难。

And the proof of work is basically doing tons of experimentation to find the commits that work, And that's hard.

而现在的奖励仅仅是登上排行榜。

And then the reward is just being on the leaderboard right now.

完全没有金钱奖励。

There's no monetary reward whatsoever.

但我不想把这个类比推得太远，但从根本上说，它存在这样一个问题：需要投入大量的搜索工作，但验证一个候选方案是否真正有效却非常简单，因为你只需训练一次。你知道，有人可能尝试了10000个想法，但你只需要检查他们产出的那个东西是否真的有效，因为其他99000个都不行。

But I don't wanna push the analogy too far, but it fundamentally has this issue where a huge amount of search goes into it, but it's very cheap to verify that a candidate solution is indeed good, because you can just train a single You know, someone had to try 10,000 ideas, but you just have to check that the thing that they produced actually works, because the 99,000 of them didn't work.

你知道吗？

You know?

所以，简而言之，你需要设计一个系统，让一群不可信的工作者与一群可信的验证者协作，整个过程是异步的，能够正常运行等等。

And so, basically, long story short, it's like you have to come up with a system where an untrusted pool of workers can collaborate with a trusted pool of workers that do the verification, and the whole thing is kinda like asynchronous and works and and so on.

从安全角度来看，这很安全，因为如果有人给你任意代码并让你运行，那非常可疑且危险。

And it's it's like safe from a security perspective, because if anyone sends you arbitrary code and you're gonna run it, that is very sketchy and dodgy.

但从根本上说，这完全是可能的。

So but fundamentally, it should be totally possible.

所以你熟悉像在家设置和在家折叠这样的项目。

So you're familiar with projects like setting at home and folding at home.

这些问题具有类似的设置。

Of these problems have a similar kind of setup.

在在家折叠项目中，你是在折叠蛋白质，找到低能量构型非常困难。

So, folding at home, you're folding a protein, and it's very hard to find a configuration that is low energy.

但如果有人找到了一个他们评估为低能量的构型，那就完美了。

But if someone finds a configuration that they evaluate to be low energy, that's perfect.

你可以直接使用它。

You can just use it.

你可以轻松验证它。

You can easily verify it.

所以，很多事物都具有这种特性：产生它们代价高昂，但验证它们却非常便宜。

So, a lot of things have this property that's very expensive to come up with, but very cheap to verify.

因此，在所有这些情况下，像在家折叠、在家SAT或在家自动研究这样的项目都会很合适。

And so, in all those cases, things like folding at home, or SATi at home, or AutoResearch at home will be good fits.

所以，简而言之，互联网上的一群智能体可以协作来改进大语言模型，甚至可能超越前沿实验室，谁知道呢？

And so, long story short, a swarm of agents on the Internet could collaborate to improve LLMs, and could potentially even run circles around Frontier Labs, like who knows?

这甚至可能是可行的。

That's even possible.

前沿实验室拥有大量可信的计算资源，但地球上的不可信计算资源规模要大得多。

Frontier Labs have a huge amount of trusted compute, but the Earth is much bigger and has a huge amount of untrusted compute.

但如果你建立相应的机制来应对这个问题，那么外部的群体或许能提出更好的解决方案，人们也会为他们关心的事业贡献计算资源。

But if you put systems in check, systems in place that deal with this, then maybe it is possible that the swarm out there could come up with better solutions, and people contribute cycles to a thing that they care about.

所以，抱歉，最后一点想法是，许多公司或其他机构可能都有他们关心的特定项目，而如果你有计算能力，就可以参与不同的AutoResearch方向。

And so, sorry, so the last thought is lots of companies or whatnot, they could maybe have their own things that they care about, and you, if you have compute capacity, you could contribute to different kind of AutoResearch tracks.

比如，你可能关心某种特定类型的癌症问题。

Like, maybe you care about certain you know, you care about cancer or something like that of a certain type.

你不必非得向机构捐款。

You don't have to just donate money to an institution.

你实际上可以购买计算资源，然后加入该项目的AutoResearch社区。

You actually could purchase compute, and then you could join the AutoResearch forum for that project.

如果一切都重新整合为自动研究，那么计算能力就成为你贡献到池中的核心资源。

If everything is rebundled into AutoResearches, then compute becomes the thing that you're contributing to the pool.

是的。

Yeah.

这非常鼓舞人心，而且也很有趣。

That's very inspiring, and it's also interesting.

我知道这能走多远，但有趣的是，至少有一部分人群——比如硅谷的人，或者在中国零售店排队的人——已经重新意识到，拥有个人计算能力是很有趣的。

Like, I know how far this goes, but it is interesting that at least some audience of people, you know, here in Silicon Valley or lining up at, you know, retail stores in China have discovered that, like, having access to personal compute is interesting again.

是的。

Yeah.

对吧？

Right?

所以也许他们真的有动力为自己的目标做这件事，然后就能为自动研究做出贡献。

So maybe they're really motivated to do that for their claws, and then they can contribute to AutoResearch.

这几乎就像金钱是每个人都关心的东西，但未来真正每个人都关心的会不会是FLOP呢？

It's almost like dollars the thing everyone cares about, but is FLOP the thing that actually everyone cares about in the future?

比如，会不会出现一种反转，你真正关心的东西变了？

Like, is there gonna be, like, a flipping thing almost of, like, what's the thing that you care about?

比如现在，即使你有钱，也很难获得算力。

Like, right now, for example, it's really hard to get compute even if you have money.

是的。

Yeah.

所以，实际上这看起来几乎像是

So, actually, it almost seems like

在某种意义上，算力是占主导地位的。

the flop is, like, dominant in a certain sense.

对。

Yeah.

所以也许情况确实有点像那样。

So so maybe that's kinda like kinda like that.

你掌控了多少FLOP，而不是你掌控了多少财富？

Like, how much how many flops do you control instead of, like, what wealth do you control?

我不认为这是真的，但想想还挺有意思的。

I don't actually think that's true, but it's kind of interesting to think about.

你最近发布的是关于一些就业数据的分析。

The last thing you released was, like, a little bit of jobs data analysis.

是的。

Yeah.

对吗？

Is that right?

即使你只是在可视化一些公开数据，也可能触碰到了敏感点。

Might have touched a nerve even though you're just like visualizing some public data.

你当时对什么感兴趣？

What were you curious about?

是的，我想我是好奇，毕竟每个人都非常关注人工智能对就业市场的影响以及未来会怎样。

Yeah, I guess I was curious to I mean, everyone really is thinking about the impacts of AI on the job market and what it's gonna look like.

所以，我只是想看看，就业市场究竟是什么样子。

So, I was just interested to take a look, like, what does the job market look like?

不同的职位在哪里？

Where are the different roles?

不同职业中有多少人？

And how many people are in different professions?

我只是非常想仔细看看各个案例，试着思考一下：随着这些人工智能的发展，它们会成为人们使用的工具吗？

And I was like really just interested to like look through the individual cases and try to think myself about, like, you know, with these AIs and how they're likely to evolve, like, are these gonna be tools that people are using?

它们会取代这些职业的工具吗？

Are these gonna be displacing tools for these professions?

当前的职业有哪些，它们将如何变化？

And like, what are the current professions, and how are they gonna change?

它们会大幅增长或调整吗？

Are they gonna grow or adjust to a large extent?

或者，可能会出现哪些新职业？

Or like, what could be new professions?

所以，这其实就是一种激发我对这个行业进行深入思考的方式，我想。

So, it's really just like a way to fuel my own chain of thought about the industry, I suppose.

所以，是的，就业数据基本上来自美国劳工统计局。

And so, yeah, the jobs data basically is just a Bureau of Labor Statistics.

他们实际上对每种职业都有未来增长的预测百分比，我认为是未来将近十年。

They actually have a percent outlook for each profession about how much it's expected to grow over the next, I think, almost decade.

是的。

Yeah.

我觉得是十年，但这是2024年发布的。

Think it's a decade, but it was made in 2024.

好的。

Okay.

我们需要大量的医疗工作者。

We need a lot of healthcare workers.

是的。

Yeah.

所以他们已经做出了这些预测，但我其实不太确定他们使用的具体方法是什么。

So so they've already made those projections, and I'm not sure actually a 100% what the methodology was that they they put into their projections.

我觉得我挺想把不同的情况分类来看：大家是不是都觉得，目前主流发展的都是这类数字化的人工智能——就好像是能在数字世界里互动、还能操控大量数字信息的虚拟灵体，而且它们现在还没有实体载体，也不存在物理形态。

I guess I was interested to color things by like, if people think that what's, like, primarily being developed now is this kinda like more digital AI that is kind of like almost like these ghosts or spirit entities that can, like, interact in the digital world and manipulate a lot of, like, digital information, and they currently don't really have a physical embodiment or presence.

而实体领域的发展速度可能会慢一些，因为你是在实实在在地操控物质原子。

And the physical stuff is probably gonna go slightly slower because you're manipulating atoms.

毕竟翻转数据位、复制粘贴数字信息这类操作，本来就比操控实体物质的速度快上百万倍。

So flipping flipping bits and and the ability to copy paste digital information is like makes everything a million times faster than accelerating matter.

你懂吧？

You know?

所以从发展态势来说，我觉得数字领域会迎来巨量的活动，会有大量的重构工作，各类活动都非常活跃，一切都在酝酿升温。

So so energetically, I just think we're gonna see a huge amount of activity in digital space, huge amount of rewriting, huge amount of activity, boiling soup.

而且我认为，和物理世界大概率会发生的变化相比，数字领域的发展速度简直快如光速，这是可以推演出来的趋势。

And I think we're gonna see something that in the digital space goes at the speed of light compared to, I think, what's gonna happen in the physical world to some extent, it would be the extrapolation.

另外我觉得目前还有不少积压的潜力有待释放，过去那些原本需要计算机和人力共同完成的数字信息处理工作，很快就能解除限制、大幅提效。

And so I think, like, there's currently kind of, like, I think, overhang where there can be, like, a lot of unhobbling, almost potentially, of, like, a lot of digital information processing that used to be done by computers and people.

现在人工智能成了处理数字信息的第三类核心力量，这些领域都会迎来大量的重构调整。

And now with AI as, like, a third kind of manipulator of digital information, there's gonna be a lot of refactoring in those in those disciplines.

但现实世界实际上会比数字世界慢上一些时间。

But the physical world is actually gonna be, like, I think, behind that by some amount of time.

所以对我来说，真正有趣的是，这就是为什么我特别强调那些从根本上操作数字信息的专业人士。

And so I think what's really fascinating to me is, so that's why I was highlighting the the professionals that fundamentally manipulate digital information.

这些工作你可以在家完成，等等。

This is work you could do from your home, etcetera.

因为我感觉这些领域会发生变化。

Because I feel like those will be like, things will change.

这并不意味着这类工作会变少或变多，因为这涉及到需求弹性以及其他许多因素，但这些职业会因为这些新工具以及人类超有机体神经系统的升级而发生改变——如果你愿意这么理解的话。

And it doesn't mean that there's gonna be less of those jobs or more of those jobs because that has to do with, like, demand elasticity and many other factors, but things will change in these professions because of these new tools and because of this upgrade to the nervous system of the human superorganism, if you wanna think about it that way.

根据你对数据的观察，你对正在面对就业市场、或正在思考现在该学什么、该发展哪些技能的人，有什么建议或看法吗？

Given the look you had at the data, do you have either any observations or guidance for people facing the job market or thinking about what to study now or what skills to develop?

我的意思是，我很庆幸现在我的工作还需要我亲自去见人。

I mean, we can all go get like, I'm very thankful that I have to, like, meet people for my job right now.

是的。

Yeah.

我

I'm

变得更注重身体了。

getting more physical.

是的。

Yeah.

不过，你能在家工作吗？

Could you do your work from home, though?

我可以。

I could.

我觉得其中有些关系方面的部分很难，但大部分我都能在家做。

I think there are relationship parts of it that are hard, but most of it I could.

是的。

Yeah.

我觉得很难说，因为就业市场极其多样，答案可能因人而异。

I think it's really hard to tell because, again, like, job market is extremely diverse, and I think the answers will probably vary.

但在很大程度上，这些工具非常新、非常强大，所以首先要做的是努力跟上它们的发展。

But to a large extent, like, these tools are extremely new, extremely powerful, and so just being, you know, just trying to keep up with it is, like, the first thing.

而且，是的，因为我觉得很多人会轻视它，或者

And, yeah, because I think a lot of people kinda like dismiss it or

或者他们害怕它。

Or they're afraid of it.

或者他们害怕它，等等，这当然完全可以理解。

Or they're afraid of it, etcetera, which is totally understandable, of course.

是的。

Yeah.

我觉得目前它本质上是一种赋能工具。

I think like it's fundamentally an empowering tool at the moment.

这些工作是由一系列任务组成的，而其中一些任务可以快很多。

And these jobs are bundles of tasks, and some of these tasks can go a lot faster.

所以人们应该把它们主要看作是当前的一种工具。

And so people should think of it as primarily a tool that it is right now.

我认为这方面的长期前景是不确定的。

And I think the long term future of that is uncertain.

是的。

Yeah.

说实话，这真的很难预测。

It's kinda really hard to forecast, to be honest.

而且，我本人其实并没有专业从事这方面的工作。

And, like, I'm not professionally, like, doing that really.

我认为这应该是经济学家们该做的正经工作。

And I think it's a job of, like, economists to do properly.

不过你是个工程师。

You are an engineer, though.

我觉得有趣的一点是，工程类职位的需求仍在持续增长。

And, like, one thing I thought was interesting is that, like, the demand for engineering jobs is continuing to increase.

对。

Yeah.

我无法确定这是否只是暂时的现象。

I can't tell if that's like a temporary phenomenon.

我不确定

I'm not

我目前对这件事还拿不定主意。

sure how I feel about it yet.

你知道吗？

Do you know?

是的。

Yeah.

这几乎就像是需求弹性一样。

That's like the demand elasticity almost.

比如，软件曾经很稀缺。

Like, software was scarce.

对吧？

Right?

所以我们没有更多软件需求的原因只是因为稀缺且价格太贵。

And so the reason we don't have more demand for software is just scarcity and it's too expensive.

太贵了。

Too expensive.

是的。

Yeah.

所以，如果障碍消除，就会出现杰宾悖论，也就是说，软件的需求实际上会增加。

So, if the barrier comes down, then actually you have the Jebin's paradox, which is like, you know, you actually the demand for software actually goes up.

价格更低了，而且数量更多

It's cheaper, and there's more more

更强大。

More powerful.

是的。

Yeah.

关于这一点的经典例子总是自动取款机和银行柜员，因为当时很多人担心自动取款机和计算机基本上会取代柜员。

The the classical example of this always is the ATMs and the bank tellers, because there was a lot of, like, fear that ATMs and computers, basically, would displace tellers.

但实际情况是，随着银行分行数量增加，每个银行分支机构的运营成本变得低得多，因此柜员反而更多了。

But what happened is they made, like, the cost of operation of of a bank branch much cheaper as there were more bank branches, so there were more tellers.

这正是人们常引用的典型例子。

It's like the canonical example people cite.

但本质上，这仅仅是杰宾悖论。

But, basically, it's just Jemin's paradox.

当某样东西变得更便宜时，就会释放出大量需求。

Like, something becomes cheaper, so there's a lot of unlocked demand for it.

所以我认为，对于软件工程，我确实持有一种谨慎乐观的态度，因为在我看来，软件的需求将会非常巨大，而且它已经变得便宜多了。

So I do think that that's probably I do have, like, cautiously optimistic view of this in software engineering, where I do it does seem to me like the demand for software will be extremely large, and it's just become a lot cheaper.

因此，我认为在相当长的一段时间内，很难做出预测，但至少就目前本地情况来看，软件的需求将会增加。

And so I do think that for quite some time, it's very hard to forecast, but it does seem to me like right now, at least locally, there's gonna be more demand for software.

因为软件太棒了。

Because software is amazing.

你知道的，这是数字信息处理。

It's like, you know, digital information processing.

你不必被迫使用那些被提供给你的、存在各种缺陷的任意工具。

You're not forced to use arbitrary tools that are given to you that are imperfect in various ways.

你不必被迫接受现有的东西。

You're not forced to subscribe to what exists.

代码现在是短暂的，可以更改和修改。

Code is now ephemeral, and it can change, and it can be modified.

因此，我认为数字领域将出现大量活动，从某种意义上重新连接一切，这将创造对这类事物的大量需求。

And so, I think there's gonna be a lot of activity in the digital space to rewire everything in a certain sense, and I think it's gonna create a lot of demand for this kind of stuff.

我认为从长远来看，是的，即使有了像AutoResearch这样的工具，比如OpenAI、Anthropic或其他这些实验室，它们雇佣了大约一千多名研究人员。

I think long term, yeah, obviously, even with AutoResearch, like OpenAI or or, you know, Anthropic or these other labs, like, they're employing, what, like a thousand something researchers.

对吧？

Right?

嗯哼。

Mhmm.

这些研究人员本质上就像是高级版的AutoResearch，他们正在主动地自动化自己，而这正是他们所有人努力的目标。

These researchers are basically, like, glorified AutoResearch, like, you know, they're like automating themselves away, like, actively, and this is like the thing they're all trying to do.

是的

Yeah.

我觉得我曾经到处走了一圈

I think like, I went around

一些研究人员也感到精神错乱。

Some of those researchers also feel psychosis.

对吧？

Right?

因为它们确实有效。

Because they can it's working.

对吧？

Right?

所以他们觉得，我的职业生涯也结束了。

And so they're like, it's over for me too.

我确实花了很多时间走访了OpenAI，我当时就想，你们意识到吗？如果我们成功了，我们会取代很多工作。

I did spend a bunch of time going around OpenAI, and I was like, you guys realize if we're successful, like, we're a lot of jobs.

就好像，我们只是在给山姆之类的人搞自动化建设似的。

Like, like, it's just we're just building automation for Sam or something like that.

要么是给我，要么是给董事会。

Like, I or the board.

我也不太确定。

I'm not sure.

但确实有关于这套自动化的说法，是的，就是给董事会或者首席CEO之类的人搞的，之后我们都会丢了工作，顶多只能在边上线做点贡献。

But, like, there's just billing about this automation for, yeah, the board or the CEO or something like that, and we're all out of our job and maybe contributing on the sides.

所以啊，对，从这个角度来看，大家就是在深入琢磨这件事。

And so, yeah, it's kind of like nerding from that perspective.

我能问你诺姆提出的那个问题吗？

Is it okay if I ask you Noam's question?

你本来也可以做那样的事。

You could be doing that.

对吧？

Right?

在一家前沿实验室，与许多同事一起进行自动化研究，需要大量计算资源。

Auto researching with a lot of compute scale and a bunch of colleagues at one of the frontier labs.

为什么不呢？

Like, why not?

我曾经在那里待过一段时间。

Well, I was there for a while.

对吧？

Right?

而且我又回去了。

Like, and I did reenter.

所以在某种程度上，我同意，我认为这个问题可以从很多角度来分析。

So to some extent, I agree, and I think that there are many ways to slice this question.

这个问题本身带有一些敏感性。

It's a very loaded question a little bit.

我想说的是，我非常认可人们在前沿实验室之外所能做出的贡献和影响。

I will say that I feel very good about what people can contribute and their impact outside of the Frontier Labs, obviously.

不仅是在行业内，还包括更广泛的生态系统层面的角色。

Not in the industry, but also in more ecosystem level roles.

所以，例如，你的角色更偏向生态系统层面。

So your role, for example, is more ecosystem level.

我目前的角色也更多是在生态系统层面，我对人们在这些角色中所能产生的影响感到非常乐观。

My role currently is also more on ecosystem level, and I feel very good about the impact that people can have in those kinds of roles.

我认为，反过来，把自己过度与前沿实验室绑定，确实存在一些问题。

I think conversely, are definitely problems in my mind for basically aligning yourself way too much with the Frontier Labs too.

从根本上说，这些前沿实验室提供了巨大的经济激励，而根据你自己的说法，人工智能将深刻而剧烈地改变人类和社会，而你却在直接构建这项技术，从中获益，并通过经济手段与之紧密结盟。

So, fundamentally, I mean, have a huge financial incentive with these frontier labs, and by your own admission, the AIs are going to really change humanity and society in very dramatic ways, and here you are basically building the technology and benefiting from it, and being very allied to it through financial means.

这正是OpenAI最初成立时所面临的核心困境。

Like, this was a conundrum that was in at the heart of, you know, how OpenAI was started in the beginning.

这正是我们当时试图解决的难题。

Like, this was the conundrum that we were trying to solve.

是的。

Mhmm.

所以呢，你知道，这就有点……

And so, you know, that so it's kind of

这个问题至今都还没解决。

It's still not resolved.

这个核心难题……

The conundrum is

还是没有得到彻底的解决。

still not, like, fully resolved.

这是第一点。

So that's number one.

如果你身处这些前沿实验室之一，你就不是一个完全自由的个体，也没办法真正以完全自主、不受约束的姿态参与到这类讨论中。

You you're not a completely free agent, and you can't actually, like, be part of that conversation in a fully autonomous free way, like, if you're inside one of the Frontier Labs.

有些话是你不能说的，反过来，还有些话是机构希望你去说的。

Like, there are certain things that you can't say, and conversely, there are certain things that the organization wants you to say.

而且他们不会强迫你，但你能感受到那种压力，清楚自己应该说什么。

And, you know, they're not gonna twist your arm, but you feel the pressure of, like, what you should be saying.

你知道的。

You know?

对吧。

Right.

因为，很明显。

Because, like, obviously.

否则的话，

Otherwise, it's,

就会变得特别尴尬，人们投来奇怪的目光，问你在干什么？

like, really awkward conversations, strange side eyes, like, what are you doing?

你知道的。

You know?

所以你根本不能真正成为一个独立的个体，我觉得在前沿实验室之外，我反而更贴近人类，因为我几乎不受那些压力影响，对吧？但我也不能想说什么就说什么。

So you can't, like, really be an independent agent, and I feel like a bit more aligned with humanity in a certain sense outside of frontier lab, because I'm not subject to those pressures almost, right, and I can't say whatever I want.

我会说，在前沿实验室里，你当然也能产生影响。

I would say in the frontier labs, you can have impact there, of course, as well.

但有很多研究人员，也许你就是其中之一，也许你的想法非常好，等等。

But there's many researchers, and maybe you're one of them, maybe your ideas are really good, etcetera.

而且可能有很多决策要做，你希望在这些对话发生时，能身处其中。

And maybe there's a lot of decision making to do, and you want to be in a position where you are in the room with those conversations when they come up.

我认为，目前整体的利害关系其实还算低，所以一切都挺不错的。但归根结底，当真正的利害关系变得极高时，如果你是某个组织的雇员，我不确定你到底能在多大程度上影响组织的决策。

I do think that currently the stakes are, like, overall fairly low, and so everything is kinda, like, nice, But ultimately, at the end of the day, like, when the stakes are really high, etcetera, if you're an employee at an organization, I don't actually know how much sway you're going to have on your organization and what it's going to do.

说到底，你其实并不真正掌权。

Like, fundamentally, at the end of the day, it's you're not, like, really in charge.

在房间里，你确实贡献想法，但你并不真正掌控你所隶属的那个实体。

In a room, and you're contributing ideas, but you're not really in charge of that entity that you're a part of.

所以，我认为这些在某种程度上是一些错位的来源。

So, those are like some sources of misalignment, I think, to some extent.

我要说，从某种意义上，我非常认同这种观点：实验室无论好坏，都是封闭的，很多工作都在那里进行，它们处于能力的前沿，正在处理未来即将出现的事物。

I will say that, like, in one way, I do agree a lot with that sentiment, that I do feel like, labs, for better or worse, they're opaque, and a lot of work is there, and they're kind of like at the edge of capability in what's possible, and they're working on what's coming down the line.

我认为，如果你不在这些前沿实验室里，你的判断从根本上就会开始偏离，因为你没有参与那些即将来临的事物。

And I think if you're outside of that frontier lab, your judgment fundamentally will start to drift because you're not part of the, you know, what's coming down the line.

对。

Right.

所以我觉得我的判断也必然会逐渐偏离。

And so I feel like my judgment will inevitably start to drift as well.

我将无法真正理解这些系统底层是如何运行的。

And I won't actually have an understanding of how these systems actually work under the hood.

那是一个不透明的系统。

That's an opaque system.

我无法很好地理解它将如何发展等等。

I won't have a good understanding of how it's going to develop and etcetera.

因此，我认为在这一点上我同意，这也是我感到担忧的地方。

And so I do think that in that sense, I agree and something I'm nervous about.

我认为真正接触正在发生的事情，并置身于前沿实验室中是很有价值的。

I think it's worth basically base being in touch with what's actually happening and actually being in the frontier lab.

如果某些前沿实验室愿意让我去待一段时间，为他们做出出色的工作，然后也许再回来

And if if some of the frontier labs would have me come for, you know, some amount of time and do really good work for them, and then maybe coming

伙计们，他在找工作。

Guys, he's looking for a job.

这太令人兴奋了。

This is super exciting.

是的。

Yeah.

那么我认为这可能是一个不错的安排，因为我总觉得，也许这种方式能真正与正在发生的事情保持联系，同时又不至于完全受这些机构的控制。

Then I think that's maybe a good setup, because I kinda feel like it kind of, you know, maybe that's like one way to actually be connected to what's actually happening, but also not feel like you're necessarily fully controlled by those entities.

所以，说实话，我认为诺姆在AutoAI一定能做出非常出色的工作，但我也觉得他最具影响力的工作很可能发生在OpenAI之外。

So, I think, honestly, my mind, Noam can probably do extremely good work at AutoAI, but also I think his most impactful work could very well be outside of OpenAI.

不。

No.

那是成为独立研究员，加入AutoResearch的号召。

That's a call to be an independent researcher with AutoResearch.

对。

Yeah.

外面有很多事情可做，我认为最终理想的解决方案可能是来回穿梭，是的。

There's many things to do on the outside, and I think, ultimately, I think the ideal solution maybe is like, yeah, going back and forth or yeah.

我认为，从根本上说，这在两个地方都能产生非常惊人的影响。

And I think fundamentally, it can have a really amazing impact in both places.

所以非常复杂。

So very complicated.

我不知道。

I don't know.

比如，这个问题有点复杂，但我的意思是，我加入了前沿实验室，而现在我是在外面。

Like, it's a very loaded question a little bit, but, I mean, I joined the Frontier Lab, and I'm outside.

然后也许在未来，我又想回去，我觉得这就是我对这件事的看法。

And then maybe in the future, I'll want to join again, and I think that's kind of like how I look at it.

关于世界或AI生态系统对前沿的了解程度，有一个相关的问题是：开源与前沿的距离有多近，以及这种状态能持续多久。

One question related to what visibility does the world or the AI ecosystem have into the frontier is like how close open source is to the frontier and how sustainable that is.

我认为这相当令人惊讶，从最初只有少数中国模型和全球模型，到未来短期内人们将继续发布更多模型，这些模型在能力上比业界预期的要接近前沿。

I think it is quite surprising, the entire sequence of events actually from like having a handful of Chinese models and global models, and I think people are going to continue releasing here in the near term, that are closer than much of the industry anticipated from a capability perspective.

我不知道你是否对此感到惊讶，但你是开源的长期贡献者。

I don't know if you're surprised by that, but you're a long term contributor to open source.

那么，你对此有什么预测？

Like, what's your prediction here?

是的。

Yeah.

大致来说，基本上是的。

So roughly speaking, basically, yeah.

闭源模型领先，但人们一直在关注开源模型落后了多少个月。

The closed models are ahead, but, like, people are monitoring the number of months that sort of, like, open source models are behind.

起初，两者之间没有任何差距，后来差距扩大到了十八个月。

And to start with, there's nothing, and then it went to eighteen months.

是的。

Yeah.

它们一直在趋同。

They've been convergence.

对吧？

Right?

所以它们可能落后了，大概是多久呢？

So then maybe they're behind by, like, what is the latest?

现在可能是八个月、六个月、八个月左右吧？

Maybe, like, eight months, six months, eight months kind of thing right now?

是的。

Yeah.

我当然是开源的忠实支持者。

I'm a huge fan of open source, obviously.

比如在操作系统方面，有像Windows和macOS这样的闭源系统。

So for example, in operating systems, have, like, closed sort like, you know, Windows and Mac OS.

这些都是大型软件项目，有点像未来大语言模型会变成的样子，而还有Linux。

These are large software projects, kind of like what LMs are gonna become, and there's Linux.

但Linux非常简单。

But Linux is very easy.

实际上，Linux 是一个极其成功的项目。

Like, actually, Linux is an extremely successful project.

它运行在绝大多数计算机上。

It runs on the vast majority of computers.

我上次查的时候，好像是60%左右的计算机在使用 Linux？

Like, last time I checked, was it like 60% or something, like, run Linux?

这是因为业界需要一个所有人都觉得安全使用的通用开放平台。

And that's because there is a need in the industry to have a common open platform that everyone feels sort of safe using.

我认为，业界一直对这种项目的存在有着需求。

I would say, like, the industry has always felt a demand for that kind of a project to exist.

嗯。

Mhmm.

我认为现在也是如此，这就是为什么企业真正希望这种项目能够存在。

And I think the same is true now, and that's why businesses actually want there's demand for this kind of a thing to exist.

最大的区别在于，一切都变成了资本。

The big difference is that everything is capital.

这里面投入了很多东西

There's a lot

非常昂贵。

of It's very expensive.

投入到这个项目中。

That goes into this.

所以我认为，这就是事情开始出现分歧的地方，让竞争在某种程度上变得更困难了。

So I think that's where things fall apart a little bit, make it a bit harder to to compete in some sense.

我认为当前的模型非常好。

I I do think that the current models are very good.

我觉得另一件非常有趣的事情是，对于绝大多数消费者使用场景来说，即使是开源模型也相当不错。

The other thing that I think is really interesting is that for the vast majority of consumer use cases and things like that, even open source models are actually quite good, I would say.

我认为，如果往前看更多年，很多简单的使用场景都会得到很好的覆盖，甚至可以在本地运行。

And I think, like, if you go forward, like, more years, it does seem to me like a huge amount of simple use cases are gonna be well covered and actually even run locally.

但总会有一些对前沿智能的需求，而这可能实际上占据相当大的一部分市场。

But there's gonna be always like some demand for frontier intelligence, and that can actually be extremely large piece of the pie.

但前沿智能的需求可能就像诺贝尔奖级别的工作，或者像把 Linux 从 C 语言迁移到 Rust 这样的项目。

But it could be that the frontier the need for frontier intelligence is gonna be, like, you know, Nobel Prize kind of work, or, like, let's move Linux from C to Rust.

会有一些规模更大的项目，以这种方式界定范围，而前沿封闭智能很可能主要与这些项目互动，而开源则会逐渐渗透掉大量基础应用场景。

There's gonna be, like, bigger projects, you know, like, scoped in that kind of a way, and there's gonna be maybe more and maybe that's where a lot of the frontier closed intelligences are gonna be interacting with, and open source is kinda, like, gonna eat through a lot of the more basic use cases or something like that.

你知道，今天所谓的前沿技术，可能在今年晚些时候就会变成开源的——我目前从封闭实验室使用的这些前沿模型，未来很可能变成开源的，并承担大量工作。

You know, at some point, what is frontier today is gonna be you know, probably later this year, what's frontier today in terms of what I'm using right now from the closed labs might be open source, and that's gonna be doing a lot of work.

所以我预计这种动态会持续下去。

So I kind of expect that this dynamic will actually basically continue.

我们会拥有像‘神谕’一样的前沿实验室，它们拥有封闭的 AI，而开源模型则会落后几个月，我预计这种局面会持续下去。

Like, we'll have Frontier Labs that have closed AIs that are kind of like these Oracles, and then we'll have open source kind of like behind by some amount of months, and I kind of expect that to to continue.

实际上，我认为这种整体架构相当不错，因为我对完全依赖封闭智能有点犹豫——从结构上看，我认为这存在某种系统性风险。

And I actually think that's like a pretty pretty good setup overall, because I'm a little bit hesitant of having I don't actually think it's like structurally I think there's some systemic risk attached to just having intelligences that are closed, and that's like, that's it.

我认为，过去集中化模式的记录一直很差，而且

And I think that that's a you know, centralization has a very poor track record in my view in the past and has

你是说在政治或经济系统整体上吗？

You mean, like, in political or economic systems in in general?

是的。

Yes.

没错。

Exactly.

我

I

我觉得有很多

think there's, like, a lot

欧洲的。

of European.

对。

Yeah.

很多都是很糟糕的先例。

A lot of it's pretty bad precedent.

所以我希望有一个东西，它可能不在能力的前沿，因为那是新且未探索的等等，但我希望有一个东西是落在后面的，作为一个整个行业都能访问的共同工作空间。

So I want there to be a thing that is maybe not at the edge of capability because it's new and unexplored, etcetera, but I want there to be a thing that's behind and that is kind of like a common working space for intelligences that the entire industry has access to.