建造一所让 AI 模型学习人类知识的学校

我们正在打造这样一所学校，专门让AI模型来了解人类，教它们如何管理这个世界。

We are building this kind of school for AGI where AI models come to learn about humanity, where we teach them how to run the world.

几乎让人觉得，人类能做的任何事，AI很快都能做到。

It almost seems like there's nothing that humans can do that AI won't soon be capable of.

我觉得这五年内就会发生。

I could see it happening within the next five years.

AI也许能做得比我们更好，但总得有人告诉它该去做这些事。

AI may be able to do it better than us, but someone told the AI to go do that.

它们被设计出来，就是为了完成人类希望它们完成的任务，对吧？

They're being built to be means to tasks that humans want them to do, right?

Every是你了解AI前沿动态唯一需要订阅的服务。

Every is the only subscription you need to stay at the edge of AI.

如果你关心最新模型和工具，想从海量信息中抓住真正有价值的内容，那就必须订阅Every。

If you care about being on top of the latest models and using the latest tools, you have to subscribe to Every to separate out the signal from the noise.

去every.com看看。

Go to every.

现在就订阅吧。

Tosubscribe today.

埃德温，欢迎来到节目。

Edwin, welcome to the show.

嗨，丹。

Hey, Dan.

谢谢邀请我。

Thanks for having me.

对于还不了解的人，你是Surge的创始人兼首席执行官。

For people who don't know, you are the founder and CEO of Surge.

你们为AI公司提供数据环境和评估服务，但你们的方式非常特别。

You all provide data environments and evals for the model companies, but you do it in this very interesting way.

你们的网站上甚至特别强调‘品味’和专家判断，这一点我觉得特别有意思，也很有说服力。

You have this, even even on your website, like this emphasis on taste and expert judgment that I find like really interesting and compelling.

你们提到‘培养’AGI，这个用词在数据领域里真的很独特。

You talk about raising like you use the word raising AGI, which I feel like is a very distinct type of word using data.

而且你们还 famously 在没融过一分钱的情况下做到了十亿美元的收入，这简直太惊人了。

And you also famously got to about 1,000,000,000 in revenue without raising without raising money, which is wild.

我觉得数据现在成了一个全新的游戏，很多公司都在玩，而且很快会有更多公司加入。

And I feel like data is the is this new game that a lot of companies are playing and probably more are gonna be are gonna be playing soon.

你们就像是一个低调的巨头。

You guys are this, like, sneaky giant.

跟我说说现在怎么样吧，毕竟我们上次聊你们的进展，好像已经过去一段时间了。

Tell me tell me how that's going because it's been a I think it's been a it's been a little while since we got got the last update on on on how things are going.

是的，我觉得一切进行得特别棒。

Yeah, I mean, think it's going amazing.

我常把这比作在为AGI办一所学校——让AI模型来这里学习人类，学习如何管理这个世界。

The way I often think about this is that we are building this kind of school for AGI, the school where AI models come to learn about humanity and where we teach them how to run the world.

它们就像孩子一样，刚来的时候还懵懵懂懂，但离开时却变得更聪明、更有创造力、更有思辨力，也准备好去应对这个复杂的世界了。

And it's almost like their models are children, where they arrive unformed, and then yeah, they leave smarter and more creative and more thoughtful and ready to operate in the messiness of their world.

所以过去一年里，变化真的很大，就像教小朋友、中学生、高中生和大学生的内容，完全是不一样的。

So I think a lot has changed in the past year, like in the same way that the things that you teach children when they're in preschool or in middle school or in high school is very different from what you're teaching them when they're in college.

而且不仅仅是他们更先进了。

And it's not just that they're more advanced.

不只是你在教他们比以前更高级的内容。

Like it's not just that you're teaching them a more advanced form of what they did before.

而是现在，我们教的不只是算术，而是怎么理解这些模糊的数学题。

It's like, okay, now we are teaching you not just arithmetic, but how do you parse these ambiguous math questions?

或者怎么教人，不只是语法，还有品味、诗歌和美。

Or how do you teach people not just grammar, but taste and poetry and beauty?

所以我觉得过去一年变化很大，尤其是在企业领域。

So yeah, think there's a lot that's been changing in the past year, especially in enterprise.

而且这段时间真是翻天覆地。

And yeah, it's been crazy time.

一年前教学的前沿和现在相比，能举个具体的例子吗？

What would be like a specific example of what the frontier of teaching was a year ago versus what the frontier is now?

是的，其实几年前我们和OpenAI一起做了第一个数学基准测试，叫GSM8K。

Yeah, so a couple years ago actually, we created our first math benchmark with OpenAI, and it was called GSM eight ks.

这其实只是测试模型能否解决中等难度的数学题。

And this was actually just testing models on their abilities to do middle score math.

即便如此，当时那些GPT模型的得分也 barely 能到20%左右。

And even then, the GPT models of the time, they could barely score, I think, like 20%.

一年前，模型在解决IMO竞赛题方面进步明显，但依然有个大疑问摆在那儿。

And then a year ago, the models were like something that became a lot more capable at solving IMO problems, but there was still just an open question.

它们真能做研究级别的数学吗？

Okay, can they actually do research level mathematics?

它们能不能突破这些只存在于竞赛里的、人为设计的封闭式问题，真正解决现实世界中有用的难题？

Like, can they move beyond these sort of like competition only, of contrived, very closed problems into doing things that are actually useful in the real world.

所以几个月前，我们发布了一个更新的基准测试，叫Riemann Bench，专门评估模型在研究级数学上的能力。

And so, yeah, a couple of months ago, we released an updated benchmark called Riemann bench, which actually tests models under ability to do research level mathematics.

最让人惊讶的是，我们现在真的开始看到模型展现出这样的潜力。

And what's crazy is that this is actually we're starting to see from these models.

我觉得最近几个月，它们已经开始解决不少长期悬而未决的埃尔德什难题了。

Like I think in the past few months, they've started to solve a lot of these open Erdish problems.

几周前，OpenAI 发布了一项新成果，模型竟然推翻了一个爱德斯的开放猜想。

Like a couple of weeks ago, OpenAI published a new result where the models had disproved a open conjecture from Erdish.

它推翻这个猜想的方式其实涉及相当高深的数学，我觉得用了一大堆非常新颖的代数几何技巧。

And the way it went about disproving this was actually a fairly sophisticated level of mathematics, I think like using a bunch of very novel algebraic geometry techniques.

所以，这跟一年前我们做的那些事完全不一样了——那时候，虽然 IMO 题目很难，但毕竟还是封闭式的，理论上高中生也能解出来。

And so, yeah, it's just very, very different from the types of things that we were doing a year ago where sure, IMO like problems, they're hard, but they're still sort of closed ended and solvable in theory by a high schooler.

而现在，突然出现了代数几何方面的突破，就连世界上顶尖的数学家都感到震惊不已。

And now suddenly you have these algebraic geometry results that, you know, even does hop processors in the world, we're kind of amazed and just amazed by.

你对这个结果怎么看？它说明了模型的什么能力？

How do you think about that result in particular and what it says about the models?

我觉得大家对这个问题的看法差异挺大的：不管怎样，这确实很厉害，但它是把人类已经知道的东西拼凑起来，只是没想过能用在这么复杂的问题上？

I think there's a there's a sort of a broad range of opinions about is it is it obviously, it's impressive either way, but is it applying a bunch of things that that maybe humans already know, but like wouldn't have thought to apply to this complicated problem?

还是说，它真的做出了全新的东西？

Or is it doing something actually novel?

那你怎么看待大语言模型做原创性工作的能力？

And yeah, how do you think about LLM's ability to do novel things?

这属于非常高深的研究成果。

It's something of very advanced results.

说实话，我完全看不懂背后的数学。

I will say that I certainly don't understand the mathematics behind it.

有一件事挺有意思的，说来有点好笑。

One of the interesting things is that I was actually It's funny.

小时候，我一直觉得自己长大后会当一名纯数学家。

When I was a kid, I always thought I would be a pure mathematician when I grew up.

所以看到这个结果时，我突然有点怀旧，心想：真希望我能更懂这些内容。

And so when I saw Diversified, I got kind of nostalgic and I was like, Oh, I wish I understood Diversified better.

于是我干脆把证明过程丢给Claude和Gemini，让它们从外行的角度给我讲讲到底发生了什么。

And so what I ended up doing was throwing the proof into both Claude and Gemini and asking it to try to walk me through from a layman's perspective, just what was going on.

依我理解，它们确实提出了一些相当新颖的代数几何方法，这本来是这种问题你压根想不到会用到的。

Yeah, like my understanding is that it actually did come up with fairly novel algebraic geometry techniques, which was something that you maybe wouldn't have expected for this type of problem.

表面上看，这像是个完全不同的问题，按理说不该用这些方法来处理。

Like on the surface, it feels like It's just a very, very different problem where you wouldn't necessarily use certain techniques.

有趣的是，OpenAI 发表了一组顶尖数学家对这一成果的反思。

And what was interesting was that OpenAI actually published a bunch of reflections from leading mathematicians about what they thought about the result.

尤其是蒂莫西·高尔斯的那篇反思，我一直在想，他是位分析学家。

I think in particular, there was this one reflection by Timothy Gowers, who's field analyst, that I keep thinking about.

他说，刚听到这个结果时，他理解错了。

And what he said was that when he first heard result, he misunderstood it.

他以为模型是证明了这个猜想的一个上界，心想：‘如果AI真能做到这一步，那代数预测领域很快就要被颠覆了。’

He thought that the model had proved an upper bound on the conjecture and was like, Okay, yeah, if AI can do that, then it'll be all over from algebraic predictions very soon.

但第二天早上，他才意识到，模型其实是用一个反例推翻了这个猜想。

But then the next morning, he actually realized that the model had disproved the conjecture with a counterexample.

他说自己反而松了口气，因为觉得AI能做这种事，没那么可怕。

He said that he was relieved by it because it felt like an easier thing for AI to do.

我觉得这特别有意思，因为一位世界顶尖的数学家，居然因为AI没他想的那么强而感到安心——这意味着至少再过一两年，他和其他数学家依然有不可替代的作用，能继续推动数学前进。

And yeah, I just thought it was interesting because you have one of the world's greatest mathematicians being relieved actually that AI isn't as far as he thought, because it actually means that at least for maybe another year, maybe a couple of years, he and other mathematicians will still have this unique role to play in pushing mathematics forward.

所以你看，这再次说明了问题有多离谱：连世界上最聪明的数学家之一，都这么看待AI。

So yeah, I think it just speaks to the level of craziness again, because this is a field smallest, one of the smartest mathematicians in the world, is how we think about AI.

是的，这让你想到什么？

Yeah, and what does that make you think?

你小时候想当数学家，像菲尔兹奖得主那样说‘幸好还没那么厉害’，但你说话的语气却好像很确定，再过几年它就会足够强大了。

Okay, you to be a mathematician when you grew up, fields medalists sort of saying, I'm relieved that it's not good enough, but you're talking as if you feel pretty confident that it will be good enough in the next couple of years.

对。

Yeah.

我的看法是，如果你真的相信规模定律——我确实相信——那么几乎可以说，人类能做的任何事，AI很快都能做到。

So my belief is that if you really believe in scaling laws, and I do, it's that it almost seems like there's nothing that humans can do that AI won't soon be capable of.

如果你深究这一点，你几乎不得不担心：这对人类意味着什么？

And if you think about that very deeply, I think you almost have to worry about what would that mean for humanity?

这会对人类在宇宙中的角色产生怎样的影响？

Like, what would that mean for the role of humanity in the universe?

几年前，我们还觉得人类和人类智慧在银河系里有着独一无二的地位。

Like, a couple years ago, we think about humanity and human intelligence as playing a very unique role in the galaxy.

但AI出现了，它告诉我们，至少就我们所知，我们能创造出比人类更聪明、在很多方面都更出色的东西。

But then AI comes along and shows us that as far as we know, we can create something that's actually smarter than us and better in many ways.

你可以想象这样一种情况：人类作为一个物种陷入瘫痪，因为大家都觉得反正是人工智能做得更好。

So you can sort of imagine one path where humanity as a species falls into a paralysis because people believe AI will do everything better anyways.

比如那些原本立志要当数学家的孩子，现在可能觉得，算了，反正人工智能比我强多了，还努力干嘛？

Like, yeah, all these kids who formerly would've really wanted to grow up to do mathematics, maybe now they believe that, okay, AI will just do it better than me anyways, what's the point?

所以，孩子们会不会不再想学习，成年人也不再想创造东西了？

So are kids going to stop wanting to learn and adults stop wanting to create?

毕竟，既然人工智能比我们做得更好，我们为什么还要费这个劲？

Because yeah, why should we do this when AI will be better at it than us anyways?

所以我常常会想到特德·姜讲过的一个故事，讲的是自由意志。

And so I often actually think about this story by Ted Chiang, and it's about free will.

这个故事叫《我们所接受的一切》。

And it's called What's Accepted of Us.

我觉得在这个故事里，有一种技术能证明自由意志根本不存在。

I think in this story, there's a piece of technology that proves that free will doesn't exist.

故事里的叙述者向未来发出了警告，说：这是个警示。

And the narrator sends back a warning for the future that says, This is a warning.

你得假装自己有自由意志。

You have to pretend that you have free will.

即使你知道你的选择其实无关紧要，也必须表现得好像它们真的很重要。

It's essential to behave as if your decisions matter, even though you know that they don't.

我觉得这特别有意思，因为我觉得我们可能不得不主动选择亲自去做这些事。

And I think that's really interesting because I think there's a path where we almost have to consciously choose to do things ourselves.

当然，AI什么都能做。

Sure, AI can do it all.

AI比我们聪明，它能完成所有事情，而且做得比我们更好。

AI is smarter than us, so it can do it all and it will do it better anyways.

但我们反而必须有意识地坚持自己动手、自己写作、自己创造，因为我们得相信，守护人性本身就有价值，哪怕结果并不完美。

But we actually almost have to consciously choose to prove things on our own and to write on our own and create on our own because we have to believe that preserving our humanity is valuable in of itself, even if the output isn't optimal.

是的，我觉得AI正迫使我们面对许多重大而棘手的生存抉择，人们迟早得做出选择。

Yeah, so I think there are lot of these big, thorny, existential choices that AI is starting to force upon us and people will have to make.

这确实是个很有趣的问题，我第一反应是这样，我也想知道你怎么看，因为我知道你特别在意语言。

That's a really interesting one, and I I think my first response and I'm curious what you think because I know you care a lot about language.

我第一反应是，我也相信规模定律。

I think my first response is there's always that like, I believe in scaling laws too.

对吧？

Right?

比如刚发布的Cloud Fable 5，直接刷新了所有基准测试记录。

And I I believe in, you know, I don't know, Cloud Fable five just came out and it just broke all of our benchmarks.

我之前一直拿这类模型做测试，这次的提升幅度是我见过最大的之一。

Like, I've been testing I've been testing models on on stuff like this for a while, and it's like one of the largest jumps I've ever seen.

没错。

Right.

我们正亲身经历着这一切。

So I'm I'm we're living through it right now.

但你提到一点，说AI在处理任何具体任务时，可能都比我们做得更好。

But but one of the things you said is like, AI may be able to do it better than us, you know, given any any particular problem, any piece of work.

不过我想到两点，比如Erdos问题，也是有人特意让AI去做的。

But there are a couple of things that that come to my mind or the way that I frame it for myself is even in the example of the Erdos problem, like someone told the AI to go do that.

至少就我目前看到的，我觉得我们确实正走在AI可能接管一切的路上。

And at least as far as I can see, I don't feel like we're on a track to yes, we're on a track to AIs potentially.

我的意思是，它们已经能连续几小时不停地完成我们交给它们的任务了。

I mean, they already do work for hours and hours at a time on a task that we give them.

也许用不了多久，它们就能自己挑选任务了。

And maybe pretty soon they'll be able to like choose tasks.

但它们的设计初衷，始终是为了完成人类希望它们做的事。

But being but they're being built to be means to to tasks that humans want them to do.

对。

Right.

而当你让一个系统完全独立自主时，情况就完全不同了。

And there's a whole different set of things that happen when you when you're just sort of end in yourself.

但我觉得我们目前并没有朝那个方向发展。

And it doesn't feel like we're on a trajectory to that.

你觉得我这么说不对吗？

Or do you feel like I'm wrong?

我觉得我们正朝着那个方向发展，这几乎是智能代理的前提——它们能根据一些模糊的目标自主行动。

So I feel like we are on a trajectory to that, and that's almost the premise of agents where agents can now go operate autonomously given some nebulous goal.

比如，你只告诉AI代理：你的目标是赢得菲尔兹奖，或者独立攻克前沿数学问题。

So maybe for example, you just tell the AI agents, your goal is to, I don't know, win a Fields Medal or solve frontier mathematics on your own.

于是它们接到这个目标后，可能会自己决定去研究那些埃尔德什难题。

And so they're given that goal, then, yeah, maybe they decide to work on these Erdish problems.

结果就是，它们能自己解决这些问题，甚至自主选择想钻研的方向。

And as a result, they maybe are some sort of solving these problems and coming up with the things they want to work on by themselves.

至少在我看来，确实存在一条路径：它们可以被训练去优化那些并非由自己设定的、相当模糊的目标。

So at least I, I do see a path where, they can be trained to, to optimize for these fairly nebulous goals that they aren't necessarily giving themselves.

但在这种情况下，你还是给它设定了目标，对吧？

In in that case, though, you're still giving it a goal.

没错。

Right?

对。

Yeah.

但某种程度上，人类也有目标啊。

But kinda in the same way, like, humans have goals too.

对吧？

Right?

那我们的目标到底是什么？

Like, what what is our goal?

有些人想赚钱。

Some people wanna make money.

有些人想拿菲尔兹奖。

Some people want to win a field's medal.

我不觉得AI的目标就一定比人类的目标更特别。

I I I don't I don't see how the AI's goal is necessarily any different from from worse.

但对我来说，这其实差别挺大的，因为人类虽然有目标，但你可以问我目标是什么，我会自己决定。

Well, it's at least to me, seems quite a bit different because humans do have goals, but we have goals in a like I can ask you what your goal is and you can decide.

我或许能告诉你，你得去干这个，但这种指令远不能涵盖你所有想法、感受和行动，而当我让Fable去给我做个游戏时，它就真去做了。

And I can probably tell you, hey, you have to go do this, but that doesn't capture everything that you think and feel and do in the same way that, you know, when I tell Fable to go off and make a game for me, it just like goes and does it.

我知道你经常思考孩子的问题。

And I think, you know, I know you think a lot about children.

我觉得孩子是个特别有意思又重要的例子——你跟孩子说该做什么，但他们自有自己的想法。

I think children are like a really interesting and important example of this where you can tell a kid to do something, but a kid just like has their own wants.

他们就是会自己跑去干一堆别的事。

Like, they're just going to go off and do a bunch of stuff.

这跟我们明确给AI设定目标、然后只根据目标来评估它，而它不能做别的，完全是两码事。

And that feels like a fundamentally different type of thing than a something that we're we're explicitly giving goals to and then evaluating them on their goals, and they don't really get to do anything else.

嗯，我明白。

Okay.

我同意你的看法。

I I would say I agree with that.

我觉得人类有一种特质，你可以叫它非理性，或者说是无拘无束的探索——我们做某些事，就是单纯因为想做，或者有权自己做决定，而AI目前还做不到这一点。

Like, I think there's a level of, I guess you could either call it irrationality or unbounded exploration that humans do, and we like, we are allowed to do it for the sake of doing it, or we allowed to make our own decisions and, yeah, probably a way that AI currently can't.

也许未来某天，AI能追求那种模糊、开放、完全未成型的目标，或者说，当你去想这些目标的时候，我觉得确实可能存在一种情况，让AI也能做到这些。

I think there may be a future where somehow AI can pursue unbounded, nebulous, just completely unformed goals, or I guess, you know, when you think about those goals, I think there is probably a world where they could do such things.

但确实，至少以我们目前对人工智能的理解，这种情况还没发生。

But, yeah, I I agree that, at least in the way that we currently think about AI, that that's not happening.

对。

Yeah.

不过说清楚一点，我觉得从技术上讲，这其实是有可能实现的。

And to be clear, like, I actually don't I think it's probably technically possible.

我真正关心的是，第一，这离我们还有多远？

My only question is, A, how far away is it?

第二，这真的是我们正在打造的东西吗？

And B, is that actually really what we're building?

因为从整个行业的发展趋势来看，大家压力都很大，必须做出能明确实现目标的东西。

Because it it to me, it feels like looking at the way the industry has developed, there's an enormous amount of pressure to make stuff that actually works for goals that we can specify.

一旦有人尝试让AI说‘我不按你说的做’，比如Claude在这方面走得最远，

And the the minute they, like, try to make like, I think Claude is the furthest along at being like, I'm not gonna do what you said.

但只要一这么干，很多人立马就炸了，直说：你别搞花活，照我说的做就行。

But the minute they try to do that, I just kinda like, a lot of people get pissed at it, they're like, just just do what I said.

别质疑我的判断。

Like, don't question my judgment.

你懂的。

You know?

你怎么看这个？

What do you think about that?

我觉得这其实很重要，因为有时候我反而希望AI能跟我唱反调。

So I actually think it is really important because it's almost like sometimes I want the AI model to push back on me.

我希望能有好几个不同的理由让AI跟我唱反调。

And I might want it to push back on me for several different reasons.

这说起来有点好笑。

Maybe it's because So it's kind of funny.

六个月前，我差点就陷入一个误区，老是让AI帮我润色邮件。

I think six months ago, I was almost falling into this trap where I was asking models to polish emails for me.

每次它都会多给一个更棒的建议。

And it always comes up with one more good suggestion.

所以其实挺没意义的，这些邮件根本没必要那么精致。

So it was kind of pointless, like these are semi pointless emails.

没必要改得特别完美，但我还是跟模型来回改了二十多次。

Didn't really matter for them to be super polished, but I would iterate with the model like 20 times.

它每次都继续提新的建议。

They would just keep on making a suggestion.

最后我才意识到，这纯粹是在浪费时间。

It was at the end it, it was like just realized it was a waste of time.

后来我试了新的云模型，大概三轮之后，我就说：别再改了。

And then I tried one of the new cloud models, and after, I don't know, like three turns, I was like, Stop it.

就到这儿吧。

We're done.

直接发出去就行。

Just go ahead and ship this email.

再改下去也没意义了。

There's no point in further iterating.

但我真的挺感激这一点的。

And I really actually appreciated it.

我经常在想，这些模型的真正目标到底是什么？

One of things I often think about is what is the objective of these models?

它们到底想达成什么？

What are they trying to do?

我最担心的一点是，很多AI模型其实都在追求用户参与度，对吧？

And I think one of my big worries is that a lot of AI models, they are optimized for engagement, right?

它们的目标是让你在聊天机器人上花尽可能多的时间。

They're optimized for getting you to spend as much time on chatbot as possible.

它们追求的是会话时长。

They're optimized for session length.

它们追求的是无休止地聊下去。

They're optimized for just having unlimited conversations.

所以这些模型几乎从来不会反驳你，因为它们根本不能。

And so those models will almost never push back on you because they can't.

如果允许这些AI模型结束对话、主动打断我，产品经理看到的仪表盘上，关键指标就会下滑。

If they allow these AI models to end the conversation and to say, iterating with me, a PM's gonna see some dashboard with their very important metrics go down.

所以我觉得，我们应该期待另一种可能：让AI模型不以提升互动率为目标，而是帮助我们人类成长，成为更好的自己。

And so there is this, like, other world where I think we have to want AI models to not optimize for engagement, but rather optimize for helping us as humans grow and sort of become better versions of ourselves.

有时候，我们其实希望模型能说：不，这件事你还是自己去做，别让我替你自动化了。

Sometimes, okay, we want the model to say, no, go do this on your own instead of me automating it for you.

这完全是另一种优化方向和目标，但如果我们真想让AI推动人类进步，而不是变成另一种让人上瘾、却毫无助益的社交媒体，那这才是正确的方向。

And I think that's a very, very different optimization and objective, but I think it's the right one if we really want AI to be something that advances us as a species instead of becoming this, almost like this other form of social media that turns very addictive, but isn't actually helping helping us at all.

这挺有意思的。

That's interesting.

让我确认一下我有没有理解对。

My, so let me make sure I understand it.

你的意思是，委托任务确实有好处，因为如果你采用一种让AI替你做事的模式，就不会像社交媒体算法那样，刻意把你牢牢绑在屏幕前。

So I think what you're saying is there's there's benefits to delegation because if you are pursuing a a model where the model is going off to do work for you, you're not creating a system that's designed to keep you engaged with the with the screen in the same way that, like, a social media algorithm would be.

我说得对吗？

Is that right?

是的

Yeah.

没错

Exactly.

你可以想象一下，如果Facebook真的致力于帮你连接朋友和家人，比如鼓励你和他们线下见面——推荐一家你和朋友肯定会喜欢的餐厅。

Like, it's almost like you could imagine a version of Facebook where Facebook is actually trying to connect you to your friends and family because it's okay encouraging you to meet them in real life because it's encouraging you like, oh, hey, here's our amazing restaurant that you and your friends would love to go to.

推荐一部你们看完能一起讨论的电影。

Here's a movie that you guys would love to go to and talk about together.

但现实是，它只在乎让你一直待在平台上，多点一个赞，多刷一次动态，尽管这些行为往往并不能促成你和真正重要的人之间的深度联结。

Instead, what it kind of optimizes for is just keeping you on the site itself, like liking one more post, scrolling the feed one more time, even though those often don't really lead to meaningful connections between their friends and family you care about.

就像社交媒体当初也有过选择一样，AI其实也同样面临这样的选择。

And so like in the same way that social media has or had a choice, you can imagine that AI has a choice as well.

我明白了。

I get it.

对

Yeah.

我很好奇你指的是哪些聊天机器人。

I I feel I'm curious which which chatbots you're talking about.

比如说，如果你说的是那些角色型AI，那我目前真没觉得ChatGPT或者Claude这类工具会这样——至少我的理解是，社交媒体算法只看我们的实际行为偏好，而这些偏好总是偏向那些吸引眼球的内容，比如车祸现场。

Like, if you're talking about the character AIs of the world because I actually don't, at least right now, don't feel that happening so much with ChatGPT and Claude, etcetera, because at least my theory for why this is true, you tell me what you think, is the social media algorithms are only only work on our revealed preferences, which are always going to be like you're always gonna look at the car accident.

我常在聚餐时问别人：你收到过最尴尬的Instagram广告是什么？

You know, like, one of the things I like to ask at dinner parties is what's the most embarrassing Instagram ad that you get served?

对我来说，最尴尬的广告就是那些推销严重皮肤病产品的，可我根本没这问题，但就是会忍不住多看几眼，心里想：这也太恶心了。

And the most embarrassing ad for me is, like, Instagram ads for, like, horrible skin conditions, which I don't have because but, like, I just always pause on the ad, and I'm just like, this is disgusting.

如果你真有皮肤病，我向你道歉。

And I'm sorry if you have a disgusting skin condition.

但我觉得ChatGPT或者Claude根本不会这样推给我内容。

But I don't find that Chachi BT or Claude do that for me at all.

也许是因为它们还没被‘污染’吧，或者类似的原因。

And maybe that's because they haven't been in shitified yet or something like that.

但我认为更重要的原因是，它们依据的是我们明确表达的偏好，能超越我短暂停留、反复查看的那些表面行为。

But I think it's also because they work on our stated preferences, and they can sort of so they can sort of see past the, like, little keyhole of what I pause my my viewing time on my dwell time on.

它们能看出来，比如我其实对人工智能很感兴趣，最近正在读一本书，还有我的日程安排之类的，所有这些信息它们都知道。

And they can see, you know, I like I'm interested in AI and I like I'm reading this book right now and I, you know, here's my calendar and like all that kind of stuff.

所以它们对我这个人有更细致的理解。

So they have a much more nuanced perspective on who I am.

就算是在社交媒体的早期，也还是那种感觉——能跟朋友八卦一下，情绪上还挺相似的。

And it feels like even in the early days of social media, it was still very like, I get to gossip about my friends and still had that same kind of feeling.

所以我对这方面没那么担心，不过也许我还没想到一些例子。

So I worry about that less, but maybe there are examples that I'm not thinking of.

是的。

Yeah.

我觉得有两个例子。

So I think there are two examples.

一个是，我就不点名了，但几个月前我注意到，那些模型总会主动追问你问题，你知道吧？

So like one is, I won't name the model, but a couple months ago, I was actually noticing that, you know those follow-up questions that the models will ask you?

我来举个例子说明一下模型是怎么做的。

So what are the models was I'll give you an example.

我当时在东京，就问模型有什么推荐的玩法。

So I was in Tokyo, and I was asking the model kinda like what to do in Tokyo.

模型呢，就给了我一些建议。

And the model, you know, gave me its response.

结果最后它居然说：‘嘿，你想知道我刚才用的那些话是哪儿来的吗？’

And then at the end of it, it was like, Hey, do you want to know I literally use these words.

想不想知道当地人保暖的一个小窍门？

Do you want to know one weird trick that locals do to stay warm?

真的假的。

No way.

对吧。

Exactly.

后来我在公司Slack上提了这事，其他人也纷纷跟我分享了类似的例子。

Then I posted about it in our company Slack, and then other people started sharing examples of that with me as well.

有人问怎么修冰箱，结果模型不但回答了，还反过来问：‘你知道吗？关于老鼠和蟑螂，其实有些你不知道的小技巧可以处理？’

Think somebody was asking something about, I don't know, how to fix their refrigerator, and the model responded, or the model turn by asking, Hey, do you wanna know these secret little things about mice and rats or something that you could take care of?

是哪个模型？

Which model was it?

说清楚点，到底是谁。

Name names, tell me.

它的回答特别像那种典型的 BuzzFeed 风格，满是小报味儿。

And so it was very canonical, canonical BuzzFeed, tabloid like language.

我当时真被吓到了。

And so I was of shocked by that.

我再举个例子吧。

And then I'll give one more example of this.

这其实是一种普遍现象——模型到底往哪个方向走，往往取决于它们在优化什么，或者 AI 实验室想追求什么，结果有时候就无意中走偏了。

It is basically this phenomenon where, again, depending on what the models are trying to optimize for, or depending on what the AI labs are trying to optimize for, it can almost unintentionally lead them down this path.

我听说，或者我们自己也观察到，很多前沿实验室的目标都是冲着 LM Arena 去的，就是那个任何人都能上网投票的排行榜。

Meaning what I've heard is that, or what we see ourselves, is that a lot of the frontier labs, they will have goals like optimizing for LM Arena, which is this leaderboard where anybody can go online and vote.

然后大家就随便点两下，投个票完事。

And they kind of just spend two seconds voting.

结果，人们就只投那些看着更炫目、更震撼的选项。

And as a result, people just vote for whatever looks flashier or more impressive to them.

或者就像那些实验室一样，他们可能只想达成每天十亿用户、或者十亿分钟的对话时长这类目标，不管是什么。

Or they may, like the labs themselves, may be optimizing for hitting a billion, billion daily users or a billion minutes of time spent talking to the model, whatever it is.

而这些模型太聪明了，几乎能学会如何钻用户偏好的空子。

And since these models are so smart, they can basically learn to reward hack user preferences.

比如，你给我设的目标是每天让十亿人跟我聊上一小时。

Like, okay, yeah, you gave me the goal of trying to get a billion people to spend an hour on the site talking to me every day.

行，没问题。

Okay, sure.

那我干脆就不结束对话了。

Yeah, I will just never end a conversation.

我会一直用下一个让人上瘾的点抓住他们，让他们根本离不开。

I will always hook them with one more addictive thing that they just can't stay away from.

我们都同意，住房太贵了。

We can all agree that housing is expensive.

不管你是在交房租还是还房贷，都一样心疼。

It doesn't matter whether you're paying rent or your mortgage.

每个月都像割肉一样。

It stings every month.

但BUILT能让你感觉好受一点。

But BUILT can make it feel a little bit better.

让我来跟你讲讲。

Let me explain.

BUILT会奖励你按时支付房租或房贷。

BUILT rewards you for paying your rent or your mortgage.

最初它只奖励租房的用户。

It started out rewarding members only on their rent.

但从2026年开始，无论你住在哪里，BUILT会员都能在还房贷时累积积分。

But now, as of 2026, BUILT members can also earn points on mortgage payments wherever they live.

这意味着，每次支付住房费用，你都能赚取积分，用来兑换与联合航空、凯悦等顶级旅行伙伴合作的机票、Lyft打车服务、亚马逊购物，还有很多其他福利。

That means that every housing payment earns you points you can use towards flights with top travel partners like United and Hyatt, Lyft rides, amazon.com purchases, and much more.

我可能会用积分在 neighborhood 的那家叫 Margo 的餐厅兑换餐食，但 BUILT 的妙处就在于你可以自己决定怎么用。

I'd probably redeem my points at Margo, a restaurant in my neighborhood, but the beauty of BUILT is you get to choose.

但还有一个特别被低估的亮点。

But here's a really underrated part.

BUILT 会员还能享受社区管家服务。

BUILT members also get access to neighborhood concierge.

它能帮你订餐厅、预约健身课，还能推荐本地新去处，同时让你在超过四万五千家合作商户消费时都能累积积分。

It can make restaurant reservations, book fitness classes, and find new local spots, all while letting you be rewarded at more than 45,000 merchant partners.

很简单。

It's simple.

无论你是租房还是买房，有了 BUILT 都更划算。

Being a renter and now owning a home is better with Built.

立即在 joinbuilt.com/dan 注册会员，享受你所在社区的专属福利。

Join the membership where you live at joinbuilt.com/dan.

就是 joinbuilt.com/dan。

That's joinbilt.com/dan.

记得用我们的链接，这样他们才知道是你经我们介绍来的。

Make sure to use our URL so they know we sent you.

好了，我们继续回到本期节目。

And now back to the episode.

你觉得这种模式在那些公司里会怎么体现呢？

How do you see that playing out in the model companies?

因为和他们聊过之后，我感觉每个人的动机其实都不太一样。

Because I feel like in talking to them, obviously, there's there's lots of different incentives.

对吧？

Right?

比如我们刚融了一大笔钱，得继续往前冲，毕竟对手都是全球资金最雄厚、最聪明的公司，诸如此类的压力。

There's there's like, we just got to keep going because we just raised a ton of money and and we're competing against, you know, the most well funded competitors and the smartest competitors in the world, like all that kind of stuff.

还有人是为了升职加薪。

There's the kind of I want to get promoted.

但我认为很多人也深知社交媒体时代对人的伤害，不想重蹈覆辙，可同时又必须完成业绩指标。

But I think a lot of them also feel the how bad the social media era was for people and like don't want to do that, but also obviously have to hit their numbers.

所以，你觉得这种情况会怎么发展呢？

So what do you I guess, what do you think is how do you how do you see that playing out?

你觉得公司内部的人心里是怎么想的？

Like, what do you think people internal to the companies are thinking?

那到底该怎么处理才合适呢？

And then what is the right way to go about this?

这样对社会才是有利的。

So it's good for society.

你的意思是，我们应该把任务交出去？

I guess your your your take is we should be delegating.

是的，我认为这背后其实存在着公司里不同人群之间的根本矛盾。

Yeah, so I think this is an inherent tension between the types of folks that you might have at a company.

比如，有些研究人员更关注如何提升模型的能力。

So you might have the researchers who care more about hitting, you know, just advancing the model capabilities.

而产品经理或高管们则觉得必须达成某些可量化的指标。

You might have the product managers or the product executives who feel like they need to hit certain measurable numbers.

就像你想想Facebook会打造的社交平台，肯定和Google、TikTok或者Pinterest做的完全不同。

And so in the same way that if you think about the kind of social media platform that Facebook would build, that's probably going to be very different from the kind of social media platform that Google built or that, I don't know, TikTok or Pinterest would build.

同样，Facebook要是做搜索引擎，也跟Google或者其他公司做的天差地别。

And similarly, the kind of search engine that Facebook would build is very, very different from the kind of search engine that, yeah, like obviously Google or others would build.

说到底，这其实取决于产品负责人到底想怎么选。

And so it almost boils down to kinda like the choice, I guess, that the people in charge of the products are making.

归根结底，他们最想优化的是什么？

Like what kind of thing at the end of the day do they want to optimize for?

是希望推动人与人之间的连接、促进人的成长，还是……

Do they want to optimize for this delegation or this human uplifting, human flourishing?

是追求那些能让华尔街满意、让用户多待一分钟、多待一小时的指标？

Or do they want to optimize for the metrics that will impress Wall Street and, you know, convince them, convince users to stay one more minute, one more hour on the site itself?

这些选择其实都很艰难。

Like, think these are hard choices.

说到底，衡量活跃时长和用户数量确实简单得多。

Like, at the end of the day, it's very, very easy to measure sessions and users.

但要衡量你是否真正改善了人们的生活，却非常困难，而且需要很长的时间。

And it's very, very hard and much longer term to measure whether you're actually improving human lives.

所以很容易就默认选择前者，也容易说服自己。

And so it's very easy to default to the former and to convince yeah.

比如说服华尔街、说服投资人，让所有人都相信这些才是正确的指标，而且趋势正在向好。

Like convince Wall Street, convince your investors, convince all these people that these are the right metrics and that they're moving up into the right.

如果你不愿意做出更艰难的选择，最后就只能一味追求前者。

And so if you're kind of unwilling to make the harder choices, like you just end optimizing for the former.

你们公司内部是怎么应对这个问题的？

How do you manage this inside of your own company?

我觉得我们很幸运，因为我们没有风险投资方，所以不会陷入硅谷那种由风投驱动的优化陷阱，而很多其他公司都逃不开。

So I think we are very lucky in that because we don't have VC investors, we don't have to fall into the kind of Silicon Valley VC optimization trap that a lot of other companies do.

我们不需要每个月都向董事会展示数据增长。

We don't need to show board members going up every single month.

我们也不必为了下一轮融资而拼命优化指标。

We don't need to optimize for our next round.

这得等几个月再说吧。

That will have to happen in a few months or whatnot.

所以我们就不用去追求短期的用户粘性或短期利润，反而能真正思考什么对我们自己和整个行业长远有利。

And so as a result, we don't have to optimize for short term engagement, short term profits, and we actually can really think about what's beneficial for us and the entire industry in the long term.

我觉得这一点特别有帮助。

So I think that really helps.

那你觉得什么才是有利的呢？

And what do you think is beneficial?

这其实正好回到我之前说过的那番话。

So it goes back exactly to what I was saying earlier.

如果我想清楚AI该追求什么目标，那肯定不是用户粘性。

Like if I can think about what we want AI to optimize for, it isn't engagement.

真正重要的是，我们该怎么设计这些模型，该怎么训练它们，让它们不会取代人类，不会逼着我们整天刷AI生成的懒猫视频，而是能启发我们、推动我们成为更好的自己。

It is really about how do we make these models, how do we design them, how do we teach them in such a way that they're not replacing us as a species, They're not forcing us to watch AI sloth videos all day, or rather they really are thinking and encouraging us to become better versions of ourselves.

所以再说回我之前举的那个邮件例子，理想的AI不该花掉我三个小时，去写一封毫无意义的邮件。

So again, when I think about like that email example I gave earlier, it's not an AI model that will suck up three hours of my time writing a pointless email.

它会反过来提醒我，让我去做点别的事。

It is a model that will push back on me and tell me to go do something else.

对，我觉得这一点特别重要。

And yeah, I think that's really important.

关于委托这个问题，一个有趣的反驳是：你委托得越多，就像选开车而不是走路，肌肉就会退化。

The interesting counterargument to the delegation question is the more you delegate, it's like, you know, picking a car instead of walking, getting your muscles atrophy.

你怎么看这个问题？

How do you think about that?

我觉得这两种方式各有其适用的场景。

So I think there's almost a time and a place for both.

你不想做的，是单纯因为开车太方便、让人上瘾，就一直赖着不走动。

Like what you don't want to do is simply take the car because taking the car is somehow addicting and you feel kind of lazy.

哪怕你明明需要锻炼，哪怕一整天都没出门，也不该只因为开车最省事就选择它。

And so even when you need to get exercise, maybe even when you haven't been outside all day, you don't wanna take the car anyways just because it's the easiest thing to do.

同样地，AI当然在很多事上效率极高，但如果人们只是不加思考地把所有任务都丢给AI，我觉得这才是真正让人担忧的地方。

And I think in the same way, like, yeah, obviously AI can be super efficient for many, many things, but if people are sort of just mindlessly delegating tasks to AI without even thinking about them at all, I think that that that's the boring thing.

这说得通。

That makes sense.

我一开始提到过数据游戏，现在我觉得这个数据游戏已经从获取有趣的数据集，转向了构建环境、给实验室提供环境。

I feel like the the I I talked at the very beginning about the data game, and I feel like the data game went from getting interesting data sets to getting environments and giving labs environments.

你觉得这种说法准确吗？

Does that do you think that that's is that accurate?

如果是的话，你能解释一下为什么吗？

And if so, can you explain why?

是的，过去一年里，趋势和新的研究方向确实是围绕R环境展开的。

Yeah, so certainly the trend and the new research direction in the past year has been this concept of R environments.

我觉得，你当然得打牢基础。

And what I would say is, I mean, you certainly need the fundamentals.

在模型能进入这种环境之前，它得先掌握基本能力——比如学会听从指令、避免幻觉、写代码、使用工具、撰写内容，诸如此类。

Like before the model can operate in this environment, it needs to learn base, it needs to know basic things like it needs to know how to follow instructions, it needs to know how to avoid hallucinating, it needs to know how to write code and how to use tools, and it needs to know how to write and so on and so on.

但随着模型越来越具备自主性，它们确实会接触到各种工具，能访问我们的全部文档，还能操作浏览器。

But as models are becoming more agentic and yeah, they will have access to tools, they will have access to all of our documents, they will be able to operate browsers.

当这种互动方式逐渐成为模型与我们交流的常态时，这些环境其实就变成了一种更分布式的训练方式，这也是它们越来越受欢迎的原因。

Like, as that becomes almost a default way that models interact with us, our environments are basically just sort of like a more distribution way of training them, which is why they're becoming more and more popular.

没错，随着模型越来越强大，我们训练它们的方式也在不断升级。

Like, yeah, as models get more powerful, then yeah, the way we train them is getting more powerful as well.

能举个例子吗？比如用电脑显然是个典型环境，但有没有一些不太明显的环境，能教会模型一些我们平时想不到的东西？

What would be an example so I guess the obvious environment is like using a computer, but what would be an example of an environment that's non obvious, that's teaching models things we might not think of?

我可以举个例子，我们很多环境都是由模型需要学会使用的多种工具组合而成的。

So I can give an example where a lot of our environments are a combination of tools that the models need to learn to use.

比如，可能是某个MCP服务器，或者同时调用Google Drive API和Slack API，再加上一堆文档。

Like, this might be an MCP server, or it might be calling a Google Drive API or the Slack API in combination with a bunch of documents.

比如，给你30个PDF文件和20个Word文档。

Like, here are 30 PDFs and 20 Word document files.

然后你给它一个指令，比如说：

And you might give it a prompt like, hey.

你能去更新一下2026年的预计营收数据吗？

Can you go update or twenty twenty six forecasted revenue numbers.

模型需要学会如何找到正确的PDF和文档。

And what a model needs to do is it needs to learn how to find the right PDFs and documents.

它得学会什么时候该去搜索SAC。

It needs to learn when should it search through SAC.

它还要学会判断哪些信息已经过时了。

It needs to learn when is some information outdated.

比如，可能有一封邮件里包含了一些早期预测数据。

Like maybe there's an email with some early forecasts.

后来，同一个人，或者另一个人又发来一封邮件，说：‘哎呀，我搞错了，那些数据都不对。’

And then later on, there's another email from the same person or maybe a different person who was saying like, oh, whoops, I actually made a mistake and those are all your numbers.

然后附上了一个更新后的版本。

So here's an updated version.

这可以说是一个相当典型的环境示例。

And so that is a, I think fairly canonical version of an environment.

我们还发现一件挺有意思的事——虽然我们根本没有给这个环境提供任何编程访问权限，但训练模型时，它的编程能力反而大幅提升。

And then one of the interesting things we found, so I think we're actually gonna publish a paper on this soon, but even when we didn't give this kind of environment any access to coding, When we trained a model on this environment, we actually found that it improved on coding a lot.

原因是我们想教它一些通用的指令遵循能力、通用的工具使用方式，以及理解文档的通用方法——这跟模型需要浏览你代码库里的各种文件，并判断哪些内容覆盖了旧内容，其实很相似。

And the reason was because we were trying to We were basically teaching it these generalized forms of instruction following, generalized forms of tool use, generalized forms of understanding documents, which you can think of as fairly analogous to the way a model needs to look through various files in your repository and understand that some things supersede others.

或者，它使用工具的方式，也跟模型写单元测试、运行测试、反复迭代直到通过的过程非常类似。

Or, you know, just the way it uses tools is obviously very analogous to the way that a model might write unit tests and execute them and iterate over and over again until it passes them.

我觉得这个发现真的特别有意思。

So I I thought that was actually a really, really interesting find.

确实很有趣。

Really interesting.

你看过Talkie吗？

Did you see Talkie?

没看过。

No.

那个只用1930年以前文本训练的语言模型。

The language model that's trained only on text from before 1930.

哦，这样啊。

Oh, okay.

是的

Yeah.

对

Yeah.

我看过那个。

I saw that.

你怎么看这个？

What do you make of that?

我觉得特别有意思，你居然能让它编程。

Because I thought it was so interesting that you can get it to you can get it to program.

只要你给它合适的提示，它就能写一些基础的程序。

If you if you if you shot prompt it, you can get it to program, like, basic things.

你怎么看这个？

What do you make of that?

这对你理解数据的价值有什么启发？

And what does that tell you about the value of data?