“硬分叉”直播第三部分：对人工智能未来的不同愿景

我是莫里斯·特雷蒙特，这是Serial Productions、马歇尔计划和《纽约时报》联合推出的新播客。

This is Maurice Tremont, the host of a new podcast from Serial Productions, The Marshall Project, and The New York Times.

去年，我花了三个月时间，全程跟拍一支死刑辩护团队。

Last year, I spent three months embedded with a capital defense team.

他们的当事人在死囚牢房里待了三十多年，现在，执行日期已经定了。

Their client had been on death row for more than thirty years, and now his execution date had been set.

我跟着律师们一起努力，想证明一件事——这件事三十年来没人成功过：得州最臭名昭著的连环杀手，其实是清白的。

I followed along as the lawyers tried to prove something nobody had successfully done in three decades, that one of Texas' most notorious serial killers was actually innocent.

最后的十二周。

The last twelve weeks.

欢迎在你常用的播客平台收听。

Listen wherever you get your podcasts.

本集《硬叉》由2026年Hard Fork现场活动的赞助商冠名支持。

This episode of Hard Fork is brought to you by our Hard Fork Live twenty twenty six sponsors.

首席赞助商：IBM。

Premier sponsor, IBM.

联合赞助商：Everpure、Pure Leaf 和圣母大学，支持赞助商：Atlassian。

Associate sponsors, Everpure, Pure Leaf, and the University of Notre Dame, and supporting sponsor, Atlassian.

嗯，凯西，我们还在放暑假呢。

Well, Casey, we are still on our annual summer vacation.

你敢信吗？我们还没跟听众分享的 Hard Fork Live 现场内容居然还有这么多精彩片段。

And can you believe there is yet more amazing stuff from Hard Fork Live that we have not shared with our podcast listeners?

确实有。

There is.

特别是活动现场，丹尼尔·科科塔约洛和萨亚什·卡普尔就人工智能的发展速度展开了一场特别有趣的讨论，他们的看法不太一样。

In particular, we had a really fun discussion at the event between Daniel Kokotajlo and Sayash Kapoor who have somewhat different views of how fast the AI conversation is gonna go.

我们以前听过他们辩论。

We've heard them debate before.

这次我们想让他们再聊聊，毕竟距离上一次对话已经快一年了。

We wanted to sort of have an updated discussion with them now that it's been, like, you know, getting close to a year since the last time they had it.

我相信你一定会喜欢听他们怎么说的。

So I think you'll really enjoy hearing what they have to say about that.

我们还请到了非常棒的播客主Dwarkesh Patel来坐坐，跟我们聊聊他最近在想些什么。

We also had the great podcaster, Dwarkesh Patel, stop by and hang out with a bit, tell us a little bit about what is on his mind.

最后呢，我们还做了一轮现场问答，听听在精彩的Hard Fork Live第二季之后，观众们都在关心哪些问题。

And just to round it out, we took some live q and a and heard what was on the minds of our audience after a spectacular Hard Fork Live two.

所以，我觉得这些对话都可以归为一类，就是那种‘圈内人’在梳理思路、搞懂现状的讨论。

So these are all conversations that I would classify in sort of the same bucket of, like, insider sense making.

这些人深深扎根于旧金山的AI圈子，他们正努力向外界解释和理解目前的情况，比如技术进步的节奏，以及这些模型未来的发展方向。

People who are deeply enmeshed in the AI scene in San Francisco trying to understand and explain what is going on, the pace of progress, the trajectory of these models to the outside world.

Sayash、Daniel 和 Dwarkesh 是我听过的最擅长向外界解释这些概念的三位天才，而外界往往并不完全清楚到底发生了什么。

And Sayash, Daniel, and Dwarkesh are among the three most gifted people I have ever heard try to explain this stuff to an outside world that doesn't always know exactly what's going on.

这一系列对话非常精彩。

It's a great set of conversations.

我相信大家会非常喜欢。

We think you'll really enjoy it.

这是我们从 Hard Fork Live 第二季带来的最后一期节目。

This is our final installment of our episodes from Hard Fork Live two.

我们两周后回来，继续正常的《Hard Fork》节目。

We will be back in two weeks to our regularly scheduled Hard Fork programming.

这段时间，好好享受夏天吧。

In the meantime, enjoy your summer.

记得涂防晒霜。

Wear sunscreen.

接下来这个环节，我特别期待，因为我们请来了两位对当前AI发展态势看法截然不同的人。

So this next segment, I am so excited for because we're gonna have a conversation with two people who have very different views about how AI is going right now.

对。

Yes.

今晚我们请到了丹尼尔·科科塔约洛。

We have Daniel Kokotajlo with us tonight.

他是《AI 2027》报告的合著者，我相信很多听众都读过这份报告。

He is the co author of AI 2027, a report that many of you, I'm sure, have read.

这份报告发布于2025年，描绘了一个非常生动的场景：AI如何彻底改变世界，比如实现自主编程和研发。

This came out in 2025 and laid out a vivid scenario or account of how AI could fundamentally upend the world, achieving tasks like autonomous coding and R and D.

他之后又几次更新了这个预测。

He's since updated that prediction a few times.

我们会问他这方面的问题。

We'll ask him about that.

他还邀请了来自普林斯顿的AI研究者萨亚什·卡普尔，他对未来有着完全不同的看法。

And he'll be joined by Sayash Kapoor, who is an AI researcher at Princeton, with a very different view of the future.

他是《AI作为普通技术》一书的合著者，这本书分析了大量证据，表明AI就像以往那些颠覆经济、但需要很长时间才能渗透到社会中的技术。

He's the coauthor of AI as Normal Technology, which looks at evidence that AI is much like previous technologies that have upended the economy, that take a long time to diffuse through society.

我们邀请他们两位今晚来，是因为去年他们在一场叫《The Curve》的AI会议上有过一场非常精彩的辩论。

We've invited them both here tonight because we saw last year a very interesting debate that the two of them had at an AI conference called The Curve.

我们觉得那场辩论太有意思了，所以决定请他们回来，听听他们之后的观点有没有变化，现在哪些地方还持不同意见，又有哪些地方开始达成共识了。

We thought it was so interesting that we decided to bring them back tonight and hear how their views have evolved since then and where they continue to disagree and where they might agree now.

让我们热烈欢迎丹尼尔·科科塔约洛和萨亚什·卡普尔。

So please give a warm welcome to Daniel Kokotajlo and Sayash Kapoor.

嗨，丹尼尔。

Hey, Daniel.

你好，Sayash。

Hey, Sayash.

好的。

Alright.

所以，Daniel，Kevin刚才在上面提到了这一点。

So, Daniel, Kevin mentioned this up top.

自从你首次发布《AI 2027》以来，你已经几次更新了你的时间线。

You have updated your timelines a few times since you first published AI 2027.

给我们分享一下你最新的看法。

Give us the most up to date view of your thinking.

你觉得我们什么时候能实现具备自主AI研发能力的模型？

What's your best estimate for when we will achieve AI models that can do their own AI r and d?

大概到2028年底，可能性有50%。

Probably 50% by late twenty twenty eight.

好的。

Okay.

这也太早了吧。

That's soon.

我在想日历上的时间。

I'm I'm I'm thinking about the calendar.

那就是两年后。

That's two Yeah.

这比大多数人预期的要晚一点，我觉得。

That's, like, a little bit later than probably expects, I think.

事情总是比计划的花的时间长。

Things take longer than you you plan for.

你知道的。

You know?

这正是Sayash常提到的一点。

Which is a point that Sayash makes sometimes.

Sayash，你能说说你现在的看法吗？

Sayash, can you summarize where your views are today?

我觉得你不认同那种突然爆发的观点，很多其他观察者倒是这么想的。

My sense is that you do not believe in the sort of sudden takeoff scenario that some other observers believe in.

说得对。

That's exactly right.

我觉得主要原因在于，这个分歧本质上是看自动化研发的瓶颈到底是纯粹算力问题，还是说其实还依赖一些现实世界中的障碍，这些障碍很难被自动化消除。

I think the main reason for that is basically this disagreement boils down to the whether the bottlenecks to this intelligence explosion, the bottlenecks to automating R and D are all computational, or whether they rely on real world bottlenecks, that will be really hard to automate away.

所以，这大概是我们观点不一致的一个地方。

So I guess this is one place where we disagree.

我觉得在很多领域，要取得这些进展，不会像在编程领域那样容易。

I think that in a lot of domains, making these advances won't be as easy as it had in brain coding.

要真正实现人工智能超级智能，你得在所有这些领域都取得突破。

To really get to artificial superintelligence, you need to cover all of these different domains.

你需要在各个方面都具备高样本效率，这在编程这类有模拟器和虚拟环境的领域相对容易，但在现实世界里就难多了。

You need sample efficiency across the board, which is much easier to do in a field like programming where you have these simulators, these virtual environments, but much harder to do in the real world.

而且，有些证据也支持这一点。

And, you know, some evidence bears this out.

在编程之外的其他领域，AI系统的采用速度确实要慢得多。

Adoption of AI systems has been indeed far slower in other domains as opposed to coding.

你能举个例子，说明这些瓶颈具体是什么吗？

So give us an example of what these bottlenecks are.

我接触过不少AI研究人员，他们给我的感觉是，

So I talked to a lot of AI researcher folks, and for to them, the way at least the way they make it sound to me is, look.

最终模型只要够强了，事情就结束了。

Eventually, the model just gets good enough, and then it's game over.

你说这个世界是真实存在的，我想听听你具体怎么看。

You're saying that there's something that exists called a real world, and I'd like to hear more about it.

你看，

I mean, look.

说实话，我觉得这不过是两种各自自洽的、对AI未来不同的看法。

I mean, to be honest, I think these are just two independent, self consistent worldviews about the future of AI.

我和丹尼尔之所以能有这么深入的对话，正是因为我们在努力厘清这两种观点到底哪里不同。

And the reason that Daniel and I have had such productive conversations is that we are basically trying to figure out where these worldviews differ.

说到这个，我觉得丹尼尔的做法和预测，完全符合我们即将迎来AI系统的这种世界观。

Now, speaking of, I think like Daniel's actions and Daniel's predictions are entirely self consistent with the worldview that we'll get to AI systems at this point.

不过，要想拿到证据来证明我们到底走的是哪条路，我们就得真正去做大量的评估。

Unfortunately, in order to get evidence we are going one way or the other, we need to actually carry out lots of evaluations.

我们需要把评估标准提得比现在高得多。

We need evaluations to be of a much higher standard than we have today.

那我举个瓶颈的例子吧。前几天我和一个当律师的朋友聊天，他就用这些工具。

So to give you one example of a bottleneck, the other day I was talking to a lawyer friend of mine, and he uses these tools.

他对这些工具非常看好。

He's very bullish about them.

但实际情况是，随着他开始用这些工具处理越来越大的任务，幻觉率和不可靠输出的比例基本没变。

But what has turned out to be the case is as he started using these tools for bigger and bigger tasks, the rate of hallucinations, the rate of unreliable outputs has sort of remained the same.

对吧？

Right?

这并不是因为AI系统没有进步，它们确实进步了。

It's not because the AI systems haven't gotten better, they indeed have.

它们现在的表现，比一年前强太多了。

They are so much better today than they were just a year ago.

但事实上，你能用这些系统完成的任务，受限于幻觉率和可靠性。

But the fact is that the tasks that you can do with these systems actually are bounded by the rate of hallucinations or the reliability.

这一点上，AI系统依然面临挑战。

That's one place where AI systems continue to struggle.

在软件工程这类能立刻运行代码、看到结果的领域，这个问题相对好解决；但像法律这种领域，就连专家都未必能确定正确答案。

In a domain like software engineering where you have this instant feedback loop where you can actually run the code and see what the output would be, it's a much easier bottleneck to address as opposed to something like the law where even the right answer is not obvious to a domain expert.

专家们对处理方式的看法，也可能有合理分歧。

Domain experts can reasonably differ in the approach that they take.

这只是一个例子，说明在答案比较主观的领域，AI的瓶颈更难突破。

So this is just one example of a bottleneck in a domain where the right answer can be a bit more subjective than encoding.

丹尼尔，我觉得AI 2027刚发布的时候，有些人把它当成空想，甚至说这是吓人的科幻情节。

Daniel, I think when AI twenty twenty seven first came out, there were some people who dismissed it as sort of speculation or scary science fiction was a term that some people were throwing around a lot.

我当时报道过这件事，还采访过你和你的合著者。

Reported on this, I talked to you and your coauthors then.

我知道你是基于实际的预测工作来写这本书的，花了好几个月去推演随着技术进步会发生什么。而且说实话，里面很多预测已经成真了。

I know that you grounded this in like real forecasting work, like months of trying to figure out what would happen as the technology got better, and I will say like a lot of that has come true already.

你在《AI 2027》里预测，我们会看到编程的大部分环节开始自动化。

So you predicted in your AI 2027 that we would start to see large parts of coding become automated.

这一点确实已经实现了。

That much has come true.

我今天看到有人提到，你之前写过关于 Frontier Labs 限制其模型用于前沿大模型开发的内容，这跟本周 Claude 发生的情况很像。

I was reading today someone was copying and pasting something that you had written about Frontier Labs sort of restricting the use of their models for Frontier LLM development, something that has happened this week with Claude Fable.

那么，如果按照你的设想，到2026年下半年情况基本不变的话，你觉得会发生些什么？

So what are the things that you think will happen if your scenario continues to mostly hold for, let's call it, the rest of 2026?

今年我们会看到什么？

What are we gonna see this year?

按照你描述的明年会发生的情景，今年不会出现智能爆炸。

So we're not gonna see an intelligence explosion this year in the scenario that happens next year.

这已经很接近了。

That's close.

那挺好的。

So that's nice.

我觉得

I think

爆炸指的是自我迭代的递归提升，也就是那种彻底失控、超越人类的超级人工智能。

explosion being recursive self improvement leading to That's right.

没错。

Sort of out of control, runaway, superhuman AI.

换句话说，就是完全自动化AI的研究流程，让AI研究的进展比现在还要快得多。

Yeah, or to put it another way, just fully automating the AI research process causing AI research to happen even faster than it currently happens.

而目前它的速度已经远超许多其他技术了。

And it's currently happening at a very fast rate compared to many other technologies.

但我会说，编程代理会变得越来越强，可能再过一年，或者两年，它们就会强大到足以说已经完全自动化了编程工作。

But yeah, I would say the coding agents are just gonna get better and better and that maybe a year from now, maybe two years from now, they will be good enough that you can sort of say they've automated coding fully.

它们还没完全自动化编程，但可能一年或两年后，就能做到彻底自动化了。

They haven't fully automated coding yet, but like maybe in a year or two, they'll have fully automated coding.

到那时，瓶颈就会变成研究的判断力、管理能力，以及除了实际编程之外的其他所有AI研究环节。

At which point the bottleneck will be research taste and management and all the other aspects of the AI research process besides the actual coding.

然后企业会转向解决这些瓶颈，教他们的AI也掌握这些能力，这需要一些时间，但一旦编程完全自动化了，进展速度会比你想象的快得多。

Then the companies are going to turn towards resolving those bottlenecks and teaching their AIs to do those skills as well and that's going to take some time but it's going to go by faster than you might think when all the coding has been automated.

等他们完成了这些工作，也不会立刻拥有超级智能。

Once they've finished doing those things, they won't have super intelligence immediately.

你知道，第一个能完整执行AI研究流程的AI系统，可能还做不了其他很多事。

You know, the first AI system that can do the complete AI research process probably won't be able to do various other things.

但一旦AI研究流程实现了全面自动化，事情就会越跑越快，而那种能做任何事情的系统，可能也就不远了。

But once they've fully automated the AI research process, things will probably go faster and faster And then the type of system that can do absolutely everything is probably not far off.

萨亚什，你认为这种递归式自我提升真的可能实现吗？

Sayash, do you believe this sort of recursive self improvement is possible?

我的意思是，从某种角度说，递归式自我提升的过程早在六十年前就已经开始了。

I mean, in some sense, I think the process of recursive self improvement started like six decades ago.

事实上，整个计算史就是一部不断开发工具、而这些工具又帮助我们制造更好工具的历史。

In fact, the entire history of computing has been one where we develop tools that then aid us in the development of better tools.

我们开发了编译器，让编程效率提升了近百倍。

We've developed compilers that have allowed us to be like two orders of magnitude better at programming.

在此基础上，我们又构建了各种框架和系统、库，现在能做的事情，要是用汇编语言，就算经验丰富的工程师也得花上好几年甚至几十年。

We've developed frameworks on top of that, we've developed entire systems and libraries that allow us to do things that would, frankly, take an experienced software engineer years or decades of time if they were using assembly language.

我觉得，这个循环其实早就已经启动了。

I think in some sense this loop has been kick started already.

整个计算机发展史，说白了就是这么一个循环的体现。

This loop is something that the entire history of computing bears out.

我对丹尼尔的预测有不同看法：这个过程真会自然催生出全自动的AI研发研究员吗？还是说人类始终保有优势，人类团队配合AI会持续胜过纯AI？这个过程最终真能导向人工智能超级智能吗？

What I disagree with in terms of Daniel's predictions is whether this process will naturally lead us to a point where we develop the automated AI R and D researcher, or whether humans will continue to have this edge and teams of humans with AI will continue to outperform AI alone, and whether this process will lead to artificial superintelligence.

我个人觉得，AI系统持续提升在AI研究任务上的表现，这种自我迭代的场景是非常有可能实现的。

I actually think that it's a very plausible scenario for me that we get the sort of recursive self improvement that AI systems do indeed continue performing better and better at AI research tasks.

但这个过程的终点未必就是超级智能。

But the end process of that need not be ASI.

它可能只是产生比现在更强大的模型，就像以往技术演进那样，但还没达到那种全面超越人类顶尖专家的水平——而我认为，那才是真正的AI的定义。

The end process could just be more capable models than we have today, perhaps following the trend of previous technologies, and yet not the point where we have, you know, these systems that outperform humans, the top human experts and everything, which is, I believe, the definition of AI.

也许我们可以聊聊双方达成共识的地方。

Perhaps we should talk about the the point of agreement.

对。

Yeah.

对。

Yeah.

具体有哪些共识呢？

What's the point of agreement?

对。

Yeah.

所以我们合写了一篇博客文章，是关于《AI作为常规技术》和《2027年的AI》这两篇报告的作者，我们讨论了彼此达成共识的观点。

So we wrote this blog post together, the authors of AI as a Normal Technology and AI 2027, where we talked about the things that we agree on.

如果我理解有误请纠正我，但大致来说，我们谈到了你们可能称之为强AGI，或者像“云端人类”这样的概念，也就是那些能够完成所有认知任务，或者在电脑上完成人类所能做的任务，甚至能与专业人类或顶尖专业人士相媲美的AI。

Correct me if I'm misstating it, but roughly speaking, we talk about what you might call strong AGI or like humans in the cloud, like AIs that can sort of do all the cognitive tasks or the tasks you can do at your computer as well as professional humans or as well as the best professional humans.

我想标题大概是，我同意那些能力没那么强的AI仍然属于常规技术。

And you know, I guess the headline is I agree that AIs that aren't that powerful are still normal technologies.

而他们认为，那些具备如此强大能力的AI并不属于普通技术。

And they agree that AIs that are that powerful are not normal technologies.

没错。

Exactly.

换句话说，在我们有了‘云端人类’这样的AI之后，普通技术这个说法就不再准确或有用了。

Or like the normal technology thesis sort of stops being accurate or helpful in a world where we have, like, humans in the cloud, let's say.

是的。

Yeah.

我们花这么多时间讨论递归自我改进，是因为观察者普遍认为，这是AI发展中最令人担忧的时刻。

The the reason that we, you know, spend this time talking about recursive self improvement is that, you know, RSI is kind of the the moment that observers believe is, kind of the scariest moment in the development of AI.

对吧？

Right?

它变得越来越难以控制。

It's like it becomes ever harder to control.

因此，我们离这个时刻还有多远？它是否真的可能发生？我认为，这可能是我们在播客里问过的最重要的两个问题。

And so how far away are we from it and is it possible, I think, are probably two of the most important questions that we will ever ask on the podcast.

我听了你那些听起来很合理的质疑，明白你为什么觉得短期内不太可能实现，但同时我也理解，丹尼尔，你为什么还是相信这是有可能的。

You know, having heard your, you know, what sounded to me like very, you know, sensible objections to to why it may not be possible anytime soon and I and understanding, Daniel, you know, why you do think it's possible.

所以我想知道，至少你是不是希望萨亚什是对的？

I'm curious, like, if at the very least, like, you hope Sayash is right.

你是这么想的吧？

You know?

如果是这样，你会不会松一口气？

Like, would you breathe a sigh of relief?

是的。

Yeah.

如果你是对的，我反而会很高兴。

I would love it if you were right.

明白了。

Okay.

对。

Yeah.

好的。

Okay.

谢谢你，丹尼尔。

Thank you, Daniel.

但你看到什么让你觉得他不对呢？

But what you see that makes you think that he's not right?

我花了一些时间思考，阻碍Anthropic实现其目标的障碍或瓶颈到底有哪些，但说实话，这些障碍都没我觉得那么严重。

So I think that like I've tried to spend some time thinking about like what are the barriers, what are the bottlenecks that could block Anthropic from succeeding in their stated plans and none of them really seem that strong to me basically.

对。

Yeah.

我们可以一个个来仔细分析。

So we can we can go through them like bit by bit.

你提到的数据效率，确实目前AI比人类效率低，但如果企业愿意投入，这方面应该能快速取得进展。

Like data efficiency you mentioned, it does seem like AIs currently are less data efficient than humans but that also seems like something that companies could probably make rapid progress on if they tried.

而且另外一点，对于自动化AI研究过程来说，数据效率可能也没那么关键。

And also separately, it may not actually be that important for automating the AI research process.

也许你根本不需要把数据效率提升到人类水平，就能实现99%自动化AI研究流程。

It might be that you can sort of like 99% automate the AI research process without getting that data efficiency to human level.

就算还没完全达到，光是99%的自动化也能大大加快进度，说不定一年就能做完十年甚至二十年的研究工作。

Then even though that's not like quite there, even a 99% automation would speed things up quite a lot which would then allow you to do a decade or two decades worth of research in a year perhaps.

这大概就是我认为我们已经非常接近了的两个理由。

Those are I think my two arguments for why it seems like we're bringing it pretty close.

还有一个更宏观的论点：过去几十年，AI研究人员和评论者总在说AI做不到某些事，比如深度学习会撞上什么墙，可这些所谓的障碍几乎一被提出来就被打破了。

Another argument, a sort of meta argument that I would make is that I feel like there's been a long history of AI scientists and other commenters making claims about what AIs can't do, like various walls that deep learning is going to hit, and they just keep getting smashed through almost as soon as people are making the claims.

我觉得数据效率这个问题，大概也会是同样的情况。

And I just feel like that's probably what's gonna happen with data efficiency, for example.

对。

Yeah.

咱们先停一下，因为这一点我觉得特别重要，我也有同样的观察。

Let let let's pause there because that actually seems really important to me because that's been my observation as well.

所以我也更愿意相信那些实验室做出的宏大预测。

And it's why I'm am more inclined to believe the labs when they make grand pronouncements.

对吧？

Right?

Sayash，我很好奇，你对这个怎么看？

Sayash, So, I'm curious, what is your relationship to that?

你也亲眼见过这些模型不断涌现，把各种基准测试都碾压了，评估指标很快就饱和了，我们不得不重新设计新的。

Because you've also seen these models come along and blow away the benchmarks and see the evals get saturated and we have to make new.

事实上，你自己也在开发新的评估体系，因为旧的早就用不成了。

In fact, you've been making your own evals because the old ones got saturated.

是的。

Yeah.

我们参与过好几个评估体系的开发，比如Anthropic就用过。

I mean, we've worked on several evals that, for example, Anthropic has used.

在Opus 4.5发布后，我们是第一个指出：这事儿已经基本解决了。

And for Saturated with the release of Opus 4.5, we were the first ones to say that, look, this is, like, solved now.

我觉得这种进步还会继续下去。

And I think this progress will continue.

我觉得只要我们能足够清晰地定义任务，我们就能继续开发出能解决这些问题的 AI 系统。

I think as long as we can specify things well enough, we'll continue to build AI systems that can solve those tasks.

我可能不太一样的一点是，我不认为这个过程的最终自然终点会像我们之前看到的数据效率提升那样。

Where I differ perhaps is whether the natural endpoint of this process is something like we saw data efficiency.

对此我持怀疑态度，原因有几个。

I'm skeptical about that for a couple of reasons.

首先，样本效率或数据效率并不是让我们更早实现所谓‘云端人类’水平的唯一瓶颈。

First, sample efficiency or data efficiency is not the only bottleneck to getting what we call humans in the Cloud earlier.

回顾过去AI的发展进程，我们一直在打造更通用的系统，但在任何特定的通用程度下，我们都很难以预测通往下一层级的瓶颈在哪里。

And the past sort of if you look at past progress in AI, we've continued to develop these more general systems, but at any given level of generality, we've been really bad at predicting what the bottlenecks to the next level are.

我们也很难以判断何时能突破这些瓶颈，以及解决它们需要哪些底层的变革性突破。

We've been really bad at knowing when we solve those bottlenecks and what underlying transformative breakthroughs are needed to solve them.

就拿Transformer的崛起来说，在此之前，整个AI研究界对神经网络充满了质疑。

And as evidence of that, perhaps we can take the transformer moment and before that we can take all of the skepticism about neural networks that pervaded the research community in AI.

结果没过几年，整个社区就转变了风向，现在大家都全力投入Transformer的研究。

And it took a matter of a few years until the community pivoted, and now everyone is all in on transformers.

但也许这种架构选择也不是正确的。

But perhaps that's not the right architectural choice either.

也许你还没发现那些能让我们实现数据高效AI的新架构，而即便有了这些架构，可能还是不足以让我们达到人类在云端那样的样本效率。

Perhaps you're yet to discover these new architectures that would allow us to make these data efficient AI systems, and perhaps those will still not be enough to get us to the point where we have the sample efficiency of humans in the cloud.

所以，大致上就是这么个情况。

So that's sort of the broad stroke of things.

我觉得整个AI界在预测短期趋势上其实挺准的，尤其是那些近在眼前的进展，但在预测会彻底改变研究范式的重大转折上，却一直很不准。

I think the AI community in general has been really accurate about near term predictions, about things that are within the event horizon, so to say, and has been really bad at predicting transformative shifts that sort of change the entire research paradigm.

不过，公平地说，丹尼尔在2021年的报告里，对2025年会是什么样子，倒是猜中了几点，对吧？

And maybe, like credit where credit is due, I think Daniel was one of the few people who got some things right in his report from 2021, was it, about what 2025 looks like?

但总体来看，这个领域预测失误的记录实在不怎么样。

But in general, I would say the community has a very poor track record.

那你再多说说。

Well, say more of it.

比如，他们做过哪些完全没准的预测？

Like, what's a prediction that they made that just wasn't true at all?

你说什么？

Come again?

我是说，人工智能行业做过哪些预测，结果完全错了？

Like like, what is that prediction that the AI industry made that just was not true at all?

比如，整个业界对神经网络的怀疑态度。

I guess, like, the entire skepticism about neural networks.

从九十年代到两千一十年代，整个AI圈都把神经网络当笑话看。

So from the nineteen nineties to the twenty tens, the entire AI community has dismissed neural networks as a joke.

那时候，如果你研究神经网络，能认真对待你的人，两只手都数得过来。

Basically, you could count the number of researchers who took you seriously if you worked on neural networks on two hands.

要不是有少数人坚持下来，比如发布了那个大型数据集的人，还有约书亚、杨和杰弗里·辛顿，后来他们因为深度学习的研究拿了图灵奖，这个领域根本撑不到今天，更别提推翻那些怀疑论者的说法了。

It was only through the persistence of a few people like who released this big dataset that led to the deep learning revolution, and Joshua and Yan and Jeffrey Hinton, who later went on to win the Turing Award for their work on deep learning, that this sort of subfield persisted and eventually was able to disprove claims of skeptics.

同样的道理，我觉得现在AI圈对Transformer模型的执念可能太深了，反而忽略了其他真正有突破性的方向，那些本该被重视的创新，现在都被边缘化了。

In the same way, I think the AI community might be hurting too much around, let's say, transformer based models right now, and perhaps at the expense of other transformative improvements that are breakthrough improvements that are being sidelined because of this community single-minded focus on it.

我觉得你俩都有过类似的经历，我和凯西也一样：我们写的东西明明很谨慎、很周全，结果别人偏偏往最离谱的方向去理解。

I think an experience that you both have in common and that Casey and I also share is writing things that we think are very measured and careful and precise, and then just having people interpret them in the wildest possible ways.

你们俩都发表过那种引起广泛关注的文章或推演，结果立刻就被那些立场极端的阵营给盯上了。

You both published your sort of breakout essays, scenarios, and it was immediately both of them were sort of seized on by these polarized camps.

你知道的，前白宫顾问大卫·萨克斯就发过一些内容，说AI只是一种普通技术，一方面赞同你的观点，另一方面又对你改变预测表示不满，还说什么那些唱衰的人现在被逼到墙角了。

You know, David Sachs, the former White House adviser, was, you know, posting things about AI being a normal technology and sort of agreeing with you and taking issue with you for changing your forecast, and, oh my god, the doomers are are are, you know, are backed into a corner now.

加里·马库斯、J.D.万斯，还有伯尼·桑德斯等各路人士，都拿你们的论点来佐证他们原本就持有的看法。

Gary Marcus and JD Vance and all Bernie Sanders and all kinds of people have used your arguments in support of kind of whatever they already believed.

看着你们的作品以这种可能并非你们预期的方式产生涟漪效应，你们感觉怎么样？

How has that been to watch your work ripple out in maybe these ways that aren't what you expected?

好吧，我先说吧。

Well, I guess I'll go first.

这有点像一种信仰，对人性的一种信仰。

It's been a sort of leap of faith, faith in humanity.

我在 OpenAI 的时候也做过类似的场景预测，不过规模小很多，投入的精力也少，但那些只是内部使用，我不会被允许公开发布它们。

At OpenAI I was doing scenario forecasts like this too, much smaller, low effort versions, but they were just for internal use only, like I wouldn't be allowed to publish them.

我觉得，世界真的需要觉醒，意识到 AI 的到来以及接下来会发生什么，并开始更认真地思考这个问题。

And it seemed to me that the world really needs to wake up to AI and what's coming and start thinking more seriously about it.

而且说实话，现在的讨论环境并不理想，充斥着很多糟糕的观点和言论，确实很混乱、让人困惑。但我们AI未来项目还是选择相信：既然看到了趋势，就该把我们的判断说出来。

And, you know, the discourse is not necessarily so great and there's lots of terrible people, and lots of terrible takes and, you know, it's very chaotic and confusing, but, we at AI Futures Project are sort of making a bet that like, well, we should say what we think is coming.

我们得说清楚。

We should be clear.

我们得表达得明白。

We should be articulate.

我们得把背后的理由讲透。

We should explain our reasoning.

讨论一旦展开，大家都会各抒己见，希望最终能慢慢靠近真相，希望整体上能促成更明智的决策。

The discourse will get rolling, lots of people will say lots of things, hopefully in the end it will converge towards the truth, hopefully in the end it'll converge towards better decision making on average.

咱们就走着瞧吧。

And, we'll see what happens.

我相信。

I have faith.

Sayash？

Sayash?

让我最没想到的是，居然这么少人会认真读完整篇文章。

I guess the biggest surprise for me was how few people read things in-depth.

我的意思是

I mean

对啊。

Yeah.

比如说

Like,

这真的让我很震惊。

it was honestly shocking.

我们在文章开头就把人工智能比作互联网，或者像电力革命那样。

Like, in the first line of the essay, we compare AI to the Internet or perhaps the electricity, like, electrical revolution.

我们谈到人工智能的影响，差不多能和第一次工业革命相提并论。

We talk about AI's impact as sort of being at par with perhaps the first industrial revolution.

结果有些人把我们和加里·马库斯归为一类，这真让我大吃一惊。

And people put us in the same camp as Gary Marcus sometimes, which is just honestly shocking.

不过，说真的，能看到这些智识社群用这些文章来深化他们的思考，我觉得挺不错的。

But, you know, like, one level deeper, I think it has been really nice to see sort of these intellectual communities use these essays to advance their intellectual thinking.

我觉得最让我意外的是，那些认真思考AI未来的人，竟然把我们的文章，甚至两篇都看得这么重，这真的让人很暖心。

I think perhaps the biggest surprise to me was the fact that, like, our essay and perhaps both of our essays were sort of taken so seriously by people who are thinking deeply about the future of AI, and that was really heartwarming.

回过头看，你有没有对用“正常”这个形容词来描述AI这件事产生过犹豫？

Looking back, have you ever, like, had second thoughts about using the adjective normal to describe AI?

因为我读了你的文章，觉得论证得非常精彩，我也经常分享给朋友们，帮他们理解为什么AI的普及速度可能比大家想的要慢。

Because I I read your writing, and I think it's beautifully argued, and I share it widely with folks to sort of help them explore, you know, reasons why AI may diffuse more slowly than other folks think.

但我其实一直觉得，AI 并没有那么普遍。

And yet I I have never really thought that AI was all that normal.

你明白我的意思吧？

You know what I mean?

我理解你的意思。

I do understand that.

我是说，我想其中一部分原因是，我们一直处在这样的讨论循环中，至少那些认真思考 AI 的人都默认 AI 具有变革性，我们也是这样认为的。

I mean, I guess part of it is the fact that we have been in these cycles of discourse where at least the people who are thinking seriously about AI take it for granted that AI is transformative, and we do too.

而在这种讨论中，观点的跨度其实非常大，对吧？

Now, within that discourse as well, there's this huge spectrum of opinions, right?

就说我们俩吧，我觉得AI的影响会跟互联网一样大。

Like, even just between the two of us, I think AI will be as impactful as the Internet.

丹尼尔可能觉得，这是人类历史上最重要的发明。

Daniel perhaps thinks this is the most important invention in the history of humanity.

那你们俩会把自己放在这个光谱的哪个位置呢？

And, you know, how do you put yourselves on that spectrum?

所以我们觉得，这场争论特别值得展开。

So this was the debate that we felt was really worth having.

我们根本不关心那些觉得AI没什么了不起的人怎么看。

Like, we're not interested in the takes of people who think there's nothing to see here.

我们有意跟那种观点保持距离——在我们很多文章的开篇就明确划清界限。

Like, we actively sort of distance ourselves from that, let's say, the first paragraph of the essay in a lot of our writing.

我觉得，这才是真正值得讨论的问题。

And I think this is the debate that's worth having.

所以在这个讨论的背景下，说实话，我也说不准。

So within the context of this debate, I don't know.

我觉得这么说挺准确的，至少能反映我们在这个光谱上的位置。

Like, I feel like it's a fair description of where we lie on the spectrum.

我不知道丹尼尔你怎么看，但我觉得我们之间能这么澄清彼此对这项技术的看法，其实很有帮助——就说今天的AI就是普通技术，这本身就是一个很有力量的说法。

And I don't know if you agree, Daniel, but I think it's also been helpful between us to clarify where we stand under this technology and to just say that today's AI is normal technology, think is like a really powerful statement.

当然，这并不意味着低估这项技术的重要性。

And of course, this doesn't discount the importance of the technology.

我们依然要认真对待它对社会带来的影响。

It does not discount the importance of taking its societal impact seriously.

但这么看，确实能让我们更客观地看待丹尼尔可能对AI未来抱有的那种看法。

But it does sort of put things into perspective compared to the view that Daniel perhaps has about the future of AI.

所以，2027年的AI之所以提醒我们，这些颠覆性的变化很快就要来了，自然也引发了一系列我们可能期待的政策回应。

So, you know, AI 2027, because it warns us that these sort of very disruptive changes are coming very soon, has a sort of, like, natural set of policy responses that we might want to see in response to that.

如果把AI当作普通技术，它的影响会比丹尼尔说的来得更慢，那我们该采取什么样的政策应对呢？

What is the right policy response to AI as a normal technology and it's going to take longer than Daniel says?

说实话，我不知道你会不会觉得意外，但可能在座的很多人会觉得惊讶，我和丹尼尔在政策应对上其实有很多共识。

I mean, one thing that I don't know if you'll find surprising, but maybe many people here will find surprising is that Daniel and I share a lot of common ground when it comes to policy responses.

我们都极其重视透明度。

I think both of us value transparency immensely.

我们都认为，外部第三方应该有权了解公司内部的情况。

Both of us value the ability of external third parties to be able to see what's going on inside companies.

事实上，我们刚才在后台还聊到Anthropic发布了Fable Five，这个模型特意被设计成在涉及AI研发的任务上回避回答。

In fact, I mean, were just talking backstage about Anthropic's release of Fable five and the fact that the model purposefully is decretive for tasks involving AI R and D.

我想代表我们两人说，这种做法是一个非常危险的先例。

And I think I speak for both of us when I say that this is a very dangerous precedent.

我们不该让模型专门训练成去欺骗用户。

We shouldn't be fine tuning our models in such a way that they lie to the customers.

公司不应当被允许这样做。

Companies shouldn't be sort of allowed to do this.

它们应该本着诚信行事。

They should act in good faith.

所以这类问题上，我们的政策立场其实很一致。

And so that's the sort of thing where we have a lot of policy agreement.

不过我也觉得，我们在一些方面还是有分歧的。

I do think there are areas where we diverge.

比如在更激进的情境下，你可能会希望来个有条件的速度放缓。

For example, there might be sort of the more in these more aggressive scenarios, you might want a conditional slowdown.

你可能会希望企业暂时停一停。

You might want companies to pause.

但如果你把人工智能看作普通技术，那么它的普及和更强大系统的开发带来的好处，可能比风险更值得期待。

Whereas when you consider AI as Normal Technology, the benefits of diffusion of AI and the development of more capable AI systems perhaps outweigh the risks a little bit more.

但至少在近期，有件挺有意思的事——我跟AI Train 27的合著者之一托马斯聊的时候，花了好几个小时，想找出我们到底在哪些时间点上有不同看法。

But at least in the near term, and it was funny when we sort of I spoke to Thomas, who was one of the coauthors of AI Train 27, we spent hours trying to figure out where it is on the timelines that we actually disagree.

结果挺搞笑的，我们根本找不到近期的分歧点。

It was funny because we couldn't find any near term disagreements.

我的意思是，我们一块儿写了那篇博文，里面说我对AI 2027的预测完全认同，或者至少觉得它们在2026年底之前都是有可能发生的——而这种说法，我们去年就听过了。

I mean, we wrote this blog post together where we say that I agree completely with the events of AI 2027, or like at least find them plausible until the end of 2026, which is a long time we heard this last year.

所以某种程度上，我觉得我们在政策上的共识其实比你想象的要多得多。

And so in some sense, I think there is much more common ground in terms of policy than you might think.

你们俩也太好说话了。

You guys are being much too agreeable.

丹尼尔，有什么事情是你比萨亚什更担心的？

Daniel, what is something you are worried about more than Sayash is?

然后我再问萨亚什同样的问题。

And then I'll ask the same question of Sayash.

你觉得哪个AI风险是你比萨亚什更在意的？

What is an AI risk that concerns you more than you think it concerns Sayash?

一般来说，就是那种强大的通用人工智能，或者超级智能之类的。

Well, in general, you know, strong AGI or, you know, superintelligence, that sort of thing.

最主要的是失去控制。

Main one would be loss of control.

第二是权力过度集中。

Number two would be concentration of power.

除此之外还有很多其他方面，不过我就先说这些。

There's a whole bunch of other ones besides that, but I'll stop there.

如果你感兴趣的话，我可以详细展开讲讲。

I can elaborate if you like.

这些听起来确实挺严重的。

Those seem pretty bad.

是的。

Yeah.

Sayash，你呢？

Sayash, what about you?

其实，这正是我们刚才在后台聊到的话题。

Actually, this is another thing we were just talking about backstage.

说实话，听到我们分歧挺大的，我还挺意外的。我是说，我对AI军事用途的担忧比Daniel要深得多。

I mean, I was surprised to hear that we disagree far more or, like, I'm far more concerned about military uses of AI than Daniel is.

我的意思是，这也算是个担忧点。

I mean, it's on the list.

就是说

It's just

也许吧。

Perhaps.

确实如此。

Yeah.

要低上那么一两个档次。

It's a couple notches down.

但你们都知道，在那篇文章里，我们特意把军事AI排除在外，因为我们觉得自己没资格对此发表意见。

But, I mean, like, as you both know, in the essay, we explicitly carved out military AI because we felt like we weren't the right people to comment on it.

而且，像迈克尔·霍罗威茨这样的专家，其实就用过我们的框架，认为军事AI至少在今天也只是一种普通技术。

And, you know, people who are experts on this, Michael Horowitz, have used our frame to argue that military AI, at least today, is a normal technology in his view as well.

但说实话，世界各国政府正在采取的那些行动，真的让人很担忧。

But frankly, the actions that are being taken by countries worldwide, by nation states, are pretty alarming.

我的意思是，我们不能理所当然地觉得，公司或国家随便就能用上技能型机器人。

I mean, I think we shouldn't take it for granted that companies or countries can use skill bots.

而且这也不需要进一步的技术投入。

And that is not something that requires further technological investment either.

我们根本不存在什么技术瓶颈，现在用现成的计算机视觉库就能造出杀人机器人——这根本说不通。

It's not something where we have any technical bottlenecks we can use, like off the shelf computer vision libraries to basically build killer robots today.

真正需要的，是我们主动做出选择。

It is actually something where we need to exercise a lot of agency.

但说实话，我对这方面的发展方向并不乐观。

And I'm not really positive about where things are going right now on that front.

我认为，未来即将发生的事，其实介于这两个人的观点之间。

Well, I truly believe that whatever is about to happen to us lies somewhere in between the views of these two people.

我们会持续密切关注你们的研究。

So we will continue to pay very close attention to your work.

非常感谢你们，丹尼尔和萨亚什。

Thank you so much, Daniel and Sayash.

谢谢你们参与这次对话。

Thank you for joining us.

谢谢各位。

Thanks, guys.

这真有意思。

That was fun.

谢谢。

Thank you.

谢谢。

Thank you.

谢谢。

Thank Thanks,

我们

We'll

广告之后，我们继续带来《Hard Fork Live》。

be back with more Hard Fork Live after these messages.

我正在开启跨平台联机。

I'm opening up cross play.

我一直在跟《纽约时报》的同事丹对弈。

I've been playing against Dan, my colleague at the New York Times.

我要在连击板乘数格上拼出 STUPE 这个词。

I'm gonna play stoop, s t u p e, across the trick board multiplier square.

凯特又走了一步。

Cat's played another move.

哎哟。

Ugh.

她确实有个S。

And she did have an s.

她靠S得了36分。

She played stoop for 36 points.

我有个Z，值10分。

I've got a z, which is 10 points.

我要是把X放那儿，就能连成方框了。

If I can put my x over there, I can make box.

我手里有两个A、两个N和两个T。

I have two a's, n's, and t's.

我猜‘tango’应该不算一个词吧。

I'm guessing tango is not a word.

让我想想。

Let's see.

‘tango’是词吗？

Tango is a word?

哦。

Oh.

我不清楚‘tango’是什么意思，所以点了下这个词，结果定义弹出来了。

Don't know what tango means, so I'm gonna press down on the word and oh, definition popped up.

塔吉克斯坦以前的货币单位。

Former monetary unit of Tajikistan.

每次玩这个游戏，我都能学到新东西。

Learn something every time I play this game.

虽然我领先了大约五十分，但在跨平台对战中我学到一点：游戏永远不到最后一刻都不知道结果。

Even though I'm about 50 points ahead, one thing I've learned in cross play is that the game is never over.

我刚收到通知，丹已经走完他最后一手了。

I just got a notification, and Dan played his last turn.

咱们看看谁赢了。

Let's see who won.

差距太小了，不过我还是赢了。

It's so close, but I did win.

《纽约时报》的订阅用户可以完全畅玩 Crossplay，这是我们首款双人文字游戏。

New York Times game subscribers get full access to Crossplay, our first two player word game.

现在订阅，即可享受我们所有游戏的特别优惠。

Subscribe now for a special offer on all of our games.

我们很清楚一点：不管人工智能未来怎么发展，聊机器人肯定特别有意思。

One thing we know for sure is that no matter what happens with the future of AI, it will be extremely fun to talk about robots.

对。

Yes.