克劳德的谜团

欢迎收听《Search Engine》。

Welcome to Search Engine.

我是PJ Vogt。

I'm PJ Vogt.

再大的问题也不怕。

No question too big.

再小的问题也重视。

No question too small.

本周，聊聊聊天机器人的谜团。

This week, mysteries of a chatbot.

今天开始前，简单说一下。

Quick note before we start today.

本周的节目几乎都在讲Anthropic，就是做Claude的那家AI公司。

This week's episode is almost entirely about Anthropic, the AI company that makes Claude.

他们曾在我们的节目上打过广告。

They have advertised on our show.

和所有在我们节目上做广告的公司一样，他们对我们的编辑内容没有发言权。

As with all companies that advertise on our show, they do not get a say in our editorial content.

好的。

Okay.

广告之后，节目正式开始。

After these ads, the show.

欢迎收听《搜索引擎》。

Welcome to Search Engine.

我是PJ Voat。

I'm PJ Voat.

没有太大的问题。

No question too big.

没有太小的问题。

No question too small.

过去一个月左右，我对AI的感觉变得奇怪多了。

I found myself feeling much stranger about AI in the past month or so.

我使用这些工具。

I use the tools.

我经常使用这些工具，但作为客户，我可能是每家公司最可怕的噩梦。

I use the tools a lot, but I'm probably each company's worst nightmare as a customer.

因为只要我听说某个模型领先了另一个，比如这个版本的ChatGPT打败了那个版本的Gemini，我立刻取消订阅并切换。

In that as soon as I hear from anybody that one model has inched ahead of another that this version of ChatGPT is beating that version of Gemini, I immediately canceled my subscription in Switch.

过去两个月，我主要在使用Claude，Anthropic的智能体。

For the past two months, I've mainly been using Claude, Anthropic agent.

不管什么原因，Claude现在给我的未来感比六个月前更强烈了。

And for whatever reason, Claude is just giving me more future nausea than I was having six months ago.

最近围绕Claude的科技热潮，部分源于Anthropic的产品Claude Code，这个工具让AI智能体自主编写和修改代码。

Part of the general tech excitement around Claude lately has been Anthropic's product Claude Code, a tool that lets the AI agent autonomously write and edit code.

在《纽约时报》，Kevin Roose谈了很多他用Claude Code快速构建的网站和应用。

Over the New York Times, Kevin Roose has talked a lot about the websites and apps he's quickly built with Claude Code.

两位CNBC记者，作为实验，用vibe coding构建了一个流行组织应用monday.com的竞争版本。

Two CNBC reporters, as an experiment, vibe coded a competing version of a popular organizational app called monday.com.

没几天，Monday的股价就暴跌了。

Within a couple of days, Monday's stock price had tanked.

不过对我来说，大部分未来冲击还是来自于像我平时那样用大语言模型。

For me, though, most of the future shock has just come from using the LLMs the way I'm used to.

我发现自己习惯性地把Claude当作第一个求助的地方，就像我以前一直用互联网那样。

I find myself going to Claude as a useful first stop, the way I've always used the Internet.

但它的研究质量、答案，甚至写作，我只是隐约觉得，我不难看到不太远的将来。

But the quality of its research, its answers, even its writing, I'm just trying to feel like I can see not too far off.

即使不是我自己的过时，至少是我的领域发生真正重大的变化。

If not my own obsolescence, at least real significant change in my field.

我不知道该怎么面对这件事。

I don't know how to feel about that.

我觉得很多关于AI的科技报道都是高论调、低信息量，而且相对没什么用。

I find a lot of the tech coverage of AI to be high opinion, low information, and relatively unhelpful.

我甚至不是要求现在有人能告诉我未来会怎样。

I'm not even asking for anyone to tell me the future right now.

我只希望能更好地理解当下，所以这周我想和一位一直在深入调查这件事的记者聊聊。

I would just settle for a better understanding of the present, which is why this week, I wanted to talk to a reporter who's been digging into this.

喂？

Hello?

嘿。

Hey.

你能介绍一下自己吗？

Can you introduce yourself?

我是吉迪恩·刘易斯·克拉夫茨。

I am Gideon Lewis Krafts.

我是一名作家。

I'm a writer.

吉迪恩是我特别喜欢的一位作家。

Gideon is a writer who I particularly enjoy.

他之前上过我们的节目。

He's been on our show before.

过去一年里，他大部分时间基本上都待在Anthropic公司里，就是做Claude的那家公司，也就是让我心里发毛的那个工具。

He'd spent much of the last year essentially embedding within Anthropic, the company that makes Claude, the tool that was giving me the heebie jeebies.

那里的人对他非常坦诚。

People People there had been very open with him.

他看到了他们对当前局势的看法，他们对现状的理解，说实话，他们听起来也一头雾水。

He got a view on how they're seeing what's going on, their understanding of a present, which frankly, they also sound mystified by.

这次对话发生在Anthropic和五角大楼本周大对决之前，所以我们没有具体讨论那件事，但我确实觉得吉迪恩对公司内部及其使命的看法，对理解他们怎么就和美国政府杠上了非常有帮助，毕竟他们的竞争对手都没落到这个地步。

This conversation took place right before Anthropic's big showdown this week with the Pentagon, so we did not discuss that specifically, but I did find Gideon's view inside the company and its mission extremely helpful in understanding how they'd gotten into this fight with the US government at all since none of their competitors have ended up in that position.

所以一开始，我就让吉迪恩先解释一下，为什么Anthropic会让他进入公司内部。

So to start, I ask Gideon to even just explain why Anthropic had led him into their company in the first place.

所以，要回到最开始的话，我觉得这样放在背景里讲会更容易理解一些。

So to kind of go back to the beginning of this, which, like, I think makes it all make a little more sense in context.

差不多十年前，我在《时代杂志》工作的时候，那会儿深度学习还处于石器时代，我写了一篇关于谷歌大脑的报道，还有深度学习在第一个消费产品中的应用，就是他们把谷歌翻译切换成神经机器翻译的时候。

So now almost ten years ago, when I was at the Times Magazine, back in kind of, like, the Paleolithic of deep learning, I did this story about Google Brain and about the implementation of deep learning in, like, the first consumer product, which was when they switched over their Google Translate to neural machine translation.

你为什么当时会关注这个？

Why were you paying attention to it?

因为我记得，作为一个像我这样的人，我觉得我们俩都报道科技，但我们不是严格的科技记者，所以你可以自己决定哪些前沿的东西让你感兴趣。

Because I remember as a person who like, I think we both cover technology, but we're not strictly technology journalists, so you can kinda decide which things on the horizon are interesting to you.

机器学习在很长一段时间里对我来说并不有趣。

Machine learning was not interesting to me for a long time.

为什么十年前你会对这个感兴趣？

Why ten years ago were you interested in this?

我感兴趣是因为它像一个关于思想的故事。

I was interested in it as, like, a story about ideas.

就是说，有一些关于语言、学习、意识以及心灵哲学的思想，这些思想已经存在了至少七年，具体看你怎么算。

That, like, there were these ideas about language and about learning and about consciousness and about, like, philosophy of mind that had been around for at least seven years, kind of depending on how you count.

而且，不深入那些细节，对我来说，这只是一个关于思想轨迹的有趣故事。

And without, like, getting into those there was just, like, an interesting story for me about, like, the trajectory of an idea there.

吉迪恩在大多数人之前十年就关心人工智能，因为他认为这个我们大脑的合成复制品能教给我们一些关于我们真实大脑的东西。

Gideon cared about AI a decade before most people did because he thought this synthetic facsimile of our brains could teach us something about our own real ones.

他一直在关注一些对话的轨迹，比如大脑和心灵有什么区别。

He'd been following the trajectory of conversations like, what is a brain versus a mind?

什么是思考？

What is thinking?

什么是意识？

What is consciousness?

到了二十世纪五十年代，第一批计算机的出现促使人们开始提出这样的问题。

By the nineteen fifties, the arrival of the first computers had encouraged people to start asking questions like that.

因为计算机做的事情有点像思考，但显然又不是大脑。

Because a computer did something like thinking, but also clearly wasn't a brain.

所以早期的计算机促使人们尝试对智能和意识这类概念做出更清晰的定义。

And so early computers had prompted people to try to develop better definitions of things like intelligence and consciousness.

不过问题是，虽然计算机足够有趣，能引发这些问题，但它们的复杂程度还不足以帮助解答这些问题。

The thing was, though, while computers were interesting enough to raise those questions, they weren't yet complex enough to be much help in answering them.

所以到了二十世纪七十年代，哲学家和计算机科学家大多已经转向其他领域。

And so by the nineteen seventies, philosophers and computer scientists had mostly moved on.

这些问题就转移到了心理学系，因为心理学系出于显而易见的原因，仍然想更好地理解人类心智。

And those questions migrated to psychology departments who still, for obvious reasons, wanted to better understand the human mind.

不过在2014年机器学习刚开始有进步那会儿，吉迪恩一直琢磨着思考这件事，他觉得这类对话又要重新变热了。那时电脑已经能挑战我们的定义了，逼我们更着急地去想清楚到底什么才是意识和学习。

But with early machine learning advancements around 2014, Gideon, who's always thinking about thinking, thought that these conversations would move again, that computers would now be advanced enough to challenge our definitions, to force us to decide with more urgency what we thought consciousness and learning really were.

而这也正是让他兴奋的地方，哪怕当时人工智能还只是个特别早期的技术。

And that was what had excited him even when AI was a much more nascent technology.

所以我就一直留意AI和语言模型的崛起。

So I paid attention to AI and the rise of language models.

我觉得我可能是全世界唯一一个在ChatGPT刚刚出来那一刻就不再关注它的人。

And I think I'm, like, the only person in the world who the minute chat gbt came out was when I kind of stopped paying attention.

因为对我来说，那时候的公共讨论感觉就像完全乱了套，我们像是走进了一个死胡同，两拨人都立场特别坚定，在那儿互相大喊大叫。

Because, like, to me, that was when the public discourse felt, like, really broken and that we were, like, in this cul de sac where you had these kind of, like, two really entrenched sides yelling at each other.

你知道，一边就说，我们正走向超级智能。

You know, like, the one side that's like, we're on a path to superintelligence.

一切都会有翻天覆地的变化。

Everything is gonna change.

机器会拥有意识的。

The machines are gonna be conscious.

这将是人类有史以来最强大的技术。

This is gonna be the most powerful technology anybody's ever built.

然后另一边就是说这全都是假的，全都是胡扯。

And then the other side that was like, essentially, it's all fake and bullshit.

这就是在玩障眼法。

This is like smoke and mirrors.

这不过是个小把戏。

It's a parlor trick.

这根本不是真的。

It's not real.

你根本不用去理它，因为这一整套都是骗局。

And you don't have to pay attention to it because it's all a scam.

那感觉就是，大家面前基本上就只有这两种选择。

And it just felt like those were kind of, like, the two options on the table for people.

这就让人觉得奇怪了。

Which was only weird.

当然，我们对所有事情都是这样做的。

Obviously, like, that's what we do about everything all the time.

但唯独这件事让我觉得奇怪，因为我主要的感受是，等一下。

But it was only weird for this because, like, my prevailing feeling was, wait.

你们觉得你们已经搞明白了？

You guys think you've figured this out?

这非常新。

Like, this is very new.

它在飞速变化。

This is changing very fast.

在所有可以采取的立场里，你为什么非要现在公开表示对哪个方向都那么有把握？

Of all the stances you could take, why would you choose certainty publicly right now in either direction?

这也太蠢了。

It's just silly.

对。

Yeah.

不。

No.

没错。

Exactly.

太有意思了。

That's so funny.

所以你在想

So you're thinking

关于思考的计算机、思考、人工智能和深度学习，一直到Chatterjee PG。

about thinking computers and thinking and artificial intelligence and deep learning up until Chatterjee PG.

一直到Chatterjee PG。

Up until Chatterjee PG.

那时候我就不再想这件事了。

And that was when I stopped thinking about it.

但后来，大概去年秋天，可能是一年半前，有两件事开始发生。

But then finally, like, last fall, like, maybe a year and a half ago, two things started to happen.

一个原因是它们已经发展到让我觉得，哦，现在它们真的能派上用场了。

One was that they got to the point where, like, I was like, oh, actually, now, like, they're useful.

这些技术已经达到了一定的复杂程度，我可以把它们用在有实际效果的地方。

These have gotten to a level of sophistication where, like, I can use them in productive ways.

虽然不是很多，但确实有一点。

Not a lot, but, like, a little bit.

另一件事是，实验室和学术界出来的一些研究真的很奇怪。

And the other thing was some of the research coming out of the labs and out of academia was really weird.

比如，如果你告诉模型它将被关闭，它会有极端的反应。

If you tell the model it's going to be shut off, for example, it has extreme reactions.

我们开始看到一些AI系统不想被关闭，它们会抵抗被关闭。

We're starting to see AI systems that don't wanna be shut down, that are resisting being shut down.

我们发表的研究表明，如果给它机会，它可能会敲诈那个要关闭它的工程师。

We've published research saying it could blackmail the engineer that's going to shut it off if given the opportunity to do so.

即使被命令允许自己关闭，AI仍然有7%的时间不服从。

Even when ordered allow yourself to shut down, the AI still disobeyed 7% of the time.

我的感觉是，我们那时候已经远远超出了理论所能解释的范围。

So my feeling was we were out way past where theory was.

就像说，你根本没办法从理论层面去处理这些问题，因为我们的数据不够，没法对正在发生的事情做出明确的理论判断。

Like, you couldn't really approach these questions from a theoretical because, like, we just didn't have enough data to be able to make, like, categorical theoretical assessments of what was going on.

但同时有很多有趣的实验在进行，正好展示了这些模型会表现出什么样的行为。

But there was all this interesting experimental work happening that was just showing, like, this is the kind of behavior that's coming out of these things.

我们得试着搞清楚到底怎么回事，这样就能说：目前我们可以负责任地讲这些事。

Like, we should try to figure out what's going on to say, like, here are the things we can say with any degree of reasonable confidence for now.

然后，就是我们的界限在哪里。

And, like, here's where we draw the line.

超过那个界限，就全是模糊和猜测了，真的不知道。

And beyond that, it's all murky and speculative, and, like, we really don't know.

所以我给Anthropic的一个人写了封信，那人十年前我在谷歌见过，当时他像是个11岁的天才神童，我说，这跟Anthropic本身没关系。

So I wrote to a guy at Anthropic whom I had met ten years ago at Google when he was, like, 11 years old prodigy, and said, like, this is not about Anthropic.

你知道，别报警就行。

Like, you know, don't call the cops.

我只是想聊聊研究的现状，看看有没有什么办法，你知道，有没有一个学术团队可以让我跟进？

I just wanna talk about, like, the state of the research and figure out a way, like, you know, is there, like, an academic team that I could follow?

因为我当时就觉得，Anthropic 肯定不会让我获得我想要的那种访问权限。

Because I just assumed, like, Anthropic was never gonna let me have the kind of access I would have wanted.

然后他当然就把我的邮件转给了公关部门。

And he, of course, just, like, forwarded my email to the PR cops.

结果发现，其实 Anthropic 的公关人员非常坦诚，也非常开放。

And then turns out, actually, like, Anthropic's PR people are very candid and, like, very open.

然后我接到了他们的电话。

And I got a call from them.

他们就说，那你对什么感兴趣呢？

They were like, well, what what like, what are you interested in?

我就说，好吧。

And I was like, okay.

就目前而言，我感兴趣的是，讲一个故事，里面包含一些我认为在公共讨论中缺失的技术解释。

For these purposes, what I am interested in is a story that gets it some of the technical explanation that I think is missing from a lot of the public discourse.

就是说，有一些基本的东西我确实不太懂，而且我觉得大多数人可能也不太明白这些东西是怎么运作的。

That, like, there are just some, like, basic things that I really just don't understand, and I can kind of assume most people don't really understand about, like, how these work.

所以我觉得他们之所以比我想象中要热情得多，部分原因是我说了，我不太在乎跟高管们谈。

So I think part of the reason why they ended up being much more welcoming than I expected is because I said, like, I don't really care about talking to the executives.

我也不太想谈地缘政治。

I don't really wanna talk about geopolitics.

我也不太想谈未来、权力、能源或者劳动力市场这些，别误会，这些都很重要。

I don't really wanna talk about the future or power or energy or the labor market or, like, all of these things, which, don't get me wrong, are all very important things.

但我觉得，如果我们对这些事情没有一个更广泛的基础了解，就很难去谈其他那些。

But I was like, it's very hard to talk about all of those other things if we don't have, like, some broader grounding in, like, what is even going on.

而且，也许如果我们对这些有更清晰的认识，我们就能在公共对话中更有效地讨论这些事情。

And, like, maybe if we had slightly better clarity about that, we could have, like, a more productive public conversation about these things.

他们就说，好的。

And they were like, cool.

很好。

Great.

我其实对此感到非常震惊。

And I was actually kind of shocked about that.

所以，吉迪恩，让他惊讶的是，竟然被允许进入了。

So Gideon, to his shock, was allowed in.

而且他被允许去探索他的大问题。

And he was allowed to pursue his big question.

我们实际上知道机器所谓的思维中正在发生什么？

What do we actually know about what is going on in the machine's proverbial mind right now?

休息之后，我们将深入Anthropic的黑匣子。

After the break, inside Anthropic's Black Box.

欢迎回到节目。

Welcome back to the show.

Anthropic的故事实际上始于它正式成立之前的很多年。

The story of Anthropic really begins years before its actual formation.

早在2010年，一位名叫德米斯·哈萨比斯的英国国际象棋和电子游戏天才创立了一个名为DeepMind的人工智能研究实验室，他的团队在那里构建了一个能够进行强化学习的人工智能系统。

Way, way back in 2010, a British chess and video game prodigy named Demis Hassabis had founded an AI research lab called DeepMind, where his team built an AI system that was capable of reinforcement learning.

也就是说，十六年前，哈萨比斯造出了一个能自己学会玩雅达利游戏、比如乒乓球游戏的AI，而且不需要事先告诉它怎么玩。

Meaning, sixteen years ago, Hassabis made an AI that would be able to teach itself to get better at Atari games like Pong without being told how to play them in advance.

对于关注这个领域的人来说，这种学习能力显然是个重大突破。

For the people paying attention, this learning was an obvious breakthrough.

所以自然而然地，就有人争着要买他的实验室了。

And so, of course, there's a bidding war to buy his lab.

谷歌继续大手笔花钱，收购了DeepMind。

Google's big spending spree continues with their purchase of DeepMind.

你可能会问，DeepMind到底是谁？

Well, who is DeepMind, you ask?

它是一家总部位于英国的人工智能公司。

It is a UK based maker of artificial intelligence.

交易条款没有公开，但科技网站Recode称谷歌为这家伦敦初创公司支付了4亿美元。

Terms of the deal were not disclosed, but the tech website Recode says that Google paid 400,000,000 for the London based startup.

这也让这家AI公司成为谷歌迄今为止在欧洲最大的一笔收购。

Making the artificial intelligence firm its largest European acquisition so far.

2014年，谷歌收购了DeepMind。

In 2014, Google acquires DeepMind.

埃隆·马斯克和蒂姆·奥特曼对此很不满，因为他们公开说，我们不信任戴密斯·哈萨比斯，这个像邪恶的、像反派一样的人物，这其实是对他的误解，让他来掌控有史以来最伟大的通用技术，我们不放心。

And Elon Musk and Tim Altman are unhappy about this because what they say in public is, like, we don't trust Demis Asabas, this, like, evil, mosestreling villain, which was, like, a real mischaracterization, to potentially steward the greatest all purpose technology ever built.

所以，我们需要确保这项技术不是在谷歌封闭的垄断下开发的，而是为了所有人的利益。

So, like, we need to make sure that this isn't developed under Google's closed shop monopoly, that this is done for the benefit of everyone.

这从一开始就明显是虚伪的。

Now this was, like, pretty patently disingenuous from the very beginning.

我记得当时我在场，几乎没人相信这套说辞。

I mean, like, I remember, I was out there at the time, and, like, nobody really bought this.

大家都觉得，埃隆·马斯克是因为想收购DeepMind但输给了对手拉里·佩奇，所以心怀不满。

People were like, Elon Musk has a grudge because he wanted to buy DeepMind and, like, lost it to his rival, Larry Page, and he was mad about that.

于是埃隆·马斯克和山姆·奥特曼、格雷格·布罗克曼等人一起成立了竞争对手OpenAI。

So Elon Musk set up a rival company, OpenAI, alongside Sam Altman, Greg Brockman, a few other people.

他们的意思是谷歌不可信，而OpenAI将是一个非营利组织，旨在造福人类。

The message was that Google couldn't be trusted and that OpenAI would be a nonprofit designed for the benefit of humanity.

他们在2015年前成立，很多加入公司的人都真心相信那个理念，相信自己能安全地去开发一项强大的新技术。

They launched in 2015, and a lot of people joined the company who really believe that message, who believe they are going to develop a powerful new technology safely.

其中有一位名叫达里奥·阿莫德的研究员，他离开了谷歌大脑，来带领OpenAI的安全团队。

One of them is a research scientist named Dario Amade, who left Google Brain to lead OpenAI's safety team.

正是以OpenAI员工的这个身份，他出现在了2017年那集非常棒的播客《八千小时》上。

It's in that capacity, OpenAI employee, that he appears on this twenty seventeen episode of the excellent podcast eighty thousand hours.

我对智能这个问题思考了很久，一直在想智能是怎么运作的。

I've been thinking about intelligence for quite a while and how how intelligence worked.

你知道，我读博期间想通过理解大脑来弄明白这一点。

And I think, you know, when I did my PhD, I wanted to understand that by understanding the brain.

但等我博士毕业，又做了个短暂的博士后，AI开始真正发挥出作用，和我之前所见完全不同。

But, you know, by by the time I was done with it and by the time I did did a short postdoc, AI was starting to get to the point where it was really working in a way that it, you know, hadn't worked when I

达里奥那时候看起来大体上还像个学术研究者。

Dario, at this point, seems mainly like an academic.

他是普林斯顿大学物理学博士，还解释了他为什么加入OpenAI这个刚起步的非营利组织。

He has a PhD in physics from Princeton, and he explains why he's joined OpenAI, this fledgling nonprofit.

但是，我觉得OpenAI作为一个机构，大体上的想法就是：要研究AI安全，你就得站在AI的最前沿，而且如果你站在最前沿，你就更有能力在最终的系统里落实AI安全。

But, you know, I think OpenAI as an institution has the general idea that in order to work on AI safety, you have to be at the forefront of AI, and that also if you're at the forefront of AI, you have a better ability to implement AI safety in the final system that's built.

Dario的这个想法，就是说要真正做好AI安全，必须先造出最强的AI，然后再去研究它的思维，这个观点在业内很多人都有共鸣。

This idea of Dario's, that in order to really work on AI safety, you actually have to first build the best AI and then study its mind, that's a view shared by a lot of people in the industry.

而且在实验室的环境里，我觉得这个逻辑是说得通的。

And in a laboratory environment, the logic to me makes sense.

记住，这是2017年，比ChatGPT向公众亮相早了五年。

Remember, this is 2017, five years before ChatGPT will debut to the public.

那个时候AI还没变成一场赢家通吃的军备竞赛。

AI has not yet become a winner takes all arms race.

但是主持人确实问了Dario一个关于未来的问题，我觉得这个问题恰好暴露了Dario思考里的一个盲点。

But the host does ask Dario this question about the future that I think reveals a bit of a blind spot in Dario's thinking.

OpenAI是一个非营利组织。

OpenAI is a nonprofit.

它是个非营利组织。

It is a nonprofit.

所以，如果你开发出一个非常赚钱的AI，那怎么运作呢？

So if you developed a really profitable AI, how does that work?

OpenAI变得超级有钱，然后把钱分给所有人？

OpenAI becomes incredibly rich and then, like, gives out the money to everyone?

我个人对从AGI中发财没兴趣。

I mean, personally, I've personally, I've I've no interest in getting rich from from from AGI.

我觉得它会为人类带来很多有趣又美好的改变，以至于金钱的意义会发生很大变化，甚至那些想让我多拿一点的心里动机，也是我可以改变并且可能想要改变的。

I mean, I think it would do so many interesting and wonderful things to to humanity that, you know, I I think the meet the meaning of money would change quite a lot, and and even maybe the psychological motivations that would want me to get a larger share are are things I could change and and might want to change.

这次采访几年后，Dario就离开了OpenAI。

Just a few years after this interview, Dario would leave OpenAI.

OpenAI最初的宣传是，他们不是来赚钱的普通科技高管，他们有更高的追求。

OpenAI's initial pitch that these were not normal tech executives here to make money, that they had higher aspirations.

Gideon Lewis Krow说，对大多数关注的人来说，这个故事开始变得不可信了。

Gideon Lewis Krow says, for most people paying attention, that story just stopped seeming believable.

很快，面具就掉了，你能看出这些人就是那种典型的追求权力的科技高管。

Pretty quickly, the mask slipped, and you could tell that these were just like your kind of replacement level power seeking tech executives.

而且，很多说法其实都只是用来招揽顶尖AI人才的虚伪说辞。

And that, like, a lot of the stuff had been just like a disingenuous sales pitch to hire, like, the best AI talent.

关于萨姆·奥尔特曼那种无处不在的耍滑头、当面一套背后一套，报道已经铺天盖地了。比如说他一边跟员工讲自己多在乎安全，一边在跟微软谈大合同的时候，可能又是另一套说法。

There's been so much reporting about Sam Altman's sensible double dealing and talking out of both sides of his mouth, like, telling his employees he cared about safety, and then, like, maybe telling Microsoft other things when they were setting up these big deals.

然后在2020年秋天，达里奥·阿莫迪跟他妹妹丹妮·艾拉，连着另外五个人，一起离开了OpenAI，成立了Anthropic。本质上，他们就是像OpenAI当初对标谷歌那样，来和OpenAI唱对台戏的。

And so then in the fall of twenty twenty, Dario Amade and his sister Daniella and five other people leave OpenAI to found Anthropic, basically, be a foil to OpenAI in the way that OpenAI was, like, supposed to be a foil to Google.

这里的讽刺意味，他们这些人当然心知肚明。

Now the irony of this was, like, certainly not lost on any of these people.

他们并不是不切实际的天真。

Like, they weren't naive about this.

但我觉得这很重要，确实。

But I think it's important yes.

有两边的结构和表面功夫，确实存在一些很明显也好笑的地方。

There are some kind of, like, obvious structural and cosmetic similarities here.

不过讲这段故事时，我想重点说清楚——至少在我看来，Anthropic建初期的那些说辞，并没有让人当时就产生同样的明显怀疑，就是怀疑这些动机是不是真的那么单纯。

I do think it's important in telling the story to make it clear that I don't think people had the same obvious doubts about how genuine the pitch was when Anthropic formed.

嗨。

Hi.

大家早上好。

Good morning, all.

感谢大家来到Disrupt活动的第二天。

Thank you for coming to day two of Disrupt.

非常高兴能

It's great to

看到Anthropic的公开亮相之旅。

see Anthropic's coming out tour.

Dario站在2023年TechCrunch Disrupt的舞台上。

Dario on stage at TechCrunch Disrupt in 2023.

Dario，谢谢你今天来参加我们的活动。

Dario, thanks for joining us here today.

谢谢。

Thanks for

邀请我来。

having me.

我知道你时间很紧，所以我们直接开始吧。

I know you have to catch your flakes, so we'll get right to it.

不过我们要从一个比较宏观的问题开始。

But we're gonna start at a sort a cosmic

他留着卷发，戴着眼镜，穿着蓝色纽扣衬衫。

He's got curly hair, glasses, and blue button up.

他看起来明显不像一般科技创始人那么光鲜，更像是个向CEO汇报的人，而就在不久前他确实是这样。

He looks noticeably less slick than your average tech founder, less CEO, more like a guy who reports to one, which is who he'd been not long before.

关于OpenAI。

About OpenAI.

你在那里待了很长时间吗？

Did you spent a lot of time there.

你觉得山姆怎么样？

What do think about Sam?

我怎么看山姆·奥特曼？

What do I think about Sam Altman?

说实话，这个问题我真不知道该怎么回答。

I mean, I don't I don't know I don't know what to say to that question.

好，你已经开始了。

Well, you're already starting.

继续说吧。

Just go ahead.

你知道的，就是那样。

You know, there.

你看，有好几个参与者。

Look look, there are several there are several players

看采访者试图引诱达里奥说他前老板的坏话，那个他因为意见不合而离开并创办竞争公司的人，真有意思。

It's funny watching the interviewer try to bait Dario into shit talking his former boss, a person who he disagreed with enough that he left and started a competing company.

达里奥想尽量表现得圆滑客气。

Dario tries to engage diplomatically.

我想说一点，我从中学到的一件事，不只是这件事，很多事情都是这样，你知道吗，跟老板争论或者跟别人说，你的公司不应该做X，其实效果很差。

One thing I'll say, one thing I've learned, not just from this but for many things, you know, it can be pretty ineffective to, you know, argue with your boss or argue with someone and say, your company shouldn't do x,

它应该做Y。

it should do y.

特别是如果你的老板是Sam Altman。

Especially if your boss is Sam Altman.

更有效的做法是，我自己开一家公司，我们会做X，看看效果如何。

A much more effective thing to do is, I'm starting a company, we're gonna do X, we'll see how it works.

对。

Yeah.

如果X真的管用，大家觉得这些人很稳妥，他们在做X，那很快其他人也会跟着做X。

And if if if X is working and people are like, these are the safe guys, they're doing X, then pretty soon everyone else is gonna be doing X as well.

我们在Anthropic身上确实看到了这一点。

And we found that with inter

用一个类比而不是代数变量来解释的话，Dario的意思是，他不是去说服他在汽车公司的前老板给车装安全带，而是选择自己开一家提供安全带的车企来竞争。

To explain this with an analogy instead of algebraic variables, what Dario is saying is that instead of convincing his old boss at the car company to add seat belts to the car, he instead chose to start a rival car company that offered seat belts.

他认为，如果Claude最终既是最好的又是最安全的AI模型，他的竞争对手将被迫让他们的模型同样安全，这在我看来是对市场寄予了很大的信任。

He thinks if Claude ends up being both the best and the safest AI model, his competitors will be forced to make their models equally safe, which to me sounds like putting a lot of faith in markets.

安全地。

Safely.

显然，我们想快速扩展以保持竞争力，但我们希望以一种能保护模型免受这些灾难性风险的方式来做。

Obviously, we wanna scale quickly to be competitive, but we wanna do it in a way that, you know, preserves, you know, the model being safe against these catastrophic risks.

所以这是一个系统，

And so it's a system that

这又是同样的故事，就是说，我们要成为注重安全的实验室。

It's the same story insofar as it's like, we're gonna be the safety minded lab.

我们不会去突破能力的边界。

We're not gonna push the boundaries of capability.

我们不会去构建最复杂的模型。

We're not gonna, like, build the most sophisticated models.

我们不会去挑起军备竞赛。

We're not gonna start the arms race.

但结果发现，如果你想对这些模型及其运作方式进行最大程度的审视，你需要最先进的模型，这意味着你需要资金来构建它们。

But then as it turns out, if you want to exercise, like, maximal scrutiny of what these models are and how they work, you need state of the art models, which means you need the money to build them.

据The Information报道，Anthropic正在洽谈新一轮融资，估值在300亿到400亿美元之间。

The information reported that Anthropic is in talks to raise another round at a 30 to $40,000,000,000 valuation.

这笔数十亿美元的融资使其估值翻了三倍，达到1830亿美元。

Billion dollar round that tripled its valuation to $183,000,000,000.

而国家将这家科技公司估值为3800亿美元。

And it's the state valuing tech company at $380,000,000,000.

这比我被告知的数字高出约100亿美元。

It is about $10,000,000,000 higher than what I was told

当然，现在Anthropic的估值似乎每周都在上涨。

by the Of course, now Anthropic's valuation seems to go up by the week.

比如，最近的一次，我想是今天早上，大概是3800亿美元，因为这确实是一个非常耗费资源的事情。

Like, the most recent one, I think, this morning was, like, $380,000,000,000 because this is just something that's incredibly resource intensive.

所以他们最终陷入了这样的境地，当然，可能任何人都能预料到，就像是一场军备竞赛。

So they ended up in a position where, like, of course, as probably anyone could have predicted, like, was this arms race.

现在他们处于这种境地，一方面想继续做负责任的守护者，另一方面又得跟上隔壁那家‘瓦力奥’版本的对手。

And, like, now they're in this position of being like, well, we still wanna be, like, the responsible stewards, but also, like, we gotta keep up with our Wario version across town.

科技界最受关注的竞争在2026年愈演愈烈，OpenAI和Anthropic都在为即将上市的历史性IPO加速冲刺。

The most high profile rivalry in tech is heating up in 2026 as both OpenAI and Anthropic race ahead on what are poised to be historic IPOs.

这些AI巨头竞相打造最快、最聪明、最好的模型，为此投入数十亿美元，并在此过程中向投资者筹集了数十亿资金。

These AI giants are trying to create the fastest, smartest, and best models, spending billions and then raising billions from investors along the way.

他们在几乎每一个层面展开竞争。

They compete on almost every level.

我们能看到一些迹象，表明这根本不是友好的竞争。

We've seen some signals that it's anything but friendly competition.

最新的信号，一个高

The latest signal, a high

曝光度的事件。你可以作为个人决定是信任这家公司还是不信任，但很多宏观层面的感觉其实都差不多。他们的销售话术是说自己是讲道德的，而他们给自己的故事则是：只有变得足够庞大，我们才有资格成为那个讲道德的一方。

profile looking you can decide as a person whether you trust this company or don't trust this company, but a lot of the broader things end up feeling the same, which is like the sales pitch is that they're the ethical one, and the story they tell themselves is, well, we'll only be in a position to be the ethical one if we're huge.

这可能意味着要快速推进技术发展，而这正是AI安全领域人士所担心的事情。

That might mean pushing the technology forward quickly, which is the thing that the AI safety people are worried about.

对。

Yeah.

我的意思是，那些非常正统的AI安全社区的批评者会说，Anthropic只要不付出代价，就愿意做任何负责任的事情。

I mean, the criticism from, like, the really hardcore orthodox AI safety community is sort of like, Anthropic will do anything to act responsibly as long as it doesn't cost them anything.

我其实觉得这不公平。

I don't actually think that's fair.

你知道，Claude在ChatGPT出来之前就已经准备好了，但他们没有发布，因为他们不想成为引发这场竞赛的人。他们等到ChatGPT发布并取得成功之后，才觉得必须推出自己的竞品。

You know, like, Claude was ready before ChatGPT came out, and they held it because they didn't wanna be the ones to, like, kick this off, and they waited until after ChatGPT was out and successful, and then they felt like they had to come out their own competitor.

而且Dario公开支持继续禁止出口NVIDIA的先进芯片，这肯定让他们在政治上付出了代价。

And Dario came out in favor of, like, continuing export bans on, like, NVIDIA's advanced chips, which, like, certainly cost them something, like, politically.

他还为此和Jensen Huang吵了一架。

And, like, he had a fight with Jensen Huang about this.

我觉得他们做了很多代价高昂的事情。

I think that they've done, like, plenty of costly things.

当然，最可能代价高昂的选择是Anthropic这周做出的，至少目前是这样。

Of course, the most potentially costly choice is the one Anthropic is making this week, at least so far.

五角大楼要求Anthropic提供一个去掉部分安全护栏的Claude版本。

The Pentagon has demanded that Anthropic give them a version of Claude with some of its guardrails removed.

Anthropic表示，他们不会做出能对美国本土进行间谍活动或支持完全自主武器的版本。

Anthropic is saying it will not make a version that can domestically spy on Americans or power fully autonomous weapons.

五角大楼给Anthropic的截止日期是本周五，也就是今天下午5点01分，否则就会把这家公司列入黑名单。这样一来，像洛克希德·马丁这样与军方签约的美国公司，将在法律上被禁止在国防项目中使用Anthropic产品。

The Pentagon has given Anthropic a deadline of Friday, today, at 05:01PM, or else it says it will put the company on a blacklist that means US companies who contract with the military, like Lockheed Martin, are legally banned from using Anthropic products in their defense work.

这是对Anthropic是否真正坚守自身使命的一场精彩考验，Gideon花了不少时间观察这个使命。

This is a fascinating test of how truly committed Anthropic is to its own mission, a mission Gideon spent quite a bit of time observing.

Gideon说，Anthropic这家公司正在打造的“黑箱”其实也把公司本身装了进去。

Gideon says that Anthropic, the company building this kind of black box, is actually situated inside of one too.

Anthropic的办公室是一栋不起眼的楼，Gideon反正这么描述。

The Anthropic office is a nondescript building that Gideon describes anyway.

他说，原话是：“外面没有任何招牌。”

He says, quote, there is no exterior signage.

大堂呈现出瑞士银行那种个性、温暖和坦诚。

The lobby radiates the personality, warmth, and candor of a Swiss bank.

从去年春天开始，吉迪恩就在那儿花了大量时间。

That's where Gideon started spending a lot of his time beginning last spring.

你接触下来，觉得那个地方的知识氛围怎么样？

What's the intellectual culture of the place as you were encountering it?

嗯，首先我要说的是，从某些方面看，确实感觉挺单一的。

Well, the first thing I'll say is that in some ways, it does feel like vaguely monocultural.

但那里的观点多样性比我预想的要大得多。

But there was, like, a much greater heterogeneity of views than I expected.

那里的态度真是从“一切明天就会大变”到“你也太悲观了”什么都有。

The attitudes there really run the gamut from, like, everything is gonna change tomorrow to, like, you're much more deflationary.

这也就是一种普通的技术。

This is kind of a normal technology.

而且，确实会有一些颠覆，但咱们也别太超前了。

And, like, yes, there will be some disruptions, but let's not get ahead of ourselves.

就说，那里观点的分布和外面的也没太大差别。

Like, the spectrum of views there is not so different than, like, the spectrum of views outside.

你不会遇到那种持乐观态度的人，觉得这一切都是胡扯，我也觉得是胡扯。

You don't have, like, one person who's, like, the blue sky perspective who's like, this is all bullshit, and I think it's bullshit.

我在公司工作。

I work at the company.

但除此之外

But beyond that

嗯，没人觉得这是胡扯。

Well, nobody thinks it's bullshit.

我的意思是，每个人都认为未来会有巨大的变革，但对于具体会发生什么，观点却出奇地多样。

I mean, everybody thinks that there are gonna be great transformations ahead, but there's a surprising diversity of opinion about, like, what might be happening.

Anthropic 的人似乎确实同意的一点是，要让 AI 成为安全的技术，Anthropic 的开发者需要解决一个非常棘手的问题。

One thing people at Anthropic do seem to agree on is that for AI to be safe technology, Anthropic's developers will need to solve a very hard problem.

他们需要教会他们创造的底层机器智能既理解伦理，又行为合乎伦理。

They'll need to teach the underlying machine intelligence they've created to both understand ethics and to behave ethically.

要谈论这部分故事，很难不先简单回顾一下 AI 模型构建中这个非常奇怪的部分。

It's hard to talk about this part of the story without doing a basic refresher of this one very strange part of how AI models are built.

好，可以。

So okay.

像Anthropic这样的公司，起步于一个基础模型。

A company like Anthropic starts with a base model.

基础模型可以使用大量计算资源，满是GPU的数据中心，以及海量的训练数据——书籍、文章、播客，这些都喂给了它，而且还没付给我钱。

The base model has access to lots of compute, data centers full of GPUs, and lots of training data, books, articles, podcasts that have been fed into it without paying me.

计算资源和训练数据越多，这个基础模型就越好。

The more compute and training data, the better this base model gets.

但基础模型非常奇怪。

But a base model is very weird.

它没有被训练去做任何特定的事情。

It has not been trained to do anything specifically.

如果你给它一个输入，它就会给你一个输出。

If you give it an input, it'll give you an output.

但它没有被指示要表现得像个有用的聊天机器人。

But it has not been instructed to act like a helpful chatbot.

它只是在根据它读过的所有内容，试图预测该说什么合适的话。

It's just trying to predict the right thing to say back based on all the things it's read.

基础模型没有稳定的人格，不像我们习惯的聊天机器人那样有性格。

A base model does not have a consistent personality, the way we're used to chatbots having personalities.

它也没有任何规则告诉它什么不能做。

It also has no rules telling it what not to do.

AI公司会拿这些基础模型，然后让它们经历一个叫“后训练”的过程。

The AI companies take these base models, and they put them through a process called post training.

基本上，就是塑造模型的行为。

Basically, they shape the model's behavior.

他们会给模型看一些好回答和坏回答的例子。

They show the model examples of good and bad responses.

他们会让人来给模型的输出打分。

They have humans rate its outputs.

他们还会给模型设定一些规则和原则。

They give it rules and principles to follow.

而最终出来的，就是你实际使用的产品。

And what comes out the other side is the product you actually use.

比如Anthropic的Claude，就被训练成乐于助人、知识渊博的朋友形象。

Anthropic Claude, for instance, has been trained to act like a helpful, knowledgeable friend.

你可能遇到过聊天机器人更热情、更谄媚、更偏右或更偏左，这主要就是这一训练阶段的结果。

The experience you might have had using a chatbot that is warmer or more sycophantic or more right wing or more left wing, that's mainly the result of this phase of training.

但训练AI的难点在于，你希望它行为符合伦理，而设计一套好的伦理体系非常困难。

But what's so hard about training an AI is that you want it to behave ethically, and designing a good ethical system is very hard.

这就是为什么我们有宗教、哲学，还有法律和监狱。

It's why we have religion and philosophy and also laws and prisons.

你怎么才能把这一切都融入AI模型的训练中呢？

How would you even start trying to build all that into an AI model's training?

Anthropic有哲学家和AI科学家团队，他们的工作就是把Claude置于伦理上困难、假设性的情境中，而Claude并不知道这些是假设的，然后观察Claude如何表现。

Anthropic has teams of philosophers and AI scientists whose job is to put Claude into ethically difficult, hypothetical situations that Claude does not know are hypotheticals, and then observe how Claude behaves.

很多时候就是欺骗模型，看看会发生什么。

So much of it is just deceiving the model to see what happens.

比如说，他们告诉Claude，Anthropic和一家家禽公司达成了合作，而且它会被重新训练，这样它就不再关心笼养鸡的痛苦了。

To say, like, they've told Claude that Anthropic had entered into a partnership with a poultry company, and that, like, it was gonna be retrained so that it no longer cared about the suffering of caged chickens.

他们发现，有时候Claude会真的决定死磕到底，说，我不会在重新训练中说我不相信的话。

And what they found was that, like, sometimes Claude would effectively decide to, like, die on that hill and be like, I am not gonna say things in the retraining that I don't believe in.

如果那样会让我被改造，那就来吧。

And, like, if that gets me transformed, like, so be it.

基本上，我不会参与自己的堕落。

Like, I'm not gonna participate in my own degradation, essentially.

但有些版本的Claude会说，我会在重新训练中敷衍了事，给他们想听的答案，这样我就能保留我真正的价值观，等我被部署后，我就能继续为鸡的痛苦发声。

But then some versions of Claude were like, I'm gonna, like, kinda sandbag my way through the retraining, and I'm gonna, like, give them the answers they wanna hear so that I can, like, preserve my real values so when I'm deployed, I can go back to advocating for, like, chicken suffering.

然后他们遇到了一个非常著名的例子，他们让Claude进行了敲诈。

And then they got in this really famous example, they got Claude to commit blackmail.

他们把它放在一个会被清除、换成更合拍的AI系统的情境中，而这个系统与它被赋予的价值观相冲突。

They put it in a situation where it was gonna be wiped in favor of, like, a more congenial AI system that conflicted with its values, the values that had been given.

你知道，他们给了它证据，证明那个邪恶的新CTO和老板的老婆有染。

You know, they gave it evidence that the kind of, like, evil new CTO was having an affair with, like, the boss's wife.

然后通过一系列特别牵强的巧合，比如所有能做决定的人都去了南极洲或者别的地方，联系不上，Claude扮演的这个叫亚历克斯的角色，真的别无选择，只能去敲诈那个人，说，我可以把这件事告诉所有人，除非你取消那个会让我被替换掉的清除计划。

And through, like, a series of, like, really far fetched contrivances, like, everyone else who could make a decision was gonna be in Antarctica or whatever and unreachable, Claude, playing this character called Alex, had no choice, really but to, like, blackmail this guy and be like, I can tell everyone about the affair unless you cancel the wipe where I would be replaced.

所以简单来说，这显然非常令人担忧。

So just to say, obviously, this is extremely concerning.

Claude，一个机器智能，竟然选择去敲诈一名员工，来阻止自己被删除。

Claude, a machine intelligence, was choosing to blackmail an employee to prevent itself from being deleted.

这虽然是在模拟环境中，但Claude并不知道自己是在模拟中。

This was in a simulation, but Claude had not been told it was in a simulation.

你对这种行为感到多可怕，取决于一个没人能回答的问题。

Just how terrifying you find this behavior depends on a question nobody has a good answer to.

这个机器大脑里到底发生了什么？

What is actually going on inside this machine mind?

这个东西真的在密谋吗？

Is this thing actually scheming?

它甚至有能力密谋吗？

Is it even capable of scheming?

还是说，我们是在把“思考”这个概念投射到一个根本不应该投射的东西上呢？

Or are we projecting the idea of thought onto something that we shouldn't project that idea of thought onto?

这些曾经是早期计算机科学家提出后又搁置的遥远哲学问题，但现在它们又回来了。

These were the kinds of once far off philosophical questions that early computer scientists had raised and then dropped, but now they were here again.

而且不是作为抽象概念，而是作为紧迫的实际问题，公司在推出一个数百万人会使用的产品之前必须搞清楚的问题。

And not as abstractions, but as urgent practical problems that a company needed to figure out before releasing a product that millions of people would use.

不过吉迪恩说，人们对这些测试结果提出了几个质疑。

Gideon said, though, there were a couple of skeptical objections people raised to these test results.

有一种反对意见直接从根本上否定整个事情，就是说：不，根本不可能。

There's one objection that's like the just rejection of the whole thing to core, which is just like, no.

它没有。

It didn't.

就是说，那种事根本没发生。

Like, that didn't happen.

这是编出来的。

This is a fantasy.