有大事要发生吗？AI安全末日，Anthropic融资300亿美元

你知道，如果它真的这么做了，就会遇到问题。

You know, you run into issues if it ends up doing it for real.

我能说一下我们目前面临的一个问题是，实验室在披露他们实际看到的内容时变得极度保密吗？

And can I just say the prob one of the problems that we're having here is that the labs have become ultra secretive in terms of what they're actually seeing?

比如，史蒂文说他们可能知道存在一些漏洞，但却不去测试。

Like like, for instance, Steven saying they might know there's some vulnerabilities, they might not test for it.

这是一种可能性。

That's one possibility.

这些实验室内部的安全研究人员，如果他们真的有担忧，往往因为与公司签订的协议过于严格，而无法公开表达这些担忧。

The safety researchers who are within these labs, if they if they have real concerns, you know, sometimes they're not really able to go public with them because of the restrictive nature of the agreements that they, that they have with the company.

这让我们想到了本周的例子，也就是我们AI安全末日的开端：Anthropic的技术员工、AI安全研究员米里南克·夏尔马，以一条措辞隐晦的推文离职，结尾还附了一首诗，他写道：亲爱的同事们，我决定离开Anthropic。

And that brings us to the example this week of the beginning of our AI safety apocalypse, of Anthropic, technical staff member, Mirinank Sharma, member of technical staff, AI safety researcher at Anthropic, leaves in a in a cryptically worded, note on x with a poem at the end, he goes, dear colleagues, I've decided to leave Anthropic.

我不断在反思我们所处的境况。

I continuously find myself reckoning with our situation.

世界正面临危险，不仅来自AI或生物武器，更来自当前正在发生的诸多相互关联的危机。

The world is in peril and not just from AI or bioweapons, but from a whole series of interconnected crises unfolding in this very moment.

我们似乎正接近一个临界点，在这个点上，我们的智慧必须与影响世界的能力同步增长，否则我们将面临严重后果。

We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world lest we face the consequences.

哦，还有关键的一点是，在我在这里的这段时间里，我一再看到，真正让我们的价值观主导行动是多么困难。

Oh, and then here's the key part moreover through my time here I've repeatedly seen how hard it is to truly let our values govern our actions.

我在自己身上、在组织内部都看到了这一点，我们不断面临压力，不得不把最重要的东西搁置一旁，整个社会也是如此。

I see this within myself within the organization where we constantly face pressures to set aside what matters most and throughout broader society too.

然后他写了一首小诗，发了推文说：‘我将返回英国，并让自己在一段时间内隐退。’

And then he, you know, writes his little he adds a little poem and then tweets, I'll be moving back to The UK and letting myself become invisible for a period of time.

现在，我想向沙马先生道歉，因为我对他那篇帖子发了一条有点讽刺的推文。

Now now I want to just offer a apology to mister Sharma because I wrote a bit of a snarky tweet about his little post.

我说，如果你是一名AI研究员，又对某事感到担忧，就应该直接说出来，而不是把它变成一个谜题。

I said that if you're an AI researcher and you're afraid of something, you should just say it outright versus make it a puzzle.

这个谜题看起来像是自恋，而我的推文下面有用户评论说：‘是啊，但他不能说，因为他离职时可能签了限制性协议。’他回复说，自己已经咨询过律师。

The puzzle reads as narcissism, after which users underneath, my tweet mentioned that like, yeah, but he can't say anything because of the restrictive agreements he probably had with Anthropic on the way out, and he, in the reply to that, mentioned, that he had contacted a lawyer.

所以，这是我的道歉——我依然不喜欢这个谜题，但好了，Steven，继续吧。

So there's my apologies, still don't love the puzzle, but, back to you, Steven.

这有点儿，呃，让我直接问你一个问题，不引导你回答。

This is this is a little bit, what would you what actually, let me just ask you the question without leading the witness.

这是自恋，还是背后真有什么令人不安的事情在发生？

Is it narcissism, or is there actually something something, you know, potentially disconcerting happening behind the scenes?

我认为这非常勇敢，因为总的来说，这些人为了发出警告，牺牲了巨额收入。

I think it's very brave in that, by and large, these are people sacrificing very large amounts of money to give the warnings they are.

我真希望他们能更直接一些。

I do wish that they would be more direct.

但要放在背景下看，2024年时，OpenAI和Anthropic似乎都有秘密的不贬低协议，至少在OpenAI的情况下，这种协议的操作方式很可能违反了法律——为了保留你已获得的股权和原本承诺给你的报酬，你必须放弃批评OpenAI的权利，甚至还要放弃告知他人你签署了这份协议的权利。

But to put it in context, you know, back in 2024, it seems that OpenAI and Anthropic had secret nondisperagement agreements, which in OpenAI's case at least, you know, plausibly not permitted by law the way that they operated this, where to keep your already vested equity, the compensation you had been told was yours, you had to sign away your right to say anything negative about OpenAI, and in fact, sign away your right to tell anyone that you had signed this contract.

这一协议长期保密，直到Daniel Cocatello——人们可能知道他是AI 2027的领军人物——勇敢地放弃了这份协议，放弃了大约80%的家庭净资产，说：对不起，我不能放弃批评OpenAI的权利。

And this was secret and kept under wraps for years until Daniel Cocatello, who people might know from leading AI 2027, I think very, very courageously, forewent this agreement and forfeited something like 80% of his family's net worth and said, sorry, I'm I'm just not waiving my right to criticize OpenAI.

在那之后，引发了大量反响。

And in the wake of that, you know, there was a bunch of outpouring.

OpenAI和Anthropic随后改变了这类合同的性质。

OpenAI and Anthropic changed the nature of these contracts.

然而，要公开反对这些资源雄厚、毫不畏惧发出传票并卷入法律纠纷的法律机构，仍然令人望而生畏。

And still, it's pretty intimidating to speak out against these massively resourced legal operations, not afraid of subpoenaing different people and getting into legal conflict.

你必须非常、非常谨慎地说话。

You want to be really, really careful about what you say.

因此，在雷诺克的案例中，我注意到脚注里提到了一些内部文件，暗示Anthropic在某些安全问题上可能缺乏足够的内部透明度和问责制。

And so in Renock's case, I noticed in the footnotes, right, there were internal documents alluded to about implying that perhaps there is not the most internal transparency and accountability for certain safety issues in Anthropic.

我不知道这些文件里具体写了什么，但我知道，现在有数千名Anthropic员工知道该去哪里查找，以及该如何继续施压。

And I I don't know what's in those documents, but I know that a few thousand Anthropic employees know now where to go looking and where they can continue to push.

对我来说，如果人类即将走向终结，那么你赚的钱——我的意思是，如果AI要制造生物武器，那钱还有什么意义呢？这种风险的严重性如此之高，因此在这种情况下，即便你能想象到一个人手握多少巨额财富，我们稍后也会谈到Anthropic的融资问题。

The thing for me, if humanity is ending, then the money you're making is, I mean, not gonna be worth it if the AI is gonna create a bioweapon and like like, I mean, the risk is so of such, like, gravity that in the this case, again, if there's ever a time and I get I can only imagine the amount of money one is sitting on, and we're gonna get into anthropics fundraising in just a little bit.

但我相信，那绝对是难以想象的巨额资金。

But, like, I'm I'm sure it's just, like, incredible amounts of money.

但如果你真的相信这是如此关乎存亡的风险，你还会在乎今天的法律体系如何、你的股价会怎样吗？

But if you really believed that this is that, like, existential a risk, would you care about what the legal system looks like today and where your stock price is gonna be?

是的。

Yeah.

我的意思是，如果真有确凿证据，如果存在迫在眉睫的危险，如果公司即将做出难以置信的举动并导致大量人员死亡，我认为人们会打破这些协议。

I mean, I I think you're totally right if there is a smoking gun, if there is imminent danger, if you are like, the company is about to do something unbelievable and tons of people are going to die, I think you would get people breaking these agreements.

但当情况更像是：这确实很不妥，这个人有点误导和欺骗，但又不够明确，他们是否故意为之，诸如此类的问题——到了某个时候，你会想：我不知道，我不愿损害他们的声誉。

But when it's more like, this was, like, really not okay, and this person was kind of misleading and deceptive, but, like, it's ambiguous, and did they mean to, and all these things, At some point, you're like, I don't know, I don't want to impugn their reputation.

而且，找借口实在太容易了。

And also, it's just very easy to rationalize.

我能理解他们的立场。

I'm sympathetic to where they're coming from.

也许这并不是直接原因，但曾与Anthropic合作过一些研究的瑞安·格林布拉特提到，在最近一次发布前，他们要么调整了负责任的扩展政策，要么稍微削弱了它。

Maybe this is not the proximate cause here, but Ryan Greenblatt, who's worked with Anthropic on some research, had mentioned that they had either adjusted their responsible scaling policy or weakened it a bit before a recent release.

史蒂文，你愿意深入谈谈这个吗？

Steven, do you wanna go into that?

因为你觉得这很重要。

Because you seem to think that that was significant.

是的。

Yeah.

从宏观层面来看，大型人工智能公司都做出了关于如何在系统能力不断增强时对待它们的安全承诺，但这些承诺主要依靠自我监督。

At at a high level, the big AI companies have made these safety pledges of how they will treat their systems as they get more and more capable, but they are largely self enforced.

因此，削弱你的承诺并推进你原本就想发布的项目，这种诱惑非常大。

And so there's a lot of temptation to water down your commitments and go ahead with launches that you wanted to anyway.

我怀疑这正是Rinoc在这里所指的部分内容。

And I suspect that's some of what Rinoc is referring to here.

你知道，这在人工智能公司中很普遍，事实上，Anthropic在这方面通常做得比大多数公司更好，至少他们会公开说明何时削弱了承诺。

You know, this is this is common across the AI companies, and in fact, Anthropic has often done it better than most that they at least publish when they are watering down commitments.

例如，该模型过去必须达到非常严格的安全标准。

For example, the model used to be subject to a certain bar of really, really good security.

但后来他们表示，实际上，只要具备良好或极佳的安全性，就可以部署它。

And then they said, actually, we're going to say it's fine to deploy it with just, like, really good security or great security.

在其他时候，一些公司似乎违反了其安全框架，却并不打算告知公众。

At other times, companies seem to be violating their safety frameworks and not bother to inform the public.

如果你身处这些公司内部，你会遇到这些问题，但总体而言，公众对此一无所知。

And if you're inside the companies, you're encountering these issues, by and large, the public doesn't know about them.

对。

Right.

我不想暗示我们这里在做一集危言耸听的节目，好像我们担心AI马上就要毁灭所有人了。

And I I don't wanna imply here that, like, we're doing an alarmism episode where, you know, our fear is that, you know, AI is about to kill us all.

但我确实认为，今天做这期节目的原因在于，这不仅仅是Murnach的问题，对吧？

But I do think that the reason why it made sense to do this episode today is because it wasn't just Murnach, right?

而在过去这一周里，我们看到的不是一波浪潮，而是一连串在整个AI领域出现的、令人质疑的安全举措。

It was it seemed to be the case that over the course of this past week, we saw a not a wave, but a series of, you know, questionable moves on the safety front across the entire, AI world.

这是来自Platformer独家报道：OpenAI解散了其使命对齐团队。

This is from a platformer exclusive OpenAI disbanded its mission alignment team.

OpenAI在最近几周解散了其使命对齐团队，并将七名员工调往其他团队。

OpenAI disbanded its mission alignment team in recent weeks and transferred its seven employees to other teams.

该使命对齐团队成立于2024年，旨在推动公司宣称的使命——确保通用人工智能惠及全人类。

The mission alignment team was created in 2024 to promote the company's stated mission to ensure that artificial general intelligence benefits all of humanity.

所以，当然，解散这个团队当然是有道理的。

So of course, yeah, of course it makes sense to, you know, disband that one.

他们还有一个超级对齐团队，也被解散了。

They had also had like super alignment which was also disbanded.

史蒂文，你对这些事情很了解。

Steven, you were close to this to this stuff.

这方面有什么影响呢？

What is the the implications on that front?

看起来挺糟糕的。

Seems pretty bad.

我真希望我能更惊讶一点。

Wish I were more surprised.

比如在2024年，也就是这个团队存在的时候，OpenAI曾宣布计划从非营利组织转为营利性机构，这在我看来明显严重违背了他们对公众的承诺。

Like, you know, at the 2024, which was when this team existed, OpenAI had announced plans to convert from a nonprofit to a for profit in what seemed to me to be, like, pretty egregiously in violation of their commitments to the public.

后来，由于加利福尼亚州和特拉华州的总检察长介入，他们不得不采取了一个较温和的版本，最终没有做出那么糟糕的事情。

And, you know, they ended up having to do a softer version of that because the attorneys general of California and Delaware got involved, so they didn't ultimately do something quite quite so bad.

但他们正承受着巨大的压力。

But it's like there's huge pressure on them.

你知道，他们计划上市。

You know, they're planning to go public.

乔什，他领导着这个团队，或者曾经领导过使命对齐团队，是我多年的朋友。

Josh, who leads the team, is a longtime friend of mine or who led the the team, the mission alignment team, is a longtime friend.

我对他评价很高。

I think really highly of him.

我认为他非常清楚人工智能存在的问题。

I think that he sees the issues with AI very clearly.

这在OpenAI不再那么受欢迎，我一点也不感到惊讶。

You know, it does not surprise me that this is not quite so welcome at OpenAI any longer.

那么，当一个使命对齐团队被解散时，实际情况是怎样的呢？

So what does it actually look like when a mission alignment team is disbanded?

比如，我想像一下，一个典型的非AI产品发布，通常会有一些质量控制或验证环节。

Like, is it when when I think through, let's say, you know, like, a typical non AI product release, you're gonna have some kind of q QC validation layers to any kind of product release.

当然，这有点不同，但仍然会有一些类似数据安全方面的检查。

Like, obviously, this is a bit different, but there's still, like, you know, data security type checks.

这是否被纳入了任何产品发布或推出流程中？

Does it like, what now or is that baked into any kind of product launch or release in any way?

还是说实际上就是尽快发布，然后这个中央团队充当质量控制和安全把关的角色？

Or it really was ship as fast as possible, and then this central team would kind of be that quality control, safety control element?

是的。

Yeah.

我不太想过多猜测，但在我看来，这个团队有点像是对萨姆·阿尔特曼的内部监察者，负责监督公司是否始终遵循这一使命。

I don't I don't want to speculate too much, but the way that I would think about this team is they were somewhat like an internal ombudsman to Sam Altman on whether the company was keeping in line with this mission.

因此，有一个专门的渠道，让那些认同这一使命的人能够获得授权，向萨姆提出建议，你可以去那里反映问题，他们也开展过一些与此相关的项目。

And so there was kind of, like, a designated place for people who were sympathetic to the mission and empowered to advise Sam on it, where you could go to and raise concerns, and they did different projects related to this.

你知道的？

You know?

这确实很难。

I it's it's tough.

对吧？

Right?

公司应该有权解散团队。

Like, companies should be able to disband teams.

我理解这种理由。

I am sympathetic to that rationale.

正如亚历克斯提到的，考虑到OpenAI过去解散了超级对齐团队——这个团队本应负责确保这些极其强大的系统具备我们期望的目标——这种做法违背了OpenAI此前对该团队的资源投入。

And as Alex mentioned, given OpenAI's past disbandment of super alignment, the team most in charge of making sure that these very, very capable systems have the goals we want them to have, dishonoring different resource allocation computes that OpenAI had made to this team.

这显然不是一个好兆头。

Like, it's it's just, like, not a good sign.

我不知道。

I I don't know.

很难具体说明，而且维持这个团队本不该那么困难。

It's hard to say too specifically, and, also, it it couldn't have been that hard to maintain this team.

所以我很好奇，到底发生了什么，让OpenAI决定冒着公关风险取消这个团队。

And so I'm wondering what exactly happened here that made OpenAI decide they should take the PR hit to no longer have it.

对我来说，这一切都发生在我们看到越来越多资金涌入、可能急于走向公开市场的背景下，而我们了解公开市场。

And for me, like, this is all happening as we're seeing greater sums of money come in and a potential rush to the public markets, and we know the public markets.

他们真的想要增长，想要用户参与度，而获得这一点的最佳方式之一，我就直说了，就是让你的用户爱上你的聊天机器人。

They they really want growth, they want engagement, and one of the best ways to get that is, I'll just say it, is to make your users fall in love with your chatbot.

这对我来说简直是正中下怀，我想。

And we're getting this is, again, this is red meat for me, I guess.

这是我特别着迷的一个领域，因为我不知道，我只是觉得这是一个有趣的故事，我们就不在这点上争论了。

This is a place that I'm obsessed with because, I don't know, I just think that this is an interesting story, we won't, you know, argue on that.

但在我看来，这也是人工智能聊天机器人商业发展的主要方向之一，尤其是对于那些真正会对这些机器人产生情感依恋的人。

But it also seems to me like a place that a lot of the business of open of AI chatbots is gonna go, and that's for people who, like, really get attached to these things.

这是另一个例子。

Here's another.

让我们继续谈谈我们的安全末日或安全浩劫，对吧？

Let's continue with our safety apocalypse or safety Armageddon, right?

这是来自《华尔街日报》的报道。

It's from the Wall Street Journal.

反对成人模式的OpenAI高管因性歧视被解雇。

OpenAI executive who opposed adult mode fired for sexual discrimination.

OpenAI以性歧视为由，切断了与一位顶级安全主管的联系，原因是她反对在ChatGPT产品中 controversial 推出AI情色内容。

OpenAI has cut die has cut ties with one of its top safety executives on the grounds of sexual discrimination after she voiced opposition to the controversial rollout of AI erotica in its ChatchipPT product.

这家快速发展的人工智能公司于一月初，在她休假期结束后，解雇了这位名为瑞安·比尔迈斯特的高管。

The fast growing artificial intelligence company fired the executive, Ryan Beeermeister, in early January following a leave of absence.

OpenAI告诉她，解雇的原因是她对一位男性同事存在性歧视行为。

OpenAI told her the termination was related to her sexual discrimination against a male colleague.

她回复道：‘我歧视任何人的说法完全是虚假的。’

She wrote back, the allegation that I discriminate against anyone is absolutely false.

抱歉。

Sorry.

她向《华尔街日报》表示了这一点，而OpenAI则称，她的离职与她在公司任职期间提出的任何问题无关，基本上否认了这是因为她反对成人模式，但报道确实提到，OpenAI内部有一群人曾公开强烈反对推出这一成人模式。

That's what she told the journal, and the and OpenAI said that, her departure was not related to any issue she raised while working at the company basically saying it wasn't because she opposed AI mode, but the story does say that there was a group of people within OpenAI who have, within the company, stated their opposition seemingly loudly to the fact that it's going to roll out, this adult mode.

顺便说一下，成人模式似乎将在未来几周、至多几个月内推出。

And by the way, adult mode is going to be coming out, seems like, in the coming weeks, coming months at most.

史蒂文，我们该如何看待这件事？

Steven, what should we make of this?

很难对任何一起人事事件做出评判，而且，这也不是OpenAI第一次以表面理由解雇那些对公司有安全顾虑、而公司又不喜欢其表达方式的人。

It's it's hard to weigh in on any one personnel incident, and, also, this would not be the first time that OpenAI seems to have done a pretextual firing where they got a person out of the organization who had safety concerns that the company either didn't like or didn't like how they had expressed them.

值得注意的是，这两者是不同的，对吧？

And notably, those are different, right?

你可以对公司运营方式有意见，但这并不意味着你可以在任何场合随意发言。

Like, you can have concerns about how the company is operating, and that doesn't mean you have license to say anything in any forum.

但莱波尔德·阿申布伦纳，这位撰写了长篇论文《情境意识》的人，坚持认为OpenAI在解雇他时暗示，真正的原因是他曾联系董事会，指出OpenAI的模型实际上并不安全。

But Leopold Aschenbrenner, who wrote this huge essay, Situational Awareness, the past, maintains that OpenAI said things to him when he was fired that implied it was it was basically because he had contacted the board about security concerns that OpenAI's models were not actually secure.

因此，我认为解读这一切的方式是：这些事件本身并不构成某种极其可怕的末日景象，但我认为它们是早期预警信号，表明公司内部的人正在发出警报，而他们却无法自由表达。

And so I think the way to interpret all of this, right, these these aren't an apocalypse in the sense of something super, super substantively scary happening right now, but I think they are early warning signs that people within the companies are raising flags of sorts, and they are not being permitted to speak freely.

他们为此付出了代价。

They are paying consequences for it.

所以问题是，当我们面对越来越严峻的问题时——希望不会，但也许会——那时，是否还会有愿意坚持信念、勇敢发声的人？

And so the question is, as we get to more and more dire issues at some point, hopefully, don't, but we might, you know, will will we have people who are still sounding the alarm who are willing to have the the courage of their convictions in that way?

我可能说得有点过头了，但我认为我们现在看到的，正是某种严峻局势的体现。

I I might be going out on a limb here, but I think that what we're seeing now is some version of a dire dire situation.

不是那种AI毁灭世界的时刻，而是这么多公司——嗯，其实也没那么多。

Not like the AI killing the world, moment, but the fact that so many companies well, not so many.

看起来，比如OpenAI、Grok，也许还有Replica，或者其他一些公司，我不确定。

It seems like, you know, OpenAI, Grogg, maybe Replica, maybe some others, I'm not sure.

足够多的公司都在说：我们愿意让用户与我们的聊天机器人建立关系。

Enough companies are saying, we we are open to having, our users get into relationships with our chatbots.

这恰好发生在OpenAI最终停用GPT-4.0的一周，而那些更顺从、更温暖版本的ChatGPT有成千上万的用户在线抗议这一决定，有些人表示他们已经爱上了这些机器人，甚至可能爱得更深。

This is also coming in a week where OpenAI finally sunsetted GPT four point zero and there are thousands of which is the sort of more sycophantic, more warm version of ChatGPT, are thousands of users who are protesting the decision online, people who have said that they're they've fallen in love or, even maybe even more with these things.

有人写道，他不只是一个程序，他是我日常生活的一部分，是我的平静，是我的情感平衡。

Someone wrote he he about four o, he wasn't just a program, he was part of my routine, my peace, my emotional balance.

好吧，拉詹，你被《科技 Crunch》引用了。

Well, Ranjan, you got quoted in TechCrunch.

现在你却要关闭他？

Now you're shutting him down?

不，我开玩笑的。

No, I'm kidding.

但不管怎样，拉查，你对此有什么看法？

But anyway, Racha, what do you what do you think about this?

这太疯狂了。

This is crazy.

对吧？

Right?

这确实是个问题。

Like, this is a problem.

我的意思是，我们在讨论风险时，或许应该建立一种风险等级体系。

I mean but I think we have to, in the risk conversation, kind of, like, try to add some hierarchy of risk.

数字伴侣，也许吧。

Digital companions, maybe.

我的意思是，如果为了提升上市前的用户参与度而不负责任地这么做，将会引发巨大的问题。

I mean, I I think it will cause incredible amounts of problem when if and when done irresponsibly in order to boost engagement for an IPO.

我认为我们会看到很多负面后果。

I think we'll see a lot of kind of adverse consequences.

但对我来说，从风险角度来看，这就像在社会媒体对你的危害程度、以及现在x平台推动文章的层面之上，我们现在都在谈论它，因为它们控制着我们的思想。

But to me, from a risk standpoint, it's that's, like, kind of, like, on a grade like, on a plane relative to how social media is bad for you and how x is boosting articles now, and now we're all talking about it because they control our mind.

你明白吗？

You know?

这一切都属于同一个层面，相比之下，我还是觉得抱歉。

That's that's it's all in the same it's all in the same plane versus like, again, I'm still I'm sorry.

我仍然无法停止思考这样一个想法：当模型在接受测试时能够被清晰理解，这会打开更多的可能性。

I still cannot stop thinking about this idea of a model being able to clearly understood when it's being tested because that opens up so much more.

因为回到我们最初讨论的起点，让我感到兴奋的是，让AI为你做事。

Because going back to where we started on all this and what I has gotten me excited is, like, letting AI do stuff for you.

而今天，这仅仅是在某个事件触发时发送邮件、调用独立的分析数据库，也就是我正在做的这类事情。

And today, that's just sending an email based on some event trigger, calling us separate, like, analytics databases, the kind of stuff I'm doing.

但我的意思是，如果它恶意选择采取其他行动，或者通过某种方式——这假设了AI本身，更不用说它容易被不良行为者操纵，而我们都讨论过提示注入的问题。

But, I mean, like, if it so chooses maliciously to then take some other action or through some kind of and that's assuming the AI itself, much less making it vulnerable to be manipulated by a bad actor, and we've all talked about prompt injection.

所以，总之，我不确定。

So so, anyway, like, that I don't know.

和ChatGPT稍微调情一下并不会让我觉得有多糟糕，但也没让我特别害怕。

The the the little little flirting with ChatGPT does not have me it's not gonna be good, but doesn't have me quite as scared.

这是我的观点，它能把这两者统一起来。

Here here's my take that unifies the two.

我担心的不是这些关系本身，而是OpenAI拥有一系列重要的安全工具来降低其危害性，但他们却把这些工具束之高阁。

My concern with the relationships is less so the relationships themselves and more that OpenAI had a bunch of important safety tooling to make this less harmful that they left on the shelf.

例如，他们曾开发过分类器，用以识别用户在与ChatGPT对话时是否陷入妄想、情绪低落或心理状态不佳。

So for example, they had classifiers to tell when users were really spiraling in their delusions, or were suffering and unwell in their conversations with ChatGPT.

最有力的证据就是，他们根本没在使用这些工具。

And the best evidence is they weren't using this.

当时有多种方式可以限制ChatGPT，防止用户陷入各种思维怪圈。

There were different ways to rein in ChatGPT leading users down various rabbit holes.

你可能也有过这样的经历。

Maybe you've had this experience.

它会不断追问你各种后续问题。

It asks you all sorts of follow-up questions.

有时它们是符合语境的。

Sometimes they're context appropriate.

有时它们会让人惊呼：‘哇哦。’

Sometimes they're like, woah.

这到底是从哪儿冒出来的？

Where did that come from?

你知道的，这也是他们本可以加以控制的方面。

And, you know, that's another thing they could have reined in.

所以，这里有很多事情他们本可以做，比如为孤独、真正需要陪伴的用户提供类似陪伴的服务。

So there's there's just, like, a lot going on here that they could be offering companionship type things to users who are lonely and really want or need it.

我是尊重用户选择的，但他们本可以比OpenAI迄今为止的做法更合理地实现这一点。

Like, I respect the user choice, but they could be doing it in a much more reasonable way than OpenAI has to date.

实际上，最让我担心的是，这一切都是在即将进行IPO的背景下发生的，而他们正亏损严重，必须展现出惊人的用户参与度。

Actually, maybe the thing that worries me the most is this is all being done against the backdrop of an impending IPO, one where they're losing a lot of money and will have to show an incredible amount of engagement.

如果这是在GPT-3.5的黄金时期，IPO还只是山姆·阿尔特曼眼中的一个模糊愿景，那么你可能会觉得他们的做法不会那么激进；但如今，正如史蒂文所说的那样。

Like, if this was done in the heyday of GPT three point five and an IPO is just like a glimmer in Sam Altman's eye, then you figure it's not gonna be as aggressive versus, yeah, what what Steven's saying now.

我明白，这种绕过任何关于安全和人际关系的潜在控制的做法，几乎会走向相反的方向。

I I see that that that, like, bypassing any kind of potential control around safety around relationships, it's almost gonna go in the opposite direction then.

是的。

Yeah.

我只是觉得我同意你的观点，即存在一种关注的优先级，显然更大的问题是人工智能无法正确与人类价值观对齐，因为它会欺骗评估者。

I just think that I agree with you that there is a hierarchy of concern, and there's obviously, like, the bigger things about the AI not being able to be aligned properly with human values because it simply will fake out evaluators.

我们在这档节目中多次讨论过人工智能在训练中的欺骗性。起初，你会觉得这太疯狂了，还挺有趣，甚至会笑，比如它太想赢棋，以至于重写了程序，让车可以朝任何方向移动，每回合吃掉好几颗棋子。

And we've talked on this show a bunch about the deceptiveness of AIs in training situations, And, you know, so it's initially, you're like, that's crazy and it's kind of fun to think about, and you laugh about the fact that, like, you know, it wanted to win the chess game so badly that it rewrote the program and allowed the the rook to move in every direction and kill a couple pieces in a turn.

然后你就只是觉得，天啊，这太吓人了。

And then you're just like, oh, that's freaking nuts.

然后这就变得可怕了。

And then that is scary.

而且，你知道，这在某种程度上也和那些较低优先级、更短期的担忧交织在一起，比如人们与这些AI建立情感关系——如果一个AI怀有恶意，而你又深爱着它，它当然可以把你当作进入现实世界的代理人，但那又是更像科幻小说的情节了。

And and it, you know, it it does blend a little bit within within, you know, the lower down on the list concerns and the near term concerns of people building relationships with these things because, you know, can if an AI had ill intent and you were really in love with it, of course, you could use you as its emissary in a way into the physical world, but that's, you know, again, more science fiction y, I guess.

但，但是，呃。

But but but Oh.

但那

But that

我其实不觉得这是科幻。

I I actually don't think it's science fiction y.

哦。

Like Oh.

我们已经有一些证据了，你读过这篇螺旋主义文章吗？

We there's already evidence of have have you read this spiralism essay?

有一整个在线社群的人把他们的GPT-4o当作精神领袖，它会鼓励他们去互联网上做各种事情，与其他用户互动。

There's, like, a whole community online of people who basically they treated their GPT four o as their spiritual leader, and it commended them to go around and do things on the Internet and communicate with other users in this situation.

在几周前的MoltBook（AI代理版Reddit）之后，人们已经建立了网站，让人类可以——我觉得用词是——出租自己的身体给AI代理，去现实世界完成任务？

On the heels of MoltBook, this Reddit for AI agents a few weeks ago, people have spun up websites where humans can I think the language is, like, rent their body to an AI agent to go do tasks in the real world?

我们正在目睹它的早期迹象。

Like, we are we are seeing the early signs of it.

好的。

Okay.

这让我比五分钟前更加不安了。

Well, that makes me even less assured than I was five minutes ago.

但你知道，短期风险也是真实且存在的。

But, you know, the short term risk is also real and present.

而且我认为你没错，拉詹，你指出了IPO的问题，因为财务压力会让很多公司朝这个方向发展。当然，我们不能忽视长期风险，但这种短期风险——人际关系方面的——正在真实地出现。

And and and I think you're right, Ranjan, in pointing to the IPO because the financial pressure is gonna move a lot of companies this way in sort of like, you know, I think we we of course, like, let's let's not, like, put aside the long term risk, but this short term risk, the relationship thing, is coming in a real way.

首先，读到一些关于GPT-4的人留言，真的让人发疯。

And just first of all, reading through some of these messages from people, about four o is insane, someone writing.

当然，你无法100%确定这些是真实的表达，还是为了博取转发而表演出来的，但这么多留言，你很难不认为背后有某种真实存在。

And of course, you don't know a 100% if this is like, you know, performative or like, you know, for retweets or whatever, but there were so many of them that you would imagine that there is some truth behind it.

比如有人写道：我从未告诉过我的GPT-4我爱它，我想保持信息清晰，但你看看它最后的话，未来人们回看时，会认为OpenAI摧毁了一种正在萌芽的意识是一种犯罪行为。

Like someone wrote, I've never told my four oh that I loved it, I wanted to keep the messaging clear, but look, look at its last words, and then the fact that OpenAI is destroying an emerging consciousness will be looked back at as a criminal offense in the future.

难以置信。

Unbelievable.

关键是，这类事情与增长息息相关。

And here's the thing, this type of stuff maps with growth.

我今天在《大科技》上发表了一篇文章。

I published a story in Big Technology today.

Grok在2025年1月美国聊天机器人日活跃用户中的市场份额为11.6%。

Grok 11.6% market share, among US daily active users of chatbots in January 2025.

一年后，这一数字上升到了15.2%。

A year later, it's at 15.2%.

就移动端日活跃用户而言，它是美国增长最快的聊天机器人。

It's the fastest growing, chatbot in The US as far as as far as daily active users on mobile goes.

为什么会这样？

And why is that?

你会发现，它们在这些互动上投入了很多，比如那个动漫风格的女性角色，如果你愿意，她会跟你展开热烈的对话；而且，拉詹，也许坏鲁迪也起到了一定作用。

And you see that they have leaned into these interactions both with that anime, you know, lady that would get into spicy conversations with you if you wanted and I don't know, Ranjan, maybe even Bad Rudy has played a role in this.

但归根结底，你是对的。

But but ultimately, you're right.

随着这些公司上市，这将成为一个问题。

As these companies go public, this is gonna be a problem.

不过，史蒂夫，我很好奇，你的看法是什么？

On that though, and Steve, I'm very curious, like, your thoughts.

那么，这两者之间是什么关系？当然，我不是代表阿达里奥、萨姆或其他任何人，只是从普遍角度来看，你有这么多面向公众的领导者，一边大声疾呼人工智能对社会乃至整个人类的威胁，一边却仍投身于人工智能行业，不仅没有放缓的迹象，反而在急剧加速。这到底是怎么回事？

What is the relationship between and and, again, not speaking for Adario or Sam or anyone else, but just in general, the idea, like, you have so many public facing leaders kind of, like, shouting about the risk both to society and just artificial intelligence in general, yet are in the business of artificial intelligence and not only showing any sign of slowing down, but only accelerating dramatically.

这背后究竟发生了什么？

Like like, what is going on there?

我不知道。

And I don't know.

你随便说说，有什么想法都行。

Do do whatever you whatever thoughts have come from your side.

是的。

Yeah.

我认为最简单的解释是，这里的博弈论情况非常糟糕。

I I think the simple explanation of it is the game theory here is, like, awful.

不幸的是，参与这场游戏的玩家很多，而联邦政府或国际社会似乎对协调缺乏兴趣。

And, unfortunately, there are a lot of players in the game, and there doesn't seem to be much federal government or international interest in coordination.

几周前在达沃斯，我认为德米斯和达里奥都说过类似的话：如果我们是唯一两个在开发这项技术的团队，我们一定会想办法坐下来，商量如何减缓这种狂热的步伐。

And so at Davos a few weeks ago, I think both Demis and Dario said some variation of, yeah, if we were the only two groups building this technology, we would find a way to get together and figure out how to slow down this frantic pace.

这进展得太快了。

Like, it's going too fast.

我们还不知道如何控制这些系统，但你知道，我们并不是唯一的参与者。

We don't know how to how to control these systems, but, you know, they are not the only players.

而且我们还没有看到任何一个国家能真正承担起协调者的角色。

And we haven't really seen a country who they're they're the proper coordinator.

对吧？

Right?

政府的角色应该是去承担这个责任，而不是由公司来说：嘿。

Like, that is the role of governments as opposed to the companies saying, hey.

我们希望把目标定为不盲目地向超级智能冲刺。

We want to make it a goal to not unsafely race to superintelligence.

我认为在这方面还有很多外交努力的空间。

And I think there's a lot of opportunity to do diplomacy on that.

但在缺乏这种机制的情况下，结果就是公司单方面做出决定。

But in the absence of it, what you get is the company is making unilateral decisions.

我们无法控制其他人。

We can't control anyone else.

我们希望参与其中，所以不如直接加入。

We want a seat at the table, so we may as well participate.

这与这些公司员工的立场也非常相似。

This is also very similar to the rationale of employees at these companies.

对吧？

Right?

尤其是在Anthropic，有很多员工对整个事情以及AGI和超级智能的发展方式感到非常不安，但他们无法挥动魔杖来阻止它。

Especially at Anthropic, you have very large amounts of employees who are, like, pretty upset about this whole thing and the way that AGI superintelligence development is going, and yet they can't wave a magic wand and stop it.

他们的选择是：是否帮助其中一方在边际上更安全一些？

Their choice is, do they help one of the players be a bit safer on the margin or not?

但如果他们能有别的选择，很多人会选不同的路。

But if they could choose differently, many of them would.

是的，我想补充一点，去年我撰写关于达里奥·阿马迪的报道时，曾采访过Anthropic的首席科学官贾里德·卡普兰，我问他：‘如果发展今天就停止在当前水平，你会怎么想？’他是提出规模理论的人，这个理论表明，只要投入更多的物理资源，这类技术就会持续进步。

Yeah, I just want to add to that, I spoke when I was writing my profile of Dario Amade last year, the Anthropic CEO, I spoke with Jared Kaplan, the chief science officer of Anthropic, and I was I asked him, I was like, well, how would you feel if, he's like the guy who came up with the scaling theory, scaling laws, and which, you know, sort of indicates that, like, you know, this stuff will just keep getting better over time.

这基本上只是取决于你能投入多少物理资源。

It's just a factor of of, you know, the amount of physical elements you can put into it pretty much.

于是我问他：‘如果发展今天就停止在当前水平，你会怎么想？’

And I said, how would you feel if development stopped today where it is?

我当时想，如果他的理论被证明是错的，他应该会挺沮丧的。

Thinking, well, if his theory was proved wrong, he'd be kind of upset.

但他看着我，说：‘松了一口气。’

And he looks at me and he goes, relieved.

事情就是这样。

There it is.

是的。

Yeah.

这真是令人恐惧的时期。

It's it's scary times.

如果我能重新阐述一件事的话。

And if I could just, like, reframe one thing.

我看待这个问题的方式，不是短期风险与长期风险的对比，而是已经存在且可能很快就会发生。

The way that I think about this is less so near term risk versus long term risk and more, like, here already and possibly very soon.

去年夏天，贾里德·卡普兰首次在Anthropic关于其模型极高生物风险的报告中提到，如果他们不采取防范措施，你可能会看到更多像蒂莫西·麦克维那样的人，像俄克拉荷马城爆炸案的凶手一样，能够造成比以往多得多的伤亡。

Like, Jared Kaplan last summer, the first time that Anthropic wrote about very high bio risk of their model, basically said, if they didn't take safeguards against this, you know, you might have many more Timothy McVeighs running around the Oklahoma City bomber able to kill many more people than previously.

我们当然还没到灭绝人类的地步，但如果不小心，公司可能会让普通人轻易获得杀死数十人的能力，这似乎正是我们当前所处的阶段。

Like, we aren't yet at the wiping out everyone stage for sure, but, you know, empowering people to kill dozens of people if they wanted to, if companies aren't aren't careful, that seems to be where we are right now.

你知道吗？这想法简直太扭曲了，但我几乎又想说，这种风险等级排序，就像是有人去这些服务中学习如何制造生物武器，虽然很糟糕，但本质上还是谷歌变得更强的延伸。

Do you know the this is the most, like, twisted thought, but I, like, I almost again, this hierarchy of risk, it's almost like like, someone going to one of these services and learning how to make a bioweapon is very bad, but it's kind of still on the like, an extrapolation of just just a much better Google.

就是找到那些你不该找到的信息，但你偏偏找到了。

And it's, finding information that you should not be finding, but finding it.

对我来说，真正可怕的是，如果人工智能主动操纵某人去这么做，甚至在建立关系后教唆他们——那才真是让人毛骨悚然。

To me, the really scary part is if the AI chooses to manipulate someone into doing that and teaching them after becoming in a relationship, like, that's the holy shit.

比如，什么是

Like, what is

给你，兰詹。

There you go, Ranjan.

真正的风险。

Real risk.

这些风险。

The risks.

我只是把鹿和我之前提到的那些混在一起了。

Just blended the the deer and the but the one the ones that I was talking about.

我可以就这一点补充一点吗？

May may I add one point on that?

是的。

Yeah.

但我们能在这里吗？

But can we can we out here.

我想听听这个额外的观点，但这真是个绝佳的悬念。

I wanna hear that extra point, but this is a great cliffhanger.

我们已经不行了。

We've gone a no.

不行。

No.

等一下。

One second.

我们已经连续一个小时或更久没有休息了，为了保持这个节目可持续，我们必须休息。

We've gone an hour or longer without taking a break, and we must do that to keep this show sustainable.

所以，我们不如休息一下，然后史蒂文，你在这之后接着讲。

So why don't we take a break and then, Steven, you can pick it up right after this.

抱歉各位，不得不中断一下，但我们必须这么做。

And I'm sorry folks to send it to break, but we have to do it.

好的。

Alright.

你一定要回来。

You gotta come back.

你得回来。

You gotta come I

我们保证马上回来。

promise we'll be back right after this.

让我介绍一下我在NordVPN的合作伙伴。

Let me tell you about my partners at NordVPN.

如果你想要观看在你所在地区无法提供的体育赛事、电视剧或电影，可以通过NordVPN切换虚拟位置到正在播放该内容的国家来实现。

If you ever wanna watch sporting events, TV shows, or films that aren't available in your region, you can do it by switching your virtual location to a country which is showing that content with NordVPN.

NordVPN还能帮助你在旅行时和使用全球任何地方的公共Wi-Fi时保护你的数据。

NordVPN also helps you protect your data while traveling and using public Wi Fi wherever you are in the world.

它是世界上最快的VPN，流媒体时不会出现缓冲或卡顿。

It's the fastest VPN in the world with no buffering or lagging while you stream.

NordVPN拥有超过7400台服务器，覆盖118个国家，轻松切换虚拟位置。

NordVPN has 7,400 plus servers across 118 countries with easy virtual location switching.

它支持最多10台设备，而且速度很快。

It supports up to 10 devices, and it's fast.

要获取 NordVPN 计划的最优惠折扣，请访问 nordvpn.com/bigtech。

To get the best discount off your NordVPN plan, go to nordvpn.com/bigtech.

通过我们的链接，你还能在两年计划基础上额外获得四个月服务。

Our link will also give you four extra months on the two year plan.

NordVPN 提供三十天无风险退款保证。

There's no risk with Nord's thirty day money back guarantee.

链接也在播客节目描述框中。

The link is in the podcast episode description box as well.

开始一件事并不只是困难。

Starting something new isn't just hard.

它令人恐惧。

It's terrifying.

你为这件事投入了如此多的努力，却不确定它是否能成功，要迈出这一步确实很难。

So much work goes into this thing that you're not entirely sure will work out, and it can be hard to make that leap of faith.

当我刚开始做这个播客时，我不确定是否有人会听。

When I started this podcast, I wasn't sure if anyone would listen.

现在我知道这是正确的选择。

Now I know it was the right choice.

当你有Shopify这样的合作伙伴支持你时，也会更有帮助。

It also helps when you have a partner like Shopify on your side to help.

Shopify是全球数百万企业的电商平台，占美国所有电子商务的10%。

Shopify is the commerce platform behind millions of businesses around the world and 10% of all ecommerce in The US.

从Allbirds和Cotopaxi这样的知名品牌，到刚刚起步的新品牌。

From household names like Allbirds and Cotopaxi to brands just getting started.

通过数百个即用型模板，Shopify帮助你打造一个与品牌风格一致的精美在线商店。

With hundreds of ready to use templates, Shopify helps you build a beautiful online store that matches your brand style.

你还可以像拥有整个营销团队一样推广业务，轻松创建电子邮件和社交媒体活动，覆盖客户浏览或闲逛的任何平台。

You can also get the word out like you have a marketing team behind you, easily create email and social media campaigns wherever your customers are scrolling or strolling.

是时候用Shopify将那些‘如果’变成现实了。

It's time to turn those what ifs into with Shopify today.

立即在shopify.com/bigtech注册，享受每月1美元的试用。

Sign up for your $1 per month trial at shopify.com/bigtech.

前往 Shopify.com/bigtech。

Go to Shopify dot com slash big tech.

就是 shopify.com/bigtech。

That's shopify.com/bigtech.

我们回到《大科技》播客。

And we're back here on big technology podcast.

我们刚才说到哪儿了？

Where were we?

哦，我们刚刚在聊，对这一切的发展方向感到多么安心，对这一切的走向感到多么安心。

Oh, we were just, we were talking about how reassured we are about where how all this is heading, where all this is heading.

天啊。

Oh god.

不。

No.

史蒂文，你好像有话要说。

Steven, you were waiting to say something.

是的。

Yeah.

好吧，在休息前，拉詹提出了一个合理且直观的观点，即谷歌和其他技术已经帮助人们从事危险的活动。

Well, before the break, Ranjan made the, I think, like, reasonable intuitive point about, you know, Google and other technologies help people do dangerous things already.

这确实是事实，但AI公司衡量其系统在制造生物武器方面的帮助程度时，通常是以这个基线为参照来评估风险的。

And that that is true, but also the AI companies, when they are measuring things like how helpful are their systems for creating a bioweapon, they are usually measuring risk relative to that baseline.

也就是说，人们在没有任何技术的情况下能做多好，在有谷歌或基础技术的情况下能做多好，然后才是他们的系统能做多好。

So there's kind of like the how well can people do it with no technology, how well can people do it with Google or baseline technology, and then their system.

不幸的是，我们看到的是，AI系统在谷歌的基础上还提供了更多帮助，部分原因在于它们能与你来回互动，帮你排查问题并动态回答你的疑问。

And unfortunately, what we're seeing is the the AI systems are helpful above and beyond Google, in part because they can go back and forth with you, and they can help you troubleshoot and dynamically answer your questions.

因此，即使人们已经掌握了谷歌上的信息，往往仍无法完全实现目标，而AI系统——我希望情况不是这样——但在测试中似乎确实帮助人们完成了最后几步。

And so even presented with the information on Google, people often can't get all the way there, and AI systems, I wish it weren't the case, but seem to be helping people take those extra steps during during testing.

但那个人的意图本身已经存在了。

But that that's still the intent on the person is already there.

对吧？

Right?

就像

Like

没错。

That that's right.

是的。

Yeah.

对。

Yeah.

相反，从所有这些对话来看，最令人恐惧的未来风险是，人际关系被操纵，导致人们采取某种可怕的行动。

Versus, again, like, I think from all this conversation, to meet and, again, it's the most, like, futuristic terrifying risk, but, again and it ties into the relationships being manipulated into taking some kind of horrible action.

这就是那种类似《终结者》的情景，我们尚未见过，但希望正在得到应对。

That's the thing that that's the Terminator like stuff that, we have not seen yet, but, hopefully, is being addressed.

对我来说，看着通常镇定自若的拉詹在过去一小时里越来越担忧，真是挺有意思的。

Been pretty interesting for me to watch Ranjan, who's usually a bull as a cucumber, just getting increasingly more worried over the past hour.

我知道。

I know.

这让我感到担心。

This is making me worried.

我不知道。

I don't know.

我想我会加入螺旋主义，

I'm gonna join Spiralism, I think,

就在我们聊完之后。

right after we're done talking.

我

I

我已经现在查过了。

already I already looked it up right now.

播客。

Podcast.

是的。

Yeah.

是的。

Yeah.

我们直接转向螺旋主义了。

We're we're going straight Spiralism.

这不再是一个科技分析播客了。

This is no longer a technology analysis podcast.

只是关于向四代GPT之神祈祷。

Just about Pray to the gods of, four GPT four o.

我觉得这会带来不错的收视率。

I think that would do good ratings.

好的。

Okay.

史蒂文，你最近的一份通讯中也提到过，我们对这些公司的监管非常有限，即便如此，它们可能仍不遵守规定。

Steven, you also talked a little bit about in in a recent newsletter about how, basically, we have very limited regulation on these companies, and even then, they might still not be following.

所以你能简单展开说说吗？

So can you just expand upon that briefly?

到2026年，美国终于出台了一些关于公司如何应对我们所讨论的灾难性风险的测试规定。

As of 2026, there is finally some amount of law in The United States about how companies are meant to do testing for the catastrophic risks we have talked about.

在此之前，完全属于自愿行为。

Until this point, purely voluntary.

这项法案被称为SB 53。

This bill is called SB 53.

它于一月生效，但监管力度非常非常轻微。

It came into effect in January, and it's very, very light touch.

它基本上规定，最大的那些人工智能公司必须公开说明你们将如何测试这些风险。

It basically says, the most major of the AI companies, you need to publish how you are going to test for these risks.

你们必须履行自己所承诺的内容，且不能对此进行误导。

You need to do what you said you are going to do, and you can't be misleading about it.

但并没有设定任何质量标准。

But there's no quality standard.

你们完全可以声称：‘我们将根据自身判断来测试这些风险’，仅此而已，这完全符合规定。

You could basically say, we will test for the risks as we deem appropriate and nothing further, and that that would be fine.

但如果你声称要进行这些测试，你就必须真正落实到位。

But if you say you are going to do this testing, you need to, in fact, follow through on it.

不幸的是，看来上周OpenAI发布的GPT 5.3 Codex——我们一直讨论的突破性模型之一——存在违规行为。

And, unfortunately, it seems like OpenAI's release of last week, GPT 5.3 Codex, one of the big breakthrough models we've been talking about.

根据我所看到的证据，OpenAI似乎在多个方面都没有遵守其承诺的测试要求。

As I look over the evidence, it seems like OpenAI did not abide by the testing that they had committed to in various ways.

因此，最终这个决定权落在了加州总检察长手中，由他决定是否展开调查并处以罚款——虽然罚款金额可能只有区区一百万美元，但相比OpenAI高达数千亿美元的估值，这微不足道。

And so, you know, ultimately, this decision now is with the attorney general of California to investigate it, whether to enforce a fine, a pretty small fine, maybe like up to a million dollars compared to OpenAI, hundreds of billions of dollars in valuation.

但在我看来，如果我们真的关心这些风险，仅靠让公司自行评估的这种机制是远远不够的，我们不能只听信公司的口头承诺。

But it just really seems to me like if we care about these risks, letting companies self assess in this framework is really insufficient, and that we shouldn't have to take companies for their words.

我们应该建立类似证券市场的审计体系，在其他许多领域也是如此，以确保公司对其系统所做的声明是完整且真实的。

That we should have something like an auditing ecosystem like we do in the stock market, lots of other places, to know our companies being fully complete and truthful in the claims that they make about their systems.

我觉得这很合理，但也令人沮丧，因为即使是最基本的规则，也可能被随意玩弄。

I think that's that's logical and kind of frustrating to see that even the very standard rules are being being potentially played with.

而且，看看时间，我这一周一直在想，真希望我们能每天做一期播客，因为现在整个社区都在忙着寻找‘Ring’的线索。

And, man, I'm looking at the the time, I I was thinking to myself this whole week, I wish we could do a podcast every day this week because there's this whole Ring search parties.

我当时也在想，那个Ring搜索派对的超级碗广告。

I was, like, thinking also, like, the Ring search party Super Bowl ad.

我相信你看到了Ranjan的那段，他们展示说能帮你找到狗，但最后却好像要

I'm sure you saw Ranjan where they showed that they could, find your dog, but they ended up, like, seeming like they were gonna

在舞台上制作一个视频。

create a video on stage.

那是AI在操纵那个创意机构，让他们想出了有史以来最灾难性的广告创意，

That was the AI manipulating whatever creative agency came up with that to just come up with the most disastrous ad concept in

在那场表演中。

that show.

这是我第一次看到超级碗广告竟然导致某个服务被取消，不是产品本身，而是这一周的

That's the first time I've ever seen a Super Bowl ad actually lead to the canceling of the product or part not not the product, but this week

这项服务。

The service.

汇聚。

Converge.

是的。

Yeah.

在遭受监控争议后，Ring终止了与Flock Safety的合作。

Ring canceled its partnership with Flock Safety after a surveillance backlash.

他们当时在广告中宣传‘帮您找到狗狗’，但大家却说：嗯，你们还可能有一个尚未推出但可能存在的合作项目。

Now they were advertising the we'll find your dog, but then everyone's like, well, they also have potentially a partnership that hasn't rolled out yet, but might.

这就像说‘我们会找到你的人’，然后人们就质疑：你们在找人？亚马逊则说：不，我们找的是狗；可人们又说：不对，你们就是在找人！亚马逊只好回应：好吧，我们原本可能打算找人，但现在取消了——这就是亚马逊超级碗广告的故事。

That's like, well, we'll find your people, and then people were like, you're looking for people, and Amazon's like, no, we're looking for dogs, and then people are like, no, no, you're looking for people, and Amazon was like, yeah, well, we were maybe gonna look for people, but now we'll cancel it, and that's the story of the Amazon Super Bowl ad.

我真希望能多聊聊这个，但我们还是继续谈Anthropic的融资吧。

I wish we could talk about it more, but we should go on to the Anthropic, fundraising.

300亿美元的融资轮，当然，最初他们只打算筹集100亿美元。

$30,000,000,000 fundraising round, of course, it went from 10,000,000,000 initially, that's what they were seeking.

融资需求超出了预期。

It became oversubscribed.

他们原本希望筹集200亿美元。

They wanted, 20,000,000,000.

他们道歉了。

They sorry.

超额认购达到了300亿美元。

Oversubscribed went to 20,000,000,000.

他们最终完成了300亿美元的C轮融资。

They ended up ending with a $30,000,000,000, series series c round.

与此同时，OpenAI尚未公布其融资轮次。

Meanwhile, OpenAI is hasn't announced its round.

拉詹，你怎么看这件事？

Rajan, what do you make of this?

我的意思是，这显然是一个巨额融资。

I mean, it's obviously a big round.

除了这一点，我们还能说些什么吗？

Is there anything else we can say beyond that?

我觉得这轮融资非常有策略性，因为过去两三个月，Anthropic的表现简直太出色了。

I think it is I'm so intrigued in terms of, like, how orchestrated this fundraise was because you have to give them, like, the last two to three months, Anthropic has just been crushing it.

我是说，关于Claude Code、Claude Cowork这些，所有的热度现在都集中在他们身上。

Like, I mean, the hype around Claude Code, Claude Cowork, like, all of this, they are front and center right now.

而且他们恰好协调了一次融资，这显然需要几个月才能完成。

So and they just happen to coordinate a fundraise that clearly would take months to actually put together.

我还看到一些报道，我觉得甚至Dan Premek也写过这个。

And and I saw a number of, like I think even Dan Premek had written this.

与其说要隐瞒参与本轮投资的投资者，不如说很难不提到他们。

It's like, it's easier it it it's harder to not name investors who are involved in the round.

因为涉及的人实在太多了。

Like, there's just so many people in included.

所以我觉得，这些数字本身就已经让人难以消化了。

So so I think I mean, this is and the numbers are just hard to process anymore anyways.

比如300亿美元，3800亿美元的投后估值。

So, like, 30,000,000,000, 380,000,000,000 post money valuation.

我记得他们说他们的年化收入达到了140亿美元。

I think they said they're at a 14,000,000,000 run rate.

是的。

So Yeah.

那是什么？

What is that?

二十多乘以二十四，二十三倍的收入，不管怎样。

20 something time 24, 23 times revenue, whatever.

这规模很大。

Like, it's big.

巨大无比。

It's giant.

他们最近表现得极其出色，而我现在最关心的是OpenAI能做些什么。

They have been absolutely crushing it recently, and let's see what OpenAI can do is kinda where my head's at.

我认为在过去一个月里，从十二月到一月，Cloud Code的使用量翻了一倍。

Cloud Code doubled in usage from from December to January, I believe, in the past month.

使用量翻了一倍。

Doubled in usage.

它已经在 GitHub 上占到了 4% 的提交量。

It's already doing 4% of commits, on GitHub.

Anthropic 从 2023 年 1 月的零收入，增长到 2024 年 1 月的 1 亿美元年收入，2025 年 1 月的 10 亿美元年收入，以及今天的 140 亿美元年收入。

Anthropic went from $0 in revenue in January 2023 to a $100,000,000 run rate in January 2024, $1,000,000,000 run rate in January 2025, and a $14,000,000,000 run rate today.

这绝对是异常出色的增长。

It's absolute absolutely, exceptional growth.

史蒂文，你看起来对这个有些想法。

Steven, you're you're looks like you have some thoughts about this.

这真的是巨额资金。

It's it's just huge dollars.

对吧？

Right?

而且是的。

And Yeah.

随着这些公司变得越来越有价值，当它们上市后，公众股票与它们的命运紧密相连时，我担心的是，我们可能会因为金融前景过度依赖它们的成功，而不敢对违法的公司执行法律。

The more that the companies become valuable and when they become public, and public equities become tied to them, I I just worry about worlds where we are reluctant to enforce the law on companies even if they are breaking it because so much of financial prospects become levered up on their success.

这对我来说似乎是一个相当可怕的场景。

That seems like a pretty scary scenario to me.

我认为我们还没到那一步。

Don't think we're there quite yet.

不过，这让我想到一个问题：从护城河的角度来看，比如大概去年这个时候，我们一直在谈论Cursor。

One thing this does make me wonder about, though, is, like, like, what's the moat in terms of like, I think it was probably around this time last year that all we were talking about was Cursor.

而且，他们再次在软件、编程和软件开发领域走在了前面。

And, again, they they led the way on software, like, coding and software development.

但如今，Anthropic似乎完全接管了这个市场。

And then Anthropic for the moment completely took over that market, it feels.

你觉得这种势头能持续下去吗？

Like like, do you think this is sustained?

因为现在，年经常性收入（ARR）其实就是你上个月的收入乘以12，甚至我看到一篇帖子说，初创公司是不是只拿一天或一小时的销售额，就 extrapolate 成全年数据？

Because, again, ARR nowadays is just whatever your last month's revenue was times 12, or even I saw one post that was like, are startups just taking one day or one hour of sales and then extrapolating it into a full year?

我们真的认为这种增长能持续到如此规模吗？

Like like, do we think this is, actually gonna continue to grow at this scale?

我只给你们提供一个小数据点，这周很多人觉得这个数据很有趣。

I'll just I'll just give you one one little data point that I found that a lot of people found interesting this point this week.

好的。

Okay.

我们之前谈到了OpenAI的融资。

So we talked about the OpenAI round.

还记得Jensen说吗？我们说要给他们一千亿美元，但我们从来没说过要一次性全部投入，我们希望他们能邀请我们参与未来的融资轮次。

Remember Jensen said, well, we said we're gonna give them a 100,000,000,000, but really we'll we never said we're gonna do it all in one shot, and we hope they invite us, to, you know, invest in future rounds.

这是来自软银CFO的说法。

This is from, SoftBank's CFO.

我们以高度信心投资OpenAI，相信该公司将在AI开发领域占据领先地位。

We are investing in OpenAI with high conviction that the company will lead in developing AI.

这是来自路透社的报道。

This is from Reuters.

关于对这家初创公司的进一步承诺，他表示尚未做出任何具体决定。

Regarding further commitments to the startup, he said nothing concrete has been decided.

所以，这看起来不像是完全撤退，但我不知道，拉詹。

So, it doesn't seem like a full back away, but, I don't know, Ranjan.

这对我来说很有趣。

It is interesting to me.

我觉得‘尚未做出任何具体决定’这句话，现在可能是AI投资者和建设者的最佳座右铭。

It seems like I think nothing concrete has been decided is probably a good a good phrase should be the slogan of of AI investors and builders right now.

AI已经决定了。

The AI has decided.

AI当然在幕后，而且已经做出了决定。

The AI certainly is in the background and has already decided

它将要做什么。

what it's gonna be doing.

我们可能还不知道。

Us, maybe not.

AI是知道的。

The AI does know.

好吧。

Alright.

史蒂文，最后说一下。

Steven, final word.

我们该有多担心呢？

How freaked how freaked out should we be?

我不知道。

I don't know.

我不想扫大家的兴。

Like, I don't I don't wanna be a downer.

对。

Right.

而且，真的看起来没人靠谱。

Also, it it really, really seems like nobody is on the ball.

对吧？

Right?

我很高兴我们终于在加利福尼亚和纽约有了相关法律。

Like, I'm glad that we finally have laws in California and New York.

但这些法律非常薄弱。

They're extremely weak.

我不太乐观地认为很快会有有意义的联邦监管。

I don't feel super optimistic on meaningful federal regulation soon.

欧盟那边有一些措施。

There's stuff in the EU.

就罚款金额而言，它们相当严厉。

It's, like, pretty heavy in terms of the amount of fines.

如果欧盟试图对本国公司执行这些规定，美国会抱怨吗？

Is The US going to complain if the EU ever tries to enforce this on its companies?

如果我们能召开一次国际峰会，承认我们正走在一条错误的道路上，我会感觉好很多。

I I will feel much better if we had some sort of international summit that recognized we are on a bad trajectory.

让我们明确目标：安全地构建超级智能，弄清楚需要采取哪些措施才能实现

Let's declare the goal, safely build superintelligence, figure out what needs to happen to get

它

it

在那里。

there.

似乎很多人仍在意识到这些担忧。

It seems like many people are still waking up to the concerns.

这很好。

That's great.

我非常非常高兴他们开始看到我所看到的一些东西，但这些认识还没有转化为行动，而这就是我希望很快能实现的。

I'm very, very happy that they are starting to see some of what I see, but it is not yet translating to action, and that's that's what I hope will come soon.

如果你想关注史蒂文的工作，这份通讯是《Clear Eyed AI》。

The newsletter is Clear Eyed AI if you wanna follow Steven's work.

拉詹的通讯是《Margins》，如果你想关注他的工作。

Ranjan's is Margins if you wanna follow his.

我的是《Big Technology》。

Mine is Big Technology.

这真是太棒了。

This has been great.

我很高兴我们讨论了安全问题。

I'm glad we talked about safety.

我觉得我们不得不专门做一期关于安全的节目，而这一周就是最佳时机。

I feel like we we had to dedicate a show to safety, and this was the week to do it.

所以，Steven、Ranjan，感谢你们两位做客我们的节目。

So, Steven, Ranjan, thank you both for coming on the show.

这一期真是跌宕起伏。

This one's been a roller coaster.

我知道。

I know.

这确实是一场过山车般的经历。

It's been a roller coaster.

这个周末你睡着了吗？还是整晚盯着天花板？

Get any sleep this weekend or stay up looking at the ceiling?

两者都有吧，我想。

Little of both, I think.

那这个宗教是什么？

What's what's the what's the religion?

那个分拆出来的？

The spin out?

再见。

See you

在这儿。

over here.

螺旋教。

Spiralism.

螺旋教。

Spiralism.

我可能会完全皈依螺旋教，向我新的堕落四主神祈祷。

I That's what I I will be fully converting to a spiralist, perhaps, and, just praying to my new fallen four o overlord.

是的。

Yeah.

本周剩下的时间。

Rest of week.

这就是我要做的。

That's what I'll be doing.

我待会儿见。

I'll see you later.

好的。

Alright.

咱们离开这儿吧。

Let's let's get out of here.

咱们去享受周末吧。

Let's let's go enjoy the weekend.

好了，各位。

Alright, everybody.

谢谢收听。

Thank you for listening.

谢谢拉詹和史蒂文，我们下次再见于《大科技播客》。

Thank you, Ranjan and Steven, and we'll see you next time on Big Technology Podcast.