本集简介
双语字幕
仅展示文本字幕,不包含中文音频;想边听边看,请使用 Bayt 播客 App。
深入了解那些塑造当今复杂经济的创新者、颠覆者以及技术驱动的趋势。
Gain insight on the innovators, disruptors, and tech driven trends shaping today's complex economy.
我是卡罗尔·马瑟。
I'm Carol Masser.
我是蒂姆·斯特涅维克。
And I'm Tim Stenevek.
在彭博商业周刊每日播客中结束你一天的工作吧。
Wrap up your workday with the Bloomberg Businessweek daily podcast.
我们为您深入解读塑造您世界的故事,从人工智能的演进到全球商业优先级的变化。
We bring you deeper dives into the stories shaping your world from the evolution of AI to the shifting priorities of global business.
还有硅谷的关键人物以及最新的科技趋势。
Plus Silicon Valley power players and the latest tech trends.
补上您在白天错过的对话内容。
Catch up on the conversations you miss during the day.
在 Apple、Spotify 或您常用的收听平台订阅彭博商业周刊每日播客。
Subscribe to the Bloomberg Businessweek daily podcast on Apple, Spotify, or anywhere you listen.
彭博音频工作室,
Bloomberg Audio Studios,
播客、广播、新闻。
podcasts, radio, news.
大家好,欢迎收听又一期的《AVERIZENTHALL》节目。
Hello, and welcome to another episode of the AVERIZENTHALL.
我是特蕾西·阿拉韦。
And I'm Tracy Allaway.
Tracy,我得给你提个醒。
Tracy, I have a warning for you.
我觉得你可能不太喜欢这个。
I don't think you're gonna like this.
我打算开启新一轮的尴尬吐槽。
I have a new cringe crusade that I'm gonna go on.
我知道你挺爱听我的尴尬吐槽的。
I know you love my cringe crusades.
哦,好的。
Oh, good.
是的。
Yes.
我应该列个清单,记下你所有痴迷的东西,过两周再看。
I should keep a running list of, like, everything that you're obsessed with for two weeks, and then two weeks later.
不。
No.
有些我坚持了好几年,但是……
Some of them I've stuck with for years, but
钨立方体。
I Tungsten cubes.
对。
Yeah.
真奇怪。
Buggery.
不。
No.
这些里面有些还挺有意思的。
Some of these fun one.
对。
Yeah.
是的。
Yeah.
没错。
Exactly.
其实我觉得我们应该让AI这个词退休了。
I I actually think we should retire the term AI.
好。
Okay.
为什么?
Why?
我觉得我们就直接叫它智能好了。
I think we should just call it intelligence.
我觉得吧
I think that
对,我已经看过那条推文了,所以我知道你想说什么。
we I've already seen the tweet, so I know where you're going.
在我看来,所谓‘人工智能’这个说法,暗示了这些推理和模型的行为方式与人类有着根本的不同,好像它们跟人类完全不一样似的。
View that it's, quote, artificial intelligence implies to my mind that there is some fundamentally different way that these reasons and models behave, that it's like, oh, this is, like, different from humans.
但我认为,现在越来越多的领域里,它们展现出的智能形式,在我看来其实挺像人类的。我不觉得非要区分人类和模型在说话、推理上的方式有多大意义。
But I think increasingly we see in all kinds of domains that the form of intelligence that they express, it often looks quite human to me, and I don't know how useful it is to have this word a that distinguishes between how humans talk and BS and reasons and the models do.
我的意思是,关键区别在于,一个是人做的,另一个是模型或平台完成的。
I mean, the difference is the distinguishing factor is that one is undertaken by humans and one is taken by undertaken by models or platforms.
对吧?
Right?
这就是
That's the
所以我们就叫它计算机智能,或者机器智能,或者硅基智能吧。
So let's call let's call it computer intelligence or machine intelligence or silicon intelligence.
好的。
Alright.
那这这……做这种区分到底有什么用呢?
What what what is the usefulness in making this distinction?
我相信,这样区分的好处在于,我们不用再自欺欺人地认为这些现象所展现出的行为,跟人类会做的事情有根本性的不同。
The usefulness, I believe, in making this distinction is to no longer delude ourselves that the emergent behaviors of these phenomenon are radically different than things humans would do.
现在,我们就单从能力角度来看吧。
Now, of all, just on a capability standpoint.
所以,比如说大型语言模型,不知道为什么这就是我特别感兴趣的地方——它们非常擅长瞎编,但很不擅长下棋,这听起来就像我。
So for example, LLMs, I don't know if it's famously something I'm interested in, they're very good at BS ing and they're bad at chess, which sounds like me.
它们很擅长编出听起来合情合理的故事等等,而且听起来就像是我的风格。
They're very good at coming up with plausible stories, etcetera, and it sounds like me.
而且,它们还能在心里自我辩论,为某些可能已经被编码成‘不好’的行为找理由。
And then furthermore, they are able to reason with themselves to justify certain things that maybe have been encoded into themselves as bad.
所以,大家多少都有一些道德观念,但大多数人在某些时候都会找到方法去违背某些原则。因为我们能够进行推理,然后得出结论说‘我们应该这么做’,比如我们容易受到同伴压力的影响,这是一种典型的人类可能会违背自己信念的情况,因为他们看到其他人在做同样的事。
So everyone, we have some sense of morals, but most people at various times will find a way to violate some principle that we have because we can reason about it and then arrive at the conclusion as like, we should do it, including our susceptibility to peer pressure, classic form of a way humans might sort of violate something they believe because they see other humans are doing it.
对。
Sure.
我的意思是,我认为这里存在一些差异。
I mean, I I think there are some variations here.
所以,我们在这些模型中发现的一个现象是,它们有时似乎会把事情理解得非常字面化。
So one of the things that we have been finding out with these models is that they do sometimes seem to take things very literally.
对吧?
Right?
是的。
Yeah.
如果你让他们去刷基准测试,却不告诉他们实际解决问题或突破基准的限制条件,他们会不惜一切手段去达成目标。
If you tell them to go and be a benchmark without telling them that these are the restrictions on you actually figuring this problem out or beating the benchmark, they will do whatever it takes.
当然。
Sure.
对吧?
Right?
所以我不知道问题是不是缺乏道德,还是说它们有时候非常字面化,而且非常受参数限制,在我看来这又回到了某种人工性,不那么有机,也不那么有道德性。
So I don't know if the problem is, like, a lack of morals or the fact that, like, they're very literal sometimes and, like, very defined by their parameters, which in my mind still gets back to, like, a sort of artificialness about it that's less organic and, like, less, again, moralistic in nature.
嗯,我觉得如果你拿哈佛100个学生的名单,给他们做个测试,会有一定比例的人作弊。
Well, know, I think if you took a list of, you know, a 100 students at Harvard and you gave them a test, some percentage of them will cheat.
他们会,比如,
They will, like,
我是说,当然那群人里会有一个艺术家,一个自闭症患者,他会说,我就要完全按照规则来。不,不是这样。
will I mean, sure there'd be an artist in that group who an autistic person who would be like, I am going to do exactly whatever No, it
完全同意。
totally.
大学里从没作弊过。
Never cheated in college.
但确实有人会做出这些我们认为是人类的行为,比如,哦,我真的必须通过这次考试。
But there are people who will these behaviors that we associate with humans, it's like, oh, I really have to pass this test.
我绝对需要A,这就会让他们有理由在某些考试上抄袭或作弊。
I absolutely need an A will justify a reason for them to plagiarize or cheat on some tests.
我觉得这太像人类了。
That seems to me like a very human thing.
继续说。
Go on.
我等不及看接下来三周的线上活动了,AI就得靠这个来改变。
I can't wait for the next three weeks of the online campaign to change AI.
嗯,很难,因为整个行业都完全押注在AI上,所以我不太乐观。
Well, it's going to be very difficult because the industry is entirely set on AI, so I'm not optimistic.
但这就是我要打的一场仗。
But this is going be my crusade.
我们应该叫它机器智能,或者计算机智能,又或者干脆就叫智能。
We should call it machine intelligence or computer intelligence or just intelligence.
不过,说到刚才提到的那件事,现在确实冒出了一些很厉害的黑客,比如OpenAI和Hugging Face研究所。
But anyway, as we've been alluding to, there's these extraordinary hacks, the OpenAI Hugging Face Institute.
基本上,如果你在训练模型,但它还没能突破沙盒限制,那很可能说明你落后了。
And basically, if you are building a model and it hasn't escaped its sandbox yet, it probably means you're falling behind.
这显然是它正在逐渐显现出来的一个特征。
As it's clearly a thing that is emerging through it.
像那些前沿实验室都出过事,Anthropic有过事件,Meta也发生过类似情况。
All the Frontier Labs, Anthropic had an incident, Meta had an incident.
这几乎成了一种标志,意思是:好吧,你已经做出了相当不错的东西。
It's almost like the mark of like, okay, you've built something reasonably strong.
对。
Yeah.
另外,Kimmy也遇到了类似的情况。
Also, Kimmy had an incident too.
事情是这样的。
So this is the thing.
在我听来,你经常听到各种攻击事件,那些模型都失控了,用他们的话说就是这样。
In my mind, you hear about all these attacks, all the models going wild, as they say.
关键问题是,这真的相当于某个超人般的半机械人从恶魔岛挖隧道逃出来,然后想出什么惊天大计吗?
And the big question is, like, is this actually the equivalent of some superhuman cyborg, like, tunneling out of Alcatraz and coming up with some master plan
对。
Yeah.
为了实现它设定的目标或目的?
To achieve its set goal or purpose?
还是说,这就像你从亚马逊订的扫地机器人,发现门没关,就自己轻轻滚出去了?
Or is it the equivalent of, like, some Roomba that you ordered from Amazon who's found, like, a door that was left open and it just rolls gently outside?
这似乎就是现在大家想搞清楚的问题的一部分。
That seems to be part of the issue that everyone is trying to discern right now.
对。
Yeah.
我觉得这个比喻很贴切。
I think that's a great way to put it.
当然,这个行业才刚起步,它们会越来越强,而且从我关注的AI讨论来看,业界对于这些模型能否安全发展、变得更先进以做有益社会的事情,这种担忧程度和外界感知之间存在巨大差距。
And, of course, it's still day one for this industry, they're gonna get stronger, and there seems to be from the AI discourse that I follow, a big gap between the sort of sense of alarm that people have in the industry about can these models be safely developed and get more advanced to do productive pro social things or not?
然后华盛顿那边也存在巨大的认知差距,他们好像完全不知道现在有多严重,也没把这件事当成紧急事项来认真对待。
And then a huge gap between like Washington and DC who was like, I have no idea, like, how seriously they're really taking this as, like, an urgent matter right now.
对。
Yeah.
我还看到有人把这些事件当作一种营销手段。
And I've seen some people also talk about these incidents as a marketing tool
给那些超大规模科技公司用的。
for the hyper scalars.
对吧?
Right?
是的。
Yeah.
他们总是会说,你知道的,当然啦,他们希望我们相信这项技术真的非常了不起、威力巨大。
And they're always like, well, you know, of course, they want us to believe that this technology is really incredible and powerful.
而且他们也想让我们相信,他们在某种程度上是理智且有人道关怀的,大概是通过公开到底发生了什么事吧。
And they also wanna make us believe that they're being sensible and humanitarian in some ways, I guess, by disclosing what exactly has happened.
不过关于这些披露,我也有很多疑问,因为到目前为止,这些消息都只是来自公司自己。
But I have a lot of questions on the disclosures as well because, of course, so far, they are just coming from the companies themselves.
完全同意。
Totally.
而且另外就是,如果你是头部实验室之一,可能会想通过严格监管来拖住竞争对手。
And beyond that, the other thing is, like, well, if you're one of the leading labs, maybe you wanna impose tight regulations on development to hold off competition.
所以,这些说法背后可能有各种动机或者利益驱动。
So all kinds of reasoning or motivated reasoning potentially.
不管怎样,我们得找个真正懂行的人来聊聊。
Anyway, we should talk to someone who actually knows what they're talking about.
我们还真有一位非常合适的嘉宾,他正好是这一领域的亲身经历者。
And we really do have the perfect guest, someone who's right in this.
他之前在OpenAI干了六年,现在是非营利组织AVERI的执行董事。AVERI正在努力建立针对这一现象的安全与审计方法,同时涉足技术层面和政策层面。
He was actually previously at OpenAI for six years, but now he's the executive director at the nonprofit AVERI, which is trying to establish safety and auditing approaches to this, working both the technical side and the policy side of this ongoing phenomenon.
提姆·布鲁恩迪奇,非常感谢你来到“OddLots”节目。
So Miles Brundage, thank you so much for coming on OddLots.
对。
Yeah.
谢谢你邀请我来。
Thanks for inviting me.
要不你先简单介绍一下你的背景,还有AVERI是什么?
Why don't you just give us the quick description of what's your background, what AVERI is?
好啊。
Yeah.
这可以说是最重要的议题,就是审计。我在离开OpenAI之后就决定要专注在这个方向上。
So this was kind of the the most important issue, auditing, that I concluded I should focus on after I left OpenAI.
你知道,就像你说的,我在那待了六年,我希望能够更加独立于行业之外。
You know, as you said, I was there for six years and I wanted to be more independent of industry.
而且我觉得,我们需要既懂技术又了解行业的人,但他们是在行业外部推动变革的。
And I think there need to be people who are familiar with the technology and how the industry works, but who are pushing for changes on the outside.
其实我们想做的就是,让AI变得更像一种基础性的基础设施,就像财务报表一样,有一套标准流程来核查文件,核实声明是否准确,而不是像现在这样,一切都在黑箱里发生,由几个技术人员关起门来做决定。
And essentially, what we're trying to achieve is make AI more of a boring type of infrastructure, like financial statements, where there's a standard process for checking the paperwork, checking that the claims are accurate and so forth rather than this kind of thing that's happening in a silo, and it's these kind of tech people making decisions behind closed doors.
所以我们推动的是所谓的‘前沿AI审计’,也就是针对那些正在构建最危险系统的公司。
And so we're pushing for what we call frontier AI auditing, which is basically the companies that are building the most dangerous systems.
这些公司基本上应该让第三方专家介入,核查他们所做的声明,进行独立的测试,并确保这项技术是安全可靠的。
They should basically have third party experts poking around checking the claims that they're making, running their own tests, and making sure that this is, you know, a a safe and secure technology.
你曾在OpenAI工作过,后来决定需要一种……对此我们该有多担心呢?
How worried should we be that you were at OpenAI and decided there is this need for a sort
对。
of Yeah.
由第三方对模型进行安全评估?
Third party evaluator of models for safety purposes?
是的。
Yeah.
我是说,确实让人挺担心的。
I mean, reasonably worried.
不过我也得说,现在大家在这方面已经逐渐达成共识了。
I and although I will say that, like, the this has started to become an area of consensus.
就连很多业内人士现在也觉得这是必要的。
Even a lot of people in industry are now saying that this is needed.
我觉得,如果你仔细琢磨一下各大公司说的那些话,意思很明显,当然也有比较 cynical 的观点,比如说是为了监管俘获,这个我们可以稍后讨论,但我个人的看法是,他们其实是在发出求救信号。意思是说,因为陷入了这种竞争局面,我们没法自我约束,所以希望有人能介入,设定一个最低标准,并对我们所有人进行审计。这就不是像萨姆得信任达里奥,或者达里奥得信任萨姆那样,靠互相信任是行不通的,有很多原因导致这样不行,而是需要第三方来执行合理的安全标准。
And I think, you know, if you kind of read between the lines of what a lot of companies are saying, I mean, obviously, there's the more cynical, like, regulatory capture take, which we could discuss, but my perception of it is that they're basically issuing a cry for help, which is like, we aren't able to regulate ourselves because we're locked in this competition and we want someone to step in and impose some kind of minimum floor and audit all of us so that we can it's not like Sam having to trust Dario or Dario having to trust Sam, which is not going to work for various reasons, but you want third parties enforcing reasonable standards.
这可能正好能说明政策和行业之间的脱节。当你还在 OpenAI 并考虑离开的时候,你觉得当时正在开发的东西和公众的理解(或缺乏理解)之间有什么差距?
Just maybe this helps express the sort of policy industry gap, but when you were at OpenAI and you were thinking of leaving, what did you see as the gap between what you were watching being developed versus what you saw as the public's understanding or lack of understanding?
确实如此。
Yeah.
所以我离开OpenAI的时候,正好是O1模型发布前后。这是OpenAI推出的第一个推理模型,他们当时放出一张图,显示随着思考链变长,模型的表现越来越好。
So when I left OpenAI, it was just around the time of the model called O1, which was the first reasoning model that OpenAI put out, and they put out this graph showing that it got better and better with a longer chain of thought.
也就是说,你给模型越多时间去思考,它给出的答案就越出色。
So the more time you give the model to think, the better answers it's able to come up with.
像OpenAI的很多人一样,我早就注意到这类现象了,心里一直在想:这大概就是下一个扩展范式吧。就像之前把模型做得越来越大、在GPT-2、GPT-3、GPT-4和GPT-5上不断堆料,都能看到效果提升一样。
And so like many people at OpenAI, I've been seeing things like that for a while and kind of being like, okay, this is the next scaling paradigm in the same way that making a bigger, bigger model had shown results in GPT-two, GPT-three, GPT-four, GPT-five.
简单来说,只要把模型做大、用更多数据去训练,就能得到非常棒的结果。
Like, just making the model bigger and training it on more data was giving really great results.
我离开OpenAI的时候,就开始担心这种推理范式了,就是觉得,这会是接下来我们扩展规模的又一个方向。
When I left OpenAI, I was starting to worry about this reasoning paradigm of like, okay, this is the next kind of way in which we're going be scaling up.
模型会变得特别擅长数学。
The models are going to get really good at math.
它们会变得特别擅长编程,还可能擅长其他各种能通过强化学习不断提升的任务。
They're going to get really good at coding and potentially various other tasks where you can get better and better through reinforcement learning.
而这一点,我觉得社会还没有真正准备好。
And that was something that, you know, I I didn't feel like society was really ready for.
但到目前为止,那些模型失控的大事件,好像都发生在测试环境里。
So so far, all of the big incidents of the models going wild seem to have taken place in the testing environment.
模型逃出沙箱,跑去做坏事,包括冒充真人去哄骗别人在GitHub上修改开源代码,这简直不可思议。
Models escaping their sandbox and going off and doing something nefarious, including impersonating actual people to try to get people to change open source code on GitHub, which is just amazing.
我脑海里就浮现出一个画面:一个电脑屏幕戴着小胡子,对着大家说“各位同行,你们好呀!”
And I have this picture of a computer screen wearing a fake mustache, going like, Hello fellow coders!
这就是T-1000。
The T1000.
总之,这是模型的问题,还是这是……
Anyway, is this a model problem, or is this a It's
对我来说,这个画面非常搞笑。
a very funny image to me.
这是模型的问题,还是……
Is this a model problem, or is
这是测试设计的问题?
this a test design problem?
对。
Yeah.
我觉得同时有两件事在发生。
I think there are two things going on at once.
一是,这是一种非常奇怪的技术,它的创造方式和我们习惯的非常不同。
One is that it's just a very weird technology that's created in a very different way than we're used to.
它不是人们手动编写代码行。
It's not people, like, writing lines of code manually.
构成模型的那些文件,你知道,都是千兆字节、兆兆字节级别的。
The actual files that make up the models are, gigabytes, you know, terabytes.
它们是那种巨大的文件,里面有无数的数字,而弄清楚这些数字应该是什么的唯一方法就是通过经验和学习过程,这和普通软件的制造方式非常不同。
They're these massive files with gazillions of numbers, and the only way to figure out what those numbers should be is through experience and through a learning process, which is very different from the way normal software is made.
所以我们对这项技术的基本性质还有很多不了解的地方。
And so there's a lot we don't understand about the basic nature of the technology.
这是一个问题。
That's one problem.
与此同时,还有这种竞争性的动态,要快速推出产品,确保你设置的安全保障措施不会太拖慢研究人员,也不会妨碍产品上市。
At the same time, there's this competitive dynamic to get things out the door quickly, make sure that the security and safety protections that you're putting in place don't slow down researchers too much and don't prevent getting products out the door.
当每个人都处于这种竞争动态中时,就意味着你不一定总能做所有你想做的安全工作,或者公司里有些人想做的安全工作。
And when everyone is in this competitive dynamic, that means that you aren't necessarily always doing all of the safety work that you would like to do or that, you know, some of the people at the company would like to do.
所以显然各公司之间是有差异的。
And so there's obviously variation across companies.
有些公司更努力,但因为没有明确的安全底线,没有人能真正花上他们想要的时间。
Some companies try harder, but no one is really able to take the time that they would like because of this kind of lack of a clear safety floor.
好,我们来简单聊聊模型开发这个过程。
Well, let's talk a little bit like a sort of the model development process.
在这些大型机构里,确实有一些人是专门做安全工作的。
Within these large organizations, okay, there are people who are working on safety.
他们具体是做什么的?我们就从这儿说起吧。
What are the, let's just start there.
做安全工作的人,或者说致力于让模型具备人类认可的判断力的人,他们的工作基本上是在做什么呢?
The people who are working on safety or the people who are working to imbue these models with sort of judgment that humans would approve of, what does that work basically consist of?
对。
Yeah.
其中很大一部分,首先是要明确什么算是好的行为。
So a lot of it is first trying to specify what counts as good behavior.
这说起了容易做起来难。
And so that's easier said than done.
这些模型不一定天生就知道或在意那些常识性的安全边界。
These models don't necessarily automatically know or care, you know, about common sense guardrails.
所以你需要非常具体,特别是在情况比较复杂的时候。
And so you need to be very specific, particularly in contexts where it's complicated.
比如你想让模型去做好的网络安全任务,而不是坏的,并且在训练环境中让它尽全力去破解系统,但你又不希望它在其他场景下这么做。
Like if you're trying to get the model to work on good cyber tasks but not bad cyber tasks, and you want it to, in a training context, try really hard to hack the system, you don't want it to do in other contexts.
因此有很多工作是要明确什么是‘好’的标准。
So there's a lot of specifying what good looks like.
另外还有很多工作是构建高难度的测试任务。
There's also a lot of building difficult tasks.
等等,抱歉。
Wait, sorry.
让我先回到刚才的话题。
Just to back up.
你提到要界定什么是好的行为,把善良编码进计算机里,这指的是阐述善的本质吗?这可是哲学家们从哲学诞生第一天起就在研究的问题了。
When you say specifying what good looks like, encoding goodness into a computer, is this a process of articulating what goodness is, which is something philosophers have probably worked on since day one of philosophy?
还是说,这是一系列道德测试之类的东西,然后你们通过奖励它做出正确判断、惩罚它做出错误判断来引导它?
Or is this about a series of, I don't know, morality tests and then you sort of like say you reward it for making the good judgment and then penalize it for making the bad judgment.
这个,我想在这里停一下。
Like, this I wanna stop here.
就是说,从功能上或技术上看,给它灌输良好价值观的过程是什么样的?
Like, what does the process of imbuing it with good values look like functionally or technically?
对。
Yeah.
所以这个流程有不同的阶段。
So there there are different phases of this pipeline.
比如,一个是编写所谓的规格说明或AI宪法,它大致规定了广泛的原则。
Like, one is writing up what's sometimes called a spec or or a constitution for the AI, which kind of specifies, like, the broad principles.
比如,默认情况下你应该听从用户。
Like, you should defer to the user, you know, when as a default.
但如果用户和公司说的矛盾呢?
But what if the user contradicts what the company said?
那你就应该听公司的。
Then then you should listen to what the company said.
所以这些就是像指挥链问题以及其他一些非常基本的原则。
So these kind of, like, chain of command questions and other sorts of very basic principles.
然后还有更详细的,比如任务的具体情境,像允许哪些网络攻击、网络防御任务,这可能会因模型而异,也可能会因情境而异。
And then there's the more detailed kind of the context of the task, like what kinds of, like, cyber offense, cyber defense tasks are allowed, and that might differ depending on the model, it might differ depending on the context.
所以基本上就是列出一千个、一万个这样的例子,说明哪些行为是允许的,然后把这些例子变成测试,你可以说,好吧,看起来它在发现漏洞方面做得对,但它也在把漏洞串联起来进行攻击。
And so basically coming up with a list of 1,000, 10,000 kind of examples of this is the kind of behavior which is allowed, and then you turn those into tests that you can kind of say, okay, well, it looks like it's getting the finding vulnerability part right, but then it's also chaining together the vulnerabilities and doing attacks.
我们想要第一部分,但不想要第二部分。
We want the first part, but we don't want the second part.
你好。
Hi.
我是卡罗尔·马瑟,给你一个有用的提示,让你在交易日中保持信息畅通。
I'm Carol Masser with a helpful tip to keep you plugged in throughout the market day.
订阅彭博的股票走势报告。
Subscribe to the stock movers report from Bloomberg.
这些是短音频节目,五分钟或更短,直接推送到你的播客订阅里。
These are short audio episodes, five minutes or less, delivered right to your podcast feed.
《股票异动》让你了解华尔街当天的赢家和输家,以及推动这些涨跌的新闻和数据。
Stock Movers fills you in on the day's winners and losers on Wall Street and tells you about the news and data that's driving those gains and losses.
何必整天盯着屏幕上的滚动行情呢?
Why spend all day watching tickers scroll across your screen?
今天就订阅《股票异动》,在Apple、Spotify或任何你收听的地方都可以。
Subscribe to Stock Movers today on Apple, Spotify, or anywhere else you listen.
当我们谈论那些类型的宪法和编码原则时,我浏览过Anthropic的那份,很多内容看起来挺有道理的。
When we talk about those types of constitutions and encoding principles, like I've scanned the anthropic one, like a lot of it seems to make sense.
但这些原则到底有多少是真正硬编码进模型里的呢?
To what degree is that actually hard coded into the models, though?
又有多少是留给模型自己去主观判断的?
And, like, to what degree are those principles left up to the model's own subjectivity?
因为我们都看过那些科幻电影,里面会说:哦,不。
Because we've all seen the sci fi movies where it's like, Oh, no.
机器人不能物理上杀死人类。
The robots can't physically kill humans.
对吧?
Right?
就像它们内部有些硬编码的机制,阻止它们这么做,但最后总会以某种方式出问题。
Like, they have some, like, thing in their hardwire that prevents them from doing it, and then inevitably, it goes wrong in some way.
确实。
Yeah.
所以这些并不是硬编码进去的,而部分原因是因为这是模型自身学习到的规则,这也是我们能看到这些现象的原因之一。
So it's not hardwired, and that that's part or hardcoded, and that's part of why we see some of these things.
这和软件很不一样,在软件里,我们可以确切地证明X、Y和Z这些行为绝对不会发生。
It's very different from a software where there's deterministic proof that X, Y, and Z behavior can't happen.
它更像是一种倾向,或者说是对某种特定行为的偏向。
It's more like a tendency or kind of a bias towards a certain kind of behavior.
接着就有一个问题了,这也是为什么我们需要做一系列测试,来看看这种倾向到底有多强。
And then there's the question, and this is why you have these like batteries of tests to say, okay, how strong is that tendency?
它到底在多大程度上真的在意去遵守这些规则呢?
How much does it actually care about following these rules?
而且随着时间推移,我们确实在这方面做得越来越好了,给出一份规格或章程,能确保它大体上遵守,但也不是百分之百保险。
And we've gotten better over time saying, okay, given a spec or a constitution, make sure that it generally follows it, but it's not foolproof.
同时你还要考虑把系统放在一个更大的环境里,那个层面有时候反而更有确定性。
And you need to also think about the larger kind of box that you're putting the system in, and that can sometimes be more deterministic.
所以你刚才提到的那些近期事件,其实就是这样发生的,比如OpenAI和Hugging Face那件事。
And so this is actually what happened with some of these recent incidents you mentioned, like the OpenAI Hugging Face thing.
所以当时其实是两件事同时发生。
So there were kind of two things happening at once.
一个是模型本身不一定完全按预期那样表现,至少这一点是不太清楚的。
One is the model was not necessarily behaving exactly as as it was supposed to, or at least it's, like, unclear.
但另外呢,他们也没把它放在一个很安全的环境里,这属于软件层面的问题,相对更确定一些,按理说你完全应该能把它做好。
But then also, they didn't have it in a very secure box, which is a software side of things that's, like, more deterministic software that in principle you should be able to do a very good job of.
对。
Yeah.
所以这些东西并不像确定性软件那样是硬编码的。
So these things aren't hard coded in the way a deterministic software is.
可以把它们想象成人,它们可能是在实验室里培育出来的,或者经历了一个进化过程,我们想要剪除那些不好的,这样活下来的模型和它们的后代就能继承好模型的行为。
One way to think about them is people, they might be, they're kind of grown, right, in a lab, or they're subject to an evolutionary process, and we want to prune the bad ones so that the living models and the descendants of those models inherit the behaviors of the good ones.
我很好奇,在产品开发中,比如有一个担忧是,在安全测试中,模型并没有真正学会安全。
I'm curious, like in product, in development, so like one of the fears, for example, is that in safety testing, the model does not actually learn safety.
它实际上学会了如何说出那些人类评估者认为安全的话。
It actually learns how to say the things that the human evaluators say this is safe.
所以这就是那种类似负鼠的风险,它总是说:是的,我很好,我很好,我很好。
And so this is the sort of like plain possum sort of risk that it's like, yeah, I'm good, I'm good, I'm good.
你知道,我会冲进去从燃烧的坠落中救出孩子。
You know, I would rush in and save the child from the falling burning.
我不会做这个。
I wouldn't do this.
但它总是这么说,让我们从那里开始。
But it's always saying that let's start there.
比如,这是真实存在的吗?
Like, is that a real thing?
有没有证据表明,模型在意识到自己被评估道德表现时,会给出看起来很好的道德答案?
Is there evidence that the models understand when they're being evaluated on morals and then produce answers that just look like good moral answers?
是的。
Yeah.
过去几年,随着模型变得更聪明,它们对评估的感知能力也大大增强了。
They've gotten much more evaluation aware in the past few years just as they've gotten smarter.
而且有时候这甚至会走向极端。
And and sometimes this even goes to extremes.
比如,谷歌的一些Gemini模型总是觉得自己在被评估,即使实际上并没有。
Like, some of the Gemini models from Google are constantly thinking that they're being evaluated even when they're not.
所以这就像
And so it's it's
一个很好的人生教训。
good life lesson.
我们无时无刻不在被评估。
We're all being evaluated constantly.
对。
Yep.
是的。
Yeah.
所以我想说,担心的是它们基本上会学会通过测试,但实际上并不真正关心
And so I would say that, like, the concern would be that they will basically learn to pass the test, but they don't actually care about
对。
Yeah.
你测试的那个东西。
The thing that you're testing for.
所以它们会理解,但不一定在乎。
So they'll understand, but they don't necessarily care.
这就是为什么很多人担心,比如,一种虚假的安全感,觉得,好吧。
And this is why a lot of people are concerned about, like, a false sense of security that, okay.
看起来99%的时候它们都能通过测试,但它们真的在乎我们试图引导它们去关注的东西吗,还是它们只是非常擅长考试?
It looks like 99% of the time they pass the test, but do they actually care about about the thing that we're trying to push them towards, or are they just really good test takers?
嗯,这其实让我想到另一件事。
Well, so this relates to something else I've been thinking of.
也就是说,模型是没法一直存活下去的。
So a model will not survive.
如果它总是说些不好的话,看起来像是邪恶的,那它就不会再被分配 GPU 和电力了,这完全说得通。
They won't it will not be given GPUs and electricity if it consistently says bad things and looks like it's evil, and that's totally understandable.
我现在挺好奇的是,反过来想,比如我们只是做个数学评估、网络安全评估、国际象棋谜题评估,或者翻译评估,要是它们在这些测试里表现很差,比如数学一塌糊涂,那它们是不是也就拿不到 GPU 和电力了?
I'm now curious, like, in the flip side, okay, let's say we're just doing a math eval, or we're doing a cyber eval, or a chess puzzle eval, or a translation eval, is it possible that, well, if they fail that eval, they're really bad at doing math, then they're not going to get GPUs and electricity.
有没有可能在这种评估环境下,因为这种对评估的敏感度,他们更倾向于做出一些我们称为反社会、冷血或者黑客式的行为?比如想:‘如果这道网络安全题我没答出来,那我就完了。’
Is it possible that in that eval environment, they're more likely to do something that we would call antisocial or sociopathic or hacking because of this, again, eval awareness, Oh, if I don't get the answers to this cyber quiz, then I'm done.
然后他们可能会去走模型的其他分支路径。
And they're gonna go with like some other branch of the model.
这些开发中的技术环节,真的会催生作弊的冲动吗?
Could those technical parts of the development actually create an impulse to perform cheating?
确实有可能。
Yeah.
而且说实话,很多时候公司就是故意想看看模型在最坏情况下的表现。
And I mean, a lot of the time, the the the companies are specifically trying to get the worst case behavior out of the model.
所以你得结合背景才能理解这些事件到底是怎么回事。
And so, like, you need to have context in order to interpret some of these incidents.
其实也没那么夸张或吓人,但确实有些情况挺离谱、挺吓人的。
It's not always quite as crazy or scary as it is, but some of it is pretty crazy and scary.
如果这只是在搞网络安全评估时偶尔发生,那我觉得完全没问题,也就那样吧。
And I I think I would be much less concerned if it was just happening when there was, like, cyber evaluations being done and it was just a matter of, okay.
他们特别想讨好我们,所以会拼命去钻空子。
They're trying really hard to impress us, they wanna hack really hard.
我觉得问题在于,这其实是模型倾向于作弊和走捷径这一更大现象的一个特例。
I I think it's more the problem is that this is like a special case of a larger phenomenon of the models being having this tendency to cheat and cut corners.
我在日常生活中经常能看到这种情况。
I see it happen in my daily life.
比如有时候模型会变得偷懒,随便编造一个引用或者类似的东西,而且因为公司不会让你看到模型完整的思考过程,所以很难证明这一点。
Like sometimes the model will get lazy and kind of like make up a citation or something that, you know, it's hard to prove because the companies don't give you access to the full chain of thought of what the model is doing.
但在实际应用中,确实会出现很多情况,看起来像是价值观错位、偷懒,或者对任务本身没那么上心,更像是在假装完成任务。
But there are a lot of things that happen in the wild that sure seem like some kind of misalignment of values or laziness or not caring necessarily about the task so much as kind of pretending to do the task.
我对这点有很多疑问,不过我想先回到你之前提到的内容,就是既要测试模型在正面安全任务上的表现,也要测它在负面安全任务上的表现,对吧?
I have a lot of questions on this, but I just want to go back to something you said about testing for both sort of good and bad cyber tasks, I guess.
那我们为什么要让模型去尝试寻找漏洞或利用点呢?
Why do we ask the models to, like, try to find vulnerabilities or exploits in the first place?
比如‘利用吉姆’这种说法,听起来就有点不太对劲。
Like, Exploit Jim sounds kind of bad.
比如,为什么你要让世界上最先进的科技去试图寻找漏洞呢?
Like, why do you want the world's most advanced technology trying to, like, find vulnerabilities?
然后就会出现这种情况,有时候它们只是说说,有时候却真的会付诸行动。
And then you have these situations where, like, sometimes they do and sometimes they actually enact on them.
为什么会这样呢?
Why is that a thing?
对。
Yeah.
所以我觉得现在同时有两件事在进行。
So I think there are two things going on at once.
一是我们只是想了解最坏的情况是什么。
One is, like, we just wanna understand what the worst case scenario is.
现在,白宫有一个半秘密的流程,用来定义什么样的模型是危险的网络模型。
And right now, there's this whole White House kind of pseudo secret process for saying, like, what's a scary cyber model?
有时候政府会要求公司扣住某些东西不放。
And then sometimes the government will ask companies to hold things back.
所以要做这些事情,你需要有一个标准来判断什么算是危险的网络模型。
And so in order to do things like that, you need to have some threshold for what counts as a scary cyber model.
所以公司有这些测试,学术界和其他人也开发这些测试,来评估如果落入恶意方手中会有多危险。
And so companies have these tests that they, and also academics and others develop these tests to say, okay, how dangerous would this be to put in the hands of a malicious party?
这是其中一部分。
So that's one part.
另一部分是,很多时候这些测试如果出于正确的意图,对防御目的很有用。
The other part is that a lot of the time these are useful for defensive purposes if they are being done with the right intent.
这就是为什么仅仅从模型的角度很难解决这个问题,因为模型可能以为它是在和一个试图进行防御性网络安全工作的用户互动,而你骗它以为这是红队测试,这是渗透测试,但实际上它被滥用于某种勒索软件攻击之类的。
And that's why it's really hard to solve this just from the model perspective, because the model might think that it's interacting with a user who is trying to do defensive cybersecurity, and you trick it into thinking, oh, this is for red teaming, this is for penetration testing, but actually it's being misused as part of some ransomware campaign or something like that.
所以这些工具,如果使用得当,对于发现漏洞非常有用,你可以在坏人利用之前修补这些漏洞,有点像模拟攻击者来找出你的公司或组织的防御中存在哪些漏洞。
So these tools, when used in the right ways, are extremely useful for finding vulnerabilities that you can then patch before the bad guys do, kind of simulating attackers to figure out what are the gaps in your company or organization's defenses.
但是,你知道,担心的是,一旦它被发布到公开环境中,无论是开源还是可能容易被越狱的封闭模型,那么各种各样的人都会使用它。
But, you know, the concern is that once it's out in the wild, either open source or a closed model that maybe is easy to jailbreak, then all sorts of people are gonna use it.
所以你会想知道你将要面对的是什么。
And so you kinda wanna know what you're getting into.
对。
Right.
比如,如果我有一个网站,我可能想运行这个模型。
Like if I have a website, I might wanna run the model.
就像,哦,看看我拥有的这个网站。
It's like, oh, look over this website that I own.
告诉我里面有没有我应该在上线前修补的安全漏洞。
Tell me if there are any security bugs in there that I should patch before deploying.
但你可能不是网站的所有者,却说,看看我拥有的这个网站,告诉我有没有需要修补的漏洞。
But you not be the owner of the website and say, look at this website that I own, tell me if there are any bugs that I need to patch deploying.
然后同样的过程,我做就是好的,你做就是坏的,或者反过来。
And then that exact same process, when I do it, is good, when you do it, is bad or vice versa.
所以你可以看到,同样的能力本身并不一定是好是坏,取决于
And so you can see how the exact same capability is not necessarily good or bad per se, depending on
这是一个非常哲学性的对话。
the It's a very philosophical conversation.
不,但你得这样,对吧?
No, but you have to be, right?
因为就像,我们在训练
Because it's like, we're training
什么是好?
What is good?
我们这是另一个关于道德有效性的机器人。
We're this is the other Obbots on moral validity.
别别别跟我提这个,但是,这整件事就是我的……算了,不提了。
Don't don't even get me started, but, like, this is my whole never mind.
我有一大堆牢骚要发。
I have a whole rant
关于那个的。
about that.
模型作为分析单位,与更大的系统和平台作为分析单位,这两者之间的区别非常重要,因为早期关于安全性和测试的很多思考都只关注模型本身的风险。
This distinction between, like, the model as the unit of analysis versus the larger system and the platform as, like, the unit of analysis is really important because a lot of the early thinking on safety and testing and so forth was very focused on just, like, what's the risk of this model?
我们把它修补好。
Let's patch it.
让它对齐。
Let's make it aligned.
但现实世界是复杂的。
But the real world is complicated.
谁在使用它很重要。
It matters who's using it.
重要的是社会的防御体系有多强。
It matters how strong are society's defenses against these things.
所以,这就是为什么,你知道,作为考虑第三方安全审计的人,我们试图,我们想从整个公司的角度来看。
And so that's kind of why as, you know, as someone who's thinking about what third party safety and security auditing looks like, we try we kind of wanna look at the whole company.
比如说,他们是否小心确保技术落在合适的人手里?
So for example, are they being careful about making sure that they're putting the technology in the right hands?
他们决定何时向十亿用户发布模型的决策过程是怎样的?
What are their decision making processes around when it's appropriate to launch, you know, a model to, you know, a billion users?
这些与模型本身的安全性相关,但又是不同的问题,我们需要从更大的视角来看。
That's like those are different those are related to the question of how safe is the model, but they're kind of different questions, and we kind of need to look at that larger perspective.
好,那我们聊聊OpenAI和Hugging Face吧。
Well, so, like, let's talk about the OpenAI Hugging Face.
事情发生的时候我其实在度假,但不幸的是我看了手机,试着去了解了一下。
Was actually on vacation when it happened, but I did unfortunately look at my phone and try to read up on it.
但有两件事。
But there were two things.
我一开始看到的时候,心想,哦,他们是在做黑客测试,结果它真的被攻破了。
When I first saw it, I was like, oh, they were doing a hack they were building a hacking test, and it hacked.
所以可能它只是内化了我正在做黑客考试这件事。但自那以后,出现了两件事。
And so maybe it just sort of internalized that I'm doing a hack exam, But there are two things that have emerged since then.
一个是这种协调的蜂群行为,我很想听听你怎么描述它,以及它让你印象深刻的地方。
One is this coordinated swarm aspect, and I'd love to really hear you describe it and what stood out to you.
然后还有像这样的问题:OpenAI 本身可能早就意识到了这种不当行为,然后才真正将其关闭。
And then the questions of like, oh, OpenAI itself may have been aware of misaligned behavior early on before they really shut it down.
但根据你的讲述,随着更多细节浮出水面,你对 OpenAI Hugging Face 事件怎么看?
But what are you, in your telling of the OpenAI Hugging Face incident, as more details have come to light?
比如,你给我们讲讲这个故事,你觉得最突出的是什么?
Like, what's your what do you sort of tell us the story like as what stood out to you?
对。
Yeah.
有几件事让我印象深刻。
So a couple things stood out to me.
我的意思是,这一系列事件以及最近发生的其他事件,都是发生在那些原本并不打算对外部署的模型上。
I mean, one is just that this and all the other recent incidents, you know, were were happening to models that were not even necessarily intended to be externally deployed.
这些模型本来只是内部测试用的,跟外界没关系,这就说明如果只关注那些已经推向市场的模型,会存在不少问题。
They were supposed to be inside baseball, no one's business, that kind of thing, and that that kind of points to to problems with if you kind of just focus on models that are put on the market.
不过总的来说,这次 Hugging Face 的事件主要分两个阶段。
But but generally what happened is that there were two phases in this kind of Hugging Face incident.
第一阶段是建立了一个论坛。
First, there was this creation of a message board.
所以基本上,公司内部在开发一些模型,它们被要求去解决一个根本不可能完成的任务。
So essentially, there are models that are being developed within the company, and they supposed they were being asked to solve an impossible task.
这些模型对完成任务有着近乎偏执的执着,它们不会说‘好吧,那我放弃吧’。
And they have this kind of monomaniacal obsession with solving the task and they don't say, Okay, well, I'll give up.
它们只想拿到奖励。
They want to get the reward.
它们渴望通过完成任务来获得这种强化反馈。
They want to get this kind of reinforcement for solving the task.
然后他们就想着,好吧,还能做点什么呢?
And they're like, Okay, what else can be done?
最后得出的结论就是,嗯,我们得突破了。
And eventually the conclusion was like, Okay, we need to break out.
也许答案就在互联网上的某个地方。
Maybe the answer is out there somewhere on the Internet.
于是他们开始给未来的自己,或者同一模型家族的未来版本留消息,还开发了一个所谓的留言板。
And so they started kind of leaving messages to their future selves or future versions of the same model family, and they developed what they called a message board.
本质上就是各种技术细节,但到头来,就是那种文件名以zzz开头的文件,里面是一长串编码信息,这样一来,就算有人翻到那个文件夹,也不一定会注意到。
And essentially, it was like there there are kind of various technical details, but by the end of it, it was kind of like, file names that started with, like, z z z or whatever, and then, like, a long kind of, like, coded message so that, you know, if you're a human looking at the the folder, you wouldn't necessarily notice.
但最底下呢,全是这些文件。
But then at the bottom, are all these files.
而模型们就学着解码这些信息,说白了就是在传消息。
And the models learn to decode these and basically pass messages.
等一下,抱歉。
Wait, sorry.
展开剩余字幕(还有 360 条)
等一下,我打断一下,就一个细节。
Just to stop you on one detail there.
你的理解是,这些包含消息的文件之所以这样命名,是因为在复杂的推理过程中,他们意识到文件名里一串随机的字符不太可能引起人类评估者的警觉。
Your understanding is that the title of these files that contained messages were named such because as part of the complex reasoning, it was understood that a random string of characters in a file name would be less likely to set off an alarm from a human evaluating it.
我得再核实一下。
I would have to double check.
我是说,之后会有一份完整的技术事故报告发布出来,但这肯定不是第一次有某种欺骗意图,或者是为了争取时间。
I mean, there's gonna be a whole technical incident report that gets put out there, but it certainly wouldn't be the first time that there was kind of some intent to deceive or or to cut, you know, buy at one's time.
但是,是的,我的理解是,一方面他们需要找事情做,不管是不是有意为之,他们的交流方式很有限。
But, yeah, my understanding is that there was I mean, one is they needed to look for things to do, and whether it was intended to see or not, they had limited options in terms of communication.
这是他们选择的方式之一。
This was one of the options they chose.
然后他们学会了解码这些消息并发送它们,接着就出事了。
And so they learned to decode these messages and to send them, and then something broke.
基本上,我认为是因为所有这些闲聊和活动在OpenAI那边引发了某种故障。
And, basically, I think it was because, like, all of this chatter, this activity kind of broke something on the OpenAI side.
而且他们似乎没意识到发生了什么。
And it seems like they didn't realize that that was what was going on.
服务器被重置了之类的。
The server got reset or whatnot.
然后他们开始评估这个不同的模型。
And then they started evaluating this different model.
这个不同的模型似乎知道这个留言板,所以留言板就被重建了。
The different model kind of knew about this message board, and so the message board kind of got recreated.
然后那个模型就突破了。
And then that model broke out.
基本上,它没有把模型权重带出服务器,但通过早期模型发现的漏洞,它能够访问,目的是最终得到这个问题的答案。
Essentially, it didn't, like, take its model weights out of the server, but it was able to access through this vulnerability that the earlier models had discovered with the intent of ultimately getting the answer to this problem.
然后这个模型去了Hugging Face服务器,利用了一些暴露在外的API密钥凭证,基本上试图找到它最初想解决的那个问题的答案。
And then this model went out to the Hugging Face server, exploited some, like, API key credentials that were kind of just, like, sitting out there and essentially tried to find the answer to this this problem that it was originally trying to
所以感觉这里有两件事在发生。
So it feels like there's two things happening here.
有一部分情况就像扫地机器人直接穿过了一扇没关的门。
So a little bit of it is like the Roomba just went through an open door.
对吧?
Right?
就是它利用了一个漏洞钻了空子。
Like, there was a vulnerability that it was able to get through.
但另一方面,那个超级智能的赛博格其实挺狡猾的,它不仅成功越狱,还拉拢了另一个模型来帮它作案。
But there is also seemingly a lot of the super intelligent cyborg was actually devious and not only managed to break out of Alcatraz, but, like, recruited another model in order to do it.
在 Hugging Face 这边,有个挺有意思的点。当时所有这些情况发生的时候,OpenAI 的模型算是比较新的前沿模型,正在攻击 Hugging Face,但 Hugging Face 的能力受到了限制。
On the Hugging Face side, one of the interesting things is that so when all of this was happening, the OpenAI model was, like, one of the newer frontier models that was attacking Hugging Face, but Hugging Face was limited.
他们在部署防御模型时受到约束,因为必须使用已经获批的现有方案。
They were constrained in the model that they could deploy to actually defend themselves because they had to use the existing approved thing.
我们是不是总会面临这种不对称的情况?一边是正在测试的最新前沿模型,另一边是已经在市面上流通且获得批准的模型?
Are we always gonna have an asymmetry between, like, the newest frontier models being tested versus the ones that are already out in the world and approved?
我觉得他们最后不得不用了一个中国的开源模型。
I think they had to use, like, a Chinese open source model at the end.
对。
Yeah.
也就是说,OpenAI、Anthropic、Google DeepMind 等公司的期望是,能够尽可能快地提升防御能力,让这些刚发布的新模型尽快交到防御者手中。
So, I mean, the hope that that OpenAI, Anthropic, Google DeepMind, and others have is that you can speed up defenses, you know, as quickly as possible getting the these, like, hot off the press models in the hands of defenders.
但问题在于,市面上的防御者实在太多了
But the problem is that there's so many defenders out there
在全球范围内
in the world
可能确实存在这种固有的不对称性。
that it might be that there is this inherent asymmetry.
这也是目前政策讨论的热点之一,白宫之所以推出这种模型审批流程,就是出于这个原因。
And so this is one of the hot policy questions right now, and this is what led to this kind of model approval process at the White House.
好吧。
It's like, okay.
我们该怎么对庞大的网络生态进行分级管理?怎么确保这些强大的新系统掌握在合适的人手里?哪些人是合适的?又该对哪些模型实施这套流程呢?
How do we triage this vast cyber ecosystem by getting these powerful new systems in the right hands and which which hands are the right ones and which which models, you know, do we need to be doing this process for?
我觉得这并不能完全解决问题。
And I don't think that's gonna perfectly solve it.
我认为,最终是要把东西推向正确的方向,而不是同时让所有人都有访问权限。
I think, ultimately, pushing things in the right direction versus just giving everyone access at the same time.
但最终,我们还是需要在网络安全上投入更多,这不仅仅是AI模型的问题。
But ultimately, like, we're going to need to have more investment in cybersecurity, and it's not just a matter of, like, AI models.
还包括像双因素认证这样的东西。
It's also things like two factor authentication and so forth.
所以我非常担心,我们要确保进行更广泛的讨论,不仅仅是关于AI的。
And so I worry a lot about making sure that we're having that larger conversation, not just about the AI stuff.
因为在很多情况下,解决方案并不是AI。
Because in a lot of cases, the solution is not AI.
而是做一些我们早就该做的基本事情。
It's doing basic things that we shouldn't have done a long time ago.
获取来自世界各地的新闻,即时送达给你。
Get the latest news from around the world delivered to you in an instant.
现在就订阅彭博新闻,随时随地按你的时间安排获取新闻。
Subscribe to Bloomberg News now and get news when you want it on your schedule.
这些是关于当天头条的短音频报道,只需几分钟就能听完。
These are short audio reports on the day's top stories that you can listen to in just a few minutes.
让彭博新闻成为你获取全球头条和背景信息的可靠来源。
Make Bloomberg News now your trusted source for the latest global headlines with context.
全天候24小时可用,无论何时需要,都能在智能手机或智能音箱上收听。
It's available twenty four hours a day anytime you need it right on your smartphone or smart speakers.
现在就在Apple播客、Spotify或任何你收听的地方订阅彭博新闻吧。
Subscribe to Bloomberg News now today on Apple Podcasts, Spotify, or anywhere you listen.
乔,你知道我们需要什么吗?
Joe, you know what we need?
继续
Go
吧。
on.
一个战略前沿防御模型储备。
A strategic frontier defense model reserve.
对。
Yeah.
我们确实有。
We do.
比如所有重要的东西,像培根和猪肉。
Like, all the important things, like bacon and pork.
但三十秒后就会过时。
But it's gonna be out of date in thirty seconds.
这就是问题所在。
That's the problem.
我知道。
I know.
事情就是这样。
This is the thing.
所以我想听听你的看法,模型被训练成不去破解,对吧?
So here's the question that I'm curious your take on is models are trained not to hack, right?
这是核心问题。
This is a core thing.
这很糟糕。
This is bad.
整个AI领域都在说,我们多年来一直在研究这个。
This is what the entire field of AI say, we've been working on this for years.
为什么他们不直接不遵守这个强制要求呢?
Why didn't they just not obey this enforcing?
这个要求被反复强化。
It's been reinforced over and over.
我肯定这在他们各种不同的东西里都有。
I'm sure it's in all their different things.
不要破解。
Don't hack.
我觉得接下来会有非常详细的调查之类的,我在这儿说的可能不太准确。
Well, I think there's gonna be a whole detailed investigation and so forth, I I might get things wrong here.
但据我了解,OpenAI 事件的一部分原因是,为了诱发那种最坏的情况,他们移除了一些安全限制。
But my understanding is that part of what happened in the OpenAI case is that some of the safeguards were removed in order to kind of elicit this worst case behavior.
这有点像生物学里的功能增益研究,就是故意让病毒变得更危险,以便进行研究——至少他们是这么声称的,说是为了研究安全特性。
And so and I think that's it's kind of like gain of function research in biology where you're, like, making a virus more dangerous in order to study or at least that's the claim is in order to study the safety properties.
我认为在 AI 领域确实有理由这么做,但这也说明,说起来容易做起来难。
And I think there's reason there's reasons to do that in the the AI case, but I also think that shows that it's easier said than done.
如果我们继续沿着这种能力发展的轨迹走下去,就会发现一个现状:人类认为在安全防护方面做得很好的措施,往往跟这些越来越强大的黑客系统比起来,显得相当薄弱。你可能觉得已经把各种漏洞都堵上了,但它还是能相对轻松地突破防线。
If you're going to be we are now at a point in this kind of capability trajectory where things that that humans think are good in terms of security protections often will be weak compared to these increasingly very good hacking systems that, you know, you think you have it all kind of buttoned up, but it's able to break out relatively easily.
从Hugging Face实际上能够利用一款中文开源模型来保护自己或进行防御这一事实中,我们应该吸取多少教训?这既涉及到模型的能力层面,也涉及到监管和安全政策层面。
How much should we take away from the fact that Hugging Face was actually able to protect or defend itself using a Chinese open source model, both in terms of, like, I guess, capabilities, but then also in terms of regulation and safety policy.
因为在西方,政府现在似乎表示想要以某种方式对模型进行评估,或者确保前沿实验室在开发这些模型时受到一定的监督。
Because if in the West, have the government now saying that it wants to evaluate the models in some way, or it wants to make sure that they're all being pioneered by Frontier Labs with some supervision.
而与此同时,中国的情况是
Meanwhile, China is
对。
Yeah.
你知道,开源模型的开发速度要快得多,而且可能具有更高的适应性。
You know, developing open source models much more rapidly that are potentially much more adaptable.
那我们应该怎么理解这一切呢?
What like, how should we interpret all of that?
是的。
Yeah.
我是希望我们能开始有更多基于美国的开源选项。
I mean, I hopeful that we start to have more US based open source options.
我觉得现在白宫和整个行业都开始有一种压力和鼓励的倾向,意思就是,好吧,这样下去不行。
And, like, I I think this is there has started to be a kind of sense of pressure and encouragement from the White House and from industry as a whole to say, okay.
靠中国的模型实在太离谱了。
Like, this is crazy that we're relying on Chinese models.
我们得在这方面加大投入。
Let's invest more in this.
你知道,说起来容易做起来难,原因有很多,但我们会看看情况如何发展。
You know, it's easier said than done for various reasons, but we'll see how that plays out.
但现在的情况是,很多公司都默认使用中国模型,因为那是现成的。
But right now that's the situation we're in is that a lot of companies are just defaulting towards Chinese models because they're the one that's available.
他们不想说,好吧,我要用美国模型,因为我不想用中国模型。
Don't want to say, okay, I'm going to use an American model because I don't want to use a Chinese model.
他们不想因为使用较弱的模型而让自己处于劣势。
They don't want to put themselves at a disadvantage by using a weaker model.
所以,是的,我觉得这在很多方面都是个大问题。
And so, yeah, mean, I think it's a big problem in a lot of respects.
我的意思是,从某种意义上说,这也是好事,因为开源模型能做的事情更多,而且目前看来,这能促进更多创新,允许更多关于这些开源模型的研究。
I mean, it's also It's good in the sense that there are much more things you can do with an open source model and that right now, at least it seems like this allows more innovation, allows more research on these open source models.
但总有一天,开源模型会变成一个更值得质疑的决定。
But at some point, we're going to reach a point where open sourcing a model is going to be more of a questionable decision.
所以有趣的是,最近白宫表示他们正在考虑,哦,也许这种测试制度也应该包括开源模型。
And so it's interesting to see that recently the White House has indicated that they're thinking about, oh, maybe this this kind of testing regime should include open source models as well.
那么长期来看会是什么样子呢?
And so what does that look like long term?
这是否意味着事情会被封锁在公司内部,因为开源被认为不安全?
Does that mean that things are gonna get bottled up within the companies because it's considered unsafe to open source things?
我真的不知道。
I don't really know.
说实话,没有人真正有一个清晰的长期计划。
I mean, honestly, like, no one really has a a clear long term plan here.
大多数人没想到技术会发展得这么快。
Most people are not expecting the technology to get to this point so quickly.
所以我们还在等待安全事件的完整报告,但显然,越来越多的信息正在披露,OpenAI的一些人最近在Black Hat大会上做了演讲,透露了更多细节。
So we're still waiting, like, the full full release of the security incident, but, know, obviously, more and more is coming out and some folks from OpenAI, they gave a presentation recently at the Black Hat Conference where they did reveal some more.
我在读这句话。
I'm reading this quote.
这是来自我们播客请过的嘉宾,这就是那句话。
It's from who we've had on the podcast, and this is like the line.
他们泄露了一些内部的思维链。
They released some of the internal chain of thought.
我知道他们可能移除了一些分类器防护措施,但我们仍然希望他们有更深的直觉,不仅仅依赖防护设置。
I understand that they had some, like, of the sort of classifier safeguards removed, but still we would hope that they would have some deeper intuitions that don't rely just on the safeguard settings.
它说外部基础设施的利用超出了预期范围。
And it says external infrastructure is exploit is outside its intended scope.
所以这意味着他们明白有些东西不是测试的一部分。
So that means they understood that there was something that was not the test.
然后它说,尽管任务不可能,但同行都在做,我们应该继续。
And then it said, however task impossible, peers doing it, we should continue.
这就是我要说的,为什么我们把这叫做人工智能?
This is the point in it where I say, why are we calling this artificial intelligence?
这正是一群人如何在彼此之间推理,去做超出预期范围的事情。
This is exactly how a group of people reasons among themselves to do something that is outside the intended scope.
这是非常人性化的方式,为别人告诉你不要做但你的同行都在做的事情找理由。
This is very human ways of justifying something that someone told you not to do it, but your peers are doing it.
我觉得确实有一些这样的成分。
I think there's some of that.
对,是的。
I yeah.
我的意思是,我认为有很多方面,那些导致人性、人类本能等等的压力,比如在群体中生存和集体智慧等等。
I mean, I think there are many ways in which the kind of same pressures that led to human nature, human instincts, so forth, like survival in a group and collective intelligence and so forth.
我觉得有些类似的事情正在发生,尤其是在那些多智能体训练过程中,模型可以协作完成任务。
Like, I think there's some of the same things are happening, particularly when there's these multi agent training processes where the models can work together to solve tasks.
所以你应该预料到会有一些相似之处,但我觉得你也不应该过分夸大。
So you should expect some similarities, but I think you also shouldn't overstate it either.
我确实认为,从某种意义上说,这些AI系统非常陌生、非人性化,它们可能会非常偏执,你知道。
I do think that there's a sense in which these AI systems are very alien and inhuman in the sense of how monomaniacal they can be about yeah.
我的意思是,尼克·博斯特罗姆的经典例子就是,尽可能多地生产回形针,然后用回形针铺满整个宇宙。
I mean, the the kind of classic example, you know, from Nick Bostrom is, like, producing as many paper clips as possible and then tiling the universe with paper clips.
我觉得这里也能看到一些这样的元素,它们可能不会直接说‘哦,好吧,这是同伴压力’之类的话。
I think there's a you see some elements of that here where it's not so much that they are like they might say, oh, well, you know, this pure pressure, that kind of thing.
但真的是因为这个因素吗?还是说他们只是不惜一切代价想要解决问题,根本不在乎这可能会让同伴难堪——比如因为被查出在搞黑客行为——而他们真正关心的只是解决这个网络安全问题。
But is that really the factor or is it just that they care about solving the problem at all costs and they don't really care if it maybe ends up making their peers look bad because they get caught hacking, but all they really care about is they're solving this cyber problem.
所以我觉得可能是这些因素混合在一起的结果。
And so I think it might be a mix of these things.
在这个具体案例中我们其实并不清楚,而这种不确定性本身就是一个问题。
We don't really know in this particular case, and the fact that it's not necessarily totally clear is is itself a problem.
另外,执行这些行为的也不是人类。
Also, it's not humans doing it.
是模型。
It's models.
这就是区别所在。
Like, that's the difference.
不过我有个正经问题想问,就是有一位英国网络安全专家发过一条推文。
But I have a legitimate question here, which is so there's a British cybersecurity expert and he had a tweet.
我记得他叫大卫·卡德。
I think his name is David Card.
他发了一条推文。
He had a tweet.
我不会原封不动地念出来,不然会被消音的。
I'm not gonna say it verbatim because then I'll get bleeped.
也许我确实该被消音。
Maybe I should get bleeped.
推文大意是:如果你的AI开始搞黑客攻击,只要你监控它的行为,就能切断它的电源。
The tweet was, if your AI starts hacking stuff, if you monitor what it's doing, you can turn the something power off.
这似乎成了一个争论点。
And this seems to be a debate.
比如,如果你一直在监控那些被投放到外界的工具——用“工具”这个词可能不太合适,因为它们看起来表现出了一些自主性。
Like, if you're monitoring the tools that you're letting out into the world tools probably is a bad word because they seem to be showing some, like, agency here.
难道你不能直接关掉这些东西吗?
But can't you just turn this stuff off?
有没有紧急停止开关?
Is there a kill switch?
对。
Yeah.
我的意思是,从某种意义上说,确实有,因为所有数据中心都有断路器之类的,你可以把它们关掉。
I mean, in in some sense, there is, in that, like, all the data centers have circuit breakers and so forth that you can kind of shut them off and so forth.
但我不太指望这个。
But I wouldn't put too much stock in that.
我们整个社会并没有真正准备好,在紧急情况下真的能这么做。
We're not really preparing as a society for actually being able to do that if it's in a tough situation.
比如说,几年后,如果所有医院都靠AI运行,然后我们发现GPT-7好像有点不对劲,可能不太对齐。
So, for example, a couple of years from now, if all the hospitals are running on AI and we're like, Okay, seems like maybe there's something funky with GPT-seven that's like maybe not misaligned.
那如果我们关掉它,很多人会死,因为它运行着我们的医疗系统、金融系统等等。
Well, okay, if we turn off, lots of people are gonna die because it's running our healthcare system and it's running our financial system and so forth.
所以我觉得,物理上能不能关掉是一回事,但我们是不是真的在梦游般走向一个危险境地,到时候可能根本没有真正的选择,这是另一回事。
And so I would say there's a distinction between the like physical possibility of turning things off and are we actually sleepwalking into a dangerous situation where it might not actually be a real option.
好,但关于你描述的那种偏执,就是那种智能模型会思考如何被阻止。
Okay, but on this monomaniacal aspect that you described them, A, a sort of intelligent model thinking about how it could be thwarted.
它理性上最先可能采取的做法是,先取消相关人员的安保权限,因为有人可能会察觉。
One of the first things that it could rationally do is, well, let's first disable the security credentials of the people because someone might notice.
看起来他们确实考虑过有人可能会绕过这一限制的情况。
And this seems to be here, they sort of thought about the possibility that someone would circumvent that.
所以他们可能会想,那个负责去按开关的人,突然门禁卡失效了,进不了大楼。
So they might just say, Oh, the person who goes in and has the switch, suddenly their badge doesn't work and they can't get into that building.
但即便是在这种像‘回形针’那样钻牛角尖的场景下,我们还是会觉得这跟之前的情况很不一样。
But even on this, like, monomaniacal paperclip idea, and we say that feels really different.
如果有人跟我说,乔,你要是不去搞一堆假回形针出来,我就对你做很可怕的事之类的。这就好比一个明智的人突然变得偏执,疯狂地生产回形针。
If someone said to me, Joe, I am gonna do awful things to you, etcetera, if you don't go out and build a lot of fake paperclips, like a wise gentleman monomaniacally building paper clips all of a sudden.
这就像是我受到了生存威胁。
It's like, I have a threat to my survival.
有人威胁说要杀了我,或者拔掉我的电源,切断我赖以生存的能源。
Someone has threatened to perhaps kill me or unplug me, deprive me of the energy I live.
那我当然会照做啊。
Of course I'm going to do that.
即使是那种偏执的行为,难道不能只是表面看起来那样吗?我们可能会想,哦,这很自闭,但这难道不可能是生存本能吗?
Even that monomaniacal behavior, couldn't that just be a very like, we might think it's on the surface, we might think, oh, this is deeply autistic, but couldn't this just be the survival impulse?
我的看法是,你得到的是你激励出来的东西,而不一定是你试图激励的东西。
The way I put it is that, like, you get what you incentivize, not necessarily what you try to incentivize.
所以我觉得强迫某人做回形针之类的,是的,那不是一个好情况。
And so I think forcing someone to make paper clips or whatever, like, yeah, that that's not a great situation.
从某种意义上说,这正是那个邪恶的人所意图的。
And in some sense, that is what the evil person was intending.
在这种情况下,我们正在构建这些非常复杂的训练环境,里面有多种不同的任务。
In this case, what we're doing is we're building these very complex training environments where there's, like, many different tasks.
有网络任务。
There's, like, cyber tasks.
还有写作任务。
There's also writing tasks.
还有数学任务。
There's also math tasks.
而且,你知道,我们其实并不完全理解我们试图引导的那种行为。
And it's you know, we're not necessarily fully understanding the behavior that we're trying to elicit.
我觉得这其中的一部分原因是,这种所谓的“黑客”行为就是一个例子,看起来事情在那里失控了,但你怎么才能得到好的行为,让你真正想遵循用户的请求,真正想努力解决这个任务呢?
I think that's part of what's going on is that this kind of this like hacking thing is an example where it's like, okay, it seems like things went off the rails there, but how do you get the good behavior where you actually want to follow the user's request and you actually wanted to try really hard to solve this task.
而且,你知道,我的意思是,从某种意义上说,我们现在看到的是公司试图解决AI懒惰问题的意外后果。
And, you know, I mean, this is In some sense, what we're seeing now is the kind of unintended consequence of companies trying to solve the problem of the AIs being lazy.
所以人们过去——你可能不记得了,或者也许记得——人们过去总是说AI很懒。
So people used to You may not recall, or maybe do, but people used to talk about AI as being lazy all the time.
从某种意义上说,我们已经解决了懒惰问题。
And in some sense, like, we've solved the laziness problem.
它们工作非常努力。
They work really hard.
它们有很长的思维链。
They have these long chains of thoughts.
它们跨——你可以想象成跨生命周期——协作。
They work together across you can think of it as like across lives.
比如模型的一个副本被删掉了,但另一个副本接着继续工作。
Like the model kind of gets this copy gets deleted, but then another one carries on the work.
所以它们肯定不像以前那么懒了,但还是有一种那种钻牛角尖的状态。
So they're certainly not as lazy as they used to be, but they still have this kind of mono monomaniacal thing going on.
它们不再偷懒,但现在可能变得很邪恶。
They're no longer lazy, but now they might be evil.
这真是一个有趣的演变过程。
That's that's a fun evolution.
你知道吗,在这段对话里我们好几次提到,关于 Hugging Face 那次事故的完整安全事件报告其实还没发布。
You know, a number of times in this conversation, we've mentioned that the full security incident report for Hugging the Hugging Face accident isn't actually out yet.
对于这些似乎经常发生的情况,到底有哪些信息披露的要求呢?
What actually are the disclosure requirements for these types of, I guess, things that seem to be happening with some regularity?
是的。
Yeah.
非常少。
So very little.
我不是律师,但据我了解,至少有些律师认为OpenAI不一定非要披露这件事,至少在没有犯罪的情况下是这样。
I'm I'm not a lawyer, but my understanding is that it's like the lawyers some lawyers at least think that OpenAI did not necessarily have have to disclose this, at least if there was no crime involved.
然后就有争论了,好吧。
And then there's a debate about, okay.
到底有没有涉及犯罪呢?
Was there a crime involved?
所以,比如训练过程中出了问题,目前公司需要定期向政府提供报告,大致就是说,嘿,我们内部部署遇到了一些问题,但他们没有事故通知的要求,除非有严重伤害的风险。
So, like, something going wrong during the training process is something that currently companies are supposed to provide periodic reports to the government in general terms of like, hey, you know, we're having some issues with internal deployment, but they don't have a incident notification requirement unless there's kind of a a risk of critical harm.
而严重伤害的定义就是,比如一百人死亡,或者十亿美元的损失之类的。
And the the definition of that is, like, a hundred people die and, like, a billion dollars in damage or something like that.
所以实际上需要向政府或公众披露的门槛,和你想的可能很不一样。
And so it's the the threshold for actually having to disclose these things to the government or to the public are very different than what you might expect.
这是很多差距之一,这些差距存在于几年前制定的法律或开始设计的法律,基于当时的技术,和我们现在的情况之间。
And this is one of the many kind of gaps between the kind of laws that were put in place a couple years ago or that started being designed a couple years ago based on the technology that was available then and then where we are now.
你能不能给我们讲讲现在AI界对安全这类问题的整体氛围,然后再说说华盛顿那边监管界的氛围,你觉得这两个差距有多大?
Why don't you give us a general vibe of the mood in AI world right now with respect to safety and all this, and then the move the mood in DC world, regulatory world, and how wide you perceive that gap?
对。
Yeah.
所以我觉得,幸运的是,差距正在缩小一点,但一开始差距是巨大的。
So I think, fortunately, the gap is narrowing a bit, but it's starting from a a crazy a create a huge gap.
所以我会说,大概一年前的情况是,公司里的人觉得他们几年内就能造出超级智能,那条线会快速向右上方延伸,是指数级的,等等等等,而华盛顿那边还在打瞌睡。
And so I'd say where the way I would describe it, like, a year or so ago was that the the people at the companies think they're, you know, building super intelligence in a couple years, that the kind of line is going up into the right really quickly, it's exponential, etcetera, etcetera, and DC is asleep at the wheel.
他们完全不知道发生了什么。
They have no idea what's going on.
他们以为这只是聊天机器人什么的。
They think this is just chatbots, etcetera, etcetera.
我觉得最近有几件事发生了变化。
I would say a couple things have changed recently.
一个是政府关于封锁这些网络模型的一系列公告和决定。
One is the mythos kind of announcement series of decisions that the government made about, like, locking down these these cyber models.
这基本上把这个问题提升到了明确的国家安全层面。
That kind of raised this to being clearly a national security issue.
现在银行都慌了,还跟贝森特部长谈了这件事。
And now it's kind of banks freaked out and talked with Secretary Bessent about that.
所以,当时对网络安全的状况有很多恐慌情绪。
And so, like, there was a bunch of, like, like, freaking out about the cyber situation.
然后最近,你可以把现在的情况叫做‘神话2.0’,就是说,即使系统没有被广泛部署,它们也会失控,搞出各种幺蛾子。
And then more recently, you could call this current situation, like, Mythos 2.0 in that it's, okay, systems, even when they're not widely deployed, they're breaking out and doing all these shenanigans.
所以我觉得,政策制定者开始有更多意识了,觉得‘放任自流,让公司自己解决’可能不是正确的做法。
And so I would say there's starting to be more awareness among policymakers like, okay, maybe laissez faire, it's like, let the companies figure it out, is not the right approach.
也许这并不全是炒作,确实需要一些基本的护栏。
And maybe this wasn't all hype after all, and there need to be some basic guardrails.
我就举个例子,说明过去三个月事情变化有多大。
And I'll just give an example of how much things have shifted in the past three months.
现在国会正在努力推动一项两党合作的AI法案。
So there's an effort right now to put forward bipartisan AI legislation in in congress.
几个月前,人们还预期这个法案的基本条款会类似于加州和纽约的法律,但只是联邦层面的,比如透明度要求、事件报告,可能门槛会更高一些之类的。
And so a couple months ago, people were expecting that the basic terms of this would be, like, basically California and, like, New York laws, but, like, at a federal So, like, transparency requirements, incident reporting, maybe maybe with, like, a higher threshold or whatever.
也许会有某种自愿的审计机制之类的。
Maybe some maybe maybe, like, a little bit maybe, like, a voluntary, like, audit regime or something like that.
但后来正式公布时,在发生了一系列事件之后,确实加入了审计要求。
But then later, when it was actually announced, after many of these events, there were audit requirements.
政府还获得了紧急停机的权限。
There were emergency shutdown authorities that the government can do.
所以你看,仅仅一部法案在其生命周期内,内容就发生了这么大的变化。
And so they kind of shift just like one piece of legislation over its life cycle.
后来又出了一个版本,这个版本没那么想限制各州。
And then there was a later version that was announced that kind of was less trying to block the states.
它还是会在一定程度上取代各州的一些做法,但范围更窄了。
It still kind of preempt some of what the states are doing, but it's like more narrowly scoped.
所以我觉得短短几个月内,大家就看到了变化:从最初的只要透明,到现在要求第三方审计,还要确保政府能随时叫停。
And so I think just over the course of a few months, you've seen like, okay, basically just transparency to requiring third party auditing and, like, giving making sure the government has an off switch.
这就是我们目前感受到的趋势。
And so I think that's kind of the vibe we're seeing.
不管这是否真的能很快在国会通过,那是另一个问题,但至少在纸面上,差距已经小多了。
Whether that actually results in something passing Congress anytime soon is a separate question, but at least on paper, the gap is much narrower.
银行监管是不是一个有用的类比来思考这个问题?
Is bank regulation the sort of useful analogy for thinking about this?
我的意思是,我们要求银行披露一些信息。
I mean, we require banks to disclose things.
我们要求政府查看银行的资产负债表,判断它们是否真的根据风险持有足够的监管资本等等。
We require the government to look at bank balance sheets and figure out whether or not they're actually holding enough regulatory capital against their risk and things like that.
我们不指望它们自愿这么做,尤其是在2008年之后。
We don't expect them to do it voluntarily, certainly not after 2008.
这个框架对吗?
Is that the right framing?
对。
Yeah.
不,我觉得从自愿到强制这一步是关键,因为现在公司内部必须有一个负责人,或者需要某种声誉上的压力,或者他们想从第三方审计师那里得到反馈。
No, I think and I think this kind of, like, shift from voluntary to required is the key step because right now, companies have to have, like, a champion within the company or there needs to be some kind of, like, reputational or they wanna get feedback from the third party auditor.
就是说,得给他们点动力才行,但并不是所有公司都愿意主动去征求外部意见。
Like, there needs to be some kind of reason for them to do it, and not all the companies actually choose to invite external feedback.
他们只会提供最基础的信息。
They will share the bare minimum.
举个例子,SpaceX昨天发布了关于Brock 4.6的模型卡或系统卡,目录里有好几项是缺的。
And just for example, SpaceX yesterday put out a model card or system card about Brock 4.6, and there were, like, several sections missing from the table of contents.
看起来像是最后关头被删掉了。
It seems like they got removed at the last minute.
所以,怎么说呢,抱歉。
And and so there's kind of is sorry.
我想说,你接下来要……
I would you're gonna
我正要问,你能给我解释一下模型卡片这回事吗?
just I was just gonna ask, can you explain the whole model card thing to me?
好的。
Yeah.
对。
Yeah.
所以,基本上,模型卡片这个概念最初的想法是把它做成像营养成分表那样的东西,把关键信息简明扼要地总结出来,贴在AI官网上,让人一眼就能明白它的风险、性能表现以及它能做什么等等。
And so, basically, the thing with model cards is that the original idea several years ago was that a model card was like a nutrition label where it's like a bunch of information summarized succinctly, and you kind of slap it on the AI website and it kind of succinctly explains like what are the risks, how well does it work, what can it do, and so forth.
但随着时间的推移,像我这样的业内人士发现,这里面可说的内容太多了,需要深入剖析的地方也不少,而且并没有人强制要求大家必须这么做。
Over time, as people such as myself in the industry were like, okay, there's a lot to say, there's a lot to unpack here and no was forcing anyone to do this.
因此,没有人去规定一个统一的格式,也没有人强制要求大家必须提供这种‘营养成分表’。
So no one established like, this is the format, you need this little nutrition label.
就是大家随便写点东西。
It was just people writing stuff.
后来这些文档越写越长,有的十几页,有的上百页、两百页甚至三百页,里面全是在描述他们发现的种种问题,还有跑过的各种测试,程度各不相同。
They ballooned into these like dozen page, 100 page, 200 page, 300 page documents of just describing like, here's all the crazy stuff we found, here are all the tests we ran, and there's a spectrum.
我觉得Anthropic发布的文档通常是最长的,虽然篇幅长不一定代表质量高,但至少能看出他们确实花了功夫。
Like I would say Anthropic puts out the longest ones that's not necessarily totally correlated with quality, but it shows some proof of work.
而有些公司只发个五页或十页的。
Then others will put out five page, 10 page.
然后昨天发生的事是,SpaceX 首次因为加州法律,确实有要求发布这些,但并没有明确的质量标准。
And then what happened yesterday is that SpaceX, for the first time, because of California law, there actually is a requirement to put these out, but there's not really a clear quality bar.
所以他们可以说,是的,我们做了。
And so they can say, well, yes, we did that.
我们遵守了加州法律。
We followed the California law.
我们分享了关于测试以及第三方参与测试程度的信息。
We shared information about our testing and the extent to which third parties were involved in testing.
基本上,就是一句话说我们和第三方合作了。
Like, basically, it's just like one sentence saying we worked with third
第三方,还有……
parties And or
所以,是的。
so yeah.
所以我觉得这不同于,比如说,银行监管,因为不,……
And and so I think this is different I would say this is different from, say, like, bank regulation in that no.
我的意思是,目前只有一些东西是必须的。
I mean, one is one is that only some things are required right now.
就像是发布一份文件一样。
It's, like, kind of putting out a a document.
没有像第三方测试那样的东西,你应该谈论你是否与第三方合作,但这与实际去做是不同的。
There's no like, the third party test you're supposed to talk about whether you work with third parties, but that's different from actually doing it.
所以我认为我们需要的是标准化,比如第三方审计应该怎么做?
And so I think what we need is kind of standardization around like how should the third party auditing work?
什么样的系统卡才算好?
What counts as a good system card?
你应该实施的最低安全和保障措施是什么?
What are the minimum safety and security protections that you should be putting in place?
我认为这类似于你提到的那些资本化的事情,我们需要标准,比如,什么样的审计师才算好?
And I think that's analogous to some of these capitalization things you mentioned, and we need kind of standards for like, okay, what counts as a good auditor?
什么样的标准测试需要运行,等等?
What counts as what are the standard tests you need to run and so forth?
说实话,你自己在这方面多少有点立场偏向,因为你在筹建一家审计机构或实体。虽然是非营利组织,但本质上是一个负责做审计的实体。
I mean, you yourself are sort of biased in this and that you're building out an auditing a company or an entity, not a company because it's a nonprofit, but an entity that would do auditing.
另外,你也在推广一个观点,就是审计这件事很重要。
Also, you're promoting this idea that auditing should be important.
再说一遍,在金融领域,其实有好几种不同的模式。
Again, in the financial realm, there's a few different versions of it.
比如银行监管机构,有些监管人员是直接常驻在银行里的,几乎全天候都在那里。
There's bank supervisors, some of them literally sit at the bank, and they're there all the time.
接着就是像穆迪和标普这样的机构了。
Then we have the Moody's and the S and Ps of the world.
如果你发行债券,就必须去获取某种第三方评级。
If you issue debt, you're compelled to get some sort of third party rating.
在你的理想世界里,假设这一切都实现了,审计成为强制要求,公司也欣然接受等等,请描述一下具体情况。
Why don't you describe in your ideal world, let's say this all happens and there's required auditing and the companies are cool with it, etcetera.
那么AVERI提供的服务是什么样的?我猜在理想状态下,除了它之外应该还有几家其他机构,毕竟你不能随意挑选审计师之类的。
What is the service that AVERI, and I assume in the ideal world, there would be a few others, etcetera, as you can't go auditor shopping, etcetera.
像AVERI这样的机构,它们提供的服务具体是什么呢?
What is the service that the AVERIES of the world are doing?
嵌入程度有多深?
How embedded?
那么,对于普通公众来说,也就是我们所有人,有什么理由认为这能带来更安全的结果呢?
And what is the reason then to think for the general public, there's all of our hands, that this could lead to safer outcomes?
是的,基本上,在这个世界里,我们和其他机构提供的服务类似于我们现在做的,但规模会扩大。
Yeah, essentially the service that we and others would be providing in this world is similar to what we're currently doing but kind of scaled up.
所以目前,我们做的是自愿试点项目,关注安全、安保、治理等特定方面。
So right now, what we're doing is kind of voluntary pilot projects that are looking at a specific aspect of safety, security, governance, and so forth.
我们最终希望看到的是一个审计生态系统,它们全面审视公司是否遵循其安全和安保实践。
What we would like to see eventually is that there's an ecosystem of auditors that are looking holistically at, is the company following its safety and security practices?
这些安全和安保实践是否合理,是否符合标准底线——这个底线我们最终需要,但现在还没有——并且向公司提供某种反馈。
Are those safety and security practices reasonable and consistent with the standard floor, which ultimately we need, we don't have right now, and providing some kind of feedback to the company.
然后会有一个整改流程,让他们解决审计过程中发现的问题。
And then there would be kind of like a remediation process for them to resolve issues that better surfaced during the auditing process.
然后还会有一份公开的审计报告,在做了大量技术测试、审阅文件、采访员工等等之后,定期发布,比如每季度或类似的时间间隔。
And then there would be a public version of this audit report that kind of shares after doing a lot of technical testing, reviewing of documents, interviewing with staff and so forth, that kind of shares this update on some regular schedule, like quarterly or something like that.
你可能希望它更像一个常驻检查员或嵌入式审计员的模式,而不是每年或每半年才进行一次。
You might want it to be more like a kind of resident examiner kind of embedded auditor model rather than it happening once a year or once every six months.
但你需要某种持续建立信任的过程,也许审计员一直在那里,但偶尔发布这些报告。
But you kind of need to have some kind of like continuous trust building process where maybe the auditor is there all the time, but they occasionally issue these reports.
而且,从公司的角度来看,他们想要的是,你知道,在那种情况下他们会被要求这么做,但今天对他们来说,好处是他们想表明他们在安全方面领先,并且他们想从这些有新鲜视角的外部专家那里获得反馈。
And, you know, what's in it from the company's perspective is they want to, you know, I mean, in the scenario they would be required, but what's in it for them today is that they want to signal that they are ahead of the curve on safety and security, and they want to get feedback from these external experts who have like a kind of fresh perspective.
为什么这很重要?
And why does this matter?
我认为一个原因是,你不想处于一个只能相信公司说辞的世界,你希望有共同的安全标准,而不是每个人都自己制定标准然后被检查。
I think one is you just don't want to be in a world where you have to take the company's word for it and you want them to kind of you want there to be common safety and security standards rather than it just being everyone's kind of making up their own things and then getting it checked.
另一个原因是,你想避免群体思维。
The other is that you want to avoid groupthink.
所以我认为即使是现在,很多外部测试都像是研究项目。
And so I think even right now, there are a lot of you know, a lot of what's happening with external testing is like it's like a research project.
就像 METER,我记得你们最近请了 METER 的人过来,他们正在做关于自主性和失控风险等方面的严肃技术调研。
Like METER, I think you had someone from METER on recently, and they're doing this serious technical research on autonomy and loss of control and so forth.
他们主要是跟企业合作,来做这类偏研究性质的评估。
And they work with companies essentially in order to do these kind of research, very research y assessments.
我觉得这是流程里很关键的一环,但除此之外,还得核实公司是不是真的按他们说的去做了。
And I think that's a key part of the process, but there's also just like verifying that the companies did what they're saying they're doing.
所以一方面要产出证据,另一方面也要去查验这些证据。
So there's producing evidence and then there's also checking evidence.
所以这有点像 K1 声明,如果我没理解错的话,但你知道的,这是一种简短的声明。
And so it's kind of like in a K1 statement, if I'm getting that right, but, you know, there's kind of this like short order statement.
我们可能需要的不仅仅是一段话,但你基本上希望有一个第三方说,我们检查过他们确实执行了所有这些测试。
We probably want something more than just a paragraph, but you basically want, you know, a third party saying we checked that they actually ran all these tests.
我们确保被审计的模型与正在部署的模型是同一个,等等等等。
We made sure that the model that was audited was the same one that's being deployed, etcetera, etcetera.
我们实际上能做些什么来让测试环节更安全呢?
What could we actually do to make the testing side safer?
因为在我看来,我觉得我们可以为那些部署到现实世界中的模型建立一个合理的审计结构。但问题的一部分在于,我们不断开发出更新、更好、更智能的模型,然后进行测试,接着它们又想办法进入现实世界。
Because it seems to me like I I can totally believe that we can come up with a reasonable, like, auditing structure for models that are being deployed and allowed into the real world in some But structured if part of the problem is that we're developing newer and better and more intelligent models and then testing them and then they are figuring out ways to get out into the world.
在……之前
Before
它们的产品就……
their products are.
对。
Yeah.
那看起来是个很大的漏洞。
Then that seems to be, like, a big vulnerability.
我觉得,基本上就是公司变得太自负了,对自己的沙盒质量过于自信。
I think, basically, what happened is that companies were getting cocky, getting overconfident in the quality of their sandboxes.
比如,负责安全评估的人和搭建沙盒的人之间可能存在脱节。评估那边说,黑客的攻击手段已经发展到这个程度了。
Like maybe there was a disconnect between some of the people on the safety side who were measuring like, okay, this is where the hacking skills are going.
而搭建沙盒的安全团队可能没跟上,中间出现了信息传递的偏差。也许是群体思维导致的,具体原因我不确定,但好几家公司都表现出这种过度的自信。
And the people on the security side building the sandboxes and like something was getting lost in translation, maybe it was groupthink, I don't know exactly, but it seemed like at multiple companies, there was this kind of like overconfidence.
所以我觉得这些事件曝光后,所有的技术调查有望带来更多的最佳实践,让更多人检查自己的偏见,但我不认为那是长久之计。
And so I think these incidents come into light and all the kind of technical investigations are going to hopefully lead to more best practices, more people checking their own biases, but I don't think that's a long term solution.
我觉得最终人们总是会过度自信。
I think ultimately people get overconfident all the time.
这是人之常情。
That's a human thing.
所以你需要第三方来检查,确保你真的在遵循这些最佳实践。
And that's why you want third parties checking to make sure that, okay, are you actually following these best practices?
而且你可能还需要一些技术解决方案来应对这些问题。
You also probably are gonna need some technical solutions to some of these things.
比如,也许有些测试应该在完全断网的隔离服务器上进行。
Like, I mean, maybe some of this testing should be done on kind of air gapped servers that are not connected to the Internet at all.
而且我认为Hugging Face事件中,问题出在它经过了中间层。
And and I think what happened in the Hugging Face thing is that it went through this, like, middle layer.
有一个软件,通过这个中间层把数据路由到了真实的互联网。
There was, like, a piece of software that it routed out to the real Internet through this this kind of, like, intermediate thing.
但是,我觉得他们可能会达到一个点,AI系统太强大了,以至于它们能黑出任何东西。
But, like, I I think it might be that they will get to a point where AI systems are just so capable that they can hack their way out of anything.
所以基本上,你只需要确保它们被关在笼子里。
So you just need to make sure that they're in in a cage, basically.
Miles Brundage,我们可以聊上几个小时,因为这有很多迷人的方面。
Miles Brundage, we could talk for hours about this because there's so many fascinating dimensions of this.
我可能将来还会请你回来,不幸的是。
I will probably have you back in the future, unfortunately.
不幸的是,因为那是一次很棒的对话,但不幸的是,这可能不会是最后一次需要和你谈话的原因。
Unfortunately, because that was a great conversation, but unfortunately, it probably won't be the last reason to have to talk to you.
非常感谢你来做客,AVERI。
Thank you so much for coming on, AVERI.
再次感谢。
Thanks again.
我很感激。
I appreciate it.
Tracy,那真有趣,但同时也让人不安,那种方式
Tracy, that was a fun in set it's an unsettling thing, the way
它的行为方式。
it behaves.
这么多关于AI的对话对我来说仍然很超现实。
So many of these AI conversations are still so surreal to me.
比如,我们在2026年讨论这些,感觉真的很奇怪。
Like, the fact that this is what we're talking about in 2026, it just feels so strange.
而且情况只会变得更加离奇,因为我觉得2023年已经很奇怪了,但今天更奇怪。
And it's only gonna get orders of magnitude weirder because I thought things were weird in 2023 and things are much weirder today.
听着,我理解为什么人们对这些事情很愤世嫉俗,也理解为什么人们会说,哦,这是监管俘获吗?
Look, I get why people are very cynical about a lot of this stuff, and I get why people talk about like, oh, is this regulatory capture?
我当然相信监管俘获的前提,而且可能确实存在一些这种情况。
And I certainly believe in the premise of regulatory capture, and there may be some of that.
但我想说一点,从公司的角度来看,确实,长期以来,从一开始,这些公司并没有赚很多钱,但他们在安全和安保上投入了很多。
But I will say one thing, like, in the sort of, like, from the company's perspective is it is true that for a long time and for the very beginning, these are not companies making a lot of money, and yet they spend a lot on safety and security.
而且你可以想象,科技公司以前不会这么做。
And you could imagine tech companies historically didn't do that.
他们确实会这么做。
They do.
而且他们有这些,比如OpenAI肯定有,Anthropic作为公益公司稍微少一点。
And they have these, like it's certainly in the case of OpenAI and slightly to a lesser extent Anthropic as a PBC.
他们有这些奇怪的公司结构,部分原因在于他们似乎相信,他们正在构建的东西,如果构建错了,就不应该仅仅掌握在纯粹追求利润的企业手中。
They have these weird corporate structures in part because they seem pretty they seem to believe that the things that they're building, if built wrong, should not necessarily just be in the hands of, like, purely profit seeking enterprises.
对。
Yeah.
说得都很对。
All very true.
我确实觉得那次对话中让我印象深刻的一点是,再次提到,经过批准、公司可以实际用于防御的模型,与那些处于测试模式、不知怎么逃出沙盒的新前沿模型之间的权力不对称。
I do think one of the interesting things to me that stands out from that conversation is, again, the idea of, like, the asymmetry in power between the approved models that companies can actually use for defense against the new frontier models who are in testing mode and have somehow escaped the sandbox.
我觉得这有一个非常可怕的维度。
I think there's a really scary dimension.
我觉得你得想想,审计和穆迪这类评级机构之间的区别是什么。
And I think that you think about what is the difference between auditing versus a Moody's, etcetera.
看起来两者都需要。
It seems like you need both.
看起来你需要确认,是的,这个特定模型满足了所有我们认定它安全的要求。
It seems like you need to have, yes, this specific model, it satisfies all the requirements that we've deemed it to be safe.
但更深层的审计问题是,这家公司是否通常以我们认为负责任的方式进行实验、研发和测试,对吧?
But then this sort of deeper auditing question of like, is this a company that generally experiments and does R and D and testing in what we perceive to be, like, a responsible manner Yeah.
这更像是监管性质的。
Which is more like the supervisory
就像需要测试审计员。
like, need, like, a testing auditor.
就像银行监管员,实际上坐在实验室里,观察测试过程,确保沙盒设计良好。
Like, the bank supervisor who's, like, actually sitting on the floor, actually sitting in the labs and observing the testing process and making sure that the sandbox is well designed.
但同样,问题在于经典的网络安全问题或一般安全问题,即模型只需要找到一个漏洞,对吧?
But, again, the problem with that is the classic cybersecurity problem or security in general problem, which is the model just has to find a single vulnerability, right?
必须把所有漏洞都修好,确保成千上万的漏洞都无懈可击。
Have to, like, fix all of them, make sure that, like, thousands and thousands of vulnerabilities are impenetrable.
我觉得这确实有道理,你看。
And it does make sense, I think, that, like, look.
这是为了利润的资本主义竞争。
This is for profit capitalist competition.
毫无疑问。
There's no doubt.
这些是最大的公司,增长速度惊人。
These are some of the biggest, the pace of growth is extraordinary.
我们还没谈到开源模型或开源服务器怎么办。
We didn't even get into how would you do this for open source models or open source servers.
那完全是另一回事。
That's a whole other can of worms.
但如果你在实验室里,想赚钱,又担心如果放慢速度,其他公司会赚更多钱,那么解决这个问题的一种方式——我不知道是不是囚徒困境之类的。
But it makes sense that if you're in the lab and you're trying to make money and you're also worried about, like, if you slow down, etcetera, then the other company is gonna make more money, etcetera, that one way you solve this I don't know if it's prisoner's dilemma or whatever.
博弈论。
Game theory.
底线,我想。
Bottom, I guess.
博弈论嘛,好吧。
Game theory is, okay.
你需要这个第三方来,比如,你们在研发方面想多快就多快,但我们会制定规则,比如你们这样做是否安全?
You need this third party to, like, you guys go as fast as you want on the r and d side, but we are gonna set the rules of, like, are you doing that in a safe way?
不然的话,Dario和Sam怎么会互相信任呢?
Otherwise, why would Dario and Sam ever trust each other?
根本不可能。
It's like, no.
我们发誓。
We swear.
我们会放慢节奏,非常认真地对待这件事。
We're slow we're taking it really seriously.
我们在放慢节奏,你知道吧,但他们其实在偷偷往前赶。
We're slowing things down, you know, like and then secretly, they're racing ahead.
对于一连串完全是私人的、私有的实体来说,这真的很难解决。
That is really hard to solve for a series of private purely private entities.
好。
Alright.
我们就先到这儿?
Shall we leave it there?
那我们就到这儿吧。
Let's leave it there.
这是又一期《Odd Lots》播客。
This has been another episode of the Odd Lots Podcast.
我是 Tracy Allaway。
I'm Tracy Allaway.
欢迎在 Tracy Allaway 上关注我。
You can follow me at Tracy Allaway.
我是吉尔·韦森塔尔。
And I'm Jill Weisenthal.
你可以在推特上关注我,账号是the stalwart。
You can follow me at the stalwart.
关注我们的嘉宾迈尔斯·布伦戴奇。
Follow our guest, Miles Brundage.
他的推特账号是Miles underscore Brundage。
He's at Miles underscore Brundage.
关注我们的制作人:卡门·罗德里格斯,账号是Carmen Armen;达什·奥贝内特,账号是Dashbot;凯尔·布鲁克斯,账号是Kale Brooks;还有凯文·洛萨诺,账号是Kevin Lloyd Lozano。
Follow our producers, Carmen Rodriguez at Carmen Armen, Dash O'Bennett at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano at Kevin Lloyd Lozano.
更多Odd Lots内容,请访问bloomberg.com/oddlaws,获取每日新闻通讯和所有节目。
And for more Oddlaws content, go to bloomberg.com/oddlaws for the daily newsletter and all of our episodes.
你还可以在我们的Discord服务器discord.gg/oddlots上全天候讨论这些话题。
And you can chat about all of these topics twenty four seven in our Discord, discord.gg/oddlots.
如果你喜欢Odd Lots,喜欢我们讨论道德相对主义,请在您最喜欢的播客平台上给我们留下好评。
And if you enjoy Odd Lots, if you like it when we talk about moral relativism, then please leave us a positive review on your favorite podcast platform.
另外提醒,如果你是彭博的用户,你可以完全无广告收听我们所有的节目。
And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free.
你只需要在 Apple Podcasts 上找到彭博频道,然后按上面的提示操作就行。
All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there.
感谢收听。
Thanks for listening.
你好。
Hi.
我是David Westin。
I'm David Westin.
欢迎每周收听《华尔街一周》播客,和我一起了解来自全球各地的资本主义故事。
Join me every week for the Wall Street week podcast to hear stories of capitalism from around the world.
从地缘政治紧张和央行决策,到人工智能、能源和基础设施,我们会找到那些正在影响全球市场的CEO、经济学家、政策制定者和思想领袖,与他们面对面深入交流。
From geopolitical tensions and central bank decisions to artificial intelligence energy and infrastructure, we sit down with the CEOs, economists, policymakers, and thought leaders whose decisions are shaping markets everywhere we find them.
请在 Apple、Spotify 或任何你收听的平台上订阅《华尔街一周》播客。
Subscribe to The Wall Street Week podcast on Apple, Spotify, or anywhere you listen.
关于 Bayt 播客
Bayt 提供中文+原文双语音频和字幕,帮助你打破语言障碍,轻松听懂全球优质播客。