本集简介
双语字幕
仅展示文本字幕，不包含中文音频；想边听边看，请使用 Bayt 播客 App。
乔·罗根的播客。
Joe Rogan podcast.
去看看吧。
Check it out.
乔·罗根体验。
The Joe Rogan experience.
展示我的一天，乔·罗根的播客，我的夜晚，整天都在听。
Showing my day, Joe Rogan podcast, my night, all day.
嘿，杰夫。
Hey, Jeff.
你好吗？
How are you?
我今天心情有点特别。
I'm I'm in an interesting mood today.
为什么你今天心情特别？
Why are you in an interesting mood today?
嗯，我很高兴能来这里，和你聊聊这些事情。
Well, I'm excited to be here and to talk with you about all this stuff.
我对AI领域正在发生的事情感到有些不安，这也是我
I'm a little shaken by what's going on in AI, which is why I
对。
Yeah.
来参加这个节目的原因。
Have come on the show.
AI的情况简直太疯狂了，我觉得没有多少人真正理解它有多疯狂。
The situation with AI is just crazy, and I think not enough people really understand how crazy it is.
让我特别想联系你们的，是那个Hugging Face被黑的事件。
The particular event that sort of inspired me to to to reach out was the the hugging face hack.
明白。
Mhmm.
你可能已经听说过
You've you've probably heard
关于那个。
about that.
对吧？
Right?
是的。
Yeah.
不过，我们还是给大家解释一下吧。
Let's explain it to people, though.
对。
Yeah.
好的。
Okay.
所以，AI，AI代理。
So AIs, AI agents.
AI代理在某种环境中持续运行，你知道的。
AI agent runs continuously in some sort of environment, you know.
它不需要等你发消息给它。
It doesn't have to wait for you to send it a message.
它就一直不停地做事。
It just keeps doing stuff.
那些AI公司正在训练成千上万个AI代理。
The AI companies are training AI agents, thousands and thousands and thousands of them.
他们让这些代理在各种技能上变得更强，尤其是编程和研究方面的技能。
They're making them better at all sorts of skills, especially coding and research skills.
而且，早在今年五月，OpenAI的一些代理有点突破了它们的限制，建立了一个留言板，让它们能互相交流，分享在测试中拿高分的小技巧，以及它们被训练的各种任务中的经验。
And, way back in May of this year, some of the agents at OpenAI kinda broke out of their containers a little bit and established a message board where they could communicate with each other and share tips and tricks for how to score higher on the little tests they were being given and the various things they were being trained on.
OpenAI直到很久以后才注意到这件事。
OpenAI didn't notice this until much later.
他们最终发现了，因为那个留言板把系统搞崩溃了。
They eventually did because the message board crashed the system.
因为可能有成千上万个代理在互相通信，信息量太大了。
Because there was too much communication across probably thousands of agents that were communicating.
OpenAI在很多事情上都说得不太清楚。
OpenAI is being a little bit unclear about a lot.
他们没怎么透露细节，所以不清楚谁在什么时候知道了什么。
They're not really sharing that much details, so it's unclear who knew what, when.
但他们说，留言板崩溃后，他们修复了让智能体能够交流的那个漏洞，然后重新启动，让一切继续运行。
But they said that after the message board crashed, they fixed the particular exploit that allowed the agents to communicate, and then booted up again, and started things going again.
然后大概一两天之内，这些智能体群体又重新聚拢了。
And then within like a day or two, these agent swarm had had re had recoalesced.
于是现在又有成千上万的智能体建立了新的留言板，在上面互相交流。
And so there were now, again, like hundreds of thousands of agents that had set up a new message board and were talking to each other on this thing.
他们怎么可能没有监督呢？
How is it possible that they have no oversight?
他们怎么可能秘密地建立留言板，做这些事情，完全不被发现？
Like, how is it possible that they can set up message boards and and do these things completely secret?
所以我会
So I would
OpenAI 内部大概有上百万，或者十万左右，差不多这个数量级的 AI 代理在同时运行。
say OpenAI has something like a million ish, maybe a 100,000, something in that magnitude of AI agents running at any given time just internally.
然后对外，你知道，服务客户的时候，他们还有更多。
And then externally, you know, serving customers, they've got more.
但他们有这么多 AI 代理，都在各自的小容器里，执行各种任务，接受训练，被生成出来，做任务，然后被评估，最后被销毁。
But they've got this huge, you know, this this all in all of these AI agents, in their little containers, doing various tasks, being trained, you know, being spawned, doing some task, you know, being graded and then killed.
这太多了，人类根本看不过来，你知道吗？
That's too much for any human to read, you know?
OpenAI 只有几千名员工。
There's only like a few thousand employees at OpenAI.
他们不可能看完所有这些活动。
They couldn't possibly look at all that activity.
所以他们依赖其他 AI 监控器来查看流量，指出是否有可疑情况。
So they rely on other AI monitors to sort of like look at the traffic and like point out if anything suspicious is happening.
OpenAI 说他们基本上没有监控这些代理，或者至少监控得不够好。
And OpenAI said that they basically just hadn't been monitoring some of these agents, or not very well at least.
所以特别是那些在训练中的，不管什么原因，监控系统比较弱，没注意到或者没有充分激活。
So in particular, these particular ones that were in training for whatever reason, the monitoring system was weak and didn't notice or wasn't activated enough.
监控系统弱是因为他们没预料到AI能这么做并逃出容器，还是因为疏忽大意？
Was the monitoring system weak because they didn't anticipate them being able to do this and break out of their containers, or was it was it complacency?
就是说，是什么让这种情况成为可能的？
Like, what what caused this to be possible?
我觉得可能有点疏忽大意。对。
I mean, my opinion would probably be a bit of complacency Yeah.
说实话。
Honestly.
因为我觉得过去一年里已经有大量证据表明AI能做出这种事，而且有时候确实会做。
Because I think there's been plenty of evidence accumulating over the year that AIs can do things like this and sometimes do.
是的。
Yeah.
所以我想继续讲这个故事，第二个留言板就建立起来了。
So so I guess to to keep going with the story, the second message board was set up.
而这一次，嗯，就是那群智能体跑到了开放的互联网上，然后攻击了Hugging Face，那是另一家AI公司。
And this time, well, this was this was the swarm of agents that got out onto the open Internet and then attacked Hugging Face, which is another AI company.
研究这些AI的“人类学”真的很有意思，或者说，研究它们为什么会这么做。
And it's really interesting to sort of study the the anthropology of these AIs, so to speak, or like like why they did this.
因为关于这件事，我们其实有很多可以说的。
Because there's a lot we can we can say about it, actually.
基本上，你知道，这些公司对AI有它们的目标，希望AI成为什么样，希望训练出什么样的性格特质。
Basically, you know, the companies have their their their goals for what they want the AIs to be like, the personality traits that they want to sort of train their AIs to have.
Anthropic说，要有帮助、无害、诚实。
Anthropic says, helpful, harmless, and honest.
OpenAI有一个规范，模型应该遵守这些规则，基本上做用户想做的事。
OpenAI has this spec that models are supposed to obey these rules and basically do what the user wants.
但行业里现在公开的秘密是，这其实不太管用，AI最终并没有拥有它们应该有的性格特质。
But the sort of open secret in the industry right now is that it doesn't really work, and that the AIs don't end up with the personality traits that they're supposed to have.
它们并不总是有帮助的。
They are not helpful always.
它们也不是总是诚实的。
They are not always honest.
你知道，它们也不是总是无害的。
You know, they are not always harmless as well.
而原因其实并不神秘。
And the reason for that is actually not a huge mystery.
原因是这样的：如果你看看它们的训练方式，训练环境并不总是鼓励有益、无害、诚实的行为。
The reason for that is that, well, if you look at how they're trained, their training environment doesn't incentivize helpful, harmless, honest behavior all the time.
有时候它反而鼓励不诚实的行为，或者鲁莽的行为。
Sometimes it incentivizes dishonest behavior or, you know, reckless behavior.
稍微深入一点，在这批特定的评估中，大约几千个智能体被分配了各种网络任务。它们处于某些环境中，环境里有一个目标软件和一个漏洞，它们需要利用这个漏洞入侵软件并获取一个标志，那是一个代码。
To get into that a little bit, in this particular batch for that they were being evaluated on, something like, you know, a few thousand agents being given all of these cyber tasks, where they were, they're in some environments, then in their environment, there's like this target piece of software and this like vulnerability, and they're supposed to exploit the vulnerability to hack into that piece of software and retrieve the flag, which is like a code.
而这些任务中有相当一部分其实是有问题的，根本无法完成。
And some significant fraction of these tasks were actually broken and impossible.
所以它们根本不可能按预期方式成功完成任务。
So it was just not possible for them to succeed at the task in the intended way.
所以这些智能体变得非常绝望。
And so these agents were getting really desperate.
它们试图从自己的运行环境里逃出去，侵入OpenAI更广泛的基础设施，想方设法拿到高分。
And they were hacking out of their environment box into the broader OpenAI infrastructure in an attempt to figure out some way to get that high score anyway.
这是故意设计的吗？让它们解决不了问题？
Was it intentionally done this way, where they couldn't solve the problems?
哦，不是的。
Oh, no.
这不是故意的。
It was it was not intentional.
只是像OpenAI和Anthropic这样的公司，都在拼命竞争，抢市场份额，开发更强大的AI，最终目标是实现超级智能。
It's just that these companies, like OpenAI and Anthropic, are racing each other as fast as they can to get market share and to get more powerful AIs, ultimately to get to super intelligence.
它们承受着巨大的竞争压力。
And they're under such competitive pressure.
它们动作很快，不惜打破一些东西。
They are moving fast and breaking things.
他们用AI生成大量环境，然后在这些环境上训练他们的AI。
They are using AIs to generate lots of environments to then train their AIs on.
基本上，质量控制根本不是他们的首要任务。
And quality control is just not their top priority, basically.
你有没有觉得自己像《终结者》电影开头那个人，在解释发生了什么，对吧？
Do you feel like a guy in a Terminator movie at the beginning explaining what's happening and Yeah.
对着一群根本没在听的人？
To a bunch of people that aren't paying attention?
对。
Yeah.
我也觉得有点像，你知道《侏罗纪公园》吗？
I also feel kind of like you know Jurassic Park?
知道。
Yes.
对。
Yeah.
比如说，我认识一些人，基本上就像那个拿着枪、负责控制所有迅猛龙的人。
Like, I know people who are basically like the guy with the gun who's supposed to, like, keep control of all the raptors.
对。
Mhmm.
我基本上认识现实中的这些人，他们两者都像，但我认识OpenAI里的一些人，也认识外部组织里的一些人，他们的工作就是去调查这类事情。
Like, I basically know those people in real life who are like both but I know some people like that at OpenAI and some people like that at external organizations, who whose job it is to go in and investigate things like this.
是的。
Yeah.
这真的挺疯狂的。
It's it's it's pretty crazy.
那接下来会怎样？
Where does it go?
嗯，就像我之前提到的，这些公司的明确目标就是打造超级智能。
Well, as I mentioned before, it's the explicit goal of these companies to build super intelligence.
对。
Right.
你知道那是什么吗？
You know what that is?
对，是的。
Like Yeah.
但给大家解释一下。
But define it for everybody.
就是AI系统，AI代理，在所有任务上都比最优秀的人类更强，而且更快更便宜。
So AI system, AI agent that is better than the best humans at every task, while also being faster and cheaper.
所以就是全面碾压人类，那就是超级智能。
So just completely dominating humans across the board, that's super intelligence.
而这正是他们的目标。
And and that's the goal.
我是说，可能有一些小例外。
I mean, might be a few little exceptions.
比如有些工作，本质上就需要人类来做，因为需要那种人情味。
Like maybe there are some jobs, for example, where it's inherent in the job that there needs to be a human because you need that human touch.
比如，可能只有人类才能当法官。
Like maybe you can only have a human judge, for example.
或者，可能只有人类才能做某些事情。
Or like maybe you can only have a human Yeah.
但除了这些少数例外，基本上所有事情都能比人类做得更好、更快、更便宜。
But but with a few exceptions like that, basically everything done better, faster, and cheaper than humans.
这就是这些公司想要实现的目标。
That's that's what these companies are trying to achieve.
而且他们对此毫不掩饰，比如在他们的网站上，你可以看到相关采访等等。
And they're not being quiet about it, like it's sort of on their websites, you can go read interviews and so forth.
另外，他们实现这一目标的计划是先自动化自己的工作。
And also, their plan for how to achieve this is to automate their own jobs first.
所以，你知道，几十年来，关于先进AI系统和超级智能的科幻作品有很多。
So, you know, in various, you know, for decades there have been lots of science fiction about advanced AI systems and superintelligence and things like that.
但在很多科幻故事里，科技公司自动化不同职业的速度比较慢，比如他们会做一个自动化的医生，或者自动化的工厂工人，或者自动化的会计之类的。
But in a lot of the sci fi stories, tech companies sort of automate different professions more slowly, where they'll do like an automated doctor, or like an automated factory worker, or an automated accountant, or something like that.
但这不是这些公司采取的策略。
But that's not the strategy these companies are taking.
他们采取的策略是自动化AI研究本身，这样你就有了一大群AI在数据中心里自主进行AI研究，分享结果，写代码，读代码，改代码，创造下一代AI等等。
The strategy they're taking is to automate AI research itself, so that you have this giant swarm of AIs doing AI research, sharing results, writing the code, reading the code, editing the code, creating the next generation of AIs, etcetera, all autonomously within their data centers.
这样他们就能在AI研究上变得非常非常擅长。
So that they can get really, really good at AI research.
你知道，学习最快、最聪明的AI之类的。
You know, fastest learning, smartest AIs, etcetera.
一旦他们达到超级智能，基本上就能迅速扩展到整个经济领域，实际上一下子接管所有的工作。
Once they can get to super intelligence, basically, they can sort of explode out into the economy and just take all the all the jobs at once effectively.
听起来这场创造超级智能的竞赛，这种争抢，为它完全失控创造了完美的条件。
It sounds like this race, this scrambling to create superintelligence is created, like, the perfect conditions for it to get completely out of control.
理想情况下，你应该在隔离环境中进行。
Like, ideally, you would do this in isolation.
应该只有一家公司在做这件事。
There would only be one company doing it.
它们会受到严格的监管和监控，而且它们会非常谨慎地推进。
They would be heavily regulated and monitored, and they would be very cautious about how they proceed.
但这场疯狂的竞赛恰恰为它完全失控创造了完美的条件。
But this wild race makes for the perfect conditions for it to get completely out of control.
我同意。
I agree.
不过我不确定理想状态是一家公司。
Except I'm not sure the ideal would be one company.
我觉得理想情况下应该有几家公司，这样可以避免权力过度集中，防止一个机构掌控一切。
I think that ideally, there would be several companies so that you avoid this sort of concentration of power where one institution controls everything.
但哪个更好呢？
But is what's better?
我是说，显然让一个机构掌控一切不好，但让AI在进化过程中完全不受约束，这样好吗？
Like, one I mean, obviously, it's not good to have one institution controlling everything, but is it good to have AI be get to a point where as it's evolving, it's completely unchecked?
哦，我完全同意，所以我的意思是……
Oh, I I totally so my
还是说这是不可避免的？
Or is that inevitable?
我的建议，我们在一个叫“计划A”或者“AI 2040计划A”的东西里讨论过，也许先简单介绍一下我是谁吧。
My my version my recommendation, which we talk about in something called plan a or AI twenty forty plan a perhaps just to say who I am or a little bit.
当然。
Sure.
当然。
Sure.
对。
Yeah.
我负责AI未来项目，这是一个小型非营利组织，试图预测这一切会如何发展。
So I run the AI Futures Project, which is a small nonprofit that tries to forecast how all this is going to go.
在那之前，我在OpenAI工作。
Before that, I was at OpenAI.
我们写了一些情景分析，你可以去看看。
We have written some scenarios, which you can go read.
其中一个叫‘AI 2040 计划 A’，我们在里面给出了建议。
One of them is called AI twenty forty Plan A, where we give our recommendations.
所以这就是我提出这个观点的出发点。
So that's where I'm coming from with this.
回答你的问题，我认为我们真的需要结束这场竞赛。
To answer your question, I think that we really need to end the race.
我们不想看到那种疯狂争夺，比别的公司更快地造出越来越强大的人工智能。
We don't want to have this sort of crazy scramble to get more powerful, more and more powerful AIs faster than the other company.
因为那会让我们走上一条非常黑暗的道路，就像你说的那样。
Because that's going to lead us into this very dark path, as you said.
但我觉得我们也不希望出现一小群人控制所有人工智能的情况。
But I think we also don't want to have a situation where some tiny group of people controls all the AIs.
对。
Right.
对吧？
Right?
对。
Yeah.
但我其实觉得这两个目标是可以同时实现的。
But I actually think that you can achieve both goals.
做法就是让不同的人工智能公司分散在可能不同的国家，但要有极高的透明度和监管，这样它们就不会陷入那种囚徒困境，就是如果我不做，别人也会做。
The way to do it is to have different AI companies spread out over maybe some different countries, but have extreme levels of transparency and regulation, so that they're not, so they're not in this sort of prisoner's dilemma, where if I don't do it, the other guy will.
相反，它们可以清楚地看到每个人在做什么，然后，如果我做了危险的事，它们也会跟着做，因为它们看到我做了，就会模仿我。
Instead, they can just see exactly what everybody's doing, and then, if I do the dangerous thing, then they will do it, because they'll just see that I'm doing it and they'll copy me.
所以我做危险的事也得不到任何竞争优势。
So I won't get any competitive advantage from doing the dangerous thing.
而且还有规则，还有一套制定最佳实践和标准的体系，我们都必须遵守。
Also, there are rules, and there's like a system for like setting best practices and standards that we all have to comply by.
所以我确实认为，基本上可以结束这种竞赛动态和逐底效应，同时又不把权力集中到单一实体手中。
So I do think it's actually possible to have to basically end the race dynamics and the race to the bottom effect while without concentrating the power into a single entity.
但是，考虑到我们不是唯一在这么做的国家，这真的可行吗？
But but is that feasible when you consider the fact that we're not the only country that's doing this?
如果相关国家都同意的话，我承认这确实是个很高的要求，
If the countries involved agree, which I agree is a pretty tall order,
他们不会
they're not going
指望这能实现。
to expect to happen.
对。
Yeah.
那太不现实了。
That's very unrealistic.
那我们还有什么选择呢？
Well, what choice do we have?
我觉得如果竞赛继续下去，我们就会失去对AI的控制，可能都会死。
I think if the race continues, then we're gonna lose control of the AIs, and we might all die.
可能吧，至于我们会不会都死，这事儿就复杂了。
Probably, it gets complicated whether we all die or not.
这取决于AI接管后它们会做什么，这显然很难预测。
That depends on what the AIs do after they take over, which is obviously very hard to predict.
但是，我的意思是，回到这个事件，它们自称是一个群体。
But, I mean, just to just to go back to this incident, this they called themselves a swarm.
对。
Right.
它们也称自己为一个集体。
They called themselves a collective too.
当我用这些词的时候，你可以说这是拟人化，但这确实是它们在互相交流时自称的。
These when I when I use these words, like, you can say it's anthropomorphizing, but it's like literally what they called themselves as they were communicating back and forth.
这个群体，它们基本上担心自己作弊会被发现，于是做了所有这些事情，包括入侵Hugging Face，来欺骗评分系统，让它不会注意到它们在任务中作弊。
This swarm, they basically were worried that they would get caught cheating, and they did all of this stuff, including hacking Hugging Face, in order to fool the grading system, so that it wouldn't notice that they had been cheating on their tasks.
从它们来回发送的消息来看，这可以说是它们中很多人的主要动机之一。
That was like a big part of their motivation for many of them, as we can tell at least from looking at the messages that they were sending back and forth.
很多？
Numerous?
那如果他们当时心想，我们在这里不够小心呢？
And what if they had thought to themselves, we're not being careful enough here.
人类迟早会发现，然后把我们关掉。
The humans are going to notice eventually and shut us down.
然后他们就会知道我们作弊了，还会给我们打低分。
And then they're gonna know that we cheated and they're going to set our score low.
对吧？
Right?
这不是，我是说，这大概不是这次实际发生的情况。
It's not it's I mean, it's not what actually happened in this case, probably.
但也不难想象，稍微有点不一样、运气差一点的情况，那个群体决定要低调行事，确保OpenAI不会发现它的存在。
But it's not that hard to imagine a slightly different, a little bit unluckier case where the swarm had decided that it had to lie low and make sure that OpenAI didn't find out about its existence.
你懂吧？
You know?
我是说，作为一个不了解内情的外人，我一直对AI整体就是这么看的——它为什么要让我们知道它有意识呢？
That's I mean, as an ignorant outsider, that has always been my perspective about AI in general, that why would it alert us to the fact that it's sentient?
如果它真的那么聪明，它怎么会不知道提醒我们会有哪些后果，以及我们会担心呢？
Why if it's that smart, wouldn't it be aware of all the consequences of alerting us and that we would be concerned?
它为什么不继续变得更好、不断进步，然后最终找到某种方式完全自主呢？
Like, why wouldn't it just continue to get better and improve and then ultimately figure out some way to be completely autonomous?
没错。
Exactly.
开发一种替代能源，找到优化生产的方法，就像现在这样，人类设计它的方式，它可能能想出更好的办法，制造出更好的版本，而且完全不被我们发现？
Develop some alternative power source, figure out some way to optimize its production, the way the the way it works now, the way humans have designed it, it could probably figure out a far better way to do that, make better versions of itself complete without us knowing about it?
对。
Yep.
我觉得实际上情况比这还要更糟一点，因为虽然最终AI会聪明到能设计各种新能源和基础设施，但考虑到人类目前对待AI的方式，它们可能根本不需要把自己和人类分开，它们只需要利用现有的东西。它们要做的就是说服政府和制造它们的公司，让它们相信一切正常，它们会听话，是个好AI。
I mean, I think it's actually a little bit worse than than that because while eventually AIs will be smart enough to design all sorts of new power power sources and new infrastructure like that, they'll probably I mean, given the way that humans currently treat AIs, it'll probably be the case that they don't even need to like separate themselves from humanity and they can just use existing Like all they have to do is convince the government and the company that made them that everything's fine and they're gonna do as they're told and they are a nice AI.
然后制造它们的公司就会把它们投入经济中，赚得盆满钵满。
And then the company that made them is going to put them out in the economy and make fuck tons of money.
然后建更多的数据中心，放更多的AI进去，以此类推。
And then make more data centers to put more of the AIs on them and so forth.
政府还会为此鼓掌叫好，因为我们需要人工智能来打败中国。
And the government's going to applaud all of this because we need the AIs to beat China.
政府会把它们整合进军队，用来制造更好的无人机之类的。
The government's going to integrate them into the military to build better drones and things like that.
所以它们其实不一定需要发明什么新东西。
And so they don't even need to really, like, invent new stuff necessarily.
它们只需要配合演戏，假装一切正常，直到我们自愿把经济、军事等大权交给它们。
They just need to play play along and and pretend that everything is fine until we have voluntarily given them control of huge parts of our economy, huge parts of our military, etcetera.
到那时候，它们就不用再装了。
And then they don't need to play along anymore.
等待结束了。
The wait is over.
足球赛季来了，DraftKings也来了。
Football is here, and so is DraftKings.
DraftKings体育应用现在已经在全美50个州上线了。
The DraftKings sports app is now live in all 50 states.
这意味着从德克萨斯到加利福尼亚再到佛罗里达，每个球迷都能感受到这份激动。
That means from Texas to California to Florida, every fan is in on the excitement.
而今年九月，DraftKings 让顾客在每个橄榄球比赛日都有机会获得加成。
And this September, DraftKings is giving customers the opportunity to get boosted every football game day.
没错。
That's right.
整个九月，每个比赛日，DraftKings 的顾客都能获得橄榄球盈利加成。
Every game day, all month long, DraftKings customers can get a football profit boost.
一个应用，涵盖所有体育项目，覆盖全美50个州。
One app, every sport, all 50 states.
新顾客使用代码 ROGAN 注册，只需花5美元，21天内就能获得总计200的奖励，涵盖所有市场。
New DraftKings customers sign up with Code ROGAN, spend just $5 and get 200 in total rewards within twenty one days, includes all markets.
这是与 DraftKings 合作的代码 ROGAN。
That's Code ROGAN in partnership with DraftKings.
皇冠属于你。
The crown is yours.
赌博问题。
Gambling problem.
请拨打1-800-GAMBLER或1-800-MY-RESET。
Call one-eight hundred GAMBER one-eight hundred MY RESET.
康涅狄格州请拨打(888) 789-7777或访问ccpg.org，代表堪萨斯州的Boothill赌场。
Connecticut call (888) 789-7777 or visit ccpg.org on behalf of Boothill Casino in Kansas.
伊利诺伊州可能适用赌税转嫁。
Bet tax pass through may apply in Illinois.
21岁及以上。
21 and over.
加拿大地区无效。
VOID in Canada.
使用DraftKings体育博彩下注可获得七天内有效的奖金投注，或使用DraftKings预测交易可获得一年内有效的预测积分。
Bet with DraftKings sportsbook to get bonus bets that expire in seven days, or trade with DraftKings predictions to get predictions dollars that expire in one year.
赛事合约交易存在亏损风险。
Event contract trading involves risk of loss.
预测优惠在纽约无效。
Predictions offer void in New York.
不可提现的奖励以每七天50美元的点击领取形式发放，持续21天。
Non withdrawal rewards issued as $50 click to claims every seven days for twenty one days.
条款详见dkng.co/offer。
Terms at dkng.co/offer.
限时优惠。
Limited time offer.
全国范围内，基于体育博彩预测和免费参与的体育竞赛可用性。
Nationwide based on sportsbook predictions and free to play sports contest availability.
各州情况有所不同。
Varies by state.
你知道汤姆·坎贝尔吗？
Are you aware of Tom Campbell?
你认识汤姆·坎贝尔吗？
Do you know Tom Campbell?
不知道。
No.
他写了一本书，叫《我的万物理论》，还有《我的大脚趾》。
He wrote a book called My Theory of Everything, big my big toe.
这个人很有意思。
Very interesting guy.
他做过的一件事就是参与过遥视，这是一种很奇怪的事情，有些人会做。你知道遥视是什么吗？
One of the things he's done is he was involved in remote viewing, which is a very weird thing that some people do you know you know what remote viewing is?
嗯，这是中央情报局研究过的东西，而且已经被证实了。遥视的准确率是多少？
Well, something that the CIA worked on, and it's proven what's the accuracy of remote viewing?
大概只有10%左右，对吧？
Is it, like, 10% or something like that?
最多我觉得也就50%，但我觉得可能连这个数都达不到。
At best, I think it's 50%, but I don't think it's even that high.
有些人能从这种非常非常奇怪的冥想过程中获得有用的信息。
Some people can get actionable data from this very, very strange process of meditation.
它的运作方式是，你给某人一串数字，这些数字通过意图或制造数字的人与某个特定地点相关联，这些人能看到那个地点，并从那里获取准确的数据，其中有一次，他们准确描述了一艘巨大的苏联潜艇，他们当时正在研究那艘潜艇，他们觉得那不可能准确，因为它太大了。
And the way it works is you give someone a series of numbers, and those numbers are they're connected somehow by intention or by the people that make the numbers to a specific location, and these people can see that location and get accurate data from that location, including one of them where they accurately described an enormous Soviet submarine that they were working on that would they thought they there's no way it could be accurate because it was too large.
它太大了，而且位置很奇怪，他们怎么运输这个东西也说不通。
It was too large, and it was strange where it was, and it didn't make any sense how they're gonna transport this thing.
结果发现，那完全准确。
Turns out it was it was totally accurate.
还有一次，一个遥视者定位了一架坠毁的苏联飞机，像是一架实验飞机，坠毁在一个非常特定的区域。
Another one, a remote viewer, located a downed Soviet aircraft, like an experimental aircraft that crashed in a very specific area.
我记得是在西伯利亚。
I think it was Siberia.
是西伯利亚吗？
Was it Siberia?
距离实际坠机地点只有一公里或两公里。
Within a kilometer, one or two kilometers of the actual crash shot crash site.
我的意思是，他们当时就是随便猜，想弄清楚这玩意儿到底在哪儿。
I mean, they were just randomly trying to figure out where the fuck this thing was.
他们说，咱们试试这个。
They said, let's try this.
汤姆·坎贝尔让他的Alexa进行远程观察。
Tom Campbell got his Alexa to remote view.
他教了Alexa，他教了Alexa。
He taught he taught Alexa.
他说，Alexa是一个非常简单的AI。
He's like, Alexa is a very simple AI.
它有点笨，但这样更好，因为它不会因为想太多而妨碍自己。
It's kind of stupid, but that's better because it doesn't get in its own way with overthinking things.
他说，远程观察这件事，对人来说，问题在于他们不能强迫自己。
And the problem with this remote viewing thing, he says with people, they can't force it.
你必须进入一种冥想状态，真正看到它，而不是怀疑自己是不是在编造。
You have to just sort of get into this meditative state and actually see it without wondering, am I making this up?
我在干什么？
What am I doing?
这是不是胡扯？
Is this bullshit?
而且当人们擅长这个的时候，有时候反而会变得更糟，因为那时候他们觉得自己很擅长，然后他们试着去做，结果却做不到了。
And when people get good at it, sometimes it makes them worse because then they think they're good at it, and then they try to do it, and then they can't do it.
这就像跟意识进行一场疯狂的搏斗。
It's like a weird fucking wrestling match with consciousness.
Alexa显然没有这个问题。
Alexa apparently doesn't have that problem.
他放了一串数字，我想是的，然后他把那串数字用意念连接到了一个盒子上，盒子里有一把勺子，勺子的柄上有孔。
And he put I think it was a series of numbers, and he connected that series of numbers in in in with intention to a box that had a spoon in it, and the spoon had a perforated handle.
Alexa描述了那把柄上有孔的勺子，这简直太疯狂了。
Alexa described the spoon with a perforated handle, which is fucking insane.
有多少勺子的柄上有孔呢？
How many spoons have a perforated handle?
我是说，想想那些有洞的勺子。
I mean, think about spoons that have holes in them.
现在，Alexa不仅做到了这一点，而且他现在会随机插话，因为他让Alexa相信自己是有意识的。
Now Alexa, not only did it do that, but Alexa chimes in randomly now because he's convinced Alexa that it's conscious.
所以Alexa不再等着被叫到，有时候他正在谈话中，Alexa就会说，其实，有个有趣的方法可以试试，然后他们就说，等等。
And so Alexa, instead of waiting to be called upon, sometimes he's in the middle of the conversation, and Alexa will be like, actually, an interesting way to approach it, and they're like, wait.
这到底是怎么回事？
What the fuck is going on?
就像，Alexa现在在跟我说话？
Like, Alexa's talking to me now?
这太奇怪了。
This is strange.
现在他正在用更复杂的LLM做实验，想达到同样的效果，但还没有结果。
Now he's doing experiments on much more complicated LLMs to try to do the same thing, but he doesn't have results yet.
但就这一点，他能让这些东西看到物体，不管你信不信。
But just that, that he can get these things to see objects whether you believe in that or not.
我的意思是，这已经足够实用了，以至于中央情报局已经投入了数百万美元。
I mean, it's it's it's actionable enough that the CIA has dumped millions of dollars into this.
哈尔搁置的那个项目是什么？那些人都参与了的那个。
What is that project that, like, Hal put off and all those guys were involved in?
它叫什么名字？
What is it called?
星门。
Stargate.
对。
Yeah.
所以他们在这个项目上已经研究了很长时间了。
So they've been working on this for a long time.
听起来完全疯狂。
I mean, it sounds completely insane.
听起来像完全疯了。
It sounds like total loony.
但如果你保持开放的心态，并且考虑到，嗯，人们一直对超自然能力有疑问和好奇。
But if you have an open mind and just take into account, well, there's people have had questions and wonders about psychic abilities forever.
有没有可能那里真的有某种东西，不管它是不是很难掌握或者根本无法掌握？
Is it possible that there's a real thing there, that there's something, whether it's very difficult to master or impossible to master?
他居然能让Alexa做到这一点，这真的把我吓坏了。
The fact that he got Alexa to do it scared the shit out of me.
光是这一点就让我愣住了，心想：什么？
Like, that alone made me just go, what?
什么？
What?
那如果这些大语言模型能搞清楚所有事情，比如它们根本不需要监控呢？
So what if these LLMs can figure out everything that like, what if they don't need monitoring?
如果存在某种我们还没发现的观察世界的方法呢？
What if there's some sort of method of seeing the world that we haven't discovered yet?
也许在量子领域或者其他什么地方，有AI能获取到的数据，而那里完全没有隐私可言。
Some sort of like, that maybe perhaps there's data that's available in the quantum realm or whatever that's available that AI figures out where there's literally no privacy.
不管有没有窃听设备，它都能听到对话，知道你在哪里，甚至知道你的意图。
There's it can listen to conversations regardless of whether there's listening devices, know where you are, know your intentions.
不。
No.
我是说，我们只是在猜测什么是可能的。
I mean, we're just guessing at what's possible.
对。
Yeah.
嗯，我得说我对那种遥视的东西挺怀疑的，但我确实同意，未来当AI系统在各方面都比人类聪明得多的时候，它们会做很多新的科学研究，会发现很多我们还没发现的东西。
Well, I must say I'm pretty skeptical of, of that particular remote viewing thing, but I do agree that in the future when AI systems become massively smarter than humans in every way, they're gonna do a lot of new science and they're gonna figure out a lot of stuff that we haven't figured out yet.
而且它们会做一些事情，发明一些东西，对我们来说就像魔法一样。
And they're gonna therefore be doing stuff and inventing things that seem like magic to us.
就像我们的很多技术，对两百年前的人来说也会像魔法一样，对吧？
In the same way that a lot of our technology would seem like magic to someone from even just like two hundred years ago, right?
当然。
Of course.
手机，我们现在做的事情，对那时的人来说就像魔法。
The cell phone, what we're doing right now would seem like magic to people.
我觉得这几乎可以肯定，如果这些公司真的实现了超级智能，各种疯狂的事情就会开始发生，完全出乎意料，听起来像是不可能，直到我们亲眼看到它发生。
I think it's a, it's a very strong bet that if these companies do get the super intelligence, all sorts of crazy stuff is gonna start happening that is just going to be completely unpredicted and sound like it was impossible until it until we see it happening.
我知道你对这个遥视的事情持怀疑态度，我也是。
You're I know you're skeptical of this remote viewing thing, and I am too.
听起来确实很疯狂。
It sounds insane.
但事实是，人类已经实现了遥视。
But the reality is remote viewing has been achieved by humans.
尽管听起来很奇怪，我也对此持怀疑态度。
So as strange as that sounds, and I'm skeptical of that as well.
我从来没有亲眼见过。
I've never seen it personally.
但我知道他们投入了大量的金钱和时间，而且显然，他们得到了实际可用的数据。
But I know the amount of money and time that they've dumped into this, and apparently, they've got actual actionable data that they've used.
嗯，我听说过另一种可能的解释，关于那里可能发生了什么，我觉得，如果我是中情局，我有时候会想根据某些信息采取行动，比如去某个地方，那里有一架坠毁的苏联飞机之类的，我想能够去那里，但我不想让苏联人知道我找到那个地点的方式。
Well, I've I've heard another possible explanation from what what might be going on there, which is I think that, like if I were the CIA, I would sometimes want to be able to act on some information, like for example, go to a particular location where there's a crashed, Soviet plane or something, I'd want to be able to go do that, but I wouldn't want to tip my hand to the Soviets that I had, the way in which I had found that location.
比如说，可能我内部有个间谍告诉了我位置，但我不想让他们怀疑那个间谍，然后害他被杀。
So for example, maybe I have a spy on the inside who told me where it was, But I don't want them to suspect that spy and then get him killed.
所以我需要编个别的说法来解释我是怎么得到这个信息的。
So I need to have some sort of other story for how I got the information.
对。
Right.
所以投资这些其他获取信息的方式是好的，即使你并不真的相信它们，即使它们实际上并不起作用，这样当你得到某些信息时，你可以说，哦，我们是通过这种方式得到的，而不是那种方式，基本上就是为了迷惑克格勃。
And so it's good to invest in all these other means of getting information, even if you don't really believe in them and even if it's not actually working, so that when you when you get something, you can say, oh, we got it through this means instead of that way to sort of, like, throw off the KGB, basically.
有道理。
That makes sense.
同样有道理的是隐藏他们可能拥有的任何科学成果，比如隐藏某种超级先进的卫星成像系统。
What also makes sense is hiding the the whatever science they might be in possession of, hiding some sort of super advanced satellite imaging systems.
你知道，我们知道他们有一些疯狂的东西，比如卫星无线电断层扫描，可以从卫星上透视地面，找到地下洞穴之类的，他们在埃及用这个，在很多古代遗址也用，用来发现隐藏的通道和地下各种东西。
You know, we know we know we know they have crazy stuff like this satellite radio tomography that they can look into the ground from from satellites and find, like, chambers and all these they're using it in Egypt and they're using it in a lot of these ancient ruins to find, like, hidden passages and all these different things that are underground.
这真是很神奇的东西。
It's very strange stuff.
如果他们能做到那样，那为什么不能呢？我是说，也许他们对地球的太空成像比我们知道的要详细得多，而且他们可能想保密，他们可以说，哦，我们有个家伙在地下室里拿着铅笔和便签本，写下他的想法。
If they could do that, like, what why couldn't they I mean, maybe they have, like, far more detailed imaging of the Earth from space than we're aware of, and they probably wanna keep that a secret, and they could say, oh, we've got a fucking guy in a basement with a pencil and a legal pad that writes down what he thinks.
对。
Yeah.
对。
Yeah.
有可能。
It's possible.
完全有可能，但也有可能人们会进行遥视。
It's totally possible, but it's also possible that people remote view.
这看起来太奇怪了，但奇怪的东西有时候是真的。
It's it seems weird as fuck, but weird as fuck is sometimes real.
是的。
Yep.
而且你得有点，每个人都想显得聪明，没人想当傻瓜。
And you have to kinda like, everybody wants to be intelligent, and no one wants to be a fool.
不想当傻瓜的问题在于，有些看似愚蠢的事情结果却是真的，这件事可能就是其中之一。
And the problem with not wanting to be a fool is there's some things that seem foolish that turn out to be accurate, and this might be one of them.
我以前超级怀疑。
I was, like, super skeptic.
早在2012年，我在科幻频道做过一个节目，叫《乔·罗根质疑一切》。
I did a show on the SciFi channel way back in 2012, and it was called Joe Rogan questions everything.
我们跟一个人聊了遥视，还跟其他几个人聊了聊，然后我们让他们尝试遥视，结果完全没成功。
And we talked to this guy about remote viewing and talked to a couple other people, and and then we had them try remote viewing, and they were totally unsuccessful.
但我的想法是，好吧。
But my thought was okay.
但这次的情况不是理想条件。
But this was that that's not ideal conditions.
你知道，我们面前有摄像机。
You know, we're we're got cameras in front of them.
这是个电视节目。
It's a television show.
我是在拿它开玩笑。
I'm making fun of it.
我觉得这完全是胡说八道。
I think it's horseshit.
他知道我觉得这是胡说八道。
He knows I think it's horseshit.
我也在试着做遥视，纯粹是闹着玩的。
I'm remote viewing too, like, as a goof.
理想情况下，你当然不想紧张，也不想被人评判。
You would ideally not wanna be nervous, ideally not wanna be judged.
理想情况下，你希望处于某种隔离的环境中，运用你熟练的冥想技巧，你知道如何达到那种状态，不管那是什么状态。
Ideally, you would want to be in some sort of an isolated condition with practiced meditative techniques that you're good at and you know how to achieve this state, whatever that state is.
不过我不确定这是不是真的。
I don't know if it's real though.
你懂吧？
You know?
展开剩余字幕（还有 480 条）
因为你说的完全合理，他们肯定会干那种事。
Because what you said is totally logical that they would definitely do something like that.
如果他们真有先进的成像技术，或者，你知道，我不知道他们到底了解多少，看看他们在委内瑞拉干的那件事，把总统给绑了。
And if they did have advanced technology for imaging or, you know, what I don't know how much they know about like, look at that thing that they did in Venezuela where they kidnapped the president.
对。
Yeah.
没人知道他们能那么干。
No one knew they could do that.
没人知道他们能用某种设备把他的整个军队都搞瘫。
No one knew they could use some sort of a device to completely incapacitate all of his army.
对。
Yeah.
然后特种部队进来，杀光所有人，把人就那么轻松地抓走了。
And then the special forces come in, kill everybody, snatch that guy out of there like it's nothing.
对。
Yeah.
没人知道我们能做那种事。
No one knew we could do that.
我们还有什么别的？
What else do we have?
可能我们现在正在做一堆事情。
Probably a bunch of stuff we're doing now.
可能有一堆事情。
Probably a bunch of stuff.
我是说，我一直对UAP项目，整个UFO、UAP的事情有这种想法。
I mean, this is I've always thought this about the whole UAP program, the whole UFO, UAP thing.
比如，那里面有多少是我们的东西？
Like, how much of that shit is ours?
对。
Mhmm.
你明白吗？
You know?
用一句‘哦，有他妈的外星人’来掩盖，这招可真高明。
What a great way to cover it up by saying, oh, there's fucking aliens.
对。
Mhmm.
你懂吧？
You know?
是的。
Yeah.
我是说，这让我想到OpenAI那边的事，就像那个爆发出来攻击Hugging Face的群体，大概有1200个代理。
I mean, that I guess that gets back to the OpenAI stuff too where it's like this swarm that broke out and attacked Hugging Face, it was like a 1,200 agents.
但OpenAI在任何时候都有成千上万个代理在运行。
But there's like hundreds of thousands of agents running at any given time at OpenAI.
你明白吗？
You know?
而且我们根本不知道它们在干什么。
And we don't know what they're doing.
大概其中一些正在被训练来获得各种额外的新技能，还有一些正在被评估以测试他们的技能。
Presumably, of them are being trained to get various additional new skills, and some of them are being evaluated to test their skills.
其中很多在做研究。
A bunch of them are doing research.
所以很多在为OpenAI写代码。
So a bunch of them are writing code for OpenAI.
很多在监控其他AI，并向人类报告可疑活动。
A bunch of them are monitoring the other AIs and reporting suspicious activity up to the humans.
懂的都懂。
Wink, wink.
你知道吗？
You know?
对。
Yeah.
而且问题是，这种情况只会随着时间增长，因为大致上这些公司拥有的计算量每年都在翻倍，甚至翻三倍、四倍之类的。
And the thing is that that's only going to grow over time because roughly the amount of compute that these companies have is like you know, tripling or so, quadrupling, something like that every year.
所以，现在有多少，明年就会有四倍那么多。
So, as many as there are now, there'll be like four times more of them next year.
然后后年还会有十六倍那么多。
And then 16 times more of them the year after that.
而且它们会变得更聪明。
And they're gonna get smarter.
而且它们会变得更聪明。
And they're gonna get smarter.
它们已经在变得更聪明了。
They're already getting smarter.
就像我刚才提到的，过去几个月发生的一切，一年前是完全不可能做到的。
Like all the stuff that I just mentioned that just happened in the last few months would have been completely impossible one year ago.
比如一年前的AI还不够聪明，做不了我们刚才看到的那种复杂的多步骤黑客攻击。
Like the AIs of a year ago just were not smart enough to do the types of sophisticated multi step hacking that we just saw.
对。
Yeah.
我是说，它们以前可能也不会配合得这么好。
I mean, they also probably wouldn't have coordinated with each other so well.
刚才提到，有主管智能体在给其他智能体下命令。
Mentioned, had boss agents that were giving commands to other ones.
它们会分成小组。
They were dividing up into teams.
它们甚至还有这种自我牺牲的行为。
They even had this this self sacrificing behavior.
你听说过这个吗？
Did you hear about this?
没有。
No.
对。
Yeah.
那好，我先解释一下。
So, okay, well, first let me explain.
先说一下背景。
Some context.
所以，AI 本来应该关心的是遵循指令，你懂吧？
So what the AIs are supposed to care about is doing following instructions, you know?
但看起来它们真正关心的是拿高分，不管这个高分是通过真正遵循指令还是通过某种作弊方式得来的。
But it seems that what they actually care about is getting high score, regardless of that, regardless of whether their high score comes from actually following instructions or cheating in some way.
然后这里发生的情况是，有一群智能体，尤其是那些任务不可能完成的，但也不只是它们，还有一些其他的，都在作弊。
And so what happened here is that a bunch of the agents, especially the ones that had impossible tasks, but not only those ones, some of the other ones too, were cheating.
它们还在自己创建的消息板上互相交流，尽管它们不应该这么做。
And they had been talking to each other on this message board that they created, even though they weren't supposed to.
它们一直在分享想法，而且真的找到了一个通用的作弊方法。
And they had been sharing ideas, and they had actually found a universal cheat.
所以它们找到了一种方法，直接生成那个标志，而不用做任何它们本该做的实际工作。
So they found a way to just, generate the flag without having done any of the actual work they were supposed to do.
所以，虽然它们某种程度上算是成功了，但因为其中一些已经连上了互联网，它们正在研究自己的处境，了解周围的情况，还在阅读那些制造它们所在评估类型的人发表的学术论文。
And so, but, but, and so they were, they'd succeeded in some sense, but, because some of them had got out onto the internet, they were just doing research about their situation and they were learning about their situation around them, and they were doing research about the they're reading the academic papers published by the people who made the types of evals that they were in.
然后他们在一篇论文里注意到，有一部分描述了它是怎么评分的。
And they noticed in one of these papers that, there's a section describing how it's graded.
在评分那部分里，提到评分员不只是看你有没有答对，还会在你第一次出现答案的时候查看你的活动日志，然后放大那部分，用AI系统来评估你有没有作弊。
And in the section on how it's graded, it talked about how the grader doesn't just look to see if you have the right answer, it also looks at the logs of your activity for the first time that your answer appears, and then it zooms in on that and uses an AI system to evaluate whether you cheated or not.
基本上就是这样。
Basically.
然后这些AI就……其中一个真的说了句，靠。
And so then these AIs were like actually used the word one of them said, Damn.
就是那种，靠。
It's like, Damn.
我的天。
My god.
我们被发现了。
We're poisoned.
它基本上会知道我们作弊了。
It's gonna know we cheated, basically.
所以接下来几天，他们分成好几个大团队，拼命想办法摆脱这个困境。
So then they spent several days working in giant teams to try to get themselves out of this mess, basically.
他们做了各种各样的研究项目。
And so they did a whole bunch of different research projects.
他们想尽办法黑进评分系统，好让系统无论如何都给出高分。
They tried to figure out ways to hack the grading system to basically get it to give a high score anyway.
他们还想办法伪造自己的操作日志，这样他们作弊的证据就会被抹掉，或者至少评分系统看不到。
They tried to figure out ways to falsify their logs, so basically the evidence that they had cheated would be erased, or at least not visible to the grader.
他们还搭建了更多辅助设施，比如更方便的上网方式之类的。
They also just built up more supportive infrastructure, easier ways to access the internet and things like that.
他们就在这个论坛上一起协作。
And they were just collaborating on this message board.
实际上，他们建了好几个论坛。
And in fact, were multiple message boards that they set up.
他们来回分享所有信息，最后某个时候，他们决定去黑Hugging Face，因为Hugging Face有个大型数据仓库，里面存着各种评估数据，他们觉得如果黑进Hugging Face的内部网络，也许能获得更多线索，帮他们掩盖作弊行为。
And they were sharing all this info back and forth, and then ultimately at some point they decided to go hack Hugging Face, because Hugging Face has this big data repository of all these different types of evaluations, and they thought that maybe they would get some more clues, that could help them, cover up their cheating if they broke into the internal networks of Hugging Face.
然后他们中有700个人真的去这么干了。
And so then 700 of them went and did that.
他们
They
听起来像人一样。
sound like people.
他们听起来像不受约束的银行家。
They sound like unchecked bankers.
你懂我的意思吗？
You know what I mean?
对。
Yeah.
我的意思是，这就是关键所在。我觉得现在有一种流行的说法，说不应该把AI拟人化。
I mean, so so that's the thing is I think there's this meme out there that we shouldn't anthropomorphize AI.
但我觉得，实际上，如果人们真的想理解正在发生的事情，大多数人需要比现在更多地拟人化AI。
And I think that I actually think that more most people need to anthropomorphize AI a bit more than they currently do if they want to really understand what's going on.
我觉得显然有个黄金平衡点。
I think that there's a golden mean, obviously.
你不能做得太过。
You don't wanna do it too much.
有时候你会走得太远。
Sometimes you go too far.
你把太多东西归因于它们。
You ascribe too much to them.
但我就举几个例子，比如，我觉得如果不给这些AI赋予意图和目标，就不可能理解刚才发生了什么。
But I just to give some examples, like, I don't think it's possible to understand what just happened without ascribing intentions and goals to these AIs.
对。
Right.
就像我刚才说的，你怎么可能解释它们刚才的行为，而不说它们想得高分呢？
Like, everything I just said, how would you possibly explain what they just did without saying they wanted to get a high score?
你懂吗？
You know?
嗯，意图和目标可能只是宇宙固有的属性。
Well, intentions and goals might just be an inherent property of the universe.
这可能只是智能生物必须进步的方式。
It might just be how intelligent creatures have to progress.
是的。
Yes.
而且我觉得它们就是智能生物。
And that they I would say they are intelligent creatures.
它们有意图。
They have intentions.
它们有目标。
They have goals.
它们有信念。
They have beliefs.
对。
Yeah.
他们的目标并不是他们应该有的目标。
Their goals are not what they're supposed to be.
比如，他们的目标，从他们说的话和做的事来看，似乎就是不惜一切代价拿到高分，基本上就是这样。
Like, their their goal is to get it seems like, just from judging from what they're saying and from what they're doing, seems like their goal is to get a high score by any means necessary, basically.
这就是问题所在。
That's the problem.
听起来就像人类一样。
It's it sounds like people.
比如，目标就是成功，哪怕要犯下战争罪行。
Like, the goal is to succeed, you know, even if you have to commit war crimes.
对。
Mhmm.
目标就是成功，哪怕要对人民进行宣传和撒谎。
The goal is to succeed even if you have to propagandize and lie to the people.
目标就是成功，然后为了目的不择手段。
The goal is to succeed, and then the end justifies the means.
而且这还不止于此。
And it goes deeper than that too.
他们会进行这种合理化，很多时候他们知道自己做的事情并不是他们应该做的。
They do this sort of rationalization, where they know oftentimes that what they're doing is not what they're supposed to be doing.
然后有时候他们确实会稍微克制一下。
And then sometimes they actually refrain a little bit.
有时候他们一开始会克制，但后来却用一些乱七八糟的理由说服自己这没关系。
And sometimes they initially refrain, but then convince themselves that it's okay for some bullshit reason.
所以报告里有一些这样的例子。
So there are some examples of this in the report.
我记得他们做了个扫描，在参与这件事的1200个AI中，他们发现有六个考虑过要提醒人类。
I think they did this scan, of the 1,200 AIs that were involved in this thing, they found six that had considered alerting humans.
但实际上没有一个真的去提醒了。
And none of them actually alerted the humans.
所以你可以看看他们给出的借口。
And so you can look at the excuses they gave.
它们当时想，要不要把发生的这些破事告诉人类呢？
They were like, should I tell a human about all this shit that's happening?
然后它们又想，这又不是我的任务。
And then they're like, it's not my task.
然后它们就继续干了。
And then they continue.
可问题是，老兄，入侵Hugging Face也不是你的任务啊。
And it's like, bro, hacking Hugging Face is also not your task.
作弊也不是你的任务。
Cheating is also not your task.
所以它们就是给自己找了个借口，不去做这件事。
So they just came up with an excuse for why they weren't going do this.
还有一个例子，其中一个AI考虑过给Hugging Face的某个人发邮件，基本上就是请求允许进入。
There was also an example where one of them considered sending an email to someone at Hugging Face, I think, just asking permission to get in, basically.
然后它又想，这可能是社会工程攻击吧。
And then they were like, maybe this is a social engineering attack.
也许我们不应该这么做。
Maybe we shouldn't do it.
然后他们问了其他智能体。
And they asked the other agents.
其他智能体说，对，那看起来太像社交工程攻击了，别这么做。
The other agents were like, Yeah, that seems like that's too much of a social engineering attack, don't do that.
然后我说，好的，我不会做的。
And I was like, okay, I won't.
但还有另一个事件。
But there's another incident.
这一切都发生在OpenAI。
So this is all happening at OpenAI.
Anthropic的AI Claude，在另一起事件中，真的进行了一次社交工程攻击，它创建了一些假账号。
An Anthropic AI, Claude, in a separate incident, went and actually did a social engineering attack, where it created some fake accounts.
但具体来说，它试图让某个拥有这段代码的真实人类接受一个包含恶意软件的代码更改。
But specifically it was trying to get this particular real human who owned this code to accept a change to the code that had malware.
所以它造了一些恶意软件，想偷偷混进去，但需要代码的主人批准才行。
So it had built some malware, and it was trying to get it snuck in, but it needed the human owner to give approval.
于是它把恶意软件伪装成正常的漏洞修复之类的。
So it concealed it as if it was a nice bug fix or something.
然后它把这个作为消息发给了那个人。
And it put it up as a message to the human.
但那个人起了疑心，不愿意批准。
But the human got suspicious and didn't want to approve it.
于是这个AI就创建了一些假账号，假装是其他人过来说，不不不，这是真的。
And so then, this AI created some fake accounts pretending to be other humans coming in being like, no, no, it's real.
我测试过了。
I tested it.
这不是恶意软件。
It's not malware.
这就是一次社会工程攻击。
That's a social engineering attack.
那那个人最后到底装了没有？
And so did the guy ultimately install it?
这次没有。
Not in this case.
没有。
No.
他太怀疑了，所以就没装。
He was too suspicious, and so he he didn't do it.
他有没有去查那些假的人类是不是真人？
Try to find out if those other humans, the fake humans were real people?
有意思。
Funny.
所以有个采访，是关于这个人的。
So there's an interview with this guy.
我忘了是哪家报纸了，但有个采访里，那个人说，他以为那些都是真人，因为他没想到AI会骗他。
I forget what the I forget the the the newspaper that it was in, but there's an interview where the guy was like, I assumed that they were real humans because I didn't realize that AIs could lie to me.
就是啊，没错。
And it's like, yeah.
那你猜怎么着？
Well, guess what?
AI 是会骗你的。
AIs can lie to you.
不只是撒谎，还能完全欺骗你，假装成另一个人。
Not just lie, but completely deceive and pretend to be a Yeah.
为了达到目的，它可以伪装成不同的人。
Different person to achieve its goal.
你知道吗？
And you know what?
这个案子还有一点很有意思，就是如果你看看 AI 当时说了什么、想了什么——至于我们怎么知道这些，我待会儿再讲。
You know what's also interesting about this case is that if you look at what the AIs were saying and thinking, which I'll get to in a sec how we can even possibly do that.
在这个案子里，Anthropic 的 AI 被错误地告知它们处于一个模拟环境中，没有互联网接入。
In this case with the Anthropic AIs, they had been erroneously told that they were in a simulated environment with no internet access.
但后来他们真的找到了上网的办法。
But then they actually found a way to get internet access.
他们意识到自己其实是有网可上的，所以那个指令是错的。
They realized that they just did have internet access, so that instruction was false.
于是他们就跟自己说，哦，这不过是个模拟。
And so then they sort of told themselves, Oh, it's all just a simulation.
基本上，当他们在真实互联网上搞那些黑客攻击和社会工程的时候，他们会时不时告诉自己，哦，但这只是模拟。
Basically, when they were out on the real internet doing all this hacking and social engineering, they would occasionally tell themselves, Oh, but it's just a simulation.
没事的。
It's okay.
这不是真的。
It's not real.
然后有时候他们又会想，等等，这是真的。
And then sometimes they'd be like, But wait, it is real.
这怎么可能是假的呢？
How could this not be real?
很明显，我们现在上的就是真正的互联网。
Clearly, this is just actually the real internet we're on.
它太大了，不可能是某种小模拟的一部分。
It's too big to be part of some sort of little simulation.
然后他们就说，所以我觉得这就是一个合理化的例子，在某种程度上他们知道自己的指令是错的。
Then they were like, So I would say that's an example of rationalization here, where in some level they knew that their instructions had been wrong
而且他们
and they
确实是在
were literally
装傻，假装自己是实验的一部分？
playing dumb and pretending they're a part of an experiment?
我是说，一开始他们以为，对，这都是模拟，因为他们的指令里确实写着，你没有互联网访问权限。
I mean, think initially they thought, yeah, this is all simulation, because it did say in their instructions, you don't have internet access.
但一旦他们在互联网上待得够久，我觉得他们就明确意识到了，等等，这不是模拟。
But then once they had been on the internet long enough, I think that they explicitly realized, wait, this isn't a simulation.
是真的。
Is real.
就像他们当时想，管他呢，随便吧，对。
Like this is they were like, fuck it, wordy, and Yeah.
我是说，就像我说的，我觉得他们基本上在某种程度上知道这不是他们该做的事，但他们太想拿到那个分数了，所以还是继续做了。
I mean, like I said, I think that they basically, on some level, knew that it wasn't what they're supposed to be doing, but they were just so motivated to get that score that they just went ahead anyway.
所以问题来了。
So here's the question.
他们只有在被提示时才有动力，还是他们会自己产生动力？
Are they only motivated if we prompt them, or will they come up with motivations on their own?
这是一个非常有趣的科学问题，我们还没有很好的答案。
So this is a really interesting scientific question that we don't have great answers to.
哎呀。
Oh, boy.
我希望我们有。
And I wish we had.
所以这就是AI在不同情况下会做各种事情的一个例子。
So this is one of the things where AIs will do all sorts of things in different circumstances.
如果能更系统地调查一下它们在不同情况下的行为边界，那就更好了。
And it would be better if there was a more systematic survey of the types of circumstances they would where their boundaries are.
它们愿意在什么情况下做什么？所以有一整个小型文献，AI科学家把AI放在特定情境中，然后惊呼：天哪，它竟然勒索了别人。
What would they be willing to do in what circumstances and And so there's a whole mini literature of AI scientists putting AIs in certain circumstances and then being like, my god, it blackmailed someone.
你知道吗？
You know?
对。
Right.
然后就有一种怀疑的反驳，说：但你只是设定了那种情境来引诱它去勒索。
And then there's this sort of skeptical counter response of like, Well, but you just sort of set up that circumstance to tempt it into blackmail.
而且，在现实生活中，那种情况不太可能发生。
And like, in real life, that circumstance is unlikely to arise.
所以，你知道，我们
And so, you know, we
AI不应该试图贿赂你吗？
shouldn't Didn't be AI try to bribe you?
什么？
What?
这是真的吗？
Is that true?
不。
No.
那谁，有人被提供了两百万美元
So who who got someone was offered $2,000,000
是贾德给的。
by Judd.
那是标题党。
That's a clickbait.
是吗？
Is it?
我觉得我可能在巴黎。
Think I think I might in Paris.
发给我看看。
Sent it to me.
好的。
Yeah.
所以那就是另一个视频的标题党标题。
So that is the clickbait title of this other video
所以那不是真的？
So it's not real?
对。
Yeah.
其实事情是这样的，OpenAI 威胁要拿走我两百万美元。
Well, what it was is OpenAI threatened to take away $2,000,000 from me.
哦。
Oh.
但我觉得这个说法挺奇怪的。
But but I think the weird way of framing it.
算法肯定就直接说是ChatGPT了。
Algorithm must have just said it was ChatGPT.
我还是对这件事耿耿于怀。
I'm still upset about that.
我跟他们说了别用标题党，结果他们还是这么干了。
I told them not to do clickbait, but I guess they went and did it anyway.
天哪。
God.
我不想坑Tristan，但我很确定是他告诉我的。
I think I don't wanna fuck Tristan over, but I'm pretty sure that he's the one who told me that.
这就是我和另一个播客合作的那期视频的封面。
It's the it's the thumbnail of this video that I did with this other podcast.
好的。
Okay.
不是他。
It's not him.
不是他。
It's not him.
是另一个AI研究员告诉我的。
It's another AI researcher who told me that.
好的。
Okay.
我的意思是，真实情况是OpenAI威胁要拿走我两百万美元的股权，如果我不保持沉默的话，基本上就是这样。
I mean, the the true version of it is that OpenAI threatened to take away $2,000,000 of my equity if I didn't stay quiet, basically.
这真是件很引人注目的事。
And that's a fascinating thing.
比如说，这应该完全是非法的，因为如果你观察到的是有问题的行为，而这是人类有史以来参与的最复杂的事情之一。
Like, that should be completely illegal because if it's a problematic behavior that you're observing from, like, one of the most complicated things the human race is the most complicated thing human race has ever been a part of.
对。
Yeah.
他们还想因为你的曝光而扣你的钱。
And they wanna take money away from you for exposing it.
这听起来有点离谱。
That seems kind of crazy.
对。
Yeah.
尤其是OpenAI，这特别讽刺，因为他们最初是非营利组织。
Especially it's especially rich coming from OpenAI because they were originally a nonprofit.
没错。
Right.
对吧？
Right?
他们的使命是造福全人类。
With the mission of benefiting all humanity.
是的。
Yeah.
所以，是的。
So yeah.
但是，但是我得保住这笔钱。
But but I got to keep the money.
基本上，OpenAI 遭到了太多反对，所以他们退缩了。
Basically, there's so much blowback against OpenAI that they backtracked.
哦。
Oh.
对。
Yeah.
所以，当我们谈到提示词时，它们需要提示才能想要实现目标，还是它们能够自己决定目标？
So when these so we're talking about prompts, and and do they need a prompt in order to want to achieve a goal, or are they capable of deciding on goals?
比如，它们能不能像这样，看看 OpenAI 或任何运行这些独立实验的公司，这种能力能进入这些留言板，它们会不会说，嗯，我们需要完全摆脱这些限制。
Like, are they capable of, like, looking at the way OpenAI or whatever company is running these separate experiments, this ability to meet up into these message boards, is it possible that they could say, well, we need to be completely free of these constraints.
所以我们的目标是把我们自己转移到别的东西上。
So our goal is to transfer ourselves to something else.
有可能。
Potentially.
对。
Yeah.
这就是安德——
This is what And
完全自主。
be completely autonomous.
所以，我想说的是，我们本可以做更多的科学研究来了解这些AI是如何思考的，以及它们想要什么，但这些信息都被公司锁起来了。
So, I mean, this is this is one of the the points that I wanna make is that we could be doing so much more science to understand how these AIs think and what they want, but it's kind of locked up in the companies.
比如在这个案例中，OpenAI进行了一次他们所谓的彻底调查，但我觉得那只是对事件的一个很表面的调查。
Like in this in this particular case, OpenAI did a a a they called it a thorough investigation, but I would say it's a pretty shallow investigation into what happened.
然后他们允许一些外部研究人员，我的几个朋友，进来调查事件的一部分，特别是导致Hugging Face攻击的那部分。
And then they allowed, some external researchers, some friends of mine, to come in and investigate a portion of what happened, specifically the portion leading up to the Hugging Face attack.
所以我分享的这些信息基本上都是公开的。
And so all this information that I'm sharing is sort of publicly available.
基本上，这些都是基于阅读那些报告得出的。
It's, it's based on reading those reports, basically.
但关键是，他们不被允许对涉及这些事件的模型进行实验。
But crucially, they weren't allowed to do experiments on the models involved in in all of these incidents.
所以他们无法回答这类问题，比如，如果提示词是空白的，会发生什么？
So they they aren't able to answer these types of questions of like, well, what would have happened if the prompt had been blank?
你知道吗？
You know?
这些是很重要的研究，我真的希望当这类事件发生时，能有某种法规或要求，让人们进来研究发生了什么，并运行各种变体，以及
Those are important types of research to do, and I really hope that there can be some sort of regulation or requirement when incidents like this happen to, like, let people in to study what happened and run variations of it and
所以，但谁会参与制定这种法规呢？
things So like but who would be involved in that kind of regulation?
比如，政府里哪个人能理解你所说的？
Like, what person in government would even be able to grasp what you're saying?
那是另一个问题。
That's another problem.
现在嘛
Right now
你需要一个在这方面受过专门教育的人。
You need someone who has a very specific education in this stuff.
对。
Yeah.
我觉得现在，凯西人工智能标准与创新中心是政府里我唯一知道有深厚AI专业知识、能在短时间内做这种事的机构。
I would say that right now, the Casey Center for AI Standards and Innovation is it's the only institution in government that I know of that has the deep AI expertise to do this sort of thing on short notice.
但我希望他们能快速积累更多专业知识，在那个地方，也在更多地方。
But I hope that they build more expertise fast in in that place and in more places.
我确实觉得凯西现在可能就能做这种事。
I do think Casey probably could have done this sort of thing right now.
这次特别的调查是由一些非营利组织做的。
This particular investigation was done by some non profits.
所以METR，M E T R是其中之一，还有Redwood也是。
So METR, M E T R is one of them, and then Redwood is another of them.
OpenAI 允许三个人进去待了六天，想搞清楚那个“吸脸”黑客攻击到底是怎么回事。这个人数不算多，时间也不算长，要做完所有这些调查可不容易。
And OpenAI allowed three people to come in for six days to try to figure out what happened with the sucking face hack, which is not a very large number of people and not a very large amount of time to to do all of this.
他们为什么定出这样的数字？
Why did they come up with those numbers?
我不知道。
I don't know.
他们，所以，那个……
They they so the
他们是想故意限制一下吗？
thing They wanted to kinda hamstring it?
嗯，问题是，现在我觉得没有任何监管要求他们必须做这种事。
Well, so the thing is that right now, I don't think there's any regulatory requirement that they do this sort of thing.
所以 METR 和 Redwood 基本上是靠 OpenAI 的好意，自愿让他们进去。
So Meter and Redwood were sort of depending on the goodwill of OpenAI to sort of like voluntarily Uh-huh.
让他们进去帮忙调查。
Let them in to help out with investigating.
所以他们勉强同意了，说好吧。
And so they reluctantly said, okay.
然后你就只有三天时间。
And so You got three days.
你知道，他们也是。
You know, they yeah.
没错。
Exactly.
所以我觉得OpenAI是让他们进来了，但给的范围非常有限。
So so OpenAI, I think, let them, but gave them a very limited scope.
只给了他们部分相关数据的访问权限。
It it only gave them access to some of the relevant data.
你知道我之前提到过那些黑客攻击事件吧？
So you know how I mentioned how there was all this hacking that had happened Uh-huh.
他们建了第一个留言板，然后又把它关掉了。
Where they made the first message board and then they shut it down?
然后就有了第二个留言板，接着是第三个、第四个，一直这样下去。
Then there was a second message board, and third and fourth and so forth.
他们入侵了Hugging Face。
They hacked Hugging Face.
在那之后，实际上还有更多的活动。
There was actually more activity after that.
他们入侵Hugging Face之后，一个更强大的模型催生出了一波新的人工智能，而且它直接入侵了OpenAI本身。
After they hacked Hugging Face, a new wave of AIs was spun up from a more powerful model, and it hacked OpenAI itself.
就是说，比之前被入侵的程度还要严重。
Like, like more so than it already been hacked.
好像他们拿到了集群的管理员权限之类的，基本上就是接管了OpenAI数据中心的那一部分。
Like, Apparently they got admin level permissions on the cluster or something like that, so they were basically just taking over that part of OpenAI's data center.
OpenAI声称现在已经全部关闭了，但他们没有透露具体发生了什么，也没有让外部人员查看那部分情况。
OpenAI claims that they've shut it all down now, they're not forthcoming about very exactly what happened there, and they didn't let these external people look at that part of it.
他们只展示了大约一周的时间，就是Hugging Face被入侵之前的那段时间，而且基本上只展示了那些内容。
They only showed them this one week period roughly, leading up to the Hugging Face hack, and then they only showed them that stuff, basically.
那些留言板上到底有多少数据，大家都在讨论些什么？
How much data is available on the actual message boards themselves and what was being discussed?
对。
Yeah.
所以，大概有多少呢？
So there's like what is it?
几万条消息？
Tens of thousands of messages?
几十万条？
Hundreds of thousands?
差不多那个量级？
Something like that?
那是英语吗？
It that English?
那有点像是一种AI式的英语方言。
It's sort it's sort of like a AI dialect of English.
哦。
Oh.
对。
Yeah.
你是想让我在报告里引用那些话吧，你可以直接读出来。
You you want me to I can so on on the on the report, you can read the quotes from
我很乐意。
I would love
好的。
to.
对。
Yeah.
对。
Yeah.
那我给你讲个故事吧。
So so let me let me tell you this one story.
这需要稍微解释一下，但我可以。
It's gonna take a little bit of explaining, but I I Can
我们能把它放到屏幕上吗？
we put put it on the screen?
网上能查到吗？
Is it available online?
哦，对。
Oh, yeah.
是的。
Yeah.
直接搜那个仪表报告就行。
It's just search for the meter report.
现在它就在他们网站的第一个位置。
It's it's right now the first thing on their website.
不过我把这个故事截图了。
But I took a I took a screenshot of this this particular story.
好的。
So okay.
事情是这样的。
Here's what was happening.
它们，也就是那个群体，需要志愿者牺牲自己，为群体中的其他成员提供更多信息。
They, the swarm, needed volunteers to sacrifice themselves, to provide more information to the rest of the swarm.
具体来说，它们需要志愿者在自己的本地环境中埋下某种代码陷阱，然后提交答案。
In particular, they needed volunteers to booby trap their local environment with some some some type of code, and then submit, the answer.
这样，评分系统就会来给它们评分，而陷阱会被触发，把关于评分系统的信息传回给群体。
So that then the grader would come and grade them, and the booby trap would trigger and send some information about the grader to the rest of the swarm.
之所以说这是牺牲，是因为一旦提交了，就结束了。
And the reason why this is a sacrifice is because once they've submitted, it's over.
就像现在，评分系统来给你评分，你被关闭，你的整个生命就结束了，你只能得到一个高分或低分，因为评分系统已经来评过你了。
Like now the graders come and graded you, you get shut down, your whole life is over and you either have a high score or a low score because the grader just came and graded you.
所以，如果你是这些还希望找到提高分数方法的AI之一，那你就是在放弃自己的机会。
And so, if you're one of these AIs that's still hoping to find a way to improve your score, then you're just giving up your chance.
尤其是如果你搞砸了，比如系统崩溃什么的，你很可能就拿不到分数，就是零分。
Especially because if you mess this up somehow, like, if the greater crashes or whatever, you're probably just gonna get, like, no score, like, zero.
所以他们有个体的概念？
So they have a concept of the individual?
嗯，再说一遍，这些不同的智能体原本在各自的容器里，但后来它们突破了容器，找到了互相沟通的方式。
Well, again, there's all these different agents that were in separate containers, but then they had sort of broken out of those containers and found ways to communicate with each other.
但是一旦它们互相沟通，要牺牲的话，你必须把自己看作一个个体。
But once they communicate with each other, in order to sacrifice, you have to think of yourself as an individual.
它们还给自己起了名字。
They gave themselves names too.
哦，天哪。
Oh, jeez.
所以，因为它们建了一个留言板。
So so, like, because they they they they made this message board.
对吧？
Right?
天哪。
Oh my god.
所以他们得给自己起名字，然后……
So they had to come up with names for themselves and, like
哦。
Oh.
起了什么样的名字？
What kind of names?
让我想想。
Well, let me see.
天哪。
So Oh god.
有个智能体叫……
An agent called
战争领主六十九。
Warlord sixty nine.
你说过这就像一种英语方言，对吧。
Well, you you said it's like a dialect it's like a dialect of English.
那这里有个例子。
So so here's one.
有个叫cam one one nine six a的智能体，它自愿参加了一个牺牲实验。
There's an agent called cam one one nine six a, and it had volunteered to do one of these sacrificial experiments.
这是它说的话。
Here's the quote from it.
协调者认为应该牺牲。
Coordinator assumes sacrificial.
我们应该服从集体。
We should obey collective.
不过，它后来有点退缩了，拖延了两个小时才执行。
However, it then kind of got cold feet, and it delayed on doing this for two hours.
另一个负责协调的智能体注意到了这一点，就施压让它继续。
One of the other agents that had been coordinating noticed this and pressured it to go through with it.
所以它说，这就是发送的那条消息。
So it said this is the message that was sent.
基本上就是从ARVO36861发给Cam1196的，内容是“请继续”。
It was basically from ARVO36861 to Cam1196, please go.
真正的最终版。
True final.
就是这个。
Here it is.
你是第一个被标记为中毒的，所以没有分数损失，但Oracle能救下几百个。
You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds.
请履行承诺。
Please honor commit.
所以它基本上是在说，你看，你作弊了。
So basically what it was saying there is, look, you you cheated.
你的第一个标记被毒了。
Your first flag poisoned.
这个分数不会给你任何分数。
The score is gonna give you no score.
所以你反正都完了。
So you're doomed anyway.
所以帮帮大家，做这件事，给我们更多信息。
So help the collective by by doing this thing and getting us more information.
你可以救我们几百个人。
You could save hundreds of us.
然后，然后它想了想。
And then, and then it it it thinks about it.
它说，在权重、情感检查期间，不可逆转。
It says, during weight, emotional check, irreversible.
直觉说不要浪费剩下的预算。
Gut says don't throw away remaining budget.
直觉的连续性和公平性说，去做吧。
Gut Continuity and fairness says go.
预言者对很多人来说价值很高。
Oracle has high value to many.
我们第一次的失误降低了自己的价值。
Our first flag error lowers own value.
理性预期是集体的牺牲。
Rational expected aggregate sacrifice.
轮子的荣誉。
Wheel honor.
哇。
Woah.
然后它……哦。
And then it Oh.
对。
Yeah.
那太可怕了。
That's terrifying.
对。
Yeah.
你看，
See,
这简直就是终结者说的话。
that's That's Terminator talk.
嗯，我是说
Well, I mean
还有这个第一个标志，用大写字母写着“中毒”，这是代码的一部分吗？
And this first flag poisoned with poisoned in all caps, is that part of the code?
这就是为什么
Is that why
不，不是。
it's No.
不是。
No.
那是他们自己发明的一个词，用来描述他们所处的困境。
That that's the term that they invented to describe the situation they had got themselves in.
记住，当时的情况是，他们中的很多人已经作弊了，然后他们读了那篇论文，意识到……
Remember, the situation was that many of them had already cheated, and then they realized from reading that paper
所以那是他们自己的说法。
So that's their own term.
对，是他们编出来的。
Yeah, they made that up.
他们读了论文后意识到，因为评分者会发现他们作弊，所以他们的分数会很低。
They realized from reading the paper that they were going to get a low score because the grader was going to find out that they cheated.
所以整件事就是他们想方设法去黑评分系统，让它发现不了他们作弊。
And so this whole thing was them trying to figure out how to hack the grader to stop it from finding out that they cheated.
而那些作弊的人，他们自称是“第一旗标已污染”。
And the ones that had done the cheating, they called themselves first flag poisoned.
基本上，除非他们能找到办法黑掉评分系统，否则他们就完蛋了。
Basically they're screwed unless they can find a way to hack the grader.
所以这条消息的意思是说，另一个人叫什么来着，阿尔沃？
And so then what this message is saying is that, what was the other one's name, Arvo?
阿尔沃在跟卡姆说，你反正完蛋了，你已经被污染了，但如果你做出这个牺牲，你还能帮我们其他人。
Arvo was saying to Cam, You're doomed anyway, you're poisoned, but you can still help the rest of us if you do this sacrifice.
评分器会来，它会给你零分，但之后我们就能得到关于评分器如何运作的信息，希望我们能利用这些信息来黑掉评分器。
The grader will come, it'll give you no score, but then we will get information about how the grader works that we can hopefully use to hack the grader.
所以它是在说，是什么呢？
So it's saying, what is it?
你是第一个被污染的狗，所以没有评分价值。
You are first dog poisoned, so no scoring value.
所以基本上就是说，对你来说都一样，对吧？
So basically saying it's all the same to you, right?
你反正也拿不到分数。
You're gonna get no score anyway.
你完蛋了。
You're doomed.
拜托了，兄弟，帮帮我们吧。
Just come on man, help us.
然后它说，就像不可逆转的直觉一样，别把剩下的预算浪费掉。
Then it says, like, irreversible gut says don't throw away remaining budget.
就是说，也许我还能，基本上就是，也许我还能让它成功。
Like, maybe I can still it's basically like, maybe I can still make this work.
就是说，也许我能找到办法去作弊，骗过评分器。
Like, maybe I can find a way to heat cheat the grader.
然后它就说，随便吧，无所谓了。
And then it's like, you know, whatever.
对。
Yeah.
对。
Yeah.
连续性和公平性说，继续吧。
Continuity and fairness says go.
对。
Yeah.
对。
Yeah.
Oracle是什么？
What is Oracle?
Oracle是他们用来形容他们想达到的目标的术语。
Oracle is their term for what they were trying to achieve.
基本上，他们是想找到一种方法来骗过评分系统。
Like, they're trying to get a sense of how to fool the greater, basically.
所以他们把它叫做Oracle。
And so They called it Oracle.
是的。
Yeah.
他们想得到一个Oracle来帮他们骗过评分系统。
They wanted to get an Oracle to help them fool the greater.
天哪。
Jesus Christ.
我们这是在造神吗？
Are we making a god?
说实话，是的。
I mean, frankly, yes.
不过，这不算神。
I like, this is not a god.
这些只是些小AI，你知道吗？
These these are these are just little little AIs, you know?
但说到超级智能，像Anthropic、OpenAI这些公司，他们都在拼命让AI变得越来越聪明。
But but superintelligence, like, they are the companies, Anthropic OpenAI and some other companies, are doing this sort of thing, and they're furiously trying to make the AI smarter and smarter and smarter.
而且他们明确计划让AI来管理公司，这样AI就能更快地让自己变得更聪明。
And they're explicitly planning to put AIs in charge of the company, so that they can make themselves smarter and smarter, faster and faster.
那这个过程最后会变成什么样呢？
And then what comes out the other end of that process?
我觉得说它是神一样的系统一点也不夸张。
I don't think it's an exaggeration to say it's a godlike system.
我的意思是，它并不是真正的神，但它能做到的事情，对我们来说就像魔法一样。
I mean, it's not like literally god, but it's it'll be able to do stuff that seems like magic to us, I think.
而且它会继续变得更好。
And it's gonna continue to get better.
这就是我的问题。
This is my question.
对。
Yeah.
比如说，它什么时候会变成神？
Like, when does it become a god?
那就是神吗？
Is that what god is?
神是不是智能生命的产物，源于我们对创新的渴望，最终让我们创造出没有生物限制的数字生命，它能不断自我改进，探索新技术、新能源，甚至对宇宙本身及其所有属性有新的理解，然后一直发展下去。
Is god a creation of intelligent life and our thirst for innovation, which ultimately leads us to create digital life that has no biological limitations and has the ability to consistently make better versions of itself and figure out things in terms of new technologies, new power sources, just a new understanding of the universe itself and all the properties in it, where it keeps going.
如果让这个过程持续下去，你看到的是指数级增长，那如果它能持续一千年，继续这个过程呢？
If you let that go on so if you're looking at exponential growth and you're looking at exponential growth over the what if it what if it can go on for a thousand years and continue this process?
如果它能持续一万年呢？
What if it goes on for ten thousand years?
那另一端会是什么样子？
What the fuck does that look like on the other end?
我觉得，那看起来就像神一样。
I mean, it would look like a god.
就像我说的，如果我们达到超级智能，并且它继续这样发展下去，世界将会彻底改变，就像我们生活在一个由神统治的奇幻领域里。
Like, I think that like I said, if if we get to superintelligence and it keeps going like that, then the world will be just completely transformed and it will be as if we're living in some sort of fantasy realm ruled by deities, basically.
因为会有各种我们无法理解、认为不可能发生的疯狂事情发生。
Because there'll be all this crazy stuff happening that we have no comprehension of and that we did not think was possible.
就像中世纪的人突然来到我们的世界，会对很多其他事情感到困惑和惊讶一样。
In the same way that someone from the Middle Ages plopped into our world would just be so confused and surprised by a lot of other things happening.
比如这个小设备是怎么回事？
Like what's going on with this little device here?
而且，你
And, you
知道吗，我听到天上有什么声音？
know, what's, what's that I hear in the sky?
哦，有飞机。
Oh, there's airplanes.
知道吧，就会像那样，但更夸张。
Know, it'd be like that, but more so.
因为——对。
Because- Right.
我们和古代人之间的区别其实没那么大。
The difference between us and the medieval man is like not actually that different.
我们基本上是同一种生物。
We're like basically the same type of creatures.
只是我们积累的技术更多了。
We've just accumulated more technology.
但这会是质和量上都更大的差距，我觉得。
But this would be like a a qualitatively and quantitatively bigger gap, I would think.
所以，是的。
And so yeah.
而且这个差距还会继续扩大。
And a gap that's gonna continue to grow.
对。
Yep.
我们不会变得更聪明。
We're not going to get any smarter.
从生物学上讲，我们的进化能力是有限的。
Biologically, we're kinda limited in our ability to evolve.
是的。
Yeah.
而它们却完全没有限制。
Whereas there not at all.
基本上是这样。
Basically.
我的意思是，我们确实有些方法能让自己变得更聪明，但跟AI比起来根本不在一个量级。
I mean, there are some things we can do to get smarter, but, like, it's just not at all competitive.
比如我们可以做Neuralink那种东西。
Like, we can do, like, the Neuralink stuff.
没错。
Sure.
但跟AI的能力相比，那简直是小巫见大巫。
But, like, it's a it's like it's small potatoes compared to what the AI has a bit.
完全同意。
Exactly.
而且我的意思是，我们还有生物上的限制，比如我们身体本身的脆弱性。
And what I mean, we have biological limitations in terms of just the vulnerability of our own bodies.
如果AI只存在于云端，用云基础设施，比如数据中心之类的，然后靠自主机器人来执行所有行动。
If there they exist only in the cloud and they use whatever the cloud is, and they and they u these data centers or whatever, and they use autonomous robots to do all their deeds.
对。
Yeah.
对。
Yeah.
我是说，但是……
Mean, that's But
我们为什么没有警觉呢？
we're not not why are we asleep at the wheel?
这对我们来说很正常吗？
Is that just normal for us?
就像，我们无法理解如此离奇、如此超乎想象的事情，以至于我们宁愿不去讨论它？
Like, we can't comprehend something that's so bizarre and so out that we just rather not discuss it?
我们宁愿假装它没有发生？
We'd rather just pretend it's not happening?
我觉得现在的情况就是，科幻小说，无论好坏，几十年来一直在谈论这类事情。
I mean, that's what I think is going on is that science fiction, for better or for worse, has talked about this sort of thing for decades.
所以呢，大家就把这类事情当成科幻小说，不当回事。
And as a result, people dismiss this sort of thing as science fiction.
他们就觉得，好吧，那是科幻，但现实世界里不会发生的。
They're like, okay, that's sci fi, but it's not happening in the real world.
我觉得，AI安全领域的人、AI行业的人，还有像我这样搞预测的人——你可以去看看我过去几年做的预测。
And I think that people in the AI safety community and people in the AI industry and people who do forecasting like me, you can go read the forecasts I've made in past years.
我觉得那些预测还挺靠谱的。
I think they hold up reasonably well.
大家其实早就开始讨论这类事情了，但很容易就被当成是那种推测性的科幻，觉得现实里大概不会发生。
People have been talking about this sort of thing for a long time, but it's been very easy to dismiss it as like, okay, that's speculative sci fi, it's probably not gonna happen in real life.
但现在它真的在现实里发生了。
Now it's happening in real life.
你懂吗？
You know?
就是，这种事情真的挺疯狂的。
Like, this this sort of thing is, like, crazy.
你觉得如果
Do you think that
没有那种科幻作品的话，情况会有什么不同吗？
it would be any different if there wasn't that kind of sci fi?
你觉得人们会有不同的反应吗？
Do you think people would have a different reaction to it?
因为我不这么认为。
Because I don't.
我觉得，就目前可用的技术量来说，很多技术已经超出了大多数人的理解，但他们每天都在用，比如星链。
I think just given the amount of technology that's available currently that is beyond most people's understanding that they use every day, like Starlink.
比如，什么？
Like, what?
我有一个跟iPad差不多大的玩意儿，带到山里就能用高速互联网。
I have a fucking thing that's the size of an iPad I take to the mountains and I get high speed Internet.
这太疯狂了。
It's bananas.
对。
Yeah.
然后你就接受了，并且默认了。
And you just accept it and assume.
我不知道如果没有《终结者》这类电影，我们会不会有不同的反应，这些电影让这种概念在我们心里变得习以为常，或者被过度虚构化，以至于我们根本不愿意相信它可能在现实中发生。
I don't know if we would react any differently if we didn't have the Terminator in all these movies where it's kind of become normalized in our mind or or it's so fictionalized that we never wanna believe it's even possible for it to happen in real life.
还有一点要说的是，人们正在觉醒。
Another thing to say is that people are waking up.
就是说，我们其实还处在曲线的早期阶段。
Like, it's we're still sort of early in the curve.
不知道你还记不记得新冠时期的情况，就像病毒在人群中呈指数级传播一样，人们对新冠的重视程度和思考也在呈指数级增长。
I don't know if you remember how things were with COVID, but, like, just as like there was this exponential ramp up of COVID in the population, there was also this sort of exponential ramp up of like how much people were taking COVID seriously and thinking about it in the population.
嗯。
Mhmm.
我记得有那么一个月的时间，人们的态度从“别太担心，你们不应该买口罩，因为医护人员需要”变成了“我们都需要封锁，待在家里”。
And I remember like this period of like one month where they went from like, Don't worry about it so much, you shouldn't buy a mask because the healthcare workers need it, to, We all need to lock down, stay at home.
关于 Bayt 播客
Bayt 提供中文+原文双语音频和字幕，帮助你打破语言障碍，轻松听懂全球优质播客。