游戏、山羊与通用智能——对话弗雷德里克·贝斯

欢迎回到Google DeepMind播客，我是主持人汉娜·弗莱教授。

Welcome back to Google DeepMind, the podcast, with me, your host, professor Hannah Fry.

可以说，目前的生成式AI与我们科幻幻想中的技术形态还存在一定差距。

I think it's fair to say that generative AI, as we have it now, is a bit of a mismatch for our science fiction fantasies about what technology might become.

但值得记住的是，这并非唯一可能。

But it is worth remembering that this isn't the only possibility.

还存在其他形式的AI，其设计本质就是自主实体。

There are other forms of AI, forms which are, by their very design, autonomous entities.

这类AI完全属于机器人管家或虚拟管家的范畴，能在你产生需求前预知每个突发奇想。

These AI are firmly in the realm of robot butlers or a virtual concierge that anticipates your every whim before you even have it.

好吧，这两个具体例子可能还需要等待一段时间。

Then, okay, we might have to wait a little while for those two particular examples.

但我这里要讨论的是具有自主能动性的人工智能。

But I'm talking here about artificial intelligence with agency.

这种AI内置了需求目标，能够为追求自身目标而持续独立做出决策。

AI that has built in wants and objectives and the ability to independently make decision after decision in pursuit of its own goals.

这类AI被称为智能体。

This kind of AI are known as agents.

虽然智能体尚未像生成式AI那样引爆公众认知，但请相信，它们将产生重大影响。

And while agents haven't yet exploded into the public consciousness quite as much as generative AI has, trust us, they are going to be a very big deal.

而目前，电子游戏世界正是训练智能体的绝佳试验场。

And in the meantime, a very good training ground for agents is the world of video games.

想想看：

Think about it.

那些精心设计、边界明确的游戏环境，正是智能体可以自由探索、自主制定规则并学习处理自主权的理想场所。

Perfectly packaged, neatly constrained environments where the agents can run wild, work out the rules for themselves, and learn how to handle autonomy.

如果您从开始就关注我们，就会知道DeepMind在AI游戏领域有着引以为傲的传统。

If you've been following us since the beginning, then you will know that AI and games is something that DeepMind has a proud heritage in.

今天为我们做讲解的是弗雷德里克·贝斯，他是谷歌DeepMind的高级研究工程师。

And so to explain today, we are joined by Frederic Bess, who is a senior staff research engineer at Google DeepMind.

弗雷德里克拥有计算机视觉博士学位，早在2015年DeepMind团队仅有约100名研究人员时，他就是最早加入的成员之一。

Frederic has a PhD in computer vision and was one of the earliest people to join the DeepMind team back in 2015 when there were only about a 100 researchers here.

在他投身人工智能前沿领域之前，还曾在视觉效果行业工作过一段时间，研究如何将CGI技术整合到实拍素材中。

And before the decade he spent at the cutting edge of artificial intelligence, he also spent some time working in the visual effects industry on how to integrate CGI into real footage.

弗雷德里克，非常感谢你参加我们的播客节目。

Frederic, thank you very much for joining us on the podcast.

谢谢你，弗雷德里克。

Thank you, Frederic.

看到这样的履历，我明白DeepMind为什么要聘用你了。

I I can see why DeepMind hired you with a CV like that.

哦，谢谢夸奖。

Oh, I mean, thanks.

好的。

Okay.

我想我们最好先从一些定义开始。

Well, so I think it's probably best if we start off with some definitions.

当然。

Sure.

我之前简单提到过智能体的概念，但你是如何定义它们的？

So, I mean, I've talked briefly about what agents are there, but but how do you define them?

在我看来，智能体是一个非常广义的概念，我将其定义为能在环境中行动的实体。

So in my mind, an agent is a very general concept, and I define it as an entity that can act in an environment.

我认为行动力是智能体的核心特征。

I think action is what define an agent.

而环境则是能为智能体提供观察数据，并与之互动的场所。

An an environment, on the other side, is a place which can provide observations for the agent and which the agent can interact with.

因此关键要理解的是，智能体通过作用于环境会改变环境，这对于开发能在环境中（如游戏、现实世界、互联网等）完成有用任务并理解其行为后果的AI系统至关重要。

And so what's important to understand is that the agent, by acting into the environment, will change the environment, which is crucial for developing AI systems that can do useful things and understand the consequences of their actions in environments, in games, in the world, on the Internet, etcetera.

不过这听起来像是超越了单纯AI范畴的概念。

This sounds like something exists beyond just AI though.

比如飞机的自动驾驶系统，这能算作智能体吗？

Mean, like autopilot on planes, would that count as an agent?

是的。

Yes.

完全正确。

Absolutely.

或许我们可以举几个智能体的例子，因为这是个非常宽泛的概念。

So maybe we can give a few examples of agents because it's a very general concept.

人类就是智能体，因为我们能采取行动。

So humans are agents because we can act.

我们用眼睛和耳朵观察世界。

We observe the world with our eyes and our ears.

我们具有本体感觉。

We have proprioception.

我们能感知自己肢体的位置。

We know where our limbs are.

所以我们都是智能体。

So we are agents.

我们能影响世界。

We can affect the world.

机器人同样如此。

Similarly, robots as well.

但若回溯到几十年前，像自动驾驶这类智能体，其逻辑都是我们手工编码的。

But if you go back to maybe decades ago where we have we had agents such as autopilot, we handcrafted the logic of those agents.

我们当时编程让它们执行我们指定的任务。

We were programming them to do what we wanted them to do.

因此它们与我们稍后要讨论的现代智能体不同，但这些确实是智能体的实例。

So they were different from the modern agents that we can talk about later on, but these were indeed examples of agents.

那么好吧，如果那算是智能体，它和自主性或者说拥有代理权是同一回事吗？

So, alright, if that's an agent then, is that the same thing as autonomy or having agency?

这些概念是否都存在微妙的差异？

Are these all slightly subtly different?

我认为它们确实存在微妙差别。

So I think they are subtly different.

我认为智能体能够执行其被编程设定的行为。

I think an agent can act and do things that that it has been programmed to do.

这就是代理权——能够采取行动的能力。

And that's agency, being able to take actions.

而自主性则更偏向于独立完成任务的能力或属性。

Now autonomy is more the skill or the property of acting by itself to accomplish a certain task.

自主性存在不同等级。

And there are various levels of autonomy.

比如自动驾驶汽车能自主做出最佳路线决策，我们就可以说它具备某种程度的自主性。

So if we have a self driving car which will take decision to best navigate the roads, you could say that there is a certain level of autonomy.

有些智能体自主性很低，需要大量人工输入。

Some agents are much less autonomous and will require a lot more of your input.

而光谱的另一端，可能存在能主动探索世界、学习新事物的智能体，这类几乎不需要我们过多指导。

And then on the other end of the spectrum, could have an agent that goes in the world and learn new things, explore, which we we don't really have to instruct much.

所以说在智能体的自主性维度上存在一个光谱分布。

So they are a spectrum in the space of autonomy in agents.

这是否部分类似于你所设定目标类型的某种层级关系？

Is part of that, like, almost a hierarchy of the type of objectives that you're setting?

比如，我不确定，像拿起杯子又放下这样的自主动作，和决定午餐吃什么还是有些区别的。

Like, I don't know, the the autonomous action of, picking up a cup and putting it down is slightly different to deciding what's for lunch.

是的。

Yes.

我们正在训练智能体完成简单任务，因为这是起点，但拥有一个能为你烹饪完整午餐的智能体

We are training agents to do simple things because does this is a starting point, but having an agent that can cook for you your whole lunch

还能决定你想吃什么就更好了。

decide what you want as well would be nice.

是的，确实如此。

Yes, as well.

它可以是个推荐系统。

It could be a recommender system.

我们首先需要许多子步骤和子任务来妥善解决这些任务。

We'll require a lot of sub steps and sub tasks to solve those tasks properly as well first.

所以我们把这些称为高层次任务，相比短视界任务会更基础，几乎是机械动作，比如抓取物体、把物品放在桌上，而不是烹饪午餐。

So we call those high level tasks compared to short horizon tasks, which will be very basic, almost mechanical actions like grab an object, put the objects on the table compared to cook lunch.

对。

Yeah.

就像你在工厂里看到的机器人，与科幻片里的那种对比。

Like the robots you see in factories, for example, versus the type of ones that we have in science fiction.

是的。

Yes.

但这是目标吗？

But is that the sort of aim?

我们讨论的是未来会进入家庭的机器人吗？

Are we talking about robots eventually that will be in our homes?

我想是的。

I think so.

我们正在努力开发通用智能体，目标是实现AGI（人工通用智能），届时我们将拥有真正通用的人工智能系统，其通用性将媲美人类。

So what we are trying to do is develop general agents, and we want to reach a point which we call AGI, artificial general intelligence, which is a point where we will have agents and AI systems that will be truly general, as general as humans.

这些系统将能像人类一样适应新情境、进行推理，甚至在某些任务上超越人类能力。

We'll be able to adapt to new situations, reasons like we do, and even surpass our capabilities in doing tasks.

一旦AI系统达到这种能力水平，将释放人类潜能的无限可能。

Once we reach this level of competency in AI systems, this will unlock a lot of potential for what we can do as humans.

这些AI系统能加速科学进步，或完成许多人类不擅长的任务。

Those AI systems will be able to accelerate our scientific progress, for example, or do a lot of tasks which maybe we are not very good for.

比如驾驶汽车、家务协助或在线购物等场景。

For example, maybe driving or help you in your home, help you do shopping online.

我认为有些应用领域我们甚至无法预想，因为这种技术飞跃需要我们逐步探索发现。

I think there are some applications which we can't even begin to conceive because this is such a technological leap that I think we will just discover it as we go.

但我们坚信，构建能在环境中行动的通用智能体是实现AGI的关键。

But we do believe that building such general agents which can act into an environment is key to achieve AGI.

你描述的方式很有趣，这意味着部分智能体本质上将是虚拟存在的。

That's interesting then the way that you're describing it that some of these agents are going to be virtual essentially.

是的。

Yes.

而另一些则会实际嵌入物理环境中。

And then some of them will be actually physically embedded in an environment.

没错。

Yes.

我认为这都归结于观察空间与行动空间的界定。

So I think it all goes back to what's your observation space and what's your action space.

我们可以开发物理智能体，如机器人这类具身智能体，它们能在现实世界中执行任务，这将非常实用。

So we can have physical agents, embodied physical agents like robots, which will be able to do things in the real world, which will be very useful.

我们已...

We already, as you

我的机器人管家。

My robot butler.

谢谢。

Thank you.

没错。

Exactly.

但还包括那些不在现实世界运作，而是在抽象虚拟空间工作的虚拟代理。

But also virtual agents which work not in the real world, but in an abstract virtual space.

这可能仍是一个物理空间，例如我们在研究中使用的游戏，用来展示和理解这些代理能做什么。

This could be still a physical space, for example, games, which we use in our research to also demonstrate and understand what those agents can do.

但你也可以拥有一个没有任何物理甚至虚拟行动空间的代理，它却能编写程序，比如生成Python代码、为你构建软件——这虽非物理空间，但它仍是代理，因为它能采取行动、编译函数、上网查询。

But also you could have an agent that doesn't have any physical, even virtual action space, but instead it can make programs, for example, write Python code, build software for you, which is not a physical space, but it's still an agent because it can take actions, it can compile the function, it can look on the internet.

因此这仍然非常有价值。

And so this is still very valuable.

这与我们从大语言模型看到的聊天机器人有何不同？

How is this different to the chatbots that we've seen from large language models?

我是说，它们也能做很多类似的事情，对吧？

Because I mean, they can do a lot of that stuff too, right?

它们能上网搜索，能理解你的指令并做出某种决策？

They can search the internet, they can understand your instructions and make decisions sort of?

是的。

Yes.

通常你从聊天机器人得到的是聊天输出。

So what you get out of chatbots generally is a chat output.

它们生成自然语言。

They produce natural language.

现在有些聊天机器人具备能替你上网搜索的功能。

Now there are some capabilities in some chatbots which could go in and search the Internet for you.

但这个空间仍然相当受限，仅限于程序员添加到聊天机器人中的自然语言或特定功能。

But this space is still fairly constrained to just being either natural language or specific functions that the programmer adds to the chatbot to do things.

现在，这些聊天机器人与你对话时，某种程度上，你就是聊天机器人的环境。

Now, those chatbots, when they speak with you, I mean, in a way, you are the environment of a chatbot.

对吧？

Right?

聊天机器人的话语会影响你，也会影响你接下来对它的提问。

What the chatbot says will affect you and will affect what you ask the chatbot next.

但可能不像聊天机器人或智能体在某个环境中行动那样会产生重大后果——当智能体采取某些行动时，环境会发生剧烈变化。

But maybe it's not as there is not as much consequence as if the chatbot or the agent was acting in an environment which drastically changes when the agent takes some action.

所以我们讨论的是被动型智能体。

So it's it's we bought for passive type of agents.

嗯。

Mhmm.

我们可以称其为智能体，但它与现实世界中行动的智能体截然不同。

We could call it an agent, but it's very different from an agent that act in the real world.

这些模型的训练方式也是如此——它们通过大量从互联网抓取的数据进行训练，而不是在一个环境中通过尝试、失败、理解原因并再次尝试来训练。

And the way also those models are trained, they are trained with a lot of data that are scraped from the Internet, but they are not trained in an environment to try two things, fail, understand why they failed and try again.

这是这些模型用人类数据训练的某些方式之间的差异。

This is a difference between some of the way those models are trained with human data.

那么我们应该将它们视为独立的，还是可以结合两者？

So then should we be thinking of these as separate or can you combine the two?

能否将大型语言模型与有目标的智能体结合起来？

Can you combine large language models and an agent that has an objective?

可以的，实际上谷歌DeepMind有一些机器人论文，比如RT2和RTX，就在做这件事。

So you can, and actually there are some robotics paper from Google DeepMind, it's called RT2 and RTX, which are doing that.

他们在训练时，语言模型从训练所用的海量数据中获取了大量知识。

So they are training, so a language model has a lot of knowledge from the internal scale data that it's been trained on.

所以如果你愿意的话，它具备很多常识，并且理解一系列概念。

So if you if you want, it has a lot of common sense and it understands a bunch of concepts.

但它不具备激活机器人关节的能力。

But it doesn't have the ability to activate joints of a robot.

但你可以通过文本训练这个模型，使其能为你的机械臂生成关节激活指令，例如。

But you can make text this model and train it to produce joint activation for your robotic arm, for example.

这正是那篇论文中他们所实现的。

That's what they've done in that paper.

我想语言模型几乎为算法提供了某种垂直商业领域中对事物的概念性理解。

I guess almost the language model gives the algorithm some sort of, in vertical commerce, conceptual understanding of things.

是的。

Yes.

我认为这些大型模型的伟大之处在于它们拥有丰富的知识，能理解概念、词汇以及如何与之对话。

I think that's what is great about those big models is that they have such knowledge and understand concepts and words and what chat to them about.

它们能制定计划，但这一切都停留在语言层面，对吧？

They can make plans, but all of this is in language space, right?

而我们需要弥合语言与现实世界或虚拟世界行动之间的鸿沟。

And we need to bridge the gap towards acting in the real world or in the virtual world.

去实际执行行动吗？

To go off and actually in acting?

对。

Yes.

不过游戏有什么特别之处呢？

What is it about games though?

为什么从游戏开始会很有用？

Why is it useful to start with games?

我认为游戏能提供非常丰富的体验，而且游戏种类极为多样。

So I think games offer a very rich experience, and games are very varied.

以电脑游戏为例，如果你看看有多少游戏存在，会发现有成千上万款游戏。

So if you look at how many games there are on on computers, for example, you have tens of thousands of games.

每款游戏都提供不同的场景、环境和可进行的活动。

And each game offers you different scenarios, different environments, different things you could be doing in the game.

因此我们认为这是训练AI系统达到人类水平的绝佳试验场。

So we think it's a great proving ground to train AI systems to match human performance.

游戏还有其他优势。

There are other advantages in games.

它们是虚拟的，所以易于扩展。

So they are virtual, so they are easy to scale.

与受现实世界限制的物理系统相比，你可以并行实例化许多不同的游戏。

You could instantiate a lot of different games in parallel compared to physical systems, are constraints to the real world.

它们也很安全。

They are also safe.

在游戏中出错并不重要。

So if things go wrong in games, it doesn't really matter.

毕竟只是个游戏。

It's just a game.

游戏是人类与智能体互动的场所，比如在多人在线游戏中，而且游戏本身对人类来说也很有趣。

Games are a place where humans contract with agents, for example, in multiplayer games and games are, yeah, fun as well for humans to play.

我认为这是我们能收集到的绝佳经验数据来源。

And I think it's a great source of experience data that we can collect.

我想你们已经在非玩家角色中部署了智能体吧？

I guess you've already got agents already in sort of non player characters, right?

那些NPC在场景中游荡。

NPCs floating around the place.

是的，没错。

Yes, yes.

所以我认为这是个非常有趣的观点。

So I think that's a very interesting point.

从历史上看，实际上游戏开发者在游戏中创建智能体时，与我们制作的智能体类型截然不同。

Historically, and actually it's still the case when game developers create agents inside of games, it's a very different beast to the type of agents that we make.

那些智能体可以访问游戏的内部代码，游戏开发者只需编写逻辑，效果很好，它们运行速度快，能完成相当惊人的任务。

Those agents have access to the internal code of the game and the game developers just, you know, write the logic, which works great and they are fast and they can do pretty impressive things.

比如站在那里，如果有人靠近就攻击他们。

Things like stand around and then if someone approaches you attack them.

我是说，这

I mean, it's

算是吧。

sort Yeah.

我是说，你可以为游戏中的NPC智能体编写相当复杂的逻辑，但用代码描述行为的复杂度存在上限。

I mean, there is some pretty, yes, some pretty complicated logic you could put in the agents in the game, in the NPCs, but there is like a limit in how complicated you can describe behavior in terms of code.

这正是深度学习和现代智能体的用武之地，因为这些智能体内部的逻辑并非人工编写。

And that's where deep learning and modern agents come in because those are not the logic inside those agents is not handcrafted by humans.

相反，我们让智能体接触大量经验数据，它会自行学习行为模式。

Instead, we subject the agent to a lot of experience data and the agent will learn the behavior for itself.

而且是以更柔和、更细腻的方式，它能掌握非常复杂的行为，这些行为可能难以用编程语言编写出来。

And in a in a more in a softer way with more, like, nuances, and it can pick up on a very complex behavior, which maybe wouldn't be feasible to to write down in in code, in programming language.

与其说只是遵循规则，不如说它是在自行摸索如何达成目标。

Rather than just following rules, it's sort of working out how to achieve its objective itself.

没错。

Yeah.

DeepMind在游戏领域有着相当长的研究历史。

Deep Mind's got quite a long history of playing around with games.

考虑到他们通过这种方式取得的成就，这么说可能有点低估了。

Maybe that's a bit trivializes it slightly considering the things that they've achieved by doing so.

是的。

Yes.

2015年，神经网络智能体首次取得突破，出现了名为DQN（深度Q网络）的技术，该智能体被训练来玩Atari游戏。

So back in 2015, there was the first breakthrough in neural network agents called the DQN for Deep Q Network, in which the agent was trained to play Atari.

这里的突破在于，该智能体仅使用像素作为观察空间。

And the breakthrough here is that the agent was only using pixels as the observation space.

所以Atari游戏是指像《太空侵略者》、《打砖块》这类游戏吗？就是那种有挡板和弹球的八十年代风格游戏？

So Atari is like Space Invaders Breakout, the one with the the the bricks where you've got a paddle and a ball bouncing around, like the sort of games you had in the nineteen eighties?

是的。

Yes.

就是那些街机游戏。

So those those arcade arcade games.

对。

Yeah.

没错。

Exactly.

它唯一的输入就是屏幕上的像素？

And the only thing it would take as an input were the pixels on the screen?

屏幕像素，还有奖励机制概念——也就是你获得多少分数，这对智能体学习至关重要。因为这些智能体是通过强化学习训练的，该方法旨在最大化得分、最大化奖励。

So the pixels on the screen, and there was also the concept reward which is like how many points you get and that is very important for those agents to learn because what those agents are doing and how they are trained, they are trained using a method called reinforcement learning which aims at maximizing this score, maximizing the reward.

因此将深度学习与深度神经网络和强化学习相结合，诞生了DQN智能体，这是该领域的首创。

And so combining deep learning with deep neural network and reinforcement learning yielded the DQN agent, which was a first of its kind.

这对DeepMind来说是一个非常重要的里程碑。

And that was a very big milestone for DeepMind.

随后另一个重要智能体是AlphaGo，它是学习围棋的智能体，围棋是一种棋盘游戏。

Then another important agent that was built was AlphaGo, which was an agent that tries to learn to play the game of Go, which is a board game.

众所周知非常难。

Notoriously difficult.

众所周知非常困难。

Notoriously difficult.

我认为当时人们不确定AI是否能在围棋上战胜人类，但后来这个观点被证明是错误的。

I think it was uncertain whether AI could ever beat the game of Go, but then that was proven to be wrong.

是指AlphaGo击败李世石的时候吗？

When AlphaGo beat Lee Sedol?

是的，正是如此。

Yes, exactly.

所以那是个重大突破。

So that was a huge result.

沿着历史继续往下说，这并非详尽清单，因为在所有这些重要里程碑之间还发生了大量研究。

Then if you continue along the history, so this is a non exhaustive list because there is a lot of research that has been happening also in between all these important milestones.

后来出现了AlphaZero，它将下围棋的智能体提升到新高度，还能下国际象棋和将棋。

There was AlphaZero, which is taking the agents that play Go and also it could play chess and shogi to the next level.

这个智能体完全没有使用人类数据进行训练。

And this agent was trained without any human data.

AlphaGo是结合人类数据、自我对弈和强化学习训练的，而AlphaZero则完全没有使用人类数据。

So while AlphaGo was trained using a mix of human data and self play and reinforcement learning, AlphaZero was trained without any human data.

它完全从零开始通过自我对弈，就发现了最佳棋步。

And just starting from scratch with self play, it discovered the best chess moves.

这个现象非常有趣。

And that was very interesting to see.

令人惊叹的是，去除人类输入反而让它更强。

Extraordinary that taking the human input out made it better.

嗯，是的。

Well, yes.

这确实很有意思，因为你实际上可以观察到国际象棋的最优走法，非常迷人。

So, yeah, it's interesting because you could you can actually look at what are the most optimal moves for chess, which is fascinating.

嗯。

Mhmm.

我发现。

I find.

但后来它又转向了电脑游戏领域，对吧？

But then it moved into computer games again, didn't it?

是的。

Yes.

所以我认为下一篇非常重要的论文是关于Alpha Star的，这是一个玩《星际争霸2》的智能体，这是一款竞技游戏，与围棋或国际象棋的环境截然不同，对吧？

So I think one next very important paper was the Alpha Star paper, which is an agent that is playing Starcraft two, which is a game, competitive game, that's very different environments from, say, GO or chess, right?

这里的挑战之一是，在《星际争霸2》中，你只能从环境中获取部分信息。

So one of the challenge here is that in Starcraft two, you only see partial information from your environment.

有一种叫做战争迷雾的机制，让你无法看到敌人在做什么。

There is this thing called fog of war where you cannot see what your enemy is doing.

因此你必须在制定策略时充分考虑这一点。

And so you really have to take this into account during your kind of strategy.

而在围棋和国际象棋中，你可以看到整个棋盘。

While in GO, in chess, you see the whole board in front of you.

所以你能掌握游戏的完整状态。

So you have access to the whole state of the game.

此外，智能体必须实时操作，因为你知道在《星际争霸》中，当你与人类对战时，必须尽可能快地行动，没有时间花几分钟来思考。

Also, agent has to play in real time because Starcraft you know, once you play against a human at this time, you really have to play as fast as you can so that you cannot take minutes to think.

而这个《星际争霸2》的智能体能够击败非常非常优秀的人类玩家。

And the Starcraft two agent was able to beat very, very good humans at the game.

所以现在这又向前迈进了一步。

So now this is one step on even further.

就像在每一步中，你都去掉了让事情变简单的因素。

So it's it's like at every step, you sort of you remove something that made it easy.

在Atari游戏中，你的每个动作都能得分。

So Atari, you get points for every action.

国际象棋则不然。

Chess, you don't.

国际象棋中，你能看到整个棋盘。

In chess, you get to see the entire board.

《星际争霸》则不行。

In Starcraft, you don't.

《星际争霸》是竞技性的。

In Starcraft, it's competitive.

而你现在做的事不具备这种特性。

In what you're doing now, it isn't.

是的。

Yes.

那些智能体的目标是最大化得分并赢得游戏。

So those agents were trying to maximize the score and to win the game.

而我们在SIMA中的工作截然不同——我们训练智能体遵循指令。

While what we do in SIMA is very different, we train agents to follow instructions.

这个项目叫什么名字？

What's this project called?

我们的项目名为SIMA。

Our project is called SIMA.

全称是「可扩展可指导多世界智能体」。

That stands for Scalable Instructable MultiWorld Agent.

我们使用各种电子游戏和研究环境来训练智能体执行我们下达的指令。

And we used various different video games and also research environments to train an agent to follow instructions that we ask it to do.

这些指令通常很简单，五到十秒内就能完成，比如「捡起苹果」「转身」「爬上梯子」等等。

Those instructions are fairly simple that you can typically carry out in five to ten seconds, such as pick up the apple or turn around, climb up the ladder, etcetera.

因此没有计分，我们实际上将游戏用作容器，作为可以执行大量指令的围墙。

And so there is no score, and we use the games really as containers, as walls in which we can carry out a whole bunch of instructions.

由于我们使用横向语言来制定这些指令，因此可设计的指令数量非常庞大。

And the amount of instruction, the number of instructions that you can come up with is is very large due to the fact that we use lateral language to formulate those instructions.

这就带来了其他挑战。

So this poses other challenges.

我们真正希望使我们的智能体具备通用性。

We really want to make our agents to be general.

虽然围棋智能体只学习如何击败围棋游戏，但我们感兴趣的是制造一个能真正理解你说的话、理解上下文、理解环境然后执行这个指令的智能体。

So while a Go agent only learns to beat the game of Go, we are interested in making an agent that can really understand what you're saying, understand the context, understand the environment and then carry out this instruction.

因此我们追求的是通用性。

And so we are aiming for generality.

我认为通用性是我们在追求AGI时希望智能体具备的关键特性。

And I think generality is a key property that we want in our agents to pursue AGI.

这就是我们使用沙盒游戏的原因。

And that's why we use those sandbox games.

你说的沙盒游戏是什么意思？

What do you mean by sandbox games?

沙盒游戏是指玩家基本上可以在游戏提供的空间内做任何他们想做的事情。

So a sandbox game is a game where the player can basically do whatever they want within the space of what the game offers.

如果这是一款有自然景观的游戏，你可以制作、建造、烹饪、探索。

So if you did if it's a game which has a natural landscape, you can craft, you can build, you can cook, you can explore.

比如《我的世界》。

Like Minecraft.

没错。

Exactly.

就像《我的世界》。

Like Minecraft.

这就是我们所说的沙盒游戏，因为游戏本身并没有明确的目标。

And that's what we call a sandbox game because there is not really like a goal to the game.

有些沙盒游戏是带有目标的。

Some games are sandboxed with a goal.

所以它有点混合性质。

So it's a bit of a mix.

不过，没错，你只需要享受乐趣并做各种事情。

But, yeah, you just have fun and do a bunch of things.

就像在真正的沙盒里一样。

Like in an actual sandbox.

正是如此。

Exactly.

是的。

Yes.

关于你们训练所用的游戏，有《无人深空》、《拆迁》、《英灵神殿》、《模拟山羊3》等等。

And in terms of the games that you trained on, so there's No Man's Sky, Teardown, Valheim, Goat Simulator three, as well as a bunch of others.

我们会确保把这些游戏的清单放在节目说明里。

We'll we'll make sure we put a list of those in in the show notes.

你自己是游戏玩家吗？

Are you a gamer yourself?

是的。

Yes.

非常热衷。

Very much so.

我六岁生日时收到了一个Game Boy游戏机。

I I was given a Game Boy for my sixth birthday.

六岁生日。

Sixth birthday.

太棒了。

Amazing.

真走运。

Lucky thing.

是的，那太神奇了。

Yeah, it was amazing.

我记得我玩的第一款游戏是俄罗斯方块，当时能在车里玩游戏让我感到非常惊奇。

I think my first game was Tetris and yeah, that was I was amazed by being able to play those games in car.

从此公路旅行再也不一样了。

Car trips were never the same.

确实如此。

It's true.

那成年后你还玩俄罗斯方块以外的游戏吗？

And as an adult, did you go beyond Tetris?

当然玩过。

Yes, of course.

我记得家里装了DSL宽带后，基本上就能随时上网了。

I remember when we at home got DSL, which would give you basically constant Internet access.

我特别开心，因为终于可以不用担心计时器，尽情玩网络游戏了。

I was very happy because I could finally play online games without being worried about the counter, know, how much time do I spend on the Internet.

所以，没错，网游、竞技游戏、沙盒游戏，各种游戏。

So, yeah, online games, competitive games, sandbox games, games.

所有这些，我都很喜欢

All of these, yeah, I I I do quite like

。

it.

听起来你基本上拥有了完美的游戏体验

I mean, so you've got basically the perfect experience then

to to

to to

身处这个空间。

be in this space.

看起来是这样。

Looks like it.

你现在正在开发的新项目，这些沙盒游戏，确实是个很大的转变。

The new projects that you're working on now, these sandbox games, I mean, they are quite a big departure.

在一个没有明确定义成功标准的环境中，让智能体完成任务有多困难？

How hard is it to get an agent to achieve things in an environment where you don't have that really clearly defined metric of success?

是的。

Yeah.

所以这与之前构建智能体训练的方式截然不同。

So it is it is a very different style of building an agent training from before.

我们深受先前工作的启发，特别是交互式智能体项目。

There is some prior work which we were very inspired by, which was the interactive agents project.

DeepMind博客上有篇文章，他们用了一个玩具屋环境，里面有很多物品，目标是让智能体在屋里执行指令。

There is a blog post on the DeepMind blog post where they used a playhouse environment with a bunch of objects and the goal was to get the agents to carry out instructions in that house.

这就像个模拟环境。

It's like a simulated environment.

没错，是个模拟环境。

It's a simulated environment, yes.

就像个儿童玩具屋，里面有彩色物品，目标是把茶壶放到床上，而且也没有奖励机制。

It's like a children's house with colorful objects and the goal was to take the teapots and put it on the beds and there was no reward as well.

所以我们想把这个推向更高层次。

So we wanted to take this to the next level.

而且我们还想使用非自研的游戏，那些所有人都能玩的游戏。

And we also wanted to use games that are not games that we make ourselves, that are available for everyone to play.

所以基本上这就是你们玩的游戏。

So basically it's the games that you guys play.

是这样吗？

Is that what it is?

其中一些确实是。

Some of them are indeed.

我是说，我只想花更多时间玩这个我一直在玩的游戏。

I mean, I just want more time to spend on this game that I've been doing.

所以我打算开发一个代理来做这件事。

So I'm gonna build an agent to do it.

当这种情况发生时

Is this interesting when

有趣吗？

it's happening?

是的。

Yeah.

你可以这么做。

You could.

对。

Yeah.

于是我们与游戏开发商和工作室建立了合作关系，以便能在研究中使用他们的游戏，我们选择了特定类型的游戏。

And so, yeah, we got into partnership with game developers and game studios to be able to use their games in our research, and we picked games which are of a certain genre.

我们想要3D游戏，没错。

So three we wanted three d games Yeah.

因为我们对这种具身化的3D环境很感兴趣。

Because we were interested in this embodied three d setup.

我们希望能够用语言定义自己的目标。

We wanted to be able to define our own objective with language.

以《英灵神殿》为例，这是一款维京生存模拟游戏，你可以为智能体设计大量任务。

So if you go into Valheim, for example, which is a Viking survival simulation, you can make up a lot of tasks for the agents.

你可以下达指令，比如收集情绪、建造房屋、制作夹克等等。

You can say, go get some moods, build a house, craft a jacket, etcetera.

那你到底要怎么训练一个智能体，当你

Then how on earth do you train an agent when you

没有数据时？我们的做法是收集人类玩家的游戏数据。

don't have So what we did is that we collected data from humans playing the game.

然后运用模仿学习技术，教导智能体模拟并预测人类的下一步行为。

And then we used a technique called limitation learning, which teaches the agent to just mimic and predict what a human would do next.

基于所有观察数据和你此前的操作，人类会怎么做？

Given all of these observations, given what you've done before, what would a human do?

这就是我们训练智能体行为模式的方法。

And that's how we train the behavior of the agent.

所以这里不存在奖励机制。

So there is no reward.

我们只是模仿人类行为。

We just imitate humans.

那么这是概率驱动的吗？

Is it probabilistic then?

是不是类似这种情况：人类在这种情境下会这样行动，所以我们会以某种概率采取相应行动？

Is it sort of like in this sort of situation, humans acted in this way, so we'll act in that way with this probability?

没错，正是如此。

Yes, exactly.

因此智能体的所有行为都是概率性的。

So all the actions of the agent are probabilistic.

在智能体行动前，需要从概率分布中采样并选择动作。

So before the agent acts, there is a probability distribution which needs to sample and pick an action from.

通常这是最可能采取的行动，但有时你可能会做出不太可能的举动，从而导致意想不到的行为。

Often it's the most likely action, but sometimes you can take an unlikely action which can lead to unforeseen behavior.

有时这是种好行为。

Sometimes it's a good behavior.

有时

Sometimes

并不是。

it's not.

继续，给我举个例子。

Go on, give me an example.

嗯，比如说我们有《模拟山羊》这样的游戏。

Well, so for example, we have games such as Goat Simulator.

告诉我们《模拟山羊》的游戏目标是什么。

Tell us the objective of Goat Simulator.

《模拟山羊》的游戏目标基本上就是作为一只山羊在世界上制造混乱。

So the objective of Goat Simulator is to basically cause mayhem as a goat in the world.

你可以开车，可以跳到物体上，作为一只山羊。

And you can drive cars, you can jump on objects As a goat.

作为一只山羊。

As a goat.

对。

Yeah.

你你你是一只山羊。

You you you are a goat.

你还能拥有翅膀飞翔。

You can have wings and you can fly as well.

但这是个完美的沙盒游戏，因为你可以做许多不同的事情。

But it's a perfect sandbox game because you can do so many different things.

在《模拟山羊》中，有一个按键会让山羊变得软趴趴的，像布娃娃一样瘫在地上。

And so in Goat Simulator, there is a key which causes the goats to become all floppy and ragdoll on the floor.

我不太确定作为玩家什么时候会用到这个功能，可能是用来装死吧。

I'm not too sure what's the when you would actually, as a player, use this maybe to pretend you're out.

装死。

Play dead.

没错。

Exactly.

但有时我们测试AI时，它们就会直接瘫成布娃娃，这确实给团队测试时带来了不少欢乐时刻。

But sometimes when we playtest the agents, just ragdolls on the floor, which, yeah, brings some funny moments in the in the team when we do the playtest.

本来期待它冲进人群，结果直接瘫成布娃娃了。

Just expecting it to, like, barge into a crowd of people and it's just ragdolls.

对。

Yes.

是的。

Yes.

这种情况确实会发生。

That that can happen.

我能看看效果吗？

Can I see what it looks like?

我这就给你演示。

I'll let I'll show you.

来吧。

So Go on.