#258 – 扬·勒丘恩：智能的暗物质与自监督学习

第一个挑战是让机器学会表征世界，并提出自监督学习。

The first one is getting machines to learn to represent the world and proposing self supervised learning.

第二个挑战是让机器以与基于梯度的学习本质上兼容的方式进行推理，因为这正是深度学习的核心所在。

The second is getting machines to reason in ways that are compatible with essentially gradient based learning, because this is what deep learning is all about, really.

第三个挑战是我们目前完全不知道如何解决，至少我不知道如何解决，那就是：我们能否让机器学会对行动计划进行分层表征？

And the third one is something we have no idea how to solve, at least I have no idea how to solve, is can we get machines to learn hierarchical representations of action plans?

我们知道如何训练它们学习感知的分层表征，比如通过卷积网络、变换器等类似技术。

We know how to train them to learn hierarchical representations of perception, you know, with convolutional nets and things like that and transformers.

但是行动计划呢？

But what about action plans?

我们能否让它们自发地学习到良好的分层动作表征？

Can we get them to spontaneously learn good hierarchical representations of actions?

也是基于梯度的。

Also gradient based.

是的。

Yeah.

所有这些都需要某种程度上的可微分性，以便应用基于梯度的学习，而这正是深度学习的核心。

All of that, you know, needs to be somewhat differentiable that you can apply sort of gradient based learning, which is really what deep learning is about.

所以这是背景知识，一种能够以可微分方式推理的能力，这种能力与背景知识有某种深层关联，或建立在背景知识之上，然后基于这些背景知识，能够制定出层次化的计划，对吧。

So it's background knowledge, ability to reason in a way that's differentiable that is somehow connected, deeply integrated with that background knowledge or builds on top of that background knowledge, and then given that background knowledge, be able to make hierarchical plans Right.

在现实世界中。

In the world.

如果你看看经典的最优控制理论，其中有一种叫做模型预测控制的方法。

So if if you take classical optimal control, there's something in classical optimal control called model predictive control.

它早在20世纪60年代初就已经存在了。

And it's, you know, it's been around since the the early sixties.

NASA用它来计算火箭的轨迹。

NASA uses that to compute trajectories of rockets.

其基本思想是，你有一个相当准确的预测模型，比如火箭，或者你打算控制的任何系统，这个模型能够根据系统在时间t的状态以及你对系统采取的行动——比如火箭的推力和所有可能的控制输入——预测出系统在时间t+Δt的状态。

And the basic idea is that you have a pretty a predictive model of the rocket, let's say, or whatever system you're you intend to control, which, given the state of the system at time t and given an action that you're taking on the system, so for a rocket to be thrust and, you know, all the controls you can have, it gives you the state of the system at time t plus delta t.

对吧？

Right?

所以基本上就是一个微分方程，类似这样的东西。

So basically a differential equation, something like that.

如果你有这个模型，并且这个模型是以某种神经网络或一组可进行反向传播的公式形式存在，你就可以实现所谓的模型预测控制或基于梯度的模型预测控制。

And if you have this model, and you have this model in the form of some sort of neural net or some sort of set of formula that you can backpropagate gradient through, you can do what's called model predictive control or gradient based model predictive control.

因此，你可以将这个模型在时间上展开，输入一个假设的动作序列，然后有一个目标函数来衡量在轨迹结束时系统是否成功达到了你期望的目标。

So you can unroll that model in time, you feed it a hypothesized sequence of actions, and then you have some objective function that measures how well at the end of the trajectory the system has succeeded or matched what you want it to do.

你知道，机器人有没有造成伤害？

You know, is it a robot harm?

你有没有抓到你想抓的物体？

Have you grasped the object you want to grasp?

如果是火箭，你是否在空间站附近的正确位置？

If it's a rocket, you know, you at the right place near the space station?

诸如此类的事情。

Things like that.

通过时间反向传播——同样，这在20世纪60年代就由最优控制理论家们发明了——你可以算出最优的动作序列，从而让系统达到最佳的最终状态。

And by backpropagation through time, and again, this was invented in the nineteen sixties by optimal control theorists, you can figure out what is the optimal sequence of actions that will, you know, get my system to the the best final state.

所以这是一种推理形式。

So that's a form of reasoning.

这本质上是规划，而机器人领域的许多规划系统实际上都基于此。

It's basically planning, and a lot of planning systems in robotics are actually based on this.

你可以将这视为一种推理形式。

And and you can think of this as a form of reasoning.

所以，以青少年开车为例，你对汽车的动力学模型有相当好的理解。

So, you know, to take the example of the teenager driving a car again, you have a pretty good dynamical model of the car.

它不需要非常精确。

It doesn't need to be very accurate.

但你知道，如果你向右打方向盘，而旁边是悬崖，你就会冲下悬崖。

But you know, again, that if you turn the wheel to the right and there is a cliff, you're gonna run off the cliff.

对吧？

Right?

你不需要一个非常精确的模型就能预测到这一点。

You don't need to have a very accurate model to predict that.

你可以在脑海中运行这个过程，并因此决定不去做，因为你能提前预测到结果会很糟糕。

And you can run this in your mind and decide not to do it for that reason, because you can predict in advance that the result is going to be bad.

因此，你可以想象不同的场景，然后选择最有利的场景并采取第一步，再重复规划的过程。

So you can imagine different scenarios and then employ or take the first step in the scenario that is most favorable and then repeat the process of planning.

这被称为滚动时域模型预测控制。

That's called receding horizon model predictive control.

所以，所有这些概念都有几十年的历史了。

So all those things have names going back decades.

在经典最优控制中，世界的模型通常不是通过学习得到的。

So in classical optimal control, the model of the world is not generally learned.

有时你需要识别一些参数，这被称为系统辨识。

Sometimes a few parameters you have to identify, that's called systems identification.

但一般来说，模型大多是确定性的，并且主要由人工构建。

But generally, the model is mostly deterministic and mostly built by hand.

因此，人工智能的重大问题，我认为是未来十年人工智能的主要挑战：我们如何让机器运行能够处理不确定性并应对现实世界复杂性的世界预测模型？

So the big question of AI, I think the big challenge of AI for the next decade, is how do we get machines to run predictive models of the world that deal with uncertainty and deal with the real world in all this complexity?

这不仅仅是火箭的轨迹，你可以将它简化为一些原理。

So it's not just the trajectory of a rocket, which you can reduce to principles.

甚至也不仅仅是机器人手臂的轨迹，你同样可以通过精细的数学建模来描述它。

It's not even just the trajectory of a robot arm, which again you can model by careful mathematics.

但它涵盖了世界上所有其他事物，我们所观察到的一切。

But it's everything else, everything we observe in the world.

人类、行为、涉及集体现象的物理系统，比如水、树、树上的枝杈，或者那些复杂的事物——人类能轻松地为它们建立抽象表征和预测模型，但我们仍不知道如何让机器做到这一点。

People, behavior, physical systems that involve collective phenomena like water or or, you know, trees and, you know, branches in a in a tree or something or or, like, complex things that, you know, humans have no trouble developing abstract representations and predictive model for, but we still don't know how to do with machines.

在这三个层面中，你在哪里融入了这个世界的博弈论特性？你的行动不仅对世界的动态环境做出反应，还会对其产生影响。

Where do you put in in these three, maybe in the in the planning stages, the the game theoretic nature of this world where your actions not only respond to the dynamic nature of the world, the environment, but also affect it.

所以，如果有其他人类参与，这是第四个层面，还是在你的行动层次化表征中以某种方式整合进去了？

So if there's other humans involved, is this is this point number four, or is this somehow integrated into the hierarchical representation of action in your view?

我认为它是被整合的。

I think it's integrated.

只是现在，你对世界的模型必须应对……这使得问题变得更加复杂。

It's just it's just that now your model of the world has to deal with you know, it just makes it more complicated.

对吧？

Right?

人类复杂且难以预测，这使得你对世界的模型变得更加复杂，复杂得多。

The fact that humans are complicated and not easily predictable, that makes your model of the world much more complicated that much more complicated.

嗯，下棋就是一个类比。

Well, there's a chess I mean, I suppose chess is an analogy.

蒙特卡洛树搜索。

So Monte Carlo tree search.

我是走一步，你走一步。

I mean, there's a I go, you go.

我是走一步，你走一步。

I go, you go.

卡帕西最近在麻省理工学院做了一场关于汽车门的演讲。

Like, Kapathi recently gave a talk at MIT about car doors.

我觉得里面也有些机器学习，但主要是关于汽车门。

I think there's some machine learning too, but mostly car doors.

汽车本身具有动态性，比如有人在开门时会进行确认。

And there's a dynamic nature to the car, like the person opening the door checking.

我的意思是，他谈论的并不是这个。

I mean, he wasn't talking about that.

他谈的是感知问题，即什么定义了车门，这是一个宏大的哲学问题。

He was talking about the perception problem of what the the ontology of what defines a car door, this big philosophical question.

但我觉得这很有趣，因为很明显，开车门的人是想下车，比如在纽约，你减速会传递某种信号，加速也会传递某种信号，这就像一种默契的互动。

But to me, was interesting because, like, it's obvious that the person opening the car doors, they're trying to get out, like here in New York, trying to get out of the car, you slowing down is going to signal something, you speeding up is gonna signal something, and that's a dance.

这就像一场不同步的象棋对局。

It's a asynchronous chess game.

我不确定。

I don't know.

所以感觉这不仅仅是，我的意思是，也许你可以把所有这些微小互动整合进一个庞大的模型中。

So it feels like it's not just I mean, I guess you can integrate all of that into one giant model, like the entirety of the the the these little interactions.

因为这并没有象棋那么复杂。

Because it's not as complicated as chess.

这就像一场小小的舞蹈。

It's just like a little dance.

我们一起跳一段小舞，然后就弄明白了。

We do like a little dance together, and then we figure it out.

在某些方面，这比国际象棋复杂得多，因为它是连续的，并且以一种持续的方式充满不确定性。

Well, in some ways, it's way more complicated than chess because because it's continuous, it's uncertain in a continuous manner.

感觉并没有更复杂。

It doesn't feel more complicated.

但感觉不更复杂，是因为这是我们进化出来解决的问题。

But it doesn't feel more complicated because that's what we're we've evolved to solve.

这是我们进化出来解决的这类问题。

This is the kind of problem we've evolved to solve.

所以我们擅长这个，因为大自然让我们擅长应对它。

And so we're good at it because, you know, nature has made us good at it.

大自然并没有让我们擅长下国际象棋。

Nature has not made us good at chess.

我们在国际象棋上完全不行。

We completely suck at chess.

是的。

Yeah.

事实上，我们设计它作为游戏，正是因为它具有挑战性。

In fact, that's why we designed it as a as a game, is to be challenging.

嗯。

Mhmm.

最近国际象棋和围棋的进展让我们意识到，人类在这些方面真的很差，差得离谱。

And if there is something that, you know, recent progress in chess and Go has made us realize is that humans are really terrible at those things, like really bad.

你知道吗？

You know?

在AlphaGo出现之前，有一个说法，当时最顶尖的围棋选手认为，理想中的玩家——他们称之为‘神’——比他们多出两三颗子。

There was a story, right, before AlphaGo that, you know, the best Go player thought there were maybe two or three stones behind, you know, an ideal player that they would call God.

嗯。

Mhmm.

事实上，并不是。

In fact, no.

落后九到十个子。

There are, nine or 10 stones behind.

我的意思是，我们就是很菜。

I mean, we're just bad.

是的。

Yeah.

所以我们不擅长，这是因为我们的工作记忆有限。

So we're not good at and it's because we have limited working memory.

我们不太擅长那种树状探索，而计算机在这方面比我们强得多。

We we you know, we're not very good at, like, doing this tree exploration that, you know, computers are much better at doing than we are.

但我们更擅长学习世界的可微分模型。

But we are much better at learning differentiable models of the world.

我的意思是，我说可微分，其实也不是指我们真的会反向传播，而是说我们的大脑具有一些估算某种梯度的机制。

I mean, I say differentiable in a kind of, you know, I should say, not differentiable in the sense that, you know, we run back far through it, but in the sense that our brain has some mechanism for estimating gradients of of some kind.

是的。

Yeah.

而这正是让我们高效的原因。

And that's what, you know, makes us efficient.

所以，如果你有一个智能体，它包含一个世界模型——在人脑中，这基本上就是你大脑的前半部分——以及一个目标函数，在人类身上，这个目标函数由两部分组成。

So if you have an agent that consists of a a model of the world, which, you know, in the human brain is basically the entire front half of your brain, An objective function, which in humans is a combination of two things.

其中一个是你的内在动机模块，位于基底神经节，也就是你大脑的底部。

There is your sort of intrinsic motivation module, which is on the basal ganglia, know, on the base of your brain.

这个模块负责衡量疼痛、饥饿以及类似的感觉和情绪，也就是即时的感受。

That's the thing that measures pain and hunger and things like that, like immediate feelings and emotions.

然后，还有一个模块，类似于强化学习中人们所说的‘评论家’，它能够预测某种情境的未来结果。

And then there is, you know, the equivalent of what people in reinforcement learning call a critic, which is a sort of module that predicts ahead what the outcome of a situation will be.

因此，它不是一个成本函数，也不是一个目标函数，而是一个经过训练的预测器，用于预测最终目标函数的结果。

And so it's not a cost function, but it's sort of not an objective function, but it's sort of a trained predictor of the ultimate objective function.

而这个模块也是可微分的。

And that also is differentiable.

如果所有这些——你的代价函数、你的评论家、你的世界模型——都是可微分的，那么你就可以使用基于梯度的方法来进行规划、推理和学习，也就是我们希望智能体能够做到的所有事情。

And so if all of this is differentiable, your cost function, your critic, your world model, then you can use gradient based type methods to do planning, to do reasoning, to do learning, you know, to do all the things that we'd like an intelligent agent

去做。

to do.

基于梯度的学习，你的直觉是什么？

And gradient based learning, like, what's your intuition?

这可能是解决智能的核心，因此在你看来，你并不需要基于逻辑的推理。

That's probably at the core of what can solve intelligence, so you don't need, like, logic based reasoning in your view.

我不知道如何让基于逻辑的推理与高效学习兼容。

I don't know how to make logic based reasoning compatible with efficient learning.

是的。

Yeah.

好吧。

And okay.

我的意思是，这里有一个很大的问题，或许是一个哲学问题。

I mean, there is a big question, perhaps a philosophical question.

我的意思是，这并不那么哲学，但我们能问的是，我们从工程和计算机科学中知道的所有学习算法都是通过优化某个目标函数来实现的。

I mean, it's not that philosophical, but that we can ask is that all the learning algorithms we know from engineering and computer science proceed by optimizing some objective function.

是的。

Yeah.

对吧？

Right?

所以我们可以问一个问题：大脑的学习是否在最小化某个目标函数？

So one question we may ask does learning of the brain minimize an objective function?

它可能是多个目标函数的组合，但本质上仍然是一个目标函数。

It could be a composite of multiple objective functions, but it's still an objective function.

第二，如果它确实优化了某个目标函数，那么它是通过某种梯度估计来实现的吗？

Second, if it does optimize an objective function, does it do it by some sort of gradient estimation?

它不需要是反向传播，但必须是一种高效估算梯度的方式，其复杂度与实际执行推理的复杂度相当。

It doesn't need to be a backprop, but some way of estimating the gradient in an efficient manner whose complexity is on the same order of magnitude as actually running the inference.

因为你无法承担通过扰动大脑中的权重来观察其效果这样的操作，也无法通过扰动来估算梯度。

Because you can't afford to do things like perturbing a weight in your brain to figure out what the effect is, and then sort of you can't do sort of estimating gradient by perturbation.

在我看来，大脑使用某种零阶黑箱无梯度优化方式似乎非常不可能，因为这种方式比梯度优化效率低得多。

It's to me, it seems very implausible that the brain uses some sort of, you know, zeroth order black box gradient free optimization because it's so much less efficient than gradient optimization.

所以大脑必须有一种估算梯度的方法。

So it has to have a way of estimating gradient.

有没有可能，某种基于逻辑的推理会在局部出现，就像你所说的，如果大脑有目标函数，也许它就是一种创建目标函数的机制。

Is it possible that some kind of logic based reasoning emerges in pockets as a useful like you said, if the brain has an objective function, maybe it's a mechanism for creating objective functions.

它是一种创建知识库的机制，例如，这些知识库之后可以被查询。

It's it's it's a mechanism for creating knowledge bases, for example, that can then be queried.

也许它是一种以梯度为基础的方式学习到的高效知识表示，或者类似的东西。

Like, maybe it's like an efficient representation of knowledge that's learned in a gradient based way or something like that.

我认为存在许多不同类型的智能。

Well, so I think there is a lot of different types of intelligence.

首先，我认为我们所理解的那种逻辑推理——可能源自上世纪七八十年代的经典人工智能——人类其实很少使用，而且也不太擅长。

So first of all, I think the type of logical reasoning that we think about, that we are, you know, maybe stemming from, you know, sort of classical AI of the nineteen seventies and eighties, I think humans use that relatively rarely and are not particularly good at it.

但我们却根据彼此解决这些罕见问题的能力来评判对方。

But we judge each other based on our ability to solve those rare problems.

这叫智商测试。

It's called an IQ test.

我觉得是的。

I think so.

比如，我下棋并不擅长。

Like, I'm I'm not very good at chess.

是的。

Yes.

我一直在评判你，因为实际上我们。

I'm I'm judging you this whole time because well, we we actually

以你的背景，我相信你下棋很厉害。

With with your with your, you know, heritage, I'm sure you're good at chess.

不。

No.

刻板印象。

Stereotypes.

并非所有刻板印象都是正确的。

Not all stereotypes are true.

我下棋很差。

Well, I'm terrible at chess.

所以，你知道，但我认为我拥有的另一种智力是这种能力——通过推理，当然还有数据，来构建对世界的模型。

So, you know but I think perhaps another type of intelligence that I have is this, you know, ability of sort of building models to the world from, you know, reasoning, obvious obviously, but also also data.

而这些模型通常更偏向于类比性质。

And those those models generally are more kind of analogical.

对吧？

Right?

所以这是一种通过模拟和类比进行的推理，你用一个模型去应用到新情境中。

So it's it's it's reasoning by simulation and by analogy, where you use one model to apply to a new situation.

即使你从未遇到过那种情况，你也能将它与你之前经历过的某种情境联系起来。

Even though you've never seen that situation, you can sort of connect it to a situation you've encountered before.

而你的推理更类似于某种内在的模拟。

And and your reasoning is more, you know, akin to some sort of internal simulation.

所以当你在建造，比如一个木盒子时，你其实是在模拟正在发生的事情。

So you you are kinda simulating what's happening when you're building, I don't know, a box out of wood or something.

对吧？

Right?

你会提前想象，如果以这种方式切割木材，结果会是什么样？

You kind of imagine in in advance, like, what would be the result of, you know, cutting the wood in this particular way?

你会用螺丝、钉子，还是别的什么？

Are you gonna use, you know, screws or nails or whatever?

当你与人互动时，你也会对这个人形成一个模型，并带着这个模型去与对方互动，从而告诉对方你觉得对他们有用的信息。

When you are interacting with someone, you also have a model of that person and and sort of interact with that person, you know, having this model in mind to kind of tell the person what what you think is useful to them.

所以我认为，构建世界模型的能力，本质上就是智能的核心。

So I think this this ability to construct models of the world is basically the essence the essence of intelligence.

当然，运用这种能力来规划实现特定目标的行为，也是必不可少的。

And the ability to use it then to plan actions that will fulfill a particular criterion, of course, is is is necessary as well.

所以我会像之前一样，问你一系列不可能的问题。

So I'm gonna ask you a series of impossible questions as we keep ask as I've been doing.

所以，如果这是智能的暗物质基础——即构建背景模型的能力，你认为需要多少知识呢？

So so if that's the fundamentals of dark matter of intelligence, this ability to form a background model, what's your intuition about how much knowledge is required?

你知道，我觉得暗物质就像宇宙组成中可以量化的一个百分比，有多少是暗物质，有多少是暗能量。

You know you know, I think dark matter, you you could put a percentage of on it of of the composition of the universe and how much of it is dark matter, how much of it is dark energy.

你认为要成为一只家猫，需要多少信息呢？

How much information do you think is required to to be a house cat?

所以当你看到一个盒子时，你要能钻进去；当你看到一个人时，要能计算出最恶劣的行动。

So you have to be able to when you see a box, go in it, when you see a human compute the most evil action.

如果有什么东西靠近边缘，你就把它推下去。

If there's a thing that's near an edge, you knock it off.

所有这些，再加上你提到的额外内容——即对自己身体和世界的物理规律有极强的自我意识。

All of that, plus the extra stuff you mentioned, which is a great self awareness of the physics of your of your own body and the and the world.

你认为要解决这些问题，需要获取多少知识呢？

How much knowledge is acquired, do you think, to solve it?

我甚至不知道该如何衡量这个问题的答案。

I don't even know how to measure an answer to that question.

我不确定怎么衡量，但不管是什么，它大约只需要8亿个神经元，抱歉，是8亿个。

I'm not sure how to measure it, but whatever it is, it fits in about about 800,000 neurons 800,000,000 neurons, sorry.

这种表示方式吗？

The representation does?

所有东西。

Everything.

所有知识。

All knowledge.

所有东西。

Everything.

对吧？

Right?

不到十亿个。

It was less than a billion.

狗有20亿个，但猫少于10亿个。

A dog is 2,000,000,000, but a cat is less than 1,000,000,000.

将这个数字乘以一千，你就得到了突触的数量。

And so multiply that by a thousand and you get the number of synapses.

我认为几乎所有东西都是通过一种自监督学习方式学会的。

And I think almost all of it is learned through a sort of self supervised learning.

尽管有一小部分是通过强化学习学会的，而经典的监督学习则更少，甚至在生物世界中，监督学习究竟是如何运作的都不清楚。

Although, think a tiny sliver is learned through reinforcement learning, and certainly very little you know, classical supervised learning, although it's not even clear how supervised learning actually works in the in the biological world.

所以我认为几乎所有东西都是自监督学习。

So I think almost all of it is self supervised learning.

但它是由猫或人类大脑底层固有的目标函数所驱动的，这些函数决定了它们的行为。

But it's driven by the the sort of ingrained objective functions that a cat or a human have at the base of their brain, which kinda drives their their behavior.

所以，大自然告诉我们：你饿了。

So, you know, nature tells us you're hungry.

但它并没有告诉我们如何进食。

It doesn't tell us how to feed feed ourselves.

这需要我们大脑的其他部分去自行摸索。

That's that's something that the rest of our brain has to figure out.

好吧？

Alright?

这很有趣，因为可能还存在更深层的底层目标函数。

Well, it's interesting because there might be more, like, deeper objective functions underlying the whole thing.

所以饥饿可能只是某种神经生物学层面的东西。

So hung hunger may be some kind of now you go to, like, neurobiology.

它可能只是大脑在试图维持稳态。

It might be just the brain trying to maintain homeostasis.

因此，饥饿只是大脑对当前状态不满时，人类能够感知到的众多症状之一。

So hunger is just one of the human perceivable symptoms of the brain being unhappy with the way things are currently.

对。

Right.

因为这可能只是核心处一个非常原始的目标函数。

Because it could be just, like, one really dumb objective function at the core.

但这就是行为被驱动的方式。

But that's how that's how behavior is is is driven.

事实上，基底神经节驱使我们做出与猩猩或猫截然不同的行为，这正是人类本性、猩猩本性和猫本性之间的区别所在。

The the fact that, you know, the or basal ganglia drive us to do things that are that are different from, say, an orangutan or certainly a cat is what makes, you know, human nature versus orangutan nature versus cat nature.

例如，我们的基底神经节驱使我们寻求与他人相处。

So for example, our basal ganglia drives us to seek the company of other humans.

这是因为自然演化发现，我们必须成为社会性动物，物种才能生存，这一点在许多灵长类动物中都是成立的。

And that's because nature has figured out that we need to be social animals for our species to survive, and it's true of many primates.

但这对猩猩来说并不成立。

It's not true of orangutans.

猩猩是独居动物。

Orangutans are solitary animals.

它们不会主动寻求他人的陪伴。

They don't seek the company of others.

事实上，它们会避开他人。

In fact, they avoid them.

事实上，当别人靠得太近时，它们会尖叫，因为它们有领地意识。

In fact, scream at them when they come too close because they're territorial.

是的。

Mhmm.

因为对于它们的生存而言，进化已经找到了最佳方式。

Because for for their survival, you know, evolution is figured out.

这才是最好的方式。

That's the best thing.

我的意思是，它们偶尔也会社交，比如为了繁殖之类的事情。

I mean, they're occasionally social, of course, for, you know, reproduction and stuff like that.

但它们大部分时间都是独居的。

But but but they're mostly solitary.

所以，所有这些行为都不属于智力的范畴。

So so all of those behaviors are not part of intelligence.

人们常说，你永远不可能创造出智能机器，因为人类的智力是社会性的。

You know, people say, oh, you're never gonna have intelligent machines because, you know, human intelligence is social.

但如果你看看猩猩，看看章鱼，

But then you look at orangutans, you look at octopus.

章鱼从来不知道自己的父母。

Octopus never know their parents.

嗯。

Mhmm.

它们几乎不与其他个体互动，却能在不到一年、甚至半年内变得非常聪明。

They barely interact with any other, and they and they get to be really smart in less than less than a year, in like half a year.

你知道，一年内它们就成年了。

You know, in a year, they're adults.

两年内它们就死了。

In two years, they're dead.

因此，我们人类认为某些事物与智力密切相关，比如社交互动、语言，但我觉得我们过分强调了语言作为智力载体的重要性，因为我们总觉得自己的推理与语言密不可分。

So there are things that we think, as humans, are intimately linked with intelligence, like social interaction, like language, we think I think we give way too much importance to language as a substrate of intelligence as humans because we think our reasoning is so linked with language.

所以，要解决家猫的智力问题，你觉得在一座荒岛上就能做到吗？

So for for to solve the house cat intelligence problem, you think you could do it on a desert island?

你可以有

You could have

差不多就是这样。

Pretty much.

你可以让一只猫坐在那里观察海浪，然后自己领悟出很多东西。

You could just have a cat sitting there looking at the ways, at the ocean ways, and figure a lot of it out.

它需要具备某种恰当的驱动力，你知道，让它去做那些事并学习合适的东西。

It needs to have sort of, you know, the right set of drives to kinda, you know, get it to do the thing and learn the appropriate things.

对吧？

Right?

但是，比如说，你知道，人类婴儿天生就有学习站立和行走的动力。

But, like, for example, you know, baby humans are are driven to learn to stand up and walk.

好的。

Okay.

你知道，这并不是说这种欲望是与生俱来的。

You know, it's not that's kinda this desire is hardwired.

具体如何操作则不是这样。

How to do it precisely is not.

这是后天学会的。

That's learned.

嗯。

Mhmm.

但这种想要

But the desire to

走路？

To walk?

四处活动和站起来的欲望。

Move around and stand up.

嗯。

Mhmm.

这种欲望某种程度上是

That's sort of

天生固有的。

hardwired.

确实是

It is

把这种东西固化下来非常简单。

very simple to hardwire this kind of stuff.

什么？哦，比如这种欲望？嗯，这很有趣。

What oh, like the desire to well, that's interesting.

你天生就想要走路。

You're hardwired to wanna walk.

这背后一定有更深层的需要促使人走路。

That's not there's gotta be a deeper need for walking.

我认为，是社会强加了这种观念，认为你必须走路，所有其他两足生物都如此。

I I think it was probably socially imposed by society that you need to walk, all the other bipedal

比如，很多简单的动物，你知道，它们一出生就。

Now, like, a lot of simple animals that, you know They start.

很可能根本不需要观察同类，就会走路。

Probably walk without ever watching any other members of the species.

这似乎是一件可怕的事，因为刚开始学双足行走时你做得并不好。

It seems like a scary thing to have to do because you suck at bipedal walking at first.

爬行看起来安全多了，更舒服，你干嘛这么着急？

It seems crawling is much safer, much more like, why are you in a hurry?

因为有一种驱动力促使你去做这件事，你知道的，这属于人类发展的一部分。

Well, because because you have this thing that drives you to do it, you know, which is sort of part of the sort of human development.

实际上，人们是否理解这一点呢？

Is that understood actually what

并不完全理解。

Not entirely.

是的。

No.

为什么要站起来用两条腿走路呢？

What is what's the reason to get on two feet?

这真的很难。

It's really hard.

大多数动物都不会用两条腿走路。

Like, most animals don't get on two feet.

为什么呢

Why do

用四条腿。

on four feet.

你知道，许多哺乳动物都是用四条腿走路的。

You know, many mammals get on four feet.

是的。

Yeah.

它们很快就学会了。

They Very quickly.

有些甚至极其迅速。

Some of them extremely quickly.

但我记得，上次我跟桌子互动时，桌子比两条腿的东西稳多了。

But I don't you know, like, from the last time I've interacted with a table, that's much more stable than a thing than two legs.

这真是个非常难的问题。

It's just a really hard problem.

是的。

Yeah.

有多少种鸟用两只脚实现了这一点？

How many birds have figured it out with two feet?

从技术上讲，我们可以讨论一下本体论。

Well, technically, we can go into ontology.

它们有四只。

They have four.

我想它们有两只脚。

I guess they have two feet.

它们有

They have

两只脚。

two feet.

鸡。

Chickens.

你知道吗，很多恐龙都是两只脚的。

You know, dinosaurs have two feet, many of them.

据说如此。

Allegedly.

我刚刚才知道T。

I'm just now learning that T.

霸王龙吃的是草，而不是其他动物。

Rex was eating grass, not other animals.

T。

T.

霸王龙可能是个友善的宠物。

Rex might have been a friendly friendly pet.

你觉得呢？我不确定你有没有看过弗朗索瓦·谢莱特设计的通用智力测试。

What do you think about I don't know if you looked at the test for general intelligence that Francois Chalet put together.

嗯嗯。

Mhmm.

我不确定你有没有机会看过这类东西。

I don't if you got a chance to look at that kind of thing.

比如，你对如何解决类似智商测试的问题有什么直觉？

Like, what what's your intuition about how to solve, like, an IQ type of test?

我不知道。

I don't know.

我觉得这完全不在我的关注范围内，我认为在短期内并不相关。

I think it's so outside of my radar screen that it's not really relevant, I think, in the short term.

好吧，也许换一种问法，更贴近你的工作的是，你怎么用很少的样本数据来解决MNIST问题？

Well, I guess one way to ask another way perhaps more closer to what to your work is, like, how do you solve MNIST with very little example data?

没错。

That's right.

而这个问题的答案很可能是自监督学习。

And that's the answer to this probably is self supervised learning.

只需学习如何表示图像，然后在此基础上学习识别手写数字就只需要少量样本了。

Just learn to represent images, and then learning, you know, to recognize handwritten digits on top of this will only require a few samples.

我们在人类身上也观察到这种现象。

And we observe this in humans.

你给一个小孩看一本画册，里面只有几张大象的图片，就这样。

You show a young child a picture book with a couple of pictures of an elephant, and that's it.

孩子就知道什么是大象了。

The child knows what an elephant is.

今天我们也在实际系统中看到这种情况：我们用海量的图像来训练图像识别系统，这些图像要么是完全自监督的，要么是弱监督的。

And we see this today with practical systems, we train image recognition systems with enormous amounts of images, either completely self supervised or very weakly supervised.

例如，你可以训练一个神经网络来预测人们在Instagram上输入的标签。

For example, you can train a neural net to predict whatever hashtag people type on Instagram.

对吧？

Right?

你可以用数十亿张图像来做这件事，因为每天都有数十亿张图像出现。

And you can do this with billions of images because there's billions per day that are showing up.

因此，可用的训练数据量几乎是无限的。

So the amount of training data there is essentially unlimited.

然后，你可以取系统学习过程中中间几层的输出表示，将其作为分类器的输入，用于识别你想要的任何物体，效果相当不错。

And then you take the output representation, you know, a couple layers down from the output of what the system learned and feed this as input to a classifier for any object in the world that you want, and it works pretty well.

这就是迁移学习。

So that's transfer learning.

明白吗？

Okay?

或者说是弱监督迁移学习。

Or weekly supervised transfer learning.

人们也在这种自监督学习的场景中取得了非常快速的进展。

People are making very fast progress using self supervised learning this kind of scenario as well.

我猜测，这将是未来的趋势。

And my guess is that that's going to be the future.

对于自监督学习，你认为需要多少数据清洗来过滤恶意信号？或者用什么更好的术语？

For self supervised learning, how much cleaning do you think is needed for filtering malicious signal or what's a better term?

但很多人在Instagram上使用标签来获得不错的搜索引擎优化效果，嗯。

But, like, a lot of people use hashtags on Instagram to get, like, good SEO Mhmm.

但这并不能完全代表图片的内容。

That doesn't fully represent the contents of the image.

比如，他们发一张猫的照片，却打上科学、很棒、有趣这样的标签。

Like, they'll put a picture of a cat and hashtag it with, like, science, awesome, fun.

我不确定。

I don't know.

各种各样的标签，我不知道为什么要打上科学？

All kind I don't why would you put science?

嗯，所以

Well, so

这种做法并不是很好的搜索引擎优化。

That's not very good SEO.

几年前，我在Facebook（现在是Meta AI）工作的同事们处理这个问题时，只选择了大约17,000个与物理物体或场景相关的标签，比如那些具有视觉内容的标签。

The way the way my my colleagues who worked on this project at at Facebook, now Meta Meta AI, a few years ago dealt with this is that they only selected something like 17,000 tags that correspond to kind of physical things or or situations, like, you know, that has some visual content.

所以，你知道，你不会用像#TBT这样的标签之类的。

So, you know, you wouldn't have, like, hash TBT or anything like that.

哦，所以你的意思是他们只保留了一小部分标签？

Oh, so they they keep a very select set of hashtags is what you're saying?

是的。

Yeah.

好的。

Okay.

但数量仍然在一万到两万左右，所以还是相当大的。

But it's still it's still on the order of, you know, 10 to 20,000, so it's fairly large.

好的。

Okay.

你能跟我讲讲数据增强吗？

Can you tell me about data augmentation?

数据增强到底是什么？它是怎么用于视频的对比学习的？

What the heck is data augmentation, and how is it used, maybe contrastive learning for for video?

这里有哪些有趣的想法？

What are some cool ideas here?

对。

Right.

数据增强，首先，数据增强是指通过以不改变图像本质的方式对现有图像进行变换，来人为地扩大你的训练集。

So data augmentation I mean, first, data augmentation, you know, is the idea of artificially increasing the size of your training set by distorting the images that you have in ways that don't change the nature of the image.

对吧？

Right?

所以你会做，你做Nlist。

So you take you you do you do Nlist.

你可以在Nlist上进行数据增强。

You can do data augmentation on Nlist.

人们从上世纪九十年代就开始这么做。

And people have done this since the nineteen nineties.

对吧？

Right?

你拿一个NIST数字，稍微移动它、改变大小、旋转或扭曲它，你知道的，

You take a Nlist digit and you shift it a little bit or you change the size or rotate it, skew it, you know,

等等。

etcetera.

添加噪声。

Add noise.

添加噪声，等等。

Add noise, etcetera.

而且效果更好。

And it it works better.

如果你用增强后的数据训练一个监督分类器，你会得到更好的结果。

If you train a supervised classifier with augmented data, you're gonna get better results.

在过去的几年里，这变得非常有趣，因为许多用于预训练视觉系统的自监督学习技术都基于数据增强。

Now it's become really interesting over the last couple years because a lot of self supervised learning techniques to pre train vision systems are based on data augmentation.

这些基本技术最初受到我在九十年代初以及杰夫·辛顿也在九十年代初所研究的技术的启发。

And the basic techniques are originally inspired by techniques that I worked on in the early nineties and Jeff Hinton worked on also in the early nineties.

当时有一些并行的研究。

There were sort of parallel work.

我以前称这个为孪生网络。

I used to call this Siamese network.

基本上，取两个相同网络的副本，它们共享相同的权重，然后展示同一物体的两种不同视角。

So basically, take two identical copies of the same network, they share the same weights, and you show two different views of the same object.

这两种不同视角可能是通过数据增强获得的，也可能是你移动摄像头后对同一场景在不同时间拍摄的两种视角，或者类似的情况，又或者是同一个人的两张照片，诸如此类。

Either those two different views may have been obtained by data augmentation, or maybe it's two different views of the same scene from a camera that you moved or at different times or something like that, right, or two pictures of the same person, things like that.

然后你训练这个神经网络——这两个相同的网络副本——生成一种输出表示（向量），使得这两张图像的表示尽可能接近、尽可能相同。

And then you train this neural net, those two identical copies of this neural net, to produce an output representation, a vector, in such a way that the representation for those two images are as close to each other as possible, as identical to each other as possible.

对吧？

Right?

因为你想让系统学习一个函数，这个函数具有不变性，即当以这些特定方式变换输入时，其输出不会改变。

Because you want the system to basically learn a function that will that will be invariant, that will not change, whose output will not change when you transform those inputs in those particular ways.

所以这很容易实现。

So that's easy to do.

复杂的地方在于，如何确保当输入两张不同的图片时，系统会产生不同的结果？

What's complicated is how do you make sure that when you show two images that are different, the system will produce different things?

因为如果你不对此做特别设计，系统在训练时就会忽略输入。

Because if you don't have a specific provision for this, the system will just ignore the inputs when you train it.

它最终会忽略输入，只为所有输入生成一个相同的恒定向量。

It will end up ignoring the input and just produce a constant vector that is the same for every input.

这被称为坍缩。

That's called a collapse.

那么，如何避免坍缩呢？

Now how do you avoid collapse?

有两种思路：一种是我和贝尔实验室的同事简·布罗梅利以及其他几位在90年代初提出的，我们现在称之为对比学习，即使用负样本。

So there's two ideas: one idea that I proposed in the early 90s with my colleagues at Bell Labs, Jane Bromley and a couple other people, which we now call contrastive learning, which is to have negative examples.

所以，你会使用已知不同的图片对，将它们输入网络的两个副本，并将两个输出向量彼此推远。

So you have pairs of images that you know are different, and you show them to the network, those two copies, and then you push the two output vectors away from each other.

这样最终就能保证语义相似的输入产生相似的表示，而不同的输入则产生不同的表示。

And it will eventually guarantee that things that are semantically similar produce similar representations, and things that are different produce different representations.

我们实际上是为了一个签名验证的项目提出了这个想法。

We actually came up with this idea for a project of doing signature verification.

我们会收集同一个人的多个签名，然后训练一个神经网络来生成相同的表示，同时迫使系统为不同的签名生成不同的表示。

So we would collect from like, multiple signatures from the same person, and then train a neural net to produce the same representation, and then, you know, force the system to produce different representation from different signatures.

这个问题实际上是当时AT&T的子公司NCR提出的，他们有兴趣将签名的表示存储在信用卡磁条的80字节中。

Was actually the the problem was proposed by people from what was a subsidiary of AT and T at the time called NCR, and they were interested in storing representation of the signature on the 80 bytes of the magnetic strip of a credit card.

于是我们想到了一个拥有80个输出的神经网络，我们将这些输出量化为字节，以便能够编码签名的表示。

So we came up with this idea of having a neural net with 80 outputs, you know, that we would quantize on bytes so so that we could encode the the

然后这种编码被用来判断签名是否匹配。

And that encoding was then used to compare whether the signature matches or not.

没错。

That's right.

所以你会让签名通过神经网络，然后将输出向量与卡上存储的内容进行比较。

So then you would, you know Interesting.

有意思。

Sign, it would run through the neural net, and then you would compare the output vector to whatever is stored on your card.

它真的有效吗？

Did it actually work?

它有效，但他们最终没有使用，因为根本没人在意。

It worked, but they ended up not using it because nobody cares actually.

我的意思是，美国的金融支付系统在这方面比欧洲宽松得多。

I mean, the American financial payment system is incredibly lax in that respect compared to Europe.

哦，指的是签名吗？

Oh, with the signatures?

签名到底有什么用呢？

What's the purpose of signatures anyway?

这非常

This is very

没人看这些签名。

Nobody looks at them.

没人关心。

Nobody cares.

你知道的吧？

You know?

就是这样。

It's yeah.

是的。

Yeah.

不。

No.

所以这就是对比学习。

So that that's contrastive learning.

对吧？

Right?

所以你需要正样本和负样本对。

So you need positive and negative pairs.

而这个问题在于，虽然我手头有这篇原始论文，但我其实对它并不太有信心，因为它在高维情况下并不有效。

And the problem with that is that, you know, even though I had the original paper on this, I'm actually not very positive about it because it doesn't work in high dimension.

如果你的表示是高维的，那么两个事物不同的方式就会太多。

If your representation is high dimensional, there's just too many ways for two things to be different.

因此你需要大量的、大量的负样本对。

And so you would need lots and lots and lots of negative pairs.

所以有一个相对近期的实现方法，来自谷歌多伦多团队，杰夫·辛顿是那里的资深成员。

So there is a particular implementation of this which is relatively recent from actually the Google Toronto group, where Jeff Hinton is the senior member there.

它叫做 SIMCLR，s I m c l r。

It's called SIMCLR, s I m c l r.

基本上是一种用特定目标函数实现对比学习思想的方法。

Basically a particular way of implementing this idea of contrastive learning with a particular objective function.

如今我更感兴趣的是非对比方法，也就是其他确保不同输入的表示会不同的方式。

Now what I'm much more enthusiastic about these days is non contrastive methods, so other ways to guarantee that the representations will be different for different inputs.

这实际上基于杰夫·辛顿在九十年代初与他当时的学生苏·贝克尔提出的一个想法。

And it's actually based on an idea that Geoff Hinton proposed in the early nineties with his student at the time Sue Becker.

它的核心思想是最大化两个系统输出之间的互信息。

And it's based on the idea of maximizing the mutual information between the outputs of the two systems.

你只展示正样本对，也就是你已知具有一定相似性的图像对，并训练两个网络使其具有信息量，同时尽可能相互具有信息量。

You only show positive pairs, you only show pairs of images that you know are somewhat similar, and you train the two networks to be informative, but also to be as informative of each other as possible.

换句话说，一个表示必须能从另一个表示中预测出来。

So basically one representation has to be predictable from the other, essentially.

你知道，他早在九十年代初就提出了这个想法，发表了若干篇论文，但此后几十年都没人继续研究。

You know, he proposed that idea had, you know, a couple of papers in the early nineties and then nothing was done about it for decades.

我和我在FAIR的博士后们，特别是博士后斯特凡·丹尼斯，重新振兴了这个想法，他现在是芬兰阿尔托大学的初级教授。

And I kind of revived this idea together with my postdocs at at Fair, particularly a postdoc called Stephane Denis, who's now a junior professor in in Finland at the University of Alto.

我们提出了一种叫做Barlow Twins的方法，这是一种通过某些假设来最大化向量信息量的特定方式。

We came up with something called that we call Barlow Twins, and it's a particular way of maximizing the information content of a vector using some hypotheses.

我们现在还推出了一个更近期的版本，叫做VicReg，拼写是v-i-c-a-r-e-g。

And we have kind of another version of it that's more recent now called VicReg, v I c a r e g.

它的意思是方差、不变性、协方差和正则化。

That that means variance, invariance, covariance, regularization.

这是我过去十五年来在机器学习领域最兴奋的成果。

And I'm it's the thing I'm the most excited about in in machine learning in the last fifteen years.

我的意思是，我对此真的非常兴奋。

I mean, I'm I'm not I'm really, really excited about this.

对于这种非对比学习方法，哪些数据增强方式是有用的？

What kind of data augmentation is useful for that non contrasted learning method?

我们说的是这并不那么重要吗？

Are we talking about does that not matter that much?

或者这似乎是步骤中非常重要的部分，是的。

Or it seems like a very important part of the step Yeah.

你如何生成那些相似但又足够不同的图像。

How you generate the images that are similar but sufficiently different.

是的。

Yeah.

没错。

That's right.

这是一个重要的步骤，但也是一个令人头疼的步骤，因为你必须了解哪些数据增强方式不会改变对象的本质。

It's an important step, it's also an annoying step because you need to have that knowledge of what data augmentation you can do that do not change the nature of the of the object.

是的。

Yeah.

因此，标准的做法是——你知道，这个领域很多研究者都在用——就是使用某种类型的失真。

And so the standard scenario, which, you know, a lot of people working in this area are using, is you use the the type of distortion.

基本上，你进行几何失真。

So so basically, you do geometric distortion.

也就是说，只是稍微移动一下图像。

So one basically just shifts the image a little bit.

这叫做裁剪。

It's called cropping.

另一种方法是稍微改变一下尺度。

Another one kinda changes the scale a little bit.

还有一种是旋转图像。

Another one kinda rotates it.

另一种是改变颜色。

Another one changes the colors.

你知道，你可以调整色彩平衡之类的。

You know, you can do a shift in color balance or something like that.

饱和度。

Saturation.

另一个是让它变得模糊。

Another one sort of blurs it.

另一个是添加噪声。

Another one adds noise.

所以你有一套标准的变换方法，人们会为不同算法使用相同的变换，以便进行比较。

So you have like a a catalog of kind of standard things, and people try to use the same ones for different algorithms so that they can compare.

但有些算法，特别是某些自监督算法，实际上能应对更强大、更激进的数据增强，而有些则不能。

But some algorithms, some self supervised algorithm actually can deal with much bigger, like, more aggressive data augmentation, and some don't.

所以这使得整个事情变得复杂。

So that that kinda makes the whole thing difficult.

但这就是我们所说的失真类型。

But that's the kind of distortions we're talking about.

因此，你用这些扭曲方式来训练，然后切断网络的最后一层或几层，将提取的特征作为分类器的输入，在ImageNet或其他数据集上训练分类器，并评估性能。

And so you train with those distortions, and then you chop off the last layer or a couple layers of the network, and you use the representation as input to a classifier, you train the classifier on ImageNet, let's say, or whatever, and measure the performance.

有趣的是，那些在消除图像间无关信息（即这些扭曲）方面表现优异的方法，确实能很好地去除这些干扰。

And interestingly enough, the methods that are really good at eliminating the information that is irrelevant, which is the distortions between those images, do a good job at eliminating it.

因此，你无法在这些系统中使用这些特征来进行目标检测和定位，因为这些信息已经丢失了。

And as a consequence, you cannot use those the representations in those systems for things like object detection and localization, because that information is gone.

所以，你需要采用哪种数据增强方式，取决于你最终希望系统完成的任务；而我们今天常用的标准化数据增强方法，仅适用于目标识别或图像分类任务。

So the type of data augmentation you need to do depends on the task you want eventually the system to solve, and the type of data augmentation, standard data augmentation that we use today are only appropriate for object recognition or image classification.

它们并不适用于诸如

They're not appropriate for things like

你能帮我理解一下吗？你说定位效果不好，是因为它在区分负样本方面表现不佳，所以才不能用于定位？

Can you help me out understand why the localization is so you're saying it's just not good at the negative, like at classifying the negative, so that's why it can't be used for the localization?

不是的。

No.

你训练系统时，会给它一张图像，然后再给它同一张图像经过平移和缩放后的版本。

It's just that you train the system, you know, you you you give it an image, and then you give it the same image shifted and scaled.

是的

Yeah.

你告诉它这是同一张图片。

And you tell it that's the same image.

是的

Yeah.

所以系统基本上被训练成消除关于位置和大小的信息。

So the system basically is trained to eliminate the information about position and size.

所以现在你想用这个

So now and now you want to use that

哦，我明白了，比如，

Oh, I to, like,

找出一个物体的位置和大小。

figure out where an object is and what size it is.

明白了。

Got it.

就像一个边界框。

Like a bounding box.

比如，他们实际上能够做到。

Like, they'd be able to actually okay.

它仍然能够找到图像中的物体。

It could still find it could still find the object in the image.

只是它不太擅长确定该物体的精确边界。

It's just not very good at finding the exact boundaries of that object.

有意思。

Interesting.

有意思。

Interesting.

这其实是一个有趣的哲学问题。

Which, you know, the that's an interesting sort of philosophical question.

目标定位到底有多重要呢？

How important how important is object localization anyway?

我们简直着迷于测量图像分割。

We're we're, like, obsessed by measuring, like, image segmentation.

着迷于完美地掌握物体的边界，但事实上，这对于我们理解场景内容来说未必那么重要。

Obsessed by measuring perfectly knowing the boundaries of objects when arguably that's not that essential to an understanding what are the contents of the scene.

另一方面，我认为从进化角度看，动物最早的视觉系统基本上都是关于定位的，识别则非常少。

On the other hand, I think evolutionarily, the first vision systems in animals were basically all about localization, very little about recognition.

在人类大脑中，你有两个独立的通路分别用于识别场景或物体的性质，以及定位物体。

And in the human brain, you have two separate pathways for recognizing the nature scene or an object and localizing objects.

所以，你使用第一个通路，即腹侧通路，来判断你正在看的是什么。

So you use the first pathway called the ventral pathway for, you know, telling what you're looking at.

另一个通路，背侧通路，则用于导航、抓取以及所有其他活动，你知道，许多生存所需的能力都依赖于定位和检测。

The other pathway, the dorsal pathway, is used for navigation, for grasping, for everything And, you know, basically, a lot of the things you need for survival are localization and detection.

相似性学习、对比学习或这些非对比方法，是否等同于理解事物？

Is similarity learning or contrastive learning or these non contrastive methods the same as understanding something?

仅仅因为你知道一个失真的猫和一个正常的猫是同一个东西，就代表你理解了‘猫’的含义吗？

Just because you know a distorted cat is the same as a nondistorted cat, does that mean you understand what it means to be a cat?

在某种程度上。

To some extent.

我的意思是，这显然只是一种表面的理解。

I mean, it's a superficial understanding, obviously.

但你觉得这种方法的上限在哪里？

But, like, what is the ceiling of this method, do you think?

这仅仅是实现自监督学习道路上的一个技巧，还是我们可以走得

Is this just one trick on the path to doing self supervised learning, or can we go

是的。

Yeah.

真的非常远吗？

Really, really far?

我认为我们可以走得很远。

I think we can go really far.

如果我们能弄清楚如何使用这类技术——也许方式非常不同，但核心是通过视频训练系统来实现视频预测。

So if we figure out how to use techniques of that type, perhaps very different, but, you know, the signature to train a system from from video to do video prediction essentially.

我认为我们会有一条路径，你知道，朝着某种程度上的、我不会说是无限的，但朝着机器具备某种物理常识的方向发展。

I think we'll have a path, you know, towards you know, I wouldn't say unlimited, but but a path towards some level of, you know, physical common sense in in machines.

而且我也认为，通过像视觉这样的高吞吐量通道来学习世界如何运作的能力

And I also think that that ability to learn how the world works from a sort of high throughput channel like like vision

嗯。

Mhmm.

是迈向真正人工智能的必要一步。

Is a necessary step towards sort of real artificial intelligence.

换句话说，我相信基于现实的智能。

In other words, I believe in grounded intelligence.

我不认为我们可以仅通过文本来训练机器变得智能。

I don't think we can train a machine to be intelligent purely from text.

因为我认为，与世界相关的信息量在文本中所包含的，相比我们需要知道的，是微乎其微的。

Because I think the amount of information about the world that's contained in text is tiny compared to what we need to know.

举个例子，而且，是的，我知道人们尝试这样做已经有三十年了，对吧，就像那个心理项目之类的，基本上就是把所有已知的事实写下来，希望某种常识能够从中浮现出来。

So for example, let's and and, yeah, I know people have attempted to do this for for thirty years, right, the the psych project and things like that, right, of basically kind of writing down all the facts that are known and hoping that some some sort of common sense would emerge.

我觉得这基本上是不可能的。

I think it's basically hopeless.

但让我举个例子。

But let me take an example.

你拿一个物体。

You take an object.

我向你描述一种情况。

I I describe a situation to you.

我拿一个物体，把它放在桌子上，然后推桌子。

I take an object, I put it on the table, and I push the table.

你很清楚，物体将会随着桌子一起被推走。

It's completely obvious to you that the object will be pushed with the table.

对吧？

Right?

因为它就放在桌子上。

Because it's sitting on it.

我相信，世界上没有任何文本能够解释这一点。

There's no text in the world, I believe, that explains this.

所以，即使你训练一个像GPT-5000那样强大的机器，它也永远学不会这一点。

And so if you train a machine as powerful as it could be, you know, your GPT 5,000 or whatever it is, it's never gonna learn about this.

这些信息根本不会出现在任何文本中。

That information is just not is not present in any text.

好吧，就像心理项目那个梦想一样，我认为目标是拥有大约一千万个这样的事实，为你提供一个起点，就像父母引导你一样。

Well, the the question, like, with the psych project, the dream, I think, is to have, like, like, 10,000,000, say, facts like that that give you a head start, like a parent guiding you.

但我们人类并不需要父母告诉我们，桌子会动——哦，抱歉，手机会随着桌子一起动。

Now we humans don't need a parent to tell us that the table will move oh, sorry, the smartphone will move with the table.

但我们通过其他方式获得了大量指导，因此我们或许可以给它一个快速捷径。

But we get a lot of guidance in other ways, so it's possible that we can give it a quick shortcut.

那猫呢？

And what about cat?

猫是知道这一点的。

The cat knows that.

不。

No.

但它们是进化而来的。

But they evolved.

所以

So

不。

No.

它们像我们一样学习。

They learn like us.

对不起。

The the sorry.

物质的物理规律？

The physics of stuff?

是的。

Yeah.

嗯，是的。

Well yeah.

所以你的意思是，你把很多智能归因于后天培养，而不是先天本性。

So you're saying it's so you're putting a lot of intelligence onto the nurture side, not the nature.

是的。

Yeah.

因为我们似乎经历了一个非常低效、可以说相当低效的进化过程，才从细菌发展到今天的我们。

Because we we seem to have you know, there's a very inefficient, arguably, process of evolution that got us from bacteria to who we are today.

从底层开始，现在我们在这里了。

Started at the bottom, now we're here.

确实如此。

So True.

问题是，这究竟怎么样。

The question is how okay.

所以你看，问题在于，这种硬件的本质究竟有多根本？如果它是根本的，有没有办法绕过它？

So you see, the question is how fundamental is that, the the nature of the whole hardware, and then is there any way to shortcut it if it's fundamental?

如果不是这样，如果我们所讨论的大部分智能、大部分精彩的内容主要来自后天培养、通过观察世界习得，那我们只需静坐就能构建出你所说的那个宏大、优美、吸引人的背景模型。

If it's not, if it's most of intelligence, most of the cool stuff we've been talking about is mostly nurture, mostly trained, we figure it out by observing the world, we can form that big, beautiful, sexy background model that you're talking about just by sitting there.

那么，好吧，也许这一切本质上都是监督学习。

Then, okay, then you need to then, like, maybe it is all supervised learning all the way down.

或者说，是自监督学习。

Self supervised learning, say.

无论是什么让人类智能区别于其他动物——很多人认为是语言和逻辑推理这类能力——

Whatever it is that makes, you know, human intelligence different from other animals, which, you know, a lot of people think is language and logical reasoning and this kind of stuff.

它不可能太复杂，因为这种能力只在最近一百万年内才出现。

It cannot be that complicated because it only popped up in the last million years.

是的。

Yeah.

而且，它只涉及基因组中不到1%的差异，也就是人类基因组与黑猩猩或其他物种之间的区别。

And, you know, it it and it only involves, you know, less than 1% of a genome, right, which is the difference between human genome and chimps or whatever.

所以它不可能那么复杂。

So it can't be that complicated.