后LLM时代：空间智能与世界模型 —— 李飞飞与贾斯汀·约翰逊，World Labs

比如，你会丢失字体、丢失换行、丢失页面上那种二维的排版布局。

Like, you lose the font, you lose you lose the line breaks, you lose sort of the two d arrangement on the page.

而在很多情况下，对于很多事物来说，这可能并不重要。

And and for a lot of cases for lot of things, maybe that doesn't matter.

但对某些事情来说，这确实很重要。

But for some things, it does.

我认为像素是一种更少损失的表示方式，能更真实地反映世界中的内容。

And I I think pixels are this sort of more more lossless representation of what's going on in the world.

在某些方面，它也是一种更通用的表示方式，更贴近我们人类在探索世界时所看到的东西。

And in in some ways, a more general general representation that more matches what what we what we humans see as we as we navigate the world.

所以，这里有一个效率方面的论点。

So so, like, there's an efficiency argument to be made.

比如，也许把文本渲染成图像再输入视觉模型，并不是特别高效的做法。

Like, maybe it's not super efficient to, like, you know, render your text to an image and then feed that to a vision model.

这正是DeepSeek所做的事情。

That's exactly what DeepSeek did.

是的。

Yeah.

它确实有点奏效。

It it was, like, kinda worked.

我

I

我认为这与整个世界模型有关。

think this ties into the whole world model.

比如，今年我看到的最喜爱的论文之一，是关于归纳偏差来探测世界模型的。

Like, one of the my favorite papers that I saw this year was about inductive bias to probe for world models.

这是一篇哈佛大学的论文，他们向大语言模型输入了大量的轨道模式，然后让模型预测行星围绕太阳的轨道。

So it was a Harvard paper where they fed a lot of, like, orbital, patterns into an LLM, and then they asked the LLM to predict the orbit of a planet around the sun.

模型生成的结果看起来不错，但如果你让它画出受力矢量，就会变得一团糟。

And the model generated looked good, but then if you asked it to draw the force vectors, it would be all wacky.

你知道的？

You know?

它实际上并不会遵循那个规律。

It wouldn't actually follow it.

那么，你怎么看待你所获得的数据中嵌入了什么内容？

So how do you think about what's embedded into the data that you get?

我们可以谈谈三维世界模型的分词问题。

And we can talk about maybe tokenizing for three d world models.

比如，信息的维度是什么？

Like, what are, the dimensions of information?

有视觉信息，但你到底需要从这些数据中提取出多少潜在的隐藏力呢？

There's the visual, but, like, how much of, like, the underlying hidden forces, so to speak, you need to extract out of this data?

在这方面，有哪些挑战？

And, like, what are some of the challenges there?

是的。

Yeah.

我觉得解决这个问题有几种不同的方法。

I I think there's different ways you could approach that problem.

一种方法是明确地去做，比如说，我想测量所有的力，然后把这些作为训练数据输入到模型中。

One is, like, you could try to be explicit about it and say, like, oh, I want to, you know, measure all the forces and feed those as training data to your model.

对吧？

Right?

然后你可以运行一个传统的物理模拟，从而知道场景中所有的力，再用这些力作为训练数据来训练一个模型，希望它能预测这些力。

Then you could, like, sort of run a traditional physics simulation and, you know, then know all the forces in the scene and then use those as as training data to train a model that's now gonna hopefully predict those.

或者你可以寄希望于某些东西能更隐式地涌现出来。

Or you could hope that something emerges more latently.

对吧？

Right?

你可以端到端地训练某个东西，针对一个更通用的问题，然后希望模型内部的某些部分能够学会模拟类似物理的机制，以便做出正确的预测。

That you kind of train on something end to end and then on on a more general problem, and then hope that somewhere some some something in in the internals of the model must learn to model something like physics in order to make the proper predictions.

而这两种方式基本上是我们目前更普遍的两大范式。

And those are kind of the two big paradigms that we have more generally.

但并没有任何迹象表明，这种隐式建模能让你得出空间和动力学的因果定律。

But there's no indication that those latent modeling will get you to a causal law of space and dynamics.

对吧？

Right?

这正是当今深度学习与人类智能真正开始分道扬镳的地方。

That's where today's deep learning and human intelligence actually start to bifurcate.

因为从根本上说，深度学习仍然是在拟合模式。

Because fundamentally, the deep learning is still fitting patterns.

能聊哲学挺好的。

Very useful to get philosophical.

你说我们也在试图拟合模式，但也许我们是在拟合更广泛的一系列模式，时间跨度更长，奖励函数也不同。

And you say that, like, we're trying to fit patterns too, but maybe we're trying to fit a more broad array of patterns, like, over a with with a longer time horizon, a different reward function.

但基本上，你提到的那篇论文就是这个问题：它学会了拟合轨道的具体模式，但却没有像你期望的那样真正泛化。

But but, like, basically, the the paper you mentioned is sort of, you know, that problem, that it learns to fit the specific patterns of orbits, but then it doesn't actually generalize in the way that you'd like.

它没有一种对引力的因果模型。

It doesn't have a sort of causal model of gravity.

对。

Right.

因为即使是大理石，你也知道，它在尝试中。

Because even in marble, you know, was trying it in.

它生成了这些美丽的景观，里面还有拱门。

It generates these beautiful sceneries, and there's, like, arches in them.

但这个模型真的理解拱门是如何像石头一样以中心为支撑，以及其实际物理结构的吗？

But does the model actually understand how, you know, the arch is actually, you know, drawing on the center kinda like stone and, like, you know, the actual physical structure of it.

另一个问题是，只要它总能渲染出符合我们想象的物理模型的东西，它是否真的理解这些有关系吗？

And the other question is, like, does it matter that it does understand it as long as it always renders something that would fit the physical model that we imagine?

如果你用‘理解’这个词的本意，我敢肯定模型并不理解。

If you use the word understand the way you understand, I'm pretty sure the model doesn't understand it.

模型只是从数据中学习，从模式中学习。

The model is learning from the data, learning from the pattern.

是的，对于这些应用场景来说，这重要吗？

Yeah, does it matter, especially for the use cases?

这是个好问题。

It's a good question.

目前，我认为这并不重要，因为只要它能完美地渲染出你需要的内容。

For now, I don't think it matters because it renders out what you need, assuming it's perfect.

是啊，这取决于使用场景。

Yeah, mean, depends on the use case.

如果使用场景是我想为虚拟电影或制作生成某种背景之类的东西，那你只需要看起来合理就行。

If the use case is, I want to generate sort of a backdrop for for virtual film or production or something like that, all you need is something that looks plausible.

在这种情况下，可能确实不重要。

And in that case, probably it doesn't matter.

但如果你打算用它来设计一座你之后要在现实世界中建造的建筑，比如你是个建筑师，那就有必要正确建模受力情况了，因为你不希望建筑在实际建造时倒塌。

But if you're gonna use this to, like, you know, if you're an architect and you're gonna use this to design a building that you're then gonna go build in the real world, then, yeah, it does matter that you model the forces correctly because you don't want the thing to break when to actually build it.

但即便如此，即使你的模型包含了语义，我仍然认为模型层面的‘理解’和人类层面的‘理解’是两回事。

But even there, right, like even if your model has the semantics in it, let's say, I still don't think the understanding of the signal or the output on the model part and the understanding on the human part is a different word.

但这又回到了哲学层面。

But this gets, again, philosophical.

是啊。

Yeah.

我的意思是，理解这件事有个技巧。

I mean, there there's this trick with understanding.

对吧？

Right?

这些模型的智能类型与人类智能非常不同。

Like, these models are a very different kind of intelligence than human intelligence.

而人类智能很有趣，因为我觉得我之所以能理解事物，是因为我能在某种程度上内省自己的思维过程。

And human intelligence is interesting because, you know, you know, I think that I understand things because I can introspect my own thought process to some extent.

然后我相信我的思维过程可能和别人的类似，所以当我观察他人的行为时，我会推断他们的内心状态可能和我观察到的自己的内心状态相似。

And then I believe that my thought process probably works similar to other people's so that when I observe someone else's behavior, then I infer that their internal mental state is probably similar to my own internal mental state that I've observed.

因此，我知道自己能理解事物，所以我假设你也理解某些东西。

And therefore, I know that I understand things, so there I assume that you understand something.

但这些模型就像一种外星般的智能形式，它们能做出非常有趣的事情。

But these models are sort of like this this alien form of intelligence where they can do really interesting things.

它们能展现出非常有趣的行为。

They can exhibit really interesting behavior.

但无论它们是否具有类似内部认知或自我反思的东西，如果确实存在的话，也与我们的完全不同。

But whatever kind of internal the equivalent of internal cognition or internal self reflection that they have, if it exists at all, is totally different from what we do.

所以这是

So it's

它没有自我意识。

It doesn't have the self awareness.

对。

Right.

但这意味着，当我们观察到这些系统表现出看似有趣或智能的行为时，我们并不能因此推断出它们的其他特性，因为它们对世界的认知模型和思考方式与我们截然不同。

But what that means is that when when we observe seemingly interesting or intelligent behavior out of these systems, we can't necessarily infer other things about them because their their model of the world and the way they think is so different from us.

所以，你认为最终需要两个不同的模型来分别处理视觉任务和建筑生成吗？

So would you need two different models to do the visual one and the architectural generation, you think, eventually?

就像，你目前在模型构建上的方法并没有什么根本性的差异。

Like, there's not anything fundamental about the approach that you've taken on the model building.

这更多是关于扩大模型规模和提升其能力？

It's more about scaling the model and the capabilities of it?

还是说，过于依赖视觉特性会阻碍你真正理解背后的物理原理，从而无法信任它生成的CAD设计在现实世界中真的能正常工作？

Or is there something about being very visual that prohibits you from actually learning the physics behind this, so to speak, so that you could trust it to generate a CAD design that then is actually gonna work in the real world?

我认为这只是一个数据规模和模型优化的问题。

I think this is a matter of scaling data and and and bettering model.

我不认为这两者之间存在根本性的差异。

I don't think there's anything fundamental that separates these two.

是的。

Yeah.

我希望它能成为一个统一的模型。

I would like it to be one model.

但我认为，从某种意义上说，深度学习的一个大问题是：如何让你的模型产生超越训练数据的涌现能力？

But I think, like, the big problem in deep learning in some sense is how do you get emergent capabilities beyond your training data?

你能否让模型在没有专门训练预测力的情况下，却能内在地理解这些力？

Are you gonna get something that understands the forces while it wasn't trained to predict the forces, but it's gonna learn them implicitly internally?

我认为我们在其他大型模型中看到的很多现象表明，这种涌现行为确实在规模扩大时会发生。

And I think a lot of what we've seen in other large models is that a lot of this emergent behavior does happen at scale.

这种能力会转移到其他模态、其他应用场景和其他任务吗？

And will that transfer to other modalities and other use cases and other tasks?

我希望如此。

I hope so.

但这将是一个需要时间逐步验证的过程。

But that'll that'll be a a process that we need to play out over time and see.

有没有可能直接使用现成的物理引擎？毕竟游戏行业已经帮我们省去了大量工作，还是我们必须因为某些根本性差异而重新发明一切？

Is there a temptation to rely on physics engines that already exist out there that are you know, basically, gaming industry has saved you a lot of this work, or do we have to reinvent things for some fundamental mismatch?

我认为这有点像

I think that's sort of

攀登技术的阶梯。

like climbing the ladder of technology.

对吧？

Right?

某种程度上，你之所以想构建这些系统，正是因为传统的物理引擎在某些情况下并不适用。

Like, some sense, the reason that you wanna build these things at all is because maybe traditional physics engines don't work in some situations.

如果一个物理引擎是完美的，我们就没必要构建模型了，因为问题已经被解决了。

If a physics engine was perfect, we would have sort of no need to build models because the problem would have already been solved.

所以某种程度上，我们之所以想做这件事，是因为经典物理引擎无法像我们希望的那样普遍地解决问题。

So in some sense, the reason why we want to do this is because classical physics engines don't solve problems in the generality that we want.

但这并不意味着我们需要抛弃它们，从头开始一切。

But that doesn't mean we need to throw them away and start everything from scratch.

对吧？

Right?

我们可以使用传统的物理引擎生成数据，然后用这些数据来训练我们的模型。

We can use traditional physics engines to generate data that we then train our models on.

然后你就相当于把物理引擎的原理提炼到了你训练的神经网络的权重中。

And then you're sort of distilling the physics engine into the weights of the neural network that you're training.

我认为，如果你对比其他实验室的工作，很多人推测，比如Sora就有一点这样的成分。

I I think that's a lot of what if you compare the work of other labs, people are speculating that, you know, Sora had a little bit of that.

Genie three，嗯。

Genie three Mhmm.

有一点那种感觉。

Had a bit of that.

Genie three 就像是一个电子游戏。

Genie three is, like, explicitly like a video game.

比如，你可以控制角色

Like, you have controls to

是的。

Yeah.

在其中走动。

To walk around in.

我总是觉得，我们为娱乐而发明的东西，最终竟然真的会进入严肃的工作领域，这真的很有趣。

And I I I always think, like, it's really funny how the things that we invent for fun actually does eventually make it into serious work.

嗯。

Mhmm.

没错。

Yep.

整个AI革命部分起源于图形芯片。

The whole AI revolution started by graphics chips partially.

从使用GPU生成大量三角形，转变为生成其他各种各样的东西。

Miss using the GPU for for generating a lot of triangles into generating a lot of everything else basically.

是的。

Yeah.

我们稍微提到了Marble。

We touched on marble a little bit.

我觉得你们选择Marble，某种程度上像是在结束低调状态，如果能这么说的话。

I think you guys chose marble as I kind of feel like you're sort of a little bit coming out of stealth moment if you can call it that.

是的。

Yeah.

也许你们能给我们一个简洁的解释，说明人们应该从中理解什么？因为在这里每个人都可以尝试Marble，但我认为他们可能无法将其与你们的愿景和其他实验室所推出的生成式世界之间的区别联系起来。

Maybe we can get a concise explanation from you on what people should take away because everyone here can try Marble, but I don't think they might be able to link it to the differences between what your vision is versus other, I guess, generative worlds they may have seen from other labs.

Marble是我们模型的一个缩影。

So Marble is a glimpse into our model.

对吧？

Right?

我们是一家空间智能模型公司。

We are a model spatial intelligence model company.

我们相信空间智能是下一个前沿。

We believe spatial intelligence is the next frontier.

为了构建空间智能模型，模型必须在理解、推理和以多模态方式生成世界方面非常强大，同时还要实现我们最终希望达到的、与人类交互世界一样复杂的互动水平。

In order to make spatially intelligent models, the model has to be very powerful in terms of its ability to understand, reason, generate in very multimodal fashion of worlds, as well as allow the level of interactivity that we eventually hope to be as complex as how humans can interact with the world.

这就是空间智能的宏伟愿景，以及我们所设想的世界模型。

So that's the grand vision of spatial intelligence, as well as the kind of world models we see.

MARBO 是这一愿景的首次展现。

MARBO is the first glimpse into that.

这是这段旅程的第一步。

It's the first part of that journey.

它是世界上首个向公众开放的、能够以这种保真度生成三维世界的同类首创模型。

It's the first in class model in the world that generates three d worlds in this level of fidelity that is in the hands of the public.

这是起点。

It's the starting point.

实际上，他写了这篇技术博客。

Actually wrote this tech blog.

贾斯汀花了大量时间撰写这篇技术博客。

Justin spent a lot of time writing that tech blog.

我不知道你有没有时间浏览一下。

I don't know if you had time to browse it.

贾斯汀详细剖析了Marble的输入方式——我们能使用哪些多模态输入，哪些可编辑性允许用户与模型互动，以及我们能获得哪些类型的输出。

Mean, Justin really broke it down into what are the inputs we can multimodal inputs of Marble, what are the kind of editability which allows user to be interactive with the model and what are the kind of outputs we can have?

是的。

Yeah.

所以，Marble，从某种角度来看，它是一个生成三维世界的生成模型。

So so Marble, like, basically, one way of looking at it, it's the system it's a generative model of three d worlds.

对吧？

Right?

你可以输入文本、图像或多个图像，它会为你生成一个与这些输入相匹配的三维世界。

So you can input things like text or image or multiple images, and it will generate for you a three d world that kind of matches those inputs.

而且它还具有交互性，你可以交互式地编辑场景。

And it's also interactive in the sense that, you can interactively edit scenes.

比如，我可以生成这个场景，然后说：我不喜欢这个水瓶。

Like, I could generate this scene and then say, don't like the water bottle.

把它改成蓝色。

Make it blue instead.

把桌子拿掉。

Like, get take out the table.

把这些麦克风重新摆放一下。

Like, may change these microphones around.

然后你可以根据这些交互式编辑生成新的世界，并导出为多种格式。

And then you can generate new worlds based on these interactive edits and export in a variety of formats.

使用Marble，我们实际上同时想实现两个目标，我认为我们很好地把握了平衡。

And with Marble, we were actually trying to do sort of two things simultaneously, and I think we we managed to pull off the balance pretty well.

一个是构建一个朝着空间智能宏伟愿景迈进的模型。

One is actually build a model that goes towards the grand vision of spatial intelligence.

模型需要能够理解多种类型的输入，需要能够在各种情境下建模世界，还需要能够模拟它们随时间变化的反事实情况。

And models need to be able to understand lots of different kinds of inputs, need to able to model worlds in a lot of situations, need to be able to model counterfactuals of how they could change over time.

因此，我们希望开始构建具备这些能力的模型，而今天的Marble已经初步具备了所有这些特征。

So we wanted to start to build models that have these capabilities, and Marble Marble today does already have hints of all of these.

但与此同时，我们是一家公司。

But at the same time, we we were we're a company.

我们是一家企业。

We're a business.

我们真正努力避免这只是一个科研项目，而是要打造一个对现实世界中人们真正有用的产品。

We were really trying not to have this be a science project, but also build a product that would be useful to people in the real world today.

因此，尽管Marble同时是一个朝着空间智能愿景迈进的世界模型，但它也被有意设计成一个今天就能为人们带来实用价值的产品。

So while Marble is simultaneously a world model that is building towards this vision of spatial intelligence, it was also very intentionally designed to be a thing that people could find useful today.

我们已经开始看到一些新兴的应用场景，涵盖游戏、视觉特效和电影领域，我认为Marble今天作为一款产品就能做很多非常有趣的事情，同时也为未来我们希望构建的宏大世界模型奠定了基础。

And we're see starting to see emerging use cases, for in gaming, in VFX, in in film, where I think there's a lot of really interesting stuff that Marvel can do today as a product and then also set a foundation for the grand world models that we want to build going into the future.

是的。

Yeah.

我注意到一个非常有趣的工具，你可以在里面录制你的场景。

I noticed one tool that was very interesting because you can record your scene inside.

是的。

Yes.

是的。

Yes.

这非常重要。

It's very important.

录制功能意味着对摄像机位置的精确控制。

The ability to record means a very precise control of camera placement.

为了实现精确的摄像机定位，你必须具备对三维空间的感知。

In order to have precise camera placement, it means you have to have a sense of three d space.

否则，你就不知道如何调整和移动摄像机。

Otherwise, you don't know how to orient your camera and how to move your camera.

因此，这是这种模型的自然结果，这也是为什么这仅仅是其中一个例子。

So that is a natural consequence of this kind of model, and and this is why this is just one of the examples.

是的。

Yeah.

当我使用视频生成模型时，我发现我必须学习导演的语言，因为我得去移动它们，比如平移，你知道的，比如，

I I find when I play with video generative models, I'm having to learn the language of being a director because I have to move them up, like pan, you know, like,

停滞不前。

stalling out.

你不能说将镜头向北平移六十三度。

You cannot say pan sixty three degrees to the north.

对吧？

Right?

你根本没有这种控制能力。

You just don't have that control.

而在Marble中，你可以精确地控制摄像机的定位。

Whereas in Marble, you have precise control in terms of placing a camera.

是的。

Yeah.

我认为这是人们需要理解的第一件事。

I think that's one of the first things people need to understand.

它不像其他许多模型那样逐帧生成。

It's like, it's not you're not generating frame by frame, which is what a lot of the other models are.

人们知道大型语言模型会生成一个词元。

You know, people understand that an LLM generates one token.

那么，原子单位是什么？

What are, like, the atomic units?

有点像是，你知道的，网格。

There's kinda, like, know, the meshes.

还有体素的溅射。

There's, like, the splash of voxels.

三维世界中有许多组成部分。

There's a lot of pieces in a three d world.

人们应该对你们的生成方式形成怎样的心理模型呢？

What should be the mental model that people have of, like, your generations?

嗯。

Yeah.

我觉得现在存在的和未来可能存在的是两回事。

I I think there's, like, what exists today and what could exist in the future.

目前存在的是模型原生输出的点云。

So what exists today is the model natively outputs splats.

高斯点云就是这些小粒子，每个都是极其微小、半透明的，具有三维空间中的位置和方向，整个场景由大量这样的高斯点云构建而成。

So Gaussian splats are these, like you know, each one is a tiny, tiny particle that's semi transparent, has a position orientation in in three d space, and the scene is built up from a large number of these Gaussian splats.

高斯点云非常酷，因为你可以非常高效地实时渲染它们。

And Gaussian splats are really cool because you can render them in real time really efficiently.

你可以在iPhone上渲染，渲染一切东西。

So you can render on your iPhone, render render everything.

这正是我们能实现精确摄像机控制的原因，因为点云可以在几乎任何客户端设备上实时渲染。

And that's that's how we get that sort of precise camera control because splats can be rendered real time on just on pretty much any client side device that we want.

因此，对于我们今天生成的许多场景来说，这种基本单元就是单个的点云元素。

So for a lot of the scenes that we're generating today, that kind of atomic unit is that individual splat.

但我不认为这是根本性的。

But I don't think that's fundamental.

我可以想象未来可能出现其他有趣的方法。

I I could imagine other approaches in the future that would be interesting.

事实上，我们World Labs也研究过其他方法，比如我们最近的RTFM模型，它一次生成一帧。

So there, like, there are other approaches that even we've worked on at World Labs, like our recent RTFM model that does generate frames one at a time.

在这种情况下，基本单元是随着用户与系统交互，逐帧生成内容。

And there, the atomic unit is generating frames one at a time as the user interacts with the system.

或者你可以想象未来其他架构，其基本单元是一个令牌，这个令牌代表三维世界中的一块内容。

Or you could imagine other architectures in the future where the atomic unit is a token, where that token now represents, you know, some chunk of the three d world.

我认为随着时间推移，我们可以在这里探索多种不同的架构。

And I think there's a lot of different architectures that we can experiment with here over time.

我想再深入探讨一下这一点。

I do wanna press on double click on this a little bit.

我理解Alessio本来想说的是：世界模型的基本数据结构是什么？

My version of what Alessio was gonna say was, like, what is the fundamental data structure of a world model?

因为正如你所说，它要么是splat的成本，要么是某种框架之类的。

Because exactly, like you said, like, it's it's either cost of splat or it's like the framework, what have you.

你在之前的陈述中还特别强调了物理和力，这在一段时间内是松散相关的。

You also, in in the the the previous statements, focus a lot on the physics and the forces, which is something over time, which is loosely.

我在Marble中没看到这一点。

I don't see that in marble.

我猜这还没实现。

I presume it's not there yet.

也许如果有Marble，就会有运动了。

Maybe if there was, a marble too, you would have movement.

或者高斯splat有没有什么修改方式是合理的，还是说这会是完全不同的东西？

Or is there a modification to Gaussian splats that makes sense, or would it be something completely different?

是的。

Yeah.

我觉得有几种修改是有意义的。

I I think there's a couple modifications that make sense.

实际上，这里有很多有趣的方式可以整合这些内容，这也是在这个领域工作的一个很好的切入点。

And there there's actually a lot of interesting ways to integrate things here, which is another nice wet place of working in this space.

实际上，这方面已经有很多研究工作了。

Then there's actually been a lot of research work on this.

当你谈到一些疯狂的想法时，其实已经有很多非常有趣的学术研究，探讨了如何赋予物理特性。

Like, when you talk about wacky ideas, like, there's actually been a lot of really interesting academic work on different ways to imbue physics.

在工业界你也可以提出疯狂的想法。

You can also do wacky ideas in industry.

是的。

Yeah.

好吧。

Alright.

但高斯溅射本身就像是一个个小粒子。

But but it's then it's like Gaussian splats are themselves little particles.

已经有很多方法，基本上是将物理属性附加到这些点上，认为每个点都有质量，或者将每个点视为与附近邻居通过某种虚拟弹簧耦合。

There's been a lot of, approaches where you basically attach physical properties to those splats and say that each one has a mass or like maybe you treat each one as being coupled with some kind of, virtual spring to nearby neighbors.

现在你可以在点的基础上开始进行物理模拟。

Now you can start to do sort of physics simulation on top of splats.

因此，为这些点添加物理、动力学或交互的一种途径是，预测每个点粒子相关的物理属性，然后使用经典物理或某种学习方法进行下游模拟。

So one kind of avenue for adding, adding physics or dynamics or interaction to these things would be to, you know, predict physical properties associated with each of your splat particles and then simulate those downstream either using classical physics or something learned.

或者，你知道，在三维空间中工作的美妙之处在于事物可以组合，你可以在不同地方注入逻辑。

Or, you know, the the kind of the beauty of working in three d is things compose and you can inject logic in different places.

所以一种方式是我们生成一个三维场景，预测场景中所有事物的三维属性，然后使用经典物理引擎模拟它们之间的交互。

So one way is sort of like we're generating a three d scene, we're gonna predict three d properties of everything in the scene, then we use a classical physics engine to to simulate the interaction.

或者你可以做这样的事情：作为用户操作的结果，模型现在将重新生成整个场景，以点或其他形式表示。

Or you could do something where, like, as a result of a user action, the model is now going to regenerate the entire scene, in in in splats or some other representation.

这可能会更加通用，因为你不再受限于你已知如何建模的任何物理属性。

And that could potentially be a lot more general because then you're not bound to whatever sort of, you know, physical properties you know how to model already.

但这也对计算要求高得多，因为你需要在用户操作后重新生成整个场景。

But that's also a lot more computationally demanding because then you'd need to regenerate the whole scene in response to the to user actions.

但我认为这是一个非常有趣的研究方向，未来可以在此基础上进一步拓展，正如你所说，应用于Marble 2。

But I I think this is a this was a really interesting area for for future work and for, adding on to to into potential Marble two, as you say.

是的。

Yeah.

在动态效果方面还有很大潜力。

There's opportunity for dynamics.

没错。

Yeah.

对吧？

Right?

那么，Splat的密度目前是什么状况？

What's the state of, like, Splat's density, I guess?

我们能否在放大时实现很高的分辨率？

Like, do we can we render enough to have very high resolution when we zoom in?

我们是否受限于可生成和可渲染的数量？

Are we limited by, like, the amount that you can generate, the amount that we can render?

比如说，这些怎么才能达到极高的保真度呢？

Like, how are these gonna get super high fidelity, so to speak?

你确实有一些限制，但取决于你的目标使用场景。

You have some limitations, but depending on your target use case.

我们场景中的一个主要限制是，我们希望内容能在移动设备上清晰渲染，也希望能在VR头显上清晰渲染。

So, like, one of the one of the big constraints that we have on our scenes is we wanted things to render cleanly on mobile, and we wanted things to render cleanly in VR headsets.

这些设备的计算能力远低于你通常在其他场景中所拥有的能力。

So those are those devices have a lot less compute than you're used than you have in a lot of other situations.

如果你想让一个splat文件在四年前的iPhone上以高分辨率、30到60帧每秒的速度运行，那么你能处理的splat数量就会受到一定限制。

And, like, if you wanna get a splat file to render at high resolution, high like, 30 to 60 FPS on, like, an iPhone from four years ago, then you are a bit limited in, like, the number of splats that you can handle.

但如果你可以使用今年或近年的iPhone、最新的MacBook，或者拥有本地GPU，又或者不需要60帧每秒的1080p分辨率，那么你就可以放宽这些限制，支持更多的splat，从而在场景中实现更高的分辨率。

But if you're allowed to, like, work on a recent like, even this year's iPhone or, like, a recent MacBook or even if you have a local GPU or if you don't need if you don't need that 60 FPS ten eighty p, like, then you can relax the constraints and and get away with more splats, and that lets you get higher resolution in your scenes.

我原本期待但没听到的一个使用场景是

One use case I was expecting but didn't hear

具身化应用场景。

from you was embodied use cases.

嗯。

Mhmm.

你现在只专注于虚拟现实吗？

Are you you're just focusing on virtual for now?

如果你访问WordLab的主页，会有一个名为MarbleLabs的页面。

If you go to WordLab's home home page, there is a particular page called MarbleLabs.

在那里，我们展示了不同的应用场景。

There, we showcase different use cases.

我们将它们分为视觉效果、游戏应用以及仿真应用等类别。

And we actually organize them in more visual effect use cases or gaming use cases as well as simulation use cases.

其中，我们展示了这项技术在机器人训练中的巨大潜力。

And in that, we actually show this is a technology that can help a lot in robotic training.

这让我回到之前提到的内容。

This goes back to what I was talking about earlier.

说到数据匮乏，机器人训练确实严重缺乏数据。

Speaking of data starvation, robotic training really lack data.

高保真的真实世界数据至关重要，但你很难获得大量这样的数据。

High fidelity real world data is absolutely very critical, but you're just not going to get a ton of that.

当然，另一个极端是纯粹的互联网视频数据，但这样你就缺乏了训练具身智能体所需的可控性。

Of course, the other extreme is just purely internet video data, but then you lack a lot of the controllability that you want to train your embodied agents with.

因此，模拟和合成数据实际上是一个非常重要的中间地带。

So simulation and synthetic data is actually a very important middle ground.

在这方面，我已经在这个领域工作了多年。

For that, I've been working in this space for many years.

其中一个最大的痛点是，你从哪里获取这些合成的模拟数据？

And one of the biggest pain point is where do you get this synthetic simulated data?

你需要整理资产并构建、组合这些复杂的场景。

You have to curate assets and build these, compose these complex situations.

在机器人领域，我们需要大量的不同状态。

And in robotics, want a lot of different states.

你希望具身智能体能够在合成环境中进行交互。

You want the embodied agent to interact in the synthetic environment.

Marble 实际上在为具身智能体训练生成这些合成模拟世界方面具有巨大潜力。

Marble actually is a really potential for helping to generate these synthetic simulated worlds for embodied agent training.

当然，没错。

Obviously, that's yeah.

它就在主页上。

That's on the it's on the home page.

它会在那里的。

It'll be there.

我只是在想，正如你所说，你还得建立一个商业模式。

I just I I was, like, trying to make the link to as you said, like, you also have to build, a business model.

机器人市场的规模显然非常庞大。

The market for robotics, obviously, is very huge.

也许你不需要那样，或者我们首先需要构建并解决虚拟世界，然后才能进入具身领域，而这正是一个必经的步骤。

Maybe you don't need that or maybe we need to build up and solve the virtual worlds first before we can go to embodied and obviously That is exactly stepping stone.

这还有待决定。

That is to be decided.

我觉得确实如此

I I do think that

因为其他人都直接往那里去了。

Because everyone else is going straight there.

对吧？

Right?

不是所有人，但我觉得确实有一种兴奋感。

Not everyone else, but there is a there is an excitement, I would say.

但你知道，世界足够大，可以容纳不同的方法。

But, you know, I think the world is big enough to to have different Approaches.

是的。

Yeah.

方法。

Approaches.

是的。

Yeah.

我的意思是，我们一直将这视为一种相当通用的技术，未来应该能够触及许多不同的行业。

I mean and we always view this as a pretty horizontal technology that should be able to touch a lot of different industries over time.

而且，你知道，Marble 目前更专注于创意产业，但我认为支撑它的技术在未来应该能应用于许多不同的领域。

And, you know, marble is a little bit more focused on creative industries for now, but I think the the technology that powers it should be applicable to a lot of different things over time.

而机器人技术就是其中之一，你知道，可能比我们想象的更快到来。

And robotics is one that, you know, is maybe gonna happen sooner than later.

还有设计。

Also, design.

对吧？

Right?

它和创意产业非常接近。

It's very adjacent to creative.

哦，是的。

Oh, yeah.

当然。

Definitely.

比如，我觉得这跟建筑相关？

Like, I I I think it's like The architecture stuff?

是的。

Yes.

好的。

Okay.

对。

Yeah.

我的意思是，我之前在网上开玩笑。

I mean, I I I was joking online.

我在Slack上发了一个视频，问谁想用Marble来规划下一次厨房翻新？

I posted this, this video on Slack of, like, oh, who wants to use marble to to plan your next kitchen remodel?

它实际上已经非常适合这个用途了。

It actually works great for this already.

只需拍摄两张厨房的照片，用Marble重建它，然后使用编辑功能，看看如果更换台面、地板或橱柜，这个空间会是什么样子。

Just like take two images of your kitchen, like reconstruct it in marble, and then use the editing features to see what would that space look like if you change the countertops or change the floors or change the cabinets.

这并不是我们专门为这个用例开发的东西。

And this is something that's, you know, we didn't necessarily build anything specific for this use case.

但由于这是一种强大的通用技术，你往往会自然地涌现出这些意料之外的用例。

But because it's a it's a powerful horizontal technology, you kind of get these emergent use cases that that just fall out of the model.

我们有一些早期测试用户，他们已经使用API密钥在做室内设计的项目。

We have early beta users using a API key that is already building for interior design use case.

我刚弄了我的车库。

I just did my garage.

我早该知道的，我真得

I should have known about I I gotta

下次你装修时，我们可以

Next time you remodel, we can

没错。

Exactly.

帮上忙。

Be of help.

接下来肯定是厨房了，我敢肯定。

Well, kitchen is next, I'm sure.

是的。

Yeah.

是的。

Yeah.

我对整个空间智能领域很好奇。

I'm curious about the whole spatial intelligence space.

我觉得我们应该更深入地研究一下。

I think we should dig more into that.

首先，你怎么定义它？

One, how do you define it?

还有，比如，传统智能和人们想到大语言模型时的智能之间，有哪些差距？你知道，达里奥说我们有一个满是爱因斯坦的数据中心。

And, like, what are, like, the gaps between traditional intelligence that people might think about a LLMS when, you know, Dario says we have a data center full of Einstein's.

那才是传统智能。

That's like traditional intelligence.

这不是空间智能。

It's not spatial intelligence.

要具备空间智能需要什么？

What is required to be spatially intelligent?

首先，我不理解这句话——一个充满爱因斯坦的数据中心。

First of all, I don't understand that sentence, a data center full of Einstein's.

那是哦。

That's Oh.

我真的不理解这个。

I I just don't understand that.

不。

No.

不。

No.

不。

No.

这不是一个

It's not a

这是个类比。

deep It's an analogy.

这是个类比。

It's an analogy.

嗯，人工智能作为一个领域、一门学科，很大程度上是受人类智能启发的。

Well, so a lot of AI as a field, as a discipline, is inspired by human intelligence.

对吧？

Right?

因为我们是目前所知宇宙中最聪明的动物。

Because we are the most intelligent animal we know in the universe for now.

如果你观察人类智能，它是非常多元的。

And if you look at human intelligence, it's very multi intelligent.

有一位心理学家，我想他叫霍华德·加德纳，在20世纪60年代，他明确提出多元智能理论，用来描述人类的语言智能、空间智能、逻辑智能和情绪智能。

There is a psychologist, I think his name is Howard Gardner, in the 1960s, actually literally called multiple intelligence to describe human linguistic intelligence, there's spatial intelligence, there is logical intelligence, and emotional intelligence.

所以对我来说，当我想到空间智能时，我认为它与语言智能是互补的。

So for me, when I think about spatial intelligence, I see it as complementary to language intelligence.

因此，我个人不会说它是空间智能与传统智能之间的对立，因为我不确定。

So I personally would not say it's spatial versus traditional because I don't know.

‘传统’到底意味着什么？

Tradition means what does that mean.

我认为空间智能确实与语言智能是互补的。

I do think spatial is complementary to linguistic.

那么，我们如何定义空间智能呢？

And how do we define spatial intelligence?

它是一种使你能够推理、理解、移动和在空间中互动的能力。

It's the capability that allows you to reason, understand, move, and interact in space.

我用DNA结构的发现作为例子。

And I use this example of the deduction of DNA structure.

当然，我简化了这个故事，但其中很大一部分都涉及对分子和化学键在三维空间中的空间推理，最终推测出双螺旋结构。

And of course, I'm simplifying this story, but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in the three d space eventually conjecture a double helix.

人类，或者弗朗西斯·克里克和沃森所具备的这种能力，很难将其简化为纯粹的语言过程。

And that ability that humans, or Francis Crick and Watson, had done, it is very, very hard to reduce that process into pure language.

这正是文明关键时刻的巅峰体现。

And that's a pinnacle of a civilizational moment.

但每一天，我都在这里努力去抓起一个杯子。

But every day, right, I'm here trying to grasp a mug.

整个过程包括：看到杯子、观察它所处的环境、看到自己的手、让手的形状在几何上与杯子匹配，并触碰到合适的抓握点。

This whole process of seeing the mug, seeing the context where it is, seeing my own hand, opening of my hand that geometrically would match the mug and touching the right affordance point.

这一切都深深植根于空间性。

All this is deeply, deeply spatial.

这非常困难。

It's very hard.

我试图用语言来描述它，但另一方面，这种描述性语言本身并不能让你真正拿起一个杯子。

I'm trying to use language to narrate it, but on the other hand, that narrated language itself cannot get you to to pick up a mug.

是的。

Yeah.

带宽限制。

Bandwidth constraint.

是的。

Yes.

我最近做了一些计算，比如，如果你整天不停地说，每天24小时，会产生多少个标记？

I did some math recently on, like, if you just spoke all day, every day for twenty four hours a day, how many tokens do you generate?

以平均每分钟150个单词的语速计算，大约每天会产生215,000个标记。

At the average speaking rate of, like, a 150 words per minute, it rough running around rounds out to about 215,000 tokens per day.

而你所生活的这个世界，其带宽远高于这个水平。

And, like, your your your world that you live in is so much higher bandwidth than that.

我认为这是对的。

Well, I think that is true.

但当我想到艾萨克·牛顿爵士时，比如，在当时，像重力这样的概念还没有被语言形式化，但人们在空间上已经直观地理解了物体下落的现象。

But if I think about Sir Isaac Newton, right, it's like you have things like gravity at the time that have not been formalized in language that people inanaly spatially understand, that things fall.

对吧？

Right?

但以某种方式将这些内容形式化是有帮助的。

But then it's helpful to formalize that in some way.

或者像，我们用语言来真正捕捉那些在经验上和空间上你也能理解的各种规则，但用语言描述起来更容易。

Or like, you know, all these different rules that we use language to, like, really capture something that empirically and spatially you can also understand, but it's easier to, like, describe in a way.

所以我很好奇，空间智能和语言智能之间的相互作用，也就是，好吧。

So I'm curious, like, the interplay of, like, spatial and, like, linguistic intelligence, which is like, okay.

你需要理解。

You need to understand.

有些规则用语言表达比用空间智能理解更容易。

Some rules are easier to write in language for than the spatial intelligence to understand.

但你不能，你知道，你不能只是把手这样放，然后放下这么多。

But you cannot, you know, you cannot write, put your hand like this, and put it down this amount.

所以我一直很好奇，你们是如何相互利用的。

So I'm always curious about how you leverage each other together.

我的意思是，以牛顿为例，牛顿之所以想到写下那些定律，是因为他拥有大量的身体经验。

I mean, if if anything, like the example of of Newton, like, Newton only thinks to write down those laws because he's had a lot of embodied experience in the world.

对。

Right.

是的。

Yeah.

没错。

Exactly.

实际上，区分你提到的理论构建与那种身体化的、日常生活中嵌入三维世界的经验是有用的。

And, actually, it's useful to distinguish between the theory building that you're mentioning versus, like, the embodied like, the daily experience of being embedded in the three-dimensional world.

对吧？

Right?

所以对我来说，空间智能在某种程度上 encapsulates 了那种身处三维空间、在其中移动、观察和行动的身体化体验。

So so so so to me, spatial intelligence is sort of encapsulating that embodied experience of being there in three d space, moving through it, seeing it, actioning it.

正如费飞所说，你可以叙述这些事情，但这是一个非常损失信息的通道。

And as Fei Fei said, you can narrate those things, but it's a very lossy channel.

就像身处世界之中并在此行动的概念，与试图描述它完全是两种不同的模式。

It's just like the notion of, you know, being in the world and doing things in it is a very different modality from trying to describe it.

但因为我们人类是始终在空间中互动进化的动物，所以根本不会觉得这有什么难的。

But because we as humans are animals who have evolved interacting in space all the time, like, don't even think that that's an that's a hard thing.

对吧？

Right?

然后我们自然而然地转向语言和理论构建，作为超越这种原始空间理解的抽象机制。

And then we sort of naturally leap to language and then theory building as mechanisms to abstract above that sort of native spatial understanding.

从某种意义上说，大语言模型直接跳到了最高层次的抽象推理，这非常有趣且很有用。

And in some sense, LLMs have just, like, jumped all the way to those highest forms of abstracted reasoning, which is very interesting and very useful.

但空间智能几乎像是重新打开那个黑箱，说：也许我们直接走向完全抽象的语言、推理和交流形式时，已经失去了一些东西。

But spatial intelligence is almost like opening up that black box again and saying, maybe we've lost something by going straight to that fully abstracted form of of language and reasoning and communication.

作为视觉科学家，你知道，这挺有意思的。

You know, it's funny as a vision scientist.

对吧？

Right?

我总是觉得视觉被低估了，因为对人类来说，视觉是毫不费力的。

I always find that vision is underappreciated because it's effortless for humans.

你刚出生睁开眼睛时，就开始

You open your eyes as a baby, you start to

看到你的世界。

see your world.

与生俱来。

Born with it.

对吧？

Right?

我们几乎是天生就具备这种能力。

We're almost born with it.

但学习语言需要付出努力，包括学习如何书写、语法和表达。

But you have to put effort in learning language, including learning how to write, how to do grammar, how to express.

这使得它显得很难。

And that makes it feel hard.

而大自然花了更多时间优化的感知和空间智能，却被人类低估了。

Whereas something that nature spend way more time actually optimizing, which is perception and spatial intelligence, is underappreciated by humans.

有证据表明我们天生就具备这种能力吗？

Is there proof that we are born with it?

你说的是几乎天生具备。

You said you said almost born.

所以听起来我们实际上是出生后才学会的。

So it sounds like we actually do learn after we're born.

我们出生时，视觉敏锐度较低，感知能力会逐渐增强。

When we are born, our visual acuity is less, and our perceptual ability does increase.

但大多数人出生时就具备了视觉能力。

But we are most humans are born with the ability to see.

大多数人出生时就具备将感知与运动行为联系起来的能力。

And most humans are born with the ability to link perception with motor movements.

运动能力本身需要一段时间才能完善。

The motor movement itself takes a while to refine.

动物的能力则非常惊人。

And then animals are incredible.

今年夏天早些时候，我刚去过非洲。

I was just seeing Africa earlier this summer.

这些小动物一出生，几分钟内就必须开始行动。

These little animals, they're born and within minutes they have to get going.

否则，狮子就会抓到它们。

And otherwise, the lions will get them.

在自然界中，优化感知、空间智能和语言花了五亿四千万年，而语言发展的最慷慨估计大约是五十万年。

And in nature, it took five forty million years to optimize perception and spatial intelligence and language, the most generous estimation of language development is probably half a million years.

哇。

Wow.

是啊。

Yeah.

这比我想象的要长得多。

That's longer than I would have gonna say.

我已经很宽容了。

I'm being very generous.

是的。

Yeah.

是的。

Yeah.

不。

No.

我其实一直在翻你的书，我突然意识到，我们播客里讨论过的一个有趣关联是语言模型的基准测试，以及Winogrand是如何融入了这些需要空间智能的、看似不可能的物理情境的。

I I was, you know, sort of going through your book and I I was really realizing that that one of the in interesting links to something that we covered on the podcast is language model benchmarks and how do Winogrand actually put in all these, like, sort of physical impossibilities that require spatial intelligence.

对吧？

Right?

比如，a在b上面，因此a不可能穿过b，这对我们来说显而易见，但对语言模型来说，这种情况是可能发生的。

Like, a is on top of b, therefore, a cannot fall through b is obvious to us, but to a language model, it could happen.

我不确定。

I don't know.

也许这属于下一个词预测的一部分。

Maybe it's like a part of the, you know, the next token prediction.

这正是我想说的，关于解开这种抽象概念。

And that's sort of what I mean about, like, unwrapping this abstraction.

是的。

Yeah.

对吧？

Right?

如果你对世界的全部认知只是看到一个接一个的词序列，那就真的很难理解，为什么不行呢？

Like, if your whole model of the world is just like seeing sequences of words after each other, it's really kind of hard to like, why why not?

这其实不公平。

It's actually unfair.

对。

Right.

对。

Right.

但对我们来说，这显而易见的原因是我们内心将它映射回了我们熟悉的三维世界表示。

But then the reason it's obvious to us is because we are internally mapping it back to some three-dimensional representation of the world that we're familiar with.

这个问题是，我想说的是，从你的世界模型中提炼出语言模型到底有多难，需要多长时间？我用了‘提炼’这个词。

This the question is, I guess, like, how hard is it you know, how long is it gonna take us to distill from, like I I I used the word distill.

我不知道你是否同意这个说法。

I don't know if you agree with that.

因为我们需要我们的模型具备特殊的智能，所以要从你的世界模型中提炼出语言模型。

To distill from your world models into a language model because we do want our models to have special intelligence.

对吧？

Right?

我们是不是必须完全抛弃语言模型才能做到这一点？

Like and do we have to throw the language model out completely in order to to do that?

或者不用。

Or No.

不用。

No.

对吧？

Right?

是的。

Yeah.

我不这么认为。

Don't think so.

对吧？

Right?

我认为它们是多模态的。

I think they're multimodal.

我的意思是，即使我们现在的模型Marble，也是以语言作为输入的。

I mean, even our model, Marble, today takes language as an input.

对。

Right.

对吧？

Right?

所以它本质上是多模态的。

So it's deeply multi model.

我认为在许多应用场景中，这些模型会协同工作。

And I think in many use cases, these models will work together.

也许有一天我们会拥有一个通用模型。

Maybe one day we'll have a universal model.

我的意思是，即使你做到了，从实用角度来看，人们仍然会使用语言，也希望用语言与系统互动。

I mean, even if you do, there's sort of a pragmatic thing where people use language and people want to interact with systems using language.

从实际角度看，构建允许人们与之交谈的系统、产品和模型是有用的。

Even pragmatically, it's useful to build systems and build products and build models that let people talk to them.

所以我认为这种趋势不会消失。

So I don't see that going away.

我认为有一种智力上的好奇心，想知道究竟能在多大程度上构建一个仅使用视觉或仅使用空间智能的模型。

I think there's a sort of intellectual curiosity of of saying how, like, intellectually, how much could you build a model that that only uses vision or only uses spatial intelligence?

我不知道这是否具有实际用途，但我认为这是一个有趣的知识或学术探索，看看能将这种模型推到多远。

I don't know that that would be practically useful, but I think it'd be an interesting intellectual or academic exercise to see how far you could push that.

我认为，不提物理学了，但我很好奇，如果你有一个高度精确的世界模型，但不给它任何关于当前标准物理模型的理解，它能从零开始推导出多少内容，以及它需要多高的语言理解能力。

I think, I mean, not to bring it back to physics, but I'm curious, like, if you had a highly precise world model and you didn't give it any notion of, like, our current understanding of the standard model of physics, how much of it it will be able to come up with and, like, recreate from scratch, and what level of, like, language understanding it would need.

因为我们有太多符号体系，无法直接用它们来重建，但也许我们会提出一种完全不同的模型，同时仍然保持准确性。

Because we have so many notations that, like, we cannot use that, like, recreate it, but, like, maybe we'll come up with a very different model of it and still be accurate.

我想知道，我们是否在某种程度上受限于这样一个观念：人们总认为人类必须像人类一样，因为世界是为人类建造的。

And I wonder how much we're kinda limited by the high you know how people say human always need to be like humans because the world is built for humans.

某种程度上，我们构建语言的方式也限制了这些其他模态所能产生的输出。

And in a way, it's like the way we build language constrains some of the outputs that we can get from these other modalities as well.

所以我非常期待关注你们的工作。

So I'm super excited to follow your work.

是的。

Yeah.

我的意思是，还有另一种思路——实际上，你甚至不需要做人工智能就能回答这个问题。

I I mean, like, there's another engine I mean, you actually don't even need to be doing AI to answer that question.

你可以去发现外星生命，看看它们拥有什么样的物理规律。

You could discover aliens and see what kind of physics they have.

对。

Right.

对吧？

Right?

而且他们

And they

可能会有帮助，但费费说，到目前为止，我们是最聪明的。

might help But FeyFey said, we are so far the smartest So so far.

宇宙中的动物。

Animal in the universe.

对吧。

Right.

所以，那你认为呢？我的意思是，这确实是个很有趣的问题。

So so what do you so, I mean but that is a really interesting question.

对吧？

Right?

比如，我们对宇宙的认知和对物理学的理解，是否在某种程度上受到我们自身认知能力或技术演进路径依赖的限制？

Like, is our knowledge of the universe and our understanding of physics, is it constrained in some way by our own cognition or by the path dependence of our own technological evolution?

一种可以用来做实验的方式。

And one way to sort of and and, like, do an experiment.

简直想做个实验，看看如果我们重新运行人类文明，是否还会以同样的顺序得出相同的物理学？

Like, almost wanna do an experiment and say, like, if we were to rerun human civilization again, would we come up with the same physics in the same order?

我认为这并不是一个非常实际的实验

And I don't think that's a very practical practical experiment

来运行。

to run.

我知道一个实验，我想知道人们是否可以做：我们现在有大量的天体物理数据，关于行星或天体的运动。

You know, one experiment I wonder if people could run is that we have plenty of astrophysical data now on the planet or celestial body movements.

把这些数据输入模型，看看牛顿定律是否会浮现出来。

Just feed the data into a model and see if Newtonian law emerges.

我猜可能不会。

My guess is it probably won't.

我的猜测也是这样。

That's my guess.

不是的。

It's not.

牛顿定律的抽象层次与这些语言大模型所代表的层次不同。

The abstraction level of Newtonian law is at a different level from what these language LLMs represents.

因此，如果给足天体运动数据，我不会惊讶于大模型能相当准确地预测运动轨迹。

So I wouldn't be surprised that giving enough celestial movement data, an LLM would actually predict pretty accurate movement trajectories.

假设我发明了一颗围绕恒星运行的行星。

Let's say I invent a planet surrounding a star.

如果数据足够多，我的模型会告诉你第一天它在哪里，第二天它在哪里。

And given enough data, my model would tell you on day one where it is, day two where it is.

我不会感到惊讶。

I wouldn't be surprised.

但F等于MA，或作用等于反作用，那是完全不同的抽象层次。

But F equals MA, or action equals reaction, that's just a whole different abstraction level.

这超出了当今大模型的能力范围。

That's beyond just today's LLM.

好的。

Okay.

你需要什么样的模型才能避免地心说？

What model would you need to not have it be a geocentric model?

因为如果我只是用一些视觉数据来训练，你认为太阳绕着地球转是很合理的。

Because if I'm training just some visual data, it makes sense that you think the sun rotates around the earth.

对吧？

Right?

但显然，事实并非如此。

But obviously, that's not the case.

那么它该如何学会这一点呢？

So how would it learn that?

我对所有这些我们所讨论的力都很好奇。

Like, I'm curious about all these, like, you know, forces that we talk about.

有时候，也许你根本不需要它们，只要看起来对，那就是对的。

It's like sometimes maybe you don't need them because as long as it looks right, it's right.

但当你转向尝试用这些模型完成更高级的任务时，我们能多大程度上依赖它们呢？

But, like, as you make the jump to, like, trying to use these models to do more high level tasks, how much can we rely on them?

我认为你可能需要一种不同的学习范式。

I think you can need kind of a different learning paradigm.

对吧？

Right?

所以，这里出现了一点混淆，即把大语言模型、语言和符号，与人类的理论构建和人类物理学混为一谈。

So, like, you know, there's a bit of conflation here happening where saying, is it LLMs and language and symbols versus, you know, human theory building and human human physics?

它们非常不同，因为人类的目标函数是理解世界并在生活中茁壮成长。

And they're very different because an l l like, the the human objective function is to understand the world and thrive in your life.

而实现这一点的方式是，有时你观察数据，然后思考它。

And the way that you do that is by you know, sometimes you observe data, and then you think about it.

接着你尝试在现实中做点什么，但结果与你的预期不符。

And then you try to do something in the world, and it doesn't match your expectations.

然后你就想在线更新你对世界的理解。

And then you want to go and update sort of your your your your your understanding of the world online.

人们时时刻刻都在这样做。

And people do this all the time constantly.

比如，我觉得我的钥匙在楼下。

Like, whether it's, you know, I think my keys are downstairs.

于是我下楼去找，但没找到。

So I go downstairs and I look for them and I don't see them.

哦，不对，它们其实在我卧室里。

And, oh, no, they're actually up in my bedroom.

因为我们不断与世界互动，所以一直在构建关于周围世界发生什么的理论，然后证伪或补充这些理论的证据。

So we're we're like, because we're constantly interacting with the world, we're constantly having to build theories about what's happening in the world around us and then falsify or add evidence to those theories.

我认为，这种过程放大并扩展后，就形成了牛顿物理学中的 F=ma。

And I think that that kind of process writ large and scaled up is what gives us f equals m a in Newtonian physics.

我认为这与我们训练的模型模态——无论是语言还是空间——有点不相关。

And I think that's a little orthogonal to, you know, the modality of model that we're training, whether it's language or or or spatial even.

我的说法是，这几乎是一种更高效的学习方式：你根据现有数据提出假设，列出所有可能的世界，然后通过实验排除不可能的世界，最终锁定正确的那个。

The way I put it is almost like this is almost more efficient learning because you have a hypothesis of here are the different possible worlds that are granted by my available data, and you then you do experiments to eliminate the worlds that are not possible, and you resolve to the one that's right.

对我来说，这也是我拥有心理理论的方式，即我对你正在想什么有一些假设，我会尝试采取行动来验证或修正我对你的想法的直觉，当然，大语言模型并不会做这些。

To me, that's also how I also have theory of mind, which is, like, I'm I have a few thesis of what you're thinking, what you're thinking, and I try to create actions to resolve that or or check my intuition as to what you're thinking, you know, and and obviously, LLMs don't do any of these.

心理理论可能还会延伸到情感智能，而当今的人工智能根本还没有触及这一点。

A theory of mind possibly also will break into even emotional intelligence, which today's AI is really not touching at all.

对吧？

Right?

我们真的非常需要它。

And we really really need it.

你知道，人们开始可能过度依赖这些东西了，这本身就是一个话题，没错。

You know, people are starting to depend on these things probably too much, and and that's a that's a whole topic of Yeah.

另一个争论的话题。

Of other debate.

我得问一下，因为很多人把这个问题发给了我们。

I do have to ask because a lot of people have, like, sent this to us.

我们到底需要抛弃多少？

How much do we have to get rid of?