李飞飞谈空间智能与机器人

我们正在打造人工智能的下一个前沿，也就是我们所说的空间智能。

We are building the next frontier of AI, which is what we call spatial intelligence.

在SceniX，我们正在开发一种从现实到模拟再回到现实的全流程。

At SceniX, we are developing what we call a real to sim to real pipeline.

我们可以通过数字世界中可规模化生成的数据，替代现实中所需的所有数据和评估工作。

We can replace all the data, all the evaluation we need in the real environment by using the data that can generate at a scalable way in our digital world.

想想人类的智能。

Think about human intelligence.

我们在脑子里经常做模拟。

We do a lot of simulation in our head.

你知道为什么吗？

You know why?

模拟有一个真实数据无法替代的关键作用，那就是反事实推理。

There's a very important role simulation plays that real world data doesn't play, which is counterfactual reasoning.

我们正在构建一个始终如一的世界，无论在空间、时间、不同视角，还是不同类型的互动中，都保持一致。

What we are building is a consistent world, consistent both over space, over time, over different view points, and over different type of interactions.

我的目标很简单，就是让机器人真正能干活。

My North Star is I want the robot to work.

我们生活的这个世界可以是多重宇宙，我们正在开发技术，让普通人、开发者能在不同的空间里自由行动。

The world we live in can be multiverse, that we create technology to allow people, builders, developers to act within different spaces.

你觉得我们有朝一日能造出像人类一样节能的机器人吗？

Do you believe we'll ever be able to build robots that have the power efficiency of a human being?

离实现这个目标还有多远？

How far away are we from this?

是五年内能搞定，还是根本不可能？

Is this, like, five years or this is, like, never?

简而言之，

The TLDR is

语言模型彻底改变了AI理解文字的方式。

Language models transformed how AI understands words.

下一个前沿，是教会AI理解和作用于物理世界。

The next frontier is teaching AI to understand and act within the physical world.

在World Labs收购SceniX之后，马丁·卡萨多与李飞飞和李云珠深入探讨了这项收购背后的愿景。

Following World Labs' acquisition of SceniX, Martin Casado sits down with Fei Fei Li and Yunzhu Li to unpack the vision behind the deal.

他们讨论了空间智能、世界模型、模拟仿真，以及为何解决机器人问题需要一种全新的AI——它必须擅长三维推理，而不仅仅是处理语言。

They discuss spatial intelligence, world models, simulation, and why solving robotics will require a new generation of AI built for three-dimensional reasoning, not just language.

明白了。

Alright.

很高兴你们两位能来。

Well, it's great to have you both here.

所以，飞飞，对于可能不太了解的听众，你能简单介绍一下World Labs是做什么的吗？

So, Fei Fei, for the listeners that may not have the background, maybe you can give an overview of what World Labs does.

是的。

Yeah.

World Labs是一家成立两年的初创公司。

Well, World Labs is a two year old startup.

我觉得我们该明确一点，它是一家前沿模型实验室。

I think we should just recognize it's a frontier model lab.

我们正在打造人工智能的下一个前沿，也就是我们所说的空间智能。

We are building the next frontier of AI, which is what we call spatial intelligence.

空间智能就是让人工智能具备生成、理解、推理并与空间互动的能力，无论是物理空间还是虚拟空间。

And spatial intelligence is about creating AI that has the ability to generate, understand, reason with, and interact with spaces, whether it's physical or virtual.

而实现空间智能的一个关键路径，就是构建大型世界模型，这也是World Labs的核心方向。

And of course, a means to an end towards spatial intelligence is building large world models, and that's what World Labs is mostly focused on.

是的。

Yeah.

从一开始你就一直在说，机器要能感知和推理空间，并在空间中采取行动。

So you've been saying this since the very beginning, which is the machine's ability to perceive and reason about spaces and act on spaces.

对。

Mhmm.

但我一直以为，让机器在空间中行动，顶多是远程操控那种，没想到你们现在收购了一家机器人公司。

But I always had the assumption that the acting on spaces was some long distance shooter thing, but now you're acquiring a robotics company.

所以能不能聊聊，现在做这个决定的时机和初衷是什么？

And so maybe talk a little bit about the timeliness of this and the intentions.

是的。

Yeah.

首先，要在这个空间里行动或互动，光靠机器人是不够的。

So first of all, it doesn't just take robotics to act within spaces or to interact.

对吧？

Right?

你看创意领域，不管是视觉特效、游戏还是设计，都是如此。

I mean, look at the creative field, whether it's VFX or gaming and or design.

很多应用场景里，你都可以在虚拟空间中进行创造和互动。

Many use cases, you can create and act within virtual spaces.

World Labs 的核心理念一直就是：我们所处的世界可以是多元宇宙，我们开发技术，让普通人、创作者和开发者都能在不同空间中行动。

And World Labs' thesis has always been that the world we live in can be multiverse, that we create technology to allow people, builders, developers to act within different spaces.

不过，能够在物理空间中行动，确实是未来人工智能最令人兴奋、也最重要的能力之一。

Having said that, the ability to act within the physical space is one of the most exciting and most profoundly important capability of the future AI world.

所以，机器人技术正是这一点的体现。

So robotics is very much that.

所以World Labs一直认为，机器人技术是空间智能和世界建模的重要应用之一。

So World Labs has always believed that robotics is an important application as well as use case of spatial intelligence and world modeling.

因此，邀请SceniX及其团队加入World Labs，正是我们长期愿景和使命的一部分。

So by joining force with inviting SceniX and SceniX team to World Labs is part of our long term vision and mission.

我们始终坚定地致力于这一方向。

We've always committed to that.

太棒了。

Amazing.

云竹，你是SceniX的联合创始人，能不能给大家简单介绍一下你的背景以及SceniX是做什么的？

So Yunzhu, you're the co founder of SceniX, so maybe provide everyone with a quick overview of your background and what SceniX does.

是的，我是云竹，目前是SceniX的联合创始人，同时也在哥伦比亚大学担任助理教授。

Yeah, so I'm Yunzhu, so I'm currently co founder of SceniX and also assistant professor at Columbia University.

我的研究始于我在麻省理工学院的博士阶段，之后在李飞飞实验室做博士后。

So my research started from my PhD at MIT and then postdoc with Fei Fei

真的吗？

Really?

在斯坦福大学。

At the Stanford University.

太好了。

That's great.

世界真小。

The world is small.

确实很小。

Very small.

是啊。

It is.

在我的职业生涯中，我的目标一直很简单，就是帮助机器人更好地感知和与物理世界互动。

Throughout my career, my goal has been very simple, trying to help the robots better perceive and interact with the physical world.

我是个很务实的人。

So I'm a very practical person.

我希望我的机器人能在真实的物理环境中工作。

I want my robot to work in the real physical environments.

所以对于SceniX来说，我们看到的独特机会在于，目前通用机器人发展面临很多瓶颈。

So for SceniX, the unique opportunity we see is that there has been a lot of bottlenecks.

尤其是训练和评估这两个环节，现在都卡得很厉害。

Right now, we see faced by the developments of general purpose robots, especially around training and also around evaluations.

因此在SceniX，我们正在打造一个所谓的‘真实到模拟再到真实’的流程。

So at SceniX, we are developing what we call a real to sim to real pipeline.

我们把真实环境精准地映射到数字世界里，确保数字世界和现实高度一致。

We're about to map the real environments into the digital world that has the best alignments with the real environments.

所谓一致，就是数字世界里发生的任何事，现实里也会同步发生，这样我们就能用数字世界里可大规模生成的数据，完全替代现实中那些耗时耗力的数据采集和评估工作。

By alignments, we mean that whatever happens in the digital world is also gonna happen in the real environments, such that we can replace all the data, all the evaluation we need in the real environments by using the data that can generate at a scalable way in our digital world.

SceniX的一切，就是从这个想法开始的。

So that is how everything started in SceniX.

我们集结了一支在机器人、机器人学习、仿真和渲染领域都非常顶尖的团队，就是为了构建这套‘真实-模拟-真实’的系统，突破这些关键瓶颈。

We put together a very, very strong and best teams around robotics, robot learning, and also simulation and rendering, trying to build this real-to-sing to real stack to solve some of the key bottlenecks.

你们两个人能合作，真是令人惊叹。

It's amazing that you two work together.

是的。

Yeah.

这里有个有趣的事，你可能会以为，因为我们一直合作，他作为我出色的博士后，我们早就一直在讨论SceniX和World Labs的整合了。

And there is a funny story here because you would think because we worked together, he was my amazing postdoc, we've been talking about this SceniX and World Labs integration for a long time.

但其实并不是这样。

It's actually not true.

他们最初是以客户身份加入World Labs的。

They came into World Labs as a customer.

真的吗？

Really?

去年冬天，大概十一月、十二月的时候，我们发布了第一个生成式模型Marble，SceniX就是那时候注册使用的。

When we released the first version of our generative model called Marble last winter around November, December, SceniX just signed up.

真的吗？他当时是客户？

No kidding, he was a customer?

对。

I Yes.

根本不知道那是啥东西。

Didn't even know what it was.

后来我才意识到，这是云珠的公司。

And then I realized this is Yunzhu's company.

我就给云珠打了电话。

I called Yunzhu.

我当时就说，天啊。

I'm like, wow.

这可是你的公司啊。

This is your company.

然后我们发现，双方的契合度特别高。

And then we realized there's so much synergy.

也许，飞，你简单介绍一下 MARBL 是什么吧。

Maybe, Fei, if you just quickly describe what MARBL is.

对。

Yeah.

MARBL是World Lab一直在训练和迭代的基础模型代号。

MARBL is the code name for the base model that World Lab has been training and iterating on.

目前MARBL公开的核心能力是接收一个提示信息。

The fundamental capability right now of MARBL that is publicly released is to take a prompt.

它可以是图片，也可以是几张图片，或者文字，然后生成一个几何一致的三维世界，无论是用高斯泼溅还是网格来表示。

It can be an image, it can be a few images, a text, and turn that into a geometrically consistent world that can be represented in three d geometry, whether it's Gaussian splat or mesh.

SceniX团队真正想解决的是机器人领域一个特别棘手的问题——数据匮乏。

Really what SceniX team is doing is trying to solve this extremely difficult problem in robotics, which is the lack of data.

训练数据不足，评估数据也稀缺，这跟语言模型完全不同，语言模型在互联网上数据多得是。

The lack of data in training, the lack of data in evaluation, this is very, very different from language models where data is abundant on the internet.

我们知道，要让机器人真正发挥作用，就必须突破规模定律的限制。

And we know that in order for robotics to work, we have to somehow unlock the power of scaling law.

但这个突破口到底在哪？

But where does that come from?

这是整个机器人领域都在苦苦挣扎的一个根本性难题。

This is something that it's a profound problem that everybody's battling with in robotics.

聊聊这个能量其实挺好的。

It'd actually be great to talk about this energy.

你组建了一支非常非常优秀的人才团队。

You have put together a very, very talented team.

你组建了一支很出色的人才团队。

You have put together a very talented team.

那这两者之间有多少重叠呢？

And so to what extent is there overlap?

又有多少是延伸出来的呢？

To what extent is this an extension?

你可以稍微讲讲这方面的情况。

Maybe talk a little bit about that.

是的，确实非常互补。

Yeah, that's actually How complementary it is.

简而言之，两者高度互补，而且目标一致。

It's actually the TLDR is very complementary and with a shared mission.

云竹是三位技术联合创始人之一。

So Yunzhu is one of the three technical co founders.

另外两位是陈昌时，另一位哥伦比亚大学的教授，在仿真领域是世界级的技术专家。

The other two are Changshi Zhen, another Columbia professor who has been a world class technologist in simulation.

哦，真的吗。

Oh, wow.

昌时的背景也涉及视觉特效。

And Changshi has his background in also VFX.

他曾在维塔工作室工作过。

He worked at Weta.

也待过腾讯。

Worked at Tencent.

他还是个创业者。

He's been an entrepreneur.

还有孙尼·胡，一位非常出色的工程领导者，多年前曾在一个被亚马逊收购的初创公司工作。

Then there's Sonny Hu, who is a phenomenal engineering leader who was also in a startup that was acquired by Amazon many years ago.

他在亚马逊的计算机视觉领域参与过很多不同的技术栈。

So he worked in many different tech stacks in the computer vision field in Amazon.

当我们开始认真聊的时候，我发现SceniX在人才方面和World Labs简直是完美互补。

So when we started talking more seriously, I recognized that a couple of things that SceniX has from a talent point of view is extremely complementary to World Labs.

首先是云竹在机器人领域的卓越领导力和顶尖技术能力。

One is obviously Yunzhu's incredible thought leadership and just technical prowess in robotics.

对吧？

Right?

没错。

So Sure.

从硬件到全栈机器人，他都精通。

From really, from hardware, full stack robotics.

就连他在我斯坦福做博士后的时候，那时候你已经拿到教职邀请了。

And even when he was my postdoc at Stanford, at that time, you already had your faculty offer.

所以你只待了一年。

So you were there only for one year.

我本来想留你更久，但他得去干正经工作了。

I wanted you for more than one year, but he had to go become a have the real job.

他在机器人领域是全栈型研究员，从建模到硬件都精通。

So he was a full stack researcher in in robotics from modeling to to hardware.

当然，云竹和他在SceniX带的学生，正是World Labs当时还欠缺的那批人才。

And, of course, Yunzhu and his student students at SceniX was that pool of talent World Labs hasn't had yet.

另外，他们在仿真能力方面也特别出色。

Then on the side is just incredible simulation capability.

对吧？

Right?

他在仿真领域是个资深的研究员和技术专家。

He's such a senior researcher and technologist in simulation.

World Labs正在做的，正是把仿真世界和现实世界连接起来。

What World Labs is doing is very much interfacing the world of simulation.

所以我觉得，他们欠缺的，还有生成模型和三维计算机视觉重建方面的能力，而这些正是World Labs的强项。

So I think what they don't have, obviously, is on the generative model side as well as the computer vision three d reconstruction side, we're also very strong at World Labs.

所以这是SceniX需要的技术。

So that's a technology that SceniX needs.

这样一来，这两方面结合起来，整个体系就完整多了。

So together, these two sides come together and make it much more complete.

Fei Fei这么做的动机，其实是想把机器人领域进一步拓展和补充。

Fei Fei's motivation in this is, like, this is an extension and a compliment to get into robotics.

你经历过类似的情况——决定什么时候卖掉公司，我很想听听你对加入World Labs的看法，你觉得双方契合在哪里，为什么最终做出了这个决定。

You know, having been in your situation, which is deciding when to sell a company, would be great to hear from you on how you think about joining World Labs and kind of the fit there and why you made the decision to do it.

是的，一开始我们还在犹豫，到底要不要继续独立发展。

Yeah, so at the very beginning, we were deciding, okay, do you want to just keep going?

但和Fei Fei聊过之后，看到中间这么多协同效应，合并就变得顺理成章了。

But after chatting with Fei Fei, after seeing all the synergies that are happening in the middle, it just makes perfect sense for the forces to join each other.

在SceniX，我们一直做的是从真实到逼真的方向，也就是对环境做高密度的三维重建。

So in any sense, at SceniX, what we have been doing is real to seem to real, is to do dense reconstruction of the environment.

我们捕捉环境的外观、几何结构，还有它的动态特性——也就是当你施加动作时，环境会如何变化。

So we capture the appearance of the environment, geometry of the environment, and also the dynamics of the environment, meaning how the environment is going to change when you apply actions.

目前这种密集重建还是有点太重了。

So this dense reconstruction right now is still a little bit on the heavier side.

而World Labs目前主要擅长的是稀疏重建和生成方面的强大能力。

And what World Labs right now has been doing involves a lot of profound capabilities around sparse reconstruction and generations.

所以我们看到很多机会，可以借助World Labs的Marble等技术，更高效地重建和建模环境。

So we see a lot of opportunities of leveraging like a marble and other like capabilities at World Labs in order to do very efficient reconstructions and modeling of the environment.

我们能期待World Labs推出一个机器人领域的基础模型吗？

Can we expect a foundation model for robotics from World Labs?

World Labs确实在构建一个基础模型，Martin，你知道的。

World Lab is building a foundation model, as you know, Martin.

我们正在打造一个底层模型。

We're building a base model.

随着技术发展，最令人兴奋的基础模型之一就是全能型模型。

And as the technology has been evolving, some of the most exciting base models are omni models.

对吧？

Right?

它们能接收多种模态的输入。

They take multimodal input.

它们也能输出多种模态的结果。

They have multimodal outputs.

那机器人领域的基础模型到底是什么？

And what is a foundation model for robotics?

它很可能涉及各种动作。

It's very likely gonna involve actions.

它很可能不仅要输出世界的状态，还要输出动作本身，我们绝对没有排除这种可能性。

It's very likely gonna involve the output of actions in addition to the state of the world, and we're definitely not ruling this out.

是的，说得很好。

Yeah, great.

比如，基础模型本质上必须是一个多模态模型。

So for example, for the foundation models, it essentially needs to be a multimodal model.

它需要能处理文本、图像、深度信息，以及其他各种模态的数据。

So it has to take into account from text, image, depths, and different kind of modalities.

动作是这些模态中非常关键的一部分。

And action is a very, very important parts of that modalities.

如果你把动作当作输入，它实际上能帮助建立相似性。

So if you think about frame actions as an input, that essentially affords similarity.

它能预测当你执行某个动作时，环境会如何变化。

That is going to predict how the environment is going to change when you apply a specific action.

当动作作为输出时，它本质上是一个策略模型，旨在根据特定目标，预测在真实环境中该采取什么动作才能更接近目标。

When the action is output, it's essentially a policy model that is trying to predict, give you a specific goal, like what should be the action you take in the real environment to get you closer to that goal.

这种全能型模型能为机器人领域带来巨大价值，帮助人们更好地理解如何建模环境，以及如何在环境中采取行动。

So this kind of omni models actually can benefit a lot and actually provide huge amount of values for the robotics communities in trying to understand how to model the environments and at the same time, how to act in the environments.

它还能作为基础框架，让你针对具体的机器人应用进行微调，确保真正达到客户所期待的可靠性和效率。

And this can also act as a backbone for you to fine tune into specific robotic applications to making sure it's really live up to the reliability and efficiency that expected by the clients.

云竹，如果不妨碍的话，我想问个外行投资者的问题：我看到很多机器人公司，现在最流行的做法是用视频模型，你知道的，这几乎是主流方法，而这里讲的是三维和仿真，完全是另一种思路。

Yunzhu, if you don't mind kind of a lay investor question, I see a lot of robotics companies, And a very popular approach right now for the robotics companies that come in is like we'll use a video model, you know, and like, you know, that's the predominant method where this is, you know, three d and simulation, it's a very different approach.

所以你能对比一下，这种只用视频的主流方法，和这里所追求的目标之间有什么不同吗？

And so maybe you could contrast this popular approach of just using video only versus kind of what the ambition here is.

是的，为了让机器人能理解这些动作，正如我之前说的，这些动作必须准确反映问题的核心结构。

Yeah, so in order to create words with the robot canner, the words, as I mentioned, need to capture the essential structure of the problem.

而这些动作的一个关键要求就是一致性。

And one of the very important necessary requirements for those words will be consistency.

这正是我觉得我们和Marble项目高度契合的地方——我们构建的是一个在空间、时间、不同视角以及各种交互中都保持一致的世界。

So that is where I actually see there's very, very strong synergies with Marble because what we are building is a consistent world, consistent both over space, over time, over different viewpoints, and over different type of interactions.

Marble生成的动作数据，正是我们所认为的机器人系统中不可或缺的一部分。

And Marble, the generated words from Marble, is also provide an infrastructure, a component of that entire words that we believe is necessary for the robot turner.

想象一下，如果机器人推一个物体，结果物体突然凭空消失——这在很多现有的视频预测模型里都是个问题。

Imagine if a robot is pushing an object forwards, the object just magically disappear, which has been a problem on many of the existing video prediction models.

但即便如此，这种表现已经足够给机器人提供有用的信号，让它知道该怎么做。

That one provides good enough signal for the robot to know what is the right thing to do.

不过，目前确实有很多研究在不断改进视频模型，让它们越来越强大。

But, obviously, right now, there has been a lot of investigation on building better and better and stronger and stronger video models.

所以我们发现，我们搭建的这套基础设施，可以为整个系统提供初始的动力。

So we actually see a way where some of the infrastructure we build can provide as initial momentums.

通过这个数据飞轮，从仿真驱动的模型过渡到机器人策略模型，让机器人在真实环境中执行任务、收集新数据，再把数据反馈回来。模型不必非得是纯物理的，也不必是纯学习的，而是可以介于两者之间——既能抓住问题的本质结构，又能随着数据积累不断优化和提升。

And to go in through this data flywheel of going from this, like, a more simulation driven models into, like, a robot policy models, which gonna do the execution in the real environment, collecting new data, the data will come back in, where the model doesn't necessarily have to be physics only or learning only, but somewhere in the middle, we should be able to capture the essential structure of the problem, but at the same time, be able to scale and become better and better as you accumulate more data.

你知道吗，我跟费飞合作很久了，你一直有个明确的方向，推动着这一切，你曾用不同的方式把它表述为‘三D’之类的概念。

You know, I've worked now, Fei Fei, very closely for a while, and you've always had this North Star, which has driven this, and you've articulated variously as kind of three d and in a number of other ways.

我想问问你，对你来说，是否也存在类似的哲学指引？还是说你更偏向务实，就像我一样？

And I'm just wondering, for you, is there also a similar philosophical North Star, or you're more of the pragmatic, like I am?

就是动手做，把系统搭起来，先把事情干起来。

Like build the system, like do the thing.

我的目标就是让机器人能在真实世界里真正运行起来。

My North Star is to make robots work in the real environment.

我是个非常务实的人。

I'm a very practical person.

我只关心机器人能不能真正工作。

I want the robot to work.

有一件有意思的事，来自我和费李在博士后期间的合作，我们当时在构建这样一个评估基准。

One interesting thing that's actually coming from my collaborations with Fei Li during our postdoc, we are building this kind of benchmark.

我们实际上会发问卷，问普通大众希望机器人帮他们做些什么。

We actually send out surveys asking the general public what they want the robots to do for them.

在我们收集的一千多项任务中，有三分之一是打扫卫生。

Among the thousand tasks we collected, one third of the tasks are bot clean.

人们就是不喜欢做这种又枯燥又脏的活儿。

People just don't like to do those, like, dull and dirty tasks.

这些正是我们特别希望用机器人来解决的场景。

And those are the scenarios that we really want to making sure we have robotic solutions to deal with.

马丁，我特别欣赏SceniX的一点，尤其是顺着你刚才的问题，现在有很多机器人公司都在做模型之类的。

One thing I really like about SceniX, Martin, especially continuing your question, there's a lot of robotics companies building models and all that.

我真正欣赏SceniX的是，云竹和他的联合创始人对机器人技术有着极其务实的态度。

One thing I truly like about SceniX is Yunzhu and his cofounders have such an incredibly pragmatic approach to robotics.

尤其是，他们都是从学术界出来的。

Especially, they come from academia.

对吧？

Right?

索尼没有，但云竹和张曦都是学术背景出身。

Sony doesn't, but Yunzhu and Zhang Xi come from academia.

但他们第一反应总是和实际行业的设计伙伴和客户合作，不管是工业实验室还是仓库。

But their first instinct is work with design partners and customers in real industry, whether it's industry labs or warehouses

电子组装。

Electronics assembly.

电子组装。

Electronics assembly.

这种做机器人的方式真的让人耳目一新。

That is such a refreshing, actually, a refreshing way of approaching robotics.

这让我特别期待能和他们合作。

And that that really true made me very excited to work with them.

宝贝，这话是说给云竹的，但我只是出于个人好奇——我觉得做机器人必须得非常精确。

Babe, this is for Yunzhu, but I I just feel this is this is personal curiosity, which is it seems to me that for robotics, you got to be pretty exact.

不是要求完美，但得接近完美；但像World Labs做的那些创意应用场景，其实没必要那么精确，因为有时候，出错本身反而是一种风格或者有意为之。从技术角度来说，你怎么看待这两种需求之间的平衡问题？

I mean, not perfect, but pretty close, but for the creative use cases, which World Labs has done a lot of, you kinda don't need to because, you know, I mean, you know, even sometimes like being wrong is stylistic or intentional or whatever, and so from a technical perspective, what is the challenge here for reconciling these two things, or

它们永远无法调和吗？

do they never get reconciled?

设计空间里总会存在两种不同的方向。

Like, there'll always be two points in the design space.

从长远来看，它们当然会慢慢融合。

So they will be like reconciled in the long terms, of course.

对环境的建模不需要做到完美。

And modeling of the environments doesn't have to be perfect.

在机器人领域，模型也不必完美无缺。

The model doesn't have to be perfect in robotics.

顺便问一下，这纯粹是出于好奇，有没有更正式的说法？

And by the way, there again, this is pure curiosity, but is there like a bit more formal way to say that?

比如说，‘不完美’具体是指什么？

Like, what does that mean not to be perfect?

但得差不多才行。

It has to be pretty close.

让我这么来说吧。

So let me put it in this way.

比如，针对各种机器人应用的建模，一直是至关重要的基础。

For example, models over the developments of all different kinds of robotic applications has been a very important cornerstones.

看看现有的所有机器人应用——飞机、无人机、扫地机器人，甚至四足和双足机器人，它们能工作、能从仿真迁移到真实环境，靠的都是模型。

If you look at all the existing robotic applications, like Plane, drones, Roomba, or even for quadruped robots, bipedal robots, model has been the way for them to actually work and be able to transfer from simulation to the real environments.

我明白了。

I see.

但你看那些运动型机器人，比如四足或双足机器人，它们能在雪地、灌木丛里走，却根本不需要仿真器。

But if you look at those locomotion robots, like quadruped robots, bipedal robots, they can walk in on snows, they can walk in on bushes, but you don't need to have a simulator.

有意思。

Interesting.

它们能非常精准地模拟出灌木和积雪的环境。

They can simulate all the bushes and snows very precisely.

你不需要一个完美还原所有细节的仿真，而是要抓住问题的核心结构，在数字环境中做各种随机变化。

You need to have a simulation that captures the essential structure of the problem and do a whole different kind of randomizations inside the digital environment.

这正是我们想要达到的目标。

So that is what we're aiming for.

所以，借助 SceniX 和 World Labs 的合作，我们正在探索：除了机器人本身，要模拟如此庞大的世界，到底需要多高的真实度。

So basically with SceniX and together with World Labs, we're trying to investigate what is the level of fidelity we need to model the massive massive worlds besides the robots.

正如我所说，我们能够把在模拟环境和数字世界中训练好的机器人系统，成功迁移到真实场景中。

As I said, we'll be able to transfer the robotic systems, training the simulated environment and digital worlds back into the real scenarios.

作为投资人，我听过其他研究者，比如谢尔盖·莱文，说模拟终究会和现实世界产生偏差，真实数据的收集才是至关重要的。

As an investor, I've heard other researchers say like Sergei Levine say simulation will always eventually deviate from the physical world and real world data collection is absolutely critical.

那你能谈谈这种以模拟为核心的方法，相比其他方式，到底靠不靠谱吗？

So maybe talk a little bit about like the viability of this approach where simulation is a cornerstone as opposed to some other approach?

其实这两者并不矛盾。

So they don't contradict with each other.

如果你想想模拟的本质，它其实就是预测：当你采取某个动作时，环境会如何变化。

So if you're thinking about the simulation, simulation is essentially trying to predict how the environment's going to change when you apply the actions.

这本质上就是在构建一个世界的模型。

And this is essentially a model of the world.

不一定非得是纯粹的物理模拟。

It doesn't necessarily have to be pure physics.

可以是物理规则和学习方法的结合。

It can be a combination between both physics and also learning.

我们正在收集真实世界的数据。

We are collecting real world data.

我们会用这些真实数据。

We will be using those real world data.

这在不同阶段就像一个数据飞轮。

It's just at different stages of this, like a data flywheel.

一开始，我们会更侧重物理模型，确保有正确的一致性和结构，以便学习世界规律、训练机器人策略。

Maybe at the very beginning, we have stronger emphasize on, we have more physics to making sure we have the right consistency and right structure for us to learn the world for us to train the robot policies.

但随着我们不断积累数据，包括通过数据采集和与客户的合作，数据会逐渐转向更多依赖学习来建模环境。

But as we accumulate more and more data, both through data collection and also through the collaboration with our clients, we'll have the data that will be moving towards more learning based, like modeling of the environments.

这种转变，还有这种数据飞轮机制，正是我们能同时发挥物理与几何一致性优势，以及数据和算力强大潜力的关键所在。

So this kind of transition and also this kind of data fly on is really an enabling factors of both getting the best of both physics and the geometry and consistency, as well as all the power and magics from the data and compute.

我想补充一点，从哲学角度来说，仿真和不仿真并不是非此即彼的选择。

I wanna add to this and be slightly philosophical here, is there isn't a binary choice between simulation or no simulation.

所有这些因素共同作用，才能让机器人真正运转起来。

All this come in together to to make robotics work.

想想人类的智能。

Think about human intelligence.

我们在脑子里经常做模拟。

We do a lot of simulation in our head.

你懂的。

You know?

为什么呢？

Why?

仿真有一个真实数据无法替代的重要作用，那就是反事实推理。

There's a very important role simulation plays that real world data doesn't play, which is counterfactual reasoning.

也就是设想那些还没发生、不可能发生，或者现实中数据不足无法实现的情景。

It's that you play out events that hasn't happened or cannot happen or you don't have enough data to make it happen in real world.

而在你模拟的过程中，你也在学习如何应对各种情况。

And while while you play it out, you learn how to act in it.

人类时时刻刻都在这么做。

Humans do this all the time.

我们可能并不总是意识到，但我知道你去过世界杯现场。

We probably don't you know, we just I know you were at the World Cups.

我确实去过

I was at the

世界杯。

World Cups.

西班牙队在每场比赛的赛前规划中，肯定都用了模拟，不管是用数字工具、白板，还是别的什么方式。

Congratulations to I'm Spain sure in the planning of every game, there is simulation, whether it's digital or on the on the whiteboard or whatever.

这种模拟的核心作用，就是进行反事实推理。

That simulation, the role simulation plays is counterfactual reasoning.

这对机器人技术特别重要，因为我们根本不可能获得足够多的真实世界数据。

And that's really important in robotics because we just do not have cannot possibly have enough real world data for that.

举个现实生活中的例子，就是自动驾驶汽车这个行业。

Here's a real life example, the industry of self driving cars.

Waymo已经公开表示，他们使用了数十亿小时的模拟数据。

Waymo has officially said they use billions of hours of simulation.

实际上，Waymo更依赖模拟数据，而不是现实世界的采集数据。

And actually, Waymo is more simulation heavy than just real world data heavy.

这些都是实实在在的例子。

So these are real examples.

你知道的，马丁和安德鲁也都知道，汽车其实是 simplest 的机器人。

And as you know, Martin and Andrew too, cars are the simplest kind of robots.

是的。

Yeah.

TD。

TD.

是的。

Yeah.

是的

Yeah.

所以

So

显然，模拟在机器人学习中起着至关重要的作用。

clearly, simulation plays a huge role in robotic learning.

我还想补充一点。

I also want to add to that.

如果具体来说，模拟能带来两个层面的好处。

So like there are, if you put things more specific, simulation can provide two levels of benefits.

一个是可靠性，另一个是效率。

The first one is reliability, and the second one is efficiency.

关于可靠性，如果你希望机器人系统在真实环境中稳定运行，就需要数据全面覆盖所有可能的状态和变化情况。

So for reliability, if you're thinking about a robotic system working reliable in the real environments, you need data to provide systematic coverage of all the state space and the variations that robots might encounter.

这样才能让系统学会如何应对各种复杂情况。

That's how you can learn of how this data is robust.

通过模拟，你可以系统性地随机调整光照、摩擦力、几何结构、物体类型，以及各种物理参数，确保覆盖到状态空间的方方面面。

So with simulation, you can do systematic randomizations and control and variations of lighting, frictions, geometries, object types, and also all different kind of physical parameters to making sure you have sufficient coverage of the state space.

这正是让机器人系统具备可靠性的关键。

So this is what can give the robotic systems reliability.

第二点是效率。

And second is about efficiency.

现在很多人在用遥操作。

So right now, many people are doing teleoperation.

如果你看看那些遥操作设备，想想你所依赖的那些机械结构，实际上你采集数据的速度，比真人亲自做任务还要慢。

And if you look at many of the teleoperation device, imagining all the actual skeletons you are using, you're actually collecting the data at a speed that is actually slower than human actually doing the task.

但对我们很多客户来说，人类的速度根本不够用。

But for many of our clients, human speed to them is not good enough.

他们想要的是比人类更快的速度。

They want faster than human speeds.

让机器人动得更快，并不是简单地加大驱动力度就行，因为重力是不会变的。

So for the robot to move faster, it's not as simple as just drive the robot faster because the gravity doesn't change.

但在模拟环境中，你可以系统性地加快机器人的行为速度，让机器人充分学习各种动力学特性和环境变化。

But in simulation, you can do systematic speed up of the robot's behaviors to train the robots such that it considers all the dynamics, changes of the environment.

所以这一点能真正帮助我们的客户提升效率。

So this is what can give our clients for them efficiency.

无论是可靠性还是效率，模拟都能带来一些独特的优势。

So both for the reliability and efficiency, though there are some kind of very unique values where simulation can provide.

你刚才讲了技术平台的功能，能不能具体说说大家通常用它来做什么？

You've talked about the technology and the platform, what it does, maybe talk about the specific use cases people use it for.

主要有两个典型应用场景，分别集中在训练和评估这两个方面。

There are essential, like two specific use cases, especially around both training and also around evaluations.

我们先说评估。

Starting from the evaluations.

评估这个环节，很多人在机器人领域都忽略了，但如果你在训练机器人模型，就必须清楚它表现到底怎么样——这可是你优化迭代的唯一依据。

So evaluation is something like people often overlooked in the robotics, but if you are training like robotic models, you have to know how well it works, and that is the only source of information for you to iterate

你的分析业务。

your analysis business.

很多搞AI的人都很清楚评估到底是什么，而且天天都在用。

A lot of every AI person really understands what devals are and uses it all the time.

但对非AI领域的人来说，这个词的意思可能不太一样，所以也许有必要具体说明一下你所说的评估指的是什么。

Non AI people, it often means something a little different, so maybe it's even worth just describing specifically what you mean by evaluation.

我说的评估，就是针对某个特定的检查点，看看它的表现到底怎么样。

Okay, so what I mean by evaluation is you'll be able to understand for this specific checkpoints, how well does it perform?

比如，它有95%的时间表现正常，还是99.9%？

Does it perform for example 95% of the time or 99.9% of the time?

明白了。

I see.

业界最看重的关键指标就是耗时多久。

And the key criteria people use in industry is how long does it take?

你要花多少计算时间，才能分辨出90%表现的检查点和92%表现的检查点之间的差别？

How long in workload time does it take for you to distinguish between a checkpoint that is 90% from a checkpoint that is 92 points?

如果只在真实环境中做这种对比，那耗时实在太长了。

And if you only do that in the real environment, it's just so long for you to do the distinguishments.

而且如果你仔细想想现在机器人在真实环境中做的评估，迭代速度比语言模型慢了几个数量级。

And if you really think about also the robotic evaluations, right now people are doing in the real environments, the iteration speeds is multiple orders or magnitude slower than iterations of those language models.

不仅机器人任务种类繁多、差异巨大，

So not only is like the robotic tasks very varied, very diverse

因为得实际动手去做才行。

Oh, because you have to do the thing.

对的。

Yeah.

没错。

Yeah.

没错。

Right.

确实如此。

Right.

就像让原子在空间中移动一样。

Like atoms to move through space.

对。

Yes.

没错。

Exactly.

但物理定律必须得遵守，你得盯着看。

But only there are The laws of physics have to And be have you watch

那些机器人视频，每个都得放慢十倍、八倍，因为它动得太慢了。

those robotics videos, every video has like 10x, 8x, because it moves so slowly.

没错。

Exactly.

不仅慢，还危险、昂贵，而且速度差了几个数量级。

So not only it's slow, it's dangerous, it's costly, but at the same time, the speed is also like multiple hours of magnitude slower.

我们的有些客户确实需要这样一个数字环境，用来评估他们的机器人系统。

So some of our clients actually needs this digital environment that can be used to evaluate their robotic systems.

因为我们的数字环境已经验证过和真实世界高度一致，也就是说，场景里发生的任何事，也很可能在真实环境中发生。

And because our digital environment has proven alignments with the real world, so meaning whatever happens in the scene is also likely to happen in the real environment.

如果某个检查点在模拟中表现更好，那么它在真实环境中也很可能表现得更好，这一点我们在博客文章里也讨论过。

If a checkpoint is working better in the simulation, it's also highly likely to also work better in the real environments, as we have also been discussed in the blog post.

这给了我们的客户极大的信心，让他们愿意用数字环境中的数据和信号，来对机器人系统进行可扩展、安全且快得多的评估。

So that actually give our clients very strong confidence in actually using the data, using the signal from the digital environment to do scalable, safe, and much faster evaluations of their robotic systems.

很好。

Great.

这是关于评估的部分。

So that is on the evaluation.

接下来是训练方面。

Then on the training.

在训练这边，就像我之前说的，关键在于可控性。

So on the training side, so basically, like I also mentioned, it's about controllability.

你需要能控制所有可能的状态变化、参数、光线、摩擦力、物理属性，甚至物体的形状和类型。

It's around you control all the different possible variations of states, parameters, lighting, frictions, physical parameters, like even object geometry, object types.

你要确保覆盖各种各样的场景，这样才能生成有足够信息量的数据，让机器人的表现更稳健。

So you want to making sure you have sufficient coverage of all different kind of scenarios such that we'll be able to generate informative data for your robots to be robust.

在真实环境中想做到这一点，难度实在太大了。

And this is just gonna be so hard to do just in the real environments.

就像我们之前说的，如果你用远程操控，数据采集的速度会很慢。

Like we discussed, if you do teleoperation, the speed at which you are collecting data is slow.

而且你还受限于有多少台设备、有多少个操控终端可用。

You're also limited by like, how many routers you have, how many teleoperation device you have.

围绕这些数据操作，还有一大堆别的挑战。

There's, like, a whole different kind of, like, challenges around all the data operations around it.

但在仿真环境中，所有东西都能掌控，都能系统化，你甚至能清楚地知道覆盖了哪些分布，做出精确的声明。

But in simulation, everything can be controllable, everything can be systematic, and everything can be understand at a at a level where you know exactly and making claims about exactly what distribution you have covered.

没错。

Yeah.

这样才能建立信心，知道机器人在已知范围内一定能正常工作。

To develop confidence about within the distribution, we know the robot will work.

这种信心、效率和可扩展性，正是我们的客户看重的，所以他们愿意用我们的数字环境来训练机器人系统。

So those kind of confidence and efficiency and scalability is something that our clients also value to use our, like, digital words for the training of robotic systems.

最离谱的是。

Here's the crazy thing.

甚至在 SceniX 和我们谈之前，我们 Marble 的潜在客户就已经有这种需求了。

Even before SceniX and we are talking, our inbound customers for Marble were already seeing this kind of demands.

我们根本忙不过来，但已经接到很多机器人初创公司的电话，这些公司从模型开发一直做到下游非常落地的应用场景，我们确实看到了这些需求。

We just cannot serve these customers, but we are already getting a lot of phone calls from robotics, early stage robotics companies who are developing their models all the way to downstream, very pragmatic use cases, and we're seeing these these needs.

当人们听说你要做机器人，他们脑子里想的肯定是，你拿出一个 3D 打印机，开始做硬件，然后给机器人写程序，把‘大脑’装进去，这样你就有了个机器人。

When people hear you're going into robotics, what they're gonna envision is you're pulling out a three d printer and you're gonna be making hardware, and then you're gonna be programming the brain of a robot and sticking it in the robot, and then you've got a robot.

我觉得你们讨论的并不是这个，所以或许可以聊聊这在整个机器人开发周期中处于什么位置，你们的环节在哪里结束，而其他的环节又从哪里开始。

And I don't think that's what you guys are talking about here, so maybe talk about what, where this fits in the life cycle of creating a robot and where you will end and where the rest

生态系统的其他部分将从哪里开始。

of the ecosystem will begin.

我们一直在构建的，你可以想象，是一个基础设施，以及围绕这个基础设施的软件，让人们能够构建像 Robocaine 和 Evaluate 这样的工具。

So what we have been building, you can imagine, is a infrastructure, like with the softwares around these infrastructures for people to, for them, build words, such as Robocaine and Evaluate.

这个基础设施天然地不依赖于特定的模型，也不依赖于特定的实体形态。

And this infrastructure is naturally model agnostic and embodiments agnostic.

我想说得清楚一点，因为这其实是个很微妙的点——对你来说可能显而易见，但对别人来说容易误解。你刚才的意思并不是在造机器人，而是在打造一个环境，让其他公司可以把他们的机器人大脑放进来，让它去导航、去学习。

So I just wanna be very clear just because this is actually a very subtle mean, for you it's obvious, but it's a very subtle point, which is from what you said, that's not building a robot, it's building an environment which another company can place their robot brain to navigate and to learn.

对。

Yeah.

我们现在的客户用的机器人五花八门。

So for our customers right now, they have all different kind of robots.

有的用单臂，有的用双臂，有的用固定臂，有的用移动式操作臂，有的用夹爪，还有的用更复杂的末端执行器。

Some are using, for example, single robot arms, some are using bimanual, some are using a fixed arm, some are using like a mobile manipulator, some using grippers, some are using some more elaborate versions of the end effectors.

所以我们的平台天生就支持各种形态的机器人。

So our platform right now is just naturally embodiment agnostic.

我们可以轻松接入不同类型的机器人，把它们接入我们创建并数字化的场景里，让每台机器人都能在真实环境中，以合适的可靠性和效率完成对应的任务。

We can very easily integrate different kinds of robotic embodiments, be able to put them into the words we generated, we digitalized, such that we will be able to give those individual robots capabilities of doing the right tasks and at the right levels of reliability and efficiency in the real environments.

是的。

Yeah.

而且我们的系统对模型也不做限定。

And we are also, for example, model agnostic.