为什么视频代理模型是下一个——伊桑·赫，xAI Grok Imagine

所以如果你做四倍的时间压缩，也就是四倍压缩的话——

So if you have like a temporal four compression, four times compression, then-

可能会有延迟。

It might be laggy.

是的。

Yeah.

这种延迟是天然存在的。

There's there's a lag there in nature.

那你对这个依赖性很强。

So you're very pilled on this.

我们干脆直接展示一下吧，反正视觉素材已经准备好了。

Let's just go ahead and bring it up because we have the visual prepared anyway.

实时视频生成有一些前沿的应用案例。

There's some frontier applications of real time video gen.

比如最近火起来的Flipbook就是一个例子。

So Flipbook is one of the examples that went viral recently.

对吧？

Right?

Flipbook 是什么？

What is Flipbook?

Flipbook 就像一个白色的浏览器界面。

Flipbook is is kinda like a white broad white browser.

你瞧，顶部是那种白色的浏览器界面。

You can see, like, it has the white blot browser UI on top.

不同的是，所有这些界面都是由生成式图像模型实时生成的。

The difference is all of the UIs are generated by generative image model in real time.

这里的一切都是虚构的。

And anything here are fake.

但你可以在这个想象的世界里自由探索。

But you can explore inside this imaginary world.

比如这里，我们展示了工程化的网格金字塔，模型为我们生成了这个，帮助理解它的运作方式。

Say, here we have engineering the grid pyramid, the model generates this for us to understand how it works.

如果我们想进一步探索和了解，可以点击这里的某些描述。

And if we want to navigate around and understand further, we can click on some of the description here.

作为模型，我们会生成一个新的页面，一个子页面，详细说明你想知道的内容。

As a model, we'll generate a new page, new sub page describing the details we want to know about.

所以这就像在看一段视频，但它会暂停，等我们下一步操作，然后根据我们的互动继续播放下一个内容。

So it's basically kind of we are playing a video, but it's pausing for our next interaction, and then it just plays the next thing based on our interaction.

对。

Yeah.

挺有意思的。

It's kinda cool.

没错。

Yes.

你其实是在决定，比如，你是怎么建这个金字塔的？

You kind of decide So your this was, know, how do you make a pyramid?

那种杠杆技术挺有意思的，对吧？

Levering technique seemed interesting, right?

它展示了你是怎么做到的，我想

It shows how do you take okay, I wanna

知道，那个演示推文在帧之间有更多的动画效果。

know The what the demo tweet had more animation between frames.

我觉得它只是跳过了

I think it's just skipping

它确实跳过了很多帧。

It was just skipping a lot of frames.

是的。

Yeah.

它还有不少视频模式，不过我猜很多人在用。

It also has a lot of video mode, but I guess a lot of people are using it.

对。

Yeah.

明白了。

I see.

完全不是这样。

I It's not at all.

有一个实时视频流。

There's a live video stream.

我们可以试试。

We can try.

对。

Yeah.

这就是你所设想的极端未来的一个例子。

So so this is an example of the kind of future that you see at the extreme.

我们当然还没达到这一步，但在一个推理完全免费的世界里，对。

We don't we're obviously not in it today, but in a world where inference is completely free Yeah.

这比用文字生成代码还要好？

This is better than generating code in text?

对。

Yeah.

所以，这就是词模型的最终形态了，我觉得。

So this is this is the final state of for word model, I think.

想象一下，如果互联网根本不存在，你输入 google.com 会怎么样？

Imagine imagine Internet doesn't exist, and then you'd type in google.com.

模型这时候该给你展示什么内容呢？

Like, what should what should a model show you?

模型能想象出东西，而这就是它想象出来的结果。

As a model can imagine something, and this is what the model imagined.

这些网页，其实根本不存在。

And these web pages, they completely do not exist.

我觉得，随着推理成本下降，我们将来会为所有东西都配上生成式界面。

So I think as the inference costs come down, we are going to have generative UI for everything.

想想代码模型是怎么工作的：它写网页代码，代码被转成二进制，二进制再渲染出屏幕上的像素。

If you think about how the coding model works, so they they write code for web page, and they render the code might be converted into binary, and the binary render the pixels on the screen.

每次机器学习有突破，事情显然都会变得更直观。

So in machine learning, every time we have some breakthrough, obviously it's more intuant.

那为什么我们不能直接对像素下达用户指令呢？

So why don't we have like a user instruction to the pixel directly?

生成式界面就是用户意图直接作用于像素。

So the generative UI, there'll be user intention to the pixels directly.

比如说，就算大家都用同样的邮箱界面，我也希望它能有点不一样。

And say like, even if I want email, let's say everyone have the same interface, but I want it slightly different.

我想让邮箱像TikTok那样，左右滑动就能切换邮件。

I want the email to show show to me like a TikTok so I can swipe left and right for the emails.

或者你可能想要别的样式。

And or maybe you want something else.

我们可以有完全不同的设计。

We can have completely different things.

比如我看Instagram动态时，从来不喜欢点赞按钮，总不小心点到，那就干脆生成一个没有它的界面。

Or like I have, I'm looking at Instagram stories, I don't like the like button, I always make click it, and generate the UI without it.

这将彻底颠覆现有的界面形态。

So it's going to be a revolutionary replacement of the interface.

所以未来，后台可能会运行更强大的操作模型和编码模型。

So in the future, we might have much more powerful OMs and coding models running behind the scene.

而前端则由扩散模型来直接向你展示内容。

And in the in the front end, the diffusion model will actually be the front end to show stuff to you.

这就是我想象中的样子。

That's how I imagine it.

是的。

Yeah.

扩散模型做前端，确定性模型做后端。

Diffusion front end, deterministic back end.

没错。

Yes.

差不多就是这么回事。

Something like that.

我觉得这成本太高了，不过呢。

I find that very expensive, but, you know.

我觉得你把大模型在后端写代码称为确定性挺有意思，好吧。

I find it interesting you called LLMs writing code on the back end deterministic, but okay.

是的，你写一次，如果

Yeah, you write it once, If you

你想想成本，比如说H100每小时一美元。

you think about the cost, say, let's say H100 costs $1 per hour.

如果你每天用八小时，一个月三十天，每个月就花2.4美元，你根本不想付这笔钱。

And if you use this eight hours a day and thirty days, so every month you're paying this $2.40, you'll actually not wanna pay for that.

这比Cloud Code Max还贵。

That's even more expensive than Cloud Code Max.

但如果你觉得算力成本每年能降一半，我觉得未来很可能很快就会到来。

But if you think about the compute costs come down like two times every year, I think that the future will likely arrive

随着算力成本下降、算力变快、模型更聪明，是的。

within Compute two cost comes down, compute gets faster, model gets smarter, Yeah.

模型会越来越

Model gets

我知道你为什么说翻两倍，我觉得其实是翻一百倍。

know I why you say two times because I think it's like a 100 times.

在语言模型上，每十二到十八个月，同等LMSYS ELO分数下，性能大概会提升一百到一千倍。

In language models, it is roughly 100 to a thousand times every 12 to eighteen months for the same given level of LMSYS ELO.

这可是综合了所有因素后的结果。

That that's a net of everything.

对吧？

Right?

这是模型性能和算力共同提升的结果。

That's model performance alongside compute.

不只是算力成本下降，而是一个非常有意思的未来。

So different than just compute cost come down, but, you know, a very interesting future.

是的。

Yeah.

所以网页设计师们得强调一下，无障碍访问是个问题。

So the the web designers will have to shout out that accessibility is an issue.

对吧？

Right?

那你怎么

How

处理屏幕阅读器之类的问题呢？

do you

怎么应对屏幕阅读器之类的呢？

deal with screen readers or whatever?

但没错，这种叙事方式的信息密度远超任何代码能生成的内容。

But, yes, this is higher bandwidth storytelling than anything you can possibly generate with code.

对吧？

Right?

所以我觉得大致就是这么个意思。

So I think that's the the rough idea.

我想补充一点。

I'd like to add a little bit.

人类在看东西、看视频的时候，信息接收能力其实是最强的。

So human naturally have the maximum bandwidth when when we are looking at things, look at videos.

而我们在说话的时候，表达能力也是最强的。

And and we also have maximum output bandwidth when when we are talking.

所以未来可能是这样：我们跟AI对话，AI则用生成式界面来回应我们。

So in the future, it might be something like we we talk to AI models, and the AI model responds back with the generative UI.

在脑机接口出现之前，这可能是与AI交互时带宽最大的方式了。

So that that will be the maximum input and output bandwidth to interact with AI models before Neuralink happens.

而且这也很个性化，对吧？

And I mean, it's also very custom, right?

有些人特别依赖视觉，有些人则没那么依赖，对吧？

Some people are very visual, some people are not as visual, right?

他们更喜欢文字，但生成式界面的好处是，它也能用文字呈现。

They prefer the text, but the best thing about generative UI, right, can also be text.

对。

Yes.

还有一个项目我想特别提一下，叫 Neural OS。

There's another project that we will I wanted to highlight, which is the Neural OS.

想法有点类似，但这次你是用视频模型来模拟操作你的操作系统。

Kinda similar idea, but here you're literally operating simulating your operating system with a video model.

对。

Yes.

你可以在里面玩《毁灭战士》，也能用 Firefox。

And you can play Doom, you can do Firefox.

我觉得这个其实没那么惊艳，毕竟这只是一个我能直接运行的操作系统。

I find this, like, mildly less impressive, obviously, because it's an OS that I can run.

但在这里，所有东西都是虚拟生成的。

But here, everything is imagined.

我以前习惯用命令 W 来关闭 Firefox 的标签页。

I I was, you know, used to the command w to close the Firefox tab.

但我从来没崩溃过。

I I didn't crash.

太

It's too

太沉浸了。

too immersive.

对我来说太沉浸了。

It's too immersive for me.

我想关掉那个标签页。

I wanted to close the tab.

不过，是的，我可以运行生成的扩散模型。

But, yes, I can play generated diffuser.

你知道吗，这速度惊人地快。

You know, this is shockingly fast.

对。

Yeah.

我记得大概一两年前有过一个演示。

Because I I I remember there was a demo about, like, maybe one or two years ago.

有人试过用图像模型做第一人称射击游戏。

Someone tried to do the first person shooter with a image model.

但画面完全不连贯。

There was no consistency.

而且运行起来特别慢。

It was very slow.

但这里看起来，真实感很强，这简直就是《毁灭战士》。

But here, looks like realistically, it's this is doom.

我的意思是，这事其实有两面性。

I mean, I think there's two sides to that.

对吧？

Right?

你说玩游戏，到底什么是关键部分？

There's like, okay, what is running a game?

真正吃资源的是游戏引擎，还有灯光、画面这些内容。

The heavy part of it is actually the game engine, all the lighting, all that stuff, the graphics.

这其实就是一段视频而已。

This is just kind of video.

对吧？

Right?

我们已经解决了一致性问题。

Like, we've solved consistency.

但这看起来还是几年前的图像生成效果。

This is still you know, it looks like a few years old image generation.

虽然有一点时间上的连贯性，但本质上还是把一张张图片拼起来当视频用，不过它确实能让人直观地想象出未来想看到的画面，对吧？

There's some temporal consistency, but it's kind of just images stitched together as frame video, but it's a good visual representation to picture the future you want to see, right?

我更看重的是这类东西所展现的这种感觉。

Like that's what I see in these more so.

这让我想到视频模型是如何不断进步的。

This reminds me of how the video models gets better and better.

如果单看 Neural OS，它就像个糟糕的 Windows 版本，本来我们完全可以有更好的。

So Neural OS is kinda if you just look at it, it feels like it's just a crappy version of like, Windows we could have.

对吧？

Right?

但区别在于，这个模型完全被现有的操作系统数据过拟合了。

And but but the difference is so the model this model is overfitted on the the existing operating systems.

它根本生成不出任何超出这些范围的东西。

It it can generate nothing different than that.

不过它其实和视频模型也很相似。

But it's actually also similar to video models.

当我们训练视频模型或图像模型时，用的是互联网上的数据。

So when we're training these video model, image model, we train them on Internet.

互联网上可没有那些虚构的、超自然的内容。

There's no imaginary supernatural stuff on the Internet.

但一旦模型训练好了，你就能让它生成出数据里根本不存在的超自然内容。

But once we train this model, you can prompt the model to generate something supernatural that have never existed in the dataset.

如果你用整个互联网上的标准屏幕录制视频来训练你的神经操作系统或神经计算机。

If you train your neural OS or neural computer on the standard screen recordings on the entire Internet.

所以这个模型能想象出完全新颖的界面来与电脑交互。

So model can imagine completely new interface to to interact with the computer.

是的。

Yeah.

这真是让我觉得特别神奇的一件事。

This is one of those things that is magical to me.

通常来说，泛化超出训练分布是件坏事。

Usually, generalizing auto distribution is bad.

但不知怎么的，我们似乎学到了某种内在的世界模型。

But somehow, we have learned some kind of internal world model Yes.

你说，这个加上那个，结果却呈现出彩虹和蝴蝶的样子。

That you say, you know, this plus but it looks like rainbows and butterflies.

它还真能做出合理的东西来。

It'll do it, and it'll kind of make sense.

没错。

Yep.

所以，确实挺酷的。

So, yeah, that's kind of cool.

对。

Yeah.

我不确定还有什么更多可说的了。

I I don't know if there's any comment more more on there.

我其实还想再多聊一点模型架构的事，我觉得你刚才也提到一些了。

I I I do I do wanted to I did wanted to touch a little bit more on the model architecture stuff, which I think you were getting.

这真的特别有意思。

It's, like, really fascinating.

我们平时很少有机会深入讨论这个。

We we don't get a chance to talk about this enough.

我们每一年发布的论文，基本都覆盖过了。

One of the papers that we covered, we've covered every annual segment anything release.

我不知道你有没有关注，不过你是做计算机视觉的，应该很清楚。

And I don't know if you follow I mean, you're a computer vision guy, so you you you know.

他们用了记忆注意力机制，这还挺有意思的。

So they they did memory attention, which is kind of interesting.

我一直觉得，只要能在时间维度上保持某种一致性，就很有价值。

And I always think, like, anything where you can across the temporal dimension keep some consistency.

我觉得这特别吸引人。

I think it's, like, very fascinating.

我觉得计算机视觉那边的技术往视频生成领域渗透，这方面还没被充分探索。

And I don't know if basically, like, does that the CV side bleeding into video gen side, I think is underexplored.

对吧？

Right?

我们平时聊这个，往往只想到标注，但实际上可以直接借用架构本身。

Like, we talk about it for labeling, but actually, you can borrow the architecture itself.

还有完全不同的其他方法。

And there's there's also complete different approaches.

对吧？

Right?

你提到了世界模型这个词，所以我们从视频模型聊到了世界模型。

Like, you you brought up the term world model, so we went from video model to world model.

现在有扩散模型，但其他人也在探索其他方法。

There is diffusion, but there's also other approaches that people are doing.

也许我们之后也可以聊聊这些方法，你觉得呢？

So maybe we get into those after as well, you know?

是的。

Yeah.

对。

Yeah.

他对世界模型有一整套自己的定义和说法。

He has a whole definition of world models and stuff.

我觉得我们刚才给了你太多信息了，你有什么想说的尽管讲。

I I feel like we threw a lot at you, whatever you wanna comment on.

我觉得我们其实应该回过头来讨论一下，比如说——

I I think one thing that we should actually comment back on is like, okay.

我们刚才在聊图像生成视频模型的训练步骤，但有一件事我们很少提到，就是你提到的训练数据差异。

So we were talking about the steps to train image gen to video model, one thing we don't see as much of is like, okay, you brought up the delta in training data.

对吧？

Right?

视频模型可能泛化能力没那么强，但训练一个大型视频模型的成本是多少呢？

So you won't have as much a video model might not generalize, but what is the cost of training a large video model?

我们大概知道大语言模型的情况，比如今天刚发布的那个Poolside项目，对吧？

So we know for LLMs roughly, okay, even like the poolside thing that came out today, right?

它是个Gemma级别的模型，用了大约四万亿个token，在这么多H200上训练了这么长时间。

It's a Gemma level model trained on roughly 40,000,000,000,000 tokens at this many H two hundreds over this much time.

对吧？

Right?

你能看到这笔训练具体花了多少钱。

You can see what is the exact cost of that.

用了多少GPU小时？用了多少台H200？

So how many GPU hours over how much H 200 costs?

那视频模型和图像模型的后台计算该怎么算呢？

So how do we do the backend math of, you know, same thing for video models, image models?

你怎么来拆解这个成本？

How you kind of break that down?

我可以分享一些粗略的估算。

I can share some backseat analog calculation.

有意思的是，视频模型的成本其实和语言模型差不多。

So surprisingly video models is like, the the cost is very is comparable to language models.

显然，规模最大的还是语言模型，中等规模的也差不多是语言模型。

And obvious, the largest scale is language model, maybe like a medium scale is language models.

光是存储视频本身，开销就已经很大了。

I said, just storing the videos alone, costs a lot.

你可以去查一下 AWS 上的定价。

You can maybe look up on AWS or something.

假设你有一百亿个视频，每个视频五兆字节，那光是存储就要五拍字节。

You really like say, if you have a billion videos, and let's just say like each video, like five megabyte, then you would need like five petabyte to just store those videos.

还有别忘了，我们用VAE来压缩视频，但这些连续特征也需要存下来，占的存储空间也不小。

And also remember, we talk about you use a VAE to compress the videos, and you also need to store typically, you need to store those continuous feature and also in your storage.

这些特征所占的空间，跟原始视频差不多大。

That's also comparable size with the videos themselves.

光是存这些视频和特征，加起来就得几十PB。

So just storing storing these videos and the features is is tens of petabytes alone.

对，没错。

And Oh, yeah.

我刚查了下计算数据。

I just I just looked up the calculation.

在S3标准存储上，五PB每月要花十万块。

Five petabytes on S3 standard is 100 k per month.

明白了。

Okay.

那几十PB的话，一个月就是二十万。

And then like tens of petabytes, 200 k.

更贵的是通过互联网传输数据的入站和出站费用。

And even more expensive is you have the ingress and egress through the internet.

你得把那些视频全部下载下来。

You have to just to download those videos.

我觉得在AWS上，光是下载的费用就比单纯存储还高，每次训练你都得拉取一次数据。

I believe it's it's more expensive on AWS than just storing And those each training runs, you'll probably need to pull them once.

如果你要反复训练，那开销就更大了。

If you're trained multiple times, it's even more than that.

光是存储和网络传输的费用，每个月就得几百万，这还没算上GPU的开销。

So it's like just storing the network, those costs is just, I guess it would be a few millions per month to just storing everything, not to mention the GPU costs.

至于我这边，谭军，像GPU租赁这种计算资源，其实挺高效的。

My side, Tanjun, like, you know, the compute rental, like GPU rental is very efficient.

一方面呢，可以像xAI那样自建数据中心。

There's one side, okay, can be xAI and build your data center.

那我们为什么不连存储和计算一起自建呢？

Should we not just build our like storage compute as well?

云服务的成本跟单纯存储相比。

Cloud costs compared to just, storage.

能省下很多钱。

You save so much.

是的，没错。

Yeah, exactly.

尤其是还要考虑数据流出之类的费用。

Especially with like egress and stuff.

你知道的。

You know.

这主意不错，但也带来一些自身的挑战。

That's a good idea, but it also comes to, there are some of its own challenges.

当然，当然。

Of course, of course.

那些建设GPU数据中心的人，可能没料到需要这么大的存储空间。

And like people who build the GPU data centers, they might not expect this much storage.

而且，人们在搭建存储系统时，通常只是随便找个地方，用普通的CPU就搞定了。

And, yeah, people build storage, typically, they just build it somewhere with just CPUs.

我刚看了下，仅AWS的出口流量费用就高达五笔。

I just looked at five AWS only charges for egress, not ingress.

五拍字节的 tiers 费用是23万美元。

Tier five for five petabytes is 230 k.

是的。

Yeah.

对。

Yeah.

甚至更贵。

Even more expensive

比Sora还贵。

than Sora.

我的Sora是按月计费的。

My Sora is per month.

对吧？

Right?

你登记了就再也登不出去了。

You check-in, then you cannot check out.

这太厉害了。

So it's so cool.

没关系。

It's okay.

真的真的真的

So so so

就是这边。

so so that side.

TL，亲爱的，我的数据其实是

TL, dear, you know, my my Data data is

比你想象的要大得多。

larger than you think.

是的。

Yes.

对。

Yeah.

我粗略算了一下GPU小时数乘以成本，数额也相当可观，有些地方我可能没考虑到。

My backhand math of GPU hours times GPU cost is also very much, you know, I'm missing some storms.

你这情况其实比普通训练更受IO限制。

You're also you're basically like also more IO bound than normal training.

是的。

Yes.

是的。

Yes.

因为数据加载和缓存这些环节都变得特别关键。

Because like data loadings and caching, everything becomes super important.

对。

Yeah.

是的。

Yeah.

在 Cosmos 里，我们做了很多优化，让它不再受 IO 限制。

So in Cosmos, we did a lot of optimizations to make it not IO bound.

说到实际训练模型，GPU 成本方面，如果你去看看那些开源模型，比如视频模型有多大，像 LTX 就有 190 亿参数。

But speaking of the training actually training the model, GPU cost, if you look up, like, the so open source model, how big these video models are, think, like, LTX has 19 b parameters.

这是一个稠密模型。

That's a dense model.

现在大家也在探索 MoE 架构，可能激活参数是 200 亿，但总参数量能达到上千亿。

And people are also exploring MoEs, so it might be like 20 b active and like a hundreds b total.

这跟中等规模的 OIM 模型差不多大了。

So that's that's even that's similar size as medium sized OIM models.

至于训练用的 token 数量，我们在 Cosmos 里已经公开了。

And if you if you look at number of tokens, we disclose that in Cosmos.

视觉 token 的总量也有好几万亿。

It's also like tens of trillions of tokens on the visual tokens.

所以综合来看，训练这些视频模型的成本其实和大语言模型差不多。

So putting this together, the cost of, like, training these video models, is actually comparable with OMs.

更别说视频数据和文本数据本身就有差异，训练起来可能效率更低。

Not to mention, the the infrared is slightly different from OMs, so it might be less efficient to train these models.

你

Do you

能享受到传统扩散模型那种推理加速的好处吗？

get the benefits of traditional diffusion speed up?

比如图像生成有LCM、LoRA这些微调方法，还有很多优化技术，这些在推理阶段确实有不少已经应用到扩散模型上了？

So for images, there's LCM, LORAS for fine tuning, a lot of stuff that's There's been some overlap that applies to diffusion on the inference side and stuff?

是的，推理方面完全是另一回事。

Yeah, the difference is the inference side is a completely different story.

我觉得在训练阶段，想大幅降低成本可能比较困难。

I think for the training side, it might be a little bit hard to reduce that cost.

而在推理阶段，最大的提升来自于这些模型的蒸馏。

And for the inference side, the biggest gain is from the distillation of these models.

这叫步数蒸馏，和一般的知识蒸馏有点不一样。

You can it's called step distillation, slightly different from knowledge distillation in all of them.

对于流匹配模型，通常需要一百步左右才能生成结果。

So you typically, for flow matching models, you need like a 100 steps or something.

像这种失真模型，甚至需要上千步才能生成一张高质量的图片或视频。

Like, a distortion model even need need even more, like, a thousand steps to to generate a good image or video.

步数蒸馏就是让模型自己学会如何从一步直接生成结果。

A step distillation is try to learn to generate your step from the model itself.

你可以先用完整模型跑一百步生成结果，再让一个只跑十步的模型去模仿那个完美的结果。

It's kind of like now you use the full model to generate in a 100 steps, and then you take a model that only generate 10 steps and let that model learn from the perfect one.

对的。

Yeah.

这为什么能奏效呢？

Why is this work?

这是强模型教弱模型，跟其他那种强教弱的思路其实是一样的。

Strong to weak, same as It's the rest kind of the of like strong to weak.

从建模的角度来看，这个强大的教师模型试图模拟整个互联网上的图像和视频，这个分布实在太复杂了。

I guess from the modeling perspective, the strong model the teacher model is trying to model the image and videos of the entire Internet, and that distribution is extremely complex.

而经过步骤蒸馏的模型只需要向教师模型学习，教师模型本身结构是固定的。

As a step distilled model is just trying to learn from the teacher, the teacher is is a model and the size is fixed.

它要学的分布比整个互联网简单多了。

The distribution is much simpler than the whole Internet.

这就是我觉得步骤蒸馏能起作用的原因。

That's the intuition I have why step distillation can work.

通常这些模型在实际应用中都只运行很少的几步。

So usually these models serve in productions, they only run-in a few steps.

在Cosmos里，我们用的是四步或者八步。

In Cosmos, I believe we have like four steps and eight steps.

如果是更简单的任务，比如图到图的转换，在Cosmos Transfer里甚至一步就能完成。

If you do some simpler task, like image to image translation, it can even run-in first step by one step in in Cosmos transfer.

是的。

Yeah.

我觉得这种思路也贯穿了大部分一致性模型的研究。

I I think this is the same intuition that guides a lot of the consistency model work.

我之前发给你一个关于sCM的链接。

I I sent you a a link for sCM.

我不确定你有没有

I don't know if you

有的。

Yeah.

看过这个了。

Covered that.

在我看来，这真是我见过的OpenAI最令人印象深刻的论文之一。

To me, that was actually one of, like, the most impressive papers I've ever seen from OpenAI.

没错。

Mhmm.

这正是一致性模型背后那个统一的核心理念。

That, like, this is the unifying grand concept of consistency models.

你不了解这方面有什么看法吗？

I don't know if you have any comments on this.

其实有几种不同的方法，比如这个。

So so there are there are a few different approaches, like Oh, Here it is.

对。

Yeah.

两步、二十步，或者一百步，都无所谓了。

Two steps versus 20 or a 100 steps, whatever.

这事儿已经解决了。

It's already done.

比如一致性模型，就是其中一种方法。

So there are there are a few different approaches, for example, consistency model.

我们也不能忘了GAN。

And there are also actually, we shouldn't forget GAN.

GAN其实是最早的一步蒸馏方法，因为它一开始就是只训练一步的。

So GAN, actually that was the OG step distillation, because it trained just one step to begin with.

实际上，很多方法比如分布匹配的消解，会把GAN作为消解过程中的一个损失函数。

So actually a lot of for example, there's a distribution matching dissolution, which uses GAN as one of the loss for dissolution.

它只是告诉你：嘿，生成一张图，然后有个判别器来判断这张图是真是假。

It, again, just tells you, hey, like, generate an image, and then it has a discriminator to to tell, is this image real or not?

所以模型只需要学习部分分布，而不是完整的分布。

So the model the model just need to learn one of the distribution, not the full distribution.

因为在训练时，模型要从网上那堆图里重建出原始图像，这难度太高了。

Because in training, the model is asked to reconstruct the ground truth image from the internet, which is extremely hard.

而训练GAN的时候，整个过程就一步到位。

And when you're training GAN, it's a one step process.

就是说：你生成一张图就行。

It's just a, hey, you generate image.

这张图看起来是不是和网上那些图一样真实？这任务简单多了。

Does this image look as real as the image from the Internet, which is a much simpler task?

而且把这么多方法结合起来，大家通常都会这么做，比如一致性模型和分布匹配。

And yeah, combining a lot of these approaches together, people typically do that, like consistency model and distribution matching.

而且同样，我们可以得到这种多步模型。

And and again, we can get these few step models.

明白了。

Okay.

然后还有一步法的模型。

Then there's one step

我想补充一点，就是音频方面。

I wanted to add, which is audio Yes.

对。

Mhmm.

是的。

Yeah.

还有视频。

And video.

是的。

Yeah.

所以Grok Imagine 0.9，我认为是首个大规模部署的音视频联合模型。

So Grok Imagine 0.9, I I believe it's a is a first first audio video joint model deployed at a large scale.

那这是你们的第一个模型吗？

So And that was your first model?

是的。

Yeah.

那是Grok Imagine的第一个模型。

That was Grok Imagine's first model.

它能同时生成音频和视频。

It's it's audio video joint generation.

我觉得最难的是模态对齐。

I think the the hard part is like the the modality alignment.

在这款联合模型之前，我们已经有文本到视频的对齐了。

Because before this joint model, like, we have we have text to video alignment.

文本和视频之间已经有了对应的关联。

We we have this correspondence between text and video.

大多数视觉语言模型都能理解图片，但对视频就很少涉及，更别提音频了。

Typically, most of the VLMs, they they understand images and videos videos very rare, and they they don't understand audio mostly.

如果你看看语言模型在音频生成方面的表现，跟它们聊天没问题，但要是让它们唱歌，通常都唱得不好。

And if you look at the audio generation on the LM side, you can talk to them perfectly fine, but if you ask them to sing a song or something, it typically is not very good.

而且它们也完全不会生成音乐。

Also, don't have they don't have music either.

难点在于，音频其实包含两个部分。

The hard part is that, actually, audio has two components.

一个是离散的，一个是连续的。

It has like a discrete component, a continuous component.

离散部分就像语言本身。

The discrete component is like the the language.

我们说话的时候，其实只是发出一些

So when we speak, it's just some

这属于语音识别的问题。

It's a ASR issue.

是的。

Yeah.

对。

Yeah.

我觉得那就是带有某些特性的文本标记。

It's it's text token with some characteristics, I I would say.

但音乐呢。

But but music

我觉得搞语音的人会不同意。

I think the speech guys would disagree.

就像说话时的停顿、重复这些情况。

It's like, disfluencies and then, you know,

那些是的。

those Yeah.

我认为主要是这样。

I'd say largely.

对。

Mhmm.

但音乐完全是另一回事。

The the me but the music is completely different.

音乐是连续不断的，没法像语言模型里的离散词元那样去建模。

It's it's very continuous, and you cannot model them like discrete tokens in language models.

这对多模态模型来说是个难点，更别说我们还得把文本、视频和音频对齐了。

This is like the hard part for foreign model, not to mention we have to align text, video, and audio together.

是的。

Yeah.

那怎么办呢？

So How?

所以，有几个重大挑战，比如我们现在说的视觉语言模型，大多数根本理解不了音频。

So significant some significant challenges are like so so first, like, we we talk about as VLMs, they they cannot understand most of them cannot understand audio.

所以你得想办法生成音频的合成数据。

So you have to have some way to to do the synthetic data generation for audio.

你得给模型加字幕，这需要大量合成数据和人工标注的工作。

You have to caption the model and that involve synthetic data and human data efforts a lot.

而且令人惊讶的是，大多数排名模型根本没法准确识别节奏、音调和音乐的细节。

And not just surprisingly, most of the RMs are very bad at recognizing like the beat, tone, and the details of the music.

它们或许能大概猜出是哪首歌，但要详细描述音乐的方方面面就太难了。

They they can they can give some general general prediction of which song is this, but it's very hard to describe the details of of the music.

就像我们在图像生成里说的，你得尽可能详细地描述画面，让盲人也能在脑中还原出来。

Like we mentioned in image generation, like, you have to describe image as detailed as possible so that someone blind can reconstruct that.

同样的道理，这里要让听障人士不靠听，也能在脑中还原出音乐的质感。

So here is, like, someone deaf can reconstruct how the music sounds like without actually listening to it.

也许你可以这么想：你得把音乐和对白的所有细节都完整捕捉下来。

Maybe like, you can think of it need to have the, what they call the You gotta have all the details of the music and the dialogue.

所以，这类挑战主要集中在音乐和音频上吗？

So is the challenge there typically stuff like music and audio?

还是说，其实连个基本的基准都还没建立起来？

Or is it just, like, is there a baseline?

明白了。

Okay.

现在数据量足够让我们理解叙述和对话，但音频里的那些细微差别才是数据问题的根源，还是说从零开始就能直接搞定？

There's enough data where we can understand, you know, narration, conversation, but there's nuances in audio that that's where you hit all the data issues, or is it just from state zero, you just do it all right?

有一件事很重要，就是对齐问题。

So one important thing is like the alignment.

模型必须清楚视频和音频在时间上是怎么对应的，也就是在哪个时间点，视频帧和音频片段是匹配的。

So the model the model has to know, like, the video and audio, they it has to have a time based alignment, like at which time step the video and the audio token correspond to each other.

但大多数其他模态的数据，我们根本没有这种精确的时间对齐。

We actually don't have this kind of alignment for for for most of the other modalities.

比如文本和图像、文本和视频，它们的关联都是松散的。

If you think about, like, text and image, text and video, they are loosely aligned.

你可以描述视频里发生了什么，但不需要精确到每一刻的细节。

So you can you can have a description of what's going on in the video, but you don't have to exactly you typically don't have exact description.

比如，某一秒发生了什么？两秒后又发生了什么？

Oh, at at a time step, one second, like what happens after two seconds?

当然。

Of course.

发生什么事了？

What happened?

是的。

Yeah.

理想的時間步長是多少？

What what was the ideal time step?

你得做消融實驗，結果發現大概是四秒左右？

You have to ablate it and then it's like four seconds or something?

這取決於你怎麼設計模型，對吧。

So that comes down to how you design the model Yeah.

要讓模型能感知到時間這個維度。

To for the model to to be aware of as a time as a time modality.

所以模型得具備時間意識。

So the model is like a time aware.

如果你想想OMs，这其实挺特别的。

And that's something pretty unique if you think about OMs.

所以如果你让OLM去完成一个任务，比如你问它，它会说这个任务大概要十二个小时才能做完。

So if you ask OLM to complete a task, say you ask them and they will say, oh, this task will probably take twelve hours to complete.

但一小时后它又回来，说我已经花了两天时间，所有办法都试过了。

And they come back in one hour, say, I've already spent two days on this and have exhausted everything.

是的。

Yeah.

所以语言模型本身，其实并没有时间概念吗？

So so the LM themselves, they they don't have a sense of time there?

我觉得这不光是它们没时间观念的问题。

I actually don't think that's just them not having a sense of time.

我觉得这背后是有原因的。

I think it's somewhat based.

对吧？

Right?

对啊。

Like Mhmm.

你让人去开发某个功能，去实现它。

You tell someone, okay, go work on this feature, go implement this.

在没有大模型以这种速度工作的情况下，你心里对这类任务需要多久是有基本判断的。

There's a general understanding you would have of how long that would take without LLMs working at LM speed.

对吧？

Right?

你回想两年前，我说你给我做个新的前端，要支持隐空间搜索，加个搜索栏，一堆功能，你大概会估个几天时间。

So you think back like two years ago, I tell you to like, build me like a new front end for latent space, have a search bar, have all this, you'll estimate that it'll take a few days.

对吧？

Right?

你跟大模型说，去把这个做了。

So you tell an LLM, go build this.

它说需要几天，但我觉得这背后还是有依据的，不是说它真有多懂，但这个例子能让你明白它这种说法是怎么来的。

It'll take me a few days, but you know, I think it's somewhat grounded as opposed to them not having the best not saying that they have a great understanding, but I think that example is like, you can see where it comes from.

你接受的是全网文本的训练。

You're trained on all over the text.

在猜测人类会怎么说。

Trying to estimate what a human would say.

没错。

Yes.

因为数据本身反映的就是这些内容。

Because that's what the data kind of represents.

这些数据来自互联网上的语料。

It came from the corpus on the internet.

人们会有一个大致的判断。

People have a estimate.

是的。

Yeah.

而且不只是直接的训练样本。

And not even just in direct, like training samples.

对吧？

Right?

就是你对文字长度和时间的日常感知。

Just your world understanding of tokens of how long stuff takes.

对吧？

Right?

去读本书吧。

Go read a book.

得花上一段时间。

It'll take you a while.

对吧？

Right?

是的。

Yeah.

对。

Yeah.

就算你什么也不做，光是读一本书，也得花上几天时间。

Even if you do nothing but read a book, it takes a few days.

所以，没错，我会去刷会儿Reddit。

So, yeah, I'll let my Reddit.

我花了几个小时才看完。

It took me a few hours.

要看完这些研究材料，我也得花上几个小时，不过这有点跑题了。

It'll take me a few hours to go through this research, but somewhat tangent.

所以我想说的是。

So I wanna yeah.

我以前从来没这么讲过，但我觉得一个完整的世界模型必须是递归的——也就是说，模型中的参与者得意识到自己拥有一个世界模型，然后这个意识还得继续层层递归下去。

This is a train of thought I haven't really expressed until now is which is basically like a full world model must also be recursive, meaning that the participant in the world model must also be aware that they have a world model, which is like this whole recursive thing down the down the line.

但没错。

But yes.

而且这个世界模型可能是错的，得不断更新，诸如此类。

And and and that the world model can be wrong, and that they need to update it, and blah blah blah.

对。

Yeah.

我们也在通讯里讨论过这个问题，认为需要某种递归式或对抗性的世界模型。

We we've argued this on the newsletter as well, that there needs to be sort of recursive or adversarial world models.

明白了。

Okay.

我的意思是，简单问一下，你怎么定义世界模型？

I mean, just, you know, to ask, how do you define world models?

那咱们就聊聊这个。

Oh, Let's go there.

是的。

Yeah.

为了背景清楚一点，我们之前聊过视频生成，如果你说世界模型有别于这个，那你的定义是什么？

So just for context, you know, we talked about video generation, and then there's a if you say there's a distinction between world models, what's your what's your definition?

你如何看待这两者之间的区别？

How do you see the two?

是的。

Yeah.

我先说明一下，我不会去争论什么是世界模型。

So so disclaimer, I'm not going to debate, like, what is world model?

对。

Yeah.

其实有好多不同的定义。

Like, there are there are many definitions.

我就直接说说我的理解。

So I'll just talk about my definition.

因为我之前做的是多模态领域。

Since I I came from the multimodal domain.

主要是在视频方向上。

So mainly talking from video.

所以在我这儿，世界模型就是那种实时、互动、能预测长远未来的视频。

So word model is like real time, interactive, long horizon videos.

所以分为三个方面。

So there are three parts.

我们一个个来聊。

Let's talk about them one by one.

首先是互动，我们看看 Flipbook 和 Neural Computer。

So interaction, so we just look at Flipbook and Neural Computer.

在互动这部分，你的世界模型需要支持通过键盘、鼠标，甚至语音来进行操作。

So the interaction part of it, so you word model can allow you to interact with them through keyboard or mouse and maybe also voice.

这些不同的输入方式，你都能和模型互动，模型也得做出合理的回应。

So these all of the also modality, you can you can interact with with the model, and the model should respond reasonably.

第二点是实时性。

Second part is real time.

比如你移动一下鼠标，如果这个世界模型在生成一个游戏，游戏能多快做出反应？

So once you once, say, you you move your mouse, like, if, say, the the world model generate the game, like, how how fast can the game respond?

像专业的 CS:GO 玩家可能会说，响应时间必须低于十毫秒，甚至更短。

So if you're like professional CS GO players, might say, oh, you have to respond in sub ten milliseconds or even less.

所以那并不是，我想大多数人都会认为的

So that's not, I guess most of the

不对。

No.

60帧每秒。

60 FPS.

我们继续吧。

Let's go.

哦，300帧每秒。

Oh, 300 FPS.

接近500帧每秒。

Almost 500 FPS.

等等。

Wait.

好的。

Okay.

是的。

Yeah.

我还没算过，没错。

I didn't do the math, yeah.

好的。

Okay.

是的。

Yeah.

300帧每秒，那就是三毫秒。

300 FPS, that's a three millisecond.

所以你必须在3毫秒内做出响应，大多数视频模型根本做不到这一点。

So you have to respond Most within 3,000,000 of the video models cannot do that.

是的。

Yeah.

但如果你说的是像数字人这样的视频模型，响应时间可能会宽松一些。

And but if you say if you have a video model that is, say, like a digital human, the the respond time might be more generous.

比如在实时语音交互中，通常延迟是两百毫秒左右。

Maybe like typically, like, for real time voice interaction, it's like two hundred millisecond.

所以这个时间就宽松多了。

So that's that's much more generous.

但就算两百毫秒也很有挑战，别忘了我们之前提到过，VAE会带来时间压缩。

But even two hundred millisecond is pretty tricky because like, remember we mentioned, you have this temporal compression coming from the VAE.

如果你不压缩时间维度，序列长度会直接爆掉。

So if you if you don't compress a temporal dimension, your sequence lens is going to explode.

所以如果你想让模型实现真正的实时响应，就得解决长上下文的问题。

So if you want to have this real time real timeless in your model, you have to deal with long context problem.

第三点是长时程预测。

And the third part is is long horizon.

因为如果你不只是玩几秒钟的视频游戏，大多数视频模型都只能处理几秒钟的内容。

Because if you're not going to just play with video games just like a a few seconds, Most of video models only a few seconds.

我们要玩的是几分钟、几小时的长度。

We're going to play with minutes, hours.

模型必须能够生成长篇内容。

The model have to be able to generate long form content.

把这三点结合起来，就是实时、长时程、可交互的视频。

So putting these three together, it's a real time, long horizon interactive videos.

我觉得最终的状态，可能会像一个视频版的Flipbook——你可以和一个神经计算机互动，移动鼠标、点击生成界面，它就会用像素实时回应你。

I think the the final state will be, for example, like a a video a video version of Flipbook, where you can you can interact with a neural computer, you move your mouse and you click on the generative interface and it will reply to you through pixels, generally in real time.

但要达到这一步，还有很长的路要走。

But getting there, it's a very long way to get there.

所以在Grok Imagine的时候，我带领一个小团队做了一个世界模型，第一步就是开发视频扩展功能。

So one of the first step at Grok Imagine, where I led a small world model team there, was to build video extension.

视频扩展。

So video extension

啊，这是交互性的第一步。

Ah, it's the first step of interactivity.

对。

Yes.

这是第一步。

It's the first step.

对。

Yeah.

所以这是第一步。

So it's the first step.

在这里，就是视频编辑。

It here, video editing.

是的。

Yeah.

对。

Yeah.

对。

Yeah.

所以这第一步之所以重要，是因为它开启了长时序视频的可能性。

So the the first step is because this unlocks long horizon videos.

大多数视频生成模型通常都是给你一个提示词或者一张图片作为初始帧。

Typically, for most of the video generation models, you you give it a prompt or an image as an initial frame.

然后生成一段视频。

You generate a video.

就这么简单。

That's it.

就只做一次就完了。

That's just one time done.

有些创作者会尝试用上一段视频的最后一帧，当作下一段视频的起始帧。

And some some creators would try to, like, use the last frame as a first frame for the second video.

有时候能行，但要是反复这么操作，画面就会越来越乱。

It can sometimes it works, but if you do it a few times, it it says a call to a degree.

而且它没法保持整段视频的连贯性。

And It doesn't have that context over the full video.

所以时间上的连贯性就没了。

So the temporal Yeah.

你当然是用最后一帧当输入的，对吧？

You gave it the last frame, of course, right?

对。

Yeah.

没错。

Exactly.

但这其实是个挺有趣的技巧。

But it's actually a pretty fun hack.

比如如果你看过的话

Like if you've seen

哎呀，糟了。

like Oh, no.

他在说更有意思的东西。

He's saying something better.

是的。

Yeah.

是的。

Yeah.

比如说，我记得 Vue 3 会用上一段视频的最后一秒作为上下文。

And for example, like a Vue I remember Vue three has like a one second context of the the last video.

这比只用最后一帧稍微好一点，但问题一样：如果你连续扩展好几次，比如延长到一分钟，视频质量就会比最初的那段差很多。

It is slightly better than using the last frame, but it has the same problem similar problem that it the quality would degree, like, if you extend a few times to, one minute, the the video quality would look much worse than the first video.

第二个问题是，模型缺乏对之前发生过什么的长期记忆。

Second, another problem is as a model doesn't have long range knowledge of, like, what's happening before.

比如生成一段对话，两个人在说话，他们的声音可能会随时间变化，尤其是当那一秒的上下文没包含之前的内容时，问题就更明显。

Say, if they generate some dialogue, some two people speaking, and their voice might change over some time, especially if the one second conditioning does not cover the previous context.

这些就是核心挑战。

So these are the core challenges.

所以 Grok Imagine 的视频扩展功能，能利用所有之前生成视频的历史信息。

So the Grok Imagine video extension, it has historical context of all of the previous generative videos.

它知道是谁在说话、哪些物体出现过，所有这些信息都会用来生成下一段视频。

It has a context of who is speaking and objects have appeared and everything, having that to generate the next video.