The a16z Show - AI视频的未来前沿是控制 封面

AI视频的未来前沿是控制

The Next Frontier of AI Video Is Control

本集简介

a16z 普通合伙人 Jennifer Li 与 fal 联合创始人 Gorkem Yurtseven 及工程负责人 Batuhan Taskaya 对谈,探讨当生成式视频速度快到足以实时运行时,会发生哪些变化。 他们深入解析了 H3 Max 背后的技术工作——这是 fal 基于 MiniMax 开源权重视频模型进行后训练的版本,并介绍了如何将模型后训练与系统及硬件优化相结合,在保持质量的同时大幅缩短生成时间。这种速度使得连续视频的实验成为可能,包括能够记住先前场景并在运行过程中响应新指令的流式视频。 他们还讨论了下一个挑战可能不再仅仅是速度,而是控制力,涉及摄像机运动、灯光、角色、动作以及口型同步等方面。此外,他们探讨了这些能力对专业创意工作流程意味着什么:艺术家和工作室需要的是可预测的工具,而不仅仅是通过提示词生成一段视频。 资源: 在 X 上关注 Gorkem Yurtseven:https://x.com/gorkem 在 X 上关注 Batuhan Taskaya:https://x.com/isidentical 了解更多关于 fal 的信息:https://fal.ai 在 X 上关注 Jennifer Li:https://x.com/JenniferHli 保持更新: 在 YouTube 上找到 a16z:YouTube 在 X 上找到 a16z 在 LinkedIn 上找到 a16z 在 Spotify 上收听 a16z Show 在 Apple Podcasts 上收听 a16z Show 关注我们的主持人:https://twitter.com/eriktorenberg 请注意,此处内容仅供信息参考;不应被视为法律、商业、税务或投资建议,也不应用于评估任何投资或证券;且并非面向任何 a16z 基金的投资者或潜在投资者。a16z 及其关联公司可能持有文中提及公司的投资。更多详情请参阅 a16z.com/disclosures。 由 Simplecast 制作,Simplecast 是 AdsWizz 旗下公司。有关我们如何收集和使用个人数据用于广告的信息,请访问 pcm.adswizz.com。

双语字幕

仅展示文本字幕,不包含中文音频;想边听边看,请使用 Bayt 播客 App。

Speaker 0

生成式媒体和编程助手市场一起,就是我们所说的代币市场契合。

Generative media is along with the coding agent market, what we call is token market fit.

Speaker 0

大家都在等人工智能迎来一个大众消费的爆发点。

Everyone's waiting for a large consumer moment in AI.

Speaker 0

我相信H3 Max能让这成为现实。

I believe H3 Max makes it possible.

Speaker 1

你对这个模型进行后训练后,速度和收益的提升感到惊讶吗?

Were you surprised by the speed up and the gain you could get from post training this model?

Speaker 2

我们有一个公开的版本叫H3 Max Turbo,它能在1.5秒左右生成一段5秒的视频。

We have a version called H3 Max Turbo that's public that can generate, like, a five second video in, like, one point five seconds.

Speaker 2

从成本角度看,它也便宜了大约一半。

From a cost standpoint, it's also, like, 2x less.

Speaker 0

人们开始用LLM模型、GPT Astra在Blender里创作这些美丽的场景,突然之间,这为好莱坞和专业人士解锁了一个全新的工作流程。

People starting creating these beautiful scenes using an LLM model, GPT Astra, in Blender, and all of a sudden, it unlocked the whole new workflow for Hollywood and professional people.

Speaker 2

我们一直非常专注于速度、性能和品质,现在我们已经有了一个非常好的节奏模型。

We have been very, very focused towards speed, performance, quality, and now we have a really good pace model.

Speaker 2

接下来的一两个月,我们会全力投入这件事。

The next month or two is going be fully focused on.

Speaker 3

当AI视频生成速度快到能实时生成时,会发生什么?

What happens when AI video becomes fast enough to generate in real time?

Speaker 3

A16z的普通合伙人Jennifer Li与fal联合创始人Gorkem Yurtseven以及工程主管Batuhan Taskaya坐下来,讨论H3 Max以及快速变化的生成式视频技术栈。

A16z general partner Jennifer Li sits down with fal co founder Gorkem Yurtseven and head of engineering Batuhan Taskaya to discuss H3 Max and the rapidly changing generative video stack.

Speaker 3

他们探讨了后训练和系统如何让视频生成速度大幅提升,从而开启了新的体验,让视频可以连续运行、记住之前的场景,并在播放过程中根据指令做出响应。

They unpack how post training and systems made video generation significantly faster, opening up new experiences where video can run continuously, remember previous scenes, and respond to direction as it plays.

Speaker 3

但速度只是故事的一部分。

But speed is only part of the story.

Speaker 3

他们还讨论了如何推动对摄像机角度、灯光、角色和动作的更强控制,以及为什么这些工具能让生成式视频对专业创意工作流程更有用。

They also discuss the push toward greater control over camera angles, lighting, characters, and motion, and why those tools could make generative video more useful for professional creative workflows.

Speaker 1

欢迎Gorkem和Batuhan再次来到我们的播客。

Welcome, Gorkem Batuhan, to our podcast again.

Speaker 1

我们去年做过一次。

We did the last one last year.

Speaker 1

这次真是久等了,而且我们要聊的模型特别让人兴奋,就是fal的H3 Max。

This is long overdue, and we have such an exciting model to talk about, which is fal's H3 Max.

Speaker 1

它发布那天,我就说它真的是独一档的存在。

The day when it came out, I was calling it, it's really in the league of its own.

Speaker 1

看那些基准测试的图表特别有意思,这个模型的点在最左边或者最右边,

Like, it's so funny to see the benchmarks where you have the dot of this model on the far left or far right.

Speaker 1

然后其他所有模型都在另一边。

And then everything else is on the other And

Speaker 0

那个图其实是对数刻度。

that graph is actually log scale.

Speaker 0

所以实际上差距更大,但我们得把它放进去。

So it's actually further, but we had to fit it in.

Speaker 0

我们只能用对数刻度。

We had to do log scale.

Speaker 1

那真是太搞笑了。

That is hilarious.

Speaker 0

时间方面,质量就不行了。

The time portion, the quality is not.

Speaker 1

确实。

For sure.

Speaker 1

网上肯定注意到了。

The internet noticed for sure.

Speaker 1

关于它的推文都爆了。

There are so many viral tweets about it.

Speaker 1

大家真的都拿这个模型玩得很开心。

Like people really played around with this model.

Speaker 1

能不能给我们讲讲背后的故事,是什么启发你们去训练这个来自MiniMax的开源权重模型,你们又是怎么做到质量和速度都这么好的?

Maybe just give us the backstory of what inspired you to post train this open weight model from MiniMax and how did you get the quality and speed to

Speaker 0

艺术家们?

artists?

Speaker 0

首先,MiniMax H3模型是第一个真正开源、能力很强的最新一代视频模型。

First of all, the MiniMax H3 model is the first truly open source, very capable, latest generation video model out there.

Speaker 0

所以,虽然我们和其他一些模型实验室合作,为他们运行推理,但我们从来没有这种能力,或者说没有权利在此基础上添加这种能力。

So even though we work with some of the other model labs to run inference for them, we never had this capability, like had the right to add this capability on top of it.

Speaker 0

所以当MiniMax推出了他们那个非常强大的开源模型,真正是新一代的,能接受参考,架构和其他视频模型很相似。

So when MiniMax came up with their very capable open source model that is truly last generation, can take references, like very familiar architecture to any other video model.

Speaker 0

我们认为这是一个很好的机会,可以全力以赴,看看我们能做些什么。

We thought this is a great opportunity to go all in and see what, like, we can do.

Speaker 0

而且,我们确实做了很多不同的事情,我们接下来会谈到,这些结合起来就产生了你在图表上看到的结果。

And again, we did, like, many different things that we are gonna talk about that combined gave the results that you show on the graphs.

Speaker 0

但这次一切能顺利整合的最大原因,是因为H3是第一个真正开源的新一代视频模型。

But the biggest reason why everything came together for this particular moment was because H3 was the first truly next generation video model that's open source.

Speaker 1

考虑到FAW一直以生成式媒体推理服务平台著称,你们是怎么想到要进入后训练开源模型这个领域的?

What is the idea given like FAW has been known to be like a generative media inference serving platform?

Speaker 1

就是,你们是怎么想到要去做后训练开源模型的?

Like what is the idea to get into post training open weight model?

Speaker 1

你在博客里也谈了不少,比如把系统工作和模型本身结合起来。

Like you talk quite a bit about it in the blog of like combining the system work with the model itself.

Speaker 1

那你能多讲讲背后的工作吗?

Maybe talk more about the work behind that.

Speaker 0

生成式媒体,我觉得,和编程代理市场一样,就是我们所说的“代币市场契合”。

Generative media is, I would say, along with the coding agent market, what we call is token market fit.

Speaker 0

我们的定义是,一个人能否高效地消耗大量代币,比如一个月消耗一万个代币左右。

And the way we define it is as can a single person productively spend a lot of tokens and the amount is like 10 ks a month, something like that.

Speaker 0

所以市场上对生成视频、同时生成很多东西的需求非常大,那些以此为日常工作的人,整天坐在电脑前做这些,他们花费数千美元,消耗大量代币。

So there is incredible amount of demand in the market to generate video, to generate many things at the same time, and a person who is doing this for their daily job, they spend in front of a computer and do this all day long, and they spend thousands of dollars, lots of tokens.

Speaker 0

而且从四月左右开始,整个行业包括fal自己,都一直受限于计算资源。

And since around April, the whole industry and fal itself, we've been compute constrained.

Speaker 0

我们增长的速度和我们增加计算资源的速度一样快。

We are growing as much as we are adding compute.

Speaker 0

我们这里那里做了一些事情,但整个行业都受限于计算资源,我们一直在寻找提高效率的方法,以便稍微缓解这个问题,让人们能更多地使用它。

There are things we do here and there, but the whole industry has been compute constrained, and we've always been looking for efficiencies where we can relieve that a little bit so people can use this more.

Speaker 0

所以这就是我们从四月以来所做一切背后的想法,而这次正好赶上了好时机,因为这让一切效率提高了大约一个数量级,从而为其他模型释放了更多计算资源,或者甚至可以用H3 Max生成更多的代币。

So that has been the idea behind everything we've been doing since April, and this just came at the right time because this makes everything maybe an order of magnitude more efficient so it gives more compute for other models or even like more tokens can be generated using H3 Max.

Speaker 0

我觉得巴图汉也会同意这个观点。

I think Batuhan would agree on that.

Speaker 2

比如,从系统级优化的角度来看,这也是我们过去三四年一直在做的事情,你也许能让模型快两倍、三倍,同时保持同样的质量。

Like, just from a system wide optimizations, which is what we have been doing for the past three or four years, you can maybe make the model 2x, 3x faster while producing the same quality.

Speaker 2

因为归根结底,模型还是那个模型,架构还是那个架构,约束条件也一样,你只是在尽量优化从芯片上能榨出的性能,而这里有一个性能上限。

Because it's, at the end of the day, same model, same architecture, you have the same constraints, you're just trying to optimize what you can get out of the chip itself, and there is a roofline there.

Speaker 2

我们一直在越来越接近这个上限,尤其是最近,因为我们整个团队都在专注于如何从单个芯片上尽可能多地生成视频像素。

And we have been approaching that roofline more and more, especially lately, because our entire team has been focusing on how do we get out more video pixels from a single chip as much as possible.

Speaker 2

而这套新的基于后训练优化的方法,加上系统与模型的协同设计,让我们能够超越那个上限,提升一个数量级。

And this new set of post training related optimizations with systemmodel co design enables us to go beyond that roofline by an order of magnitude.

Speaker 2

我们确实感受到了压力,之前我们一直在开源图像模型上做这些工作,在视频模型之前,我们用ideogram做过一个版本,用flux也做过一个版本,所以我们一直在实验如何构建后训练基础设施,拿一个现有模型,围绕它构建内核和系统设计,让它运行得非常非常快,得到一个专门的版本,能击败任何其他直接运行这个模型的方式。

And we just felt the pressure, we have been working on it on top of open source image models before the video models, we did one version with ideogram, we did one version with flux, so we have been experimenting with how can we build post training infrastructure to take an existing model, build kernels and systems design around it, to run it very, very fast for a specialized version that can beat anything else that we would get just by running the model itself.

Speaker 2

而且,你知道,把这些结合起来,再加上我们手里有了前沿视频模型和所有这些专业知识,我们就能在速度上实现一个数量级的提升。

And, you know, combination of that plus just getting a frontier video model on our hands and all this expertise, were able to go by like an order of magnitude in terms of speed.

Speaker 1

太厉害了。

Incredible.

Speaker 1

我们来深入聊聊这个。

Let's dig into that.

Speaker 1

我可能会记错一些数字,但是——

I may get some of the numbers wrong, but-

Speaker 0

有关于效率的数字,有关于成本的数字,还有关于加速倍数的数字。

There's like efficiency numbers, there's cost numbers, there's speed up numbers.

Speaker 0

不是每一项都直接代表效率,但加起来整体上非常高效。

Not everything means efficiency, but it all adds up to be very efficient.

Speaker 1

我觉得最让我震惊的是,成本低了大概一个数量级,而且速度也快了很多。

I guess what is stunning to me is there is like a magnitude lower cost and also much faster.

Speaker 1

我记得好像是35倍的加速,

I think it was like 35x Compared speed

Speaker 2

相比原来的MiniMax版本。嗯,

to the original MiniMax entry Well,

Speaker 1

在ELO评分上,你并没有牺牲质量。

at the ELO score, you didn't really sacrifice quality.

Speaker 1

所以,能不能稍微透露一下背后的秘诀?这主要是你之前做的那种系统工作吗?

So yeah, just maybe reveal a little more of the secret sauce behind of, is this more of the type of system work you have done?

Speaker 1

比如,需要改变模型架构吗?

Like, you have to do like model architecture change?

Speaker 1

还是说,就是那些真正降低了成本和延迟的工作?

Is this just work that really brought down the cost and latency?

Speaker 1

那下一代芯片,比如GB200,在整个故事里又扮演什么角色呢?

And how about the next generation of chips, like GB200 fits into the whole story?

Speaker 2

这其实就是我们一直在优化的多个不同变量的叠加效果。

It's just a compounding effect of multiple different optimization variables that we have been targeting.

Speaker 2

首先,显然,从基础模型到一个经过后训练、效率更高的模型。

The first one is obviously, okay, you go from the base model to a model that's post trained to be more efficient.

Speaker 2

对于扩散模型来说,基本上就是怎么从跑50步变成跑20步,对吧?

For diffusion models, this is just essentially how do you go from running 50 steps to running something like 20 steps, right?

Speaker 2

你就是在优化那个流程,但一旦从50步降到20步,质量就会下降,所以你得在优化过程中设定目标:先提升质量,然后再应用优化。我们有一些检查点,质量明显更高,但速度也明显更慢。所以我们最初的做法是,先运行后训练和强化学习流程来提升模型质量,然后再在上面应用优化栈,这样最终结果能达到甚至超过原始模型的质量,同时速度还能快一个数量级。

You're just trying to optimize that pipeline, but as soon as you go from 50 steps to 20 steps, you lose quality, so you need to target in the optimization scene, okay, I want to improve the quality, and then I want to apply the optimization, so we have checkpoints of this that are significantly higher quality, but obviously slower, so what we initially did was, okay, let's run our post training and RL pipelines so that we can improve the model's quality, and then apply the optimization stack on top of it, so that the end result gets you to the same quality, or even higher quality than the original model, but at the same time you're an order of magnitude faster.

Speaker 2

所以大部分提升来自于对模型进行后训练,让它能在更少的步骤下运行,但除此之外,你还加入了所有的内核和系统工程优化,把硬件利用率从40%提升到80%,40%是很多推理工作负载的标准水平,而80%是理论上的MFU,几乎不可能达到,所以你基本上已经达到了硬件性能的极限。

So most of the gains come from post training this model to be compatible, that you can run this on less amount of steps, but on top of that, you add all the kernels and systems engineering work that you do that brings your hardware utilization from 40%, which is standard in many inference workloads, to 80%, and 80% on theoretical MFU, which is impossible to reach, so you're essentially at the roofline of what you can get out.

Speaker 2

而且这些模型并不是简单的,哦,你给一个提示词就能得到视频,它们实际上是一套流水线。

And then these models are not just like a single, oh, you just give a prompt and you get a video they're actually pipelines underneath.

Speaker 2

你需要接收提示词,运行一个LLM,一个非常大的LLM,把提示词扩展成模型最初训练时的格式,在潜空间生成视频,然后把那些潜变量解码回像素,根据工作负载的不同,可能还会涉及放大组件。

You need to take a prompt, you need to run a LLM, like a very large LLM, go expand that prompt to a format that the model was initially trained at, generate the video in the latent space, and then decode those latencies back into pixels, and then depending on the workload, there might be an upscaling component involved.

Speaker 2

所以有多个组件,而每个组件默认都是未优化的。

So there's multiple components, and every single component by default is unoptimized.

Speaker 2

那里还有很多提升空间,我们只是从每个组件都要榨干最大性能的角度来看待这个问题。

There's still lots to be gained there, and we just looked at it from the perspective of we are gonna get the maximum out of every single component.

Speaker 2

这让我们在LLM上实现了超高速运行,对吧?

This made us go around LLMs at super high speeds, right?

Speaker 2

确实有那个组件,但对于不同的工作负载,这不像那种代理解码LLM的工作负载,那种负载有很高的缓存命中率,有更多的会话。

There is that component, but for a different workload, this is not like something like an agent decoding LLM workload, where you have very high cache rates, where you have higher sessions.

Speaker 2

这是单次请求,给一个提示词,得到一个结果,没有缓存,批量大小也很低,所以在提示词扩展方面有一整套完全不同的优化,扩散模型方面也有一整套完全不同的优化,VAE从潜变量到像素的转换也有完全不同的优化,你把所有这些结合起来,效果就会叠加。

It's a single shot, give a prompt, you get a prompt back, and there's no caching, you're operating at low batch sizes, so there's a completely different set of optimizations on the prompt expansion side, completely different set of optimizations on the diffusion model, completely different set of optimizations on the VAE that you take from latents to pixels, and you just combine all of these to have an effect that compounds.

Speaker 2

从硬件角度来看,从Hoppers升级到BlackVals,本身就能带来大约2到3倍的性能提升,但从成本角度看,两者差不多,因为成本也差不多在这个范围内。所以我觉得这主要是缩短了你的实际运行时间,而不只是提升效率本身,但如果你想要远超实时性能,那显然是有帮助的。

From a hardware standpoint, going from something like Hoppers to BlackVals, you see something like 2x to 3x improvement by itself, but from a cost standpoint, it's like pretty comparable, because the cost is also like in that leak, so I would say it only reduces your like wall clock time, but not just the efficiency itself, but it obviously helps if you wanna go significantly beyond real time.

Speaker 2

如果你想在不到三秒或两秒内生成五秒钟的视频,那今天就需要用到这些最新一代的硬件才能实现。

If you wanna generate five seconds of video in less than three, two seconds, then you need some of these latest generation hardware today to unlock that possibility.

Speaker 1

这可能是个细节问题。

Maybe this is a detail of a question.

Speaker 1

这个模型是在单个GPU上运行的吗,还是……

Is the model being served on single GPU or

Speaker 2

目前大多数视频模型都是在单节点配置下运行的,也就是八块GPU。因为一旦超过八块GPU,由于通信开销,效率会越来越低。现有的MiniMax H3端点以及其他视频模型可能都是从单节点配置开始的,这个也一样。

It is, majority of the video models today run-in a single node configuration, which is is eight GPUs, because once you start scaling beyond eight GPUs, the efficiency gets less and less because of the communication overhead, and existing both, like, the existing MiniMax H3 endpoints, as well as, like, other video models are probably getting started at like, know, single node configuration, same with this.

Speaker 2

它是在八块GPU上并行运行的。

It's like running on in parallel across eight GPUs.

Speaker 1

你觉得在效率方面还有提升空间吗?比如通过牺牲一些用户体验,比如缩小输入输出的类型范围,来优化中间的步骤,或者在并行化方面,还有没有更多潜力可以挖掘?

And do you think there'll be more efficiency gains in there that you can either, optimize more of the steps in between by sacrificing maybe some of the, like narrow down the user experiences, let's say like the different type of inputs and outputs, or like as you're thinking of parallelism, like is there more juice to squeeze?

Speaker 1

可能这就是……

Maybe that's the

Speaker 2

问得好。

best question.

Speaker 2

我们发布了H3 Max的Turbo版本。

We released a turbo version of H3 Max.

Speaker 2

最初的想法是叫它H3 Turbo,但我们觉得不能叫Turbo,因为质量比原版还好。

So initial idea was calling this H3 Turbo, and we were like, we don't wanna call this turbo because the quality is better than the original one.

Speaker 2

这个名字得体现出这个成就有多厉害,所以我们发布了H3 Max,但一周后,团队说,我们可以以97%的质量水平跑出2倍的速度。

This needs to signify how good of an achievement it is, so we released H3 Max, but a week later, our team was like, We can run the 2x faster at ninety seventh percentile of quality.

Speaker 2

我们跑了评估,结果几乎一样。

We run EVALs, they're almost the same.

Speaker 2

质量上有一点明显的损失,但我们有一个公开的版本叫HD Max Turbo,能在1.5秒内生成5秒的视频,这简直不可思议,而且成本也降低了一半。

There's a small noticeable loss in quality, but we have a version called HD Max Turbo that's public that can generate like a five second video in like one point five seconds, which is like insane, and that's also like 2x the from a cost standpoint, it's also like 2x less.

Speaker 2

所以这取决于你对质量损失的接受程度,你知道的,你可以选择降低质量,而且现在这些模型已经这么便宜、这么快了,我觉得人们不需要更快或更便宜的了,从成本角度看,相比前沿模型,已经便宜了一个数量级,从速度角度看,也快了一个数量级,你就能实现所有体验。

So they're like depends on like how how okay you are with like losing quality, you know, you can go down, and like today these malls are so cheap and so fast that I don't think people need any faster or like any cheaper, like it's already like at a point where from a cost standpoint compared to the Frontier itself, it's an order of magnitude cheaper, compared from like a speed perspective it's more than an order of magnitude faster, and you just enable all the experiences.

Speaker 2

我觉得我们需要看看人们还需要什么其他杠杆,但我现在的判断是,在目前的速度下,我们更需要提升质量,而不是速度。

I think we would need to see what else levers that people would need, but my bet today is we just need to improve quality more than the speed at these speeds.

Speaker 2

就是说,我们把速度搞定,然后努力提升这些模型的质量和可控性,这也是我们过去两三周一直在推的方向。

Like, let's fix the speed, and let's try to push for quality and controllability of these models, which is, you know, what we have been pushing in the past two or three weeks.

Speaker 0

我觉得可控性很关键。

I think controllability is is key.

Speaker 0

比如我们最开始做的是文字生成视频,然后是图片生成视频,后来才加入参考图功能,这大大增强了可控性,现在基本上大家用这些模型都是默认用参考图。

Like, when we first did it, we did text to video and then image to video, and then references came later, which which adds a ton of controllability, and it's basically the default mode how people use these models these days, references.

Speaker 0

然后我们现在也在给基础模型加各种LoRA微调版本。

And then we are now adding different LoRa's fine tunes of of the base model as well.

Speaker 0

我们还在做口型同步的版本。

We are working on like a lip syncing version.

Speaker 0

我们在做不同镜头角度的LoRA,还有不同风格的LoRA。

We are working on different different camera angle LoRa, different style LoRa's.

Speaker 0

再说,这给模型周围建立了一个完整的生态系统,真的非常有用。

Again, source adds a whole ecosystem around the model, and it really, really helps.

Speaker 1

你对这个模型后训练带来的速度提升和效果增益感到惊讶吗?

Were you surprised by the speed up and the gain you could get from post training this model?

Speaker 1

我觉得挺意外的,比如有一天,我记得是个周六,我发布了模型,结果周日就有人把它放到Twitch上,变成了实时模型。

Like, Like I saw it as a little bit of a surprise that like one day, I think it was a Saturday, I launched the model and the Sunday people put it on Twitch, become a real time model.

Speaker 1

我觉得当你减少某些东西的时候,这才是最有趣的部分。

Like what, think that's the interesting part of like when you reduce.

Speaker 0

我们做了评估。

We did evals.

Speaker 0

我们自己花了很多钱做评估。

Like we spent a ton of money doing evals on our own.

Speaker 0

我不知道,甚至花了几万美元,结果简直难以置信。

I don't know, like tens of thousands of dollars even, and, like, the results were unbelievable.

Speaker 0

然后,原本的计划是不做外部评估就直接发布模型,但后来我们决定先等等。

And then, like, the plan was to just release the model without doing external evals, and then, okay, we decided let's let's hold off.

Speaker 0

我们先别告诉别人这个模型快了多少、好了多少,除非我们有一些外部的验证。

Let let's not let's not tell people that this is, like, so much faster and so much better before we have some external validation.

Speaker 0

所以我们等了三四天,让其他评估平台真正跑一下评估。

So so we waited, like, three, four days to all these other, like, eval platforms to actually run the eval.

Speaker 0

所以我们的外部测试结果也一致,就这样我们才发布的,因为就像你说的,结果好得有点不真实。

So we matched the results that we have externally as well, and that's how we launched it because as you said, the results were a little too good to be true.

Speaker 1

确实如此。

And it was.

Speaker 1

我想,让你感到意外的是由此产生的实时应用场景吧,或者你觉得这个模型解锁了哪些以前模型做不到的体验?

I guess, well, you're taking it by surprise that the real time use case that came out of it, or what are some examples that you think this model has unlocked of the experiences

Speaker 0

对,以前模型做不到的。

that prior models couldn't?

Speaker 0

这种情况在 fal 每隔几个月就会发生一次,整个公司都会沉迷于某样东西,然后创意就爆发了,每个人都在开发新的小应用,或者做不同的优化,比如 LoRA 之类的。

This happens at fal once in every couple of months where, like, the whole company gets gets hold of something and and the creativity just explodes, and everyone is just working on a new little app or or a a different optimization, Laura, whatever it might.

Speaker 0

整个公司都围着这个模型转,一些前端工程师开始做有趣的应用。

Like, the whole company gathered around this this model, and, like, some some front end engineers started working on, like, interesting applications.

Speaker 0

我们可以聊聊我们的世界模型加速团队,这是个全新的团队。

We can talk about our, like, world model accelerator team, which is which is brand new.

Speaker 0

他们开始研究实时处理。

They they started working on the the live

Speaker 1

体验。

experience.

Speaker 1

实时通信。

RTC.

Speaker 0

WebRTC实时体验。

The WebRTC live experience.

Speaker 0

所以,公司内部有五六七个并行的小项目。

So, like, there were five, six different parallel little projects within the company.

Speaker 0

我觉得那天我们在Slack上打破了记录,因为公司里发了太多消息,而且我们是个分布式团队。

And, like, I think we broke a record on Slack that day how many messages were were sent in the company because like and and and, like, we have a distributed team.

Speaker 0

我们的人遍布世界各地,主要在旧金山,但当你看到24小时不间断的开发时,真的很惊人,比如有人连续工作十六七个小时,然后另一个人醒来接手继续干。

We have we have people all around the world, like, mostly in San Francisco, but it's it's, like, incredible when you see, like, the twenty four hour development, like, when people, like, work sixteen, seventeen hours, and then someone else wakes up and picks up that.

Speaker 0

就这样持续了三四天。

Like and that went on for, like, three, four days.

Speaker 0

然后我们就发布了所有这些项目。

And that's that's when we released all these projects.

Speaker 1

给我讲讲那个吧。

Take me into that.

Speaker 1

这太有意思了,因为你想象一个模型或产品发布,都是提前规划好的,所有评测供应商都准备好,排好队,然后发布出去。

It's so interesting because you imagine a model or product launch being planned out, having all these eval vendors being ready, lined up and ship something out.

Speaker 1

然后你让全世界或者外部用户去使用它,去实验,去创建体验并放到网上。

And then you let the world or the external users take it and then experiment and build experiences put online.

Speaker 1

对,看起来公司内部那些很有创意的人,直接拿这个,放下手头所有事情,发布了一个在推特上特别火的体验。

Yeah, it seems like people internally who are very creative just took this, dropped everything they were doing, launched an experience that got really popular on Twitter.

Speaker 1

你想跟我们说说那个吗?

Do you wanna tell us about that one?

Speaker 0

当然。

Of course.

Speaker 0

我们有个工程师,叫Rehan,就他一个人,完全自己开始从他的笔记本电脑上直播H3 Max的连续生成。

One of our engineers, Rehan, just just by himself completely started streaming a livestream of continuous generations of H3 Max from his laptop.

Speaker 0

就像,他的电脑……

Like, he was His computer.

Speaker 0

他的电脑。

His computer.

Speaker 0

没错。

Exactly.

Speaker 0

他当时在玩一些提示词技巧,想保持一个连贯的故事,然后就开始在Twitch上直播了。

He was doing some, like, prompt tricks, trying to keep, like, a coherent story, and then, like, he started livestreaming that on on Twitch.

Speaker 0

与此同时,现在已经是知名Twitter网红的Yep。

In parallel, famous Twitter influencer at this point Yep.

Speaker 0

他也有类似的想法,而且联系了我们,说他已经准备好了一个网站。

Had a had a similar idea, and he reached out to us that he has a website ready already.

Speaker 0

他想自己直播,并且搞一个能无限生成的网站。

He wants to, like, post the streaming himself and and have have a website that does infinite streaming.

Speaker 0

在公司内部,还有另一个团队在做H3 Max的连续版本。

Internally, also, we had another team who was working on a continuous version of h t max.

Speaker 0

所以H3 Max就是Rehan的版本,还有Levels的版本。

So h t max is, like, the the Rehaut's version and and Levels.

Speaker 0

版本是独立的片段。

Version were independent clips.

Speaker 0

速度还是很快,但片段有开始有结束,然后你取片段的最后一帧,试着把它放到下一个片段里,努力让最后一帧延续下去。

It's still very fast, but the clip starts, it ends, and then you take the last frame of the of the clip, try to put it in the next one, and try to create a continuous work into, like, the last frame.

Speaker 0

没有记忆。

There's no memory.

Speaker 0

比如,第二个片段除了最后一帧,其实不记得第一个片段的任何内容。

Like, the the second clip doesn't really remember anything from the first clip other than the last frame.

Speaker 0

但在内部,我们的机器学习团队在做一个版本,过渡更自然。

But internally, we were the the ML team was working on a version where it's the the transition is more seamless.

Speaker 0

有两分钟的记忆。

There is, two minutes of memory.

Speaker 0

所以,比如你身处一个场景,当你指挥模型或者有人走进房间,实际上每个人都会看向那个进来的人,场景是连续的。

So, like, you're in a scene, and when when you direct the the model or someone else enters the room, it actually like, everyone looks at that person entering, and the the scene is continuous.

Speaker 0

所以我们在内部就做这个。

So internally, we were working on that.

Speaker 0

然后另一个团队在做一个我们叫“fall live”的体验,是那个连续版本的。

And then another team was working on an experience we we called fall live for the continuous version.

Speaker 0

所以我们有三条线在同时推进,而且每条都在推特上独立爆火。

So we had three parallel efforts going on that were all independently going viral on Twitter.

Speaker 1

这些都是自发发生的。

And these were all spontaneous.

Speaker 1

你

You

Speaker 0

计划

plan for

Speaker 2

它

it at

Speaker 1

针对任何一个

for any of

Speaker 0

它们

them.

Speaker 0

对。

Exactly.

Speaker 1

然后它们就在接下来的几天里变成了产品和体验。

And they just became products and experiences in the following days.

Speaker 0

但是,Toren,我们来聊聊我们是怎么让模型变得更连续的。

But, Toren, let's talk about how we made the model more continuous.

Speaker 0

这让我很惊讶,因为我以前从没见过视频模型能做到这样。

That was very surprising to me because I've never seen that actually work on a video model before.

Speaker 2

是的。

So yeah.

Speaker 2

我们回到之前,我们一直非常专注于角色模型,基本上就是动作控制,或者说,你知道的,动作驱动的实时连续视频流。

We so going back, we have been like very, very focused towards role models, and essentially like action controlled, or like, you know, action driven real time continuous streams of video.

Speaker 2

而问题在于,像H3 Max这样的东西,质量完全不够好。

And the problem till like, you know, something like H3 Max was quality was not good enough at all.

Speaker 2

它退化得很厉害,记不住之前的内容,但我们把基础设施建好了。

It degraded a lot, it didn't remember the past before, but we built the infrastructure.

Speaker 2

我们搭建了基础设施,可以实时流式传输视频,让人们实时控制,还能同时分发给多个人,延迟非常低。与此同时,我们的机器学习团队基本上在尝试把每一款视频模型都拿来,应用这一系列的优化和技巧,看看能不能把生成时长从五秒延长到十五秒、三十秒。

We built the infrastructure that we can go stream video, have people control it in real time, being able to multiplex it to multiple people, very low latency, and at the same time, our ML team was essentially trying to take every single video model and try to apply this set of optimizations and tricks to, okay, how can we make this journey instead of a five second video, fifteen second video, thirty second video?

Speaker 2

但之前总是达不到实时生成的要求,你总是没法在五秒内生成五秒的视频。

But you were always below the real time factor, where you were always like, you never could generate five seconds under five seconds.

Speaker 2

一旦H3 Max突破了这一点,机器学习团队就震惊了,说这太不可思议了。其实我们内部是分开的团队。

Once the H3 Max unlocked it, the ML team was like, this is insane, which are separate teams internally.

Speaker 2

我们有研究团队。

We have a research team.

Speaker 2

有推理团队。

Have an inference team.

Speaker 2

我们有机器学习团队。

We have an ML team.

Speaker 2

他们看到这个结果后,都觉得太疯狂了。

They're like, they they saw this, and this is insane.

Speaker 2

我们可以把之前模型上尝试过的所有经验都应用过来,不再试图生成五秒的片段,而是尝试生成五十秒或十秒的视频,同时前一个五秒的内容仍然被关注到。

We can apply all these set of learnings that we had in previous models where we attempted to do this, where instead of trying to generate a five second chunk, let's try to generate a fifty ten second video, and then the five seconds from previous one is still attended.

Speaker 2

我们还记得它。

We still remember it.

Speaker 2

随着视频的推进,我们可以把记忆扩展到两分钟。

And as the video goes up, we can extend that memory up to two minutes.

Speaker 2

而且你需要做非常巧妙的优化,因为处理两分钟的视频计算量极大,而且从计算角度来看,复杂度是指数级增长的。

And you need to do extremely clever optimizations because attending to a two minute video is just extremely, extremely compute intensive, and just like it goes up exponentially from like a compute standpoint.

Speaker 2

所以我们在这方面做了很多优化,但最终我们能够做到记住两分钟的内容,从记忆角度来看这通常已经足够了。而且通过提示词轨道,你还可以在两分钟之后继续记住更精细的细节,基本上可以无限流式生成,我们把它保持在一小时。然后那个团队就以H3 Max Director的名义发布了这个模型,供公众使用,我认为这是唯一一个能够生成长达六十分钟连续视频并且支持动作控制的模型。

So like we did like lots of optimizations there, but at the end of the state we were able to okay, we can remember back to two minutes, which is like generally good enough from a memory perspective, and then obviously with like prompt tracks you can still like continuously remember more finer grain details above the two minute mark, and can essentially stream infinitely, we kept it at an hour from that perspective, and then that team just like released that model under H3 Max Director, which is public for people to use, and I think it's the only model that can generate like you know up to sixty minutes continuous videos that is action controlled.

Speaker 2

你可以从一个提示开始,比如一个办公室场景,有人在办公,然后三十秒后,它会自己想象。

You can start with a prompt, there's an office setting, and someone is working, and then thirty seconds later, it just imagines by itself.

Speaker 2

三十秒后,你可以说,一个女人走进门。

Thirty seconds later, you can say, a woman walks through the door.

Speaker 2

它能立即接受提示并反映出来,这是最有趣的部分。

It can take the prompt and reflect it immediately, which is the most fun part.

Speaker 0

而且办公室还是同一个办公室。

And the office is still the same office.

Speaker 0

同一个办公室。

Same office.

Speaker 0

镜头可以转回原来那个人,那个人还在那里,状态也没变。

The camera can pan back to the original person, and the original person is still there in the same state.

Speaker 2

所以我们发布了那个模型,还搞了个秋季直播网站来展示,因为大家得亲眼看看这有多酷。

So, you know, we we we released that, and it it got like, we we we did this fall live website to just like demonstrate it because it's like, people need to see how cool this is.

Speaker 2

这是项新技术。

This is a new technology.

Speaker 2

我觉得大家还没真正意识到这一点,而且它很快就火了,因为我们还让大家投票决定下一个场景是什么。

I I don't think people are like really aware, and it it got also like very viral immediately because we also let people vote on what the next section is.

Speaker 2

就像是一种众包的形式。

It was like, you know, like a form of a Crowdsource.

Speaker 2

众包。

Crowdsource.

Speaker 2

聊天室在控制剧情走向,这挺有意思的,但显然我们限制了选项,然后他们可以选,比如“一个香蕉走进办公室”而不是别的,这样更有趣,大家就开始玩起来了,然后我们加了更多频道,每个频道都有自己的主题。

The chat was controlling what was happening, which is fun, but obviously, you know, we limited on, like, the options, and then they could pick, oh, like a banana enters the office instead of a moment, and it's more fun, and people start having this, and we start adding more channels, and every channel had a concept.

Speaker 2

有一个频道完全是混乱的,还有一个频道是80年代的卡通,这个模型非常厉害,能记住很多不同的概念。

There's a channel where it's full chaos, there's a channel where it's cartoons from '80s, and the model is extremely capable, and it just like remembers like so many different concepts.

Speaker 2

而且它从风格和概念的角度来说,记忆量非常大。

And there's like it has like a big, big memory from from like a styles and like, you know, concept perspective.

Speaker 2

所以这整个体验就变得非常有趣。

So it it just became like a very fun experience underneath.

Speaker 1

再说,从这里面涌现出了很多非常棒的作品,比如H3 Max导演对我来说就是另一个巨大的惊喜。

Again, like there are so many really incredible experiences coming out of this, like H3 Max director was just another huge surprise to me.

Speaker 1

我觉得在生成媒体市场里,有趣的是,语言模型并不是一条直线,持续地在智能能力上叠加。

It's like, I found it interesting in gen media market that you, it's not like language model have this linear graph of just continuously compounding on intelligence capability and so on.

Speaker 1

感觉在你这个领域,总是有几个月比较安静,但很多东西在酝酿。

Feels like in the field you're operating in, it's always like a few months of sort of quiet time, but a lot of things are bubbling.

Speaker 1

但在很短的时间内,一切都会爆发。

But in a very short period of time, everything bursts.

Speaker 1

所有这些因素结合在一起,基础模型足够好。

All these things in combination come together of the base model being good enough.

Speaker 1

比如你可以把延迟降到很低,低到能……

Like you can get the latency down to the point where you can like

Speaker 0

参考素材。

References.

Speaker 1

就是既能获得实时体验,又能在这种实时体验之上施加控制。

Like get the real time experience, but also like apply controllability on top of that real time experience.

Speaker 1

这真的为现场体验打开了无数可能性,让最终用户能控制屏幕上发生的一切,太不可思议了。

Like this just opens so many opportunities of live experiences where end user can control what's happening on the screen, which is incredible.

Speaker 1

我们以前想象过很多这样的体验,但从来没能真正上手玩一玩。

We have imagined a lot of these experiences, but never been able to really play around with it.

Speaker 1

能不能多跟我们聊聊你在市场上看到的,大家是怎么用这个导演功能的?

Maybe just tell us more about what you're seeing from the market of how are people using the director capability?

Speaker 1

你看到创作者们做出了哪些以前没见过的东西?

What are you seeing creators are creating that you haven't seen before?

Speaker 1

你觉得这能解锁哪些新的可能性,让人们能用这种媒介做些什么?

And what do you think that unlocks as far as what people can do with this medium?

Speaker 0

H3 Max 发布差不多三周了,它已经是平台上最受欢迎的视频模型了,在文件平台上的使用量几乎是第二名的两倍多。

It's been like almost three weeks since we released H3 Max and already it is the most popular video model on the platform, on on the file platform by, like, double almost, like, little more than double in terms of, like, volume.

Speaker 0

在很多其他平台上,它也成了大家默认使用的模型,因为它又快又便宜,用起来很合理。

So in in a lot of other platforms, it's also becoming the default model that people interact with because it's so fast, so cheap, it just makes sense.

Speaker 0

如果你来到一个平台,这就是你想看到的体验。

If you come to a platform, this is the experience that you wanna see.

Speaker 0

所以从受欢迎程度和使用量来看,至少在我们看来,它正在占据主导地位。

So in terms of popularity and volume, it's taking over at least from our vantage point.

Speaker 0

至于 Max Director,已经有好几十个不同版本的直播了。

And for Max Director, again, there has been, I don't know, tens of different versions of these livestreams.

Speaker 0

其中一些还在继续,而且越来越受欢迎。

Some of them are still going on and like becoming more and more popular.

Speaker 0

我们正在和一些 AI 知识产权持有者合作,比如那些在 Instagram 和 TikTok 上有 AI 节目的人,我们会根据他们的风格训练一个 LoRA,然后做他们节目的直播版。

We are trying to work with some like AI IP holders, people who have like AI shows on Instagram and TikTok and and do train a Laura on on their style and do a live version of their show.

Speaker 0

我们已经有好几个在筹备了,所以这会非常令人兴奋。

So we have couple lined up already, so that's gonna be very exciting.

Speaker 0

而且,如果你跟创意技术专家聊,语音提示已经成了他们经常用的东西。

And, like, the way people like, if you talk to creative technologist, prompting voice has already become something that, like, they use all the time.

Speaker 0

就像用 WhisperFlow 或者 ChatGPT 的语音模式一样。

It's like using WhisperFlow or the ChatGPT voice mode.

Speaker 0

对。

Yep.

Speaker 0

现在呢,你可以一直跟模型对话,它几乎就像真实片场里的导演,指挥镜头,指挥人往哪儿走,你都能做到。

And and now, like, you can keep talking to the model and it it like, almost as if it's a real director in a real movie set directing, like, the camera, directing people where to go, you you can do that.

Speaker 0

而且,我们的创意工程师已经开始这样用这些模型了。

And, like, our creative engineers started using these models like that.

Speaker 0

所以,我们说话这会儿,就会有很多有趣的体验被开发出来。

So we'll see, like, a lot of interesting experiences are built as we speak.

Speaker 1

很有意思。

Very interesting.

Speaker 1

就是说,视频正在播放的时候……

As in, like, the video is playing on

Speaker 0

视频正在播放。

Video is playing.

Speaker 0

你在跟视频对话,显示的内容会随之变化。

You are, talking to the video and what's being displayed changes accordingly.

Speaker 1

这太不可思议了。

That's incredible.

Speaker 1

说到记忆功能现在是怎么实现的,这可能有点技术性。

And talking about how the memory piece holds now, again, this may be a technical detail.

Speaker 1

它记住上一幕或最近几分钟场景的能力,是通过画面、图像来记住的,还是通过文字?

The capability of remembering what happened in the last scene or in the last couple minutes of scene, like, are you remembering that through like the frames, the images, or is it like through text?

Speaker 2

基本上——不是,它基本上是通过视频来记住的,当然,信息被压缩得很厉害,因为你不可能处理整个视频,但它基本上知道过去两分钟内自己生成的大部分细节,超过两分钟的部分,你可以把它想象成一个不断演进的系统提示,覆盖从两分钟到六十分钟的内容,它知道整体的结构和细节。

Essentially- Against No, it's essentially like it remembers throughout video, obviously very, very compressed because you can't attend to the fall video, but it essentially knows like most of the details happened in the past two minutes from its own generations, and above the two minute mark, has like think of it as like a it has an evolving system prompt on top of the two minute mark from two to sixty minutes where it knows like the overall structure, overall details.

Speaker 2

所以它能记住最近几个场景,如果一个场景大概十五到三十秒,那它就能记住最近四到八个场景,然后在此基础上,还有一个持续演进、逐步更新的系统提示,用来保持整个世界的连贯性。

So it remembers like the last few scenes, if you think a scene is like fifteen thirty seconds, then it remembers like the last four to eight scenes, and then on top of that there's like a continuously evolving, gradually evolving system prompt that like keeps remembering the, like, the overall coherence of the of the world.

Speaker 0

大家都在期待AI迎来一个大规模消费级应用的时刻。

Everyone's waiting for a a large consumer moment in AI.

Speaker 0

现在它已经足够好、也足够便宜,可以在这个基础上构建真正新颖的社交AI体验了。

Now it's, like, good enough and and cheap enough that, like, a truly novel social AI experience can be built on top of it.

Speaker 1

那我们多聊聊经济方面吧。

Maybe let's talk more about the economic side of this.

Speaker 1

比如,我想问一下,用H3 Max处理同样时长的视频,服务成本大概是多少?

Like what is the, I guess one, just like talking about serving cost for like same minutes of video with H3 Max.

Speaker 1

这又是怎么改变你对芯片库存和规模的想法的?

And how has it changed your thinking around like your footprint of inventory of chips?

Speaker 1

你希望如何为终端用户提供不同层次的体验?

Like how do you want to have different steps of experiences serving to the end user?

Speaker 0

对,我稍微提过一点,大家都说下一代大语言模型很复杂,但其实视频模型也非常复杂,因为整个流程包含不同的组件,有时候为了效率,它们需要不同的硬件配置,等等。

Yeah, but I mentioned this a little bit, like everyone talks about how complex the next generation LLMs are, but video models are actually very complex as well because the pipeline has different components and sometimes they require different hardware configuration for efficiency, things like that.

Speaker 0

所以,如果想做得更高效,甚至更便宜,我们可能会在不同类型的硬件上运行流程的不同部分。

So if you were to do this even more efficient, let's call it maybe even cheaper, we would probably run different parts of the pipeline in different types of hardware.

Speaker 0

另一个有意思的方向是,让它在消费级硬件上运行,这样人们就可以在自己家里的机器上使用。

Another interesting thing would be to run it on consumer hardware for people to run it in their own machines at home.

Speaker 0

这些优化虽然不能百分之百转化,但也能转化得差不多。

Like optimizations don't translate 100%, but translate somewhat close to that.

Speaker 0

然后我们可以再做额外的工作,让更多部分得到转化。

And then we can do extra work to translate more of it.

Speaker 0

所以,在不同类型的硬件上做这些优化,并把整个流程组合得更高效,我觉得这就是我们接下来几周要做的事情。

So doing these optimizations in different types of hardware and combining the pipeline in a way that it's even more efficient, I think that's what we are gonna do in the next coming weeks.

Speaker 1

太棒了。

Amazing.

Speaker 1

所以,你会让像Rohan这样的人能够部分地流式体验。

So you will have people like Rohan that can stream partially of the experience

Speaker 0

但从他的

but from his

Speaker 1

同时,导演和控制平面更多是存在于,对。

also having like the director and the control plane more living on the yeah.

Speaker 1

有道理。

Makes sense.

Speaker 1

我们聊了这款模型能为消费者带来哪些体验,看起来Batuhan对从GPU中榨出的效率很满意。

So we talk about all the consumer experiences this model could unlock, and it seems like Batuhan is happy with all the efficiency, like squeeze out of the GPUs.

Speaker 1

现在我们更多在讨论如何提升这些模型的质量和可控性,以便高端市场的好莱坞创作者和导演能把它提升到新的高度。

Now we're talking more about how do we like improve quality and controllability of these models so that the high end of the market, the Hollywood creators, directors can take this to the next level.

Speaker 1

我碰巧看到一些演示,这款模型和Astra在同一周或前一周发布。

I saw some demos coincidentally, this model came out the same week or week prior to Astra.

Speaker 1

人们把Blender的体验和fal的H3 Max结合起来。

People were combining the Blender experience with H3 Max from fal.

Speaker 1

比如,谈谈

Like, talk

Speaker 0

它将如何影响好莱坞。

about how it's gonna impact the Hollywood world.

Speaker 0

将Blender与这类AI模型结合使用,是专业工作中非常流行的工作流程。

Using Blender with one one of these AI models together is a extremely popular workflow for for professional work.

Speaker 0

基本上,你用Blender和之前的非AI技术,先渲染出场景的低分辨率版本,也就是你想做的内容。

Basically, you you you render a low resolution of your scene, what you wanna do using using Blender, previous, like, non AI technology.

Speaker 0

然后,一旦你把那段视频作为参考输入到AI模型里,基本上就能达到接近100%的可控性。

And then once you add that video as a reference to an AI model, you basically get close to 100% controllability.

Speaker 0

这对VFX艺术家来说是一个非常流行的流程,那些专业做这个的人,因为他们想要精确地得到他们输入到模型里的东西。

And this this is an incredibly popular workflow for VFX artists, people who are doing this professionally because they want exact they they wanna get exactly what they put in into into the model.

Speaker 0

正如你提到的,在我们发布H3 Max一周后,人们就开始用GPT Astra这样的LLM模型在Blender里创建和生成这些漂亮的场景,这突然解锁了一个全新的流程,用LLM来创建Blender场景,然后把它传给H3 Max模型或任何视频模型。

And as as you mentioned, a week after we we launched H3 Max, people started creating, generating these beautiful scenes using an LLM model GPT Astra in Blender, and all of a sudden it unlocked a whole new pipeline using LLM to create a Blender scene and then passing that to the H3 Max model or or any video model.

Speaker 0

但它在H3 Max上效果特别好,因为速度极快,你可以同时并行尝试很多东西。

But it it works very well with H3 Max because it's extremely fast, and you can, like, try many things all at once in parallel.

Speaker 0

这为好莱坞和专业人士解锁了全新的工作流程,让你能达到接近100%的可控性。

And that that unlock the whole new workflow for Hollywood and professional people, and it gets you to, like, close to 100% controllability.

Speaker 2

就像我说的,我们一直非常专注于速度、性能和质量,现在我们有了一个非常好的基础模型。

As as I said, we have been very, very focused towards speed, performance, quality, and now we have a really good base model.

Speaker 2

我觉得接下来的一个月或两个月会完全专注于,我们能给这些模型增加多少可控性,这样工作室的专业人士,那些想要真正制作出完全符合他们使用场景的内容的专业人士,就能利用这些模型。

I think the next month or two is gonna be fully focused on, okay, how much controllability we can add to these models, so that professionals at studios, professionals who want to actually produce content that fits their use cases perfectly can leverage these models.

Speaker 2

团队一直在开发一个很棒的唇形同步模型,你只需要提供音频,再提供一个视频或图像参考,它就能完美地同步嘴唇动作。

The team has been working on an amazing lip synchronization model, where you can just supply the audio, you can supply a video or an image reference, and then it can like synchronize the lips perfectly.

Speaker 2

动作控制也是一样的。

Same with like motion controls.

Speaker 2

你可以直接拿一段某人跳舞的动作,套用到你AI生成的角色上,效果会非常完美。

You can just take a motion of someone dancing and apply it to like your AI generated character, and it fits perfectly.

Speaker 2

而且这些结果基本上靠简单的提示词就能得到,可靠性大概有80%到90%。

And this is like, you can like get these results with like basic prompting, and you're gonna get like 80%, 90% reliability.

Speaker 2

我们的目标是让输出的可靠性达到99.9%,这样你才能真正信任模型在生成的每一个细节上都做得完美,这也是我们一直在努力的方向。

What we are targeting is like 99.9% reliability in the outputs, so that you can actually trust the model did every single aspect of this generation perfectly, and that's like what we've been pushing.

Speaker 2

上周我们发布了一个重要的功能,就是摄像机控制,基本上你可以精确到每一度来控制视频中摄像机的运动方向。

One big launch that we had last week was the camera controls, which is essentially you can direct where the camera is going within the video perfectly to the to to the degree.

Speaker 1

这可以通过在提示词里描述,或者直接生成……

And this is by, like, describing in the prompt or, like, generating the

Speaker 2

我们其实就是在底层提供一个JSON,比如我想让摄像机在t0时刻位于零、零、零的位置。

We just essentially, like, underneath, you give a JSON of, like, I want camera at, like, zero zero zero at t zero.

Speaker 2

我想让摄像机在t1时刻转到90度角。

I want camera at like 90 degree angle at t one.

Speaker 2

基本上就是提供一个结构化的描述,说明你的摄像机在任何时间点需要在哪里,然后模型就会完美地把它当作唯一的信息来源,不会自己乱猜摄像机该往哪走。这样你就能从单个输入重建出三维场景,因为模型本身就是一个非常好的视频模型,但同时它又完全遵循摄像机的指令。

Like, essentially supply a structured description of where your camera needs to be at any point in time, and then the model is like perfectly conditioned to to regard it as like the only source of truth, it doesn't hallucinate where the camera should go, and you can essentially reconstruct three d scenes from a single input, because the model itself is a very good video model, but at the same time, it's perfectly adheres to the camera itself.

Speaker 1

这是因为基础模型本身已经对摄像机角度有了理解,你可以……

And this is because the base model itself already has the understanding of the camera angle that you can

Speaker 2

它不会遵循这个。

It doesn't respect it.

Speaker 2

你需要全部调整。

You need to tune them all.

Speaker 2

你需要对模型进行大幅度的调整,而这正是大规模后训练基础设施所支持的。

You need to tune the model to a significant degree, and this is what enables at large scale post training infrastructure.

Speaker 2

我们现在有了这样的基础设施,可以给H3 Max添加任何功能。

We now have the infrastructure to take H3 Max, add any capability to it.

Speaker 2

这对任何新模型也同样适用。

Same applies for any new model.

Speaker 2

如果有新的视频模型,我们大部分时间其实是花在构建基础设施上,而不是只做一次性的训练,这样我们就能围绕这个来构建服务,不仅针对开源模型,也针对前沿的闭源模型,因为我们在市场上看到最大的缺口就是这些模型的可控性。

If there's a new video model, we essentially spend most of the time building it as an infrastructure than just one off training runs so that we can build services around this for not just open source models, but for frontier, closed source models as well, because we see in the market this is the biggest gap, is just how controllable these models are.

Speaker 2

一开始我们做的是文字转视频,你输入一段提示词,就能得到一段视频,效果不错,但你永远没法描述出你想要的完美角色。后来我们有了图像转视频,先用图像编辑模型生成第一帧,然后模型生成的内容就更贴合了,但你仍然没法说,哦,我希望这个新角色在第三秒出现。

First we start with text to video where you put a prompt, you got a video back, it was good, but you never could describe the perfect character for you, and we had image to video where you used an image editing model and then generated the first scene, and then the model was obviously much more fitting, but you still couldn't say, oh, I want this new character appear at second three.

Speaker 2

你得把它放到第一帧里,或者用提示词去描述,但模型永远做不到完美。

You need to put it to the first frame, or you can prompt it, but this model is never perfect.

Speaker 2

然后我们加入了参考视频功能,你可以提供一个起始帧,还可以指定角色和他们的声音。

And then we added reference to video where you can provide an initial starting frame, and you can also provide, I want these characters with these voices.

Speaker 2

这也是一个很大的突破,你可以直接说,这个角色就用这个声音。

That's also a big unlock where you can essentially say, this is the voice for this character.

Speaker 2

现在我们在加入新的功能,比如在场景中,我希望在零秒时摄像机从这个角度拍摄,在三秒时摄像机转到那个角度。

And now we are adding, oh, within the scene, want camera to look at this degree at t zero, I want camera to look at this degree at t three.

Speaker 2

我们还在加入灯光控制,你可以直接指定光源的方向。

And then we are adding lighting controls where you essentially say where the light is coming from.

Speaker 2

这些功能都是层层叠加的,而且我们现在已经有了统一的架构,可以随时把这些功能应用到任何模型上。

These are all compounding on top of each other, and we just have the unified infrastructure to just apply this to any model at this point.

Speaker 0

这太厉害了。

That's incredible.

Speaker 0

好莱坞是我们增长最快的领域,关于AI可能颠覆好莱坞的讨论很多,但一年前好莱坞的使用量几乎为零。

Hollywood is our fastest growing segment, and there there's a lot of noise about how AI might disrupt Hollywood, but Hollywood usage was nonexistent a year ago.

Speaker 0

在过去一年里,它增长了,现在成了增长最快的领域。

And in in the past year, it grew, and now it's it's the fastest growing segment.

Speaker 0

比如亚马逊、米高梅影业,在他们的会议上,他们发布了他们的NARA工具。

Like Amazon, MGM Studios, in in their conference, they they released their NARA tool.

Speaker 0

这主要靠背后的文件基础设施支持,我们看到了来自好莱坞的巨大需求。

It's it's mostly backed by file infrastructure behind the scenes, and we are seeing incredible, incredible pull coming from Hollywood.

Speaker 0

他们需要的正是这些小的点解决方案,而不是从头生成一切。

And exactly what they need, these, like, small point solutions rather than generating everything from scratch.

Speaker 0

他们希望能够稍微延长视频。

They wanna be able to extend the video a little bit.

Speaker 0

他们希望能够改变摄像机控制。

They wanna be able to change the camera controls.

Speaker 0

他们想要改变灯光。

They wanna change the lighting.

Speaker 0

总得有人帮他们把这些解决方案做出来。

And someone has to build these solutions for them.

Speaker 0

好莱坞需要的、创作者真正需要的,和研究实验室正在做的,这中间有点脱节。

What Hollywood needs and what the creators actually need and what the research labs are working on, there's a little bit of a disconnect there.

Speaker 0

我们觉得我们可以介入,做一些小的后期训练项目来弥合这个差距,因为我们和所有好莱坞工作室都有合作,能听到他们真正需要什么。

And we believe we can come in and do these little post training projects to to close that that gap because we work with all the Hollywood studios, and we hear from them what they need.

Speaker 0

而这些正是他们需要的东西。

And these are exactly the things they need.

Speaker 0

这些小的点状解决方案能真正提高他们的效率,产出更多视频,AI 确实能很好地弥合这个差距。

These small point solutions that actually make them more efficient, push out more more video, and AI can actually close that gap very nicely.

Speaker 1

换个角度说,我们过去三年也一直在盯着这个问题。

Maybe say in a little bit different way, we have been staring at this problem for the last three years as well.

Speaker 1

我们看到很多公司在尝试打造一个电影导演级别的视频模型,要么通过预训练,要么通过后期训练视频部分。

We see companies trying to build a movie director, video model by either pre trained or post trained the video side.

Speaker 1

但我听到的是,不同的人对输出结果的要求表达得非常不一样。

But what I'm hearing is different people expressing the way they want the output to come out very differently.

Speaker 1

普通消费者谈论它、写提示词、生成结果的方式,和好莱坞导演完全不一样,这很明显。

Consumers talk about it and then write the prompt and generate the results very differently from a Hollywood director, which is obvious.

Speaker 1

专业人士想讨论的是这些镜头角度。

Professionals wanna talk about these camera angles.

Speaker 1

他们想讨论的是灯光。

They wanna talk about the lighting.

Speaker 1

你基本上建立了一个后训练的库或集合,比如数据和工具包,可以应用到任何你能拿到权重的模型上,让它们适应不同的受众,让他们能以稍微不同的方式表达创意,从而控制模型。

You sort of have built a library or a collection of post training, like call it data and toolkits that can apply these, any model that you can grab the weights on so that they are adapted to a different audience where they can express their creativity in a bit different fashion to control the model

Speaker 0

完全正确。

100%.

Speaker 1

这能解锁底层的很多能力。

When it unlocks a lot of capability underneath.

Speaker 0

而且问题的一半在于这些模型的能力。

And half the problem was capabilities of these models.

Speaker 0

我们正在解决这个问题。

We are solving that.

Speaker 0

另一半问题则是法律和数据驻留方面的。

The other half of the problem was legal and data residency, things like that.

Speaker 0

我们在这些方面也取得了很大进展。

We we we made a ton of progress there as well.

Speaker 0

现在我们有了一个系统,人们可以用自己的知识产权来申请,我们会在模型中解锁他们的知识产权。

We now have a system of people can apply with with their own IP, and we unlock their own IP in in the models.

Speaker 0

我们会继续扩大这个业务,这将是我们在好莱坞制片厂方面非常有力的一项举措。

We are gonna grow that, and that's gonna be a very powerful thing we do with Hollywood Studios.

Speaker 0

另外,我们现在也在美国托管了C Dance。

Also, we we now have C Dance US hosted as well.

Speaker 0

之前我们已经有了其他中国模型。

We already had previously other Chinese models.

Speaker 0

C Dance是缺失的那一块。

C Dance was the missing part.

Speaker 0

每个好莱坞制片厂都希望我们在美国托管它。

Every Hollywood studio wanted us to have it US hosted.

展开剩余字幕(还有 23 条)
Speaker 0

现在这个已经可以用了。

Now that's available.

Speaker 0

所以现在这些好莱坞制片厂面前已经没有任何障碍了。

So there are no no obstacles in front of these Hollywood studios now.

Speaker 0

一切都准备好了,我们相信他们在接下来的几个月里会把AI的使用量提升十倍、百倍。

Everything is ready, and we we believe they're gonna 10x, 100x their AI usage in the coming months.

Speaker 1

对于电影爱好者、消费者以及大量观看视频和创意内容的人来说,这真是个令人兴奋的时代。

It's such an exciting world for movie lovers, consumers, people who consume a lot of video and creative content.

Speaker 0

我们还有自己的大会,就是生成式媒体大会周。

And we have our conference, generative media conference week.

Speaker 0

这是我们第二次举办这个活动。

This is our second time we are doing it.

Speaker 0

去年主要聚焦在消费级AI上。

Last year, it was mostly consumer AI.

Speaker 0

当时只有少数几个好莱坞高管出于好奇来参加,而现在,制片厂已经成了主力。

There were, like, maybe couple Hollywood executives here and there just curious about it, and now it's dominated by studios.

Speaker 0

现在有很多新的人工智能工作室,它们是大制片厂的分支,想专门做纯AI的节目,但同时,好莱坞最大的那些制片厂也都在场,因为他们现在有宏伟的计划,要把AI整合到他们的工作流程和现有系统里。

New AI studios who are offshoots of the bigger studios trying to do like only AI AI shows, but also like the the biggest of the Hollywood studios are also there because now they have big plans integrating AI into their workflows, into their existing systems.

Speaker 0

所以从会议的参会情况也能看出这种变化。

So you can see the change in the attendance of the conference as well.

Speaker 1

太棒了。

That's awesome.

Speaker 1

好的,观众朋友们,请关注GEM媒体大会发布的内容。

Well, for the audience, check out, the content coming out of the, the GEM Media Conference.

Speaker 1

那将会非常非常令人兴奋。

It's going to be very, very exciting.

Speaker 1

非常感谢Gorkem和Batuhan今天来到节目,这对GEM Media来说是一个非常激动人心的时刻。

And thank you so much Gorkem and Batuhan coming on It's to our super exciting time for GEM Media.

Speaker 0

谢谢。

Thank you.

Speaker 3

感谢收听本期a16z播客。

Thanks for listening to this episode of the a 16 z podcast.

Speaker 3

如果你喜欢这期节目,记得点赞、评论、订阅,给我们留个评分或评价,也别忘了分享给你的朋友和家人。

If you like this episode, be sure to like, comment, subscribe, leave us a rating or a review, and share it with your friends and family.

Speaker 3

想听更多节目,可以去YouTube、Apple Podcast和Spotify。

For more episodes, go to YouTube, Apple Podcast, and Spotify.

Speaker 3

在X上关注我们@a16z,并订阅我们的Substack:a16z.substack.com。

Follow us on x at a sixteen z, and subscribe to our Substack at a16z.substack.com.

Speaker 3

再次感谢收听,我们下期见。

Thanks again for listening, and I'll see you in the next episode.

Speaker 3

提醒一下,这里的内容仅供参考,不构成法律、商业、税务或投资建议,也不应用于评估任何投资或证券,并且不针对任何a16z基金的投资者或潜在投资者。

As a reminder, the content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a sixteen z fund.

Speaker 3

请注意,a16z及其关联公司可能也持有本播客中讨论到的公司的投资。

Please note that a sixteen z and its affiliates may also maintain investments in the companies discussed in this podcast.

Speaker 3

更多详情,包括我们的投资列表,请访问a16z.com/disclosures。

For more details, including a link to our investments, please see a 16z.com forward slash disclosures.

关于 Bayt 播客

Bayt 提供中文+原文双语音频和字幕,帮助你打破语言障碍,轻松听懂全球优质播客。

继续浏览更多播客