掌控AI帕累托前沿——杰夫·迪恩

好的。

Okay.

对。

Yeah.

我没有一个特别好的，因为这里不是物理常数，你这里有物理常数。

I don't have a great one because it's not physical constants like you have physical constants in here.

但我确实觉得一个简单的就是参数数量到磁盘大小的转换，如果需要转换的话，那只是一个简单的字节转换。

But I do think a simple one would be number parameters to disk size if you need to convert that, which is a simple byte conversion.

那没什么意思。

That's nothing interesting.

我想知道你有没有什么。

I wonder if you have any.

如果你要更新你的

You were to update your

我觉得考虑模型中的计算，无论是训练还是推理，都是很好的。

I mean, think it's really good to think about calculations you're doing in a model, either for training or inference.

通常一个好的看法是，你需要从内存中引入多少状态，无论是芯片上的SRAM、HBM、加速器附带的内存、DRAM，还是通过网络？

Often a good way to view that is how much state will you need to bring in from memory, either on chip SRAM or HBM from the accelerator attached memory or DRAM or over the network?

然后，相对于矩阵乘法单元中实际乘法运算的成本，这些数据移动的成本有多高？

And then how expensive is that data motion relative to the cost of, say, an actual multiply in the matrix multiply unit?

而且这个成本实际上非常非常低，对吧？

And that cost is actually really, really low, right?

因为根据你的精度，它大概是亚皮焦耳级别的。

Because it's order, depending on your precision, think it's like sub one picojoule.

哦，好的。

Oh, Okay.

你通过什么来衡量它？

You measure it by

能量。

energy.

对。

Yeah.

我的意思是，一切都关乎能源，以及如何打造最高能效的系统。

I mean, it's all going to be about energy and how do you make the most energy efficient system.

然后，把数据从芯片另一侧的SRAM移过来，甚至都不用移出芯片，只是同一颗芯片的另一侧，就可能要消耗上千皮焦耳。

And then moving data from the SRAM on the other side of the chip, not even off chip, but on the other side of the same chip can be, you know, a thousand picojoules.

哦。

Oh.

对。

Yeah.

所以突然之间，这就是为什么你的加速器需要批处理。

And so all of a sudden, this is why your accelerators require batching.

因为如果你把模型参数从芯片上的SRAM移到乘法单元，那就要花掉上千皮焦耳。

Because if you move, like say, the parameter of a model from SRAM on the chip into the multiplier unit, that's going to cost you a thousand pico tools.

所以你最好能多次利用你移动过来的那个东西。

So you better make use of that thing that you moved many, many times.

这就是批处理维度的意义所在，因为突然之间，如果你有256的批次，那还好，但如果批次是1，那就真的不行了。

So that's where the batch dimension comes in because all of a sudden, if you have a batch of two fifty six or something that's not so bad, but if you have a batch of one, that's really not good.

对。

Yeah.

对吧？

Right?

因为那样的话，你花了一千皮焦耳的能量，却只做了一次一皮焦耳的乘法运算。

Because then you paid a thousand picojoules in order to do your one picajoule multiply.

我从来没听过从能量角度来分析

I have never heard an energy based analysis

批处理。

of batching.

是的，我的意思是，这就是人们为什么要做批处理，对吧？

Yeah, I mean, that's why people batch, right?

理想情况下，你希望批大小为一，因为那样延迟会很好。

Ideally you'd like to use batch size one because the latency would be great.

但能源成本和计算效率低下的问题会非常严重。

But energy cost and the compute cost inefficiency that you get is quite large.

有没有类似你把所有东西都放进内存的那种对称做法？

Is there a symmetric like you did with putting everything in memory?

我觉得显然英伟达在SRAM上大力押注，用Croc引起了很大的轰动。

I think obviously NVIDIA has caused a lot of waves with betting very hard on SRAM with Croc.

我很好奇，这是不是你在TPU上已经看到过的情况？

I I I wonder if, like, that's something that you already saw with with the TPUs.

对吧？

Right?

就像，你在那种规模下必须得那样服务。

Like, that you had to to to serve at your scale.

你可能早就预见到了。

You probably sort of saw that coming.

比如，你看到的情况催生了哪些硬件创新或见解？

Like, what what what hardware innovations or insights were formed because of what you're seeing there?

是的。

Yeah.

我觉得，你知道，TPU有一个很好的规则结构，就是二维或三维的网格，上面连接了很多芯片。

I mean, I think, you know, TPUs have this nice sort of regular structure of two d or three d meshes with a bunch of chips connected.

每一个芯片都配有HBM。

And each one of those has HBM attached.

我觉得在服务某些模型时，从HBM读取数据的成本和延迟要比从芯片上的SRAM读取高得多。

I think for serving some kinds of models, pay a lot higher cost and time latency bringing things in from HBM than you do bringing them in from SRAM on the chip.

所以如果你的模型足够小，你实际上可以做模型并行，把它分散到很多芯片上，这样做确实能获得相当好的吞吐量和延迟改进。

So if you have a small enough model, you can actually do model parallelism, spread it out over lots of chips, and you actually get quite good throughput improvements and latency improvements from doing that.

所以你现在把你的小规模模型分散到比如16或64个芯片上。

And so you're now striping your smallish scale model over, say, 16 or 64 chips.

但如果这样做，而且所有数据都适合放在SRAM里，那可能是一个很大的优势。

But if you do that and it all fits in SRAM, that can be a big win.

所以这并不意外，但确实是个好技术。

So that's not a surprise, but it is a good technique.

对。

Yeah.

那TPU的设计呢？

But what about the TPU design?

你觉得哪些地方需要改进，你有多大决定权？

How much do you decide where the improvements have to go?

这倒是个好例子。

So this is a good example.

有没有办法把那一千皮焦降到五十？

Is there a way to bring the 1,000 picojoules down to 50?

而且值不值得为此设计一款新芯片？

And is it worth designing a new chip to do that?

最极端的说法就是，有人会说，你应该把模型直接烧录到ASIC上，那是最极端的做法。

The extreme is when people say, Oh, you should burn the model on the ASIC, and that's kind of the most extreme thing.

在变化这么快的情况下，有多少东西值得用硬件来实现？

How much of it is worth doing in hardware when things change so quickly?

你们内部是怎么讨论的？

What's the internal discussion?

是的，我们和TPU芯片设计架构团队以及更高层的建模专家之间有很多互动，因为我们真的想利用共同设计的机会，根据我们认为机器学习研究的发展方向，来规划未来的TPU应该是什么样子。

Yeah, I mean, we have a lot of interaction between, say, the TPU chip design architecture team and the higher level modeling experts because we really want to take advantage of being able to co design what should future TPUs look like based on where we think the ML research puck is going in some sense.

因为作为机器学习领域的硬件设计师，你从今天开始就要设计芯片。

Because as a hardware designer for ML in particular, you're trying to design a chip starting today.

而这个设计可能需要两年时间才能进入数据中心。

And that design might take two years before it even lands in a data center.

然后芯片还得有合理的寿命，能撑三到五年。

And then it has to sort of be a reasonable lifetime of the chip to take you three, four, or five years.

所以你要预测未来两到六年，人们会想运行什么样的机器学习计算，而这是一个变化非常快的领域。

So you're trying to predict two to six years out what ML computations will people want to run two to six years out in a very fast changing field.

所以，有那些对机器学习研究有有趣想法的人，我们认为这些想法在那个时间框架内会开始起作用或者变得更加重要，这真的能让我们把有趣的硬件特性加入到TPU n加二代中，其中TPU n是我们现在拥有的。

And so having people with interesting ML research ideas of things we think will start to work in that time frame or will be more important in that time frame really enables us to then get interesting hardware features put into TPU n plus two, where TPU n is what we have today.

哦，周期是加二代。

Oh, the cycle time is plus two.

大致是这样。

Roughly, I mean.

哇。

Wow.

因为有时候一些小改动可以塞进下一代，但大改动就得在芯片设计流程更早的阶段就定下来。

Because sometimes you can squeeze some changes into n plus one, but bigger changes are going to require the chip design be earlier in its lifetime design process.

所以只要能做到，通常都是好事。

So whenever we can do that, it's generally good.

有时候你还可以放一些试探性的功能，可能占不了多少芯片面积，但如果成功了，就能让性能提升十倍。

And sometimes you can put in speculative features that maybe won't cost you much chip area, but if it works out, it would make something 10 times as fast.

就算不成功，也就浪费了一点点芯片面积，不是什么大问题。

And if it doesn't work out, well, you burned a little bit of tiny amount of your chip area on that thing, but it's not that big a deal.

有时候改动非常大，我们就得确保它真的能成。

Sometimes it's a very big change, and we want to be pretty sure this is going to work out.

所以我们会做大量细致的机器学习实验，来验证这确实是我们要走的方向。

So we'll do lots of careful ML experimentation to show us this is actually the way we want to go.

对。

Yeah.

有没有反过来的时候？我们已经定下了芯片设计，所以模型架构不能往那个方向走，因为它不太合适？

Is there a reverse of, we already committed to this chip design so we cannot take the model architecture that way because it doesn't quite fit?

对。

Yeah.

确实会有这种情况，你会调整模型架构，让它在训练和推理那代模型时，在已有的芯片上更高效。

I mean, you definitely have things where you're going to adapt what the model architecture looks like so that they're efficient on the chips that you're going to have for both training and inference of that generation of model.

所以我觉得这是双向的。

So I think it kind of goes both ways.

有时候你可以利用未来一代芯片带来的低精度优势。

Sometimes you can take advantage of lower precision things that are coming in a future generation.

所以你可能用那个低精度来训练，即使当前一代芯片还不完全支持。

So you might train it at that lower precision, even if the current generation doesn't quite do that.

对。

Yeah.

精度能低到什么程度？

How low can we go in precision?

因为大家都在说三进制之类的，

Because people are saying like ternary is like,

对。

Yeah.

我是说，我非常喜欢极低精度，因为我觉得那能节省大量能源，对吧？

It's I mean, I'm a big fan of very low precision because I think that saves you a tremendous amount of energy, right?

因为传输时每比特消耗的是皮焦耳能量，减少比特数确实是降低能耗的好办法。

Because it's picojoules per bit that you're transferring and reducing the number of bits is a really good way to reduce that.

我觉得大家从极低比特精度中获益良多，但还要配合对整个权重组应用缩放因子。

I think people have gotten a lot of luck mileage out of having very low bit precision things, but then having scaling factors that apply to a whole bunch of weights.

缩放，那是怎么运作的？

Scaling, how does that work?

有意思。

Interesting.

所以是低精度，但权重被放大了。

So low precision, but scaled up weights.

对。

Yeah.

对，这个我还真没想过。

Yeah, never considered that.

有意思。

Interesting.

既然说到这个话题，我觉得当我们做采样的时候，精度这个概念本身就挺奇怪的。

While we're on this topic, I think there's a lot of this is the concept of precision at all is weird when we're sampling.

到最后，我们会有所有这些能做非常精确计算的芯片，然后我们却在开头扔一个随机数生成器。

At the end of this, we're going to have all these chips that all do very good math, and then we're just gonna throw a random number generator at the start.

我是说，现在有一种趋势是往基于能量的模型和处理器方向发展。

I mean, there's a movement towards energy based models and processors.

我很好奇，你显然已经考虑过这个问题，你的看法是什么？

I'm just curious if you've obviously, you've thought about it, but what's your commentary?

是的，我觉得有很多有趣的趋势。

Yeah, I mean, I think there's a bunch of interesting trends.

所以基于能量的模型是其中之一。

So energy based models is one.

扩散模型是另一种，它不像那样逐个顺序解码令牌。

Diffusion based models, which don't sort of sequentially decode tokens, is another.

投机解码是一种方法，可以让你获得等效的、非常小的批次因子，比如你预测出八个令牌，这样就能将你正在做的事情的有效批次大小增加八倍。

Speculative decoding is a way that you can get sort of an equivalent, very small batch factor for like you predict eight tokens out, and that enables you to sort of increase the effective batch size of what you're doing by a factor of eight.

然后你可能会接受其中五六个令牌，这样在将权重移入乘法器以进行令牌预测时，摊销效果就能提升五倍。

And then you maybe accept five or six of those tokens, so you get a 5x improvement in the amortization of moving weights into the multipliers to do the prediction for the token.

所以这些都是非常好的技术。

So these are all really good techniques.

我认为从实际能量——不是基于能量的模型——以及延迟和吞吐量的角度来看待它们是非常好的，对吧？

And I think it's really good to look at them from the lens of energy real energy, not energy based models and also latency and throughput, right?

如果你从这个角度来看，它会引导你找到更好的解决方案，能够以更低的成本和更低的延迟来服务更大的模型或同等规模的模型。

If you look at things from that lens, that sort of guides you to solutions that are going to be better from being able to serve larger models or equivalent sized models more cheaply and with lower latency.

是的。

Yeah.

嗯，我觉得这从智力上很有吸引力。

Well, I think I think I it's appealing intellectually.

我还没看到它真正成为主流，但我确实觉得有一种诗意，如果我们从根本上把它设计进硬件里，就不需要做那么多花招了。

I haven't seen it, like, really hit in the mainstream, but I do think that there's some poetry in the sense that we don't have to do a lot of shenanigans if we fundamentally design it into the hardware.

对。

Yeah.

我觉得还有一些更奇特的东西，比如模拟计算基底，而不是数字的。

I I think there's also sort of the more exotic things like analog based computing substrates as opposed to digital ones.

我觉得那些非常有趣，因为它们可能功耗很低。

I think those are super interesting because they can be potentially low power.

但我觉得你通常最终会想把它和数字系统连接起来，结果在系统的边界和外围进行数模和模数转换时，会损失很多功耗优势。

But I think you often end up wanting to interface that with digital systems and you end up losing a lot of the power advantages in the digital to analog and analog to digital conversions you end up doing at the sort of boundaries and periphery of that system.

我仍然认为，从今天的能源效率来看，我们还有巨大的提升空间，通过为关心的模型设计更好、更专用的硬件。

I still think there's a tremendous distance we can go from where we are today in terms of energy efficiency with much better and specialized hardware for the models we care about.

你见过其他有趣的研究想法吗？或者有没有你在谷歌无法做，但希望看到其他研究者尝试的事情？

Any other interesting research ideas that you've seen or maybe things that you cannot pursue at Google that you would be interested in seeing researchers take a step at?

我猜你有很多研究人员。

I guess you have a lot of researchers.

我猜你有很多——

I guess you have a lot of-

我们的研究范围相当广泛，可以说。

Our research portfolio is pretty broad, would say.

我的意思是，从研究方向来看，有很多开放性问题，比如如何让这些模型更可靠，能够完成更长、更复杂的任务，这些任务包含很多子任务。

I mean, I think in terms of research direction, there's a whole bunch of open problems and how do you make these models reliable and able to do much longer, more complex tasks that have lots of subtasks?

如何编排一个模型，让它把其他模型当作工具来使用，从而构建出能够集体完成比单个模型更重大任务的东西。

How do you orchestrate maybe one model that's using other models as tools in order to build things that can accomplish much more significant pieces of work collectively than you would ask a single model to do.

这非常有趣。

So that's super interesting.

如何获得可验证性？如何让强化学习在不可验证的领域发挥作用？

How do you get verifiable how how do you get RL to work for non verifiable domains?

我认为这是一个相当有趣的开放性问题，因为如果我们能把这些在数学和编码方面看到的改进应用到其他不太可验证的领域，那将拓宽模型的能力。如果我们能想出有效的强化学习技术来实现这一点，那真的会让模型进步很多。

Think it's a pretty interesting open problem because I think that would broaden out the capabilities of the models, the improvements that you're seeing in both math and coding, if we could apply those to other less verifiable domains because we've come up with RL techniques that actually enable us to do that effectively, that would really make the models improve quite a lot, I think.

我很好奇。

I'm curious.

之前诺姆·布朗上我们播客的时候，他说他们已经用深度研究证明了这一点。

When we had Noam Brown on the podcast, he said they already proved you can do it with deep research.

在AI模式下，你其实已经做到了，只是这种模式不是可验证的。

You kind of have it with AI mode in a way that's not verifiable.

我很好奇，你觉得这里面有没有什么有意思的点？

I'm curious if there's any thread that you think is interesting there.

比如说，那是什么？

Like, what is it?

两者都像是JSON的信息检索。

Both are like information retrieval of JSON.

所以我在想，是不是检索这部分就是可以打分、可以验证的，还是说……对。

So I wonder if it's like the retrieval is like the verifiable part that you can score or what are like Yeah.

你会怎么建模这个问题？

How would you model that problem?

对。

Yeah.

我觉得有办法让其他模型来评估第一个模型的结果。

I mean, I think there are ways of having other models that can evaluate the results of what a first model did.

比如在检索的时候，能不能让另一个模型来判断，你检索到的这些内容相关吗？

Maybe in retrieving, can you have another model that says, are these things you retrieved relevant?

或者能不能让你检索到的这2000条内容评分，挑出最相关的50条之类的？

Or can you rate these 2,000 things you retrieved to assess which ones are the 50 most relevant or something?

我觉得这类技术其实挺有效的。

I think those kinds of techniques are actually quite effective.

有时候甚至可以用同一个模型，只是用不同的提示词让它当评论者，而不是当检索系统。

Sometimes that can even be the same model just prompted differently to be a critic as opposed to a actual retrieval system.

对。

Yeah.

我确实觉得存在那个奇怪的瓶颈，感觉我们已经解决了容易的部分，但总是有这种感觉。

I do think there is that weird cliff where it feels like we've done the easy stuff, but it always feels like that.

每年都是这样，觉得我们已经知道了很多，然后下一部分特别难，没人能搞定。

Every year it's oh, know what we know, and the next part is super hard and nobody's figured it out.

就像这个ROVR项目，大家都在讨论，好吧，那不可验证的那部分下一步该怎么做？

And exactly with this ROVR thing where everyone's talking about, well, okay, how do we do the next stage of the non verifiable stuff?

大家都说，我不知道，艾伦，我……

Everyone's like, I don't know, Ellen I

我觉得这个领域的好处是，有很多很多聪明人在思考各种创造性的解决方案，来应对我们都能看到的一些问题。

I feel like the nice thing about this field is there's lots and lots of smart people thinking about creative solutions to some of the problems that we all see.

因为我觉得大家都看到了，模型在某些方面很擅长，但在这些方面的边缘地带就会出问题，能力达不到我们期望的水平。

Because I think everyone sort of sees that the models are great at some things and they fall down around the edges of those things and are not as capable as we'd like in those areas.

然后想出好的技术，去尝试，看看哪些真的有效，这正是这个领域整个研究方面在推动的事情。

Then coming up with good techniques and trying those and seeing which ones actually make a difference is sort of what the whole research aspect of this field is pushing forward.

我觉得这就是为什么它特别有意思。

And I think that's why it's super interesting.

想想两年前，我们还在为GSM 8K的问题头疼，对吧？

If you think back two years ago, we were struggling with GSM eight ks problems, right?

比如弗雷德有两只兔子，他又得到了三只兔子。

Like Fred has two rabbits, he gets three more rabbits.

那他总共有多少只兔子？

How many rabbits does he have?

这和模型现在能处理的数学问题相比，差得可远了。

That's a pretty far cry from the kinds of mathematics that the models can

而现在你们在做IMO级别的题目，

And now you're doing IMO,

还有埃尔德什问题，纯粹用语言。

and Erdos problem Pure language.

纯粹用语言。

Pure language.

所以这真的是一年半左右的时间里，能力上非常惊人的飞跃。

So that is a really, really amazing jump in capabilities in a year and a half or something.

我觉得在其他领域，如果能实现这样的跨越就好了。

And I think for other areas, it'd be great if we could make that kind of leap.

有些领域我们确实还不知道怎么做，但有些领域我们已经看到了方向，我们会努力让它变得更好。

And we don't exactly see how to do it for some areas, but we do see it for some other areas, we're going to work hard on making that better.

对。

Yeah.

比如YouTube缩略图的生成。

Like YouTube thumbnail generation.

那会非常有帮助。

That would be very helpful.

我们需要这个。

We need that.

那简直就是AGI了。

That would be AGI.

我们需要这个。

We need that.

对于内容创作者来说。

As far as content I creators

我想我不是YouTube创作者，所以我不太在意那个问题，但我想很多人都在意。

guess I'm not a YouTube creator, so I don't care that much about that problem, but I guess many people do.

对，没关系。

Yeah, it doesn't matter.

事实证明，人们确实会以貌取人。

People do judge books by their covers, as it turns out.

再提一下AI大亨。

Just to draw a bit on the AI mogul.

我到现在还不敢相信，一年前我们有了AlphaProof和AlphaGeometry这些东西。

I'm still not over the fact that a year ago we had alpha proof and alpha geometry and all those things.

然后今年我们就说，管他呢。

And then this year we were like, screw that.

我们直接把它扔进Gemini里。

We'll just chuck into Gemini.

是的。

Yeah.

你有什么看法？

What's your reflection?

我觉得关于符号系统和大型语言模型融合的问题，这曾经是一个核心信念。

Like, I think this this question about the merger of symbolic systems and LLMs was a very much core belief.

但后来某个时候，我们就说，不，我们直接在大型语言模型里搞定一切。

Then somewhere along the line, just said, Nope, we'll just all do it in the LLM.

对。

Yeah.

我觉得这对我来说很有道理，因为人类会操作符号，但我们脑子里可能并没有符号化的表示。

I mean, I think it makes a lot of sense to me because humans manipulate symbols, but we probably don't have a symbolic representation in our heads.

我们有的是某种分布式表示，就像神经网络那样，由很多不同的神经元和激活模式组成，当我们看到某些东西时会触发。

We have some distributed representation that is neural net like in some way of lots of different neurons and activation patterns firing when we see certain things.

这让我们能够推理、规划、进行思维链，并且可以回溯。

And that enables us to reason and plan and do chains of thought and roll them back.

现在这种解决问题的方法看起来行不通了。

Now that approach for solving the problem doesn't seem like it's going to work.

我来试试这个。

I'm going try this one.

在很多方面，我们其实是在模仿我们直觉认为真实大脑中发生的事情，用基于神经网络的模型。

And in a lot of ways, we're emulating what we intuitively think is happening inside real brains in neural net based models.

所以对我来说，把完全独立的离散符号和完全不同的思考方式分开，从来就没有道理。

So it never made sense to me to have completely separate discrete symbolic things and then a completely different way of thinking about those things.

有意思。

Interesting.

对。

Yeah.

我是说，这对你来说可能显而易见，但一年前对我来说并不是这样。

I mean, it maybe seems obvious to you, but it wasn't obvious to me a year ago.

对。

Yeah.

我是说，我确实认为，在IMO中，用Lean来翻译和推理，然后第二年还用了专门的几何模型。

I mean, I do think that IMO with translating to lean and using lean and then the next year and also a specialized geometry model.

然后今年，转向一个统一的模型，基本上就是生产模型，只是推理预算稍微多一点，这其实挺好的，因为这表明那个通用模型的能力已经大幅提升，现在你不需要那些专门的模型了。

And then this year, switching to a single unified model that is roughly the production model with a little bit more inference budget is actually quite good because it shows you that the capabilities of that general model have improved dramatically, and now you don't need these specialized models.

这其实和2013到2016年的机器学习时代非常相似。

This is actually sort of very similar to the twenty thirteen to 'sixteen era of machine learning.

以前人们会为每个不同的问题训练单独的模型。

It used to be people would train separate models for lots of each different problem.

比如我想识别路标，就训练一个路标识别模型。

I want to recognize street signs and something, so I train a street sign recognition model.

或者我想做语音识别，就有一个语音模型。

Or I want to decode speech recognition, I have a speech model.

我觉得现在真的是统一模型的时代了，一个模型能做所有事情。

I think now the era of unified models that do everything is really upon us.

问题是，这些模型在面对它们从未被要求做过的新事物时，泛化能力有多好？

And the question is, how well do those models generalize to new things they've never been asked to do?

而且它们正在变得越来越好。

And they're getting better and better.

而且你不需要领域专家。

And you don't need domain experts.

所以我采访了那个团队的Itay，他说，对，我也不知道它们是怎么工作的。

So I interviewed Itay, who was on that team, and he was like, yeah, I don't know how they work.

我不知道IMO竞赛在哪里举行。

I don't know where the IMO competition was held.

我也不知道它的规则。

I don't know the rules of it.

就是训练模型，计算机训练模型。

Just train the models, the computer training models.

有趣的是，拥有这种通用机器学习技能的人，只要给他们数据和足够的算力，他们就能处理任何任务，这大概就是所谓的惨痛教训吧。

And it's kind of interesting that people with this universal skill set of machine learning, you just give them data and give them enough compute, they can tackle any task, which is a bitter lesson, I guess.

不知道。

Don't know.

对。

Yeah.

我觉得在大多数情况下，通用模型会胜过专用模型。

I mean, think general models will win out over specialized ones in most So

我想在这里稍微深入一下。

I want to push there a bit.

我觉得这里有一个漏洞，就是关于模型容量的概念。

I think there's one hole here, which is like, there's this concept of like maybe capacity of a model.

抽象地说，一个模型只能包含它拥有的比特数。

Abstractly, a model can only contain the number of bits that it has.

所以，天知道Gemini Pro有多少参数，一万亿？我们不知道。

So, know, God knows Gemini Pro is one to 10,000,000,000,000 parameters, we don't know.

但是，比如Gemma模型，对吧？

But the Gemma models, for example, right?

很多人想要那种开源的本地模型。

Like a lot of people want the open source local models that are like that.

它们有一些知识，但这并不是必需的。

And they have some knowledge, which is not necessary.

对吧？

Right?

它们不可能什么都知道。

Like, they can't know everything.

你有那个奢侈，你有大模型，大模型应该什么都能做。

Like, you have the luxury of, you have the big model and big model should be capable of everything.

但当你蒸馏到小模型时，你实际上是在记住一些没用的东西。

But like when you're distilling and you're going down to the small models, you know you're actually memorising things that are not useful.

所以，我们是不是应该把那些东西提取出来？

And so like, how do we, I guess, do we want to extract that?

我们能把知识和推理分开吗？

Can we divorce knowledge from reasoning?

是的。

Yeah.

我觉得，如果模型能检索信息，你确实希望它最擅长推理，对吧？

I mean, think you do want the model to be most effective at reasoning if it can retrieve things, right?

因为让模型把宝贵的参数空间用来记住那些可以查到的冷门事实，其实并不是对参数空间的最佳利用，对吧？

Because having the model devote precious parameter space to remembering obscure facts that could be looked up is actually not the best use of that parameter space, right?

你可能会更希望模型具备更通用、在更多场景下有用的能力，而不是记住某个冷门事实。

Like you might prefer something that is more generally useful in more settings than this obscure fact that it has.

所以我觉得这始终是个值得关注的问题。

So I think that's always attention.

但同时，你也不希望模型完全脱离对世界的了解。

At the same time, you also don't want your model to be kind of completely detached from knowing stuff about the world.

比如，知道金门大桥有多长可能挺有用的，至少能对桥梁的长度有个大致概念，对吧？

Like it's probably useful to know how long the Golden Gate Bridge is, just as a general sense of like how long are bridges, right?

模型应该具备这类知识。

And it should have that kind of knowledge.

它可能不需要知道世界上某个偏僻角落的小桥有多长，但拥有相当丰富的世界知识确实对它有帮助。

It maybe doesn't need to know how long some teeny little bridge in some other more obscure part of the world is, but it does help it to have a fair bit of world knowledge.

而且模型越大，能容纳的知识就越多。

And the bigger your model is, the more you can have.

但我确实认为，把检索和推理结合起来，让模型在多次检索和中间结果的推理上做得很好，会是让模型看起来能力更强的一个非常有效的方法。

But I do think combining retrieval with sort of reasoning and making the model really good at doing multiple stages of retrieval and reasoning through the intermediate retrieval results is going to be a pretty effective way of making the model seem much more capable.

因为你想，比如个人版的Gemini，我们大概不会用我的邮件去训练它。

Because if you think about, say, a personal Gemini, we're not going to train Gemini on my email probably.

我们更希望有一个单一的模型，然后我们可以把它当作工具，用来检索我的邮件，让模型去推理，再检索我的照片之类的，然后利用这些信息，进行多阶段的交互。

We'd rather have a single model that we can then use and use being able to retrieve from my email as a tool and have the model reason about it and retrieve from my photos or whatever and then make use of that and have multiple stages of interaction.

你觉得垂直模型是个值得探索的方向吗？

Do you think the vertical models are an interesting pursuit?

当人们说，哦，我们在打造最好的医疗大语言模型，我们在打造最好的法律大语言模型，这些是不是短期的权宜之计？

When people are like, oh, we're building the best health care LLM, we're building the best law LLM, are those short term stop gaps?

不是。

No.

我觉得垂直模型很有意思。

I think vertical models are interesting.

它们应该从一个相当好的基础模型开始，但你可以把它们看作是在丰富特定垂直领域的数据分布，比如医疗，可能不会去训练，或者说，比如机器人技术。

You want them to start from a pretty good base model, but then you can sort of view them as enriching the data distribution for that particular vertical domain, for health care, probably not going to train or for, say, robotics.

我们可能不会用所有能拿到的机器人数据来训练Gemini，因为我们希望它具备均衡的能力。

We're probably not going to train Gemini on all possible robotics data you could train it on because we want it to have a balanced set of capabilities.

所以我们会让它接触一些机器人数据。

So we'll expose it to some robotics data.

但如果你想打造一个真正顶尖的机器人模型，你会想从那个基础开始，然后用更多的机器人数据来训练它。

But if you're trying to build a really, really good robotics model, you're going to want to start with that and then train it on more robotics data.

然后这可能会削弱它的多语言翻译能力，但会提升它的机器人相关能力。

And then maybe that would hurt its multilingual translation capability, but improve its robotics capabilities.

我们一直在训练基础Gemini模型时，在数据组合上做这样的权衡。

And we're always making these kind of trade offs in the data mix that we train the base Gemini models on.

我们很想加入另外200种语言的数据，并且尽可能多地包含这些语言的数据，但这会挤占模型的其他能力。

We'd love to include data from 200 more languages and as much data as we have for those languages, but that's going to displace some other capabilities of the model.

它在Perl编程上就不会那么出色了。

It won't be as good at Perl programming.

它在Python编程上依然会很好，因为我们会包含足够多的相关数据，但在其他冷门的编程语言或编码能力上可能会受影响，多模态推理能力也可能因为数据不足而有所下降，不过它在多语言方面会非常强。

It'll still be good at Python programming because we'll include enough of that, but there's other long tail computer languages or coding capabilities that it may suffer on or multimodal reasoning capabilities may suffer because we didn't get to expose it to as much data there, but it's really good at multilingual things.

所以我觉得可能是专门模型和模块化模型的某种结合。

So I think some combination of specialized models, maybe more modular models.

所以最好能同时支持那200种语言，再加上这个超棒的机器人模型和这个超棒的医疗模块，它们可以组合在一起协同工作，在不同场景下调用。

So it'd be nice to have the capability to have those 200 languages plus this awesome robotics model plus this awesome health care module that all can be knitted together to work in concert and called upon in different circumstances.

比如我遇到健康相关的问题，就应该能调用这个医疗模块，和主模型一起配合，在这些问题上表现更好。

Like if I have a health related thing, then it should enable using this health module in conjunction with the main base model to be even better at those kinds of things.

可安装的知识。

Installable knowledge.

对。

Yeah.

就像下载一个

Just download as

软件包一样。

a package.

其中一些可安装的内容可以通过检索获得，但有些可能应该通过训练千亿甚至万亿级别的医疗数据来获得。

And some of that installable stuff can come from retrieval, but some of it probably should come from training on 100,000,000,000 tokens or a trillion tokens of health data.

对。

Yeah.

对于听众来说，我想提一下Gemma 3N的论文，里面有一点这方面的内容。

And for listeners, I think I will highlight the Gemma 3N paper where there's a little bit of that, I think.

是的。

Yeah.

对。

Yeah.

问题是，你需要多少亿的token才能超越前沿模型的改进？

Guess the question is how many billions of tokens do you need to outpace the frontier model improvements?

比如，如果我要让这个模型在医疗方面更出色，而主Gemini模型还在不断进步，我需要500亿个token吗？

It's like, if I have to make this model better at health care and the main Gemini model is still improving, do I need 50,000,000,000 tokens?

100个够吗？

Can I do it with 100?

如果我需要一万亿个医疗token，这些数据可能你手头没有。

If I need a trillion healthcare tokens, they're probably not out there that you don't have.

我觉得这真的是个

I think that's really the

挑战。

challenge.

嗯，我觉得医疗保健确实是个特别有挑战性的领域。

Well, mean, I think healthcare is a particularly challenging domain.

有很多医疗数据我们没法合理获取，但也有很多医疗机构想用自己的数据来训练模型，这些数据并不是公开的医疗数据。

So there's a lot of healthcare data that we don't have access to appropriately, but there's a lot of health care organizations that want to train models on their own data that is not public health care data.

不是公共卫生数据，而是公开的医疗数据。

Not public health, but public health care data.

所以我觉得有机会和大型医疗机构合作，为他们训练更定制化的模型，这些模型可能比用公开数据训练出来的通用模型效果更好。

So I think there are opportunities there to say partner with a large health care organization and train models for their use that are going to be more bespoke, but probably might be better than a general model trained on, say, public data.

顺便说一句，这跟之前聊的语言话题有点相关。

Believe By the way, this is somewhat related to the language conversation.

我记得一个很好的例子是，你可以把一种低资源语言放到上下文里，它就能在上下文中直接学习。

I think one of favorite examples was you can put a low resource language in the context and it just learns in context.

哦，对。

Oh yeah.

我记得我们用的例子是Kalamong，那真的是资源极少的语言，因为全世界大概只有120个人说，而且没有文字记录。

I think the example we used was Kalamong, which is truly low resource because it's only spoken by, I think, 120 people in the world and there's no written text.

所以你可以直接那样做，把它放到上下文里。

So you can just do it that way, put it in the context.

把整个数据集都放到上下文里。

Put your whole dataset in the context.

如果你拿像索马里语这样的语言来说，世界上确实有不少索马里语文本，还有埃塞俄比亚的阿姆哈拉语之类的。

If you take a language like Somali or something, there is a fair bit of Somali text in the world Ethiopian Amharic or something.

我们可能并没有把那些语言的所有数据都放进Gemini的基础训练里。

We probably are not putting all the data from those languages into the Gemini based training.

我们放了一部分，但如果放更多的话，就能提升那些模型或那些语言的能力。

We put some of it, but if you put more of it, you'll improve the capabilities of those models or of those languages.

对，不错。

Yeah, cool.

我对语言学有点兴趣。

I have a side interest in linguistics.

大学时上过几门课，而且如果我是语言学家，能接触到所有这些模型，我就会去问一些关于语言本身的根本性问题。

Did a few classes back in college and part of me, if I was a linguist and I could have access to all these models, I would just be asking really fundamental questions about language itself.

有一个很明显的问题，就是萨丕尔-沃尔夫假说，比如你所说的语言在多大程度上影响你的思维？

There's one very obvious one which is Sapir Wharf, like how much does the language that you speak affect your thinking?

但也有一些语言里有些概念在其他语言中找不到对应，但也有很多是重复的。还有一篇人们很喜欢的论文叫“柏拉图式表征”，就是说一张杯子的图片，如果你用大量包含“杯子”这个词的文本训练一个模型，它最终会在潜在空间里映射到大致相同的位置。

But then also there are some languages where there's just concepts that are not represented in other languages, But many others that are just duplicates, where there's also another paper that people love called the Platonic representation, where an image of a cup is, if you learn a model on that and you you have a lot of text with the word cup, it eventually maps to it, like roughly the same place in latent space.

所以这应该适用于各种语言，除非在某些情况下不适用。

And so, like, that should apply to languages except where it doesn't.

而这正是人类发现的概念中非常有趣的差异，也许英语里没有这些概念。

And that's actually, like, very interesting differences in what humanity has discovered as concepts that maybe English doesn't have.

我不知道。

I don't know.

这只是我对语言的一些看法。

It's just my rant on languages.

对。

Yeah.

我之前做过一个早期模型，把语言模型和图像模型融合在一起，语言模型有很好的基于词的表示，图像模型是在ImageNet这类数据上训练的。

I did some work on an early model that fused together a language based model where you have nice word based representations and then an image model where you have trained it on ImageNet like things.

然后把顶层融合起来，不，这是devise。

And then you fuse together the top layers of No, this is devise.

你再多做一些训练来融合这些表示。

You do a little bit more training to fuse together those representations.

结果发现，如果你给一张新图像，它不在图像模型训练过的任何类别里，模型往往也能给这张图像分配一个标签。

And what you found was that if you give a novel image that is not in any of the categories in the image model that was trained on, the model can often assign the label to that image.

比如，我记得望远镜和双筒望远镜都在图像模型的训练类别里，但显微镜不在。

So for example, I think telescope and binoculars were both in the training categories for the image model, but microscope was not.

所以如果你给一张显微镜的图像，它实际上能给出一个带有“显微镜”这个词的标签，尽管它从未见过标有该标签的图像。

And so if you're given an image of a microscope, it actually can come up with something that's got the word microscope as the label that are designed, even though it's never actually seen an image labeled that.

哦，那挺好的。

Oh, that's nice to know.

挺酷的。

It's kind cool.

对。

Yeah.

对。

Yeah.

很有用。

Useful.

不错。

Cool.

我觉得有一些更泛泛的、更宽泛的问题，但我想，总的来说，你希望别人多问你些什么？

I think there's more general, like broad questions, but like, I guess what do you wish you were asked more in general?

你的领域太广了。

You have such a broad scope.

我们已经聊了硬件。

We've covered the hardware.

我们已经聊了模型的研究。

We've covered the model's research.

对。

Yeah.

我觉得有意思的一点是，我本科毕业论文做的就是神经网络训练，并行神经网络训练，那是在1990年，当时我就接触到了神经网络。

I think one thing that's kind of interesting is I did a undergrad thesis on neural network training, parallel neural network training back in 1990 when I got exposed to neural nets.

我一直觉得它们是正确的抽象，只是我们需要比当时多得多的计算能力。

I always felt kind of they were the right abstraction, but we just needed way more compute than we had then.

所以系里那台有32个处理器的并行计算机，能让你训练稍微有趣一点的模型，但还不足以解决实际问题。

So like the 32 processors in the department parallel computer could get you a little bit more interesting model, but not enough to solve real problems.

所以从2008年或2009年开始，世界通过摩尔定律拥有了足够的计算能力，加上大量有趣的数据集，我们才能真正开始训练神经网络，解决人们关心的实际问题，比如语音识别、视觉，最后是语言。

And so starting in 2008 or 'nine, the world started to have enough computing power through Moore's Law and large interesting data sets to train on to actually start training neural nets that could tackle real problems that people cared about, speech recognition, vision, and eventually language.

所以当我在2011年底开始在谷歌做神经网络时，我真的觉得我们应该扩大神经网络的规模，用大量的并行计算来训练它们。

And so when I started working on neural nets at Google in late twenty eleven, I really just felt like we should scale up the size of neural networks we can train using large amounts of parallel computation.

所以我实际上重新拾起了我本科论文里的一些想法，当时我做了模型并行和数据并行训练，并进行了比较。

And so I actually revived some ideas from my undergrad thesis where I'd done both model parallel and data parallel training, and I compared them.

我当时给它们起了不同的名字。

I called them something different.

好像是叫模式分区和模型分区之类的。

There was like pattern partitioned and model partitioned or something.

这个公开了吗？

Is it public?

我们能去看看吗？

Can we go to

是的，网上有。

Yeah, it's on the web.

好的，那挺好的。

Okay, that's nice.

但我觉得，把那些技术结合起来，并且在过去十五年里不断推动规模化，真的非常重要。

But I think combining a lot of those techniques and really just trying to push on scaling things up over the last fifteen years has been really important.

这意味着硬件上的改进，比如推动开发像TPU这样的专用硬件。

And that means improvements in the hardware, so pushing on building specialized hardware like TPUs.

这也意味着要推动软件抽象层的进步，让人们能够更有效地表达机器学习的想法。

It also means pushing on software abstraction layers to let people express ML ideas effectively.

然后还要研究像稀疏模型这样的东西。

And then also working on things like, say, sparse models.

我长期以来一直觉得，稀疏激活的模型非常重要，因为就像我们之前讨论的，你希望模型有大量的容量，能记住很多东西。

I've felt for a long time that sparsely activated models are a really important thing because you want the models to have a lot of capacity, to our earlier discussion about remembering a lot of stuff.

但同时也希望激活模型的方式非常高效。

But you also want to be super efficient in how you activate your model.

所以你可能想要数万亿的参数，但只激活其中的1%或5%或10%。

So you'd like trillions of parameters, but activate only 1% or 5% or 10% of that.

我们在这方面早期发表了一篇论文，真正把超大规模的神经网络推到了极致。

And we did an early paper on this where we really scaled up outrageously large neural networks.

那篇论文的标题，我觉得是Noam起的，挺吸引人的。

That was the title of the I think that's Noam's wording in the title, which is a good catchy title.

2017年的时候，他就在讲万亿参数模型了。

I mean, in 2017, he was out there talking about 1,000,000,000,000 parameter models.

对，这确实很棒，因为相比非稀疏模型，它在达到相同质量水平时，时间或计算成本上带来了10倍的提升。

Yeah, so that is really good because that gave you a 10x improvement in time to quality or compute cost given the quality level relative to non sparse models.

同样，Transformer相比当时的LSTM，在达到相同质量水平时，计算成本上带来了10到100倍的提升。

Transformers similarly gave you a 10x to 100x improvement in compute cost to a given quality level versus, say, LSTMs at the time.

而且这些改进都是叠加在一起的。

And all of those things multiply together.

所以我觉得所有这些方面都很值得去研究。

So I think all those things really are important to work on.

硬件、系统基础设施、模型架构的算法层面、改进数据、改进强化学习的方法，所有这些因素叠加在一起，相乘起来，才让我们2026年的模型比2025年好得多，也比2024年、2023年和2022年强太多了。

The hardware, the systems infrastructure, the algorithmic aspects of model architecture, improving the data, improving the RL recipes, all these things are what are stacking together and multiplying together to give us models of 2026 are much better than models of 2025 and are awesomely better than 2024 and 2023 and 2022.

说实话，这也是一个巨大的组织挑战。

And a huge, honestly, organizational challenge.

大概有1000人，甚至更多。

There's like 1,000 people or maybe more.

你知道，第一代Gemini论文发表的时候，有大约1000位共同作者。

Know when the first Gemini paper came out, it was like 1,000 co authors.

对，对。

Yeah, yeah.

技术报告里有十页的合著者名单。

We have 10 pages of co author within the tech report.

那挺好的。

Was nice.

我是说，你知道，大家都想在一篇可能具有历史意义的论文上被署名。

I mean, you know, people want to be acknowledged on probably a historical paper.

是的。

Yeah.

我觉得其实有很多合著者完全没问题。

I mean, I think it's perfectly good to have actually a lot of coauthors.

而且我认为，组织这么多人，让他们有效地朝着共同的方向努力，让他们的工作最终能相乘，产生下一代模型这样的成果，其实相当棘手。

And I do think organizing that number of people so that they're effectively pushing in common directions, that all their work actually sort of multiplies together in the ultimate output, which is the next generation of model, is actually pretty tricky.

我们Gemini团队里有很多优秀的人来帮忙协调这件事。

And we have awesome people throughout the Gemini team to help orchestrate this.

我自己、诺姆和阿里尔在帮忙主导这件事。

Myself, Noam, and Ariel are helping steer this.

然后我们有团队在思考预训练的设置应该是什么样的。

And then we have people thinking about what does the pre training setup look like?

基础设施应该是什么样的？

What does the infrastructure look like?

后训练的流程应该是什么样的？

What does the post training recipe look like?

还有数据准备、评估、多模态能力以及国际化能力。

And what does the data preparation and evals and multimodal capabilities and I18N capabilities?

有很多不同的领域，比如编码能力，所有这些领域都非常重要。

There's a lot of different kinds of areas, coding capabilities, all these areas super important.

而且有专人密切关注这些方面，同时也关注其他所有事情，这真的很好。

And it's really good to have people, paying close attention to those things and then, also paying close attention to all the other things.

对。

Yeah.

我听说谢尔盖现在非常活跃，而且深度参与编程相关的工作。

I'm told Sergei is very actively back and very much involved in coding stuff.

是的。

Yep.

对。

Yeah.

没错。

Yeah.

我们都在同一个茶水间。

We all use the same micro kitchen.

对。

Yeah.

哦，好的。

Oh, Okay.

有太多可以展开的话题了。

There's so many jumping off points.

顺便说一下，我最近才知道，你可能讲过这个故事好几次了，但显然谷歌大脑也是在微厨房里起步的。

By the way, I found out from the recent I mean, you've probably told this story a few times, but apparently Google Brain was also started in a micro kitchen.

对对。

Yeah, yeah.

就是，

It like,

你们的微厨房真的很重要。

Your micro kitchens are very important.

我不知道大家是否明白这一点。

I don't know if people understand.

是的，我其实碰见了吴恩达，他是斯坦福的教员。

Yeah, I actually bumped into Andrew Ng, who's a Stanford faculty member.

我认识他是因为几年前我在斯坦福做过几次演讲，所以算是认识他，我就说，哦，你在这里做什么？

And I knew him from, I'd given talks at Stanford a couple of years before, so I sort of knew him and I'm like, oh, what are you doing here?

他说，哦，我还不确定呢。

He's like, oh, I'm not sure yet.

我几周前才刚开始。

I just started a couple of weeks ago.

我打算每周来这里一天做咨询。

I'm going to spend one day a week here consulting.

我还不确定要做什么，但我在斯坦福的学生在语音识别上使用神经网络已经取得了一些好结果。

I'm not sure what I'm working on, but my students at Stanford are starting to get good results on using neural nets for speech recognition.

我当时就想，哦，神经网络。

I'm like, oh, neural nets.

我喜欢神经网络。

I like neural nets.

我回想起1990年的时候，哦，那听起来挺有意思的。

I remembered back to my I'm nineteen ninety like, oh, that sounds interesting.

我们应该训练非常非常大的神经网络。

We should train really, really big neural nets.

所以那就是……

So that was the

你这么说，我觉得这是个很有意思的第一反应，就是我们应该大幅扩大规模。

Which you say that, and I think that's a very interesting first instinct, which is that we should scale this up a lot.

对。

Yeah.

嗯，我是觉得谷歌有很强的计算能力。

Well, I mean, I felt like Google has lots of computational capability.

而且如果他们在单GPU或类似规模的模型上已经看到了好结果，而我们当时的数据中心里没有GPU，也没有其他加速器，只有大量的CPU。

And so if they were seeing good results on what were effectively single GPU or models, if we actually didn't have GPUs in our data centers then, we didn't have any accelerators, we had lots of CPUs.

但我们可以构建一个软件系统，支持模型并行和数据并行，分布在很多计算机上，最终我们训练了一个相当大的模型。

But we could build a software system that would enable you to distribute with both model parallelism and data parallelism across lots of computers, and we ended up training a pretty big model.

据我们所知，它比之前的任何神经网络都大50倍。

Was 50x bigger than any previous neural net, as far as we could tell.

这是一个有20亿参数的视觉模型，在16000个CPU核心上训练了好几周。

So it's 2,000,000,000 parameters vision model trained on 16,000 CPU cores for multiple weeks.

这让我们在ImageNet 22k（也就是22000个类别的那项任务）上取得了70%的相对错误率改进。

That's what gave us really good it gave us a 70% relative error improvement in ImageNet 22 ks, which which is the 22,000 category thing.

就是这样，我们才真正意识到，扩大规模确实很重要。

And that's how we really saw, Okay, scaling this up actually matters.

我们没有写什么复杂的扩展分析，但我们有个说法：更大的模型，更多的数据，更好的结果。

We didn't write a sophisticated scaling analysis, but we had a saying, bigger model, more data, better results.

这个说法我们坚持了六七年，一直在扩大规模。

And that was our mantra for like six or seven years of scaling.

每次这么做，我们在语音、语言、视觉方面都看到了更好的结果。

Every time we did that, we saw better results in speech, in language, in vision.

说到下注，这可能有点敏感，但你显然对此有很多看法。

Speaking of bets, and this might I'll preface with, like, this might be a little bit more sensitive topic, but you have obviously a lot of opinions about this.

我们之前有位嘉宾，David Wang，他以前在你手下工作，他几乎把谷歌在语言模型上投入不足归咎于大脑市场。

We had a previous guest, David Wang, who used to work for you, he kind of blames almost the brain marketplace as the reason that Google didn't invest enough in language models.

我想知道，你当时是否同意这种看法，还是有不同的复盘？

And I wonder if that's something you would agree with at the time, or is there a different sort of postmortem?

大脑市场指的是计算配额。

The brain marketplace for compute quotas.

计算配额。

Compute quotas.

对。

Yeah.

基本上他就是说，好吧。

Where basically he was like, Okay.

大卫之前在OpenAI做工程副总裁，后来去了谷歌。

David worked at OpenAI as VP Engine, then he worked at Google.

他觉得，从根本上说，OpenAI愿意全力以赴，把一切都押在一件事上，而谷歌则更民主。

He was like, Fundamentally, OpenAI was willing to go all in, bet the farm on one thing, whereas Google was more democratic.

每个人都有配额。

Everyone had quota.

然后我就想，好吧。

And I was like, Okay.

如果你相信规模化很重要，那这就是一个重要的全组织决策。

If you believe in scaling as an important thing, that's an important organizational wide decision to do.

对。

Yeah.

我觉得我某种程度上同意这个说法。

I think I would somewhat agree with that.

其实，我记得我写过一份一页的备忘录，说我们分散资源是愚蠢的。

I mean, think I actually wrote a one page memo saying we were being stupid by fragmenting our resources.

具体来说，当时我们在谷歌研究院，特别是Brain团队，有多个大语言模型的项目。

So in particular, at the time, we had efforts within Google Research and in the Brain team in particular on large language models.

我们还在Brain和谷歌研究院的其他部门做多模态模型。

We also had efforts on multimodal models in other parts of Brain and Google Research.

然后原来的DeepMind在做Chinchilla模型和Flamingo模型。

And then legacy DeepMind had efforts like chinchilla models and flamingo models.

所以实际上，我们不仅把算力分散到这些不同的项目上，还把最优秀的人才和最好的想法也分散了。

And so really, we were fragmenting not only our compute across those separate efforts, but also our best people and our best ideas.

对吧？

Right?

所以我就说，这太蠢了。

And so I said, this is just stupid.

我们为什么不合并起来，集中力量训练一个模型呢？

Why don't we combine things and have one effort to train

这就是那次合并，对吧。

This is the merge, yeah.

训练一个从一开始就是多模态的、各方面都很强的统一模型。

To train an awesome single unified model that is multimodal from the start, that's good at everything.

这就是Gemini项目的起源。

And that was the origin of the Gemini effort.

我那页备忘录还真管用了，挺好的。

And my one page memo worked, which is good.

名字是你起的吗？

Did you have the name?

因为可能还有人不知道，Gemini这个名字是你取的。

Because also for those who don't know it, you named Gemini.

是的，是我起的。

I did.

当时也有人提了别的名字，但我说，你知道吗，这两个组织某种程度上就像双胞胎一样走到了一起。

Was another name proposed and I said, you know, it's sort of like these two organizations really are like twins in some sense coming together.

所以我挺喜欢这个名字的。

So I kind of like that.

而且还有NASA那边的解释，早期的双子星计划是通往阿波罗计划的重要一步。

And then there's also the NASA interpretation of the early Gemini project being an important thing on your way to the Apollo project.

所以这名字听起来挺合适的。

So it seemed like a good name.

双胞胎走到一起。

Twins coming together.

对。

Right.

是的。

Yeah.

不错。

Nice.

我知道我们时间快不够了，但我很好奇你现在是怎么用AI来写代码的。

I know we're already running out of time, but I'm curious how you use AI today to code.

你可能是计算机科学史上最多产的工程师之一。

You're probably one of the most prolific engineers in the history of computer science.

我读了那篇关于你和Sanjay友谊以及你们如何合作的文章。

I was reading through the article about you and Sanjay's friendship and how you work together.

你有一句话是，你需要找到一个思维方式和你兼容的结对编程伙伴，这样你们两个人在一起就能形成互补的力量。

And you have one quote about, You need to find someone that you're going to pair program with who's compatible with your way of thinking so that the two of you together are a complementary force.

我就在想，你是怎么看待编码智能体在这方面的作用的。

And I was thinking about how you think about coding agents in this.

你是怎么塑造编码智能体，让它们适应你的思维方式的？

How do you shape coding agents to be compatible with your way of thinking?

你觉得现在的工具怎么样？

How would you rate the tools today?

事情应该往哪个方向发展？

Where should things go?

对。

Yeah.

我的意思是，首先，我觉得编程工具比一两年前好太多了。

I mean, first, I think the coding tools are getting vastly better compared to where they were a year or two years ago.

所以现在你真的可以依靠它们来完成更复杂的任务，这些任务正是你作为软件工程师想要实现的。

So now you can actually rely on them to do more complex things that you as a software engineer want to accomplish.

而且你可以把相当复杂的事情委托给这些工具。

And you can sort of delegate pretty complex things to these tools.

我觉得人类软件工程师和他们使用的编程模型之间互动的一个很好的方面是，你与那个编程模型对话的方式实际上决定了它如何与你互动。

And I think one really nice aspect about the interaction between a human software engineer and a coding model that they're working with is your way of talking to that coding model actually sort of dictates how it interacts with you.

你可以让它帮你写一堆好的测试。

You could ask it, please write a bunch of good tests for this.

你也可以让它帮你头脑风暴一些性能优化的想法。

You could ask it, please help me brainstorm performance ideas.

你这样做的方式会影响模型如何回应、处理哪些问题，以及你希望模型是更独立地去完成更大的任务，还是更多地与你互动，确保你引导的方向是正确的。

And your way of doing that is going to shape how the model responds, what kinds of problems it tackles, how much do you want the model to go off and do things that are larger and more independent versus interact with it more to make sure that you're shaping the right kinds of things.

我觉得并不是某一种风格适用于所有情况，对吧？

And I think it's not the case that any one style is the right thing for everything, right?

比如有些问题，你可能希望和模型有更频繁的互动。

Like some kinds of problems you actually want maybe a more frequent interaction style with the model.

而其他问题，你可能会说，对，你直接去写吧，因为我知道我需要这个东西。

And other ones you're just like, yeah, please just go write this because I know I need this thing.

我能把它描述清楚，然后你去完成，做完再回来。

I can specify it well enough and go off and do it and come back when you're done.

所以我认为未来会有更多独立的软件代理替你处理事情，同时需要找到合适的人机交互模式和界面，来决定什么时候该打断你，说：嘿，这里需要更多指导，或者我已经完成了这件事，接下来该做什么？

And so I do think there's going to be more of a style of having lots of independent software agents off doing things on your behalf and figuring out the right sort of human computer interaction model and UI and so on for when should it interrupt you and say, hey, need a little more guidance here, or I've done this thing, now what should I do?

我觉得我们还没有找到这个问题的最终答案。

I think we're not at the end all answer to that question.

随着模型变得更好，你为交互方式设定的那些决策可能会改变。

And as the models get better, that set of decisions you put into how the interaction should happen may change.

比如说，如果你有一个50人的实习生团队，如果他们是真人的话，你会怎么管理？

Like if you have a team of 50 interns, how would you manage that if they were people?

我觉得问题不是你想不想要50个实习生。

I think it's not Do you want 50 interns?

如果他们真的很优秀，你可能会想要，对吧？

You might if they're really good, right?

那管理起来可够多的。

That's a lot of management.

但确实很多，是的。

But it's a lot of, yeah.

我觉得，很多人拥有50个实习生，这种情况是有可能的。

I mean, I think that is probably within the realm of possibilities that lots of people could have 50 interns.

那么作为一个人，你实际上会怎么应对呢？

And so how would you actually deal with that as a person?

我想你可能会希望他们组成小团队，这样你就不用跟50个人都打交道了。

I think you would probably want them to form small sub teams so you don't have to interact with 50 of them.

你可以和其中五个团队互动，他们替你去做事情。

You can interact with five of those teams, and they're off doing things on your behalf.

但我不太确定这具体会怎么发展。

But I don't know exactly how this is going to unfold.

你怎么看待引入新人？结对编程总是有助于带来全新的想法，可以说是这样。

How do you think about bringing people The pair programming is always helpful to get net new ideas in the distribution, so to speak.

感觉随着越来越多的编码代理来写代码。

It feels as we have more of these coding agents write the code.

就很难把其他人带入到问题中来。

It's hard to bring other people into the problem.

比如说你有50个条目，对吧？

Say you go to, you you have 50 entries, right?

然后你想去找Noam Shazeer，说，嘿，Noam。

And then you want to go to Noam Shazeer, be like, Hey, Noam.

我想和你一起结对做这件事。

I want to pair on this thing.

嗯。

Mhmm.

但现在有大量并行完成的工作，你需要让他跟上进度。

But now there's this huge amount of work that has been done in parallel that you need to catch him up on.

对。

Right.

我很好奇，人们会不会在某种程度上变得更孤立，各自为战。

And I'm curious if people are going to be in a way more isolated in their teams.

就像，好吧，这50个实习生带来的上下文太多了，我很难把所有的东西都转述给你。

It's like, Okay, there's so much context in these 50 interns that it's just hard for me to relay everything back to you.

也许吧。

Maybe.

我的意思是，另一方面，想象一个没有AI辅助工具的经典软件组织。

I mean, on the other hand, imagine a classical software organization without any AI assisted tools.

你会有50个人在做事，他们的互动方式自然会非常层级化，因为这50个人会专注于系统的这一部分，而不会和那边的那些人有很多互动。

You would have 50 people doing stuff, and their interaction style is going to be naturally very hierarchical because these 50 people are going to be working on this part of the system and not interact that much with these other people over here.

但如果有五个人，每人管理50个虚拟代理，他们之间沟通的带宽可能比五个人各自协调一个50人的软件团队要高得多。

But if you have five people each managing 50 virtual agents, they might be able to actually have much higher bandwidth communication among the five people than you would have among five people who were also trying to coordinate a 50 person software team each.

我很好奇你怎么调整你的工作节奏。

I'm curious how you change your just working rhythm.

你会花更多时间提前跟人一起过规格和设计目标吗？

Do you spend more time ahead with people going through specs and design goals?

我觉得有意思的是，人们学写软件时，总被教导说写规格要特别清晰，但没人真信这个。

I mean, I do think it's interesting that whenever people were taught how to write software, they were taught that it's really important to write specifications super clearly, but no one really believed that.

就像，对，随便吧。

It was like, yeah, whatever.

我不需要那么做。

I don't need to do that.

真的吗？

Really?

我不知道。

I don't know.

其实，英语从来都不是一个真正被高度重视的东西。

Mean, the was English language never kind of an artifact that was really paid a lot of attention to.

我的意思是，它很重要，但它并不是真正驱动创意过程的核心，不像如果你要指定软件让代理为你编写，你就得非常小心地描述你的需求，因为那将决定输出的质量。

I mean, it was important, but it wasn't sort of the thing that drove the actual creative process quite as much as if you specify what software you want the agent to write for you, you'd better be pretty darn careful in how you specify that, because that's going to dictate the quality of the output.

比如，如果你没有提到它需要处理这类情况，或者这是一个非常重要的边界情况，或者你非常在意这部分性能，它可能就不会按你的想法去做。

Like if you don't cover that it needs to handle this kind of thing or that this is a super important corner case or that you really care about the performance of this part of it, it may not do what you want.

而且你越擅长与这些模型互动，我认为人们会变得更好的方式之一，就是他们会非常擅长清晰地描述需求，而不是留下模糊不清的地方。

And the better you get at interacting with these models and I think one of the ways people will get better is they will get really good at crisply specifying things rather than leaving things to ambiguity.

这实际上可能不是一件坏事。

And that is actually probably not a bad thing.

无论你是不是软件工程师，或者在做其他类型的任务，这都不是一个坏技能。

It's not a bad skill to have, regardless of whether you're a software engineer or trying to do some other kind of task.

能够清晰地表达你想要什么，将会变得非常重要。

Being able to crisply specify what it is you want is going to be really important.

对。

Yeah.