解码谷歌双子星：与杰夫·迪恩同行

这就是为什么这是一个重大突破。

And so that was why that was such a big advance.

但我想这还会衍生出其他东西。

But then I guess there's other things that seem to emerge from this.

我是说，某种概念性的理解，或者说仅通过序列和语言本身就能实现的抽象。

I mean, sort of a conceptual understanding or maybe sort of abstraction that's that's possible just through sequencing and and language alone.

我是说，那很意外吗？

I mean, that was that a surprise?

是啊。

Yeah.

我是说，我认为我们在Google Brain团队最早做的一些语言建模工作，真正关注的是如何将单词建模为高维向量，而非其表面形式（比如h e l l o或c o w），这些向量能体现单词的使用方式。

I mean, I think some of the earliest work we did on language modeling in the the Google Brain team was really about modeling words not as their surface form of, like, h e l l o or c o w, but really about a high dimensional vector that represents kind of the way in which that word is used.

你知道，人类习惯在二维和三维空间思考。

You know, we're used to thinking in two and three dimensions as humans.

但当你有100维或1000维时，千维空间里就有很大余地。

But when you have a 100 dimensions or a thousand dimensions, there's a lot of room in a thousand dimensional space.

但当你让某些事物在空间中邻近——比如通过训练模型使牛、羊、山羊和猪彼此靠近，同时远离咖啡机——

But when you have things that are nearby and you've trained the model in such a way that, you know, cow and sheep and goat and pig are all near each other, and they're very far apart from espresso machines.

虽然牛奶可能介于两者之间。

Although milk could be in between the two.

牛奶可能更靠近牛，但某种程度上介于两者之间。

Milk milk would probably be nearer the cow, but kind of in between the two.

对。

Yeah.

它们可能会位于那个100维空间中的100维线上。

They would probably be kind of on that 100 dimensional line in 100 dimensional space.

所以我认为，这些模型之所以具有惊人的强大能力，是因为它们用如此多的高维度表示事物，能同时捕捉词语、句子或段落的多个层面——毕竟它们的表示空间如此广阔。

So this is kind of why these models have surprisingly powerful capabilities, I think, is because they're representing things with so many high dimensions that they can actually really latch on to many different facets of a word or a sentence or a paragraph simultaneously because there's so much room in their in their representation.

某种程度上它提取了我们赋予语言的基础意义吧。

It sort of extracted the the grounding that we ourselves have given language, I guess.

没错。

Yeah.

我是说，当我们听到一个词时，我们不仅仅想到这个词的表面形式。

I mean, we when we're when we hear a word, we don't just think of the surface form of the word.

我们想到的是牛。

We think cow.

哦，这会触发一系列其他联想，比如牛奶、咖啡机，还有挤奶、小牛和公牛之类的。

Oh, that triggers a bunch of other things like milk or espresso machine or, you know, milking and calf and bull.

我们发现这些早期词汇表征的一个特点是：方向具有意义。

And one of the things we found with those early word representations was that directions had meaning.

比如你想想动词的现在时态，像'walk'，在这个100维空间里，从'walk'到'walked'的方向，与从'run'到'ran'的方向是一致的，就像从'reed'到'red'那样。

So if you think about present tenses of of verbs like walk, you would go in the same direction in this 100 dimensional space to get from walk to walked as you would get from run to ran, as you would go from, you know, reed to red.

哇。

Wow.

是啊。

Yeah.

所以它其实是理解'理解'的。

So it actually understands understands.

我总是不自觉地用这个词，但并非刻意为之。

I keep using that word, and I don't mean it.

但这些'是啊'的结构里确实存在时态的某种表征。

But there is, like, some representation of tenses within the structure of these Yeah.

这只是训练过程中自然涌现的特性。

And it's it's just emerged from the training process.

这不是我们刻意设计的。

It didn't it's not something we told it to do.

这只是我们使用的训练算法，以及语言中存在大量特定形式的使用方式，导致它们产生了融合。

It's just the training algorithm we used and the fact that language has lots of ways in which particular forms are used cause that to merge.

你还可以实现词汇的性别转换，比如从阳性形式变成阴性形式，反之亦然。

And you could also, for example, change from male or female versions to words and vice versa.

所以母牛到公牛的关系，就像皇后到国王、女人到男人等等，是同一个方向。

So cow to bull is the same direction as queen to king or woman to man and so on.

太神奇了。

Amazing.

但这只是我们讨论的语言层面。

But this is just with language that we're talking about here.

好吧。

So okay.

那么多模态方面是如何改变这一点的？

Tell me how does the multimodal aspect of this change?

具体来说，它有什么不同之处？

What what what is it how does it make it different?

是的。

Yeah.

因为你仍然在高维空间中表示输入数据，关键是如何从图像的像素转换到某种表示——理想情况下，你希望多模态模型拥有与我们相似的机制。

Because you're still representing the input data in these high dimensional spaces, and it's really a matter of how do you get from the pixels of an image, say, into something where ideally you'd like the multimodal model to have the same kind of thing that we have.

当我们看到一头牛时，这会在大脑中触发与读到'牛'这个词或听到牛走动时相似的神经活动。

When we see a cow, that triggers kind of similar activations in our brain to reading the word cow to hearing a cow move.

你希望训练模型使其具有这种联合意义和表征，无论它们是如何获取这些输入数据的。

And you kinda wanna train models so that they have that joint meaning and representation regardless of the way they arrived at that input data.

所以如果模型看到一个牛走过田野的视频，这应该会触发模型中与之相关的一系列反应，基于模型通过深度分层结构建立的激活模式。

So if they see a video of a cow walking through a field, that should trigger a whole bunch of things that are related to that in the model based on the activations that the model has built over typically, these are very deep layered models.

通常最底层具有非常简单的表征，而模型更高层则在这些表征基础上构建更复杂有趣的特征组合，无论是文字、图像还是其他形式的表征。

And so the lowest layers typically have very simple representations, and then the higher layers in the model build on those representations and build more interesting and complex combinations of features and and representations of be it words or images or

所以你说从底层开始就是多模态的，对吧，

So when you're saying multimodal from the ground up, right,

也就是

which is

关于Gemini你常听到的一种说法是，并不是说这边有个词汇区，那边有个像素区，然后你在两者之间进行翻译。

kind of a big phrase that you hear about Gemini, it's not that you've got, like, the word section over here, pixel section over here, and you're translating between one and the other.

对。

Right.

但在模型内部，这些表征是

But, like, in the model itself, those representations are

是的。

Yeah.

在模型非常早期的阶段。

Very early in the model.

这是否会让初期搭建时变得更困难？

Does that make it harder at the beginning when you're setting it up?

会让操作变得更复杂吗？

Does it make it more difficult to do?

没错。

Yeah.

我认为，弄清楚如何将不同模态整合到模型中，以及如何训练多模态模型，确实比单纯的纯语言或纯字符模型要复杂得多。

I mean, I think figuring out how to integrate different modalities into the model and, you know, how should you train a multimodal model is more complex than a simpler pure language or pure character based model.

但你会从中获得很多好处，比如有时会出现跨模态迁移——看到关于牛的视觉内容实际上能辅助语言理解。

But you get a lot of benefits from it in that you get sometimes cross modal transfer where now seeing visual stuff about cows actually helps inform the language.

比如你可能读过很多关于草地上奶牛的描述，但现在它突然看到了相关图片和视频。

You know, maybe you'd seen a bunch of descriptions of of cows in meadows or in something, but now it some suddenly has seen images of that and videos of that.

它实际上能以某种方式将这些表征结合起来，使得模型内部对'牛'这个概念的触发机制变得统一——无论你看到的是'牛'这个词还是牛的图像。

And it's actually able to bring those representations together in a way that makes similar things trigger inside the the model regardless of whether you saw the word cow or the image of cow.

能举个未来应用场景的例子吗？

Give me an example of the type of situation you you see this being useful in in the future.

其实现在已经很有用了，这很好。

Well, I think it's already useful, which is good.

举个例子，你希望能够输入一张手写白板上的数学解题过程图片，然后判断学生这道题做对了吗？

I mean, as one example, you wanna be able to take in an image of kind of a handwritten whiteboard worked out math problem and say, the student get this problem right?

所以现在你需要在一个例子中真正展现多模态能力。

And so now you need to really bring in the multimodal capabilities in one example.

实际上你需要进行手写识别，并从中理解内容，明白吗？

You need to actually, you know, do handwriting recognition, understand from that, okay.

这是某人写在白板上的物理题，在早期的Gemini技术报告中可能还配有一张滑雪者下坡的图片。

It's a physics problem that someone's written on the board, and it's got maybe a picture of a skier going down a slope in one of the early Gemini tech reports.

我们有个很好的例子：一个学生在白板上解题后，你可以直接问Gemini'这个学生做对了吗？'

We have this good example of a student who'd worked out a problem on on a whiteboard, and you 'd actually ask Gemini, you know, did the student get this problem right?

如果错了，他们错在哪里？能否解释正确解法？

If not, where did they go wrong, and can you explain how to solve the problem correctly?

它确实能判断出学生错误地应用了无摩擦斜坡滑雪者的公式，他们用了斜边而非高度。

And it was actually able to tell that, you know, the student had incorrectly applied, the formula for a skier going down a frictionless slope, and instead they used the hypotenuse instead of the height.

然后它会说'哦不对'。

And it said, oh, no.

实际上你应该用这个公式，这是完整的解题过程。

Actually, you should have used this, and here's the problem worked out.

它完成了所有这些，识别了所有手写内容，并判断出这是道物理题。

And it did all that and recognized all the handwriting and the fact that this was a physics problem.

模型已有的这类物理知识正好适用这种情况。

This kind of physics knowledge that the model already had sort of was the right thing to apply.

我觉得这是将现有Gemini模型与现有教育模式结合的绝妙方式。

I mean, that's a really neat way that you could use the existing model of Gemini in the existing model of education, I guess.

完全同意。

Totally.

没错。

Yeah.

但实际上，我认为这些系统彼此之间并非完全孤立。

But but I suppose, actually, these are not kind of isolated systems from one another.

那么从某些方面来说，你是否认为这些多模态模型会彻底改变我们的教育方式？

So in some ways, do you think that these multimodal models will change the way that we have to do education, full stop?

我认为AI工具助力教育的潜力非常惊人，作为社会整体，我们才刚刚踏上这段旅程的起点。

I mean, I think the potential for using AI tools to help education is really amazing, and we're sort of just at the beginning of this journey as a society, I think.

比如我们知道，接受一对一辅导的学生，其教育成果比传统30人课堂的学生高出两个标准差。

You know, we know, for example, that educational outcomes of students who get one on one tutoring from another person are two standard deviations better than students who are have a traditional classroom setting of a teacher and, you know, 30 or so students.

因此如何让每个人都能享受到个性化教育辅导的益处——了解学生已知和未知的领域，用最适合的方式辅助学习——这正是AI在教育领域的潜力所在。

So how could we get everyone to the point where they feel like they have the benefit of an educational tutor that's one on one, understands what they know, understands what they don't know, can help them learn in the way that they learn best, that is the potential of AI in education.

实际上我们距离这样的场景并不遥远：你可以指着某份材料对Gemini模型说'能帮我学习这个吗？'

And I think, really, we're not that far away from something where you could point a Gemini model or a future Gemini model at something, some piece of material, and say, can you help me learn this?

比如打开生物课本第六章，里面有许多图片——

Take the chapter six in your biology textbook or something, and it's got a bunch of images.

还有大量文字内容——

It's got a bunch of text.

可能还包含你观看过的讲座视频——

Maybe it's got a lecture video that you watched as well.

这时你完全可以说'这部分我真的不明白'

And then you can actually say, I really don't understand this thing.

'能帮我理解吗？'

Can you help me understand it?

它可以向你提问，

It can ask you questions.

你也可以向它提问，

It can you can ask it questions.

然后你来回答问题。

You can answer the questions.

它能评估你的对错，并在学习旅程中真正引导你，因为这是个性化的。我们应该能让全球许多人接触到它，不仅限于英语，还包括世界各地数百种语言。

It can assess are you right or wrong and really guide you in your learning journey because it's individualized, and we should be able to get that to many, many people around the world in not just English, in, you know, hundreds and hundreds of languages all around the world.

我是说，我...我理解你提到的多种语言问题，以及努力让这些工具尽可能广泛普及。

I mean, I so I I take what you said about the the lots of different languages and and trying to make these as broadly available as possible.

但这是否存在制造双重体系的风险？一方面，如你所描述的，能接触这些工具的人会获得更好的结果，加速他们的学习和生产力？

But is there a danger of creating a bit of a two tier system here where where on the one hand, people who have access to these tools, as you described, get far better outcomes, you know, accelerate their own learning and and and productivity?

而任何不够幸运、无法接触这些工具的人将真正陷入困境。

And then anybody who is not fortunate enough to have access to the tools really struggles.

我是说，这是否是你担忧的问题？

I mean, is that is that something that concerns you?

绝对是。

Absolutely.

是的。

Yeah.

我认为确实存在制造双重体系的风险。

I mean, I think there is definitely a risk of creating two tier systems.

我认为我们应该努力使这些技术尽可能广泛普及，如果可能的话让所有人都能使用，并真正发挥其对社会的作用，使其在教育、医疗等领域变得可负担甚至免费——我认为医疗是AI能真正改变可及性的另一个重要领域。

I I think what we should strive to do is make these technologies as broadly accessible, universally accessible if we can for everyone and really try to lean into the strengths of what that will do for society and to make it affordable or free for people to take advantage of the capabilities for education, for health care, I think, is another area that has huge potential for AI to really make a big difference in health care accessibility.

那么我想回到Gemini这个话题，如果可以的话。

So I wanna go back to Gemini, if we can.

当然。

Sure.

好的。

Okay.

我想如果你从谷歌搜索起步，准确性必定是你最关心的核心要素。

So I guess if you started off with Google search, factuality must have been absolutely at the cornerstone of everything that you cared about.

但Gemini...我是说，我一直在使用它。

But Gemini, mean, I work with it all the time.

我想你应该见过它说一些相当离谱的话。

I imagine you've seen it say some quite outlandish things.

是的。

Yeah.

你是如何在脑海中调和这种矛盾的？比如在某些时候放弃对绝对事实性的执着需求？

How are you sort of squaring that circle in your head of, like, releasing perhaps some of the need for absolute factuality at all times?

是的。

Yeah.

作为一家公司，这实际上是一个微妙的平衡，因为我们从根源上是一家以搜索为基础的公司。

It's actually a tricky balance as a company because we are sort of from our origins a search based company.

正如你所说，提供准确的事实信息是搜索引擎体验的巅峰。

And as you say, providing accurate factual information is kind of the pinnacle of a search engine experience.

但我认为我们其实已经构建了有趣的大型语言模型，人们很享受与之对话。

But I think we actually had built interesting large language models internally that people enjoyed conversing with.

事实上，其中一些模型在疫情期间就已在内部开放使用，当时大家都在家办公。

Actually, some of them were available internally during the pandemic, so people were all at home.

你甚至能看到午餐时间内部使用量激增，因为人们都在和虚拟聊天机器人对话——当你独自在家时还能和谁聊天呢？

And you could actually see internal usage spike during lunchtime because people were having conversations with their their virtual chatbot because who else are you gonna talk to when you're home alone or whatever?

但这些模型本质上是被训练来预测合理的下一个标记。

But these models are trained to predict plausible NEXT tokens, essentially.

所谓标记，你可以把它理解为一个单词或单词的一部分。

So a token, you can think of it as a word or a piece of a word.

所以当你预测合理的下一个标记时，这与绝对真理是两回事。

And so when you predict plausible NEXT tokens, that's a different thing than that is absolute truth.

对吧？

Right?

这是概率上合理的句子，与事实是不同的概念。

It's a probabilistically plausible sentence, and that's different than a fact.

我认为随着时间推移我们意识到的一点是，这些模型即使不是百分百准确，实际上也相当有用。

And I think one of the things we we realized over time was these models can actually be quite useful even if they're not a 100% factual.

所以我意识到存在所有这些其他用例，比如能否用五个要点总结这份幻灯片？

And so I think realizing there's all these other use cases or can you summarize this slide deck in five bullets?

当然你可以争论第五个要点是否完全正确，但能获得4.5个关于幻灯片的准确要点仍然非常实用。

And, yes, you could argue about is that fifth bullet exactly right, but still pretty useful to get 4.5 bullets that are factually accurate about the slide deck.

而且你知道，我们正在努力做到五个要点都完全准确。

And, you know, we'd we're striving to make it five factually accurate bullets.

但即便如此，我认为这些模型的实际效用已经相当高了。

But even without that, I think the utility of these models is actually quite high.

这个认知过程是否令人感到不安？

Was it an uncomfortable realization?

因为当然，其他实验室确实更早推出了他们的模型。

Because, of course, other labs did push out their models earlier.

是的。

Yeah.

你们是否因为这种准确性问题而表现得过于谨慎？

Do you think that that that you guys had an abundance of caution because of this factual issue?

我认为我们有多方面的顾虑。

I mean, I think we had a number of different concerns.

准确性只是其中之一，模型训练方式和输出结果中的毒性和偏见问题也是我们重点改进的领域。

Factuality being one of them, toxicity and bias in the way the model is trained and the outputs that it can produce is an area where we wanna make the model less biased in a lot of ways.

因此在向公众发布前，我们在许多方面都希望保持相对谨慎的态度。

And so there were a whole number of areas where we wanted to sort of be relatively cautious before releasing things to the general public.

我认为我们已经解决了大部分问题，虽然像准确性和偏见等领域仍有改进空间，但当前发布的产品已经具有实用价值。

And I think we've we've gotten a lot of those issues kind of sorted out enough that we think the products we put out in this space are useful, even though there are obviously room for improvement in things like factuality and in bias and other areas.

所以这对人们来说需要些适应——既要追求尽善尽美，也要明白不发布产品反而可能让很多人错失虽有瑕疵但仍有价值的东西。

So I think that's taken a little bit of an adjustment for people is you know, strive for the best you can be, but also realize that by not releasing something, you're sort of holding back something that could be useful for a lot of people even with its sort of foibles.

但既然有这些缺点，那么我们该何去何从？

But then with those foibles then, so in which direction do we go from here?

在我看来，计算方式确实发生了实质性的转变。

I mean, it sort of seems to me that there's been this this real shift in the way that computing happens as it were.

你知道的，用计算器算同样的算式两次，会得到同样的答案两次。

You know, you get a calculator, you put in the same sum twice, you get the same answer twice.

是的。

Yeah.

而我们现在正处于概率计算的时代。

Whereas we're now in an era of probabilistic computing.

所以我在想，公众是否必须接受这个现实，接受我们处在一个事物更像人类、会犯错的时代？还是你认为这是可以修正的？

And so I wonder whether the the is it that the public has to come to terms with that and sort of accept that we're in an era where things are much more human like and that they can make mistakes, or is it something that you think is fixable?

我认为两者兼而有之。

I think it's some of both.

对吧？

Right?

我觉得有很多技术手段可以改善事实性领域的问题。

I mean, I think there's a bunch of technical approaches to some of these problems that will make the factuality area issues better.

举个例子，模型训练所用的数据——数万亿的文本标记和其他数据混合在这个由数百亿参数组成的巨大‘汤’里。我喜欢把这理解为：你见过很多东西，但记得不太清楚。

One instance is if you think about the data the model is trained on, like trillions of tokens of text and other data that are then mixed together in this giant soup of, you know, billions and billions of parameters, I I like to think of that as, like, you've seen a lot of stuff, but you don't recall it very well.

而我们Gemini一直在推进的一个方向就是扩展上下文窗口。

Whereas one of the things we've been pushing on in Gemini is having a long context window.

当你拥有一个能容纳大量直接信息的长空间时，你可以用各种方式对这些信息进行总结、操作、比较或提取。

So when you have a long bit of space where you can put a lot of direct information that you're trying to summarize or manipulate or compare in various ways extract information from.

模型对上下文窗口中的信息实际上有更清晰的把握。

That information in the context window, the model actually has a much clearer view of.

对吧？

Right?

它包含了实际文本及其表征，这些内容与模型见过的其他信息是分离的。

It's got, like, the actual text and the representations of that text not tangled together with everything else it's seen.

所以这个上下文窗口某种程度上是模型在那一刻认为重要的部分。

So this context window is sort of the bit that the the the model can see as important at that moment.

是的。

Yeah.

它能比其他训练过程中见过的内容更精确地对此进行推理。

It can sort of reason about that in more fidelity than other things that it's seen in its training process.

比如它可以处理五篇科学文章的PDF，然后你可以提问：'能告诉我这些文章的共同主题吗？'

So it can take, like, five PDFs of scientific articles, and then you can ask questions about it, like, can you please tell me common themes across these articles?

由于它已构建了这些文章内容的内部表征，因此确实能完成这个任务。

And then it's actually able to do that because it has its own representation of all the contents of those.

这就是我们大力推动Gemini模型超长上下文窗口的原因之一——这种能力对事实核查、视频摘要等各类任务都非常实用。

That's one of the reasons we've been pushing a lot on very long context windows for Gemini models is because I think that's a really useful capability for factuality, for, you know, video summarization, for all kinds of things.

但上下文窗口是否存在上限呢？

Is there a limit, though, to the context window?

难道可以不断扩展直到引发某种无限冲突？

You just push and push and push and push until it's sort of an infinite conflict?

这是个绝妙的问题。

That is an excellent question.

目前注意力机制的计算成本相当高昂。

Currently, the computational aspects of the attention process are quite expensive.

尝试扩展得越长，所需代价就越大。

So the longer you try to make it, the more expensive it gets.

耗时方面的代价，但

Expensive in terms of time, but

还包括计算时间，最终会涉及时间、资金、算力等各类成本。

also Compute time and Eventually, time and money and compute and and all kinds of things.

但我们认为通过算法改进，有可能突破目前200万token的上下文窗口限制。

But we think it may be possible to come up with algorithmic improvements that enable you to go beyond that 2,000,000 token context window, which is what we have now.

100万token已经相当多了。

A million tokens is quite a lot.

100万token大约相当于600页文本。

A million tokens is about 600 pages of text.

所以基本上能涵盖大多数书籍，或者20篇文章。

So that's most books, you know, 20 articles.

相当于1小时的视频内容。

It's an hour of video.

另一方面的看法呢？

About on the other side?

因为你刚才提到需要两方面考虑，可能人们需要调整预期。

Because you said it was a little bit of both, that some that perhaps people have to adjust their expectations.

我认为这些模型是工具，人们需要了解工具的能力边界，也要明白哪些场景不适合使用。

So I think these models are tools, and people need to understand the capabilities of their tools, but also some of the ways in which you probably don't wanna use the tool.

所以这某种程度上是个用户教育的过程。

So think it's a bit of an educational process for people.

不要盲目相信语言模型输出的所有内容。

Don't just trust every fact that comes out of a language model right off the bat.

需要保持一定的审慎态度。

You need to apply a bit of scrutiny to that.

就像我们现在教育人们：网上看到的内容未必真实。

Sort of like I think we've taught people these days that if you see something online, that doesn't necessarily make it true.

我认为对语言模型的某些输出保持同等程度的怀疑是必要的。

I think a similar degree of skepticism for some kinds of things from language models is probably also appropriate.

随着模型进步这种怀疑可能会减少，但对某些内容保持'这可能不准确'的谨慎态度仍是明智的。

That skepticism may decrease over time as the models improve, but it's good to take it with a healthy dose of, you know, oh, that might not actually be true for some of these things.

除了上下文窗口，是的。

Aside from the context windows Yeah.

有没有什么方法，在你编写提示词时，嗯。

Are there ways that you can, yourself, when you're sort of writing in prompts Mhmm.

能尽量降低最终得到完全胡编乱造内容的风险？

Sort of minimize the risk of ending up with something that's that's a complete hallucination?

谷歌研究人员提出的一种技术叫做'思维链提示'。

So one technique that Google researchers kind of came up with is what's called chain of thought prompting.

同理，如果你给模型一个有趣的数学题并直接问：'好，答案是什么？'

So in the same way, if you just give the model a sort of interesting math problem and you say, okay.

答案是什么？

What's the answer?

它可能答对，也可能出错。

You know, I may get it correct, but it may not.

但如果你说：'这里有个有趣的数学题，

And if instead you say, here's an interesting math problem.

能否逐步展示你的解题步骤？'

Can you please show your work step by step?

就像回忆你四年级的数学老师

So if you remember back to your fourth grade math teacher

噢，确实。

Oh, I did.

他/她总会说：'你应该逐步展示解题过程，最后再写下答案。'

He or she was probably saying, you should really show your work step by step and then get to the final answer and then write the final answer down.

这部分是因为这能帮助你完成多步骤思考——从理解题意，到计算这个基于那个，最终得出答案。

And that's partly because that helps you get through that, you know, multistep thinking process of how do you actually go from, you know, what's being asked to, okay, I need to calculate this, calculate this based on that, and so on, and finally get to the answer.

事实证明，这不仅让模型的输出更可解释（因为它展示了思考步骤），还能提高答案的正确率。

And it turns out that not only makes the model's output more interpretable because it kind of tells you what steps it's going through, but it also makes it more likely to get the correct answer.

但如果这不是数学问题呢？

What if it's not a math problem, though?

是啊。

Yeah.

我是说，即使在答案范围不明确的情况下，这种方法也勉强可行。

I mean, even in noncrisply defined right answer domains, this approach kind of works.

这其中有些微妙之处，我认为人们需要真正学习如何使用这些模型，而提示的方式实际上会极大影响输出质量的高低。

And there's a bit of subtlety, and and I think people need to actually learn how to use these models, and the way in which you prompt them is actually a big differentiator in how high quality the output is.

比如，如果你说'总结这个'，可能会得到一个结果。

Like, if you say, Summarize this, that might lead to one outcome.

但如果你说'请总结这篇文章，给出五个要点突出文章重要内容，并指出作者提到的两个缺点'。

If you say, please summarize this and give me five bullet points that highlight the major important pieces of the article and identify two cons that the author wrote down.

这样说就比单纯说'总结这个'给模型提供了更清晰的操作指令。

If you say that, that's a much clearer set of instructions to what the model should do than just summarize this.

所以当我们把这些因素结合起来时，既需要分步拆解流程，也要理解更多上下文和多模态内容。

So when we put these things together then, so sort of breaking down step by step processes, but also understanding more context and the multimodal stuff too.

嗯。

Mhmm.

我们是否正在走向这样一个未来：这类多模态模型能理解我们作为个体的个人偏好？

Are we moving towards a situation where these kind of multimodal models will understand us as individuals and our preferences?

是的。

Yeah.

我认为你真正想要的，是一个高度个性化的Gemini版本，它既要理解你当下想做的事，也要明白你做这件事的具体情境。

I mean, I think what you really want, I think, is a sort of very personal version of Gemini for you that understands, you know, what it is you're trying to do right now, but also understands the context in which you're trying to do that.

我是素食主义者。

I'm vegetarian.

所以如果我问Gemini伦敦餐厅推荐，而它知道我是素食者的话...

So if I'm asking Gemini about restaurant recommendations in London and it knew that I was a vegetarian.

它的推荐会与我不在时不同。

It would recommend different things than if I was not.

我认为一个对所有用户需求一视同仁的通用模型，远不如真正了解你和你的背景的模型来得好。

And I think a general model that is serving the needs of every person the same is not gonna be as good as one that actually understands a lot about you and your context.

要知道，有些查询你可能想对模型提出，但目前用Gemini还无法完全实现，但你能想象未来会需要这些功能。

You know, there are some kinds of queries you might like to ask a model that you can't quite do today with Gemini, but you could imagine wanting to do.

明白吗？

You know?

你能把我上周徒步时拍的照片做成插画故事书，今晚给孩子当睡前读物吗？

Can you take the the pictures I took on the hike last week and make an illustrated storybook for my kid's bedtime tonight?

它会知道这些徒步照片的拍摄背景，并懂得如何制作吸引孩子的插画故事书。

And it would know where those pictures from on your hike were and how to make an illustrated storybook that would appeal to child.

它或许还能根据你孩子的年龄调整内容使其适龄。

They would maybe know how old your child was to make it age appropriate.

所以虽然现在做不到，但这会是很有用的功能。

So I think you can't do that now, but that could be something useful.

你需要让人们主动选择使用这个功能。

You want people to opt in to that.

我认为，你希望模型掌握并用于背景分析的资料越多，就越需要让人们理解正在发生什么。

I think the more information you want the model to know and have in context, I think the more you wanna sort of have people understand what what is happening.

我认为我们将能做到的不是用这些数据训练模型版本，而是让正确信息在生成回答时可被随时调用。

One of the things I think we'll be able to do is not train a version of the model on that data, but just have the right information available in context in order to sort of call upon it when generating responses.

我觉得这会很棒。

And I think that would be pretty nice.

所以就像你们已经建立了某种通用框架 对

So as in you've got, like, this sort of this general structure Yep.

可以把自己的背景信息加载上去 没错

That you can almost imprint your own context onto Yeah.

那对你来说有点私人了。

Then that's kinda private for you.

那是

That's

没错。

right.

不错。

Nice.

是的。

Yeah.

听起来似乎相当不错。

That seems like it'd be pretty good.

是的。

Yeah.

那确实会相当不错。

That would be pretty good.

我们是否仅限于音频、视觉以及屏幕上可见的内容、语言这类东西？

Are we limited here to just audio and visual and, you know, things that you can see on a screen, language, whatever?

还是说我们预期这类助手最终会从电脑里走出来？

Or or do we ever expect that these kind of assistants will come out of our computers as well?

对。

Yeah.

我认为实际上有很多新型数据模式并非严格意义上的人类模式，而我们希望这些模型能够理解。

I mean, I think there's actually a lot of different kinds of new modalities of data that aren't sort of strictly human modalities that we want these models to understand.

比如全球各地的温度读数有助于天气预报，或者基因序列，自动驾驶汽车的激光雷达数据，以及机器人应用。

So lots of temperature readings across the earth to help with weather prediction or genetic sequences or LiDAR data for autonomous vehicles or robotics applications.

在某些场景下，你希望这些模型或许能协助现实世界的机器人应用，能够与机器人设备对话，用简单语言给它下达指令。

And then in one setting, you want these models to perhaps be able to help with real world robotics applications, be able to talk to a robotic device, give it sort of instructions in plain language.

你能不能去厨房擦一下台面，把我留在台上的汽水罐回收了，然后给我带一袋开心果什么的？

You know, can you please go into the kitchen and wipe the counter down and recycle the soda can I left on the counter and then bring me a bag of pistachios or something?

对吧？

Right?

传统上机器人一直无法理解这种语言，但我认为我们正处在实现这种能力的临界点，然后能让机器人在像这个房间这样杂乱的环境中完成50或100项实用任务，而不是传统上机器人已被部署的那些非常受控的环境，比如工厂流水线那种它们从这里到那里的非常可预测的场景。

Like, robots have traditionally not been able to understand language like that, but I think we're on the cusp of enabling that kind of capability and then being able to have robots do 50 or a 100 useful tasks even in messy environments like this room rather than kind of the traditional setting in which robots have already been deployed in the world, which is sort of very controlled environments like, you know, factory assembly line kind of things where they go from there to there, and it's a very predictable thing.

我们一直在讨论这些作为助手的东西，某种程度上是以这种方式增强人类能力。

We've been talking here as as assistants, you know, these things as being sort of augmenting human abilities in that way.

我能在医疗环境和教育环境中看到它的应用。

And I can see it in in medical settings and education settings.

但这种多模态特性是否还能为我们提供更多，比如在理解世界方面？

But is there is there more that the multimodal aspect of this offers us in terms of, I don't know, like, how we understand the world.

是的。

Yeah.

我认为这些模型现在能做的是通过几步推理，从你的要求出发去完成某些事情。

I mean, I think what these models can do now is often do a few steps of reasoning to get from what you asked it to do in order to sort of accomplish something.

随着这些模型能力的提升，你将能让模型与你协作完成更复杂的任务。

And I think as these models improve in capability, you'll be able to sort of get models to work with you to do much more complex tasks.

这就像是区别'你能从椅子租赁店订一批椅子'和'帮我策划一个会议'。

And it's sort of a difference between can you order a bunch of chairs at the chair rental place versus plan me a conference.

对吧？

Right?

后者层级更高、复杂得多。

The latter is much higher level, much more complex.

合适的模型会问你一堆后续问题，因为这里面存在模糊性。

You know, the right model would kind of ask you a bunch of follow-up questions because there's ambiguity in there.

比如，有多少人要来？

You know, how many people are coming?

你知道这是关于什么的吗？

You know, what's it about?

就像人类一样

Just like a human

你加入吗？

are you in?

是的。

Yeah.

你在哪个国家？

What country are you in?

你想在哪里进行？

Where do you wanna have it?

什么时候？

When?

然后我们就可以出发，真正完成许多为了实现那个高层次目标可能需要做的事情。

And then we kind of set off and actually be able to accomplish a lot of the kind of things that you might want done in order to do that high level goal.

但如果你有这种概念链接或这些概念链接，我现在要回到牛这个话题上。

But then if you have this this sort of conceptual links or these conceptual links I'm I'm going back to cow here.

对吧？

Right?

它能理解图片，也能理解，我不知道，我猜是重力，通过看网上的视频学会的。

And it understands pictures, and it understands, I don't know, I guess, gravity, having seen videos of on the Internet.

它可能看过物理入门讲座之类的。

It probably watched, like, introductory lectures on physics.

对吧？

Right?

哦，哇。

Oh, wow.

好的。

Okay.

所以它从这个角度理解。

So it understands it from that perspective.

对。

Yeah.

对。

Yeah.

同时还看到一堆东西在坠落。

And while also seeing a bunch of things falling.

好的。

Okay.

那么你能在某天进去说，给我画一张非常高效飞机的蓝图吗？

So then could you go in one day and say, draw me the blueprint for a really efficient airplane?

可以。

Yeah.

我是说，我认为这些模型需要配合的是一个探索过程，这个探索过程可以表现为，比如它不需要在两百毫秒内给你答案。

I mean, I think one of the things these models need to be partnered with is some exploratory process, and that exploratory process can come in the form of, know, maybe it doesn't need to give you an answer in two hundred milliseconds.

也许你明天拿到飞机设计图就会满意。

Maybe you'd be happy with your airplane tomorrow.

对吧？

Right?

所以我认为到那时，你在设计系统时会有更多自由，让它们能高效完成这类任务，比如可以离开去尝试一些实验，可能在它们能访问的模拟器中，或者它们可能为基本流体动力学之类创建一个模拟器，然后尝试多种设计。

And so I think at that point, then you have a lot more freedom in how would you design systems to be able to efficiently do things like that where they can go off and try a few experiments maybe in a simulator that they have access to or maybe they create a simulator for basic fluid dynamics or something, and they try a bunch of designs.

也许它们对飞机形状有些想法，因为见过许多现有飞机。

Maybe they have some ideas about what airplane shapes make sense, having seen a bunch of existing airplanes.

这样它们就能尝试完成你要求的任务。

And so then they can kind of try to accomplish what it is you asked.

希望他们首先会问你，你希望你的飞机具备哪些特性？

Hopefully, they first ask you, well, what characteristics do you want your airplane to have?

原来一直都是纸飞机。

It was a paper airplane all along.

是啊。

Yeah.

纸飞机。

Paper airplane.

没错。

Yeah.

重要的是知道它是不是纸做的。

It's important to know if it's paper.

比如，那样能大幅降低成本。

Like, that that reduces the cost a lot.

所以

So

我认为这类功能最终都会实现。

I think those kinds of things will come eventually.

很难准确预测这些能力何时会出现，因为这涉及到模型推理能力、所需知识、任务要求及交互方式的复杂整合。

It's a little hard to tell exactly when those capabilities you know, that's a pretty complicated sort of integration of what you want the reasoning in the model to do, the knowledge it needs, what you're asking it to do, and how you're asking it.

但我们已经看到这些模型在五年、十年间取得了相当大的能力进步。

But we're already seeing pretty big advances in capabilities of these models over five year, ten year periods.

因此在五年或十年内，这可能会成为现实。

And so over a five year or ten year period, that might be possible.

甚至可能更快实现——比如‘你能帮我设计一架具备这些特性的飞机吗？’这样的需求。

It might even be sooner than that for, you know, can you help me design an airplane with these characteristics?

但我想这些大概算是我们期待中阿波罗系统的早期雏形吧。

But I guess these are, like, the early, early precursors to what we might hope Apollo would be.

是的。

Yeah.

没错。

Exactly.

所以它才叫双子座。

That's why it's Gemini.

所以它才叫双子座。

That's why it's Gemini.

太棒了。

Amazing.

杰夫，非常感谢你参加我的节目。

Jeff, thank you so much for joining me.

很荣幸来到这里。

It's a pleasure to be here.

谢谢邀请。

Thank you for having me.

从很多方面来说，我认为杰夫的整个故事就是关于规模的故事。

In a lot of ways, I think Jeff's whole story is is what about scale.

对于谷歌搜索来说，关键在于如何获取更多网页、更多用户、更快的查询。

For Google Search, it was about how do you get more of the web, more users, faster queries.

对于神经网络来说，关键在于更强的计算能力、更多的机器。

For neural networks, it was about more computing power, more machines.

而在最近的机器学习时代，关键在于越来越多的数据。

And in the recent era of machine learning, it's been about more and more and more data.

但从这一切中诞生了某种东西——一个真正意义上的世界概念模型，它能够进行抽象思维，并且已经展现出提升人类生产力的能力。

But something emerges from all of that, a genuine conceptual model of the world, one that is capable of abstraction and has already a proven ability to enhance human productivity.

值得注意的是杰夫并没有就此止步。

And it's telling that Geoff isn't finished there.

未来还会有更多进展，更多传感器，更多模式。

There is more to come, more sensors, more modes.

当与这栋大楼诞生的强化学习工具相结合时，或许还能在通往AGI的道路上取得更多进展。

And when combined with the reinforcement learning tools that were born in this building, maybe also more progress on the path to AGI.

接下来，在本系列节目中我们还将带来更多精彩对话，话题涵盖人工智能如何加速科学发现进程，到探索智能体将如何推动人工智能领域发展。

Now, we have got plenty more amazing conversations coming up later in this series on topics ranging from how AI is accelerating the pace of scientific discoveries to exploring how agents will advance the field of artificial intelligence.

如果您喜欢本期节目，请务必订阅我们的播客。

Now if you've enjoyed this episode, please make sure that you subscribe to our podcast.

如果您有任何反馈或想推荐希望听到的嘉宾，何不在YouTube上给我们留言呢？

And if you have any feedback or you want to suggest a guest that you'd like to hear from, then why not leave us a comment on YouTube?

下次再见。

Until next time.