推理工程大师课——Philip Kiely & Ali Taha，Baseten

对。

Yeah.

我的意思是，视觉编码器之间的标准化程度要低一些。

It's I mean, there's a little bit less standardization among vision encoders.

对。

Yeah.

所以支持矩阵可能会稍微稀疏一些。

So the sort of support matrix can be a little bit sparser.

但总的来说，对，它只是整个系统中一个很小的部分。

But overall, yeah, it's it's a pretty it's a pretty minor component of the overall system.

而且最终，你从这个系统中得到的是，突然之间，你有了Kimi视觉、GLM权重和DeepSeek注意力机制，全部集成在一个模型里。

And and ultimately, what you get out of the system is all of a sudden you have Kimi vision, GLM weights, and DeepSeek attention all in one model.

我认为，这正是开源的力量和美妙之处——你可以把所有这些不同的组件组合在一起，形成一个比任何单独组件都更强大的系统。

And that's, I think, a lot of the power and beauty of open source is that you can take all of these different components and and combine them together into a system that's better than anyone can be individually.

人们以前常说，你还可以做Franklin合并，就是从每个模型中提取一些层。

People used to say that you would also do Franklin merges, where you would take like layers from each model.

现在还有人这么做吗？

Does anyone do that anymore?

嗯，回到你之前提到的，就是新模型刚出来时支持它需要做的工作，比如GLM-5.2或者Max M3之类的，有时候确实得换掉一些东西。

Well, to your point previously when you were mentioning like the work that goes into supporting a model when it first comes out, like GLM-5.2 or Max M3 or whatever the case is, sometimes you do have to like you do have to switch out some things.

比如说，Mini Max M3用的是全注意力机制。

Like for instance, Mini Max M3 had uses full attention.

在全注意力下，你会遇到一个非常严重的瓶颈，因为你在做

And with full attention, you end up with this, like, insane bottleneck inspector because you're

做

doing

自回归生成三个token，而且你要对所有序列中的token做O(n²)的计算。

autoregressive token generation for three tokens, and you're doing this, like, like, o n squared over all of the tokens that are in your sequence.

你的KV缓存会非常大，因为它不是稀疏的，也不是top-k的。

Your KV cache is like very large because it's not sparse, it's not top k.

所以我们觉得更好的做法是，嗯，把这个层换成另一个模型里用GQA之类的层。

So we find it better to like, okay, we're gonna replace this, you know, we're gonna replace this layer with a layer from another model that's using like GQA for instance.

然后只要训练得当，就能让它达到同样的接受率。

And then just with the right training, you can get it to have the same acceptance rate.

所以从其他模型改造层是完全可行的，而且实际上非常必要。

So it is it is very possible to to retrofit layers from other models and very much needed actually.

如果某一层效率不高，那训练就成了挑战。

If a layer is like inefficient, the training just becomes the challenge.

比如，怎么确保训练得当呢？

Like how do you ensure that you train it properly?

这又回到了之前提到的训练和推理之间的衔接问题。

Which again to earlier point is like the the mesh between training and inference.

也就是说，要想推理速度快，训练必须非常扎实，我觉得这一点越来越明显了。

As in like you need very good training in order to do fast inference, that's like I feel like more and more becoming true.

对。

Yeah.

在支持方面，还有别的吗？比如你说要让它完全达到生产就绪的状态。

Anything else on the support side when you say like get it to fully production ready?

对。

Yeah.

我觉得还有一个问题就是，你知道，我们可以对模型进行相当广泛的测试，但我们想尽快推出它，然后你会看到很多人测试它，得到一些有趣的结果。

I think that there's also a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly, and then you see a bunch of other people test it and you get interesting results.

GLM 曾经短暂出现过一个问题，就是有些模式崩溃，在某些提示和某些温度下，它会一遍又一遍地输出同一个 token。

There was an issue with GLM briefly where we had some, like, mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures.

一旦你把一个接口暴露给真实世界，就会有更多各种各样的输入，这样你就能发现并修复问题。

Like, once you expose an endpoint to to the real world, there's going to be, you know, so many more varieties of of things given to it that that you're able to, you know, discover and and patch things.

所以这不仅仅是一个上线当天的事情。

So it's not just a, you know, day zero process.

然后，比如第一周、第一个月，如果模型仍然受欢迎，你既要修复 bug，又要继续推动性能提升，这怎么平衡？

It's then, like, for the first week, for the first month, if a model remains popular, like, how do you both fix bugs and then continue to push the envelope on performance?

你是什么意思，你

What do you mean you

不想让你的模型输出 SSSS 吗？

don't want your model outputting SSSS?

是

Is

顺便问一下，那个东西有循环检测吗？

there loop detection on that stuff, by the way?

这种情况还是经常发生，挺让人意外的。

It still happens, like, quite a lot, which is surprising.

我们在我们的接口里做了处理，如果模型连续输出完全相同的标记四次以上，我们就直接截断生成，然后说，抱歉，这个应该重试一下。

We have, like, in our in our endpoint, like, if a model was to output the same exact token, like, four plus times, we just cut the generation and we say like, oh, sorry, this should like try again.

或者重新处理一下请求。

Or like you reprocess the request.

因为我们知道，如果连续四次都是同一个标记，那很可能是模型崩溃了。

Because we know then like if it like if it It's four times the same token, it's probably collapsed.

对。

Yeah.

有没有办法选择退出，万一我真的想要那种输出呢？

Is there a way to opt out in case I really actually want that?

没有。

No.

其实我觉得我们是有办法处理这个的。

Actually think I there's a way that we have to handle it.

不过我不太确定。

I'm not exactly certain.

我觉得在某些模型里，比如它们输出表格之类的东西时，会想画12条横线，再画12条横线。

I feel like in certain models, like when they output something like, you can imagine a table for instance, and so they want they want to draw like 12 dashes and 12 dashes.

对，我觉得这种情况是有办法处理的。

Yeah, I think there's a way for that to happen.

我觉得我们只对某些特定的标记做这个限制。

I think we only do it on certain tokens.

比如我们会排除一些特殊字符。

Like we exclude certain special characters.

对。

Yeah.

所以我们只在某些特定的情况下这么做，比如最常见的，GLM-5.2。

So we only do it on like certain like like s is the most common almost, GLM-5.2.

哦。

Oh.

我觉得DSV4也是这样的吧？

And I think it was d s v four as well?

就像，你会遇到循环问题，比如它只会输出类似这样的东西。

Like, you'd just have like looping issues where like it would be just something like

有没有

Is there

关于s有什么特别的吗？

special something special about s?

没有。

No.

只是随机的。

Just so It that's random

看起来就是它在说话。

seems to be the one talking.

对。

Yeah.

而且它只在温度为零的时候，甚至

And it's and it's only temperature zero or even

在其他温度下呢？

at other temperatures?

0.9或者别的什么，它还是会崩溃。

Point nine or whatever, it will still it will still collapse.

那真奇怪。

That's weird.

说实话，这其实是个推理问题。

It's it's an inference it's an inference problem to be honest.

就像软件问题一样，比如你用的镜像，NVIDIA有时候会发布一个镜像。

Like a software problem, like oftentimes the image you're on will like NVIDIA will release an image for instance.

如果我们把他们最新的STLM镜像中的改动合并到我们的技术栈里，就会发现这个问题被修复了。

And if we will upstream the changes from their latest STLM image into our stack, we'll find that it it fixes it.

或者这种情况经常只出现在你使用的推理引擎里，比如SGLAN。

Or oftentimes this will only happen in an inference engine that you're using like SGLAN.

但如果你换成VLM，就不会这样。

But if you were to switch to VLM, that isn't the case.

所以这看起来更像是一个极其不确定的软性问题，而不是模型本身的问题。

So it seems to be like an extremely like nondeterministic kind of soft ratio and not really a model issue.

不是权重的问题。

It's not like a weights problem.

比如可以说，这是量化的问题。

Like, be able to say like, it's problem with the quant.

我们做PTQ的时候搞错了。

We did PTQ wrong.

对吧？

Right?

但这说不通，因为同样的权重，用不同的推理引擎，问题就不会重复出现。

But that isn't that doesn't make sense because the same exact weights used with a different inference engine does not repeat the problem.

有时候是后端用的内核有很细微的竞态条件，如果你把模型部署在一个集群上，可能永远不会遇到这个问题。

And sometimes it's the kernels that are being used in the back end have like these very subtle, sometimes race conditions where if you were to use this model hosted on one cluster, you will never get this problem.

天哪。

Oh my god.

但如果你部署在另一个集群上，就会遇到。

But if you host it on a different cluster, you will.

原因是一个集群里节点间传输KV缓存用的是更慢的互联，而另一个集群的节点间互联更快，所以前者暴露了竞态，后者没有。

And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster, so that exposes the race, whereas in another cluster it doesn't.

所以最后你就只能想，好吧，这个模型不能部署在这个集群上。

So then you end up just like, okay, this model is not gonna be hosted on this cluster.

我们会把它部署在另一个集群上，因为那个集群暴露了这个问题。

We're gonna host it on, you know, another cluster because that cluster exposed that problem.

但最后又会想，那到底是软件的问题吗？

But then it ends up with, okay, is it the software?

是模型权重的问题吗？

Is it the model weights?

还是硬件的问题？

Or is it the hardware?

有个问题，即使温度设为零，结果也不是确定性的。

There is a thing about this with temperature zero still not being deterministic.

对吧？

Right?

对。

Right.

比如说因为硬件的原因。

Let's say because of hardware.

即使温度为零，嗯。

Even at temperature zero Mhmm.

同一个模型，你也不总是能得到相同的输出。

Same model, you won't always get the same output.

对。

Mhmm.

不过我对那个竞态条件的问题感到惊讶，因为我以为PyTorch是一个图，能保证至少按正确的顺序执行。

Even but I'm surprised by the race condition one because I thought PyTorch was a graph that, like, guarantees that you at least execute things in the right order.

嗯，那就别这么干。

Well, don't do it.

我不是说这不好，比如有PDI优化，可以在前一个内核结束之前启动下一个内核。

I guess guess I'm not saying that this is well, have things like PDI optimizations where you can start a kernel before the end of the previous kernel.

这样做是有原因的，因为确实需要。

And that's because you want to do that because Exactly.

但执行得不够干净。

But you don't do it cleanly.

比如，你会让执行过程有一点重叠。

Like, you overlap a little bit of the execution.

不。

No.

我觉得很有可能内核本身，就是那个应该在这个时间点运行的块，那个内核本身就有竞态条件。

I guess it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition.

比如，缺少一个屏障。

For instance, like a missing barrier.

通常在设计内核时，如果你想让它非常快，如果没有进行充分的测试，某些线程可能会在数据被其他线程写入之前就从寄存器中读取数据，因为你的屏障或同步机制有问题。

Like often if you're designing a kernel and you want it to be very fast, If you don't test it extensively, you'll have certain threads access data points from registers before they've been written to by other threads, for Because like your barrier is wrong or your synchronization is wrong.

但确实，测试本身在很多情况下都非常困难。

But yeah, like the testing itself is very, very difficult in most like

而且也没有像借用检查器那样的东西。

And there's no like borrow checker for

那是什么意思？

What does that mean?

就像Rust。

Like Rust.

就像如果你想要内存安全，这听起来是一个类似的问题。

Like that if you're trying to have like memory safety, it sounds like a comparable problem.

是的，没错。

Well, yes.

但你现在用的是CUDA，对吧？

But you're working in CUDA, right?

NVIDIA的GPU就像……

NVIDIA GPU is like

你只需要一个更高级的语言。

you just need a higher level language.

就像Modular那样。

It's like modular.

也许这就是Modular想做的事情。

Maybe that's what modular is supposed to do.

我不太清楚。

I don't know.

你觉得怎么保持模型的质量？

How do you see keeping quality of the model?

你刚才提到了所有这些步骤，比如要做量化、训练自己的投机解码器、在不同硬件上运行。

So you talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder, run on different hardware.

再看看其他模型提供商，你在消费端掀起了一场推理速度竞赛，那在保持质量不变方面，需要做些什么呢？

Looking at other model providers, okay, you kicked off a inference speed race on the consumer end, what goes into keeping quality the same across them?

当然。

Sure.

可以跑基准测试，但你怎么确定量化的标准是多少呢？

Can run benchmarks, but, like, how do you determine how much quantization are the standards?

实际上，这涉及到什么？

What actually goes into

关于质量，有几个方面。

There's a few things on quality.

大多数推理优化都是无损的。

Most inference optimizations are lossless.

比如KV缓存。

KV caching, for example.

你只是在避免重复计算相同的值。

You are just recomputing preventing recomputing the same values.

推测解码当然，如果草稿标记错了，就会被拒绝。

Speculation, of course, if a draft token is wrong, it gets rejected.

主要的损失性优化是量化。

The main lossy optimization is quantization.

这真的归结为，第一，数据格式。

And that really comes down to, number one, data format.

第二，你选择量化模型的哪些部分，哪些层。

Number two, which parts of the model you choose to quantize, which layers.

第三，对量化后的权重做大量校准，确保你保留了所有的异常值。

And number three, like, doing a lot of calibration on the quantized weights to ensure that you're sort of preserving all the outliers.

不过还有其他一些技巧。

There's other sort of tricks that you can do though.

一个重要的就是长上下文。

A big one is long context.

因为你一开始问的那个问题，就是如果我发送一个20万token的请求会怎么样。

Because one thing you asked at right at the beginning is, oh, what's gonna happen if I send a 200,000 token request in?

所以很明显，输入序列很长的话，你需要存储更多的信息，处理更多的token。

So obviously, with a long input sequence, you need to, you know, store a lot more information, you need to process a lot more tokens.

所以即使一个模型有特定长度的上下文，作为推理服务提供商，你可能会选择构建一个上下文更短的API，当然也会有一个完整长度的。

And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length, and of course, a full length one as well.

因为如果用户不需要完整的百万token上下文，比如，你可以给他们提供更好的性能。

Because if someone doesn't need the full million token context, for example, you can get them better performance.

不确定这算不算模型质量的问题。

Don't know if that's exactly like quality of the model.

我对质量的理解是，我们多大程度上忠实地还原了原始模型。

The the way that I think about quality is to what degree are we faithfully serving the original model.

如果你想象一个模型的黄金实现，它完全按照模型设计的方式运行，那么质量就是我们离那个100%保真度有多近。

If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, you know, 100% fidelity of the model.

当然，你也可以从训练的角度来考虑质量，以及如何超越100%。

You can also, of course, think about quality from the training side and how do you push yourself past a 100%.

但当我纯粹考虑推理优化时，就是在保持尽可能接近100%保真度的同时，让速度更快。

But when I think about purely influence optimizations, it's getting faster while staying as close to that 100% fidelity mark as as possible.

当然，我们内部的标准就是，你应该分辨不出我们的API和官方API有什么区别。

And certainly our standard internally is that, like, you should not be able to tell the difference between our API and a, you know, sort of official API.

我觉得Kimi在这方面做得特别好，他们做了很好的供应商基准测试。

I think Kimi in particular does a good job of vendor benchmarking here.

对。

Yes.

他们发布了一个实际的

They released an actual

供应商基准测试。没错。

vendor Exactly.

因为他们指责了一些人。

Because they accused some people.

亚马逊，有个供应商在Kimi的基准测试上表现不太好。

Amazon, there was some provider that was not doing it very well on Kimi's benchmark.

对。

Yeah.

所以对于Reflective，它会反映我们……

So with Reflective, would reflect what we

这是很久以前的事了。

This was a long time ago.

对吧？

Right?

不。

No.

大概三四，

Like like three four,

五个月前。

five months ago.

这事也发生在某个模型上，我不记得是哪个了，但他们撤下了不少，然后还专门做了个图表来说明这件事。

This also happened with I don't remember which model, but they pulled out quite a few and then they started a whole chart about this.

可能是

It might

吉米供应商验证器。

have been Jimmy Vendor Verifier.

对。

Yeah.

因为你会很生气，对吧？

Because you'd be pissed, right?

比如如果你是……对。

Like if you're Yeah.

比如如果我是消费者，我用亚马逊的接口，用Kimi，然后我会觉得，天哪，这太糟糕了。

If like if I'm a consumer and I'm using Amazon's endpoint for instance, and I'm using Kimi and I'm like, oh my god, like this is bad.

我不会说，哦，亚马逊量化模型的方式有问题。

I'm not gonna say, oh, Amazon quantized the model in a bad way.

我会说，哦，Kimi真差劲。

I'm gonna say, oh, Kimi sucks.

对吧？

Right?

所以看起来就是这样。

So it seems like that.

是的。

Yeah.

小心点。

Take care.

小

Take

心。

care.

有道理。

Justifiably.

这可能是个愚蠢的问题，但只是确认一下。

This is probably a stupid question, but just checking.

量化之后有什么改善吗？

Has anything improved from being quantization?

比如，量化是不是总是更差？

Like, is quantization always strictly worse?

嗯，从技术上讲，不是的。

Well, technically, no.

它是有损的。

It's it's a loss.

量化是一种有损的实现方式。

Quantization is is a lossy it's it's a lossy implementation.

速度提升了。

Speed improves.

显然，第八点，

It obviously, number eight,

我一直在寻找逆缩放定律。

I always look for inverse scaling laws.

这是我从Nom Brown那里学到的，就是那些通常朝一个方向起作用的东西，有时候它们会……

This is something I learned from Nom Brown, where, like, things that normally act in one direction, sometimes they

嗯，严格来说，当你跑基准测试的时候，因为这些模型是非确定性的，有时候你的MPFD四量化版本，你知道的，会比这个版本高出两个基点。是的。

Well, technically, when you run a benchmark, because these models are nondeterministic, sometimes your, you know, m p f d four quant is, like, you know, two basis points higher than your this this one's Yeah.

没错。

Exactly.

对。

Yeah.

这就是为什么我总说在误差范围内。

It's it's it's within that's why I always say within margin of error.

我其实已经不再这么说了，因为大家都以为我的意思是，在某个误差范围内，我们勉强在最差的那一边。

And I actually stopped saying that because everyone assumes that what I mean is, well, within some margin of error, we're barely inside of that to the worst.

所以我们会说，是的，有时候它只是给你一个更高的分数，但就像Ali说的，那是噪音。

So we're saying yeah, sometimes it's just like gives you a higher output score, but like like Ali said, that's noise.

据我所知，你并不一定能让结果变得更好。

To my knowledge, you're not necessarily making the results better.

你只是再次尝试，尽量让保真度接近原始模型的百分之百。

You're just trying to, again, like, keep your fidelity as close to a 100% to the original model.

关于你提到的这一点，我们在MP上做过研究。

There is, to your point, research that we did on MP.

不知道你能不能找到我们研究实习生Joshua发的一条推文。

I don't know if you are able to pull a tweet with one of our research interns, Joshua.

我记得那条推文是关于我们量化后的GLM五点二比NVIDIA的好百分之二十。

I think it's a tweet on on how we have 20% better quantized GLM-five point two than NVIDIA.

基本上，我们在这两个月的研究中发现的是，好吧。

Essentially, what we found throughout, like, this two month research is okay.

量化是有损的，就是把数据从16位压缩到4位，比如说。

Quantization is is a lossy it's it's you're compressing the data from, you know, occupying 16 bits to occupying, you know, four bits, for instance.

所以显然会丢失一些信息，你要尽量把损失降到最低。

And so you're obviously losing some information and you're trying to minimize that.

所以当我说要量化模型时，任务就变成了怎么找到可以量化的层，以及哪些层不能量化。

And so when I say that I'm gonna quantize the model, job becomes how do I find the layers that I can quantize and how to find layers to not.

比如说，对于图像模型，我不会量化调制层，也不会量化输出投影层，因为这两个层，输出投影是用户看到的，调制层是模型看到或理解的。

For instance, with image models, I don't quantize modulation layers and I don't quantize out projections because those two are like, out projection is what you see as the user, modulation is what the model sees or understands.

对。

Right.

没错。

Exact.

所以我想，关于他的论文，你有没有……我想它没有那个，是的。

And so I guess to his his paper, do you have the I guess it doesn't have the yeah.

这是一篇很长的论文。

It's a long paper.

我不知道能不能找到。

I don't know if I can find

如果有搜索的部分，或者它可能在讨论串里。

If there's a part to search or it's probably in the thread.

它可能在讨论串里。

It's probably in the thread.

对。

Yeah.

但基本上，长话短说就是，量化更多层很可能会让结果更好。比如我有一个模型，量化了第1、5、10层，另一个模型只量化了第1和第2层，那么量化更多信息的那个模型可能表现更好，因为量化误差相互抵消了。

But basically the long and the short is it is very possible that quantizing more of the model makes the results Like if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and two, it is possible that the model in which I quantize more information is going to perform better because the quantization errors have canceled out.

所以Joshua在他的数学证明中，用了一个验证器，展示了你可以预测哪些层的量化误差会相互抵消，然后选择量化那些层。

And so what Joshua showed in his mathematic proof where he had like a verifier in is that you can predict which layers are going to have quantization errors that will cancel out with each other and you choose to quantize those layers.

这样做的结果是，通过这种数学量化方法，你最终得到的模型比其他提供商的模型量化程度高出20%。

And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider.

所以你能获得20%的额外吞吐量，因为量化的层数更多，比之前运行MDF时更多。

So you get 20% more throughput of it because there's more layers than running an MDF before.

而且质量也比其他量化模型更好，因为你选择的那些层的误差相互抵消了。

And your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out.

比如一层偏右，一层偏左，一层又偏右。

Like one layer is skewed to the right, one layer is skewed to the left, one layer is skewed to the right.

你最终的logit分布更接近模型的原始分布，所以保真度更高。

Your final logit distribution is more similar to the original distribution of the model, so you have better fidelity.

所以我们是通过KL散度来证明这一点的。

And so the way we proved this was with KL diversions.

我们不只是看基准测试的分数，而是比较量化模型和原始全精度模型在逻辑分布上的KL散度。

So instead of just scoring on the benchmarks, we scored the KL diversions between the logic distribution of the quantized model and the logic distribution of the original full precision model.

我们证明了，用这种技术，如果你在逻辑上选择下一个词的概率分布和原始模型更接近，那么你很可能保持对原始模型的忠实度。

And we showed that with this technique, we get if your probability distribution on the logic switch token it wants to select is more of the same as the original model, you're probably gonna end up staying true to the original model.

所以，对。

So, yeah.

所以看起来在这之前，行业里的看法是，量化越多，效果越差，因为引入的损失越多。

So it seems like previously before this, it seemed like the industry was, well, the more you quantize, the worse it's gonna be because the more loss you introduce.

这不一定完全正确。

That's not exactly not necessarily true.

所以，对，它不会改善，但可以相互抵消。

So, yeah, it doesn't improve it, but can cancel out.

我觉得这有点像剪枝，你可以剪掉某些层。

I think it might be this, but reminds me a good bit about pruning actually, where you can prune off certain layers.

对。

Mhmm.

但非常有意思。

But very interesting.

没想到你们还专门发了篇论文。

Didn't know this was a whole paper you guys put on.

这确实是个有趣的事实。

It's a fun fact.

这篇论文原本有72页，后来我们觉得实在没法……我们真的坐不住。

It was originally 72 pages, this paper, and then we decided we can't We couldn't really sit.

所以现在变成了45页。

So it's now 45.

虽然还有39页，但内容非常充实。

Still still 39 pages, but very very substantive.

我们聊了评估和其他各种事情。

We talked about evals and and all these things.

那，在提速方面，能做到什么程度呢？

And, like, what's possible in terms of speed up?

我觉得这可能是大家最关心的问题，而且你在文章里也写了这个。

I guess, like, it's, like, probably, like, the number one thing that people do wanna care about, and it's something that you wrote about in your post.

比如，官方API是每秒70个token，你把它提升到了90。

Like, official API is 70 tokens per second, and you push it up to 90.

这是很常见的情况吗？

Is that, like, a normal thing?

所以，在Influence工作的好处是——

So what's cool about

我觉得Influence会是一个长期适合做工程的地方，原因是，如果你看看那些高度优化的领域，比如金融，在金融行业，你会用基点来衡量你进步了多少。

working in Influence, the reason that I think Influence is going to be a useful place to do engineering for a long time is that if you look at highly optimized domains, like say finance, if you're in finance, you measure how much better you got in basis points.

就像，哦，我进步了5个基点，也就是0.05%。

It's like, oh, I got five basis points better, like, twentieth of 1% better.

这已经是很大的新闻了，因为一切都优化得很好了。

That's huge news because everything is is so optimized.

我们发布优化成果时，提升幅度是20%。

When we publish optimizations, it's it's 20%.

有时是100%。

It's a 100%.

有时是200%。

It's 200%.

所以老实说，可能还有很大的提升空间。

So there's still probably, like, a lot further to go, honestly.

当研究人员开始发表关于某个方面提速1%的成果时，你就会知道这个领域已经基本被研究透了。

Like, you'll you'll know that influence is pretty much solved when researchers start publishing about how they got 1% faster at at something.

顺便说一句，因为我来自金融背景，在七十年代，做量化金融研究时，那就是当时的利润率。

Which by the way, because I am from the finance background, in the seventies, that was the margin at the time when you did quantitative finance research, you would find

那20%呢？

And, like, with 20%?

百分之零点几？

Tens of a percent?

是的。

Yes.

对。

Yeah.

现在呢，

And now it's

先从小处说起。

tiny first.

有兴趣的朋友可以去看看安德鲁·洛的论文。

For for those those people interested, look up Andrew Lowe's paper.

他有一个很有意思的展示，讲的是量化统计套利的收益分布，从七十年代那种20%的差距，一路收窄到今天几乎为零，非常有意思。

He had a really interesting illustration of quant statarb distribution, narrowing down from, like, those kinds of 20% differences in the seventies down to nothing today, which is very cool.

没错。

Exactly.

而这正是同类趋势的开端。

And and what the beginning of of the same type of of of thing.

现在做基准测试很难。

Now benchmarking is hard.

我想任何人都会这么告诉你。

I think any anyone will tell you that.

而且，对服务提供商的速度做基准测试也很难，因为涉及太多变量了。

And and benchmarking provider speeds is hard because there's so many variables that go into it.

你用什么硬件？

What hardware are you using?

系统负载有多大？

How much load do you have on the system?

提示词、输入输出序列的具体情况等等，所有这些因素。

What's the exact nature of the prompts and input and output sequence links, all that kind of stuff.

但总的来说，当你把这些改进叠加起来，你看到的是成倍的提升。

But overall, when you start stacking these improvements, you're looking at multiples.

最常见的衡量方式当然是TPS，也就是每秒处理的令牌数，这其实是我们行业里一个不太好的叫法，因为实际上有两种每秒令牌数。

You can look at it the the most common form, of course, is TPS, tokens per second, which is bad naming by us in the industry because there's actually two tokens per second.

每秒的吞吐量token数和延迟的token数是两回事，对吧。

There's tokens per second of the throughput number and and the latency Yeah.

比如，从GPU输出的总每秒token数，这是一个吞吐量指标。

To like, total tokens per second out of the out of the GPU as a throughput number.

大多数人只关心作为延迟指标的每秒token数，这个我们应该叫ITL，也就是token间延迟，但我们没这么叫。

Most people only care about tokens per second as the latency number, which we should call ITL into token latency, but we don't.

总之，你可以想象一个标准的API，没有太多优化，针对一万亿参数的模型，在合理的流量下，大概每秒30到50个token。

Anyway, so you can imagine a sort of standard API without many optimizations for 1,000,000,000,000 parameter model operating somewhere in the 30 to 50 tokens per second range for for a reasonable traffic profile.

我们通常的目标是，你知道的，把它提升10倍。

And we generally see the goal of, you know, pushing to 10 x that.

但，不一定是一开始就能达到，而是通过叠加足够的优化。

But, you know, not necessarily day zero, but by stacking enough optimizations.

如果你有，比如说，四个优化，每个都能让性能翻倍——哦，抱歉，是三个优化，每个都能让性能翻倍，然后你把这些叠加起来。

If you have, say, like, four optimizations, each of which doubles performance oh, sorry, three optimizations, each of which doubles performance, then you you stack that up.

那就是8倍的提升。

That's an eight x gain.

这就是我们在这个领域里努力达到的数量级。

That's kind of the the order of magnitude that we're working with in this in this space.

我们追求的是大幅提升性能，而不只是从70提升到90。

We're trying to make things substantially faster, not just go from, like, 70 to 90.

你是说你们已经做到了吗？

Are you saying you've you you have done that?

假设一个合理的基准是每秒30到40个token。

So let's say you have, as a reasonable baseline, 30 or 40 tokens per second.

你可以实现10倍的提升。

You can you can achieve 10 x that.

比如在GLM 5.2上，如果你想用未量化的模型，甚至是在Hoppers上，而且只是用现成的推理引擎，没有特别的优化，没有投机采样，没有KV路由，也没有分离式推理，那你可能就只能达到每秒30到40个token。

So, like, on GLM 5.2, if you wanna get unquantized, perhaps on hoppers even, and you're just using an off the shelf inference engine with no particular optimizations, no no speculator, nothing nothing extra around, like, KV routing, no disaggregation, you you're probably, yeah, looking at that, like, 30 to 40.

你觉得这个基准合理吗？

Do you think that's, like, a reasonable baseline?

对。

Right.

对。

Right.

要达到十倍的速度，需要做出很多取舍。

To get to something like 10 x, there's a lot of trade offs that you're making.

如果我们运行在每秒300到400个token的范围，显然你用的是最好的硬件。

If we're running at sort of more like a 300, 400 tokens per second range, obviously, you are using the best hardware possible.

你有一个优化过的投机解码器。

You have a optimized speculator.

你完成了所有的量化工作。

You have done all of your quantization work.

你的缓存命中率很高。

You are seeing a pretty high cache hit rate.

你用的是较小的批次大小，并且并行配置是针对延迟和吞吐量调优的，但这是可行的。

You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible.

所以，如果你去 artificial analysis 或者 OpenRouter 上看，从最差的提供商到最好的提供商，往往能达到那个范围。

So these spreads that you see if you, like, go on artificial analysis or you go on OpenRouter and you look at, you know, the the worst provider to the best provider oftentimes can can hit that kind of range.

十倍当然是非常激进的。

10 x is, of course, very aggressive.

通常可能只是四到六倍的提升，但真正让我们兴奋的是能获得这些巨大的收益，而不只是从每秒70个token提高到90个。

It's oftentimes maybe more of a four to six times improvement, but that's the the kind of performance that makes us really excited is when we can get these huge gains, not just go from 70 to 90 tokens.

这也跟硬件有关。

It's also like hardware dependent.

比如，如果你只是在一个H100节点上提供服务，然后你把模型分片到四个B200节点上。

Like, if you if you obviously have a thing where you're serving it on just like a node of h one hundreds and then you throw like, you you shard the model across like four nodes of b two hundreds.

你肯定可以通过增加硬件来提升速度，但如果硬件相同、GPU数量相同，那就另说了。

Like like, you can definitely increase the speed with just throwing more hardware at it, like normalizing for the same exact hardware and the same number of GPUs.

对。

Yeah.

那就要看推理优化了，通常能有两到四倍的提升。

Then you're looking at like a two to four x improvement depending on the influence optimizations.

所以，是的，一部分取决于怎么调用，一部分取决于谁在驱动。

So, yeah, it's some of it's, you know, what's the call, and some of it's who's the driver.

如果把两到四倍的提升拆开来看，比如说在B200单节点上跑GLM-5-2这个例子。

If you break down the two to four x, say say the example is run GLM-five-two on b two hundreds, single node.

对吧？

Right?

为了榨出最后那点性能，投入的精力跟成本相比，到底值不值？大家应该怎么看待这个问题？

What's like the cost trade off for effort to get like the last bit of juice out versus what should people just think of.

对。

Right?

光谱量化。

Spectre quantization.

光谱量化。

Spectre quantization.

那大概能占到95%的效果。

That's that's like 95%.

那这能带来多大的提升？

And how far does that get you?

那对普通人来说，做这件事有多容易呢？

And how how easy is that for the average person to do?

比如说现在，我想把GLM-5.2的权重放到一个B200的节点上。

So say right now, I wanna throw the weights of GLM-five two on a node of B-two hundreds.

找到投机解码模型或者已经量化好的模型，有多容易？

How easy is it to find speculative decoder decoder model or already quantized model?

需要做多少工作？

How much work goes into it?

如果你从头开始做，那工作量相当大。

If you're doing it upfront, it's quite a lot of work.

如果现在做的话，已经有人发布过相关的东西，你可以直接拿一些NVFP4的权重。

If you're doing it today, there's going to be people who have published things that you can just you can just grab some NVFP four weights.

你可以直接拿一个投机模型。

You can grab a speculator.

对。

Yeah.

如果我们想想，我们叠加的两个两倍是什么，从BF16到NVFP4，其实不完全是两倍。

If we're thinking about, like, what are the two x's we're stacking, going from b f 16 to NVFP four is it's not quite a two x.

对吧？

Right?

我觉得大概是从16位到8位能省30%到40%，然后从8位到4位再省30%到40%。

It's like, I think it's about, like, thirty thirty to 40% from 16 to eight and then another 30 to 40% multiplied from eight to four.

所以加起来不完全是两倍，但差不多是两倍。

So that doesn't quite get you a two x, but, like, roughly a two x.

投机解码也差不多是两倍。

Speculate a roughly a two x.

如果硬件够多，流量也够大，再加上去，又是差不多两倍。

Disag on top of that if you're able to get enough hardware and and put enough traffic through it, another roughly a two x.

然后再加上一些，你知道的，两位数百分比的提升，因为用了最新的内核之类的，运行效率更好。

And then you add in some, you know, double digit percent increase from having just a a better run time with, you know, the the latest kernels and stuff behind it.

大概就是这么叠加起来的。

And that that's kind of how it stacks up.

对。

Yeah.

所以，构建这些每一个，比如构建量化权重，对于真正懂行的人来说，是几小时到几天的工作，构建投机模型，同样也是几小时到几天，而分布式设置，也是几小时到几天。

So building each of those, like, building the quantized weights is for someone who really knows what they're doing hours to days of work, building the speculator, again, like hours to days of work, and the disag setup, hours to days.

好吧。

Well, okay.

一旦你有了它，一旦设置好了。

Once once you have it Once it's set up.

第一次让分布式工作跑起来，当然，我说的是，确实非常困难。

Once it's Getting disag working for the first time, I'm saying, of course, is is very difficult.

但是，嗯。

But Mhmm.

边际实现是，如果你……

The the the marginal implementation is If you're

只是拿，比如，如果你是一个普通消费者，手头有比如两百张B200，你想知道怎么自己托管，你不需要自己量化模型，对吧？

just grabbing like, if you are a person, just a normal consumer who has access to, like, a note of b two hundreds and you're wondering how can I just host it myself, you don't need to quantize the model yourself?

总会有开源的量化检查点，如果没人做的话，NVIDIA也会推一个出来。

There's always gonna be, like, an open source quantized checkpoint, and NVIDIA's gonna push one out if no one else does.

通常，服务提供商会自己训练好的投机模型。

You usually, the providers will have their own spectacle that they've trained as well.

你不需要自己训练投机模型，直接用他们的就行。

You don't need to train your own SPECT deck, you can just use that as well.

对。

Yeah.

比如Kimi，GLM-5.2就有自己的MTP。

Like like Kimi, GLM-five two has its own MTP.

对。

Right.

对。

Right.

很多银行。

Many banks.

多词预测。

Multi token prediction.

对。

Yes.

我就直接用这个，万一我说错了，你懂的。

I'm just gonna use it you in case I get it wrong, you know?

其实，如果我没记错的话，他们的多词预测可以用于自推测解码。

Actually, you should correct if I'm wrong, their multi token prediction can be used for self speculative decoding.

这个我还真不确定。

I'm actually not sure.

我不太

I'm not

我大概有把握，但可以让人再确认一下。

exactly I'm semi confident in that, but someone can check.

不过，你知道，这有助于讲清楚一个场景：不只是普通人，比如一家公司想从无服务器推理切换到，我想把这个部署上去，我想租一些GPU，直接跑起来。

But you know, it's useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to, I wanna this up on, I wanna rent some GPUs, throw it up.

这些步骤能让你比直接放在Veo后面快得多。

These are the steps you take to do significantly faster than just put it behind Veo.

我是说

I mean

对。

Right.

我一直在等有人提到Dynamo。

I was waiting for a mention of Dynamo.

我觉得那应该是你用来对比的基准。

I feel like that's supposed to be the baseline that you measure against.

我觉得Dynamo与其说是一个开箱即用的系统，不如说是一个用来构建的工具包。

I would think of Dynamo as less of a sort of out of box system and more of a toolkit for building with.

所以当我们讨论KV感知路由、KV卸载、PD分离这些话题时，Dynamo基本上——顺便说一下，Dynamo是NVIDIA的一个开源库。

So when we talk about doing KV aware routing, when we talk about doing KV out offloading, when we talk about doing PD disaggregation, Dynamo fundamentally is by the way, Dynamo is an an open source library from NVIDIA.

我们和Kiely已经完成了那部分。

We we've done that part with Kiely.

Kiely。

Kiely.

不错。

Cool.

所以，这样你的听众就知道了，它支持所有不同的推理框架，而且实际上它支持多种硬件，这挺有意思的。

So so then your your listeners know then that it supports all the different inference frameworks, and it actually is kind of multi hardware, which is interesting.

但是

But

它只是一个路由器。

it's just a router.

它并不是一个优化层。

It's not like an optimizer layer.

对。

Yeah.

它所做的就是，Dynamo擅长的是，它是一个库，用于在你的集群和硬件之间移动信息。

All it does like, what Dynamo is good at, it is a library for moving information around your cluster, around your hardware.

所以如果你有KV缓存放在一个地方，需要把它移到另一个地方，Dynamo会帮你协调Nixle来完成这个移动。

So if you have, you know, KV cache on one place and you need it to be somewhere else, Dynamo coordinates Nixle for you to to move that around.

但这并不意味着你直接pip install Dynamo，就能立刻获得巨大的性能提升。

That doesn't mean that, like, out of the box, you just say, you know, PIP install Dynamo, and then you get, like, a massive performance speed up.

它更像是一个开发者工具包。

It's it's more of a developer toolkit.

对。

Yeah.

我觉得它自带一套默认配置，你可以替换掉。

I would have said it would it it comes with a set of defaults that you can then swap out.

确实。

It does.

如果整个行业都标准化地部署这些，那它应该算是一个可信的基准。

If the industry at large, I think, was, like, rolling out all of these deployments standard, then I think it would be, like, a a credible baseline.

但我们有我们自己的基准，对比的是实际运行中的情况。

But we've got a we've got a benchmark against, like, what we're seeing in the wild.

我还想再多聊一点关于PD分离的内容，因为那可能排在量化之后，位列第三，仅次于量化和投机解码。

I did wanna talk a little bit more about PD dis disag because that's probably, like, number three after quantized and speculative decoding.

在你的书里，我本来想直接翻到那本书的。

In your book, though I was just gonna pull out the book.

对。

Yeah.

比如，5.2.2讲的是美杜莎，5.2.3讲的是鹰。对。

Like, five two two on Medusa, five two three on Eagle Yeah.

5.2.4讲的是NNI。

Five two four on n n I

我想5.5应该就是讲分离了。

guess it's five five would be would be disaggregation.

嗯，不是。

Well, no.

我只是想多谈谈另一个话题。

I just wanted to dwell a little bit on the other.

那你选择收录哪些内容呢？

Like, so what do you choose to include?

哪些内容你又选择不收录？

What do you choose to not include?

因为我觉得还有其他很多技术吧。

Because there was all these other techniques, I guess.

对。

Yeah.

这些技术现在还适用吗？

Are these still relevant?

因为我觉得它们大概是差不多一年半前出来的。

Because I think they came out, like, a year and a half ago maybe.

Medusa已经挺老的了。

Medusa is quite old.

是的。

Yeah.

美杜莎确实很老了。

Medusa is old.

但书里有没有把它当作一个基准，就是标准模型的基线？

But is it in the book as a good, here's the baseline for vanilla.

就像，

Like,

你应该知道这个。

you should know this.

比如，去读那篇论文。

Like, read the paper.

我就说，对。

I'm like, yeah.

非常有道理。

Makes so much sense.

是的。

Yeah.

所以写这本书的时候，我有几个目标。

So with with the book, I had a couple goals.

一个是给大家提供整个领域的基础词汇，另一个是让大家对每种技术的工作原理有一些直观的理解。

One was to give people just a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works.

正如我在AI工程师演讲中提到的，那算是这本书的第一个公开补充，推测这个领域的发展速度比其他所有领域都要快得多。

As I mentioned in in my AI engineer talk, which is kind of the first public addendum to this, the speculation space has moved much faster than everything else.

所以，是的，即使在我写这本书的时候，梅杜莎，我很大程度上是把它作为一种方式，让大家理解这个领域是如何演变的，而不是什么是最现代的技术。

So, yeah, even at the time that I wrote the book Medusa, I very much included as a way for people to understand how the how the space evolved rather than what the most modern technique is.

现在当然有了deflash、despark。

And now, of course, there's deflash, despark.

还有比Eagle更新的技术，虽然Eagle仍然很常用。

There's there's newer techniques even than than Eagle, although Eagle is still very commonly used.

Spec specta？

Spec specta?

是的。

Yes.

投机解码。

Speculative decoding.

什么？

What?

能

Can

这是基于一篇论文的，基本上就是在做投机解码。

It's you it's based it's a paper by and it's like it's basically doing speculative decoding.

嗯哼。

Uh-huh.

对于投机解码来说。

For the speculative decoding.

天哪。

Oh my god.

这其实就是另一种方式，对，这是最重要的解释方式。

It's it's literally just another it's like, yeah, it's the most important way to explain it.

而且看起来他在那里确实获得了不小的加速，但训练的复杂度，至少在我们看来，几乎和训练Ganzle一样复杂。

And it seems like he got nontrivial speed ups there, but it seems that the complexity with training it's almost like like in our mind at least, it's almost as complex as training Ganzle.

这就像是一个非常微妙的平衡，而且很多时候你只是稍微调整一下，但这实际上就是投机解码套投机解码。

It's like a very delicate balance, and oftentimes you it's just a little bit, but it's literally speculative decoding on speculative decoding.

投机。

Speculative.

对。

Yeah.

我们看过这篇论文。

We we saw this paper.

挺有意思的。

It's interesting.

对吧？

Right?

我甚至没想到现在训练它会特别有针对性。

I wouldn't even expect it to be very particular to train now.

对。

Right.

对。

Right.

我天真地想，好吧，训练一个投机解码器。

The naive part of me is like, okay, train speculative decoder.

有道理。

It makes sense.

投机解码的核心思想就像是，嗯，有点像iPhone的自动更正版，但用在普通模型上。

Like, the whole idea of speculative decoding is you you it's like almost like the iPhone auto protected version but for a normal model.

对吧？

Right?

就像你生成三个token，然后说，好，对它们做预填充。

Like, you're just you're just, you know, generating three tokens and you're like, okay, do prefill on them.

这样你就为原始模型省下了那三个步骤。

And so you save those three turns for your original model.

现在你的投机解码器要做三轮自回归了。

Now your speculative decoder is doing three turns of autoregression.

那为什么不干脆用一个更小的模型呢？

So why not just have an even smaller model?

我觉得吧

I guess

另一个问题是，投机解码器的规模是多大？

the the other question there is, what are the size of speculators?

比如说对于GLM——对。

So say for GLM- Right.

大概十亿到十亿参数左右。

It's like a billion to a billion foot Amadeus.

比如对于MiniMax，是的。

Like for min max it's it's yeah.

对。

Yeah.

就像一层。

It's like one layer.

通常是原模型的十六分之一。

It's like one sixteenth of the original model, usually.

对。

Yeah.

其实我觉得我们回办公室后应该写篇论文。

Actually, I think we should do a paper when we get back to the office.

投机，投机，投机。

Speculative, speculative, speculative.

不。

No.

这看起来像是，什么时候停下来？

It does seem like like how when do you stop?

但这也像是，比如说，如果你能训练投机解码，对吧，如果你能有一个小模型，它能准确预测中间投机者会预测什么，也能预测原始目标模型会预测什么，那为什么不直接用那个最小的模型呢？

But then it also seems like kind of like if you're able to train spec spec decode, for instance, right, like if you're able to have a small model that is accurately predicts what the intermediate speculator is gonna predict, that is able to predict what the original target model is gonna predict, then why not just use that smallest model directly?

对吧？

Right?

这跟路由问题很接近。

This is adjacent to the routing problem.

对。

Right.

是的。

Yeah.

对。

Right.

关于投机解码，实际使用中的一个限制是，你必须在运行大模型的同一台硬件上运行一个小模型。

The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you're running the big model on.

这本身就存在一个编排和资源竞争的问题，这也是投机解码普遍面临的一个约束：生成草稿token需要消耗资源，管理起来也增加软件复杂度。

There is a orchestration and resource competition problem inherent in that, and that is one of the sort of constraints on speculation in general is that draft tokens cost resources to create and cost software complexity to manage.

所以如果你有无限递归的投机解码器，你不仅在训练过程中，在推理引擎的实际实现中也会增加不少这样的复杂度。

And so if you have sort of like infinitely recursive speculators, you add in quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process.

我正想说，不知道能不能做类似的蒸馏和剪枝，你知道的，这其实是一回事。

I was gonna say, would wonder if you could do similar, like, distillation and pruning of you know, it's the same thing.

它就是个模型。

It's just a model.

我们能不能直接蒸馏很多权重，量化投机模型，但这超出我的专业范围了。

Can we not just distill a lot of the weights, quantize the speculator, but out of my domain?

我猜还有个问题，这些都是针对大型服务器工作负载的。

I guess the the question that also comes up is this is all for big server workloads.

对吧？

Right?

这些有多少适用于，比如说，这台MacBook？

How much of this applies to, say, have this MacBook.

我想高效地运行Gemma。

I wanna run Gemma really efficiently.

是不是类似的问题？

Similar problems?

不一样吗？

Not the same?

差别挺大的。

Pretty different.

几周前我在他的播客上聊过这个话题。

I talked to about this on on his podcast a couple weeks ago.

数据中心和生产负载的推理工程与本地AI的推理工程之间的区别在于，我们一开始就有根本不同的约束条件和目标。

The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI is that we start with fundamentally, like, different constraints and different goals.

对于本地AI，问题是：如何把这个模型塞进我的硬件里，然后让它不那么笨？

With local AI, it's how do I fit this model onto my hardware and then make it less dumb?

而对于数据中心推理，问题是：如何加载这个模型，然后让它不那么慢？

And with data center influence, it's how do I load this model and then make it less slow?

显然，我们关心的是别太笨，他们关心的是别太慢。

And obviously, you know, we care about less dumb and they care about less slow.

但我觉得，本地AI的推理工程生态系统，其实有很多地方是数据中心领域可以借鉴的。

But the local AI influence engineering ecosystem, I think, actually has a lot for us to learn from in the data center space.

他们是各种量化技术的专家，包括动态量化，这些我们在剪枝、蒸馏和层移除方面基本上都不碰。

They are experts in various forms of quantization, including dynamic quantization that we just kind of don't touch in the pruning and the distillation and the, you know, layer removal.

他们的移除影响不大。

There's Their removal matters less.

对。

Yeah.

不。

No.

那些剪枝，确实。

Those pruning, really.

对。

Yeah.

但是，但是，但是有一个令人惊讶的方式，

But but but there's surprising way,

但那只是为了

but that's to just to

把东西塞进笔记本电脑里。

fit something on the laptop.

对。

Right.

对。

Right.

是的，我是说，这确实是个有趣的领域。

Yeah, I mean, it's it's a it's an interesting it's an interesting space.

倒不是说他们的技术对我们数据中心来说一定适用，因为我们显然有不同的资源和目标，但更重要的是，那个领域的过程和开放性确实值得敬佩。

Not not necessarily that, like, their techniques make sense for us to do in the data center because obviously we have different resources and different goals, but more that the the process as well as the openness of that field is is something to, you know, admire.

是的。

Yeah.

就像你说的，有些优化技术，比如Turbo Quantum，你肯定听说过，它在网上引起了很大的轰动。

Like, to your point, like, certain optimizations that would like, for instance, turbo quantum sure you've heard, like, it made such huge hype on that.

然后我们在Twitter上深入讨论了一番，大家都在问，这到底是什么？

And we we did, a whole deep dive on Twitter and they said, what is it?

它是怎么工作的？

How does it work?

它到底好不好？

Why is it good or not?

然后它就火了，而且被用在了本地设备上，因为比如在MacBook上，内存带宽太慢了。

And it took off, and it was implemented on local devices because your memory bandwidth is so slow on like a MacBook for instance.

但要是把同样的东西放到NVIDIA的B200 GPU上试试。

But try putting the same thing on like an NVIDIA GPU on a B200.

TurboQuant就不会被用到了。

Turbo quant would not be like it would not be used.

就像NVIDIA明确表示的那样，这不是一个好的优化。我们亲眼见过，在TurboCon内核里做反量化和量化的开销，实际上比省下的带宽时间要慢得多。

Like NVIDIA made it clear that this is not a good optimization, we've seen it firsthand where the overhead of doing de quantization, quantization of, you know, in the kernel itself, the TurboCon kernel, each into and is actually much, much slower than the time that you save from doing the bandwidth.

因为在B200上，带宽有每秒3.5TB。

Because on the B200s, you have like 3.5 terabytes per second.

你根本不需要把存储压缩得那么厉害。

You don't need to decrease the storage that much.