Baseten 首席执行官 Tuhin Srivastava 谈 AI 推理瓶颈、自定义模型与构建推理云

大家好，各位听众。

Hi, listeners.

今天我和埃拉德请来了Baseten的创始人兼CEO图欣·斯里瓦斯塔瓦。

Today, Elad and I are here with Tuhin Srivastava, the founder and CEO of Baseten, inference cloud.

咱们今天要聊聊AI算力的瓶颈问题，为什么推理是最后一个被攻克的市场，工作负载正在怎么变，开源和可能的多芯片未来，还有如果一年内规模扩大30倍会是什么样子。

We're here to talk about capacity constraints for AI compute, why inference is the last market, how the workload is changing, the open source and perhaps multichip future, and what 30x scale in a year looks like.

图欣，欢迎再次做客。

Tuhin, welcome back.

嗨。

Hi.

见到你真好。

Good to see you.

谢谢你们邀请我。

Thanks for having me.

好的。

All right.

你们正处在最疯狂的市场之一——AI推理。

You are in one of the craziest markets, AI inference.

这事儿特别关键。

It's very important.

现在各种事儿都在发生。

There's a lot going on.

过去一年，你们的规模扩大了30倍。

You guys have grown 30x over the last year.

我觉得可以说，今年你们的营收要突破十亿美元了。

And I think I can say you're expecting to do more than a billion dollars in revenue this year.

对。

Mhmm.

到底发生了什么？

What's going on?

跟我们说说你们的规模扩张吧。

Tell us about scale.

是的。

Yeah.

不是。

No.

这事儿简直太疯狂了。

It's been it's been nuts.

说实话，过去两年里，事情一直在不断升级，大家越来越意识到，AI 能用在任何地方。

I I think what's happened over the last, honestly, at twenty four months, but just kinda keeps getting bigger and bigger, is that I think everyone is realizing that you can put AI everywhere.

现在从闭源到开源的模型，选择多得不得了。

You have all these great options available from closed source to open source models.

开源模型的基线能力已经突破了关键门槛。

The open source models have crossed some sort of chasm in terms of their baseline capability.

而且，像 RRL 这类微调技术和专有模型的后训练方法，现在也早就普及了，成功的案例多到数不清。

And then I think RRL techniques and post training for specialized models has become mainstream enough, you know, there's enough examples of it work of it working.

客户们也慢慢明白，他们完全可以自己掌控推理环节，越来越自主。

The customer's realizing they can, you know, kind of own their inference more and more.

对我们来说，这意味着长尾模型真的实现了，客户自己承载了大部分智能。

And what that's meant for us is more, you know, the long tail of models coming true, customers in housing a lot of that intelligence themselves.

随着应用层变得越来越大，我们只是顺势而为，一直在那儿收集需求。

As the application layer just gets bigger and bigger and bigger and that's growing, we are just someone index on that and we've been around to be able to collect the demand.

这里有个根本性的问题，我觉得每个人都在不断问：独立的应用层到底能不能存在，还是会被大实验室吞并？

There's an existential question in here that I think everybody is continually asking of does the independent application layer get to exist at all versus the labs?

你得相信这一点。

You have to believe this.

你为什么这么相信？

Why do you believe it?

是的。

Yeah.

听我说。

Look.

如果独立应用层真的不存在了，那会挺让人难过的，我觉得这确实是我的感受，不过嘛，难过就难过吧。

I I I think it'd be it'd be a sad thing if it didn't exist in general, and I think that's, like, my but, you know, sadness is fine.

这个

The

我整天都挺难过的。

I'm sad all the time.

哦，是啊。

Oh, yeah.

难过没什么大不了的。

Sadness is fine.

但我认为应用层能存在，原因可不止这个。

But, like, that that that's not the reason why I think the application layer will exist.

我觉得应用层能存在，有好几个原因。

I think the application layer will exist for a number of reasons.

一个是，我觉得公司真正有价值的东西，是那些只有它们自己才能收集到的用户行为数据。

One is because, you know, I think this idea that what is valuable to a company is, you know, the user signal that they can gather that only they can gather.

如果这些数据被塞进模型里，它们的生意就危险了；但如果这些数据体现在工作流程里，它们才能真正建立起自己的优势。

And to the extent that that is encoded in a model, I think a lot of their business will be at risk, but to the the extent that it is encoded in workflows, that is where they will be able to develop mode.

举个例子，像Abridge这样的公司，医生会修改录音生成的病历，之后怎么用这些病历才是关键。

So a good I think a good example of that is, say, a company like Abridge where the clinicians edits off the notes and what they do with those notes after the fact.

而在电子病历系统里往下走三步，那些操作就变成了只有他们自己才有的工作流程。

And the thing that happens in inside the EMR three steps down, and that becomes a workflow that only

你能说说Abridge是做什么的吗？

Can you explain what Abridge does?

不好意思。

Sorry.

Abridge是个智能录音助手，美国几乎所有的医院都在用，医生靠它自动生成病历。

Abridge Abridge is a ambient scribe that is used by physicians in, like, you know, almost all hospitals in The US.

我觉得Elad是他们的投资人。

I think Elad's an investor.

Shiv太厉害了。

Great Shiv's amazing.

真是家了不起的公司。

Great great company.

团队太棒了。

Great team.

产品也绝了。

Great product.

而且他们已经深度融入了医院和医生的工作流程里，简直无缝衔接。

And, you know, they they've basically, you know, got this very, very deep integration into into hospitals and to clinician workflows.

我觉得关键在于，前沿模型公司根本没法抢走这块市场，因为他们根本接触不到这些用户数据。

And my argument would be here is that actually, you know, it's very, very hard for a frontier model company to be able to eat up where that could they just don't have access to that user signal.

随着时间推移，能拿到这些用户数据的人，就能用这些反馈信号来训练模型，慢慢做出能看长远、有持续性的系统。

And what will happen over time is the folks who have access to that user signal can start to post train models on that reward signal and and start to get long horizonogenic models running that.

如果这种数据真的独特、稀有、难以获取，那肯定会出现一个专门的应用层。

And I think to the extent that that is possible and that signal is differentiated and unique and is somewhat rare to to get access to, there will be an application layer.

就像客服公司也是个例子——客服任务根本不是一次性的。

And I think, you know, it's like support company is another example of that where, you know, a support a support task isn't one shotted.

像Baseten这样的公司，客户一提工单，往往要走一连串步骤，少则十来个，多则二十个，而正是这些流程，让专门的模型有了用武之地。

Usually, at a company like Baseten, when a ticket comes in, there's like, what, like, one, two, ten, twenty actions that get taken, and that is where, you know, someone can develop a specialized model.

所以这其实有两种不同的情况。

So there's almost two versions of this then.

一种是像Abridge、Decagon这些新公司，你刚才提到的，它们用AI做新型应用，然后卖给出客户。

There's new companies like Abridge or Decagon or some of these other things that you mentioned that are doing these new types of applications that are using AI and they sell it to customers.

另一种是大企业自己搞研发，开发内部模型。

The other is enterprises building things in house or building their own models.

你觉得现在市场上，这两种新应用公司和企业自研AI的比例大概是多少？

What proportion of the market today do you think is these new application companies versus enterprises adopting AI?

你觉得几年后会变成什么样？

And how do you think that looks in a couple years?

是的。

Yeah.

我觉得这问题你两年前也问过我，对吧？哦，对，想起来了。

I I think that's a that's a you we did I think you asked me the same question two years ago Oh, yeah.

在互联网这个领域。

On on the file of the Internet.

我得重复一下。

I have to be repetitive.

这简直太疯狂了。

It it is crazy

太不一致了。

At inconsistent.

答案就是太疯狂了——我觉得，如果你从间接数据来看，99%都还没用上。

The answer is just that it's crazy that the answer is still, I think I think if you look by inference count, it'd be 99% the fall.

对。

Yeah.

这也说明了这里的机遇有多大——大部分市场还没上线，还没把AI加进去。

And that is that kind of represents the scope of the opportunity here is that the majority of the market hasn't come online and and added AI into

系统里。

the system.

企业采用AI已经远远走在前面了，我觉得这正是AI最令人兴奋的地方之一。

Enterprise adoption is well ahead of us, and I think that's one of the very exciting things about AI.

是的

Yeah.

还有很多潜力没释放出来，我觉得大家都低估了这一点。

There's just so much still to come, and people are underestimating that, I think.

完全没错。

100%.

但有意思的是，我们正亲眼看着这场转变发生。

And I but but what's cool is that we're seeing the transition happen.

对吧？

Right?

以前呢，大家还都在问，

Before, it was like, hey.

他们有没有在用AI工具？

Are they are they using AI tools?

对啊。

Mhmm.

两年前这事儿可没那么明显，但现在谁都看得出来，他们确实用了。

I don't think that was immediately obvious two years ago, and think it's obvious now that, yes, they are.

他们在用闭源模型的API吗？

Are they using closed source model APIs?

我觉得他们正慢慢朝这个方向走。

I think they're starting to get there.

一旦你开始用上这些，真正看到能做什么之后，定制模型的普及就跟着来了。

I think once you do that and then you kind of see what is possible, then comes the whole custom model adoption.

我觉得，我们未来要面对的，就是这些了。

I think that is all that is ahead of us today.

所以，如果你现在的客户主力，像你刚才说的，是那些应用型公司、AI原生企业，那些增长快的，现在大多都已经很有规模了，比如Abridge、Cursor这些。

So if the majority of your customer base today is, as you described, the the former, like, application companies, AI natives, the fast growing mean, of them are at considerable scale now, like the Abridge, Cursor Yeah.

公开的证据。

Open Evidence.

这个世界的公开证据。

Open Evidences of the world.

他们教你什么？

What do they teach you?

这会迫使公司怎么做？

What does that push the company to do?

你觉得该怎么服务他们，同时又为大企业做升级？

How do you think about serving them versus evolving for the enterprise?

对。

Yeah.

我觉得首先呢，和那些规模最大、最前沿的公司一起做项目，真的能学到很多东西。

I I think, firstly, like, you just learn a lot by building with the company's greatest scale, doing the most interesting things.

我们从两个角度来看这个问题。

We we think of it two ways.

最明显的一点就是，先瞄准最高规模的需求——那些最能推动你技术突破的客户，其他方面自然就跟着水到渠成了。

Like, I think there's, like, the the the most obvious way, which is just build for the highest scale, you know, most the the customers that will push you the most from technologically and everything kinda will fall into play.

你看Stripe的发展历程就说明了这一点，现在Stripe服务的客户里，大企业已经占了很大一部分。

Think the Stripe evolution as a company showed that was like Stripe now, like, serves like so many enterprises.

但十二年前，情况可不是这样。

But twelve years ago, that wasn't the case.

他们当时就直接冲着前沿去，跟着客户一起走。

But they just built for the frontier and kind of went with them.

我觉得第二种思路是，专门为那些服务企业的公司做产品。

I think the second way we think about that this is to just think about building for companies that are serving enterprises.

没错，我们自己不直接服务企业，但我们的客户都在服务企业。

So, yes, we don't serve the enterprise, but our customers serve enterprises.

Abridge、OpenEvidence、Decagon、Ryta、Gamma、Clay，这些公司全都在大规模服务企业。

Abridge serves enterprise, open evidence, Decagon, all these Ryta, Gamma, all these Clay, all all these companies serve enterprises en masse.

所以我们实际上得到的是他们需求的‘翻译版’，比如说，他们会说：嘿。

And what we actually get is, like, a translation of the requirements from them, which is, you know, they're like, hey.

我们需要这样的数据留存机制。

We need this sort of data retention.

我们需要这些网页模型能被部署出来。

We need this type these web models need to be deployed.

这就是他们能接受的GPU类型和延迟范围。

This is the types of GPUs or the latencies they're okay with.

从透明度角度来说，他们关心的模型要求是这些。

This is the model requirements from, like, a transparency perspective that they care about.

所以我觉得更细致的答案是：只要你认真听他们说什么，就能完整地了解到企业到底需要什么。

And so I think that is actually the more nuanced answer, is that if you listen to what their needs are, we actually get a full translation of what the enterprise will require.

举个例子，我们服务像Abridge和Open Evidence这样的公司，其实就已经很贴近医疗系统的需求了，毕竟他们就是在给医院做产品。

Like, I would say that by serving companies like Abridge and Open Evidence, we're probably pretty well suited to go serve the health care system given that they are selling and latent health given that they are selling to them.

你们有没有注意到，现在大家用的开源模型类型有什么变化？

How much of a shift are you seeing in terms of the types of open source models that are being used?

我觉得这几年确实有演变——两三年前主要是Mistral这类模型，后来Meta推出了Llama，再后来，性能不太行的模型，或者来自中国的模型，开始在不同方面冒头。

And so I think we've seen an evolution where two, three years ago, I think the main thing was of Mistral and then a few other things, and then Meta kind of came along with LAMA, and then it kind of really shifted in terms of the misperformant models or of Chinese origin in different ways.

你们客户用的模型，也有这种混合趋势吗？

Do you see that sort of mix reflected in terms of what's being used by your customers?

是的。

Yeah.

我觉得我们服务的客户，尤其是那些全球增长最快的AI公司，都非常前瞻。

I think customers, at least the customers we are serving, are very and these are, like, the fastest growing AI companies in the world that are very forward thinking.

他们想用最好的模型，而且确实在优化——目前还只是很小一部分任务，大家才会优先考虑成本。

They they wanna use the best model, and they they are optimizing I think there is there there are a there's a subset of tasks, which I think is small today, where people really start to start with cost.

对。

Mhmm.

但所有人都是先看能力，因为真正的经济价值和增长点就在这里，之后才去优化。

But everyone comes from capability first because that's really where the economic growth is being unlocked, where the value is being delivered, and then they optimize.

这么说吧，从GPT-4o到月之暗面、深求、Canopy，再到像Orpheus这样特别出色的语音合成模型，客户们普遍都想要最前沿的那一个。

And I think that's, like, actually been, you know and so with that in mind, you name everything from GPTOSS all the way to Moonshot model to DeepSeeks to Canopy or Orpheus, which is, like, really good text to speech models, customers generally wanna use whatever's at the frontier.

而且我觉得，我们现在对怎么跑好这些模型，看得比以前清楚多了。

And and I think the the difference has just been I think we have a lot more visibility into how to run these and how to run these really well.

其次，现在这些模型真的够好了。

And secondly, that they're good now.

有人对使用中国模型提出过不少担忧，尤其是安全问题——比如模型里是不是藏了后门、木马之类的东西？

There have been a number of different concerns raised about the use of Chinese models, in particular security, or is there something embedded in the models or Trojan horses or other things?

A，你认为这真有什么值得担心的吗？

A, do you think there's any real concern there?

B，人们经常说，美国应该推出一些对冲力量来应对这种情况。

And B, people often talk about how there should be US counterweights to this.

从地缘政治的角度看，你觉得这种担忧有道理吗？还是说我们真该为此紧张？

From a geopolitical perspective, do think that's something that's legitimate or something we should be worried about?

或者你怎么看这些模型的源头和它们的实际用途之间的关系？

Or how do you think about the Yeah.

嗯，说到底……

Sort of origins of these models versus their uses?

是的。

Yeah.

听我说。

Look.

我觉得这些模型，首先，真的非常出色。

I I I think these these models, firstly, are fantastic.

这些模型太厉害了。

They're amazing.

我们跟这些团队合作。

We work with these teams.

真的超棒。

They're truly awesome.

我说啊，听好了。

I'd say, look.

说实话，我很难想象，也可能我想错了——但如果你把这些模型绑在某个网络里，它们不可能凭空跨越网络边界，跑到数据中心去。

I I don't it it is hard for me it is hard for me to see and I I could be wrong, but, like, you know, if if I if I network bound these models that they're not magically, gonna be able to cross those network boundaries and to data center.

而且说实话，我从来没真见过什么证据，除了最早那几款模型，大家很快就注意到，好像里面真带点倾向性或者偏见。

And, you know, I I don't and we I've never seen any real evidence except from some very early models that I think people picked up on very quickly that there is some agenda or bias built into these.

我觉得在某种程度上，美国自己开发模型确实很重要。

I do think that to some extent is I I think there is importance to The US that we develop our own models.

要是中国有五家实验室都在搞开源模型，而我们连一个都搞不定，那损失就太大了。

I think that that would be a massive loss if that there are five companies, you know, five different labs in China that are creating open source models, and we're struggling to get one set up.

所以这是必要的。

So it's necessary.

我觉得这也是不可避免的。

I also think it's inevitable.

你知道吗，就像一年前深海时刻那样，我记得有人跟我说过，这话讲得真好——世界变化太大了，但他们说，嘿。

You know, like the sea the deep seak moment a year ago, I remember someone saying to me, I thought it was very well said, which is like, and the world's changed a lot, but they said, hey.

咱们干脆就别管它了。

You know, we should kinda just forget Mhmm.

这只是一个中国模型。

That this is a Chinese model.

我们就当它是来自Meta的，然后照着这个思路去搞。

We should just act like this came from Mhmm.

从Meta那儿来的，就这么干，心里有数就行。

From meta and and build and build with that in mind.

对。

Mhmm.

你这是只见树木不见森林啊。

It's like, know, I I think you're kinda missing the forest from the trees.

说白了，就两种情况。

Like, there's two there's two scenarios.

对吧？

Right?

要么美国永远搞不出像样的开源模型，嗯，

Either America does not ever come up with good open source models Mhmm.

那说明背后肯定有根本性问题；要么他们迟早能搞出来，咱们得提前做好准备。

And there's probably a fundamental problem there, or we will get there, and we need to be ready for that world.

是的。

Yeah.

这话有道理。

That makes sense.

挺有意思的是，我觉得美国在这儿搞强盛的开源生态特别重要。

It's interesting because, you like you, I think it's very important for The US to have a strong open source footprint here.

至少目前来看，中国政府似乎在补贴这些模型中的很大一部分，而这种补贴或盈余，实际上转嫁给了使用这些模型的美国企业。

At least for now, it looks like effectively the Chinese government is subsidizing at least a large subset of these models, and that subsidy or surplus is effectively just being passed on to US enterprises who are adopting these models.

换句话说，这相当于中国政府以一种间接方式补贴美国企业，而这一点现在似乎被忽略了。

In other words, it's a way for the Chinese government to effectively subsidize US enterprise in an indirect manner, and I think that's a little bit lost right now.

但把这一点和其他一些被提出的担忧放在一起权衡，总是挺有意思的。

But it's always interesting to weigh that against some of the other concerns that are raised.

谢谢你的这些见解。

I appreciate your comments on this.

而且我觉得，大家担心的其实是，如果我们没法跟上，万一真像你说的那样——你想想这里的经济逻辑，大多数人都觉得DeepSeek是个非常厉害的模型。

Well, and I think the concern also just there just becomes it's like, what happened if we aren't able to like, if it is fun like, I I think if you think of the economics here, which is deep deep sea by most, deep seq's a very good model.

对。

Mhmm.

你懂我的意思吧？

You know?

你可以说它是不是最前沿的，但咱们回头看看三个月前，它就已经在那里了。

And, like, you can argue whether it's at the absolute frontier or not, but, like, let's let's go back three months and it's there.

想想三个月前，我们当时在做那么多事情。

And so think about every and we're doing a whole lot of things three months ago.

是的。

Yeah.

对。

Yeah.

所以咱们就从这个角度想想吧。

And so let's just think about it that well.

你知道吗，如果能用DeepSeq，生产环境里的成本可能只有OpenAntrropic模型的百分之二十，延迟还更短，稳定性也可能更好。

You know, if it you could run deepseq, probably 20% of the cost of running open andropic models in production with comparable better latency, probably better reliability.

如果我们没法用上这种技术的话

If we don't have access to that intelligence

对。

Mhmm.

那损失可就太大了。

In that form, I think it's just a massive loss.

作为一个国家，我们没法跟上创新的步伐，因为智能的成本在下降，而智能的掌控权却越来越集中，这其实意味着更多的智能集中在少数人手里。

And as a country, we won't be able to innovate as fast because the cost of intelligence going down in control of intelligence, what we have seen just means more intelligence.

是的。

Yeah.

智能正越来越多地嵌入到各种地方。

Intelligence being embedded in more places.

对。

Yeah.

这里有个重要的点我们之前没明说：目前最前沿、最先进的模型，其实还是那些闭源的，比如Anthropic、OpenAI、谷歌这些公司的。

An important note here that we didn't mention explicitly is that the state of the art models, the ones that are most far ahead on the frontier, are actually still the closed source, anthropic, OpenAI, Google, etcetera.

没错。

Yeah.

一直以来——其实你能不能简单说说，现在Baseten上处理的工作负载大概是啥情况？

Has been- Actually, maybe you can just characterize like workload a little bit.

比如，现在服务的token里，有多少是来自自定义模型的，有多少是纯粹的开源模型？

Like how of tokens being served on Baseten, like how many of them are from custom models of some kind versus like vanilla open source today?

全是定制的。

It is all custom.

差不多就是这么回事。

It's basically yeah.

所以，95%以上？

So, like, 95% plus?

百分之九十五左右。

9095%.

说实话，我觉得这太棒了。

Like and I think that's really cool, to be honest.

咱们有两项业务。

I mean, look, we have we have two businesses.

我们有三项业务。

We have we have three business we have we have three businesses.

对。

Yeah.

我们要不要

Should we

帮你数一数？

help you count?

不用了。

No.

真不用。

No.

所以我们有专门的推理服务，其实就是定制模型的推理。

So we have, like, dedicated dedicated inference, which is basically custom model inference.

你的服务等级协议就是你的服务等级协议。

Your SLA is your SLA.

然后是共享推理，用同一个端点，共享服务等级协议；再就是训练业务。

Then we have shared inference with a shared inference endpoint, shared SLAs, and then we have a training business.

我觉得现在95%的token都用在第一个业务上。

I'd say 95% of the tokens today are on the the first business.

而且几乎所有的案例里，客户都会说‘是的’。

And almost all of them, there's probably a yeah.

几乎每个客户都会用自己的数据对模型做些调整，专门适配他们的使用场景。

For almost all of them, the customer is making some modifications to the model with with their own data specialized for the use case.

我觉得更重要的是，他们可能还会用不同的方式来编译模型。

And I think what's even more important is they might be compiling it in different ways.

根本没人直接用原始的开源权重跑模型。

No one is just running the vanilla open source weights.

你可能是为了提升质量做定制，但也可能是为了优化性能。

Like, you you might be customizing it for quality, but you also might be customizing it for performance.

几个月前，你们收购了一个研究团队。

You made an acquisition of a research team a few months ago.

你们之前提到过训练后的定制化。

You've mentioned post training customization.

当初收购这个团队的考虑是什么？

What was the rationale behind the acquisition?

现在这个团队主要在做什么？

What is that team doing today?

是的。

Yeah.

我们收购他们的原因呢，说白了，我们本来就是搞基础设施和产品的。

So the the rationale around the acquisition was, you know, we we are infrastructure and product people.

我们是产品团队，现在也成了顶尖的基础设施专家。

We are product people and now are really good infrastructure people.

但我们自己其实没什么研究能力。

And the and we didn't have much of a research capability ourselves.

我们看到市场正飞速转向，靠后训练的资源就能大大加速这个趋势——不管是做成产品，还是单纯提供资源给市场。

And and what we saw was the market moving heavily and heavily, like, that we could accelerate the market itself with post training resources, either productized or onto even just as resources for that market.

Parzed 是 Baseten 的一个客户。

So Parzed was a company that was a Baseten customer.

他们做的是模型后训练，然后在 Baseten 上跑起来。

So they were post training models and running them on on on base Baseten.

我觉得他们也意识到，迟早得转型成做推理服务的公司。

And I think what they realized was that they would eventually need to become an inference company.

而我们当时就想，诶，这不正好吗？

And what we realized was, like, hey.

我们真的很需要这种能力，因为它能让我们更早地接触客户，更好地支持他们。

We we really needed that expertise because represents a way for us to get closer to the customer earlier and be able to support them all.

把这两拨人凑在一起，简直再合适不过了。

And it just made sense pairing them together.

就像我开头说的，随着越来越多模型进入后训练阶段，我们发现，无论是做后训练的软件流程，还是需要后训练专家，需求都特别大，我们也在大力投入这方面的资源。

And just as I said in the opening statement here, which is, you know, as more and more post training models have come up, we've realized that the demand for people to either for software loops to do post training or for post training expertise is very high, and we're really, really investing in that.

另外，团队里还有不少澳大利亚人。

There are also a bunch of Australians.

我觉得咱们在这块儿还挺有先发优势的。

You know, I like to think that we had a bit of alpha there.

不过说真的，这事儿确实特别棒。

But, yeah, that's been fantastic.

他们跟各种各样的客户都在合作。

They're working with all sorts of customers.

而且当你开始深入做研究时，我们会发现，以前我们主要关注性能方面，对后训练这块关注得比较少。

And it's also very interesting when you start you know, we were doing a lot of research on the performance side and less so on the post training side.

但当我们越来越多地投入后训练研究后，就会发现推理和后训练其实紧密相连。

It's interesting as we've started to do a lot more research on the post training side, you start to see how linked inference and post training are.

比如你想想量化这件事——什么时候做量化，模型训练的方式怎么影响你做推理时的量化，这些问题简直密不可分。

And, like, you know, even even when you think about stuff like quantization and when you should do that and, like, you know, how how training how how you train the model affects how you need to quantize for inference and how paired these problems are Mhmm.

这一点现在变得特别明显了。

Has become, like, very apparent.

越来越多的情况是，后训练和推理其实是同一个问题的两面。

And more and more, rely on the post training inference are kind of both sides of the same problem.

因为推理会反过来催生更多后训练——推理产生数据，你做评估，然后就能用这些评估结果来指导后训练，优化奖励函数，慢慢就形成一个完整的闭环。

So because inference will ideally will beget more post training, where inference creates data, you do eval, you can now post train post train on the on that reward function that you that you found with those eval and and hopefully just set up an entire look.

最近几个月，来自蚂蚁和OpenAI的不少人，比如Sam、Greg，都说推理非常关键，推理人才、推理能力都具有战略意义。

Plenty of folks from Ant and OpenAI, Sam, Greg, etcetera, have said in recent months that, like, inference is super strategic, inference talent is strategic, capacity is strategic.

所以不管是推理还是训练后优化，要攒出这些能力都特别难。

So between that and post training, these are very difficult to gather, like, capabilities.

没错。

Yep.

我猜很多客户找你们咨询，都是想问怎么一步步过渡到用自定义模型，是吧？

I imagine that lots of your customers go to you guys for advice on like how to do this progression of moving to custom models.

那一般你会怎么跟他们讲这个发展周期，还有什么时候该在这方面投入呢？

Like, what do you tell people about the life cycle and when they should invest in that?

对

Yeah.

我觉得吧，话说回来

I I think it's, hey.

先去用顶级的模型验证一下，你手上确实有值得优化的东西

Go find go prove to yourself with the best in class model that you have something worth optimizing.

而且我觉得很多时候，要是有客户来找我们，你还记得两年前那个梗吗？

And and I think, you know, a lot of you know, if a customer comes to us, you you there was was that meme which was like it was like two years ago.

感觉就像没GPU，还没找到产品市场契合点。

It feels like no GPUs, pre product market fit.

我的意思是，没经过微调、还没找到产品市场契合点，就是这样。

It's like no post training pre product market fit is what I know.

是的。

Yeah.

对。

Yeah.

没错。

Yeah.

这正是我想说的。

It is what I'd say.

所以你这边合作的客户，

So people that you're working

都是从一开始就冲着大规模去的。

with here are very very at scale first.

对。

Yeah.

他们有一套明确的用户反馈信号，知道怎么优化，也证明了自己能真正为客户提供价值，而且在这块儿有独特的优势。

They they they have a user signal that they know how to optimize, and they've shown that they can, you know, they can serve customer value and that value and that they have something special around that value.

一旦你有了这种价值，那就没问题了。

And once you have that value, it's like, okay.

接下来怎么做得更好、更快、更省钱？

Now how can I do that better, faster, and cheaper?

核心想法是，比如——

With the idea being that, hey.

如果你特别需要做好客户服务，那可能真没必要在编程上多厉害，用个专门的模型反而更合适，还能做得更好、更快、更便宜。

If you need to be very good at customer support, you can you maybe don't need to be that good at coding and that a specialized model might be a better fit for that problem, and you can do it better, faster, cheaper.

那算力这块儿呢？

What about the capacity side?

你一开始说的是要把所有云和新云的算力统一起来。

You started with unifying capacity across all the clouds and neo clouds.

当所有人都在谈论供应短缺，而且是持续多年的供应短缺时，你怎么看这个问题？

How do you think about this when everybody keeps talking about a supply crunch and a multiyear supply crunch?

我觉得，现在关于供应短缺的传言实在太多了。

I think, you know, there's so much narrative around the supply crunch.

不管我们听多少遍，人们还是没意识到情况有多严重。

And no matter like, as much as we hear about it, I don't think people realize how bad it really is.

说实话，现在可用的计算资源简直少得可怜。

Like, there is, you know, there is very, very little slack compute available.

我们自己就运行着挺大的集群，而且利用率高得让人心里发毛。

You know, we we run pretty large clusters ourselves, and we run them at, like, uncomfortably high utilization.

我的意思是，我们大部分时间的利用率都在百分之九十五左右。

You know, we when I'm saying we're, like, mid nineties utilization most of the time.

我们现在已经接入了十八个不同的云平台。

There is we have made we we have we sit in 18 different clouds now.

全球范围内，我们在十八个云平台上部署了九十多个集群。

We have 90 clusters around the world across 18 different clouds.

一开始，我们开发这项技术，就是为了打造一个能覆盖所有这些云平台的统一运行时架构，把可靠性、延迟、故障切换这些事儿都藏在背后，不让客户操心——毕竟这些对关键任务场景太重要了。

And, like, you know, initially, we started we, like, built this technology to be able to, like, kinda create one runtime fabric that spans all these different clouds and try to abstract that away from our customers as a way to think about reliability, latency, failover, all these things that we think are gonna be very important for very mission critical use cases.

同样的技术，只要我们能搞到算力，不管在哪都能拉起来，这帮了我们大忙，真的解决了供应问题。

That same technology, just our ability to get compute wherever humanly possible has been really, really helpful in our ability to get supply.

我的意思是，只要我们接触到一个新的云服务商，哪怕在别的国家，也能在半天内把整个 Baseten 推理系统搭起来。

And and what what I mean by that is we can be introduced to a new provider in a different country and have it up and running with the whole Baseten inference stack.

作为架构的一部分。

As part of the fabric.

对。

Yeah.

甚至不到半天就能接入，成为架构的一部分。

Part fabric in half a day, half maybe less.

这给我们带来了极大的灵活性。

Even for and that gives us enormous flexibility.

就连我们自己，想扩大规模都挺难的。

Even for us, it is hard for us to grow.

我们有个……我觉得是吧。

We have a we have a I think it's yeah.

我来说吧。

I'll say it.

我们公司每天下午四点固定开个会，主要就是讨论：眼下怎么应对需求，怎么调配算力？

We have a a 4PM standing meeting for the company where we basically, like like, how do we, like, how do we how do we manage capacity for the demand right now?

我觉得第二点，很多人其实没搞明白：现在市面上很多供应商其实不太靠谱。

I think the second part which people don't really the two the the second part that people don't really understand is that there are also a lot of suppliers right now that it's kinda grifty.

你知道吧，他们以前根本没运营过数据中心。

You know, like, I I I think, you know, they haven't run they haven't run data centers before.

他们根本不了解服务等级协议，尤其是推理服务的SLA。

You know, they don't understand SLAs for especially for inference.

所以就算有算力空闲，我们也得跑很多倍的冗余，毕竟不能出岔子，现在这样其实挺稳的。

And so, like, you know, even when there is capacity available, there's a lot of like, there's probably we we run a lot more than this, and we have redundancy, and so it's fine.

但说实话，真正靠谱的云服务商大概就那么十来家，我觉得能算得上顶级的也就三四家。

But if you you know, there's probably like a dozen good clouds, and I'd probably put three or four of them in the gold tier.

我觉得这说明我们不仅供不应求，连能运营这些数据中心的供应商都捉襟见肘。

And I think that just means that not only are we supply crunched, we're supplier and operationally crunched onto people who can run these data centers as well.

现在你到底能提前多久预订算力？

How far ahead can you actually buy capacity right now?

换句话说，如果你提前两年甚至五年下单，市场上还有没有余量？

In other words, like, is there any any slack in the market if you buy two years ahead or five years you know,

你的意思是合同周期，还是说……嘿。

what mean, contract length or actually like, hey.

我需要在明年1月28号就用上。

I want this in January 28.

是的。

Yeah.

哪个都行。

Either one.

对。

Yeah.

是的。

Yeah.

我的意思是，我就是想确定明年1月28号能拿到货，至少能有个预期。

I mean, it's more the I want this in January 28, at least I have some visibility into

我的未来供货就有了保障。

my future supply.

对。

Yeah.

你当然能提前买，但别忘了这市场变得有多快。

You could buy that, but you gotta also remember how quickly the market how quickly the market is moving.

而且你也知道，H100这芯片实在太抢手了，多少能平衡一下紧张的局面。

And, like, you know, that gets balanced somewhat off, like, the fact that the h 100 is such a great chip.

对。

Yeah.

然后呢，说实话，这情况简直疯狂。

And, like and then set you know, it's crazy.

要是四年、四年半了，价格还在涨呢。

If it's four years, four and a years old, the price is going up still.

对。

Yeah.

也许它的实际使用寿命能有九年。

Maybe it has a useful life of nine years.

对。

Yeah.

所以你看，这挺好的。

So, you know, that's good.

但与此同时，你确实能这么做，可你这也赌得太狠了吧。

But at the same time at the same time, you know, yes, you can do that, but, you know, you're making a lot of bets Yeah.

这本身就是其中一部分。

As part of that.

而且最近六个月最大的变化就是，大家想要的租期明显变长了。

And then in terms of I think that's the big thing that's changed over the last six months is that the term length that people want has just gone up.

所以如果你想要一个24台、两百台的规模，是吧。

So if you if you wanted a thou a 24 b two hundreds Mhmm.

现在从主流云服务商那儿拿资源，没个三到五年合约根本拿不到，对吧。

Which is, you know, from a good cloud, right now, you're not getting that less than a three to five year contract Mhmm.

现在通常还得预付20%到30%的总合同价值。

Right now with a probably a 20 to 30% t c TCV prepay.

所以真正关键的是，你得有足够的需求来填满这些算力，还得有很低的资本成本——这整个格局其实已经彻底变了。

So, like, actually, what becomes important when acquiring capacity is you need to have enough demand to supply it and to serve, and then you also need, like, a low cost of capital, which is which is actually changing the dynamic pretty significantly.

这会不会影响你们对上市的考虑？

Does that does that impact how you think about going public as a company?

因为从某种程度上说，

Because arguably

是的。

Yeah.

我觉得会更早上市。

I think you'd go sooner.

是的。

Yeah.

没错。

Exactly.

对。

Yeah.

我觉得确实有这个需求。

I I think you need like, I I think the and I think there is demand for that.

但说实话，我们最近才意识到，毕竟我们是做软件的，平时根本不会这么想。

But I think, you know, the pull the you know, and also, you know, one of our one of the one of the realizations that we had recently and we're we're software people.

我们的业务在营运资金方面其实特别有意思。

And so we don't we don't think like this all the time is that, you know, our business has, like, very interesting working capital requirements.

而且正因为这样，我们的融资方式也很特别，确实如此。

Like, we don't and and I and I think, you know, even and that as a result of that, it has very interesting financing Yeah.

对。

Yeah.

需求。

Requirement.

至少目前，我们根本没考虑去借债。

And we're not at least right now, we're not even going down to the debt.

我们其实还可以用其他方式，比如债务或者其他结构。

There's also other things we could do in terms of debt or other structures.

对。

Yeah.

是啊，我最近学到了很多关于债务的知识。

Yeah, I've learned a lot about debt recently.

考虑到现在供应紧张，推理算力是你重点瞄准的几个市场之一，懂这个问题的人不少，竞争自然也少不了。

Given the supply crunch, inference being one of, you know, the top couple markets you've been going after, you have plenty of people who understand this problem and therefore, you know, some competition.

你觉得，到底是什么因素能让某家公司在这里脱颖而出、成为赢家？

How do you think about, like, what are the factors that create a dominant player here or a winning player?

就像你提到的，是资本成本吗？

Is it, as you mentioned, cost of capital?