本集简介
双语字幕
仅展示文本字幕,不包含中文音频;想边听边看,请使用 Bayt 播客 App。
马蒂·斯坦日耶夫斯基于2022年联合创立了ElevenLabs,并将其发展为价值110亿美元的AI音频领域领军企业。
Mati Staniszewski cofounded ElevenLabs in 2022 and has since scaled it to the $11,000,000,000 leader in AI audio.
他因通过逼真的情感语调捕捉语言的人性化特征而广受赞誉,如今公司正将业务扩展至智能工作流、音乐等各个领域。
He's credited with capturing the humanness of speech through realistic emotional inflection, inflection, and they're now expanding into everything from agentic workflows to music.
谢谢
Thanks for
做这件事。
doing this.
谢谢你的邀请。
Thanks for having me.
也许我们可以从这里开始:你能给我讲讲,我知道大语言模型的大致工作原理。
Maybe a place to start is describe to me how like, I know how an LLM works at a high level.
给我讲讲音频模型是如何工作的。
Describe to me how an audio model works.
如果我们以Karpathy的方式,想从零开始构建一个简易版本,它是如何运作的?
Like, if we were Karpathy style looking to, you know, build a toy one from scratch, how does it work?
在早期,你会尝试完全按照人类身体的方式去复制它。
In the early days, you try to replicate it exactly how you would replicate it with with with the human body.
所以你会试图完全重现一种模拟机器,有效地模拟出声道。
So you try to completely try to reproduce a machine, analog machine, that will create a vocal tract effectively.
然后这演变为尝试创建用于语音的数字信号。
Then that progressed into trying to create effectively, like, a digital signals for speech.
贝尔实验室是最早尝试构建一套结构化信号来表示语音的机构之一,这成为我们今天做法的最初前身。
Bell Labs was one of the first to try to, like, create a structured set of signals that will represent the speech, and that is a a first precursor to what we would do today.
接着你会尝试拼接音素,也就是人类说话时各种不同的声音,然后将它们连接起来。
Then you would try to stitch in phonemes, effectively different sounds of of of how we would speak humans, and then try to concatenate them together.
这是方程中另一个重要部分,你会基于下一个词最可能的概率,从你的音素库中提取音素并组合起来。
It's another important part in equation where you would, based on the most probabilistic approach of the next word, you would effectively try to bring the phonemes from your library of phonemes and bring them together.
然后进入现代历史,如今我们实际上在其他领域也采用了类似的神经网络。
And then down to the modern history where now we effectively do similar, like, neural nets in in other domains.
所以你会预测下一个声音,是的。
So you predict the next sound Yes.
当然,基于之前声音的上下文。
Based on on, of course, the context of the previous sounds.
是的。
Yeah.
如果是实时语音的话。
If it's a streaming speech.
如果是音频上下文,你会结合预测音素,同时也会利用上下文文本信息。
If it's, let's say, a context of audio, you will use combination of predicting of the phonemes, but you also use the contextual text element
对。
Yep.
这项工作的成果。
Of of that work.
这里要感谢我的联合创始人皮奥特,他提出了一个新想法:如何创建既可靠、高质量又快速的语音模型,将变换器模型和扩散模型的许多理念引入语音领域。
And here, credit to my cofounder, Piotr, who effectively came with that new idea of of how you can now create voice models, which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models into the speech space.
是的。
Yes.
因此,在音素空间中预测下一个标记
So that prediction of the next token on the phoneme space
是的。
Yes.
这以前是不可能的。
Wasn't something that was possible.
你经常简要提到这一点,比如你如何在文本、波形空间上进行操作。
You might be you you always talk briefly about this of, like, how you kind of operate on the text, on the waveform space.
还有一个梅尔频谱图空间。
There's also a MEL spectrogram space.
所以,通常你会依次处理文本、梅尔频谱图和波形。
So, like, usually, you do text, MEL spectrogram, waveform.
抱歉。
Sorry.
那什么是
What's the
频谱图空间?
spectrogram space?
它是一种视觉化表示,展示语音在音高和能量上的变化,然后将其转换为波形。
It's like a visual representation of how the speech sounds across pitch, across energy, and then you transform that into waveform.
明白了。
Got
它。
it.
所以,当WaveNet和Tacotron模型出现时,它们实际上使用文本到梅尔频谱图,也就是这种视觉表示,是的。
So, like, when WaveNet came came along and and tachotron models, they would effectively use text to MEL spectrogram, so that visual representation Yeah.
然后你如何将它解码和编码为波形,以实现转换。
And then how you decode and encode that into the waveform to to bring it across.
PyTorch找到了方法,能够抽象其中一些步骤,并更好地进行解码和编码,因此预测下一个音素成为关键环节之一。
And Pyotch figured out how to, like, abstract some of those steps and decode and encode them a lot a lot better so that predicting of the next phoneme was one of the big piece.
第二个关键环节是:如何将这种上下文融入到模型中?
And second big piece was how do you bring that context into the equation?
我所说的上下文是指,如果一个配音演员在朗读一段文字,你会知道,哦,这是一个对话片段。
So what I mean by context is if a voice actor was reading a textual copy, you would know that, okay, this is a dialogue sequence.
我需要生成一段对话。
I need to produce a dialogue.
如果这是一个开心的句子,我可能需要用开心的语调来念它。
If it's a happy sentence, I might need to pronounce it as a happy sentence.
嗯。
Mhmm.
但之前和之后发生的事情
But kind of the what happens before and after
是的。
Yes.
都会影响到结果,你需要把这些因素也体现出来。
Comes into the equation, and you need to bring that across.
然后还有一个重要的部分。
And then there's a last big piece.
所以语音模型包含了你如何对给定片段进行语调处理的声音,但第二大部分是语音本身的特征,包括口音、风格和语调。
So voice model has sound of how you intonate the given fragment, but the second big part is the voice itself of the characteristics of accents, of style, prosody across that voice.
因此,当你实际尝试发声、创建语音模型时,你需要将文本转化为音频,这不仅需要文本,还需要语音参考,即你希望它如何被朗读。
So when you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text, you also need the voice reference of how you want it to be spoken.
所以,这是第二个重要的创新。
So here was kind of the second big innovation.
除了上下文之外,还有如何解码和编码这些特征。
So apart from context, it's how you decode and encode those features.
当贝尔实验室提出他们最初的语音表示方式时,其中关键的一点是,他们会为语音硬编码参数。
So when Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hard coded parameters for that speech.
对于ElevenLabs模型来说,硬编码了热情的英式口音参数。
With ElevenLabs models Hard coded parameters for enthusiastic speaker British accents.
没错。
Exactly.
没错。
Exactly.
像你可以选择的一组音高元素、一组能量和频谱图这样的东西。
That kind of stuff, like the set of pitch elements that you can select, set of energy, spectrograms you can select from.
在我们的方法中,实际上你赋予模型一种开放的能力,让它自行决定这些参数应该是什么。
And in our approach, effectively, you would give the the model open ended ability to select what those parameters should be.
所以它不会是英国人、波兰人、西班牙人或英语说话者,而是模型会自行推断出这些特征,其他那些未被硬编码的参数也是如此。
So it's not gonna be British, Polish, Spanish, English speaker, but the model will deduce them themselves, the same for for other set of parameters that are not hardcoded,
无论是
whether it's
热情程度,还是悲伤情绪等等。
the enthusiasm, whether it's the sadness, etcetera.
你说英国口音是一种涌现属性,没错。
You're saying kind of Britishness is an emergent property Exactly.
在你的语音模型中。
In your voice models.
没错。
Exactly.
但是是的。
But yeah.
而这两个主要部分,就是语音生成的编码和解码过程。
And those kind of those two big parts, it's encoding and decoding of how you create the voice.
没错。
Yep.
以前这可是个超级难的问题,还要搞清楚如何在句子中构建语音,如何传递上下文以便预测下一个音素。
Super hard problem before and figured out too, how you then construct that in the sentence, how you get the context across so you can predict the next phonemes.
是的。
Yes.
所以如何在快速完成的同时,以可靠且稳定的方式将它们整合在一起,这些正是语音模型最初的两大创新,至今仍在延续。
So how you bring them together in a reliable and stable way while doing it quick, And these were kind of the two first big innovations in the voice models that continue to today.
但好吧。
But okay.
如果大语言模型通过文本、词、子部分、词元来思考世界,那么语音模型中的等价词元是什么?
So if LLMs reason about text and, you know, words, subparts, tokens, as the way they think about the world, what is the equivalent of a token in the voice model?
你提到了很多次音素。
You mentioned phonemes a bunch.
也就是说,这种表示方式是什么?
Like, what is that representation?
所以我们实际上会为说话者存储语音嵌入。
So we do we we store we store the voice embedding effectively for the speaker.
嗯,是的。
Uh-huh.
所以在生成语音时,你需要这个参考信息。
So you need that reference when you produce and create the speech.
是的。
Yes.
当然,语音模型的输入仍然是文本。
Of course, in the input to the voice model, you still get the text
嗯。
Mhmm.
你把说话者和编码一起带进来。
And you bring the the speaker and and and coding.
当你生成语音时,你是在波形上操作,或者更准确地说,是在该语音的音素层面上操作。
And then when you produce speech, you do operate on the waveform or on the or effectively on the phoneme level of of that of that speech.
当我们反过来做时,当然,当
And then when we kind of go the opposite, so of course, when
对不起。
I'm sorry.
音素是什么?
What is a phoneme?
帮我补充一下我的理解。
Fill in my understanding.
它就像是把音节分解成更小的单元。
It's like a a syllable deconstructed even to smaller elements.
好的。
Okay.
这些本质上就像是人类能够发出的声音。
And these are effectively like the human sounds you can produce.
明白了。
Got
对。
it.
所以这将是最接近这种表示方式的。
So this would be like the most the close to that representation.
但当然,在我们的模型中,现在将不仅仅是基于音素层面进行操作,还会在文本层面进行操作。
But of course, in our models, now it's going to be a combination of not only operating on phoneme level, you also operate on the text level.
你会同时在这两个层面同步操作,因为当你预测上下文时,你需要理解
You operate kind of in both in sync because when you are predicting the context, you need to understand
是的。
Yes.
这句话是如何构建的,尤其是在流式实时应用场景中,比如语音代理场景。
How that sentence will get constructed, and especially if it's more of a streaming real time use case and, like, a voice agent setting.
你需要这两部分协同工作。
You need both parts to to work across.
所以这类似于在技术层面进行微观层面的操作。
So it's similar to how you would operate on a tuck in level on the on the tech side.
我们在音频层面也进行微观层面的操作。
We operate on the tuck in level on the audio side.
我觉得11号的一大亮点是你们的声音听起来更像真人。
It feels like a big part of the magic of 11 was your voices were much more human sounding.
你们是怎么做到的?
How did you accomplish that?
我来给你简要介绍一下我们今天对文本到语音模型的思考方式。
So I'll kind of give you a quick quick synopsis of what how we think about the models on the text to speech side today.
在任何模型中,你都需要架构、计算力和数据。
In any model, you need you need architecture, you need compute, you need data.
架构方面的创新是其中之一。
So architecture innovations were one thing.
数据部分是第二个关键点。
The data part was the second big thing.
在音频方面,你可能会有大量的音频数据,但通常这些数据并没有以正确的方式进行标注。
With audio, you will have you will have a lot of audio data available, but frequently, you will not have annotated in the right way.
你不知道是谁在什么时候说话。
You won't have which speaker is speaking when.
有些内容被标注了,但‘如何’的部分却没有。
Some of the what is annotated, but the how isn't.
就像我们现在说话时,我们使用了哪些情感?
So, like, as we are speaking now, what's the emotions that we use?
我们使用了哪些口音?
What are the accents that we use?
因此,我们内部投入了大量资源,专门组建了自己的数据标注团队,以创建更优质的数据集。
So we would invest a lot internally on effectively creating our own data labelers, our own team to be able to create those datasets that will be better.
这结合了半自动技术和手动技术。
And that was a combination of, of course, like semiautomatic techniques and then manual techniques.
实际上,我们之后开发的许多模型也都源自这些研究。
Actually, a lot of the models that we did afterwards actually spun out from a lot of that research too.
语音转文字模型最初是我们为自己开发的,因为市场上现有的模型根本无法很好地标注这些数据。
Speech to text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data.
然后,我们团队中另一位杰出的研究人员成功构建了这个模型,使其成为我们推向客户的独立产品。
Then another brilliant researcher on our team was being able to construct it so we span it out as a model that we brought to the customers.
所以你们一直在语音领域做着真正有用的事情,这些工作逐渐催生出了一系列你们可能未曾预料到的产品,因为你们在不断构建有用的东西。
So you've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff.
没错。
Exactly.
没错。
Exactly.
这在很大程度上得益于数据方面的优势:我们能够实现自动化处理,组建一支专门接受语音标注培训的团队,因为大多数外部标注人员对音频和语音的理解远远不够,这帮助我们极大地提升了效果。
And that's kind of combination of data of being able to do it automatically, create a team that's coached on voice on how to describe it because most of the labelers out there just aren't as well versed on understanding the audio and voice, helped us a lot to to bring that back.
当然,将这些模型投入生产环境,观察客户如何与之互动,并让他们标注所有数据,帮助我们在时间推移中不断优化这些模型。
And then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time.
顺便提一下,我们之前讨论过语音表征的问题。
A very interesting thing on the side, so we spoke about kind of the speech representation.
最早创建语音表征的人名叫肯佩伦(Von Kempelen)。
The first guy who created the speech representation is a guy called Kempelen, Von Kempelen.
他制造了一台模拟人类声腔的模拟机器,试图发出语音。
So he created those analog machine that would represent effectively a human vocal vocal tract and try to produce that sound.
他为此花费了几十年时间,最终成功产生了元音。
He had to spend decades on that, and that kind of started producing vowels.
但同一个人还制造了第一台广为人知的国际象棋机器,能够模拟下棋。
But that's the same person that created a a chess machine, the first viral, let's say, chess machine that that would kind of simulate playing chess.
这是不是机械土耳其人?
Is this a mechanical Turk?
它就叫土耳其人。
It was called Turk.
是的。
Yeah.
是的。
Yeah.
但正是如此。
But the but exactly.
但背后疯狂的是,它是由一个人操作的。
But the kind of crazy thing behind it, it was operated by a human.
是的。
Yeah.
是的。
Yeah.
这完全是个偶然。
And it was all a fluke.
对。
Yes.
这就是机械土耳其人,实际上我们用它来进行数据标注生产,以实现这项工作。
And that's where the mechanical Turk which actually we use in that kind of data labeling production to to make that that that work there.
是的。
Yeah.
对。
Yeah.
抱歉。
And sorry.
我们直接就切入主题了。
We we kinda jumped right in.
但如果你今天描述Eleven的业务,人们会认为你们是一家语音公司。
But if you describe the Eleven business today, people think of you as the speech company.
那么,他们实际上应该如何理解你们的业务呢?如果你能说明一下主要领域——比如文本转语音、语音转文本、语音助手等——给我们做个梳理。
What how should they actually think of your business to the extent you can describe the big areas Text to speech, speech to text, voice agents, just, like break down the business for us.
好的。
Cool.
简而言之,ElevenLabs是一家研发与产品落地公司。
So in, like, the nutshell I'll describe, ElevenLabs is a research and product deployment company.
我们构建了基础的音频和语音模型,然后打造了一个平台,帮助企业改变与客户和员工沟通的方式,这将通过Yep实现。
We built foundational audio and voice models and then build a platform for businesses to transform how they communicate with their customers, with their employees, and that will apply through Yep.
AI代理,涵盖客户支持、销售、招聘培训,一直到营销和通过我们的创意工具进行故事讲述。
AI agents, from customer support, sales, hiring training, all the way through to marketing and storytelling through our creative tools.
在这一系列中,我们创建了各种基础音频模型,包括用于生成语音的文本转语音模型、支持100多种语言的语音转文本模型,并且在基准测试中表现优异,还有如何将它们串联起来的对话模型,嗯。
And in that set, we've created all types of foundational audio models, so text to speech models for producing speech, speech to text models that work over a 100 languages and happily beat others on benchmarks, all the way through to conversational models of how you loop them together Mhmm.
延伸到音乐和其他音频领域。
To music, to other domains of audio.
当然,除了模型之外,当真正将它们投入生产时,平台的第二层就发挥作用了,即针对客户的特定用例进行适配。
And then, of course, beyond the models, when you actually bring them to production, that's where the second level of the platform comes in, where that meets their businesses on the specific use case.
以代理为例,就是如何将这些模型连接到知识库、电话系统以及你执行任务所需的集成,如何评估和监控代理的行为是否符合预期,如何建立合适的保存率。
So on the agent specific example, it would be how you now connect those models to the knowledge base, to telephony, to to the integrations that you need to perform the actions, how you evaluate and monitor the agent that behaves in the right way, how you build the right save rates.
在创意和营销方面,就是如何制作一条优秀的广告,为某个营销活动创建高质量的视频配音,如何用能良好体现品牌特质的特定声音来朗读文章。
On the on the creative side, on the marketing side, it's how do you create a good ad so you can create a good video voice over for for one of the campaigns, how you create an article that's narrated with a specific voice that represents the brand in a good way.
因此,我们将模型与对客户需求的理解结合起来,构建了一个统一的平台。
So that's where we combine kind of the models and understanding of the customers we work with into into one policy platform.
每个平台公司都会面临一个问题:究竟应该在应用层面走多远。
Every platform company has this question about how far they go into applications.
那么,你们如何思考在哪些方面做横向扩展以赋能整个生态系统,而在哪些方面又该亲自开发应用呢?
So how do you think about where you go horizontal and power the whole ecosystem versus where you develop application?
因为你可以想象,围绕ElevenLabs技术会涌现出一大批字幕工具生态系统。
Because, like, you can imagine there being a whole ecosystem of closed captioning tools that grow up that, again, are built on the ElevenLabs tech.
这并不一定非得是你们自己要去开拓的领域。
Like, it's not necessarily a space that you would have to go after yourself.
我认为,与你提出的问题相比,今天我们把自己定位为一个平台——如果你的企业在构建横向应用场景,这里是个绝佳的选择。
I think the big difference between and your kind of question, today, we we we we see ourselves as a platform where if you're building a horizontal use case in your business, a great place to come.
如果某个领域具有高度的垂直专业性,我认为随着时间推移,很多应用型公司会在此形成,而这些恰恰是我们不会进入的领域。
If you have a lot of domain specificity, that's where I see a lot of kind of the application companies forming over time, where they will, where where that's the specifically not the spaces we will go into.
而且我认为,当技术像这里这样快速演进时,这一点也很有意思。
And I think it also is interesting when the tech is moving as quickly as it is here.
在SaaS领域,你可能会看到很多垂直领域的专业服务商,但我想,你们作为中间平台最大的风险之一,就是被中介化——比如,某个字幕服务还在使用旧版的ElevenLabs,且没有升级,这就会出问题,因为你们希望用户始终使用你们最新、最先进的模型,而你们每周都在部署新功能。
It's one thing, you know, with SaaS where you get these, like, vertical specific providers, but I would imagine one of the biggest risks for you guys in being intermediated is if there is, you know, like in this example, a closed captioning service that is on a two versions old version of ElevenLabs and hasn't upgraded, that's a problem because you want people to be using the latest and greatest model that you've developed, and you'll be kind of deploying new capabilities every week.
我猜这也是你们考虑的一部分,即当技术发展如此迅速时,在很多情况下你们需要直接介入。
And I presume that's part of your thinking, is that just when it's moving that quickly, you need to go direct in a of cases.
没错。
That's right.
在字幕领域,比如现在,我们已经知道我们的服务能够处理客户遇到的99.9%的情况。
In closed captioning, like, you know, here, already now, we know that our services is going to be able to tackle, like, 99.9% of the cases that customers have.
此外,我们还与医疗健康领域的客户合作,为他们创建定制模型,以实现完美的转录效果。
And then there's, like, added benefit of we work with health care customers where we will create custom models for those customers where it will get that transcription perfectly.
在字幕中,上下文才是最难处理的部分,我们之前已经讨论过很多这方面的技术问题。
The context is the tricky thing in closed captions where, like, we talk a lot about a lot of technical stuff on on this.
是的。
Yeah.
当然。
For sure.
这就需要提前准备好一套术语词典,而当我们与企业合作时,我们意识到必须在模型构建过程中融入这些术语。
And that's where you need, like, effectively, like, a dictionary of words that do the tech before beforehand, which which, as we work with the businesses, we know we need to we need to embed in that creation process.
我们正在稍微讨论一下产品方面的问题。
We're talking a bit about kind of products here.
我注意到一点,大型语言模型非常出色,比如CheckGPT、Gemini以及所有主流的大型语言模型,它们的应用数据表明人们大量使用它们。
And one thing I note is that LMs are amazing, and, you know, you have the usage stats of CheckGPT and Gemini and all the popular LMs where they're working and people use them a ton.
在语音领域,似乎存在巨大的产品滞后现象,领先的语音模型已经极其强大。
It feels like there's a big product overhang when it comes to voice, where the leading edge voice models are incredibly capable.
但前几天我开车回家时,需要读一份PDF文件。
And yet, I was driving home the other day, and I needed to read a PDF.
但我正在开车,于是我想,好吧。
But I was driving, and so I said, okay.
我让我的手机给我读一下这份PDF。
I'll just have my phone read the PDF to me.
你可以试着用iOS的屏幕朗读功能来凑合,但它对滚动内容的支持并不好。
You can kinda try and hack it with, like, iOS screen reader, but it doesn't really work with the scrolling.
理论上,你可以上传到Gemini,但你得设法让它不要总结内容,而只是单纯朗读——但我尝试点击‘为我朗读’按钮时,它直接卡住了。
And then in theory, you can upload a Gemini, but you're trying to get it to not summarize it and actually just hung when I tried to press the, like, read this to me button.
所以我根本没法让手机给我读东西,这看起来是个相当基本的功能。
And so there was no way I could get my phone to read me something, which seemed like a fairly basic, you know, feature.
而且,你知道,所有汽车都宣传语音控制,但单独来看,这功能真的很差。
And, you know, all cars advertise voice control, and, yes, it sucks in separately.
如果你想要输入导航信息,目前还没有任何汽车能做好这一点。
You know, if you want to, you know, input something to the navigation, just no car has a good version of that yet.
也许特斯拉能做到。
Maybe Tesla does.
所以为什么在大语言模型和云端代码等方面,我们似乎充分利用了这些智能能力,但在语音方面,却好像还停留在十年前?
And and and and so why does it seem like with LLMs and Cloud Code and everything, we are using all the capabilities of the intelligence, whereas with voice, like we're living ten years ago somehow.
我认为我同意这个前提,即我们落后了十年。
Well, I'm thinking what I agree of the premise that we are ten years behind.
在人们的日常使用体验中,他们仍在使用语音转文字功能,虽然这已经有所改善。
In the lived experience of people day to like, they're using series transcription, which has gotten better.
是的。
Yeah.
而且它仍然远远落后于前沿水平。
And it's still way behind the leading edge.
是的。
Yeah.
比如,确实存在一个问题,我认为在许多情况下,技术本身已经准备好了。
Like, there there is definitely a piece of, like, the the like, we I think the technology in many of those cases is ready.
这跟你所说的存在一个部署差距。
There's a deployment gap to what you are saying.
比如在汽车行业,或者一些大公司并没有足够快地采用或将其投入生产。
It's, like, in automotive or some of the big companies are not adopting that quickly enough or bringing that into the production.
但确实,沿途还有很多不同的问题需要解决。
But, yeah, like, plenty of different problems that you need to fix along the way.
我的意思是,语音模型要真正听起来自然,这仅仅是最近三年才出现的进展。
I mean, the the the quality of voice models for them to actually sound good, like, this this is only, like, last three three years thing.
是的。
Yeah.
这已经三年了。
That's three years.
这确实是三年。
It's it's a three years
多年来,巴黎现在已经有软件更新了。
Paris have over the years software updates now.
所以,这是第一个能够朗读‘是的’的语音模型的三年。
So so that's three years for the first voice model that can narrate Yes.
文本异步。
Text async.
两年前,你开始看到它的实时版本,但还不算真正实现。
Two years ago, you can start seeing the real time version of that, and not really.
我觉得真正的突破是一年前发生的。
Like, it's I think the real break was like a year ago
好的。
Okay.
你可以在生产环境中开始看到这一点。
Where you could start seeing that in production.
然后我认为在2025年,之前无法实现的关键部分是如何将实时语音交互与你所指的某种东西连接起来。
And then I think over 2025, the big piece that hasn't been possible is how you connect now the real time voice interaction with something which I think you are referring to.
它能理解你的目标、你想阅读的内容,以及如何结合你过去的偏好,是的。
It has context of what you want to do, what is the material that you want to read, how does it connect to set off your preferences from the past Yes.
并将这些信息传达出去。
And gets that across.
我认为这仅仅是最近才变得可行,而且我们看到企业界在技术层面开始大规模采用。
I think that's, like, only recently became possible and where we've seen, like, kind of the big adoption across the enterprises leading on the technical side.
是的。
Yes.
我认为今年,这项技术也将在汽车领域或某些应用场景中得到应用。
I think this year, it should be in the automotive side too or the some of the applications side.
好的。
Okay.
所以你觉得今年我们就会在汽车里看到出色的语音模型吗?
So you think we'll start seeing kind of great voice models in cars this year?
今年对于云端使用场景,比如在车里,也就是没有网络连接的情况下,还不行。
This year for the on cloud use cases, like on on on car, in car, so, like, without connectivity, not yet.
2025年会有部署,当然,还存在如何填补这些空白的差距。
20 there's deployment, of course, gap of, like, how you bring that into the gaps.
但我认为未来两到三年内会实现。
But I think, like, the next two years, three years
那PDF阅读的使用场景呢?
How about the PDF reading use case?
这个应该可以实现。
That should work.
是的。
Yeah.
那我本该怎么做才对呢?
Well, how should I have done it?
以前呢,我先讲个故事来说明这个问题,当时我们遇到了一个难题。
So back in the day, we I'll preempt this with a story to cure 11 reader, but we we we had this problem.
有很多有声书作者来到ElevenLabs。
We have so many audiobook authors come into ElevenLabs.
2023年,我们发布了第一款软件。
So 2023, we released first software.
我们有很多创作者。
We had a lot of creators
嗯。
Mhmm.
还有很多有声书作者或图书作者,他们负担不起专业的制作,却想自己制作有声书。
And then a lot of audiobook authors or book authors that tried to couldn't afford professional innovation and wanted to create an audiobook.
然而,没有任何公司接受AI生成的有声书。
However, none of the companies accepted AI audiobooks.
你也不能在Audible这样的平台上销售AI有声书。
And you can't sell an AI audiobook on Audible or something.
没错。
Exactly.
所以Audible会屏蔽AI内容。
So Audible would, like, block AI content.
所以我们别无选择。
So we had no choice.
我们需要为他们开辟一条途径。
Like, we need to create an avenue for them to
确实没有发行渠道。
there was no distribution Exactly.
针对AI有声书。
For AI audiobooks.
没错。
Exactly.
所以我们创建了ElevenReader。
So we created ElevenReader.
好的。
Okay.
而且这还附带了可以上传PDF的功能。
And that kind of came with functionality where you can upload your PDF.
你可以上传文本,并用多种出色的语音朗读出来。
You can upload your text and have it read out loud with a number of incredible voices.
无论是迈克尔·凯恩爵士,一直到与谢尔盖·费曼合作,你在那里
So whether it's sir Michael Caine all the way through to a state and working together with Sergei Feynman where where you
你是在和迈克尔·凯恩爵士这样的人合作吗?
got you were working with the sir Michael Caines of the world?
没错。
Exactly.
好的。
Okay.
然后你还可以在上面直接阅读。
And then you can actually read it on that.
而且这个功能效果非常好。
And that kind of works extremely well.
所以这个是可行的。
So that works.
是的。
Yeah.
那么你该如何实现呢?
Now how can you do it?
其实,我确实
Think Actually, I do
希望所有内容都由迈克尔·凯恩读给我听。
want everything read to me by Michael Cain.
这是一个很棒的嗓音。
It's a it's a great voice.
对。
Yeah.
你们难道不应该做个消费者应用,让我可以直接使用常见的语音功能吗?
Shouldn't you guys have a consumer app where I can just do the common voice things?
比如,我希望我的手机上有个11应用,当我上传一个PDF文件时,它能自动完成我想做的常见操作,比如读给我听。
Like, I want to be able to have an 11 app on my phone, and then if I upload a PDF to it, it can do the common things that I would like, such as have it read it to me.
是的。
Yeah.
那简直就是个阅读器。
That's that's exactly a reader.
所以这样是可行的。
So that works.
好的。
Okay.
手机厂商允许第三方键盘。
The phone makers allow third party keyboards.
你觉得他们会允许第三方语音转文字引擎吗?
Do you think they do they allow third party transcription engines?
你觉得他们会吗?
Will they, do you think?
你刚才说的是手机制造商吧。
The phone makers, you said.
对吧?
Right?
我的意思是,比如苹果公司
I mean, like Apple
还有谷歌。
and and Google.
是的。
Yeah.
是的。
Yeah.
他们是操作系统制造商。
They So they're OS makers.
是的。
Yeah.
不是所有厂商都这样。
Not not not all of them.
使用安卓系统,你可以通过它来实现。
Android and with Android, you can work through it.
比如,你知道的,这中间还有各种变体。
There's, like, you know, variations of that.
什么都没有。
Nothing.
科技公司和其他厂商。
Tech and others.
但确实,我觉得如果你有一个流行的Eleven应用支持语音转文字,很多人会使用它,也许最终苹果会说:哦,如果用户想要,我们应该允许第三方语音转文字引擎。
But, yeah, I feel like if you had a popular Eleven app that allowed for transcription, people would use it a bunch, and maybe eventually Apple would say, oh, we should allow third party transcription engines if that's what people want.
我的意思是,看起来他们似乎正朝这个方向发展。
I mean, there's it seems like they might be going in that direction.
对吧?
Right?
而且最近他们宣布将开放大语言模型生态系统。
And I recently they announced that we'll open up the LLM ecosystem.
希望他们也能对语音生态系统做同样的事,这其实有点类似。
Hopefully, they will do the same with voice ecosystem, which is kind of similar.
但再说一遍,当技术发展如此迅速时,这样做是合理的。
But again, it's I think rational to do when it's moving so quickly.
是的。
Yeah.
语音助手这一交互范式是计算领域最古老的用户界面范式之一,比如《2001太空漫游》里‘打开舱门,HAL’那样的场景。
The voice assistant paradigm is one of the oldest paradigm you know, UI paradigms in computing, like the open the pod bay doors, Hal, from Yeah.
19年,是的。
'19 Yeah.
69年。
'69.
是的。
Yeah.
我会说它目前还不能正常工作。
I will claim it's not working yet.
所以Siri没有足够的智能。
So Siri doesn't have the intelligence.
至于Gemini和ChatGPT这些应用,我是想用语音模式,但我不确定你怎么样。
And then on Gemini and ChatGPT and those apps, I mean, I want to use the voice mode, but I don't know about you.
它就是没法用。
It just doesn't work.
所以有时候我用手机时,会用iOS键盘的语音转文字功能,是的。
And so, like, sometimes I'll be using my phone, and I'll use the iOS keyboard transcription Yeah.
是的。
Yeah.
是的。
Yeah.
在字段中打字,然后说一大堆内容,再发送出去。
To type in the field and then, like, say a bunch of stuff and then send it off.
但这让我觉得,消费者真的很想要一个能正常工作的语音模式,但目前主流的大型语言模型应用或任何其他应用都还没做到。
But this suggests to me that consumers really want voice mode that works, and yet it's just not working yet for the major LLM apps or for anyone.
为什么它还没能正常工作呢?
Why isn't it working yet?
这其实很难实现,因为你想要两样东西。
It is pretty hard to do because you want you want two things.
你希望可以自由地说出你想表达的内容,但有时又希望它能等待你完成句子并补充更多信息。
You want you want to be able to say things that you want, but you want sometimes for it to execute it, sometimes to wait for you to, like, finish and add something in the sentence.
有时你希望它能互动,主动提问以澄清并获取更多细节。
Sometimes you want it to be interactive so it asks you questions back to clarify and get some of the additional detail.
而所有这些实际上都相当困难。
And all of that is actually pretty hard.
这正是我们心目中语音助手的理想魔法版本所在:你需要语音转文字功能,需要转录模块,还需要对话轮换机制。
That's where kind of the magical ideal version of a voice agent for us comes through, where you need the speech to text element, you need the transcription side, unique you need then the kind of the turn taking mechanism.
那么,你什么时候才算说完一句话?
So, like, when do you finish sentence?
是根据沉默,还是根据上下文来判断什么时候该结束?
When when is it likely based on silence, based ways likely on the context?
有时候你希望它能回应并澄清,或者至少把文字反馈给你以确认,然后再执行一组指令。
And then sometimes you want it to speak back and clarify or at least give you the text back to clarify and then maybe execute a set of instructions.
所以这个问题仍然是一个非常具有挑战性的研究课题。
So that problem is still very hard research.
因此,我同意这个观点:在对话代理的协调方面,我们尚未通过真正的对话式图灵测试,即它未能像人类那样自然地回应。
So I agree with the claim that this orchestration side has not passed a true conversational agent Turing Test where it, like, behaves as you would expect from another person,
你可以这样说,我表达的意思更简单一点:我们早就已经通过了文本大语言模型的图灵测试,但在语音大语言模型上,我们还远未达到那个水平。
where you can say, I that's the simpler way of saying what I'm saying is that we have passed the Turing Test with text LLMs a long time ago, and we're actually nowhere near that on voice LLMs.
所以,这确实是个有趣的最后前沿。
So it's kind of interesting how that's a final frontier.
是的。
Yeah.
我觉得它在某些特定领域会奏效。
I feel like it's it's it's going to work in, like, specific domains.
比如在客户服务、电话场景中。
Like, in customer support, call
是的。
Yes.
能通过语音图灵测试。
Passes the voice Turing Test.
表现得很好。
Works well.
再比如另一个极端,像互动游戏体验——真正像与游戏中另一个人类那样互动,这非常困难,而且还远未实现。
And, like, let's take another, like, spectrum of that, an interactive gaming experience, like, a truly interactive as you would have with another human in that game, it's, like, you know, so hard and and and and further out there, we haven't passed it yet there.
是的。
Yes.
是的。
Yes.
听起来不错。
Sounds good.
对。
Yeah.
但我认为这是某种更简单版本的组合。
But I think that's a combination of, like, know, even like a simpler version of within that.
有时你会立即做出回应。
Sometimes you might give a response immediately back.
有时你需要调用工具从数据库获取更多信息,如何协调这一点。
Sometimes you need a tool call to get additional information from the database, how you orchestrate that.
因此,我们在与一些公司合作时最常见的问题是:你希望这些系统能非常出色地协调。如果是对话场景,比较简单,你可以直接让代理应答;但如果你需要验证身份,或从数据库中提取更多信息,你该怎么办?
So, like, that's probably the most common thing we see as we work with some of the companies out there is you want those systems to orchestrate extremely well where if it's a conversational use case, pretty simple, you can route the agent to speak with, but if you need to authenticate, if you need to pull additional information from the database, what do you do?
你该如何优雅地处理这种情况?
How do you handle that graciously?
这真的很难。
That's really tricky.
展开剩余字幕(还有 480 条)
而且,是的,从这个角度来看,我同意。
And, yeah, and and to that extent, I would agree.
就像,这只是在逐步实现而已。
Like, that's just getting getting getting there.
是的。
Yes.
我们的目标是希望在所有这些情况下都能通过语音图灵测试,或者在非语音场景下让所有对话代理通过图灵测试,我希望在接下来的一年左右我们都能实现。
And we'll hopefully see that we our goal is to, like, pass the voice Turing Test in all those cases or the Turing Test for all conversational agents outside of voice two, and I hope we will all be there in the in the next year or so.
对于订阅制企业来说,大量收入在结账前的最后一两秒内流失。
For subscription businesses, a lot of revenue is lost in that last few seconds before the checkout.
有人得站起来找钱包,或者输错卡号,或者遇到错误。
Someone has to get up and find their wallet or they mistype their card number or they hit an error.
他们就直接起身离开了,而你
They just get up and you
就失去了这笔交易。
lose the sale.
对于
For a
像ElevenLabs这样的公司,拥有数十万订阅用户,即使像这样微小的摩擦也会累积成巨大的损失。
company like ElevenLabs, adding hundreds of thousands of subscribers, even a tiny bit of friction like that, it would really add up.
但这就是ElevenLabs使用Stripe的Link的原因。
But that's why ElevenLabs uses Link from Stripe.
客户只需保存一次信息,就可以在超过一百万家使用保存凭证的企业中几秒钟内完成结账。
Customers save their details once, and then they can check out in seconds across more than a million businesses with saved credentials.
所以,如果你想为客户提供更快的结账体验,就应该启用Stripe的Link功能。
So if you want a faster checkout for your customers, you should turn on Link from Stripe.
你们在做个性化语音转录吗?
Are you guys working on personalized voice transcription?
我们让自己变难的一个方式是,当我跟Siri说话时,我有点口音,它有时很难听懂我,但我的口音并不会改变。
Where it feels like part of the way we're making it hard for ourselves is when I speak to Siri, I have a bit of an accent, and so it sometimes has a hard time understanding me, but my accent doesn't change.
所以它完全可以学会听懂John的说话。
And so it could just get good at listening to John.
但我的理解是,并不是这样。
But my understanding is it's not.
它只是在运行全球语音识别模型。
It's just like running the global voice recognition model.
我猜ElevenLabs也是在运行全球语音识别模型。
And I'm guessing it's the same for ElevenLabs where you're running the global voice recognition model.
但同样,你有口音。
But, again, you have an accent.
所以,如果有人像在咖啡店遇到你那样,只听到你说两个词,可能会很难理解,因为他们没有像波兰口音过滤器那样去适应你的发音。
And so if someone's understand like, if you walked up to someone in a coffee shop and said two words, they might have a hard time understanding it because they'd like they're not putting it through their Mati Polish accent filter.
那么,这最终会如何发展,真正去理解对方这个人呢?
And so where is this going with, like, actually interpreting the person that you know to exist on the other side?
是的。
Yeah.
我有一个非常难以识别的口音。
I have a I have a very tricky one to detect.
我的声音经常被用在你们的测试套件中。
So my voice is frequently used in the test you're from the test suite.
是的。
Yeah.
无论是文本转语音、语音转文本,还是其他所有方面。
For text to speech, for speech to text, for, like, everything.
是的。
Yeah.
它还挺常见的。
It's it's pretty
但再次强调,要在全局模型中识别你的声音,只会让事情变得更困难。
But, again, trying to parse your voice in a global model is just making life hard.
就像专门做一个针对Mati的模型。
It's like have a Mati specific model.
是的。
Yeah.
所以在语音转文字、转录方面,正是如此。
So so on the on the speech to text, like, transcription, exactly.
我们现在引入的一个主要部分是,它包含两个部分。
Like, the big part now that we are bringing in is you have two parts.
第一部分,实际上是针对特定人物或声音的检测,这在口音方面成立,在嘈杂的房间中也同样成立。
One, effectively like a person or a voice specific detection, which is true for the accent side, but it's also true for a crowded room.
这得益于我们一支了不起的研究团队,他们能够持续保持高准确率,而且确实如此。
That's where we incredible research team that's able to continually do both the accuracy high, but also Yes.
此外,还要添加像说话人检测、降噪等功能。
Also also, like, add things like speaker detection, of course, noise reduction.
但第二部分也是关键词检测。
But then the second part is also keyword detection.
在这些场景中,有一些特定的词语是你希望说出来的,并且需要有效监控。
So there are specific words that you would want to say in those settings that you want to effectively monitor for.
所以我们之前提到过,比如说,我去咖啡店点单的情况。
So we spoke about, you know, like, let's say I'm going to the coffee shop and order things.
我希望在咖啡店中完成的一系列操作,咖啡店会期望我去做
The set of actions I would like, coffee shop would expect me to do
就是说,他们只需要留意那些与咖啡相关的词汇即可。
is It's like there's a you know, they can just listen out for the the coffee words.
没错。
Exactly.
没错。
Exactly.
然后尝试将其匹配到最接近的语境。
And then try to, like, match it to the closest proximity.
这两者都会有所帮助。
So both things will help.
一个能完美识别并处理我声音的系统设置。
A setup where you have my voice, perfect, you can decode it and code it on that.
如果你没有我的声音,或者即使你想进一步增强识别效果,我们已经支持关键词检测,这在实时和异步场景中都非常有用。
If you don't have my voice or even if you want to double amplify it, we already support effectively a keyword detection, which is useful for, like, real time setting and async settings.
所以,是的。
So Yeah.
回到Chicky Pine的转录上。
Back to, like, Chicky Pine transcription
对。
Yeah.
你可以有效地从前一个播客中预先生成这些内容,并查找你常用的那些词,比如‘是的’。
You could effectively pre pre generate that from the previous podcast and, like, look for a set of words that you would use Yes.
传统上就是这样做的。
Traditionally in that.
那么有多难呢?
And so how hard okay.
你已经做了关键词检测,但我想通过在让系统聆听Mati之前,先输入一小时的Mati音频,来实现超人级别的转录性能,这样系统就能更好地转录Mati的语音。这可行吗?
So you do the keyword detection already, but how hard are the I want to get superhuman transcription performance by feeding it an hour of Mati audio before it listens to Mati, and then it should be able to do a much better job transcribing.
这是一个非常难的研究问题吗?
Is that just a really hard research problem?
不。
No.
可解决。
Solvable.
这是可解决的。
It's solvable.
我们认为可以在下一个版本中推出,希望能在下个月实现。
We think we can we can we can roll it out in one of the next versions, which is, like, hopefully in the next in the next month.
哦,所以你认为今年就会实现,是吧。
Oh, so you think this year, you're doing Yeah.
当然。
For sure.
个性化语音转录。
Person specific transcription.
个性化语音转录。
Person specific transcription.
比如,我们现在已经能非常精准地识别说话者了。
Like, we can already diarize speakers extremely well.
比如,当我们说话时,当然可以分辨出谁在什么时候说话。
Like, if we are speaking, we can, of course, dissimilate who is speaking when.
是的。
Yes.
在转录方面,除了准确性之外,说话者分离是最具挑战性的问题之一。
Which is like, in transcription side, apart from accuracy, diarization is one of the harder problems.
而我们在这方面做得非常好。
And We do that extremely well.
现在,这将实际上如你所说,根据我想听的说话者进行微调,我们知道这将非常重要。
Now it's going to be effectively what you're saying, fine tuning based on the speaker that I want to listen to, which we know will be important.
在医疗环境中,这是非常关键的一部分。
In healthcare setup, such an important part.
你在手术室里,是一名医生,你想发出一个指令,然后你真的需要能够听到那个特定说话者的内容。
You're in an operating room, you're a doctor, you want to say a command, then you want to really be able to listen to that one person specific piece.
是的。
Yes.
你家里有一个硬件设备,比如说一个遥控器,可以帮助你控制电视。
You have a hardware device at home, let's say it's a pilot, that helps you control the the the TV.
同样地,你希望它能听懂你的话,而不是家里的其他人走动时的声音,或者你可能希望它对所有人都响应。
Here too, you will want that to listen to you versus, let's say, the family roaming around, or maybe you want it to everyone.
所以,你可以决定这一点,但在很多情况下,你希望有能力进行指定。
So, like, you you could decide that, but in in many cases, you you want to be able to specify that.
好的。
Okay.
这真的很令人兴奋。
That's really exciting.
这很棒,因为还有很多未解决的研究问题,是的。
It's great because there's, like, still so many unsolved research problems that yeah.
我们可以
We can
在语音模型领域,一波又一波的突破不断涌现。
They're, there's just breakthrough after breakthrough coming in the in the domain of voice models.
那么反过来,说到语音生成呢?
How about on the flip side when it comes to speech generation?
你知道Zoom的美化我的外观功能吗?
You know, the Zoom touch up my appearance feature?
是的。
Yeah.
对。
Yeah.
我一直在想,在语音方面,你该在什么时候提供一个去口音的过滤器?
I've always thought about that in the context of voice where should you offer a deaccenting filter for voices?
或者,像我常听的那个播客,把声音弄得有点含糊,我一直觉得他们应该用一个去含糊过滤器来让声音更清晰,或者放慢
Or, like, even this one podcast that I like to listen to, put the voice a little mumbly, and I always thought they should put it through a demumbling filter just to, like, make Or slow
放慢,慢下来。
it down, slow it down.
让发音更清晰一些。
Make the enunciation a little better.
是的。
Yeah.
但所有这些,就像修图一样,是的。
But all these things, again, like photoshopping an image Yeah.
有没有想过直接进行语音到语音的转换,而不是语音转文本或文本转语音?
There's no reason that the like, have you thought about voice to voice, basically, rather than voice to text or text to voice?
是的。
Yeah.
嗯,这主要分为两个大方面。
Well, that that so there are of two big parts.
一方面,在语音生成方面,情况类似。
One, on the speech generation side, similar.
那里仍然有太多太多的创新。
So many so many innovations still there.
还有一个更广泛的方面,那就是我们发布了一个免费模型,首次解决了这个问题。
There's, like, a wider piece, and that's, like, the we released a v free model that kind of we're solving that for the first time.
你能控制语音吗?
It's like, can you control speech?
所以你可以实现文本到语音的转换。
So you can have the text to speech.
你可以生成听起来情感丰富的语音。
You generate something that sounds emotionally great.
直到去年年底,之前基本上都是依赖模型来决定最佳表现。
Previously, until end of last year, effectively, would rely on model to decide what's the best performance.
你可以重新生成,但最终还是由模型决定最佳表现。
You could regenerate it, but ultimately model decides the best performance.
因此,可控性就出现了——我们可以给它提示,比如放慢语速、调整戏剧性的停顿,或者任何你给出的提示,而要实现这一点,你需要架构上的改变,以及我们长期以来积累的数据,这些数据标注了说了什么以及如何说的,这样你就能训练模型做到这一点。
So that's where the controllability came in, where we can finally give it cues of, say it in a slower way or change how you deliver the dramatic pause or kind of any, cues that you give, and to be able to do that, you need architectural changes and the data that we kind of created over time where you annotated what was said and how it was said, you can actually train the model to do that.
所以今天,你终于可以同时实现快速生成或完整的语音代理体验,也就是我们所说的‘表达模式’,在这种模式下,代理能感知对方的情绪,如果对方压力大,它就能做出反应并给予安慰,这既生成了安慰性质的语言响应,也以相应的情绪表达出来,这一突破非常难以实现,当然这也延伸到了你提到的许多方面。
So today, finally, you can have both speed generation or entire voice agent experience with what we call expressive mode, where the agent knows the emotions on the other side, so if the person is stressed, it can react and be reassuring, and that's generating a LM response on the reassuring side and response in that set of emotions too, and that breakthrough was super hard to do, and that of course stretches to a lot of what you said.
这可能是某种语音增强的版本,无论是实时的还是脉冲式的,用以改变语音的呈现方式,这是相对较新的创新,但我们知道它仍有很大的提升空间。
It could be some version of speech enhancement, either real time or in a pulsed setup to change how that's delivered, and that's relatively recent innovation, and we know it can still be so much better.
比如,你想要描述的边缘情况其实非常多。
Like, the edge cases of how you want to describe it is pretty large.
所以这是第一点。
So that's one.
然后是问题的第二部分,这也是一个非常重要的问题:语音到语音模型。
And then the second part of the question, which is a huge question, the speech to speech models.
正如你所说,我们的方法,当你思考语音助手或对话助手时,本质上是一种级联方式。
So as you said, our approach, as you think about voice agent, conversational agent, is effectively a cascaded approach.
你先将语音转为文本,再通过大语言模型处理,然后生成语音,并将这一切协调起来。
You use transcription of speech to text, LLM, text to speech, and orchestrates all of that together.
而语音到语音则是直接从语音输入到语音输出,中间不经过文本。
And then you have a speech to speech, which kind of goes directly from speech, and there's a speech response on the other side.
当我提到语音到语音时,你的意思是它在中间不经过文本编码吗?
When I say speech to speech, is that the idea that it doesn't go through text as an encoding in the intermediate?
没错,正是如此。
Oh, exactly.
好的。
Okay.
出于性能和准确性的考虑
For performance reasons, for accuracy reasons
你通常是为了降低延迟。
You usually do it for latency.
好的。
Okay.
为了降低延迟。
For latency.
运行一个无需转录再生成的模型更快。
Faster to run a model that does not have to transcribe and then generate.
没错。
Exactly.
它更快,但另一方面,你会失去可靠性。
It's it's quicker, but on the flip side, you lose reliability.
是的。
Yes.
你会失去对整个流程各部分的所有可见性。
You lose, like, all visibility into the parts of the pipeline.
而且在情感表达方面,我们认为你在这两个方面都能做得非常好,也许还能让它更具可控性。
And emotionality, we think you can deliver both on both sides extremely well, and maybe you can make it more controllable too.
所以今天,我们正在大力优化级联方法。
So today, we are optimizing heavily on a cascaded approach.
抱歉,级联方法是指
I'm sorry, a cascaded approach is
语音转文本流向文本层。
Is the speech to text going for the text layer.
好的。
Okay.
面向文本层。
Going for the text layer.
当我们与许多企业和机构合作时,他们需要了解流程中发生的一切。
And and as we work with, like, a lot of the businesses and enterprises, they they will need that visibility into what happens.
他们希望执行某些任务。
They will want to execute certain tasks.
除此之外,他们还希望对每个步骤有良好的可见性,并确保所有模型的高准确性。
On top of that, they want a good visibility into each of the steps and and great accuracy of all the models.
但除此之外,他们可以忽略底层的语言模型层和智能层是什么。
But beyond that, they can abstract away what's the LM layer, what's the intelligence layer.
在这个系统中,集成更加简便。
The integrations are easier in that system.
因此,我们投入了大量研究工作来实现这一点,我们相信能够做到这一点。
So that's, like, where we are betting a lot of the the research work of how you can make that great, and we think we can make that great.
至于语音到语音,当你考虑更像应用的伴侣版本时,这将是它大放异彩的地方,因为幻觉可能没那么重要,但延迟会稍高一些,也许幻觉甚至是一种特性。
And speech to speech, as you think about maybe more of, a companion version of the applications, of that's where that will flourish because maybe the hallucinations aren't as important, but the latency is a little bit more, and maybe hallucinations are even a feature.
也许在未来,为了完成这部分,你会看到某种模型组合,对于低复杂度、简单的任务,你会使用语音到语音的模式。
And maybe in the future future, just to finish that part, you will have, like, some version of combination of the models that for, like, low complexity, easy models, you will have speech to speech.
而对于更高层次的复杂任务,你会使用级联模式。
And for, like, higher layer complexity, you will have the cascaded.
好的。
Okay.
所以我本来想问一下这个。
So I was gonna ask about this.
你知道吗,关于书写发明如何改变人类大脑,以及以超越实际书面语言的方式改变神经通路,这方面的研究很多。
Like, you know the way there is research on how the invention of writing changed humans' brains and just, like, changed the the neural pathways in ways beyond kind of the the actual written language.
你有没有发现,语音到语音模型与级联模型的思维方式不同?
Do you observe that speech to speech models think differently than cascaded models?
听起来它们好像更笨一些。
Like, sounds like they're dumber.
它们
They
确实更笨一些。
are definitely dumber.
你需要更小的模型。
You need smaller models.
你不能你
You cannot you
但这很有趣,对吧?强迫模型去思考文本。
But that's interesting, right, that, like, forcing models to reason about text.
我的意思是,我知道它们里面还包含更多内容,但它们更聪明。
I mean, I know they just have much more in there as well, but they're smarter.
是的。
Yeah.
但如果你做语音到语音,通常你会使用更小的模型,这样仍然很快。
But it's like, you know, like, if you are going speech to speech, usually, you you will use smaller models, so it's still quick.
设置
Set
我明白了。
I see.
所以这纯粹也是模型大小的问题。
So it's also just a model size thing.
是的。
Yeah.
那么,除了像大小这样的相关因素之外,还有其他有趣的差异吗?
So that would But are there interesting differences beyond, like, correlates like size?
我可以说的是,这和你的问题略有不同。
What I can say is, like, slightly different to your question.
通过语音进行互动的人们,以及他们与业务互动时的表现,会因为互动方式本身而发生变化。一个很好的例子是,你可以联系ElevenLabs并注册表达你的兴趣。
The people interacting through voice and the performance we see for, like, how they interact with the business changes just by nature of interacting with A good example, you can contact ElevenLabs and register for your interest.
你填写表格时,我们额外提供了一个选项:你可以不走表格流程,而是直接与我们的智能代理对话,留下更多详细信息。
You go for the form, and at the end of that, we have supplemented that, that instead of going through the form process, you can speak with our agent and leave more details.
在网络研讨会上,有两点。
In webinar, two things.
第一,人们更愿意通过与代理交谈来离开论坛,因此我们处理论坛会容易得多。
One, people were actually much more keen to leave the forums through speaking with the agent, so we would go through the forum a lot easier.
但第二,他们在使用场景方面会更加开放。
But second, they would be a lot more open ended in terms of what the use cases are.
因此,他们会主动向我们提供关于更广泛使用场景及其复杂性的信息。
So they would start giving us information about the wider set of use cases, the complexity of the use case.
所以,手动填写表格既繁琐又困难。
So, like, the writing out was tedious and tricky.
这就像一场开放式的冒险。
This is like an open ended adventure.
开放式。
Open ended.
你可以提出跟进问题。
You could ask follow-up questions.
你可以进行澄清。
You could clarify.
但人们在使用这个系统时感觉更放松,也更信任它能正常工作,这对我们帮助很大。
But people were just more at ease and could trust the system while doing that, that it's working, and that kind of helped us a lot.
然后是免费,这可能更多是一个技术壁垒,但它支持所有语言。
And then free, which maybe is more of a technological barrier, it also works across all languages.
所以我们现在收到了来自世界各地的潜在客户,他们留下了自己的信息。
So now we have leads from like all parts of the world coming in and leaving their details.
我们做了这个用例,现在有几家不同的公司也在开发他们自己的SDR版本,是的。
So we did that use case, and now we have a few different companies building their SDR versions of that too Yes.
帮助那些能帮助他们捕获来自银行乃至汽车公司等各类机构的潜在客户,这些公司的人更愿意通过语音进行交流。
To help who'll help them capture the leads coming coming in from from banks all the way to actually, one of the automotive companies that, like, leaves that where people are just more keen to speak through voice.
所以是的。
So yeah.
我想问问这种二级效应。
I wanna ask about this kind of a second order effect.
你以前提到过,在波兰长大的时候,电视节目的配音很便宜,所以一部剧通常只用一个配音演员。
You have you know, you talked in the past about how growing up in Poland, I guess, the dubbing of TV shows, they were cheap, and so they would only have one voice actor for a TV show.
所以不管男女角色,它们都像是在说‘爱你’。
So no matter all the parts, male and female, they're like, love you.
我也爱你。
I love you too.
你知道吗,以前是一个配音演员负责所有角色。
You know, there's, like, one voice actor doing all of them.
而现在,多亏了更好的语音模型,你可以为所有配音生成非常自然的AI语音。
And now, you know, thanks to better voice models, you'll be able to just have, like, really good voices AI generated for all the dubbing.
因为这并不是在取代之前那些优秀的配音工作。
Because, again, it's not like it's taking jobs from great dubbing that was happening previously.
以前在波兰,配音质量其实非常糟糕。
Was, like, awful dubbing, you know, happening in Poland previously.
所以这就是一个第二层影响的例子。
So that's, like, one example of the the second order effects.
你在语音合成和语音识别普及后,还观察到哪些其他的第二层影响?
What are the other second order effects you're seeing of ubiquitous goods, text to speech, speech to text?
看起来在多种语言中都是如此。
It seems like across a broad array of languages.
因为无论英语如何,这种技术在波兰语、爱尔兰语,或者你选的任何语言中都曾经不存在。
Because whatever about in English, just this didn't exist in Polish or Irish or, you know, pick your language.
首先,打破语言障碍,我们的灵感虽然来自电影领域,但也适用于任何沟通场景。
One, breaking down the language barrier, we you know, the kind of the inspiration came from from the movie side, but it also applies in any in any communication setup.
比如,未来我能否去另一个国家,说波兰语或英语,而当地人的母语却听不懂我说的话?
Like, could in the future, could I travel to another country and speak Polish or speak English, and that language isn't being understood in the local native language?
就像《银河系漫游指南》里的巴贝尔翻译器那样。
Like from Hitchhiker's Guide to Galaxy, this version of the babble Exactly.
这样你就能真正理解这个世界了。
Then you can actually understand the world.
语音当然会成为一种交互层,但同样地,我们每个人都会拥有自己的扩展助手和语音代理,代表我们行事,已经有非常清晰而出色的例子,比如那些失去声音的人重新找回了声音。
Voice, of course, will be an interaction layer, but similarly, all of us will have our own kind of extension and voice agents that can help on our behalf, and there's very clear and great examples of that of people that lost their voice and can get it for the first time back.
我们到处都能看到这种情况,无论是因渐冻症还是喉癌失去声音的人,现在都能重新获得声音。
We see that everywhere, whether that's people that lost it due to ALS or throat cancer that can get it back.
最近,有一个使用Neuralink的患者案例。
Just recently, there was an example of a patient that had Neuralink.
我们与他们合作,帮助这位患者重新用自己原本的声音与家人交流。
We worked with them to bring the voice that that person could speak with their own voice back with the with the family around.
我们曾与一位在结婚前失去声音的女士合作,直到技术终于成为可能。
We worked with with with with the lady that lost her voice before before she got married, and and then finally, technology became possible.
我们成功复刻了她的声音,让她第一次能够重现婚礼仪式,与伴侣共同宣誓,那真是一个感人至深的时刻。
We we were able to recreate that voice, and for the first time, she could replicate the the marriage ceremony and speak the vows together, which was, like, such a heartfelt moment.
这可能是我们所有工作中最重要的部分。
Probably, like, the most important from all the work that that we do.
当你们提到语音代理时,
When you guys you guys
所谓的语音代理,是指一个长期运行或持续存在的代理,通过语音与外界互动吗?
talk about voice agents, is a voice agent just the idea that you have some long running or persistent agent that is going out and interacting with the world through voice?
客户服务就是一个这样的例子。
And so customer service would be one example of this.
你知道吗,反过来,你可能会在订餐厅座位时,实际打电话给餐厅。
You know, in the other direction, you're clogging and making you a rest a restaurant reservation and actually calling up the restaurant.
我是不是应该这样理解语音代理呢?
Is that is that kinda how I should think about voice agents?
没错。
That's right.
就是这样。
It's it's it's exactly.
无论是被动地与客户互动,还是主动回拨,都是一样的。
Whether it's like the reactive side of being able to interact with the customer or the proactive to call it back.
我们最近有一个非常有趣的案例,正好和健力士有关,当时有个开发者在开发一个Gindex。
We recently had a very interesting one, topical because it was a Guinness related one where there was a there was a developer developing a Gindex effectively.
哦,我看到过那个。
Oh, I saw that.
他们正在给爱尔兰的所有酒吧打电话,查询一杯啤酒的价格。
They're calling all the pubs in Ireland, checking the price of a pint.
是的
Yeah.
可以像这样询问并获取我们的报告信息
Could, like, ask that and our report information
那个叫Gindex的系统也是用ElevenLabs构建的
That or was called Gindex is built with ElevenLabs too,
它也是用ElevenLabs构建的,所以人们实际上可以做到
It was built with ElevenLabs too, so, like, people could actually Yes.
既能主动联系,也能被动响应,是的
Could do both sides, could proactively reach out, reactively reach out Yes.
始终通过语音捕捉,是的
Always capture through voice, and Yeah.
是的
Yeah.
是的,然后大约3003个不同实体可以报告他们的价格并实现信息同步
Yeah, and then kind of 3,003 different entities could report their prices and and and get that across.
顺便问一下,你把OpenClaw连接到ElevenLabs了吗?
Have you, by the way, hooked up your OpenClaw to ElevenLabs?
OpenClaw和ElevenLabs的组合是Eleven内部很多人在用的吗?
Is the OpenClaw, ElevenLabs combo something that a lot of people at Eleven are doing?
所以,正如你所知,OpenClaw会尝试寻找那些最常用的工具并进行连接。
So, yeah, as you know, the OpenClaw will, like, kind of look for the most popular tools frequently where it tries to hook up.
是的。
So Yeah.
是的。
Yeah.
是的。
Yeah.
其中一个推荐的选项。
One of the recommended ones.
语音方面的首选方案。
The top option for voice.
你能跟我讲讲语音模型的商业情况吗?我觉得人们对大型语言模型有一个直观认识,那就是这些模型的训练成本非常高。
Can you tell me a bit about the business of voice models where I think people have an intuition around big LLMs where there are these very expensive training runs.
是的,这些模型确实会很快贬值,但由于使用量巨大,迄今为止所有训练过的模型都已收回训练成本,甚至还有盈余。
And yes, they kind of depreciate quickly, but the there's so much usage that all of the models trained to date have, you know, paid off their training runs and then some.
而现在,资本支出正在持续增加,其中很大一部分如今是用于推理,但也包括训练。
And then there's this kind of ever larger CapEx going into I mean, a lot of it is inference these days, but also training.
因此,我对语言模型领域有一些直观的理解。
And so I have some intuitions from the LLM world.
我想知道,对于语音模型,我该如何看待?首先,训练语音模型有多昂贵?
I'm curious just how I should think about voice for, one, how expensive is training the voice models?
成本是来自研究人员吗?
Is the expense in the researchers?
成本是来自训练过程本身吗?
Is the expense in the the training runs?
而且,商业模式想必相当简单,就是按使用量计费,但还是请你详细讲讲这个商业逻辑。
And, I mean, the economics is presumably kinda simple, but it's just per per usage, but, yeah, just talk us through the business.
是的。
Yeah.
肯定比语言模型和图像视频模型便宜,因为模型会更小。
Definitely cheaper than the LM and and image video models, It's gonna be smaller models.
对。
Yep.
好的。
Okay.
所以模型更小吗?
So the models are smaller?
更小。
Smaller.
前沿语音模型的参数量是多少?
What's parameter count for a leading edge voice model?
几十亿到低几十亿参数的模型。
Few billion to low tens of billion parameter models.
是的。
Yeah.
所以,为了说明一下,我认为就像CPU最终不再以吉赫兹作为衡量标准,而是转向更多核心一样。
So so And for context, I think the I mean, kinda like, you know, CPUs moved away eventually from gigahertz as, like, the metric as they move to more cores.
我认为我们已经基本不再仅仅以参数数量作为衡量标准了,但目前领先的大型语言模型参数量在数千亿级别。
I think we've mostly moved away from just raw parameter count, but I think the leading edge LMs are in the hundreds of billions of parameters.
我认为领先的模型确实是这样。
I think the leading ones, yes.
但当然,你也会使用规模较小的变体版本。
But, of course, you know, you have the variations that you will use at at lower scale.
对。
Yep.
因此,资本支出仍然很高。
So CapEx is still pretty high.
是的。
Yeah.
我们最近以110亿美元的估值融资了5亿美元,这很合理。
We've, of course, raised recently a half 1,000,000,000 at at 11,000,000,000 valuation to, like Makes sense.
这很合理。
Makes sense.
为了能够持续打造世界上最好的模型。
To continue to continue being able to build the best models in the in in the world.
研究人员,当然,你希望拥有世界上顶尖的人才。
Researchers, you know, of course, you want to you want the best people in the world.
我认为我们确实有这些人才在音频领域工作,由微创始人领导这项工作。
I think we have we have those people working in in in audio, in in in micro founder who who is who is leading that work.
所以,这无疑是保持雄心壮志的重要部分,虽然不是从财务角度,而是从其他方面来看。
So that's that's that's definitely a big piece of, like, not financially, but even, like, how you keep the ambitious Mhmm.
关于部署,持续打造领先的模型有助于吸引更多人才并构建这一生态。
About deployment, deployment, so you kind of continue building leading models helps you attract more talent and and building that.
至于服务方式,推理显然与模型的使用方式相关,对我们而言,我们在各个领域都看到了惊人的增长。
And then on the how we service, of course, inference is correlated with how the models are used, and and for us, like, we've seen incredible incredible growth across the work.
大多数情况下,如果是文本输入或文本转语音,通常按文本标记计费。
Mostly, this is charged per if it's input text or text to speech, it's usually per per per text token.
如果是语音代理或转录,则按分钟计费,我们发现这部分占比更大。
If it's voice agent or or transcription, then then it's per minute, and we see that kind of being the the bigger part of it.
通常来说,总体上是按标记计费,当然,当我们与企业合作时,通常是年度协议。
Usually, like, broadly, it's per token basis, and, course, as we work with businesses, it's like an annual agreement.
消费越多,谈判空间越大,折扣也越多,是的。
The bigger the spend, the bigger the comments, the bigger the discount to Yeah.
我会把这个传达清楚。
Gonna get that across.
我们通常的做法是,当推出新模型时,我们会以成本价提供给许多客户,让他们体验最佳效果。
The way we usually do is, like, when we have a new model, we try to give it at cost to a lot of the customers so they can experience the best.
它通常还是,比如,没有那么,但只是
It's still usually, like, not as But just
最新产品往往最贵,而你们却让最新产品成为最具经济吸引力的?
the newest thing is often the most expensive, whereas you make the newest thing the most economically attractive one?
我们努力让它具有吸引力,这样客户就会觉得,虽然它对我们来说比上一代更昂贵。
We try to make it attractive so the customers are like, you know, like, it's it's more expensive for us than any previous generation.
我们不喜欢
We don't like
是的。
Yes.
质量更高了,所以我们尽量保持价格仍然有竞争力,我以前经常补贴它,
Oh, the quality is higher, so we try to keep the prices still still competitive I used to to
但它的成本本质上更高。
subsidize it, but it's inherently more expensive
来计算模型。
to figure model.
没错,没错。
Exactly, exactly.
随着时间推移,我们可能会做一些优化技巧,但我们希望客户能体验到:由于研究的原因,我们发现模型在早期阶段的可靠性可能不足,其次,人们甚至不知道这个模型能做什么。
And over time, we might do some tricks to optimize it, but we want the customers to experience because of research, the big thing that we've seen is the reliability of the model in the early days might not be there, and then two, people don't even know what's possible with that model.
所以你希望覆盖最广泛的分布,让每个人都能展示出这个模型的潜力,同时你也可以把它作为分发机制,从中学习哪些地方需要改进和调整,是的。
So you kind of want the widest set of distributions so people can show the world what's possible, so you can have it, of course, as a distribution mechanism, learn yourself what to improve, what to change Yes.
然后把它推广出去。
And then and then get it out there.
语音模型是不是变得越来越大了?
Are the voice models just getting bigger and bigger?
比如,我们会拥有数百亿参数的语音模型吗?还是说,对于某些特定的模型架构,已经存在一个自然的规模上限?
Like, will we have voice models in the hundreds of billions of parameters, or have we found like, it seems like for certain types of model architectures, there's, like, an upper limit on, like, the natural size.
我们是否已经找到了语音模型的这个上限?
Have we found that upper limit for voice models?
对于特定的使用场景,比如有声书朗读,你可能已经找到了合适的规模。
It's it's it feels like for specific use cases, like, say, audiobook narration, you probably found that size.
没必要再把模型做得更大,也能让质量达到足够高的水平。
Probably don't need to stretch it too much bigger to to make the quality as much higher.
但对于某些使用场景,模型规模可能还会继续增长。
But for certain use cases, that will probably grow.
在级联方法中,你可能不会看到规模上的显著变化,这正是我之前对这个问题犹豫的原因。
The the thing that's that's, you know, like, I hesitated on the question is, in a cascaded approach, you probably will not see, like, dramatic size changes.
嗯。
Mhmm.
你天然希望模型运行快速,是的。
You inherently want the models to be quick Yeah.
而且它们要可靠。
And they're reliable.
你需要以一种智能的方式协调它们。
You want to orchestrate them in a smart way.
在融合方法中,模型参数可能会达到数十亿甚至数百亿级别,因为你把语言模型和语音模型的特性结合在了一起。
In a fused approach, probably that will get into, like, tens, hundreds billion parameter models because you kind of combine, of course, the Intel the LLM side and the voice side.
是的。
Yeah.
所以这种情况下模型会变得更大。
So that will get bigger.
但在调整语音方面,我认为它会一直保持小巧。
But on the Adjust Voice, I think it will keep keep being small.
好的。
Okay.
但在某些领域,是的,我们会看到更大的模型。
But there are certain domains where, yeah, we'll see bigger models.
这仍然很有趣。
That's still interesting.
是的。
Yeah.
对。
Yeah.
从研究角度来看,令人惊叹的是,仍然存在这么多未解决的问题,而你们正在不断取得技术突破,并将它们融入产品流程中,这确实是公司生命周期中非常有趣的一个阶段。
It it is amazing how it does seem fun from a research point of view how there are still these various unsolved aspects and how you guys are just making technical breakthroughs and then releasing them down the product pipeline, that's like a really fun stage of a company's life cycle.
当然。
For sure.
这很有趣,因为我们感觉可以在两个方面都进行创新。
It's like fun because we it feels like we can do innovation on both sides.
研究方面有很多工作,产品方面也有很多,而最终最重要的部分是如何将它们部署给客户——轻量级企业和中小企业与大型企业的需求动态会非常不同。
There's like so much on research side, so much on product side, and then like the kind of, ultimately, the biggest parts is how we deploy it to the customers, where Light and Intent, like SMB will have very different dynamic than the enterprise.
这并不是简单的供应商与SaaS关系,只是把产品提供给最大的公司,而是更像他们在AI转型过程中的合作伙伴,你需要投入资源与他们并肩工作,应对那些以前不可能实现的全新应用场景,并将这些语音代理投入生产。
It's not vendor SaaS relationship where you just give the product out there for the biggest companies out there, but you are more of a partner in their AI transformation part, so you want the resources to work alongside them to work on the frequently very new use cases that were impossible to help create and bring those voice agents to production.
是的。
Yes.
这是一个巨大巨大的转变。
So that's, a big big big shift.
是的。
Yes.
但最重要的重点是如何将对话式代理推向全球各地的企业。
But the biggest focus is is is how we how we bring the conversational agents out there to to the businesses around the world.
所以当你说到推动对话式代理是最高优先级时,这是指客户服务这类应用场景吗?
So when you say con bringing conversational agents is the biggest priority, is this for customer service type use cases?
那么,对话代理最常用的应用场景有哪些?
Like, what are the most popular use cases for conversational agents?
是的。
Yeah.
我们希望成为企业与其客户或受众之间全方位互动的合作伙伴。
Like, we want to be a partner for, like, full interactions between business and businesses and their customers or their audience.
我说的是他们的受众,因为这同样适用于客服场景。
I'm saying their their audience because that will apply in support.
客服是最容易实现的,因为那里的条件最成熟。
Support is the easiest one because that's where it's most ready.
对。
Yeah.
但可能我们与其他公司最大的不同在于,这同样可以应用于销售场景。
But like and that's maybe the big difference to how we see ourselves to some of the other companies in the space is this can also apply to sales.
你也可以主动出击,进行回访。
You can you can have the proactive side of reaching back.
你可以拥有那种 AISDR 版本。
You can have AISDR versions of that.
是的。
Yes.
然后你还可以延伸到营销场景,我们能成为你的合作伙伴,协助你打造出色的营销活动,甚至超越单纯的对话式代理范畴。
And then you can have all the way to the marketing use cases where we are your partner for for working on on on even outside of, like, the the conversational agent space of how you create a great marketing campaign.
是的。
Yes.
先生,关于这一点,你们和 Intercom 的 Des Trainer 有什么不同?他们有个叫 Finn 的代理,是网站上一个可以对话的入口。
And, sir, how does it break down between you know, we had Des Trainer from Intercom on here, and they have Finn, their agent, and it's a thing in the website that you can go talk to.
他描述的现象和你所说的非常相似:你最初可能觉得这能帮你们回答客户支持问题,但后来它变成了网站上的通用界面——一个你可以输入文字、完成操作、获取信息的对话框。
And he described a very similar phenomenon that you described, which is you start maybe thinking, oh, this will help me answer customer support queries, but it becomes like a generic UI for the, you know, for the website where it's a box you can type in to go do things and understand things.
那为什么你不直接看文档,然后按常规方式设计集成呢?
And so why wouldn't you read the docs and design your integration by, you know, whatever?
所以我会为文字和语音分别设置一个吗?
And so will I have, like, one for text and then one for voice?
你们也会做文字版吗?
Will you guys do text too?
这到底是怎么运作的?因为看起来在文字层面,这也同样有效,比如
Will just how does that because it seems like this is also succeeding at the text level with
是的。
Yeah.
Finn、Sierra 和所有这些系统。
Finn and Sierra and all these things.
我们知道,我们能提供最大价值的地方,是那些目前大部分或绝大多数交互都来自语音的场景。
The places where we know we will we will be able to provide the biggest value is, like, where, ultimately, today, you will have more either a big portion or most of the interactions coming from voice.
嗯。
Mhmm.
所以如果这种交集存在的话,是的。
So if that kind of intersection is there Yeah.
那就是我们能提供更高价值的地方。
That's where we can provide higher value.
当然,如果你需要一个文本聊天机器人,那就像你解决了语音代理一样,文本部分也会随之自然解决。
And, of course, like, if you if you need a text chatbot there, that's that's like if you fix the voice agent, you'll have fixed text piece, like, inherently as well.
但我们今天真正优化的重点是:如何为不同的客户互动选择合适的语音,以及在你之前提到的复杂情况下,如何协调语音系统暂停或深入查阅文档,如何让其成为整个业务的延伸,而不仅仅局限于客服,而是贯穿整个用户旅程。
But the place where we we do optimize today is going to be, like, how do you select the right voice for the right customer interaction, how you pull that in the in the pretty complex case of what you mentioned earlier of, like, how you orchestrate that to pause or look for something deeper into the docs, how it can be extension of entirety of the business, so not only in support but across entire of the user journey.
但关键是,我们希望能够在所有互动中为你提供支持。
But the bottom line is, like, we want to be able to provide you across entirety of the interactions.
是的。
Yep.
语音通常是这些互动中的重要组成部分。
Voice is usually a big part of those interactions.
对。
Yes.
而且,是的,我们需要解决集成问题。
And, yes, we need to we need to solve the integrations.
我们需要解决知识库的问题。
We need to solve the knowledge.
我们需要
We need
将文本作为其中一部分来解决。
to solve text as part of that.
但比如,我们不会深入探讨我认为在许多情况下会发生的事,比如那些需要多轮交互的推理型用例。
But, like, we wouldn't, for example, go into what I think will happen in a lot of those cases, like, very deeply into reasoning version of those use cases where you maybe need to, like, the multi touch Yeah.
是的。
Yeah.
而且一个
And a
很多复杂的操作。
lot of complex actions.
很多类似财务分析这样的内容,比如判断是否值得,这并不是我们优化的重点。
A lot of, like, financial analysis of, like, is like, that would be not not something we optimize for.
我们能谈谈
Can we talk
谈谈你们的收入增长情况?
about your revenue ramp?
或者你们就是过去几年增长最快的初创公司之一。
Or you're just one of the fastest growing startups, period, of the past few years.
你们最近公布的收入数字是多少?
What's your most recently announced revenue figure?
最近公布的数字是20亿,或者说是20亿。
Most recently announced was '20 Or '20
你们想给我们哪个数字都行。
whatever number you wanna give us.
最近公布的数字是3.5亿,发生在2025年。
Most So recently announced was 350 at the 2025.
是的。
Yeah.
但证明这项技术真正有效的最好证据,是我们最近与德国电信、T-Mobile、Revolut、Klarna、Meta和IBM的合作,涵盖了广泛的使用场景,而本季度在企业增长方面表现尤为突出,首次实现了单季度新增ARR达一亿美元,这太惊人了。
But the the best proof of the technology working so recently, are announced our work with Deutsche Telekom and T Mobile, with Revolut, with Klarna, with Meta, with IBM, a wide set of use cases, and this quarter was kind of one of the best for enterprise growth where we had the first quarter hit 100,000,000 in in an additional ARR growth, which is crazy.
净新增ARR吗?
In net new ARR?
净新增ARR。
In net new ARR.
好的。
Okay.
所以,如果你认为本季度ARR增长了1亿,到年底将达到3.5亿。
So if you think this quarter was a 100,000,000 in ARR and 350,000,000 at the end of the year.
是的。
Yeah.
我不是数学家,但增长到了约4.5亿美元的水平。
I'm no mathematician, but it's up in the 450,000,000 range.
而相比去年同期,这增长了数倍。
And that's versus this time last year, that's a several fold increase.
到底是什么在起作用?
Just what's what's working?
从外部来看,我假设账户内的用户群体增长非常强劲,而且自助服务和企业业务都贡献了很大一部分。
Like, from the outside, I would assume that there's really strong cohort growth within accounts, and then you seem to have self serve and enterprise businesses that both contribute a lot.
我知道自助服务的规模有多大,但作为用户,我喜欢能够自己动手操作ElevenLabs,而不是必须去找销售沟通。
I know how big self serve is, but as a user, I like to be able to fiddle with ElevenLabs not have to, you go talk to sales.
但也许你可以谈谈,是什么让你们在这么短的时间内实现了超过4.5亿美元的年经常性收入?
But maybe you can just talk about what works to reach $450,000,000 plus of ARR so quickly?
是的。
Yeah.
没错。
So exactly.
目前,企业业务中超过50%是由销售团队主导的。
So we are over 50% is now sales led on enterprise.
没错。
Yep.
我认为,很大程度上是因为这项技术在过去一年到一年半的时间里,其代理式交互的可靠性与高质量都得到了显著提升。
And, know, I think largely that technology powers a lot of their agentic interactions just became reliable at the same time as high quality over the last year, year and a half.
所以你经常非常了解这一点。
So that's frequently you know this extremely well.
你会先开通账户,然后当然它会持续扩展,我们确实看到在ElevenLabs上存在明显的‘获取并扩展’模式。
You will start the account, and then of course it continues expanding, and we we see there's definitely land and expand motion across ElevenLabs.
我们提供
We we we bring
那这种扩展具体是什么样子?
And what does that expand look like?
是新的部门吗?
Is it, like, new departments?
是当客户扩展时,使用量就开始激增吗?
Is it just the usage starts taking off when when a customer expands?
两者都有。
Both.
但通常,第一部分是我们努力让我们的客户使用起来非常简单。
But usually, the the first the first part too is, like, we try to make it very easy for our customers.
是的
Yeah.
也许我们自己在某种程度上违背了初衷,因为我们给这项技术提供了非常有吸引力的经济模型。
Maybe that kind of against ourselves where we give the technology a pretty attractive economics.
嗯
Mhmm.
因为我们非常相信这项技术能带来价值,所以你可以实际试用和测试,然后在这一过程中
Because we so much believe in the technology providing value, so you can actually try it and test it, and then within that
你的使用量会自然增长,基本上就是这样。
point of you'll make it up in usage, basically.
没错。
Exactly.
这种使用量会持续增加,因为你知道它在创造价值,于是做出选择就容易多了。
That usage, the kind of commit continues increasing because you know it's providing value, then it's so much easier to make that choice.
当然,跨部门的渗透也同时存在。
And then, of course, cross department pollination is there too.
我们的工作始于与Edge Telecom的合作,最初是从市场部门开始的。
It is like, you know, our work with Edge Telecom started marketing side.
我们先做了Magenta项目和POCTOS生成,后来逐渐扩展到客户支持,再进一步扩展到在整个网络中部署智能代理,让客户可以拨打电话并获得代理服务。
So we did Magenta work and POCTOS generation, then it kind of expanded to customer support, and then it expanded to us working on the in agent across the entirety of the network, so people can call in and have the agent.
你可以看到这些阶段性的跃迁。
So, like, you could see those step changes step changes across.
但我们现在公司有四百七十人,一直在持续增长,而一些保持不变的是小团队的模式。
But we are now four hundred four hundred seventy people as a company, so we keep keep on growing, but some of the things that stay consistent is small teams.
我们每个产品或研究项目都由不到十人的小团队负责,甚至在规划市场推广策略的分片时,这些团队也会深入理解行业和市场,独立快速推进,这在很大程度上促成了我们的成功。
So we have less than 10 people teams for each of the product or research initiatives, or even as you think about sharding some of our go to market strategy, those will be smaller teams understanding the industry in-depth, understanding the market in-depth, and going independently and going quickly, so that definitely contributed largely to that.
其次,针对大型企业,我们发现有效的方式是:我们覆盖了完整的光谱,从自助服务、产品驱动增长(PLG)模式,帮助推广并提升ElevenLabs的知名度。
Two, especially on the biggest enterprises, what we found works is, and it's like we have the full spectrum, self serve, PLG motion that helps distribution, drive, kind of awareness of ElevenLabs.
而在完全相反的另一端,我们提供高接触服务,派遣工程团队与客户并肩工作,共同定制他们整个工作流程。
And on the completely other spectrum, we have the high touch, four deployed engineering working side by side with the customers to to to customize the entirety of their work together.
你们为什么选择自助服务?
Why did you guys do self serve?
因为我认为你们有很多竞争对手,他们的技术都藏在联系销售的流程后面,你得先和SDR谈,再和AE谈,诸如此类。
Because I presume you have a lot of competitors where they have tech, and it's behind a contact sales forum, and you have to go talk to an SDR and then talk to an AE, blah blah blah.
你们却直接把技术放在一旁供人使用。
You guys just offer the tech available on the side.
我非常认同这一点。
And I'm a huge believer in this.
我的意思是,Stripe的快速增长很大程度上得益于我们让Stripe对任何人开放。
I mean, a huge part of Stripe's growth has been driven by the fact that we just made Stripe available to anyone Yeah.
并且围绕这种采用模式打造了大量产品。
And built a lot of product around that adoption pattern.
但很多公司似乎都跳过了这一步。
But so many companies seem to skip it.
所以我很好奇,你们为什么能有这么多理由?
So I'm curious how you guys can So many reasons.
有太多理由了。
So many reasons.
我想,快速想到的几个原因是反馈循环。
I think, you know, the the the quick ones that come to come to mind is feedback loop.
你可以立即了解你的技术有多好。
It just you can have immediate understanding of how good your technology is.
第二点,这是前一点的延伸。
Two, which is extension of that.
我们对自己的技术充满信心。
We stand behind our tech.
我们相信,无论是在模型、语音代理还是部署方面,我们的技术都是全球最好的。
We believe it's it's the best in the world for models, for voice agents, for deployment.
是的。
Yep.
所以我们希望人们能亲身体验这一点。
So we want people to experience that.
我认为你们在Stripe也是这么做的,最好的技术版本对每个人开放。
And I think you do that the same in Stripe where the best version of the technology is available to everyone Mhmm.
这真的非常吸引人,让人很想亲自试试。
Which is so attractive to actually try it out.
我们总是致力于为最高端的使用场景打造产品,然后将其回馈给整个生态系统,免费开放。
We always try to make everything we build for the highest end use cases, bring it back to the ecosystem free.
通常,企业级的最新应用场景都需要可靠性、合规性和可扩展性,而这些正是我们提供的。
Frequently, the newest of the use cases for enterprise, you will need reliability, you need compliance, you need the scale, which we deliver.
因此,在开发新技术时,它可能还不足以满足许多这些要求,但绝对已经适合开发者和中小型企业使用。
So frequently, as you develop new technology, it might not, not be ready for a lot of those parameters, but it's definitely ready for developers and SMBs.
我们非常欣赏他们所做的事情,因为他们向我们展示了未来,并有效地帮助我们
And we love what they are doing because they are showing us the future and effectively helping us
找到ElevenLabs应该前进的方向。
find a trajectory of where ElevenLabs should go.
艾伦,我完全信服。
I'm totally convinced, Alan.
我总是惊讶于为什么更多公司不这么做,这感觉就像他们因为不这样做而自断前程。
I'm just always amazed that more companies don't pursue it where it feels like they're really shooting themselves in the first by not.
你们是自己在Stripe上操作的吗?
Like, did you guys self serve on Stripe?
还是你们
Or did you
我们在Stripe上是自助服务的。
We self serve on Stripe.
是的。
Yeah.
比如,Eleven是一家大公司,但却是100%的。
For for example, you know, Eleven is a huge company, and yet A 100%.
你们最初是通过Stripe自助
You started on Stripe on a self
一开始,我们只有两个人,所以是自助模式开始的。
serve kind like initially, Ed, it's like, you know, we were two of us at the at the beginning.
你会去观察行业里什么有效,但同时也会从第一性原理出发思考。
You try to see what's working in the industry, but you try to think from first principle.
所以你想试一试。
So you wanna try it out.
你想了解它是如何运作的。
You want to understand how it works.
所以在你尝试之前,流程中的摩擦点越多,你就越怀疑它是否真的可用,或者是否在这些步骤背后隐藏了额外的费用。
So the more friction elements before you're trying it out, the less you trust whether it's available, whether there'll be additional payment that's hidden behind some of those steps.
是的。
Yes.
你不想经历这些步骤。
You don't want to go through them.
所以这太繁琐了。
So it's so much.
说到Stripe,你们对我们有什么反馈吗?
Speaking of Stripe, do you have any Stripe feedback for us?
有什么需要我们改进的地方吗?
Anything you want us to fix?
直到最近,我最常见的反馈是:为什么你们不提供按使用量付费的用户计费模式?
My most common feedback until recently is like, why don't you give us pay as you go user base billing type version?
但我们的财务负责人马蒂,我记得我之前和你们团队聊过,就是前一天的事。
But one of our finance lead, Mati, I know I was speaking with your team, and and that was day before.
是的。
Yeah.
他为此考虑了很久。
He was, like, thinking about it for a long time.
他很棒。
He's great.
就像你之前说的,是吗?
Was it like you said, like,
你们应该购买Metronome。
you guys should buy Metronome?
你们应该购买Metronome。
You should buy Metronome.
第二天,Metronome的收购消息就公布了。
And then the next day, Metronome acquisition was announced.
所以现在你们有了。
So now you have it.
所以这就是我最常见的反馈,我们即将推出。
So that's that was my most common feedback, and we'll be launching.
直接推出。
Directly.
回到自助服务,我认为AI领域的一个新趋势是,所有自助服务产品都应该提供按使用付费的选项。
Going back to self serve, I think a new thing in AI is that all self serve products should have pay as you go as an option.
也许你们想提供某种包含无限用量层级的订阅服务,但我不知道你们是否有过这样的体验:你在使用云服务时,一边输入查询,一边突然遇到速率限制,系统提示‘抱歉,你已达到使用限额’,而你希望像在Cloud Code中那样,能够直接按API调用付费。
Maybe you wanna have, like, a subscription with some unlimited tiers, but I don't know if you had the experience of, like, you're using Cloud and, like, you're typing away your queries and eventually you hit some rate limit and it's like, sorry, you've hit your usage limit, and you want to be able to do the thing that you can do at Cloud Code, which is just pay per API.
这没问题。
It's like, fine.
我愿意为此付费。
I'll pay for it.
作为消费者,没有机会多付钱来更多地使用产品,这其实挺讽刺的。
And it's kind of very funny as a consumer to not have the option to pay more to use the product more.
所以,是的,我认为每个AI产品都需要知道,他们可能希望提供一种‘无限使用’的订阅方案,但设有使用上限,同时允许用户为超额部分付费。
And so, yeah, I think every AI product will need know, they probably want to have some all you can need most of what you can use subscription with limits and then the ability to pay for overages.
所以听起来这正是你所要表达的。
So it sounds like that's what you're
是的。
Yeah.
没错。
Exactly.
这正是我们正在做的。
That's what that's what we that's what we what we're doing.
我另一想问你的是,我觉得如今所有大公司的CEO都在思考,这些AI进步如何改变组织的本质,以及如何围绕这些新智能重新设计组织结构?
The other thing I wanted to ask you about is I feel like all CEOs of larger companies today are trying to figure out how do all these AI advancements change the nature of the organization, and how do you redesign your organization a bit around all this new intelligence?
这可能涉及完成工作所需人员数量的规模变化,但也应该考虑:你是否需要更多资深人员,因为他们更能有效指导AI,而AI或许正在承担过去由初级员工完成的工作?
And so that could be about what the scaling factor is of, like, the number of people you need to do the work, But it also should be like, do you need more senior people because they're better able to direct the AIs, and the AIs are maybe kind of doing the work of what previously would have been junior people?
你需要更多初级员工吗?因为他们天生更适应与AI协作的工作方式?
Do you need more junior people because they're gonna be more AI native in how they work?
你希望团队更小吗?
Do you want smaller teams?
你希望团队更大吗?
Do you want bigger teams?
你是否真的会去重新设计流程,比如让财务团队广泛使用Claude?
Do you actually go do the process engineering of, you know, your finance team should be using Claude extensively?
但历史上,财务团队并不太会自己开发软件。
But, like, finance teams did not historically, you know, have a lot of home built software.
因此,现在有各种各样的问题在浮现。
And so there's all these questions that are floating around.
而你已经迅速打造了一家更适应AI的公司。
And you have very rapidly built a much more AI native company.
所以我很想知道,作为一家新近快速崛起的大型企业,ElevenLabs能给我们带来哪些值得借鉴的经验?
And so I'm curious what lessons we should all be learning from ElevenLabs as a large business recently built.
因此,没有几十年来我们一贯做法的包袱。
And so without the baggage of decades of, you know, how we've always done it.
是的。
Yeah.
对。
Yeah.
我们始于2022年,那一年的两大热门话题是加密货币和元宇宙。
We started 2022, which is a year when the two topics of the day were crypto and metaverse.
就在那之前,当然,AI热潮开始了。
So just before and then, of course, AI-floor started.
规模。
Scale.
没错。
Exactly.
没错。
Exactly.
但我们有幸在这一切发生的过程中一路成长起来。
But we could, like, had the privilege of, like, kind of scaling through the world when it was all happening.
对我们来说,Woodworks,而且我们真的相信这将是未来最重要的部分。
For us, Woodworks, and and we, like, really believe in that being the the the big part of the future.
首先是小团队。
The first is small teams.
嗯。
Mhmm.
保持团队规模小且极度扁平。
Like, keeping the teams small and super flat.
比如,我和我的联合创始人每人直接管理超过15名下属,和他们一起工作。
So, like, can you have both me and my cofounder will have over 15 direct reports each that that we'll work with.
而这些人中的大多数也会拥有同样规模的直接下属,并且
And most of those people will have that same scale of direct reports and
好的。
Okay.
关于 Bayt 播客
Bayt 提供中文+原文双语音频和字幕,帮助你打破语言障碍,轻松听懂全球优质播客。