AWS S3的构建原理

对于大多数人来说，除了像S3这样少数领域的从业者，可能都不太熟悉这个概念。

For those of us who are not as familiar with this, which will be most people outside of very few domains like S3.

是的，S3到处都在用自动化推理，这点没得说。

Yeah, it's, I mean, S3 uses automated reasoning all over the place, okay?

自动化推理是计算机科学里的一个专业分支。

And automated reasoning is a specialized form of computer science, okay?

你可以想象一下，要是计算机科学和数学结了婚，生了孩子，那孩子就是自动化推理。

And, girly, if you if you kinda think about if computer science and math got married and had kids, right?

没错，就是自动化推理。

It would be automated reasoning.

你指的是形式化方法吗？

Is formal methods?

对，就是基于形式化方法的，我学过计算机科学，所以这个挺有意思的。

Or That's based on formal exactly I mean, I've I studied computer science, so, yeah, that's fun.

你们实际用的是严谨的形式化方法。

It's actually proper formal methods that you're using.

没错。

That is right.

我们在S3的很多地方都用到了形式化方法，但最早应用的一个地方，就是为了确保每个请求都能实现强一致性。

And we use formal methods in many different places in S3, but one of the first places that we adopted was for us to feel good that we actually had delivered strong consistency across every request.

所以我们做了证明，对吧？

So what we did is we proofed it, right?

我们实际上构建了一个完整的证明过程。

We basically built a proof for it.

然后我们把这个证明集成到了我之前提到的索引区域里，每次提交代码时都会检查。

And then we incorporated our proof on check ins into this index area that I talked about, right?

那里有缓存层，也有存储子层，都是索引功能的一部分。

Where you have your caching and then you have your storage sub layers of the index capabilities.

现在，任何人在修改我们索引子系统的代码时，只要涉及一致性相关的路径，

And so when somebody, anybody is working on our index subsystem now, and then they're checking in code into the code paths that are being used for consistency.

我们都会通过形式化方法验证，确保一致性模型没有倒退。

We are proofing through formal methods that we haven't regressed our consistency model.

你能大概给我们讲一下吗？

And can you just give us a rough idea?

我之前学过的形式化方法，都挺抽象的。

Because the formal methods that I have have studied, they they were pretty abstract.

比如设计语言，怎么定义各种操作符之类的。

The things like designing languages, how to have, like, the the the different operators.

当然，里面也涉及不少数学内容。

And, of course, there are some maths involved as well.

但你们用的形式化方法，具体是针对哪些基本元素？比如服务器、网络这些，还有数据流的模型？

But what are are they, like, primitives, like servers, network, etcetera, and models being built, data flows?

你能用一个最简单的例子，大概讲讲在 S3 里是怎么做形式化证明的吗？

Like, how how can I imagine a simple proof of of something inside S3 roughly at a really high level?

是的。

Yeah.

如果你回到证明的根本目的，其实就是为了确认某件事是正确的，对吧？

I mean if you go back to the fundamental notion of a proof, you are proving something to be correct, okay?

我们使用这些证明的地方之一是一致性，我们通过覆盖所有可能的组合来构建证明，确保一致性模型是正确的。

And so the places that we use these proofs, we use them in consistency, where we built a proof across all the different combinatorics to make sure that the consistency model is correct.

在跨区域复制时，我们也用它来证明数据从一个区域传到另一个区域确实已经抵达。

We use it in cross region replication to prove that a replication of data from one region to another arrived.

在S3的其他多个地方，我们也用它来验证API的正确性。

And we use it in different places within S3 to prove the correctness of API.

在所有这些情况下，我们都会谈到持久性、可用性和成本，但对我们而言， correctness（正确性）同样是一个核心的设计原则。

In all of these cases, we talk about durability, we talk about availability, we talk about cost, but just as strong of a principle, a design principle for us across S3 is correctness.

这里的正确性，指的是API请求或操作本身是否准确无误。

It's a correctness of a thing, an API request, an operation as it were.

对我们来说，关键在于：证明只需做一次，但每次提交代码、每次处理请求时都要重新验证，这样才能确保你真正做到了你承诺要做的事。

And the key thing for us too, is that you wanna just proof it once, you wanna proof it on every single check-in and you wanna proof it on every single request so you can verify, you can validate and verify that you are doing in fact what you say you do.

我认为，在某个规模下，数学必须成为我们的保障，对吧？

And I think for us, at a certain scale, math has to save you, right?

因为在足够大的规模下，你不可能穷尽每一个边缘情况的组合，但数学可以。

Because at a certain scale, you can't do all the combinatorics of every single edge case, but math.

在 S3 的规模下，数学真的能帮上大忙。

Math can save you and help you on this at S3 scale.

所以我们把形式化方法用在了 S3 的很多地方。

And so we use formal methods in many different places of S3.

我们还有一些研究论文。

We have some research papers too.

我可以发你几篇论文的链接，里面详细讲了这些内容。

I can send you some links to some research papers where you talk about them.

是的。

Yeah.

一定要发给我，我们会把链接放到节目笔记里，方便大家查看，我觉得这些内容特别有意思。

Please please do, and we will and we will put it in the show notes below so anyone can check it out because I think it's it's really interesting.

我觉得很多初创公司，甚至基础设施类的初创公司，还很少关注形式化方法，但听到你们一直在做这种持续验证，我感觉特别安心。

I feel formal methods are not really a thing in a lot of startups and even infrastructure startups yet, but it sounds very reassuring to me to actually have an ongoing proof of that.

说到这个，我想问一个相关的问题——关于持久性。

And speaking of which, I want to ask about one thing that is related to this, durability.

Amazon S3 的持久性承诺非常高，我记得是十一个九，我一开始还特意查了两遍，因为在后端系统里，提到三个九的可用性就已经很厉害了，四个九的可用性都很难实现，再往上成本就飙升了，我从来没听说过有哪个系统敢承诺十一个九的持久性。

Amazon S3 has very very high durability promises, I think it's 11 nines, which I had to do a double check on because in back end systems whenever you say three nines it's like, when you say four nines of availability, we're not talking durability, availability four nines is already hard to achieve and beyond that it just gets very expensive and I have never heard of 11 nines of durability.

不过这里说的是持久性，不是可用性。

Now this is durability and not availability.

我之前公开分享这个数据时，有人问了我一个问题，我自己也在想：你怎么能证明这一点？不只是在理论上，而是你们现在存储了五千亿亿个对象，按这个规模，光是靠概率，理论上也该丢掉一些数据了，你们怎么确保没丢呢？

One question question that I I got when I when I shared this that publicly, what people one thing people were asking and I was also thinking, how can you prove that, not just in a formal way, but you're now storing as you said 500,000,000,000,000 objects, which is now large enough that just by this durability promise, you should be you might be losing some of them.

你们会不会实际去检查真实数据，而不仅仅是靠理论证明？

Do you actually, like, validate it on the actual data as well on outside of the proof?

因为我知道理论证明里会假设硬件故障率，但这些假设未必完全符合现实。

Because I assume in the proof you will have assumptions on hardware failure rate, which might or might not be not be true.

所以我想问的是，在 Amazon S3 这个层面，你们怎么验证自己是否真的兑现了持久性承诺？具体是怎么做的？

So my question is that at Amazon S3 level, when you are able to look at the, are we living up to, for example, our durability promise, how do you go about that?

你们发现了什么结果？

And what are your findings?

是的。

Yeah.

我们花了大量时间讨论索引子系统，因为它是跟一致性直接相关的部分。

So we just spend a lot of time talking about our index subsystem because that is the subsystem that is related to consistency.

但说到持久性，其实你需要从S3架构的各个层面来看，而我们最关注的是存储层。

But when you think about durability, I mean, you think about it all, you know, on different levels of the S3 stack, but we really think about it in the storage layer.

在存储层面上，你面对的是这套设计所承诺的可靠性保障。

And so if you think about it in the storage layer, you have this design, this promise of the design care.

而这个保障背后，是一系列技术的组合。

And underneath that is a combination of things.

它不仅是软件层面的，还包括我们在S3中所有数据的物理分布方式。

It's software, but it's also the physical layout of where our data is across everything that we have in S3.

我之前提到过，我们的硬盘和服务器被安置在机架上，机架又放在数据中心里，我们总共有上千万块硬盘。

And, you know, one of the things that I talked about is that we have disks and servers which sit in racks, which sit in buildings, and we have tens of millions of these hard drives.

我们有数百万台服务器，分布在38个区域的120个可用区中。

We have millions of servers, and we have a 120 availability zones across 38 regions.

是的。

Yeah.

一个可用区指的是两个物理上完全独立的地点，只是为了

And one availability zone is like, two availability zones are two physically separate locations, just to

说清楚一点。

be clear.

它们在物理上是分开的，有时候相隔还挺远。

They're physically separate, and sometimes they're a ways away from each other.

在我们的一些区域里，可用区不止三个，我是说，这些可用区为我们提供了不同的域，也就是故障域。

And in some of our regions, we have more than three avail I mean, the availabilities loans gives us a different domain, a fault domain.

如果要谈持久性，对我们来说最重要的其实是审计系统。

If I were to think about durability, I think the most important thing for us is our auditors.

你想想这个分布式系统，我们之前聊过上传和下载，后台其实有成百上千个微服务，每个都专门负责一两件事，明白吗？

So you think about a distributed system, we talked about the put and the get, we have many, many, many microservices that are all doing one or two things very well in the background, okay?

所以我们有各种各样的健康检查机制。

And so we have many different varieties of health checks.

但同时我们也有修复系统和审计系统。

But we also have repair systems and we have auditor systems.

我们的审计系统会遍历整个集群，逐字节检查数据。

And our auditor systems go and they inspect every single bite across our whole fleet.

一旦发现需要修复的迹象，修复系统就会自动介入。

And if there are signs that there is repair needed, another repair system will come in place.

在分布式系统中，这些全部都是微服务，彼此松散协作，通过标准化的接口进行通信。

And these are all, in the world of distributed systems, these are all microservices working together, loosely correlated, but communicating through well known interfaces.

目前，这套系统已经包含超过两百个微服务，全部都隐藏在一个S3区域端点后面，其中相当一部分专门负责数据持久性保障。

And so that collection of systems, which are over 200 microservices now, that all sit behind one S3 regional endpoint, and a fair number of those subsystems, those microservices, are all dedicated to the notion of durability.

它们会去检查、记录并反馈结果。

So, will go and check and log and report back.

所以我的理解是，在S3的任何时间段里，都有人或系统能回答过去一周、一个月、一年的数据持久性情况，对吧？

So, do I understand correctly that in any given timeframe at S3, someone or some people or some systems can actually answer the question of what is our durability the past week, month, year, and so on.

是的。

Yes.

很好。

Okay, great.

所以，你们实际上能验证自己的持久性承诺，确认数学模型是否准确。

So, you can actually verify your durability promise that check if the math is mapping.

是的。

Yes.

我们的设计之一就是，无论何时——就像今天你我这场对话中，服务器都在不断出故障，因为服务器本来就会坏。

And, you know, part of our design is that at any given moment, in this conversation that you and I have had, just today, we're having servers fail because servers fail.

因此，我们在 S3 中构建的一切，都是基于一个前提：服务器会出故障。

And so what we are building and what we've built in S3 is an assumption that servers fail.

所以，我们的很多系统一直在做一件事：首先检查任何故障可能影响哪个节点，会对某个字节造成什么影响，然后自动触发相应的修复机制。

And so a lot of our systems are always, know, first of all, they're checking to see, you know, where any failure might hit an individual node, how does it affect a certain byte, what repair needs to automatically kick in place.

这套系统其实一直在后台默默运行，而它和用户发起的读取或写入操作完全是两回事。

And so this system is constantly moving behind the scenes, if you will, while And that is a completely separate thing from the get in the put.

用户看到的只是读取和写入操作。

The get in the put is what the customer sees.

在底层，还有一个庞大的世界，专门负责如何在大规模下管理每一个字节。

There's this whole universe under the hood of how do we manage the business of bytes at scale.

我只是在想，对我们这些构建中等规模系统的工程师来说，虽然比起S3可能算小了，但一旦出故障，影响还是挺大的。

I'm just thinking because for a lot of us engineers who are building like moderately sized systems, I'll say compared compared to s three, they can already be big, but a failure is is a big deal.

比如，机器又宕机了。

Like, you know, like a a machine going down again.

我有个小副项目，存储空间满了，就开始报错。

I have a small side project and my storage filled up and I started to get errors.

这事儿挺严重的，因为我很少遇到这种情况。

And this is a big deal because it rarely happens to me.

这三年来还是第一次。

This is the first time it happened in three years.

是的。

Yeah.

但我明白，在你们这种规模的业务里，或者在S3这种级别，这种事每天都在发生。

But I understand that in your business or or when you work at S3 scale, this is just every day.

问题不是会不会发生，而是多久发生一次，你们怎么应对。

And and the question is not when, it's just how often, how do you deal with it.

这简直就是另一个世界。

I guess it's it's a different world.

确实不一样，关键是要真正考虑到关联性故障。

It is a different world, and the the trick is to really think about correlated failure.

明白吗？

Okay?

不管在什么规模下考虑可用性，真正会拖你后腿的都是关联性故障。

So if you're thinking about availability at any scale, it's the correlated failure that'll get you.

那什么是关联性故障？

And what is a correlated failure?

嗯。

Okay.

这其实特别有意思。

So that's super interesting.

比如我之前说的最终一致性，还有投票机制，你记得吧？

So if you think about what I talked about with, you know, eventual consistency, we talked about quorum, okay?

投票机制可以容忍单个节点故障，但如果所有节点都出问题了，比如它们都在同一个可用区或者同一机架上，那你的底层存储可用性就真要出大问题了，明白吗？

And quorum is okay for one node to fail, but if all of the nodes go south, for example, and they're in the same availability zone or on the same rack, then you're really gonna be messing with your availability of the underlying storage, okay?

你刚才说的缓存容错能力就全没了，因为它们全都一块儿挂了。

You just lost your failure allowance that I talked about with the cache because they all fail together.

所以，在考虑系统可用性时，相关性故障是绝对不能忽视的关键问题。

And so like a correlated failure is an incredibly important thing to think about when you're thinking about availability.

因此，在设计时应对相关性故障，我们需要思考的是：这些工作负载是如何暴露在不同层级的故障风险下的？

And so when we're designing around correlated failures, the thing is that we have to think about is like, do we expose or how are those workloads exposed to different levels of failure?

当你通过 PUT 请求上传一个对象到 S3 时，我们会把这个对象复制多份，明白吗？

So when you upload an object to S3 with a PUT, we replicate that object, okay?

我们不会只存一份，而是会存好多份。

We don't just store one copy of it, we store it many times.

这种复制机制非常重要。

And that replication is important.

它对数据持久性很关键，但更有趣的是，它对可用性也同样重要。

It's important for durability, but what's interesting about it, it's also important for availability.

因为如果某个关联故障区域出问题了，比如整个可用区瘫痪，数据在其他地方还是有备份的。

Because if any of those correlated failure domains fail, like if a whole AZ fails, there's still a copy somewhere else.

就算某个可用区、机架或者服务器出了故障，数据依然能在别处访问到，明白吧？

And the data is still available somewhere even though an availability zone has failed or a rack has failed or a server has failed or so forth, okay?

所以，无论是物理设施还是逻辑架构，如何应对关联性故障，这对S3的可用性和持久性都至关重要。

And so that idea of how do you manage and design around correlated failures with both our physical infrastructure, as well as our logical infrastructure, is super important for S3 for both availability and durability.

我们还会考虑一种叫崩溃一致性的机制。

We also do things like we think about something called crash consistency.

说实话，我现在都能没完没了地讲下去，你得随时喊停才行。

I mean, currently, you can tell I can go on and on about this, so you just have to stop me.

不用。

No.

但这些内容才真正有意思。

But this is the interesting stuff.

好。

Alright.

所以，崩溃一致性这个概念就是，你构建的任何系统，在遭遇突然中断后，都必须能恢复到一个一致的状态。

So the whole idea of crash consistency is that a system, any system that you build, it should always return to a consistent state after a fail stop failure.

如果你能思考系统在故障发生时可能到达的所有状态，始终假设故障一定会出现，那自然也会默认一致性和可用性必须保证，这样你就能设计出各个微服务，让它们在S3这样的底层能力上协同工作。

And if you can like do things like reason about the set of states that a system can reach in the presence of failure, and you just always assume the presence of failure, then you also assume the presence of consistency and availability, then you just design all of these different microservices to all work together in an underlying capability like S3.

这正是我们工程师每天在做的事情。

But that's what our engineers do.

他们天天琢磨崩溃一致性的问题。

They think about like crash consistency.

他们也时刻关注相关性故障。

They think about correlated failures.

他们还会考虑容错机制、缓存这些事。

You know, they think about failure allowances and caches.

对吧？

Right?

我们工程师每天投入的，正是这些深入的分布式系统工作。

And it's it's all that deep distributed system work that that our engineers come in every day to work on.

咱们聊聊你是怎么看待容错设计的？

Can we talk about how you think about failure allowances?

因为其实其他公司也有错误预算这个概念。

Because again, there there is a concept of error budgets outside in other companies as well.

我觉得那玩意儿管得比较松，而容错设计可是你们的核心业务。

I feel it's a bit like loosely handled, whereas I feel this is kind of your bread and butter.

那到底什么是容错设计？你们怎么衡量它？要是超支了或者用过头了，又该怎么办？

So what is a failure allowance and how do you measure it, and what do you do if you if you overstep it or overspend it?

对。

Yeah.

我的意思是，我觉得故障容忍这个概念就是必须得有，就像你不得不接受它一样。

I mean, I think that the idea of a failure allowance is want to have it, like you have to have it.

如果你假设系统永远不会出故障，那你实际遇到的日子会非常难熬。所以我们得考虑故障容忍，但最重要的，我们先聊聊缓存里的故障容忍问题。

If you assume for enough, you know, that you'll never have a failure, you will you'll actually have a very bad day for your And so we account for failure allowances, but the most important thing is, let's just talk about the failure allowance in our cache.

那我们要怎么管理它呢？

So how do we manage that?

我们是通过合理配置容量来确保你根本察觉不到它的存在。

Well, we manage it in such a way that you'll never experience it, because we size it, right?

当你在调整缓存大小时，必须确保底层能力和硬件始终可用，就像我之前提到的那些分布式子系统和微服务，它们都在后台默默运行。

And if you're sizing the cache and you're making sure that the underlying capabilities and the hardware are always there, and we have, like I talked about those distributed subsystems, those microservices that are all in our operating under the hood.

我们有大量这样的服务，专门用来监控各项指标。

We have a ton of them that do nothing but just track metrics.

对吧？

Right?

我们的缓存大小完全依赖于这些指标和系统健康状况。

And like, you know, the sizing of our cache is all related to the metrics and then Health.

还有底层系统的规模。

The size of our underlying system.

所有的指标。

All the metrics.

没错。

Yeah.

是的

Yeah.

没错

That's right.

在S3上运行的一个巨大优势在于，我们的系统规模庞大，有着层层叠叠的架构，你知道的。

And so one of the really big benefits of running on S3 is because our system is so huge, you have these massive, you know, layers, right?

这些庞大的层级都在处理相关故障和容错机制。

And the massive layers are all managing things correlated failures and failure allowances.

由于S3的规模如此之大，任何运行在其上的应用都能从中受益。

And because they are so huge at the scale of S3, any application that's sitting on top of S3 gets the benefit of it.

咱们先暂时放下S3，聊一聊我首次组织的一个特别活动——与Statsig联合主办的《务实者峰会》。

Let's take a break a minute from S3 to talk about a one of a kind event I'm organizing for the first time, The Pragmatic Summit in partnership with Statsig.

你有没有想过，能见到《务实者工程师》播客里的那些特邀嘉宾，还有来自前沿科技公司的专家，一起探讨在AI新时代下，软件开发中哪些做法有效、哪些行不通？

Have you ever wanted to meet standout guests from the Pragmatic Engineer podcast plus folks from cutting edge tech companies and learn about what works and what doesn't in building software in this new age of AI.

欢迎二月来旧金山，参加这场为期一天的特别线下活动。

Come join me on live in February in San Francisco for a very special one day event.

Pragmatic Summit 汇聚了众多行业传奇和往期播客嘉宾，比如劳拉·塔乔、肯特·贝克、西蒙·威利森、黄智、马丁·福勒，还有很多人。

The Pragmatic Summit features industry legends and past podcast guests like Laura Tacho, Kent Beck, Simon Willison, Chip Huynh, Martin Fowler, and many others.

我们还会分享 Cursor、Linear、OpenAI、Ramp 等团队如何打造前沿产品的内部故事。

We'll also have insider stories on how engineering teams like Cursor, Linear, OpenAI, Ramp, and others build cutting edge products.

现场还设有圆桌讨论，精心安排的观众群体都彼此志同道合，大家乐于交流互动，我希望这能让这个活动格外特别。

We'll also have roundtables and carefully curated audience where everyone and everyone is interested to meet and chat with, something I'm hoping will make this event extra special.

席位有限，想参加的话可以去 pragmaticssummit.com 申请。

Seats are limited, and you can apply to attend at pragmaticssummit.com.

所有演讲都会录制并分享，付费订阅者还能提前观看，这也是对你们额外支持的一点心意。

Talks will be recorded and shared, and paid subscribers will get early access afterwards as well as a thank you for your additional support.

我期待在现场见到你们很多人，对这个活动我真的很兴奋。

I hope to meet many of you there, and I am so excited about this event.

现在我们回到 S3，聊聊这个服务的庞大规模。

And now let's jump back to S3 and the massive scale of the service.

为了让大家感受一下在这样的组织里当工程师、当技术主管是什么体验，我读一段杰出工程师安迪·沃菲尔德说过的话——我直接引述他的原话：我刚入行的时候，对构建大规模商业软件有个很天真的想法，觉得无非就是写代码而已。

To get a sense of what what what the reality is like working as an engineer, an engineering leader inside an organization like this, I read a quote from distinguished engineer Andy Warfield, who said, I'm just quoting what he said, early in my career, I had this sort of naive view about what it meant to build large scale commercial software, that it was basically just code.

我在做S3的时候很快就意识到，代码根本离不开团队的组织记忆和运维实践，还有整个系统的规模。

The thing I realized very quickly working on S3 was that the code was inseparable from the organizational memory and the operational practices and you know the scale and and the scale of the system.

你已经在S3待了十多年了，你是怎么看待这个庞然大物的？几百个微服务，数据量大到难以想象——除非你想象硬盘堆到空间站那么高。

Since you've you've now been more than a decade in S3, how do you think of this this beast, this this really complex system, hundreds of microservices, data that is hard to fathom, you know, unless you think of the hard drive stacking all the way to the space station?

你们这些工程师是怎么应对这种复杂性的？

And how do you and engineers kind of wrangle this?

说实话，这确实有点让人望而生畏。

Because it does feel a bit intimidating, I'm not gonna lie.

我觉得这一切归根结底还是靠团队的文化和那份执着。

Well, I think so much of this just comes back to the culture and the commitment on the team.

我确实在S3工作了很长时间，对这个团队的工程师们充满敬意。

And yeah, I've worked on S3 for a very long time now, and I have such deep respect for the engineering community on S3.

说实话，这不仅适用于S3，我们整个数据与分析平台的每个服务都是这样——S3的工程师每天都在为数据的持久性、可用性和一致性全力以赴。

And honestly, I mean, this is true for all of the services in our data and analytics stack, but we have engineers on S3 and they come in every single day with this deep commitment to the durability and availability and the consistency of your bite.

所以我们讨论的问题特别有意思，因为团队里有很多人，甚至有些刚毕业不久。

And so the type of conversations that we have are so interesting because we have people, and really, you know, these are people who are early out of school.

这些人一直在做S3的相关工作。

They're people who've been working on S3.

我们有一些工程师，已经在S3上干了十五年。

We have engineers who've been working on S3 for fifteen years.

从S3的创意和早期开发到现在，中间各种情况都有，你得面对一种张力：一方面，你得对底层代码保持极度谨慎，对吧？

Everything in between, creativity and the invention of S3, like you have this tension, which is like, on one side you're like, you have to be very conservative with tree, right?

另一方面，我们有个核心工程原则，叫‘尊重过往’。

And on the other hand, I mean, we have this principle engineering tenant called respect what came before.

这是亚马逊的工程准则：如果某个东西已经稳定运行了很多年，你就得尊重它。

And that's an Amazon engineering tenant, which is if it has worked for many, many years, you have to respect that.

但同时，还有另一个原则。

But then there's also this tenant.

这两个原则彼此有点冲突，但也正是这种张力让工作特别有意思。

These two tenants are a little bit in tension with each other, which is kinda what makes it so fun.

亚马逊的另一个工程原则叫‘技术无畏’。

Amazon engineering tenant is called be technically fearless.

我相信，S3 的工程师在尊重既有成果方面做得特别出色。

And I believe that the S3 engineers are just amazing at this, at respecting what came before.

因为当我们为 S3 添加新功能时，必须保留它的核心特性——比如高持久性、高可用性这些关键表现。

Because if we build new capabilities in S3, we have to maintain the properties, the traits of S3, which is adjust works, you get that durability, availability, etcetera.

但同时，我们也必须保持技术上的无畏精神，因为能够支持条件查询、原生集成 Iceberg 或向量数据，意味着我们在原有存储基础上不断拓展，帮助客户构建当下和未来所需的各类应用。

But at the same time, we have to be technically fearless because our ability to go into the world of conditionals, our ability to go into the world of native support for Iceberg or for Vectors means that we are extending this this foundation of storage in a way that helps customers build whatever application they need now and in the future.

所以，当我想起我们的 S3 工程团队时，我觉得他们每天都在践行这种双重精神。

And so that combination of the two things, that is sort of when I think about our S3 engineering team, I think they come in every day and they embody that.

回到 S3 从非结构化到结构化数据的演进，你之前提到，Hadoop 和数据仓库曾是客户在 S3 上的主要用例，后来你们发现很多大客户都在这么用，于是你们自己也开始往结构化数据方向发展。

Now, going back to the evolution of S3 from unstructured to structured data, you were mentioning how Hadoop, the data warehouse, what what was a big use case where customers started to use it on top of S3, and then at S3 you noticed, you're like what a lot of customers or some of your biggest customers doing, and then you kind of built it yourself with more structured data.

接着就有了 S3 表，再后来是向量功能。

And then S3 tables came along and then Vectors.

你能再详细说说 S3 是怎么一步步演进的吗？

Would you mind sharing a little bit more on on how you evolve S3?

因为这也是很多人关心的问题——S3 到底算完成了吗？

Because this was another question that when I asked people about what they like to know about S3, one of the question was like like, is it done?

它完成了吗？

Is is it finished?

还是说它还在不断演进？

Or or is it still evolving?

大家都觉得 S3 已经能存任何东西了，对吧？不管是哪种对象、哪种二进制数据，那还有什么新东西可加的呢？

Because there is this notion that S3 can store anything already, right, like any object, any blob, what new thing is there?

但事实上，我们一直在推出很多新功能。

And yet, we have a lot of new things.

是的。

Yeah.

如果你回过头看看，比如 Parquet 这种格式的兴起，对吧？

And if you kinda go back in time a little bit and you think about, you know, the rise of Parquet, okay?

Parquet 数据在 S3 上开始流行，大概是 2020 年左右。

So the rise of Parquet data in S3 started about 2020.

从那以后，越来越多的人开始把表格数据存到 S3 里。

And we started to see more and more people store their tabular data in S3.

而Iceberg提供的，其实是Hive的替代方案。

And if you think about what Iceberg provided, it provided a replacement for Hive.

明白吗？

Okay?

想想Hive和Hadoop，Hive本质上是让文件系统能访问S3里的非结构化数据。

So if you think about Hive and Hadoop, Hive was basically giving your file system access into S3 unstructured storage.

而Iceberg则为你的Parquet数据提供了类似的表格化访问能力，还包括了压缩和所有相关的表维护功能。

Iceberg is giving that Iceberg that tabular access, including the compaction and all the table maintenance that goes along with it, into your Parquet data.

我甚至觉得，未来所有的结构化数据都会存放在S3里。

And I actually think that the world's data for tabular data is going to live in the future in S3.

就拿上周Supabase发布的功能来说吧，他们宣布自家的PostgreSQL数据库现在可以直接把写入操作发到S3的表格里。

And if you just think about the launch that, for example, Supabase did last week, Supabase announced that their Postgres database is now gonna Is just gonna do secondary rights directly into an S3 table.

就像他们之前为向量搜索开发的PostgreSQL扩展一样，现在也会直接对接S3中的向量数据。

Just like their Postgres extension for vector is going to integrate directly with S3 Vectors.

如果整个数据库世界、整个数据源体系，都直接把数据写进S3表格，那对整个数据生态意味着什么？

And so if the world of database, if the world is data as a source, if you will, goes directly into an S3 table, what does that mean for the world's data?

好的，SQL 作为数据领域的通用语言，这一点我们都清楚。

Okay, so SQL, as we know, is a lingua franca of data.

而且全球的大型语言模型都训练于数十年积累的 SQL 数据。

And the world's LLMs have all been trained on decades of SQL.

因此，

And therefore,

所有这些都离不开它。

all it.

还有 Python。

Python.

SQL 和 Python。

SQL and Python.

Python 以及现有的各种工具。

Python and the stuff that's already out there.

所以你想想，我们现在有大量 AWS 客户对 S3 的 API 非常熟悉，这个 API 本身也很简单，但现在你可以直接用 SQL 来操作 S3 里的数据了。

And so if you think about this, you know, we have many, many AWS customers who know the S3 API pretty darn well by this point, it's pretty simple API, but now you have the ability to interact with data in S3 through SQL.

这意味着你不需要去开发云应用，也不用懂S3，只要会SQL就够了。

And what that means is that you don't have to be, you know, somebody who's building cloud applications or know S3, you just need to know SQL.

这指的是S3表，对吧？

And this is with S3 tables, right?

就是S3表。

With S3 tables.

你可以直接对S3表写SQL语句。

And so you can just write SQL into an S3 table.

不管是AI代理还是人类，你都是把数据的通用语言，作为S3表的内置功能引入进来了。

And whether you're an AI agent or a human, right, you're introducing the lingua franca of data as a native property of S3 with S3 tables.

我觉得未来几年，这种做法一定会迅速普及。

And I think you're just gonna see that take off in the upcoming years.

你们最新推出的是S3 Vectors。

And your latest launch is S3 Vectors.

能聊聊打造像向量这样的新数据基础组件，需要哪些条件吗？

Can can you share a little bit what it takes to build a new data primitive like Vectors?

幕后具体是怎么运作的？开发周期多长？各个部分是怎么衔接的？在推出这样的功能时，遇到了哪些工程上的挑战？

Just just behind the scenes, how long it takes, how the seams comes together, and and maybe what are some engineering challenges of launching some something like this.

我们说的向量，其实你用大模型的时候都会生成嵌入向量，它本质上就是一个向量，得存起来，还得能搜索，以前都得靠专门的向量数据库，还得加各种向量索引之类的。

Again, we're talking about Vectors, right, so, like, you can use use embeddings whenever you have LMs, you create an embedding, it's a vector, you wanna store that somewhere, you will need to search on it, there's specialized vector databases, there's specialized vector additions, etcetera.

所以我猜，S3 向量就是很好地支持了这些功能？

So, I'm assuming this is the function that S3 Vectors supports very nicely.

是的，现在很多人用向量数据库，就像以前很多人把表格数据都塞进传统数据库里一样。

Yeah, and you know, I mean, today, a lot of customers use vector databases, just like back in the day, a lot of people put their tabular data in just databases, okay?

他们只是借数据库的结构来查询数据，但其实根本不需要数据库，只是因为没别的选择才这么干。后来 S3 出现了。

And they just use the structure of the database in order to take advantage of being able to query their data, but they didn't really need to use a database, they just put it in a database and then S3 came along.

我们借助 Apache Parquet 这类开放格式，把结构化数据直接存到 S3 里，就是这么做的。

And then we introduced this way, you know, with the help of open formats like Apache Parquet, and being able to store that structured data in S3.

现在我们对向量也是同样的思路。

That's kinda what we're doing with vectors right now, okay?

你想想，向量本质上是一种定制化的数据类型。

And if you think about vectors, vectors are basically a bespoke data type.

说到底，向量就是一个特别长的数字列表。

A vector at the end of the day is a very, very long list of numbers.

向量其实早就存在了，也用在向量数据库里很久了，但最近几年，随着你提到的嵌入模型兴起，它们才真正开始在数据领域大放异彩。

And vectors have been around for a long time and they've been in vector databases for a while, but they really kinda took off in people's data worlds in the last couple of years with the rise of, as you said, the embedding models, okay?

如果退一步看，数据领域有个特别有意思的地方：你得先了解你的数据，才能真正理解它，对吧？

And so if you take a step back and you think about one of the great ironies of data, it is that you have to know your data to know your data, right?

你得知道数据的结构，知道字段类型，知道数据存哪儿。

You have to know what your schema is, you have to know what the data types are, you have to know where it is.

随着数据湖变成数据海洋，你越来越难搞清楚自己手里到底有什么数据。

And as these data lakes become data oceans, you have this situation where it gets harder and harder to know what's in your data.

对吧？

Right?

而嵌入模型的妙处就在于，它能帮你理解数据，让你不用自己去琢磨数据的含义。

And the beautiful thing about embeddings is that embedding models will understand your data so that you don't have to understand your data.

这些嵌入模型把对数据的语义理解，最终转化成了一种向量形式。

And the format that these embedding models puts this semantic understanding of your data is in fact a vector.

当我们和客户交流时，他们对这些嵌入模型越来越强大感到非常兴奋，都想把越来越多的语义理解应用到他们的底层数据上，不管是非结构化数据，还是存在存储里的数据。

And so when we talk to customers and they're so excited about how these embedding models are getting better and better, they want to apply more and more basically semantic understanding to their underlying data, whether it's unstructured or that they have in storage.

所以他们希望存储数十亿个向量。

And so they kinda wanna store billions of vectors.

不过我插一句，你说他们想理解数据，我纠正一下——举个例子，你有一堆文本数据，或者一些图像数据，很多客户、团队都希望写查询，比如‘能不能找一张看起来像小狗的图？’或者‘能不能找到一篇包含这些内容的文章？’

But but just to say, when they wanna you say they wanna understand, just correct me if I'm but hypothetically, you have a bunch of text data or maybe some image data, and you're saying that a lot a lot of people, customers, teams, they would like to write queries to say like, hey, can you find an image that looks like a puppy, or can you find an article that contains this or this?

而嵌入模型确实很擅长做这些事，但你得实际去生成这些嵌入，搭建整个系统等等。

And and embeddings are, as we know, are great for that, but then you need to actually create the embedding, build the system, etcetera.

对吧？

Right?

没错。

Yeah.

你说得一点都没错。

And like exactly what you're saying.

你想啊，向量能做的事太多了，想想一家公司拥有的所有数据，你在业务中或生活中积累的知识，从来都不是像数据库那样整齐地排成行和列的。

I mean, if you think about what Vectors can do, if you think about all the data that a given company has, your knowledge across your business, or your knowledge across your life isn't organized into rows and columns like a database.

都在PDF里。

It's in PDFs.

也在你的手机里，对吧？

It's in your phone, right?

还在客服的语音录音里，这些录音真实记录了客户对与你互动的感受。

It's in audio customer care recordings, which capture the sentiment of how a customer actually feels about their interaction with you.

还有白板上的内容。

It's whiteboards.

到了今天结束时，这块白板早就写满了想法，而这些想法还散落在几十个系统的文档里。

By the end of this day, this whiteboard is totally filled up with ideas and it's in documents across dozens of systems.

所以不是你没数据，而是你数据多得不得了。

And so it's not that you don't have data, you have tons of data.

但问题是，你很难搞清楚这些不同格式的数据里到底有什么，而AI模型正好能帮上这个忙。

But understanding what data you have across all of those different formats is a real problem and it's one that AI models can help you with.

过去一两年里，这些AI模型的能力提升得飞快，但我们得有个地方来存放数十亿个向量，也就是数十亿份语义关系的理解。

And so the capabilities of those AI models have gotten so much better in the last eighteen to twenty four months, but we needed a place to put billions of vectors, billions of the semantic understanding of relationships.

我们开发S3就是为了这个目的。

And that's what we built S3 for.

最先进的嵌入模型，加上在S3上存储向量的能力，这一点非常关键。

The state of the art embedding models combined with the ability to have vectors across S3 is like a really important part.

它并不是一个数据库。

And it's not a database.

我是说，S3的成本结构和规模本身就适合做向量存储。

I mean, it's the cost structure and scale of just S3, but it's for vector storage.

那我理解得对吗？

And then do I understand that?

你们得从头开发新的底层机制来存储这些数据吗？

Did you need to build new primitives to store this?

我是说，得深入到最底层，搞清楚具体怎么实现，还是说你们只是在现有的对象存储这些基础组件上搭建的？

Like, going down to the metal, figuring out exactly where we do this, or did you build it on top of your existing, you know, like, again, existing primitives of all, blob stories, etcetera?

这其实是一个全新的底层机制。

It's actually a new primitive.

所以我们之前聊过S3表。

And so, you know, we had talked about S3 tables.

S3表是基于对象构建的，因为那些单独的Parquet文件，说到底也就是个对象。

S3 tables is building on objects, because those individual Parquet files, at the end of the day, they're an object.

向量就完全不一样了。

Vectors totally different.

所以对于向量，我们设计了一种全新的数据结构和数据类型。

So with Vectors, we built a new data structure, a new data type.

而且你会发现，当你在高维空间里搜索最近的向量——其实就是向量空间——找最近邻通常特别困难。

And, you know, it turns out that when you're building vectors, searching for the closest vector in a very high dimensional space, which is basically vector space, it's often really hard to find the nearest neighbor.

所以在数据库里，你基本上得把所有向量一个个比对一遍。

And so you basically, in a database, you have to essentially compare every vector in a database.

这通常成本高得吓人。

And that's often like super expensive.

所以我们用S3的做法是，既然不是把所有向量都存在内存里，而是存在庞大的S3集群上，我们还是得保证极低的延迟。

And so what we do in S3 is because we aren't storing all of our vectors in memory, we're storing it on our fleet of S3, very large fleet, we still need to provide a super low latency.

上周发布时，我们的向量空间在热查询下能实现一百毫秒甚至更短的响应时间，这其实已经很快了。

And in our launch last week, we were getting about a hundred milliseconds or less for a warm query to our vector space, which is actually pretty fast.

虽然算不上数据库级别的速度，但已经相当不错了。

It's not database fast, but it's pretty fast.

我们是通过预先计算一批向量邻域来实现的，你可以把它们想象成向量的聚集区。

And the way that we do that is we pre compute a bunch of Think of them as vector neighborhoods, okay?

简单说，就是把相似度高的向量聚成一群，比如都属于同一种狗的向量。

And so it's basically a cluster, a bunch of vectors that are clustered together in similarity, a type of dog as an example.

这些向量邻域都是离线、异步提前算好的，这样你做查询的时候，完全不会影响响应速度。

These vector neighborhoods, if you will, they're computed ahead of time offline, they're computed ahead of time asynchronously so that when you're doing your query, it's not gonna impact your query performance.

每当有新的向量被写入S3，系统就会根据它的位置，把它加入一个或多个已有的向量邻域中。

And then every time a new vector is inserted to S3, the vector gets added to one or more of these vector neighborhoods based on where it's located.

当你在S3向量中执行查询时，系统只需要在小得多的范围内搜索最近的邻域。

And so when you are executing a query on S3 Vectors, there's a much smaller search that's done to find the nearest neighborhoods.

真正加载到高速内存里的，只有相关的向量和它们所属的邻域。

And it's just the vectors and the vector neighborhoods that are loaded from S3 into a fast memory.

我们就是在这里应用最近邻算法的，这样 Warm 查询的响应时间可以低到一百毫秒以内。

That's where we apply the nearest neighbor algorithm and it can result in like really good sub one hundred millisecond query times.

说到 S3 的规模，每个索引最多能支持二十亿个向量。

And so, if you think about the scale for S3, we'll give you up to 2,000,000,000 vectors per index.

想想 S3 向量存储桶的规模——最多能到两万亿个向量，再配上一百毫秒以内的查询响应速度，这彻底打开了你理解数据语义、查询数据的可能性。

You think about the scale of a S3 vector bucket, which is up to 20,000,000,000,000 vectors, and you think about that combined with a hundred milliseconds or less for warm query performance, that just opens up what you can do with creating a semantic understanding of your data and how you can query it.

听起来非常有趣，但也很有挑战性，因为你们从第一天起就得为大规模做准备。

It sounds very interesting and also challenging because you have to build this for scale from day one.

我想这大概就是待在 S3 的好处和麻烦吧——你每次上线的东西，都得为极端规模做好准备，但在这里，这不过是周一的日常。

Guess that's one of the, I guess, benefits and curses of working at S3 that everything that you launch, you need to prepare for what will be extreme data elsewhere, but here it's just Monday.

我们也有 S3 服务的租户。

We have S3 service tenants as well.

其中一个租户，也是我经常说的一句话，我们工程师也这么讲：规模是你的优势。

And one of the tenants, and one phrase that I use all the time and our engineers do too, is scale is to your advantage.

如果你是一名工程师，能时刻记住这一点——你做的任何东西，规模都必须成为你的优势，这会彻底改变你的设计思路。

So if you are an engineer and you think about that, and you think about one of your tenants for anything you build is that scale must be to your advantage, it just changes how you design.

这意味着你不能搞出那种越做大、性能越差或者某个指标越糟的东西。

It means that you can't actually build something where the bigger you get, the worse your performance gets or the worse some attribute gets.

必须从设计上就保证，越做大，性能反而越好。

It has to be constructed so that the bigger you get, the better your performance gets.

S3 越大，跑在它上面的工作负载就越分散、越独立。

The bigger S3 gets, the more decorrelated the workloads are that run on S3.

这正是规模对你有利的绝佳例子。

That is a great example of scale is to your advantage.

所以我们做向量功能时，就像做 S3 里所有东西一样，我们问自己：怎么才能让规模成为我们的优势？

And so when we built vectors, just like we build everything in S3, we ask ourself, how can we build this such that scale is to our advantage?

怎么才能让一百毫秒甚至更短，只是我们追求的性能起点？

How can we build this such that a hundred milliseconds or less is just the start of the performance that we're going after?

怎么才能确保存储的向量越多，S3 向量功能的表现就越出色？

And how can we make sure that the more Vectors we have in storage, the better the traits of S3 for Vector.

我有个关于 S3 局限性的问题。

I have a different question about the limitations of S3.

我看到说 S3 里单个对象最大能存 50 太字节。

I read that the largest object you can store in S3 is 50 terabytes.

那为什么会对单个对象的大小设个上限呢？

Why is there a limit on the largest object?

我的意思是，理论上可以用多块硬盘来存更大的文件，但你们为什么还是定了这个限制？

I mean, I can think we can imagine this will be through either multiple hard drives and so on, but why did you decide to have a limit?

我只是想了解，你们团队当时是怎么考虑的，为什么最终定下这个数值？

You know, I'm just interested in more in the thought process of how the team comes up with, okay, this will be the limit and this is why.

其实这个 50 太字节的限制，已经是当初上线时的十倍了。

I mean, I think, first of all, that limit of 50 terabytes is 10 times greater than what we launched with.

我们刚推出的时候上限是 5 太字节，现在提升到了 50 太字节。

We launched with five terabytes, and now we're 50 terabytes.

有时候我们跟客户说这个数字，他们还会问：我哪会存那么大的文件？

And sometimes we sit and tell customers that, and they go, what am I gonna store that's gonna be 50 terabytes?

我们就回他们：高清视频啊？

And we're like, high resolution video?

对吧？

Right?

所以呢，我觉得

And so, you know, if I think about

单个客户。

Lone customer.

对吧？

Right?

你想想这类问题，比如文件大小限制，我们通常会针对某些使用模式做优化。

And so, you think about this sort of thing, you know, like, if you think about, I don't know, size size limits, generally speaking, we we do try to optimize for certain patterns.

当我们把单个对象的大小提升十倍时，主要就是为了提升底层系统的性能和扩展性。

And when you raise the size of an object by 10 times like we did, we're just optimizing for the performance and scale of the underlying systems.

上周我们也是把批量操作的规模扩大了十倍。

It's like we increased the scale of our batch operations by 10 times last week too.

这么做的目的是，底层系统要适应现在人们普遍使用的新型工作模式。

And the idea behind that is that the underlying systems, we're just optimizing for distributions of work that are the new norm for how people are doing things.

我们会持续不断地调整。

And we'll just keep on changing.

说实话，我们的限制并不多，但我们会一直关注客户在各种工作负载中的使用情况，看看有没有需要改进的地方。

We don't have too many limits, to be honest, but we'll just keep on looking at what customers are doing across the distribution of workloads and seeing if there's something that needs to be changed.

对我们来说，最重要的是，我们和客户聊了很多，他们都说：真的吗？

The big thing for us, again, we did have a lot of conversations with customers and they're like, really?

我根本没有那么多那么大的独立对象。

Like, I don't have that many individual objects that are that big.

但随着摄像头、手机这些东西越来越多，我们看到更大的文件也越来越多，所以我们希望他们在S3里能毫无限制地增长。

But with the increase of, you know, cameras and phones and things like that, we are seeing more larger size objects and we just wanted them to be able to grow unfettered in S3.

那S3是怎么演进的？你们的路线图又发生了什么变化？

And so, how does S3 evolve and and how has the roadmap changed?

到目前为止，我理解的是，你刚才说的每一句话都在讲：客户在这么做、那么做，而你们整天跟数据打交道，自然能看到趋势、数据、文件形态，也经常和客户交流。

Because so far what I picked up is everything that you told me is saying, well, you know, our customers were doing this or that and you obviously hear, you live and breathe data, so you see the patterns, you see stats, you see the objects, you also talk with them.

是只有你们通过和客户沟通，了解他们的痛点、他们在用什么、用得越来越多的是什么，然后才决定去改进——比如调整限制，或者发现需要新增一种数据类型，因为客户已经在上面自己构建数据结构了？还是说，你们也有一个更宏观的愿景和路线图，会主动规划未来要做什么？

Is it only you talking with customers seeing what's happening, what they're struggling with, what they're using more of, and then deciding to improve that, may that be the limits, may that be figuring out we need a new data type because they're now building their own data types on top of it, Or is there also some kind of more, kind of, alright, here's a vision, here's a roadmap of what we'll do?

这是个很好的问题。

It's a great question.

事实上，我们经常讨论的一个话题就是 S3 的一致性，对吧？

And in fact, one of the things that we talk about all the time is the coherency of S3, right?

所以，人们对 S3 有一些一直期待的特性。

And so there are certain things that people always expect from S3.

这些就是 S3 的核心特质。

It's the traits of S3.

比如我们之前提到的持久性和可用性这些方面。

It's the durability availability attributes that we talked about.

为了保证这些，背后其实投入了大量的工程工作，明白吗？

And so a fair amount of engineering goes on under the hood for that, okay?

这些能力中，有些我们今天可能提到了，也可能没来得及细说。

And it's a set of capabilities that, you know, we may or may not have talked about today.

说起来，如果回想一下 2020 年，我觉得自那以后 S3 已经推出了上千项新功能。

In fact, if you think about, I think back to 2020, I think we've launched over a thousand new capabilities since 2020 in S3.

其中一些是我们认为占路线图90%的内容，也就是客户明确提出来的需求，明白吗？

And some of them are what we think of as the 90% of the roadmap, which is what people ask for explicitly, okay?

比如，我们的某些媒体客户希望支持更大的对象尺寸，我们就实现了这个功能。

And so for example, some of our media customers want the bigger object size, and so we delivered that.

还有不少客户大量使用批量操作功能。

We have other customers that do a lot with batch operations.

但我们也有一些功能是自己发明的，因为我们观察客户如何使用数据，然后问自己：我们怎么才能帮他们做到这一点？

But then we have some things that we invent because we look at what customers are doing with the data and we ask ourselves, how can we build that?

向量功能就属于这一类。

Vector kind of falls into that category.

关于向量，当我们审视S3的发展方向时，我们告诉自己：我们完全可以继续让S3成为全球最优秀的数据存储库。

For Vector, when we looked at S3 and how S3 is evolving, we told ourselves like, look, we can continue to make S3 the best repository for data on the planet.

我们一定会做到，一定会。

And we will, we will.

每天都有工程师在为此努力工作。

We have engineers that come in every day working to make that so.

但还有一个关键问题，就是你怎么确保手里的数据真的能用？

But there's this other element of how do you make sure that the data that you have is in fact usable?

怎么确保它能以行业标准的方式使用，比如在我们的表格数据上加一层Iceberg？

And how do you make sure that it's usable in a way that's, you know, industry standard, like that Iceberg layer on top of our tabular data?

但真正能用，是因为现在的AI模型在嵌入技术上太强大了，只要你能把数十亿个向量存下来，AI就能帮你全面理解数据，让你以全新的方式使用它。

But it's usable because AI models have now gotten so good at embeddings that you can have AI give you a symmetric understanding of your data if only you had the cost point of putting billions of vectors into storage so you could actually understand and use your data in a different way.

对我们来说，很多工作其实是退一步看，不只关注客户明确提的需求，更要打破数据使用的成本限制——这正是S3在做的事。

And so for us, a lot of it is kinda taking a step back and looking not just at what customers ask us for, but we want to remove the constraint of a cost of data, which is what we do in S3.

我们还要打破操作数据的限制——S3同样也在解决这个问题。

And we want to remove the constraint of working with your data, which is what we do in S3 too.

如果我们能同时做到这两点，让数据随着业务需求自由扩展，让你能充分释放AI带来的新能力，顺应数据世界的变革，那我们就有了一个清晰的形态。

And when we can do both of those things, if we can make it possible that your data grows as your business needs it, and you can tap into all the capabilities that you're getting with AI and how the world is changing for data, then we have a shape.

我们管这叫产品形态。

We call it a product shape.

然后，我们就有了一个产品形态。

Then we have a product shape.

产品形态。

Product shape.

什么叫产品形态？

What's a product shape?

这有点像一种不断演化的趋势。说到S3，我觉得它就像一个有生命、会呼吸的系统，产品形态在持续变化，但始终围绕着你对S3核心特性的期待，同时又让你能灵活地调整数据的使用方式。

It's sort of like an emerging When I think about S3, I think of it as almost like this living, breathing organism where the shape of the product is evolving, but it's evolving with coherency around what you expect for the traits of S3, but it's evolving in a way that lets you steer into how you wanna use data.

不只是现在怎么用数据，还包括未来打算怎么用。

And how do you wanna use data not just now, but in the future.

我们会根据你对数据的使用需求，持续优化S3的产品形态。

And we will continue to evolve the product shape of S3 based on what you wanna do with data.

在很多方面，我们已经突破了传统对象存储或数据库的界限。

And so in a lot of ways, we're sort of transcending the boundaries of what object storage was, or what a database traditionally was.

现在我们有了表格格式，支持条件查询，正在朝一种全新的形态发展。

Because now we have tabular formats, we have conditionals, and we're evolving into this new shape.

而这一切，最终都独一无二地属于S3。

And it is ultimately uniquely S3.

听起来你这些微服务就像在不断生长，有点像植物或者活物，对吧？

It of sounds like you have all these microservices, it's kind of evolving, almost like a plant or a living organism, no?

没错。

Yes.

我以前确实当过和平队的志愿者，搞林业的。

I am, in fact, a former Peace Corps volunteer from forestry.

所以我常从自然界找比喻来理解事情。

And so, you know, a lot of times I will go back to the natural world for my metaphors.

S3就像是一个有生命的数据仓库，让人能用数据做出以前根本不敢想的事。

And, I mean, S3 is this living, breathing repository of data that lets people do things with data that they never thought possible.

这挺有意思的，因为工程师通常不会把我们搭建的系统跟活的生命体联系起来。

It's just interesting, because I think as engineers, don't often think to relate the systems that we build with a living organisation.

其实系统里有代码，但像你说的，还有人、服务器，还有按固定节奏发生的故障。

When in fact, obviously there's code, but as you said, there's people, there's servers, there's failures that now happen a cadence.

你几乎能预测今天有多少硬盘会出问题，尤其是在你们这种规模下——这会不会是因为系统大到一定程度，自然就出现了这种特性？我跟你聊觉得特别有意思的是，S3内部的工程方式，跟小公司、初创团队完全不一样，那些团队可能处理几TB，最多几PB数据，也就到头了。

You can almost just you you can probably predict how many hard drives are failing today, in fact, at at your scale already, which, again, maybe it's do you think it's because of the scale when things become large enough, they start to have these characteristics because what I find fascinating talking to you is the way engineering works inside of S3 feels very different to how it works inside a smaller organization, your kind of startup, which again does, you know, like terabytes of data or maybe even a few petabytes, but but that's kind of it.

你见过一些这样的团队。

And you've seen some of these organizations.

在这么大的规模下，到底哪些东西会变？

What what changes at at at this large scale?

你觉得是什么让你们和 S3 团队的工作环境显得这么不一样？

What what do you think that makes it it feels pretty different, the the world that you and and the S3 teams work in.

确实如此。

It does.

但为了维持 S3 的特性并让它持续演进，我们得不断回归简化。

But, you know, in order for us to sustain the traits of S3 and to evolve it over time, we have to constantly go back to simplification.

我们的系统确实很复杂，有这么多微服务，但我始终认为，每个微服务都得把一两件事做到极致。

We have a very complex system with all of our different microservices, but I kinda go back to those microservices have to do one or two things really well.

我们必须坚持这一点。

And we have to stay true to that.

否则，分布式系统一旦变得过于复杂，迟早会变得无法维护。

Otherwise, you know, the complexification of a distributed system, you know, it's unmaintainable over time.

对于S3来说，这个理念就是——S3的核心是简单。

And for S3, this concept of, okay, there's a simple in S3.

而S3的简单体现在几个方面。

And the simple in S3 is a couple of things.

首先，是用户模型的简洁：不仅API设计得简单，还能直接用SQL查询S3里的数据，或者接入AI嵌入模型，让数据的语义理解变得特别轻松，根本不用费劲去标注一堆元数据。

One, it's a simplicity of the user model, where not only do you have a simple API, but now you have the simplicity of using SQL with S3 or you have the simplicity of being able to leverage these AI embedding models, which makes semantic understanding of your data so much easier than having to annotate, you know, a whole metadata layer.

所以，这种简洁性已经融入了S3的用户模型。

And so that concept of simplicity is in the user model of S3.

但在底层，如果你参加我们的工程会议，就会听到工程师们一直在讨论：怎么用最简洁的方式实现这个功能。

But under the hood, if you sit on any of our engineering meetings, you will hear our engineers talk about how do we make sure that we implement this capability with the greatest simplicity that we possibly can.

说到这个，你们通常会招什么样的工程师来负责S3？更看重哪些特质或者过往经验？

Speaking of which, what type of engineers do you typically hire to to work at S3 in terms of what kind of traits, potentially past experience do you look for?

我们招各种背景的工程师。

Well, we hire all kinds of engineers.

S3团队里有很多刚入行的年轻工程师。

You know, we have a lot of engineers on S3 who are early career.

他们刚毕业，要么是本科生，要么是研究生。

They're straight out of school or they're at a undergrad or graduate school.

就像我说的，我们团队里有大量工程师在S3待了很长时间，也有刚加入的，各种情况都有。

And like I said, we have like a ton of engineers who have been on S3 for a long time and everything in between.

我们做数据的团队里，有一种很强的主人翁意识。

I think there's a really strong element in our teams that work on data around ownership.

每个人都有一种发自内心的使命感。

People feel this like personal sense of commitment.

我确实有这种感觉。

I feel it.

我每天上班都能感受到，我对自己负责的部分有强烈的责任感——确保它稳定、好用，让你能专心思考应用下一步该做什么，而不是纠结于该用什么存储、怎么扩展。

I feel it every day I come in, where I feel a personal sense of commitment to your bite, to the preservation of your bite, to the usefulness of your bite, to the ability for you to think about what your application does next and not the types of storage that you need or how you grow it.

这种深刻的主人翁意识和使命感，在我们的数据团队里非常普遍。

And that deep sense of ownership and that deep sense of commitment is a very, very common thread across our data teams.

因为我们清楚，归根结底，每个现代企业都是数据驱动的企业，无论人们想用传统系统还是AI，核心都是你的数据在塑造应用体验。

Because we know that at the end of the day, every modern business is a data business and everything that people are trying to do with traditional systems, AI, whatever, is based on your data as shaping the core of your application experience.

所以这些数据就是我们的责任，我们对此有着深切的使命感。

And so that data is our responsibility and we feel it very deeply.

那对于一位有几年工作经验、在不同公司待过的中级软件工程师来说，你有什么建议？如果听完这些后，他深受鼓舞，想有一天加入像S3这样扎实的基础设施团队，或者对更有经验的人而言，你会建议他们关注哪些经历或活动，来判断这样的人才是否合适？

And what would your advice be to, let's say, mid career software engineer, someone who has a few years experience working at different places, who would who is actually, after listening to this, gets really enthusiastic and decides like, one day I'd love to work on a deep, strong infrastructure team like S3, or like, let's say, like more experienced folks, what are experiences, activities that you might look for that might help you consider these folks more?

持续的好奇心非常重要。

There's a strong value in relentless curiosity, okay?

我之前提到过‘循规蹈矩’，但当你在S3或者一个大规模分布式系统上工作时，这个系统本身就在不断重新定义存储的意义，你根本不是在框框里画画。

And, you know, I talked a little bit about coloring within the lines, and how when you work on S3 or a large scale distributed system, which continues to reinvent what storage means, you're not really coloring within the lines.

你得退后一步，重新思考：今天我来画这些边界，但我也清楚，未来很可能要擦掉它们，重新划定新的界限，去适应技术的发展方向。

You're just kinda looking, you're taking a step back and you're saying, you know, I will draw what the lines are today and I will know that I might have to rub those out and draw new lines in the future for wherever things go.

我有三个孩子，都在上大学。

And so, you know, I have three kids who are in university.

我有两个孩子在读大学，还有一个在读研究生。

I have two kids in university and one in grad school.

我觉得特别重要的一点是，永远要抽身出来，去看看最新的研究进展。

And that is one thing that I, you know, I think is really important, is to always take a step back, take a look at the latest research.

我待会儿跟你们分享的一些论文，讲的是我们如何把形式化方法应用到存储系统中。

And some of the papers that I'll share with you are around how we, you know, we either took formal methods and we brought them into storage systems, right?

或者我们换了一种思路来看待故障，这种创造力、这种永不满足的好奇心，再加上工程上的巧思，绝对错不了。

Or we thought about failure in a different way where that creativity, that relentless curiosity and that creativity with engineering, I don't think you can go wrong with that.

我认为，无论下一代软件是在S3上开发，还是在别的地方，归根结底都靠工程师的创造力驱动。

I think the next generation of software, matter if it's built in S3 or elsewhere, it is all driven by the creativity of the engineering mind.

这种能力，其实我们每个人都有。

And it is in all of us.

我们只需要把它激发出来，释放出来，就能打造出像S3这样了不起的东西。

We just have to kinda unlock it and unleash it and we will brand we'll build amazing things like S3.

我也特别喜欢S3这一点——它不仅创造出了以前根本不存在的东西，甚至以前根本没人敢想象，现在我听到不少初创公司都在S3的基础上做创新。

And I also love that with S3, not only has S3 created something that did not exist, I think it just was unimaginable because it didn't exist, but now I'm hearing startups that are building on top of S3.

比如Turbo Puffer，就是个很好的例子。

Think Turbo Puffer is a good example.

他们之所以能不断创新，是因为有了一个稳固的底层基础，我觉得创新其实分不同的层次。

You know, they're building innovation because now they have a base layer, and I I feel there's different levels of innovation.

你可以选择在最底层、再高一层，或者更高层去创新，关键是用对了基础工具，对吧？

You decide where you wanna innovate at the very lowest level, one level higher, and so on, and you just use the right primitives, right?

对你来说，就是把硬件和存储做得比谁都好。

In your case, this is just doing hardware and storage better than anyone.

而在其他层面，就是比别人更擅长运用合适的基础工具。

In the other layers, it will be using the right primitives better than anyone.

看到现在有这么多不同类型的基础设施基于 S3 构建起来，确实让我们非常兴奋。

Yeah, it's very exciting for us to see so many different types of infrastructure built on S3 now.

最后，你有没有哪本书或哪篇论文特别推荐？你读过之后觉得特别棒，为什么？

And as closing, what is a book or a paper that you would recommend a reading that you enjoyed and why?

我读过很多不同领域的论文。

I read a lot of different papers.

我现在特别着迷于嵌入模型发展得这么快。

I am fascinated by how quickly the evolution of embedding models are coming along now.

尤其是我特别感兴趣的一个领域，是多模态嵌入模型。

And in particular, a field of science that I'm quite interested in is the multimodal embedding model.

因为你知道，我们所体验的世界是多模态的，所以对数据的理解也应该是多模态的。

Because as you know, the world that we experience is multimodal, and therefore the understanding that we have of data should be multimodal as well.

因此，围绕多模态嵌入模型的一个全新研究领域正在迅速兴起。

And so there's this whole field of science that's emerging quite rapidly around multimodal embedding models.

所以我建议从事数据相关工作的人多关注这一点，因为我觉得这将是下一代数据的核心。

And so that is something that I encourage people who are working in the field of data to look at, because I think that is the next generation of data.

如果你想想未来的数据湖，我觉得真正关键的其实是元数据。

If you think about, you know, the next world of data lakes, I think it's actually gonna be on metadata.

是数据的语义理解，而不是数据本身。

It's gonna be on the semantic understanding of our data.

理解这些语义如何通过向量生成，以及如何在多种模态间进行搜索和处理，这既是研究的重点，也是技术突破的关键。

And understanding how that is created through vectors and how it's being searched and done across multiple modalities, think is an important area of both research and advancement.

所以这就是我建议大家在数据领域重点关注的方向。

And so that's what I would encourage people to look at in the world of data.

我觉得向量技术会大有作为，尤其是我们为S3向量存储设定的这个价格水平，更是会加速它的普及。

I think Vectors is gonna be quite big, particularly at the price point that we've introduced for S3 storage for Vectors.

我对这个特别期待。

And I'm excited about it.

我觉得我们对数据的理解才刚刚开始，真想看看接下来会发生什么。

I think, you know, I think we're just getting started with data and an understanding of our data, and I can't wait to see what comes next.

太棒了。

Amazing.

你有什么书可以推荐吗？

And and do you have any book recommendations?

我来给你推荐一本书，万一你的读者感兴趣呢。

I will give you a book recommendation just in case your readers are interested.

这本书不是关于计算机科学的。

It won't be in the field of computer science.

它讲的是我们周围生态系统的演变，以及如何保护本地的蜜蜂和昆虫。

It will be about the evolution of the ecology around us and supporting the bees, the native bees and insects around us.

虽然稍微有点偏题，但我还是推荐一本，如果你的读者想了解如何保护地球上的蜜蜂，不妨看看。

So a tiny bit farther afield, but I'll give you a book recommendation, and if your your readers are interested, they can they can take a look at how to support the bees of the planet.

谢谢您，Mai-Lan。

Well, Mai-Lan, thank you very much.

这次对话太精彩了，能深入了解数据世界的庞大规模，以及如何妥善处理、保障数据的持久性，真是令人着迷。

This was fascinating and and very interesting to get a peek into this massive world of scale of data and respecting the bite and treating it and making sure that it's durable.

和您聊天非常愉快，也感谢您本人，我知道您是S3的粉丝，更要感谢所有使用S3的听众——我们今天能做这些事，完全离不开大家的反馈和鼓励。

It was great talking to you, and thank you to both yourself, I know you're a fan of S3, and to all of your listeners who use S3, we quite literally wouldn't be able to do what we do without the feedback and the encouragement from everybody who uses S3 today.

真的很感谢你们。

So thank you for that.

太惊人了。

Just wow.

我早就猜到S3这样的系统背后一定很复杂，但没想到规模竟如此惊人。

I always suspected there's a lot of complexity behind a system like S3, but I just did not realize the scale of it.

以前我做系统时，哪怕只有几百台虚拟机，单台机器出故障都是罕见的事，根本不会当成常态来应对。

Whenever I worked on systems with even hundreds of virtual machines, failure of one machine was a rare event and not something that we counted on.

和Mai-Lan聊的时候，她轻描淡写地说，我们对话期间就有好几台机器宕机了——而S3早就预料到这种情况，把它当成日常事务来处理。

During my conversation with Mai-Lan she casually mentioned that several machines have failed during our conversation, which is something that the S3 knows and prepares for, and treats it like an everyday event.

我特别欣赏AWS在S3团队里同时坚持的两个看似矛盾的原则：尊重历史，同时技术上无所畏惧。

I personally really liked how AWS has two conflicting tenants heavily used on the S3 team, respect what came before and technically fearless.

对于这样一个庞大的系统，很容易因为太多公司依赖它就选择保守行事，但如果真这么做了，S3迟早会被甩在后面。

For such a massive system it will be easy to say let's move conservatively because of how many companies depend on us but if they did so S3 would fall behind.

最让我惊叹的是，AWS居然实现了强一致性，还向所有客户全面上线，既没涨价，也没增加延迟。

Finally, I'm still in awe that AWS put strong consistency in place, rolled it out to all customers, and did not increase pricing nor did they increase latency.

在S3这样的规模下，这绝对是顶尖级别的工程成就。

At S3 scale, this is an absolutely next level engineering achievement.

事实上，这可能是过去十年里最不为人知的工程壮举之一。

In fact, it was probably one of the lesser known engineering feats of the decade.

希望你觉得这期节目和我一样精彩。

I hope you found the episode as fascinating as I did.

如果你想更深入了解亚马逊和AWS，可以在节目简介里查看我与AWS故障应对团队做的独家深度访谈，讲他们如何处理服务中断。

If you'd like to learn more about Amazon and AWS, check out the exclusive deep dive I did with AWS' incident management team on how they handle outages in the show notes below.

在《务实工程师》里，我还做过其他关于亚马逊和AWS的深度分析。

In The Pragmatic Engineer, I also did other deep dives about Amazon and AWS.

这些链接也放在节目笔记里了。

They are also linked in the show notes.

如果你喜欢这个播客，别忘了在你常用的播客平台和YouTube上订阅。

If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube.

如果你还给节目打分，那就太感谢了。

A special thank you if you also leave a rating on the show.

谢谢收听，我们下期再见。

Thanks, and see you in the next one.