How Meta reduced diff authoring time by 40%

我得说，直到最近，Meta的硬通货还是代码。

I would say until quite recently, the currency at Meta was code.

所以代码评审在公司的地位极其核心。

So diffs played an extremely central element at the company.

因此大家整天都在聊这个，那公司自然就想知道，我们到底花了多少时间在这件天天挂在嘴边的事情上，也就是这个。

And so everyone talks about this all the time, and it is then natural that the business wants to know how much time are we actually spending on that very central thing that we're talking about all the time, which is this.

于是，“代码评审耗时”这个概念就诞生了。

And thus diff authoring time was born.

欢迎收听工程效能播客。

Welcome to the engineering enablement podcast.

我是本周的主持人，霍克。

I'm your host this week, Houck.

在进入正题之前，我想先介绍一下我们第二季度的AI与工程报告。

Before we dive into the episode, I wanna tell you about our q two state of AI and engineering report.

这份报告分析了超过500家工程组织的数据，探讨了AI在哪些方面带来了可衡量的吞吐量和时间节省，哪些方面的提升并没有转化为实际交付的软件或改善的开发者体验，以及数据对可持续AI投资有什么启示。

It looks at data from over 500 engineering organizations and examines where AI is generating measurable gains in throughput and time savings, where those gains are not translating into shipped software or improved developer experience, and what the data suggests about sustainable AI investment.

要阅读完整报告，请访问 gitdx.com/reports。

To read the full report, head over to gitdx.com/reports.

你好。

Hello.

欢迎收听工程赋能播客。

Welcome to the engineering enablement podcast.

我是主持人霍克。

I'm your host, Houck.

今天的嘉宾是莫里茨·贝勒，Meta的软件工程研究员，他的工作处于实证软件工程和现实世界开发者生产力之间。

Today's guest is Moritz Beller, a software engineering researcher at Meta whose work sits at the intersection of empirical software engineering and real world developer productivity.

多年来，莫里茨帮助推进了我们对软件测试、开发者工作流程、持续集成以及最近（毫不意外）AI辅助软件工程的理解。

Over the years, Moritz has helped advance our understanding of software testing, developer workflows, continuous integration, and more recently, not surprisingly, AI assisted software engineering.

今天，我们将深入探讨他在Meta领导的一些工作，包括衡量工程生产力的新方法，以及这些发现告诉我们关于软件开发未来的信息。

Today, we're going to dig into some of the work he's been leading at Meta, including new ways of measuring engineering productivity and what those findings tell us about the future of software development.

另外，从个人角度来说，我要说莫里茨很多年前在微软研究院实习过。

Now, on a more personal note, I will say that Moritz did an internship with Microsoft Research many, many years ago.

我第一次见到他时，我的团队帮他做了一项研究，那项研究对我后来正式成为软件工程研究员起到了奠基作用。

And I first met him as my team helped on one of his studies, which was foundational in my transition into a more formal software engineering researcher.

所以莫里茨对我职业生涯的影响非常大。

And so Moritz has played a huge role in shaping my career.

因此，这周能请他上播客，我特别激动。

And so I am incredibly excited to have him on the podcast this week.

莫里茨，欢迎你。

Moritz, welcome.

谢谢你，布莱恩。

Thank you, Brian.

能和你聊聊，是我的荣幸。

Well, it's an honor to talk to you.

为了给大家一些背景信息，你能简单介绍一下你在Meta的职位吗？

So just to, you know, help set set some background context, could you tell us a little bit about your role at Meta?

就像你说的，我基本上是在做那种……

Like you were saying, I'm basically doing sort of...

嗯，可以说我同时扮演着三个不同的角色。

Well, wearing three different hats in in a sense.

我有做软件工程研究的学术背景。

I have this academic background on doing software engineering research.

所以很自然，我们在Meta处理的很多问题也带有研究成分。

So naturally, a lot of the problems that we then tackle at Meta also have a research component to them.

这不仅仅是拿一个需求，然后从头到尾把它落地到生产环境。

So it's not like just taking a requirement, right, and seeing it end to end through to production.

实际上，我们通常要先弄清楚我们真正感兴趣的是什么，然后设计一些研究来进一步明确它。

But actually, usually, you know, kind of ironing out what it actually is that that we are interested in, and then designing some studies to further identify that.

这通常会产生一些实证结果。

That typically then yields some empirical results.

我们就是这么做的。

We do that.

如果我们觉得合适，自己能做的话，那我们就会自己来做。

If we think right, we we can do it ourselves, then then we do that.

但如果我们觉得需要更严谨的分析，可能会请你这样的人来帮我们做靠谱的统计。

But if we think that's a more, you know, rigorous need that we maybe pull in folks like you to actually get us good statistics.

然后，一旦有了这些数据，我们就会和真正的工程师团队一起，根据数据来决定，好吧，最好的实现方式是什么，或者只是把它做成一个仪表盘，给领导看或者自己用。

And, once we have that, then with with a team of actual engineers, right, we we then decide based on the data, okay, what should be the best approach to implementing this or just, you know, kind of surfacing it as a dashboard, be it for leadership or for direct personal consumption.

所以这真的就像是一个研究负责人、一个软件工程师负责人，然后还要像个技术负责人一样，协调不同角色之间的工作。

So it's really like basically a research head, a kind of software engineer head, and then, we'll kind of tech lead and kind of orchestrating between the different roles.

我很喜欢这个。

I love it.

我们有点过度使用“应用科学家”这个词了。

We've sort of overloaded the term applied scientist.

现在通常指的是真正在构建机器学习模型的人。

Typically, now it means people actually building ML models.

但在我看来，那才是真正的应用科学。

But in my mind, that's what real applied science is.

你是在做实际的研究，而且不仅仅是为了推动知识进步——这当然是有价值的——但关键是怎么把它应用到实践中去？

You're doing actual research and then you're doing it not just for the sake of advancing knowledge, which is a valid cause, but how do you put it into practice?

说到这个，我看过你过去几年的一些工作，里面提到一个我觉得特别有意思的指标，叫作 diff 编写时间。

And so speaking of that, some of the work that I've seen from you over the last couple of years talks about a metric that I think is incredibly interesting called diff authoring time.

那我们先从基础讲起吧，到底什么是 diff 编写时间？

And so just to help us start with the basics, what exactly is diff authoring time?

对于不太熟悉我们术语的朋友，我们可能得先解释一下，diff 到底是什么。

We should maybe prefix this by asking the question, what is a diff for those that are not super familiar with our terminology?

它基本上就相当于 pull request。

So it's essentially the equivalent of a pull request.

所以，这是一个包含代码变更的完整包，附带你对这些变更（或者说是 AI 代理所做的变更）的总结，以及一份测试计划。现在的世界里就是这样，审查者可以针对这些内容留下评论。

So it's a self contained code change together with a summary of the changes that you're making or the agent is making in today's world with a test plan, and then reviewers have the possibility to leave comments on this.

而且，这个过程可能会有多轮迭代，对吧？也就是代码变更会有多个版本。

And you may have multiple rounds, right, multiple versions of that code change.

所以，基本上我想说，直到最近一段时间，在 Meta 的核心衡量标准还是代码。

And so, basically, I would say until quite recently, the currency at Meta was code.

因此，代码差异（diffs）在公司里扮演着极其核心的角色。这不仅仅是出于性能方面的考虑，更因为这是我们沟通的方式，也是推动工程层面发展的关键动力。

So diffs played an extremely central element at the company, not just, you know, for kind of performance reasons, but simply because this is how we were talking about things, and this is what moved the company forward from an engineering perspective.

所以基本上，大家一直都在谈论这个。

And so everyone talks about this all all the time, basically.

那么很自然地，公司就想知道，嘿，我们到底花了多少时间在那件我们一直挂在嘴边的最核心的事情上，也就是这个。

And it is then natural that the business wants to know, hey, how much time are we actually spending on that very central thing that we're talking about all the time, which is this.

于是，diff 编写时间这个概念就诞生了，它主要衡量的是人类在工程、审查，甚至准备和测试 diff 上花费的主动时间。

And thus diff authoring time was born, which basically measures the active time that humans spend engineering, reviewing, maybe even preparing, testing diff.

这对我来说非常有意思。

That's super interesting to me.

我的很多工作都在研究开发者是如何分配时间的，哪些活动算是高效的，哪些不是，还有那些……

A lot of my work has looked into how are developers spending their time, which of those activities are sort of productive, what aren't, what are the...

它们之间是怎么相互关联的？

You know, how do they relate to each other?

而且，我确实研究过，开发者到底花了多少时间在主动编写代码上。

And, I definitely have looked at, well, how much time are developers spending actively writing code?

我知道，因为 AI 的出现，现在这种情况可能正在改变，但总的来说，这是开发者最想做的事情。

And I know that it is maybe changing today because of AI, but in general, that's the activity developers want to do the most.

那你能把这种编写时间跟具体的diff联系起来吗？

Now, are you able to tie that sort of authoring time to specific diffs?

还是说只是对某个人参与的所有diff取个平均值？

Or are you just like averaging out over all of the diffs that that someone may participate in?

一开始其实就是个平均值。

It started out as practically being an average.

那是个比较粗糙的指标，在DAT之前就有了，主要是为了满足业务上的紧急需求。

This was a kind of cruder metric that predated DAT to fill the immediate business need.

但后来我们意识到，这样其实有很多漏洞，对吧？

But then we realized, hey, this leaves a lot of gaps, right?

有很多事情我们做不了。

There are a lot of things that we can't do.

其中一个比较重要的就是，我们没法像在产品里那样做受控的AP实验。

One of the more important ones is that we can't actually run controlled AP experiments much in the same way that we do in product.

所以当然，任何……

So of course, any...

做软件工程的人都知道，像 Facebook、Instagram 这样的平台，还有其他一些公司，一直在不停地跑 A/B 测试，就是为了看看哪些功能能带来更好的用户体验，哪些功能能在其他指标上表现更好。

Anyone who's interested in software engineering knows this, that sites like Facebook, Instagram, but also others run lots of a a b experiments all the time to find out, hey, which of these features gives a better user experience, which of these features optimizes for certain other metrics, right?

所以性能也可以作为其他的衡量指标。

So Performance could be other metrics as well.

但在内部，我们还没达到那个阶段，因为我们缺乏针对自身内部开发的评估指标，没法很好地衡量 Meta 工程师们的开发体验。

But internally, we weren't quite at that point because we didn't quite have that metric for our own internal development and how we could assess basically the development experience of our own engineers at Meta.

所以这些粗略的指标做不到这一点。比如，如果我们在某个代码变更中引入了某项功能，或者人们在某个变更中使用了它，但同一天却没用到这个功能……

So these cruder metrics wouldn't allow us to do that because say if we had introduced a certain feature or people were using a certain feature in a diff, but were also not using that feature on, say, the same day where...

或者在之前取平均值的那个时间段里，我们也得不到清晰的数据信号。

Or at the same period where we were averaging over, we wouldn't get clean signal.

所以很明显，如果我们想让内部研发工作更科学化，并且达到产品团队那样的标准，我们就需要依靠 diff 编写时间所提供的这种高精度数据。

And so it was pretty apparent that if you wanted to make our internal development more scientific and basically bring it on par with the standards that we had in product, we would need the fidelity that diff authoring time allows us to do.

你说得完全对。

And so that's exactly what you say.

基本上精确到毫秒级别，我们就能知道你在某个时间点具体在处理哪个 diff，前提是你当时确实在做那个 diff。

Pretty much on a, you know, millisecond basis, we can tell sort of what diff you were working on at that point in time, if you were working on the diff.

这对我来说太不可思议了，那种精确度。

So that's like, that's wild to me, sort of the the level of of precision there.

但你刚才提到的用这个做AB测试，真的勾起了我的好奇心。

But you said something there that really piqued my curiosity around using this for AB testing.

所以我觉得大家自然的反应可能是，哦，用不同的开发工具。

And so I think probably the natural reaction to that is, oh, well, using different development tools.

就像，哦，开发工具。

It's like, Oh, development tool.

如果我用VS Code而不是Emacs，那对我的代码提交时间会有什么影响？

If I'm using Versus Code versus Emacs, what does that do to my diff authoring time?

如果我们做不同类型的构建优化，诸如此类的事情。

If we do different sorts of build optimizations, all of these sorts of things.

但我很好奇，你们有没有做过一些比较大胆的AB实验？

But I'm curious, do you ever run wild AB experiments?

比如，在我们的环境里。

Like, with our environment.

比如说，如果我们放一堆植物会怎么样？

Like, oh, well, what happens if we put a bunch of plants in?

如果我们换成紫外线灯会怎么样？

What happens if we put UV replacement?

红墙、蓝墙、米色墙，哪个更好？

Red walls versus blue walls versus cream walls.

你们可能会做哪些AB测试呢？

What sorts of AB tests might you be running?

我觉得更接近你第一个想法。

I think it's closer to your first.

我也这么觉得。

I suspect it.

但第二个想法，因为成本问题，我觉得我们可能没有专门的团队去优化开发环境里的墙色，不过这个想法挺有意思的。

But then the second one, because there is a cost, I don't think we have a team, right, that optimizes some like wall color in developer environments, although that's an interesting thought.

不过举个例子，其实我有两个很好的受控AB实验的例子。

But so one example would be, and actually I basically have two really good examples that are controlled AB experiments.

其中一个例子就是在Hack编译器里用自动记忆化，以前人们基本上得自己手动写那种缓存机制。

And one of them is using auto memoization in the hack compiler, where basically people had to previously hand roll their own kind of, if you will, caching make...

在UI里。

Mechanism in the UI.

所以如果某个状态变了，你就得重新计算它的所有子节点，看它们的状态是不是也得变，然后可能还得重新渲染。

So if the state of something changed, then you would need to kind of, you know, recalculate for all of its children whether their state also had to change and then possibly rerender this.

这样做既容易出错，成本也高，因为每个你想启用缓存的UI元素都得这么搞一遍。

That is both error prone and pretty costly to do because also to be redone for every single UI element that you wanted to have caching enabled.

所以React编译器团队就打算把这个做成默认支持的功能，那个编译器叫React Forget。

And so then the React compiler team set out to basically make this a supported default in the React forget compiler is is the name of that.

所以我们就能比较用这个新React Forget编译器特性的改动，和以前手动写记忆化或缓存的改动。

And so we were able to compare diffs that were using this feature of the new React forget compiler versus diffs that were hand rolling their own memorization slash caching in the past.

然后……

And so...

我们甚至都没看那些能带来的质量上的好处，光是在写改动的规模上，就看到了相当大的提升。

And we didn't even look at these sort of, you know, quality benefits that you would get from that, but just on the on the diff authoring scale, we saw quite massive improvements.

我记得使用记忆化功能的差异，大约减少了30%的DAT。

It was like, I think around 30% fewer DAT for the diffs that were using, you know, the the memorization.

当然，这是一个...

Of course, it's a...

React团队为了引入这个功能付出了很大的努力。

It was a big lift from the React team to get that in.

所以最终你需要在成本和一次性投入之间做权衡。

So you would have to eventually, you know, trade off those costs against the one time investment.

但如果这个功能后来被大量复用，那肯定是值得的。

But if you have enough reuse of that feature later on, it's it's certainly paying off.

我确实认可这种用法...

I mean, I certainly acknowledge that use...

比如运行AB实验，把差异编写时间作为结果变量，来观察工程工具和工程流程的实际变化，这是有道理的。

Like running AB experiments, using diff authoring time as sort of the outcome variable to look at practical changes in engineering tools and engineering workflows like that make sense.

就像，那是...

Like, that is...

我觉得可能更有用的用例是那个。

You know, I I I think probably a more useful use case.

但我心里那个疯狂科学家想让你做这样的实验，比如，嘿。

But the mad scientist in me wants you to run things like, hey.

如果我们找一群远程工作的人，在他们工作环境里放一只狗，对比放一只猫，会怎么样？

If we take a bunch of people who work remotely, what happens if we drop a dog into their, like, environment versus a cat?

哪个对他们的代码评审时间影响更大？

What's more disruptive to their diff authoring time?

不过也许这些疯狂的科学实验，就留到以后我们俩合作的时候再做吧。

But maybe we'll save the mad science y experiments for for some future collaboration, you and I.

我知道微软研究院在这方面有一些有趣的研究，而且我觉得他们还在继续做。

I mean, I I know Microsoft Research has some interesting stuff on on that, and I think are still doing it.

我觉得我们的做法可能更保守一些，但我同意你的看法。

I think we are a little bit more conservative in that approach, but but I do agree with you.

那确实会很有意思。

It it will be interesting.

而且我觉得肯定有一些因素，可能比我们现在衡量的那些更有影响力。

And I think there's certainly factors that are maybe much more influential than what we're currently measuring.

我们是复杂环境的产物。

We are products of complex environments.

完全同意。

Absolutely.

没错。

Exactly.

是的。

Yes.

所以每个人都很关心自己的开发者体验指标，对吧？

So everyone passionate about their own developer experience metrics, right?

比如，关于哪些指标最能衡量开发者生产力，有很多不同的看法。

Like there's lots of opinions on what metrics are best for capturing developer productivity.

我很好奇，你觉得Meta之前用的那些指标里，缺少了什么，而diff编写时间这个指标正好能补上，让它特别适合做你刚才提到的那些实验？

And I'm curious, what do you think was missing from the metrics that Meta was already using that diff authoring time sort of solves that makes it so suitable for running experiments like you just mentioned?

所以这是一个速度指标。

So it's a velocity metric.

从某种意义上说，它必须嵌入到……

In a sense, it has to be embedded in...

如果你想描绘一个相对完整的图景，如果你关心质量的话，它必须与某种质量度量相平衡。

If you're interested in kind of drawing a relatively complete picture, it has to be counterbalanced against some measure of quality if you care about that.

当然，不同的业务会对这个赋予不同的权重。

And of course, different businesses would associate different weight to that.

而且基本上也是通过吞吐量来体现的。

And also through basically the throughput.

如果你……我觉得这些就像是，如果这基本上就是它起作用的三角关系。

And if you, I think these are like, if this is basically the kind of triangle that it works in.

我们还有另一个指标叫DDM，即每位开发者每月提交的差异数，这基本上是由我的好朋友Karim Nakat首创的，你也许也应该邀请他。

So we have another metric called DDM, diffs per developer per month, that was basically pioneered by my good friend, Karim Nakat, who you should maybe also invite.

我们共同的好朋友，Karim Nakat。

Our shared good friend, Karim Nakat.

DDM很有意思，因为它基本上给你一个在特定时间范围内的吞吐量指标，这个时间范围最好足够大，以便排除一些日常生活中的小变化，比如你请个假什么的。

DDM is interesting because it gives you gives you basically a throughput metric in a given timeframe that's hopefully large enough to, you know, factor out some minor changes in like day to day life, if you take a PTO or something like that.

然后我觉得还有一个质量方面的考量，但如果你想继续留在diff这个体系里，另一个有趣的点就是看看这些diff有多大，对吧？

And then I think there's a, there's a quality aspect, but then you could, like, if you want to stay sort of in the in the diff ecosystem, another thing that will be interesting is to look at like how large are those diffs, right?

这三个因素——吞吐量、大小和持续时间——会自然地相互平衡。

And those three kind of factors like throughput, size, and duration, they would naturally balance each other out.

比如说，如果你想在其中一个上做手脚，那其他两个就会有所变动。

Say if you wanted to game one of these, then you would see movement in the others.

所以，但我觉得可能回到你的问题上来，我们之前那个体系里缺少的是，我们有一个相对粗糙的指标，按天平均。

So, but I think maybe coming back a bit to your question, what were we missing in that landscape that we had was that we had this relatively crude metric that averaged on a daily basis.

我们发现它根本提供不了我们需要的精确度。

And we found that it just didn't provide the fidelity that we needed.

实际上，现在DAT已经存在大约三年了，我们确实还能从我们做的投资中受益，因为我们看到它在AI时代比我们之前那些平均指标适用得多。

And in fact, now that DAT has existed for some three years, we actually can, we still benefit from, from the investments that we're making because we see that it applies much better in the in an AI world than our previously averaged metrics.

它对我们预期中的变动要敏感得多，不像那些平均化的指标。

It's much more sensitive to movements that we were, that we would expect than these averaged out metrics.

所以其实我们还在持续投入。

So there's actually continued investment in it.

这对我来说非常有意思，因为我猜当人们听到像“每位开发者的差异数”或者“差异提供时间”这样的说法时，可能会担心，嗯，不是每个差异都是一样的。

That's very interesting to me because I suspect when people hear something like diffs per developer or even diff offering time, there might be some concern on, well, not every diff is created equal.

对于资深开发者和初级开发者，期望值是不同的。

There are different expectations for more senior developers versus more junior developers.

所以如果这些东西被用来做绩效评估，可能并不能真正反映我在做的事情。

So if these things are being used for performance assessment, it might not be truly capturing what I'm doing.

我能理解像你们这样的团队，有扎实的研究背景，会负责任地使用这些数据，比如做AB测试。

And I can understand teams like yours, where you have a rich background in rigorous research, using this responsibly for things like running AB tests.

我很好奇，你们在实际中是怎么把这些应用到一线开发团队里的？

I'm curious how you actually see this used on development, your frontline development teams in the wild?

比如，有哪些不同的实际使用场景？

Like, what are the different real world use cases for for this?

我觉得我们已经提到了最科学上激动人心的一个，就是做AB实验。

So I think we covered, I think, the scientifically most exciting one already, which is running AB experiments.

即使你有一种直觉，觉得这确实帮助了我们的开发者，能够量化它真的很有价值。

Even if you had an intuition, right, of like, hey, this helped our developers, it really is beneficial to be able to quantify it.

这样组织就能据此进行规划，并优先考虑那些原本可能不会优先处理的变更。

And that then allows, you know, organizations to plan for it, and to prioritize certain changes that they might otherwise not have.

比如，我们在这次实验中看到的DAT节省幅度，还有我之前简单提到的第二个实验，都大得超乎预期。

So for example, the magnitude of those DAT savings that we saw in this experiment, but also in the second one that I was briefly referring to earlier, were unexpectedly large.

所以我们觉得需要在一些基础性的改变上投入更多，比如在……

And so it seemed to us that we needed to invest more into kind of foundational changes in the...

在框架和工具层面，而不是在UI上做微小的优化，这跟产品领域的情况有点不同，我记得Airbnb有一篇论文，他们……

In frameworks and in in tooling rather than maybe micro optimizing UIs a little bit, right, which is a bit different than what you see in the product landscape, where anecdotally, I think there's paper by Airbnb where they...

他们把按钮改了几个像素，或者改了颜色，结果带来了统计上显著的……

That they change the button by a couple of pixels, or they change the color, right, and it yielded a stat sake.

巨大的改进。

Huge improvement for for them.

但我们在DAT上并没有看到这种情况。

We we haven't actually seen this with DAT.

也许是因为人群不同，但这确实很有意思。

Maybe it's a different population, but this is interesting.

所以基本上就是做实验，然后这些实验决定了我们整个组织怎么规划、怎么决定给哪些改动拨款。

So it's basically experiments, and then they informed how we, as an organization, think about planning and which changes are being funded.

然后我觉得更重要的是，它还会影响开发者的思维方式……

And then I think even more so, it also informs the way of developers for...

即使他们不用DAT——其实也不是每个功能都必须用——也会影响他们怎么设计产品和改动。

Even if they're not using DAT, which they really don't have to for every single feature, how to how to think about designing their products and their changes.

所以这……

So it's...

可以说，它让你在开发时更像一个科学家，心态完全不一样。

Instills a different, I would say, state of mind when when you're developing more like a scientist.

这是其中一类。

And then this is one class.

然后我觉得第二类主要是给领导层做汇报用的，对吧？

And then I think the second one is for basically leadership reporting, right?

确保基本上在团队层面，或者再说一遍，在公司层面，指标表现怎么样？

Making sure that basically at the team or again, then company level, how, like, how are metrics doing?

我们有没有看到某些东西出现意想不到的严重回退？

Do we see kind of unexpected crazy regressions in in something?

或者，如果我们设定了某些目标，因为它们是跟不同的项目挂钩的，我们有没有达成这些目标？

Or are we hitting our goals if we if we set certain goals because they're associated with different initiatives?

把目标跟代码审查时间联系起来想也很有意思，因为不是所有的代码审查，当然也不是所有类型的活动都是一样的。

I mean, thinking about goals as it relates to diff authoring time is really interesting too, because not all diffs, certainly not all sort of class of activities are created the same.

我很好奇，你们有没有试着把代码审查分类，比如这是新创新，还是维持业务，还是还技术债之类的？

I'm curious, do you try to sort of bucketize diffs by, like, is this new innovation versus sort of running the business versus some sort of, like, you know, paying down some sort of tech debt?

你们是怎么考虑这个的？

How do you, like, think about that?

这正是我们在做的。

That's exactly what we're doing.

我们确实在按某种方式对代码审查进行分类。

We're categorizing diffs a certain way there.

而且有趣的是，你会预期不同组织或团队的这些类别分布会有所不同。

And what's also interesting is you would expect different distribution of those categories per org or team.

对于基础设施团队来说，如果他们的diff大部分是内部的，那完全正常，没问题。

A team that's an infrastructure, if most of their diffs are internal, that's totally expected and fine.

但如果你是产品功能团队，而你的diff大部分是内部的，那就有点奇怪了。

However, if you're a product feature team and most of your diffs are internal, something's strange.

可能是被错误分类了。

It could be iFlassified.

也可能是因为你正在做的工作类型。

It could be, like, the work you're doing.

研究已经说得很清楚了。

Research is pretty clear.

文献明确指出，开发者希望把时间花在实际开发上。

The literature is pretty clear that developers want to be spending time actually building.

他们希望把时间花在开发上。

They want to be spending time building.

作为整个行业，我们当然希望工程师们，这也是我们在DX Core four中提到的，我们希望花更多时间在创新交付上。

We certainly, as an industry, want engineers, and we talk about this as part of the DX Core four, we wanna be spending more time delivering innovation.

那么，我们如何花更多时间在创新交付上呢？

And so how do we spend more time on delivering innovation?

而diff编写时间，特别是如果你能对diff进行分类，我觉得这是一个很好的衡量标准。

And diff authoring time, especially if you can categorize the diffs, I see how it's a good measure for that.

但在我看来，这不仅仅是速度指标。

But it's not just a velocity metric in my mind.

比如，还有一个效率的角度。

Like, there's also sort of an efficiency standpoint.

所以，这真的是关于我们能否提高或降低总的diff编写时间，还是你实际上把它看作每个diff的时间？

So is it really about, can we drive total diff authoring time up or down, or do you actually look at it as, like, time per diff?

我们是否在获得更多...

And are we getting more...

我们是否在单位时间内创建更多的diff？

Are we creating more diffs per unit of time that we're spending?

所以我们从来没有用它来衡量性能。

And so we've never used it for performance reasons.

不像其他用于评估性能的指标，diff 编写时间从未出现在那些工具里。

Unlike other metrics that were surfaced in the tools that were used to assess performance, diff authoring time never appeared in there.

这是我们有意为之的选择。

That that was conscious choice in our part.

话虽如此，尤其是现在，对吧？

That said, especially now, right?

但对 Meta 来说，2022 年是效率之年，而 diff 编写时间的开发也正是从那时开始的。

But also for Meta, the year of efficiency was 2022, which is when diff authoring time development started.

所以这显然不是巧合。

So that was not a coincidence, obviously.

所以我们对此很感兴趣，即使可能不是针对个人或基于绩效的。

So we are interested in that, even if it's not maybe on an individual basis or, or on a performance base.

但没错，我们基本上会看，比如每个月创建了多少个 diff。

But, but yes, so we basically look at, okay, how many just have been created in a certain time period, like per month.

那当时的DAT是多少？

And what was the what was the DAT for that?

之所以用这个配对，基本上是因为，你可以轻易地把你的DDM翻倍，只要把你的DAT减半，但最终的产品是一样的。

And the reason for that pair is basically, well, you can trivially, you know, two x your DDM if you half your DAT, but the end product is the same.

我们想避免这种情况。

We wanna avoid that.

可能不是完全那种关系，因为创建更多的diff会有一些额外的开销。

Probably, it's not exactly that relation because there's some overhead associated with creating more diffs.

所以实际上，激励并不是为了你自己或你自己的审查去做这件事，因为理论上，你被评估的是你交付的影响，而不管是用10个还是20个diff完成，单独来看会有区别。

So actually, like, the incentive is not to do that for yourself, for your own review, because in theory, what you're evaluated on is is the impact you shipped and whether that's done in 10 or 20 diffs in isolation would make a difference.

有意思。

Interesting.

所以这确实涉及到了它的可操作性。

So that certainly looks at the gamability of it.

也就是说，个人能做什么来钻空子。

And it's like, what can individuals do to game it?

但我想到的是更宏观的层面，就是说，如果你想看系统效率，如果你以某种方式对diff进行标准化，考虑大小和复杂度这些因素，那么如果我花一个小时写diff，能创建10个，而别人能创建20个，那可能是因为我的构建工具更快或者更可预测。

But I was thinking even more high level, which is just like, if you want to look at system efficiency, if you sort of normalize the diffs in some way for size and complexity and all of these things, it's if I can create 10 diffs with an hour of diff authoring time versus I can create 20, and maybe that comes from my build tools are faster or more predictable.

所以我不需要花那么多时间等待。

So I'm not spending as much time waiting around.

我们的代码审查流程也更好了，因为我们用雷达之类的工具自动化了代码审查。

Our code review processes are better because we're automating code review with radar or something.

我很好奇，它是不是真的被这样用了。

I'm curious if if it's used in that way at all.

哦，当然。

Oh, certainly.

我们看到的那个最大的改变显然是AI。

And, I mean, the biggest game changer that we've seen there obviously was AI.

所以我们看到的是，我们的DAT同比降低了超过40%。

And so what we've seen there is basically our year over year DAT is down by over 40.

同时，DDM大幅增加了。

At the same time, DDM has increased substantially.

不过……

While...

现在你可能会说，也许只是变更规模变小了。

And now you could say, well, maybe just maybe just smaller.

这可以解释那个现象，但实际上变更规模反而变大了。

That would explain that, but actually just are getting larger as well.

所以我们从来没有过……

So we've never had...

而且这是在公司层面。

And this is at the company level.

这显然是你能看到的最有力的证据。

We're obviously the strongest indication that you could see for this.

以前我们偶尔会在个别团队看到类似的情况，当他们有不同举措的时候。

So we were, you know, occasionally observing something like this for individual teams when they had, you know, different initiatives.

说实话，有时候我觉得有些数据只是噪音，因为毕竟公司这么大，团队这么多。

Sometimes also, to be honest, I think some of those are just noise because at the end of the day, it's a very large company with so many teams.

总会有一些异常值，所以你得小心，别从这些数据里轻易下结论。

You're always gonna have some outliers, so you need to be careful not to draw conclusions from this.

但如果你能像我们在AI采用中看到的那样，得到一个如此清晰而强烈的信号，而且我们还能把时间上的特定事件和图表中的特定反应对应起来，那这就非常明确了。

But if you get such a clean and strong signal like we've seen with AI adoption, where we can also ascribe certain events in time to certain reactions in the graphs, it is it is very clear.

这对我来说太不可思议了。

So this is all wild to me.

就是……

It's...

首先，我注意到……

First off, I saw that...

其实，我最近发了一篇通讯，里面提到微软的Alex Hallman做的一项研究，也发现微软每小时活跃编码的拉取请求数量增加了大约40%。

In fact, I've recently published a newsletter where I talked about a study from Microsoft, from Alex Hallman at Microsoft, that similarly found that the number of pull requests per hour of active coding had increased about 40% in Microsoft.

看起来我们看到的数量级非常相似。

And it seems like we're seeing very similar orders of magnitude.

我很好奇，如果我们写的代码越来越多，但主动编写的时间却越来越少，那这些省下来的时间都去哪儿了呢？

And I'm curious if we're having more and more code, but having to spend less and less time actively authoring it, do you have some idea of where does that time go?

这些时间是不是花在了其他仍然属于构建活动的方面，但我们并不真正把它们看作是编写代码的时间，因为实际上不是在写代码，而是在生成规格说明？

Is it going to other things that are still building activities, but we don't really think of them as diff offering time because it's not actually writing code, it's generating specs?

是不是花在了验证时间上？

Is it going to validation time?

这些时间到底去哪儿了？

Where is that time going?

对，这现在只是我个人的看法，我来说说。

Yeah, this is this is only an informed opinion that I'm going to usher now.

这其实不是科学结论。

It's not actually scientifically.

我们还在研究这个问题，对吧？

We're still looking into this, right?

我觉得是花在收集上下文上了。

I think it's context gathering.

我觉得你说得也对，很多我们传统上认为不是编码的活动，现在可能应该被视为编码的相邻部分。

I think you're also right in pointing out that a lot of the activity that we traditionally considered not coding are now perhaps, or should now perhaps be considered as kind of adjacent part to coding.

我觉得很明显，写提示词花的时间并不多。

I think it's also pretty obvious that not a lot of time goes into prompt writing.

对，确实会花一些时间。

I mean, yes, some some time goes into it.

但我觉得更多的是，像……

But I think it's more so, like...

我觉得一开始我们有过一个阶段，就是，哦，对。

I I think initially, we had a point where, like, oh, yeah.

我就把所有能拿到的信息都塞进提示词里。

I'm just gonna stuff my prompt, right, with, like, all of the information I can get.

但现在我觉得更高效的做法是直接链接信息，因为代理可以自己去获取。

But now it's, I think, more efficient to just link the information because the agent can then go ahead and retrieve that itself.

所以我觉得确实有一些这样的情况。

And so I think there's there's some of that going on, for sure.

但我觉得我们还没完全搞清楚这些时间到底花在了哪里。

But I don't think we have quite found where all of that, you know, time is exactly spent.

我觉得还有一种说法是，也许我们现在需要花更多时间来审查。

I think there's also a narrative that you could say, well, maybe we need to spend more time reviewing now.

而且我注意到，GitHub 好像也得出了类似的结论。

And I've seen that I think GitHub has made some similar findings.

但另一方面，我想问，你觉得一行代码的价值还和以前一样吗？

On the other hand, wonder, do you think that a line of code still has the same value?

我们真的需要那么仔细地审查吗？

Do we even need to review it that thoroughly?

嗯，我觉得这里面其实有两个问题。

Well, I mean, I I think there's two questions in there.

比如，一行代码的价值是否还一样？

Like, does a line of code have the same value?

我的意思是，对业务来说可能一样，但对开发者生产力来说，就不一样了，因为生成那行代码的成本低太多了。

And I mean, to the business, maybe, but to sort of developer James Pughlestone developer productivity, we'll know because the cost of generating that line is so much less.

你最开始提到，历史上代码是软件工程师的货币，是你最看重的东西。

I mean, you said at the very, very beginning, historically code was the currency of software engineers, the currency you cared about.

而现在，真正重要的是意图。

And now, really, intent is the currency we care about.

比如，我们的想法能产生多大的效果？

Like, what is the throughput of our our ideas?

我觉得，我们该怎么衡量这些东西？

And I think how do we measure those things?

它们就像……

They're, like, the...

会变得越来越重要。

Gonna be increasingly the important thing.

对。

Yes.

对。

Yes.

而且，我们输入系统的想法，真的就是最终出现在代码差异里的内容吗？

And is is the idea that we put into the system actually what's coming out in the in the diff?

还是说这两者之间有什么偏差？

Or is there some drift between the two going on?

对。

Yes.

而且很多方面……

And so much around...

你提到现在很多时间都花在收集上下文上，我们怎么看待这些上下文的质量呢？

You know, you mentioned so much time now is sort of going into gathering context and how do we think about the quality of that context?

这包括你所有的提示词……

Which would include all of the prompts you are...

不仅仅是输入的数据，还要看你能否清晰地表达自己的想法。

It's not just the data you're feeding in, but are you able to clearly express your ideas?

然后从利用模型能力开始，这是否能一直贯穿到最终产品？

And then from harness the model capabilities, does that translate all the way through to finished product?

而且事实证明，人类与智能体协作在很多环节都可能出现问题。

And there's lots of places where the human agent collaboration can break down, as it turns out, think.

所以，当我们看……

So, as we look at...

你们有很丰富的数据，关于开发者实际花多少时间来创建这些原子工作单元。

You have this rich dataset on how much time are developers actually spending to create these atomic units of work.

也许现在这些是未来智能体的原子单元，而意图就是开发者的原子工作单元。

And maybe these are now the atomic units of agents moving forward and sort of intent is the atomic unit of work for developers.

你们看到模式在广泛地变化。

And you're seeing how patterns are changing sort of broadly.

当你们按不同群体细分时，你们看到差异了吗？

Do you see differences as you slice into different cohorts?

比如，初级开发者因为使用AI，他们的代码提交时间下降得比资深开发者更快吗？

Like, for example, are junior devs seeing their diff authoring time drop faster because of using AI than very senior devs are?

我们最初观察到的情况恰恰相反，这也让我们很惊讶，因为……

We have initially observed the opposite of that, much to our surprise as well, where...

我一点也不惊讶。

It's nothing to my surprise.

我觉得这个发现也得到了业界的证实，看起来资深开发者确实能更有效地使用这些AI工具。

And I think sort of this finding's been corroborated in in the industry, where it seems like, you know, seniors were able to more effectively use these AI tools.

不过我觉得现在这个差距已经慢慢缩小了。

But I think that has evened out some over time now.

所以如果你...

So if you...

至少从数据指标的角度来看是这样。

At least from a metric perspective.

我在想这一切的时候，感觉...

As I think about all of this, it's...

我一直在纠结一个问题，就是传统的遥测数据其实漏掉了软件工程中很多看不见的工作，比如画白板、头脑风暴、和同事讨论架构对齐等等。

You know, I'm I'm struggling with this idea that traditional telemetry misses a lot of the invisible work that goes into software engineering, sort of white boarding and brainstorming and talking with colleagues to sort of do architectural alignment.

我很好奇，我们能不能利用大语言模型来整理会议记录或者会议音频，把这些越来越重要的缺失部分捕捉下来，然后反馈到diff编写时间的定义里？

And I'm curious, can we start leveraging LLMs to sort of synthesize our meeting transcripts or our meeting audio to capture some of that missing piece that is going to become increasingly important and have that sort of feedback into the definition of of diff authoring time?

我们在Meta刚开始的时候，我就试过这个想法，就是内部录制会议。

I played around with this idea at the very beginning of when we started at Meta to record our meetings internally.

现在我们基本上可以自动转录所有内容，如果你想自动描述的话。

Now now we're basically auto transcribing everything if you want it to be auto described.

所以我觉得，要在会议和现有的代码改动之间做相似度匹配，其实挺容易的。

So then it's pretty easy to do a similarity match, I think, between this meeting and the diffs that are out.

我觉得这会很有意思，能加强那些目前虽然存在但还比较隐性的联系。

I think that that would be very interesting to kind of strengthen these links that that are latent, but do exist right now.

另外，我觉得还有一个有趣的转变。

I also think it's, you know, there's also an interesting shift.

一方面，我们说，生成代码的成本低多了。

So on the one hand, we say, well, it's much cheaper to generate the code.

所以也许我们不需要在架构设计上那么谨慎。

Therefore, maybe we don't need to be as careful in architecting it.

但你也可以反过来想，就拿我们最近一版的DAT来说，现在已经到第八版了。

But you could have the, you could have the counter argument actually for a very recent iteration of DAT, we're now at version eight.

实现它的人并没有写设计文档。

The person who implemented it didn't have a design doc.

但代价是我得花更多时间去审查这些代码，基本上要从代码细节里反推设计思路。其实我不需要看代码本身，我的想法是，如果你信任智能体能按照你的架构去实现，那直接拿到架构部分会容易得多，因为那可能才是我们真正关心的东西。

But then what that cost is that I needed to spend a lot more time reviewing this and basically extracting the design from the details of the code, I didn't need to look at the code, where I'm thinking, well, if you trust the agent to implement your architecture, then it would have been much easier to just get the architecture because that's the piece maybe we care about.

所以，也许仅仅因为我们能跳过架构步骤，我们就真的不该这么做。

And so maybe just because we can skip the architecture skip, we we can skip the architecture step, we we shouldn't.

我觉得佩吉·斯托里博士最近提出的三重债务框架很有说服力，她谈到不仅是技术债务，还有认知债务和意图债务的兴起，这有力地论证了我们不应该跳过架构阶段。

I think doctor Peggy Story's recent triple debt framework talking about the rise of not just technical debt, but cognitive debt and intent debt would argue pretty strongly that we shouldn't just skip the architecture phase.

关于开发者如何分配时间以及这种变化，让我回想起咱们初次见面的时候。当时你在做一个叫《Mind the Gap》的研究，那个研究对我的工作影响非常大。

So this notion of how developers are spending their time and how it's changing sort of makes me reflect back on where you and I first met, where you were working on a study called Mind the Gap, which was hugely influential in my own work.

我的几项特性研究都 heavily 依赖了《Mind Your Gap》里的概念。

And multiple of my feature studies relied on sort of the concepts of Mind Your Gap very heavily.

你能用两分钟时间讲讲你当时想研究什么，以及从这个项目里学到了什么吗？

And so could you give us a two minute version of what you were trying to study and what you learned from that project?

谢谢，Brian。

Thanks, Brian.

挺简单的。

Pretty simple.

这讲的是AI出现之前的开发者生产力。

This is about developer productivity way before AI.

基本上，我们有一些传统的生产力衡量标准，比如每人工作量的代码行数、功能数量，甚至可能是功能点之类的。

Basically, we have these traditional productivity measures, such as lines of code per our worked number of features, maybe even something like function points on the one hand.

然后还有一个相对新兴的研究趋势，就是关注感知到的或自我感知的生产力。

And then there was a relatively nascent research trend to look at perceived or self perceived productivity.

但这两者基本上是各自独立存在的。

But those two lived in isolation, basically.

在那项研究中，我和我在微软研究院的导师托马斯·齐默尔曼想做的，就是弥合这两者之间的鸿沟。

And what we wanted to do in that study with Thomas Zimmerman, who was my mentor at Microsoft Research, was to basically bridge the gap right between the two.

所以我们想同时评估这两者，基本上就是用一方的优势来抵消另一方的劣势。

And so we just wanted to assess both at the same time and basically counterbalance the drawbacks that each of them has with the strengths of the other.

所以，是的，我当时对此非常兴奋。

And so, yeah, this is like, was super excited about doing that.

我非常支持混合方法研究，我认为大规模遥测数据与开发者感知相结合是非常有价值的。

I mean, I am a huge proponent of mixed methods research, and I think, you know, measuring large scale telemetry and blending it with sort of developer perception is so valuable.

我觉得你的研究里，对我来说一个非常重要的发现就是这两者之间是有关系的。

And I think one of the, for me, really important findings of your study is that there is a relationship between those two.

开发者在评估自己的生产力时，这和他们实际做的、可以通过遥测数据测量的活动是有关系的。

The developers actually are, you know, when they are assessing their own productivity, that has a relationship in the actual activities they are doing that you can measure with telemetry.

没错。

That's right.

我们在那里看到的最重要的一点，我想你在这一集里已经提到过几次了，就是花在编码上的时间。

The number one thing we saw there, which you you, I think, mentioned here and there in this episode already was time spent coding.

而且，你猜怎么着，那时候这个时间其实挺低的。

And we were also, lo and behold, it was pretty low back then.

在微软，你会觉得，你是软件开发者，所以大部分时间应该花在开发上。

At Microsoft, it was like, yeah, you're a software developer, so like most of your time should be spent developing.

但实际上并不是这样。

But it's actually not the case.

核心开发时间其实……

Like the core development time was...

这不是一天中最主要的活动。

It's not the number one activity in the day.

这让很多开发者感到失望。

Much to the dismay of many developers.

现在很多研究也陆续证实了这一点，确实，很多开发者每天花在编码上的时间可能只有13%、14%或15%。

And now lots of studies have sort of come along and reconfirm that, that, yeah, time spent coding, like, you know, may only be 13, 14, 15% of the day for many developers.

没错。

That's right.

是的。

Yep.

还有一点很有趣，我们发现，总的来说，这是一个很大的驱动因素，但我们也看到了一些其他因素，比如你是否在值班，或者我记得微软称之为DRI，也就是指定响应人。

What's also interesting, we we did find that, I think, overall, this was a large driver, but we also saw things like whether you're on call or I think Microsoft calls the DRI designated response individual, I believe.

这已经有一段时间了。

It's been a while.

这导致了……

That that made...