Ryan Lopopolo: OpenAI's Framework for Shipping Code at 70 PRs/Week

我们希望Codex能够连接到我们的Electron应用。

We wanted Codex to be able to connect to our Electron app.

我们最初是通过Chrome DevTools协议和MCP来做的。

Chrome DevTools protocol, MCP was how we started out.

后来，我稍微看了一下代码，发现完全不一样了。

Eventually, I kinda took a peek at the code, it was entirely different.

不过，Codex仍然通过Chrome DevTools协议连接到Electron应用，但实际上是一个本地TypeScript守护进程在运行，它提供了一个小的CLI接口，而不是MCP，因为我们发现我们只需要两三个工具调用。

Still, Codex was connecting to the Electron app over Chrome DevTools protocol, protocol, but it was actually a local TypeScript daemon that was being spun up that was providing a little CLI interface instead of the MCP because we found that there were only two or three tool calls that we actually needed.

这样更节省上下文。

It's more context efficient.

速度也更快。

It's faster.

而且这一切在我不知情的情况下就发生了，我的工作流程没有被打断。

And this had happened underneath me without me knowing my workflow was not interrupted.

同样的依赖关系和实体关系依然存在，但我几乎不需要知道它是怎么实现的。

That same dependency edge and entity relationship existed, but I almost didn't have to know how it was implemented.

你觉得这很酷还是让人担心？

Did you find that cool or worrying?

我觉得很震惊。

I found it shocking.

《AI原生开发者》是一档面向开发者和工程负责人的播客，聚焦人工智能和智能体编程的前沿领域。

The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding.

每周，主持人盖伊·帕贾尼和我，西蒙·梅普尔，都会与人工智能领域最激动人心的声音对话，探讨当今开发者面临的最大问题。

Join your hosts, Guy Pajani, and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today.

这里是《AI原生开发者》。

This is the AI Native Dev.

大家好。

Hey, everyone.

希望你们到目前为止喜欢这期节目。

Hope you're enjoying the episode so far.

我们的团队在幕后非常努力，为你们带来最好的嘉宾，这样我们就能进行关于智能体开发的最有见地的对话。

Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.

无论是讨论最新的工具、最高效的工作流程，还是定义最佳实践。

Whether that's talking about the latest tools, the most efficient workflows, or defining best practices.

但不知为何，你们很多人还没有订阅这个频道。

But for whatever reason, many of you have yet to subscribe to the channel.

如果你喜欢这个播客，希望我们继续为你带来最好的内容，请帮个忙，点击订阅按钮。

If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button.

这真的很有帮助，能让我们继续提升嘉宾的质量，为你打造更好的产品。

It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you.

好了。

Alright.

回到正题。

Back to the episode.

大家好，欢迎收听新一期的《AI 原生开发者》，我们正在伦敦的 AI 原生开发者大会 2026 现场。

Hello, and welcome to another episode of the AI Native Dev, and we're here live at AI Native DevCon twenty twenty six in London.

今天和我一起的是 Ryan Lopopolo，他是 OpenAI 的技术人员。

And joining me is Ryan Lopopolo, who is a member of technical staff at OpenAI.

那么

So

嘿，大家好。

Hey, folks.

Ryan，你好吗？

Ryan, how are you?

我很好。

Doing great.

很高兴来到这里。

Excited to be here.

这次会议气氛很热烈。

The conf is popping.

真的很有趣。

It's really fun.

我今天晚些时候要上台，我已经迫不及待了。

I go on stage later today, and I'm just excited to get after it.

没错。

Absolutely.

很期待。

Looking forward.

跟我们聊聊你稍后要做的那个分享吧。

And tell us a little bit about the session that you're giving later.

我要讲的是关于 Harness Engineering 的一切，这算是今年早些时候我参与定义的一些讨论吧，就是讲怎么设置代理，让它们能在你的代码库里做高度复杂、自主的软件工程工作，而且要有很高的信心，如果能做到的话，几乎是无人值守的状态。

So I'm gonna be talking about all things Harness Engineering, which is sort of, I guess, kinda like some of the discourse that I helped define earlier in the year here where talk about what it means to kind of set up agents to do highly complex, autonomous, software engineering like work in your code bases with high confidence and almost headless nature if you can get to it.

需要很多花哨的上下文管理技巧，才能让这些代理从自己的错误中学习，产出高质量的代码，并且能在没有人工干预的情况下长时间运行。

Lot of fancy context management techniques in order to get these agents to learn from their own mistakes, produce high quality code, and go for a very long time without human intervention.

没错。

Absolutely.

我们会聊聊 Harness Engineering。

And we're gonna talk about harness engineering.

我们会谈谈你在 OpenAI 内部用的一些方法，也会聊聊哪些部分很有趣，哪些部分有点吓人，肯定会很有意思。

We'll talk about some of the methodologies that you've used internally at OpenAI and and talk through maybe some some of the bits that have been enjoyable, some of the bits that have been scary, and it's gonna be great fun.

那么，Ryan，我们为什么不从 Harness Engineering 开始呢？

So so, Ryan, why don't we kick off with Harness Engineering?

对于外行人来说，什么是 Harness Engineering？

For the for the layman, what is Harness Engineering?

我们有这些很棒的技术和编码代理，还有这些非常聪明、能生成大量代码的出色模型。

So we have these wonderful technology and these coding agents, and these fantastic models that are very smart and able to produce a ton of code.

而 harness engineering 就是这样一个理念：要生成可接受的、我们信任的、能在业务系统中建立信心的代码，在生成 PR 和系统的过程中，我们需要做出很多小决策。

And harness engineering is this idea that to produce code that is acceptable to produce code that we trust, that we can build confidence in business systems around, there are a bunch of little decisions that go into the process of producing PRs, producing systems that we have to make along the way.

为了让代理知道并理解这些决策，我们必须把它们写下来。

And in order to get the agent to know and understand those, we have to write it down.

我们必须确保这些选择的内容，以及所有构成好代码的非功能性需求，在正确的时间呈现给代理。

We have to make sure that like the content of those choices, all those nonfunctional requirements that go into making good code, get surfaced to the agent at the right time.

所以，这既要把上下文放到仓库中，让编码代理能看到，又要在正确的时间呈现给代理，还要确保我们生成的所有代码都遵循我们生产高质量软件的金线标准。

So it's a combination of getting context into the repository in a place where these coding agents can see it, surfacing it to the agent at the right time, and making sure that all the code that we produce kind of like follows the golden thread of what it means for us to produce high quality software.

审查代理、测试、lint、大型重构循环，这些都是让代理闭环的方式，确保它向我以及团队其他成员证明，它生成的代码是可接受的，可以合并。

Reviewer agents, tests, lints, big refactoring loops, all sort of ways to get the agent to close the loop and make sure that it proves to me and the rest of the team that the code it has produced is acceptable and can be merged.

现在我们已经有提示工程、上下文工程和约束工程了。

So we've had prompt engineering, context engineering, and harness engineering now.

你觉得上下文工程和约束工程之间主要有什么区别或者重叠的地方？

What would you say are the kind of like the main differences or overlap between context engineering and harness engineering?

我觉得实际上

I think really

你与智能体交互时只有两个杠杆：你提供哪些上下文，以及你提供哪些工具。

the two only levers you have when interacting with agents is which context you provide and which tools you provide.

而约束工程是这两者的结合，目的是确保我们提供给智能体的上下文是正确的，这样它就能发挥推理能力，生成好的代码。

And harness engineering is kind of a combination of both with the goal of making sure the context that we give the agent is correct so that it can work its reasoning magic to produce good code.

我认为约束工程的一个独特之处在于，我们实际上利用工具调用本身来向智能体注入提示，以管理它的上下文窗口。

And one thing that I think is unique about Harness Engineering is we're actually using the tool calls themselves to prompt inject the agent to manage its context window.

这里很酷的一点是，我们在代码库中组织测试、lint 和格式良好的测试的方式，与我为人类组织这些信息的方式非常不同，对吧？

One thing that's pretty neat here is the way that we structure tests and lints and well formatted tests in the code base is actually very different than how I would structure those messages for a human, right?

我会给人类一份详尽的失败列表，让他们在日志中仔细查找，以找出问题所在。

I would have an exhaustive list of failures that I give the human and have them wade through the logs in order to figure out what's wrong.

但对于智能体来说，由于它们调用工具的方式，我们希望这些消息尽量精简，但又要语义明确。

Whereas with agents, due to the nature of how they invoke tools, we want those messages to be somewhat compressed, but also semantically meaningful.

比如，与其给出机械的ESLint报错，我可以给一个PR错误信息，说你在某个文件里这样搞砸了，你知道的，请打开它，按照xyz文件里的操作手册去修复。

Rather than giving sort of a mechanical ESLint failure, for example, I can give a PRs error message that says, you screwed it up in this way, in this file, you know, please open it and follow the runbook in file x y z to go fix it.

因为我们见过你犯过这个错误，而且我们知道可以信任你去做正确的事。

Because we've seen you make this mistake before and we know that we can trust you to do the right thing.

这种层面的整合，这种更深入的整合，真的能解锁下一步。

And that that that level of integration, that deeper integration really unlocks that next step.

现在，有意思的是，我在看一些……

Now, interesting, I'm I'm, like, looking at some of the...

说到你的背景，你组建了Stripe的效率工程团队，在Brex领导了350名工程师的开发者生产力工作。

So some of your background here, you built Stripe's efficiency engineering team, led developer productivity at Brex for 350 engineers.

你有很多基础设施和开发者体验方面的背景。

There's a bunch of kinda, like, infrastructure and DX background there.

你觉得这些经验对你有帮助，还是反而成了阻碍？

Would you say that helped you or hindered you from the point of

从Harness工程的角度来看？

view of Harness Engineering?

当然有帮助。

Definitely helped.

这让我习惯了通过他人来推动工作。

It kind of got me used to this idea of working through others.

你知道，当我要提升一个350人工程团队的生产力时，我不可能亲自去审查每一个PR的细节。

You know, when I'm trying to improve productivity for an engineering organization of 350 engineers, I can't be hands on in the weeds of every PR that's being produced.

我也不能直接去修改底层系统。

And I can't work directly on the underlying systems.

我必须在幕后引导，确保默认情况下事情朝着正确的方向发展，建立合适的系统，让整个组织有足够的信任去做好工作。

I kinda have to steer from the background, make sure that the right things are happening by default, the right systems are in place in order to have high trust in the organization to do good work.

这正是高效使用这些代理所需要的。

And that's exactly what it means in order to be high functioning with these agents.

要想超越结对编程的编码代理，实现大规模并行，比如同时打开15个终端窗口，我必须在某种程度上放手。

In order to move beyond sort of coding agents as pair programming assistance to go massively parallel where I've got the 15 t mux windows open, I need to be able to take my hands off the wheel to a certain extent.

所以这种运作方式，这种系统思维的思路，对我来说挺自然的，因为这正是我职业生涯中一直在积累的专业能力。

So this way of operating the sort of systems thinking mentality kind of comes pretty naturally to me because that's kind of the expertise I've had to build up over the course of my career.

有意思。

Interesting.

嗯，其实我想稍微聊一下这部分，就是，你需要多少信任才能同时开那15个窗口？

Well, actually touch a little bit about, I guess, the part of that, which is, well, how trust do you need then to have those 15 open?

不过我们一会儿再聊这个。

But we'll cover that in just a second.

我很想……

I'd love to...

你知道，当你同时开着那15个窗口时，显然有大量由代理编写的代码，而且在工作流和流水线中扮演着各种不同的角色。

You know, when you have those 15 open, there's obviously a huge amount of agent written code and various different roles in the in the in the workflow and the pipeline.

你给自己和团队设定了一个非常有趣的限制，就是不能有人类编写的代码。

Now a really interesting constraint that you imposed on yourself and the team was no human written code.

没错。

That's right.

而且，我想首先问一下，当时你说‘让……’的时候，你有多害怕？

And, I guess, first of all, how how fearful were you at the time when you said, let...

这个项目里谁都不许写代码。

No one's to write any code here in this project.

你当时有多大把握觉得这是正确的做法？

How confident were you that that was the right course?

这确实在当时算是个挺激进的想法。

It was definitely a bit of a radical idea at the time.

差不多在我刚开始的时候，用这种方式构建，甚至在我们有像样的推理模型之前，GPT-5是在2025年8月发布的，而我从6月就开始用这种放手的方式工作了。

Kinda when I started out, building in this way was, even before we had proper reasoning models, GPT five came out in August 2025, and I started working in this hands off way in June.

所以那时候还是o3时代，Codex Mini刚出来，还有Codex CLI的最初版本。

So this is, you know, o three era Codex Mini had just came out, the very first versions of Codex CLI.

说实话，那时候的模型能力差远了。

And to be honest, back then the models were way less capable.

要这样完全放手操作，过程非常痛苦。

It was very painful in order to operate hands off like this.

所以，出于必要，和模型、和编码代理一起工作时，我不得不把自己当成一个非常笨重的工具。

And sort of by necessity, working with the model, working with the coding agent, I kinda had to stick myself in as a very chunky tool

让模型可以委托任务给我。

For the model to delegate to.

如果它卡住了或者需要做点什么，一开始它唯一能做的就是让Ryan帮它做。

If it got stuck or if it needed to do something, the only thing it could do at the beginning was ask Ryan to do something for it.

后来，它总是问同样的事情，我就烦了。

Eventually, get tired of being asked the same thing all the time.

最开始，那会儿是用cargo安装依赖。

Initially, back then, it was installing dependencies with cargo.

那时候Codex根本没法可靠地完成这件事。

This is just not something that that Codex was able to do reliably back then.

所以我就在终端里敲命令，然后给代理一些依赖让它用。

So I would type the commands in my terminal and, you know, give the agent some dependencies to work with.

自动化这个过程就是第一步。

And sort of automating that was the first step.

这让我进入了一种模式，非常关注我把时间花在哪里，想办法构建一个工具调用，省去我花在那上面的时间。

And it kind of got me in this mode of paying very close attention to where I'm spending my time, figuring out ways to build up a tool call that removes the need for me to spend that time.

这样我就能找出其他方式，帮助智能体获得更多更强大的工具，提升它的能力。

So then I can kind of figure out the other ways that I can help the agent get more and more chunky tools to give it more capability.

从零开始，你真的能切身感受到智能体的失败。

And to start with zero like that, you kind of viscerally experienced the failure of the agent.

我觉得，要真正了解工具调用，以及智能体在写代码过程中犯的错误，这对建立信心非常重要。

I think to kind of have a good sense of the tool calls as they are making the mistakes the agent is making sort of in the process of writing the code is really, really important in order to build confidence in

对。

it.

因为看到它犯的错误，我也能看到它做得好的地方，然后开始调整我一周的安排，去利用Codex擅长的部分。

Because to see what mistakes it's making, I'm also able to see the parts that it does really good at and can start to retool the way I go about my week to lean into the stuff Codex is really good at.

它非常擅长遵循指令。

It's really good at following instructions.

它非常擅长写测试并运行它们。

It's really good at writing tests and calling them.

所以把很多工程流程都铺成这种非常顺畅的路给智能体走，这样我就能对它建立信心。

So collapsing a lot of the engineering process into these like very well paved roads for the agent helps me build confidence in it.

你这么说挺有意思的。

It's funny how you say it.

它确实很擅长遵循指令，也很擅长写测试。

It's really good at following instructions and really good at writing tests.

我在想，哇，这跟大多数开发人员完全相反。

I'm thinking, wow, the complete opposite of most developers.

也许这么说不太对。

Maybe that's wrong to say.

但这真的很有意思。

But so that's really interesting.

而且我想，当你考虑你的团队时，这是你做的决定，但当你考虑你的团队时，每个人有多...

And I guess when you think about your team, like it's a decision that you made, but when you think about your team, how was every...

大家有多愿意加入这个？

How onboard was everyone in joining this?

我很幸运，随着团队扩大，我们招的都是新员工。

I got very lucky that as we grew the team, we grew it all with new hires to the company.

所以他们一进来就被扔进这个疯狂的环境，心里想，哦，看来这就是我们现在的做事方式。

So they kinda get dropped into this insane environment and being like, oh, guess this is just the way we do things now.

那我就全力以赴吧。

Let me just go whole hog at it.

通常人们需要大约两周时间才能适应这种工作方式，在团队扩张初期也有一些成长的烦恼。

It usually took about two weeks for folks to ramp up to this way of operating some growing pains very early on in growing the team.

可能还吃了不少狗粮吧。

And probably a good amount of dog food eating as well.

你还能回到团队里去。

You were able to go back to the team.

其实我们当时做的产品也帮了不少忙。

I mean the product we were building too also helped a bunch.

我们在构建一个数据分析师代理，用Codex来写SQL、调用Databricks等等，这意味着我们写代码用的很多技术，也正是让这个代理有效运作所需的技术。

We were building sort of a data analyst agent, a data analyst agent using codex inside in order to write SQL and call Databricks and all these sorts of things means a lot of the techniques that we were using to write the code were techniques we needed to deploy into this agent to have it be effective.

这真是个很棒的良性循环。

A very cool virtuous cycle there.

新员工入职大概需要两周时间，一开始非常痛苦，因为要琢磨怎么指导大家别写出垃圾代码。

Took about two weeks for folks to onboard initially very painful in terms of figuring out how to coach folks to not produce slop.

用这些工具保持高速度的一个关键点，就是绝不允许垃圾代码进入代码库。

A big part of maintaining high velocity with these tools is just not permitting slop to enter the code base.

但一旦我们摸索出来了，等到招到第五、第六、第七个员工时，在两周内，我们真的看到速度和PR吞吐量提升了5%、10%、15%，这可不是通常快速入职团队时能看到的，尤其是在“前奇点”时代。

But once we sort of had figured it out, by the time we got our fifth, sixth, seventh hire onto the team, within those two weeks, we were actually seeing like velocity and PR throughput go up 5%, 10%, 15%, which is again, not the common path when you onboard quickly to a team, like pre singularity type.

原因在于，随着我们在代码库中积累越来越多的上下文和能力，每个人把Codex作为代码库的唯一入口，默认就能获得所有人的最佳成果。

And the reason for that is because as we accrued more and more context, more and more capability into the code base, sort of everyone invoking codex as the sole entry point to the code base was getting the best parts of everybody by default.

不用花上一到三个月去吸收团队的最佳实践，这些实践已经在那里了。

Rather than having to spend one, two, three months absorbing the best practices of the team, they're already there.

这意味着新员工能很快贡献他们的最佳判断和上下文，所以大家都能非常迅速地提高效率。

Which means new hires to the team were able to very quickly supply their best judgment and context, which means now everybody is being more effective very, very quickly.

你已经在通过这个代理来让新员工入职了。

You're onboarding the new hire through the agent already.

没错。

That's right.

太棒了。

Brilliant.

有个很好的说法，你肯定听过，我也说过无数次了。

So there's a really nice phrase, I'm sure you've heard, I've said it a million times as well.

你可以快。

You can fast.

只有休息好了，才能跑得快。

You can only go fast when you have breaks.

对。

That's right.

现在说到……

Now going...

在这种情况下，快就是利用编码工具和AI代理，以更快的速度帮我们生成代码，这样……

In in this scenario, going fast is using, you know, codecs and AI agents to to generate that code for us at a faster velocity that...

比我们手动写要快得多。

Than than we would be able to manually write.

当然，刹车就是那些测试和审查，它们验证代码，指出我们需要花时间修复和调整的地方，然后才能上线。

The brakes, of course, are those tests, those reviews that that, you know, validate the code that show and highlight where we need to actually spend time and fix and adjust, before we go live.

另一个有趣的点是零人工审查。

Another interesting thing, that is the zero human review.

这几乎就是……

Now this is almost...

现在你让AI来踩刹车，这有点让人担心。

You're you're putting AI to beat the brakes now, which is which is a little bit more worrying.

从某种角度看，这算是一个挺吓人的提议。

In in terms of, I guess, you know, an alarming an alarming proposal.

当你说“好，我们现在要对这段代码做零人工审查”的时候，工程师们有多自在？

How comfortable were the engineers when you said, right.

我们现在要对这段代码做零人工审查。

We're now gonna do zero human review on this code.

我们要……

We're gonna...

我们希望，你知道，让智能体能够做那种审查。

We want, you know, agents to be able to do that review.

所以，这有点像合并后的代码审查，如果真有那么回事的话，其实不是什么新鲜事，对吧？

So sort of like the idea of post merge code review, if at all sort of thing is, you know, kind of like not a new thing, right?

我职业生涯刚开始的时候，大概2010年代初，这就是写软件的一部分常态，就是和团队建立高度信任的环境，偏向行动，偏向吞吐量，然后，你知道，抽查一部分代码看看。

When I started my career back in the early 2010s, like this was a normal part of what it meant to write software is kind of have a high trust environment with your team, bias to action, bias to throughput and, you know, have some representative sample of the code that you are looking at.

但除此之外，团队之间要有大量的同步沟通，确保大家对系统的心智模型是一致的。

But, you know, otherwise have lots of synchronous communication as a team to make sure that everyone has the same mental model of the system.

现在这种模式因为智能体又流行起来，真的很酷，因为如果代码库的入口总是智能体，那团队里每个工程师就不需要完全掌握系统的完整心智模型了。

And it's very cool to kind of like have this come back into vogue with agents because if the entry point to the code base is always the agent, it's really not required for every engineer on the team to have a full and complete mental model of the system.

只要他们能信任团队里的其他人都在让Codex发挥作用，

So long as that they are able to trust that everyone else on the team is enabling codex

去

to

具备那种专业水平。

have that level of expertise.

这并不是说系统里完全没有审查。

Which is not to say that like there is no review at all in the system.

为了更细致一点，我们确实有些情况要求合并前进行传统的双人审查。

Like to kind of provide a little bit more nuance here, we do have some cases where we require, like, two party traditional review pre emerge.

大多数时候，这集中在那些高度复杂的计划、跨周进行的阶段里程碑这类事情上。

And most of the time that is centered around things like highly complex plans, sort of phase milestones that would take place over the course of an actual week, these sorts of things.

原因在于，这些计划、里程碑、具体方案实际上就是你要给智能体的提示词。如果你依赖文本来驱动实现，那么措辞是否清晰、是否完全覆盖或错误描述了任务，就非常重要，因为如果一开始就描述错了，你得到的就是垃圾结果。

And the reason for that is these plans, these milestones, these exact plans function as the prompt you're gonna give the And if you are relying on a text document to drive implementation, being crisp around what words are in there and whether or not they fully under or misspecify the task is quite important because if you misspecify the task upfront, you are gonna get garbage.

所以这就是我们主要投入人工审查的地方。

So that's kind of where we focus the bulk of the human review.

而且这里的审查，和我们通常理解的代码审查不太一样。

And to the review here, like, it's different to how we would normally think of a code review.

首先，代码审查时，我们会争论细节、命名，甚至更多是风格指南式的评论，或者说有时候确实会这样。

It's like, first of all, a code review, you know, we will argue about details and namings and things like, you know, almost more style guide style comments or rather that's what sometimes happens.

很不幸，是的。

Unfortunately, yes.

而现在我们看的其实是更……

Whereas now what we're looking at is actually a much more...

我觉得可能有几个原因。

I guess there's probably a couple of reasons.

第一，因为从更高的层面去审查本身就更有意思，因为从我们的角度来看，这些输入更有意义，我们需要更多地思考我能在这里提供最大的价值是什么。

One, because it's actually more interesting to actually review at that higher level anyway, because they're more meaningful input from our point of view where we need to kind of like much more think about what's the greatest value I can provide here.

而且，正是这种更高层的决策才更有意义。

And it is that more meaningful decision making at the top.

但另外，我想我们可能离代码更远了。

But also, guess we're probably more abstracted away from the code.

所以有一部分我们只能选择信任，因为如果我们深入实现细节，那几乎是在浪费时间，我们需要把它抽象出来，并且需要有测试机制来确保它成功。

So there is an amount that we just have to trust where actually it's almost a waste of our time to go too deep into the implementation because we need that to be abstracted away and we need mechanisms of where testing just occurs to make sure that that succeeds.

没错。

That's right.

这又回到了那种团队技术负责人的思维方式，对吧？

This is sort of, again, that sort of like group tech leads sort of mentality, right?

比如我在Bricks负责开发者生产力的时候，对吧？

Like when I led developer productivity at bricks, right?

我并不会深入到每个团队写的每一段Terraform代码里去。

Like I don't have my hands in the weeds of every bit of Terraform every team is writing.

但我确实会在宏观层面关注一些指导原则，对吧？

But I do care at a high level about like guiding principles, right?

我希望基础设施是可复现的。

I want the infrastructure to be reproducible.

我希望它能快速部署。

I want it to provision quickly.

我希望我们有现成的模块来解决常见的开发者工作流。

I want us to have modules in place that solve common developer workflows.

但至于你底层模块怎么组织，或者你是否用惯用的方式使用for each，这些我倒不太在意。

But like how you structure your underlying modules, whether you're using for each in an idiomatic way, don't really care too much about that.

只要代码在局部上是连贯的，并且真正解决了我想解决的业务问题就行。

As long as the code is like locally coherent and actually solves the business problems that I'm hoping to do.

而且理想情况下，这个决定甚至可能随着时间而改变，取决于某件事应该如何构建。

And ideally that's a decision that maybe even change over time, depending on how something should be structured.

所以这不是我们应该一直紧紧抓住不放的东西。

So it's not something we should always be so connected to at all times.

总之，我觉得我们稍微提到了信任这个话题。

Anyway, I think we mentioned kind of like trust a little bit.

我不记得信任的神奇比例是多少了，关键不在于某件事成功了多少次，而在于它失败的频率，这才是破坏信任的关键。

I can't remember what the magic ratio of trust is where it doesn't matter how many times something works, it's how frequently something doesn't work is the key, which breaks the trust.

但我想知道，你和团队在哪个阶段真正开始觉得，哦，其实你知道吗？

But I guess at what stage were you and the team, like, truly thinking, oh, actually, do know what?

我完全信任这个。

I completely trust this.

不是，你知道，从我的角度来看，我在这里说的是自己的懒惰，但有时候我在使用代理式开发时，只是假设一切正常，懒得深入审查，不够仔细。

Not, you know, I guess from my point of view, I'll talk about myself here when I say laziness, but there is a time when I'm using agentic development where I'm just kinda assuming and just being lazy about not reviewing things in enough depth.

但说到真正的信任，就是，是的，我打算把这个交给智能体，并且相信它质量很高，可以直接推到生产环境。

But in terms of like true trust is in, yeah, I'm gonna hand this off to the agent and believe that this is gonna be high quality that I can actually push to production.

你觉得在过程中哪个时刻，你会说你是真正信任所交付的东西的？

At what moment would you say during the process, would you say you were truly trusting of what was being delivered?

我觉得这里有两个阶段。

I think there were two sort of phases here.

一个是，我是否信任智能体能写出好的代码？

One was, do I trust the agent to be able to produce code well?

那个神奇的时刻是在开发一个非常早期的版本，也就是今天Codex应用的前身。

And the magic moment for that was sort of working on what was essentially a very early version of what the Codex app is today.

跟队友聊天时说，天哪，我真希望这里有语音输入。

Talking with a teammate around, man, I wish I had voice input here.

然后只需要给一个提示，半小时后就能在真实世界的应用里看到效果，那真是太棒了。

And then being able to just give a prompt and half an hour later, see that in the real world in the app was fantastic.

你会产生一种信心，觉得这真的取决于我安排提示词的能力，把这些提示词送进这个神奇的机器，让我的想法变成现实。

You kind of got this feeling of confidence that it really is down to my capacity for scheduling prompts to go into this magic machine to see my ideas come to life.

所以这算是第一阶段，就是进入这种心态：我们作为工程师的角色，应该是想办法给机器喂料，安排工作，让我们的愿景得以实现。

So that was sort of like the first phase of getting into this mindset of our role as engineers should be to figure out ways to feed the machine, to schedule work in order to kind of have our vision come to fruition.

接下来的阶段就是，这些代码可信吗？

The next sort of phase was, is this code trustworthy?

质量高吗？

Is it high quality?

这个其实是团队出于需要才想到的。

And this sort of came to us as a team sort of out of necessity.

当我们第三位工程师加入团队后，我们的PR处理量大幅增加了。

When we onboarded our third engineer to the team, We significantly increased our PR throughput.

但与此同时，我们跟不上审查的速度，也没法确保代码库里没有混入垃圾代码。

But at the same time, we're unable to keep pace with reviews and making sure that swap didn't enter the code base.

我们开始有点放慢节奏，就像你提到的，每周五花一天时间做垃圾清理，在这种工作方式下，有很多长时间的同步站会，对吧？

We kinda started to kind of pump the brakes a bit, like you mentioned, and take one day a week, Fridays to do garbage collection where a lot of long synchronous standups in this way of working, right?

就是为了让大家接受这种思维模式。

In order to socialize that mental model.

所以我们会在这一周里整理一个清单，把那些不太满意的地方记下来。

So we would kind of compile a list over the course of the week of things that we didn't quite like.

我们倾向于追求吞吐量和合并速度，但同时也想确保我们的质量标准不降低。

We want to bias toward throughput, bias toward merging, but still we want to make sure we maintain our quality bar.

所以每个星期五，我们都会努力清除那些不好的代码，同时想办法在以后系统地避免这些问题。

So every Friday was trying to eliminate that slop, but also finding ways to systematically eliminate it going forward.

我们从来不想重复给出同样的反馈，就像你希望新加入团队的工程师真正理解什么是做好工作一样。

We never wanted to give the same feedback twice in the same way that you want, you know, a new engineer you're onboarding to the team to actually internalize what it means to do a good job.

所以这些就逐渐变成了程序化的防护措施，再加上一个很长的外部循环，也就是那些自动化的CI任务，它们专门用来查找和识别那些不好的代码，整理出一份我们写好代码的黄金原则清单，然后在代码库中找出偏离这些原则的地方，并提出合并请求。

So that kind of stacked into sort of these programmatic guardrails and then a very long outer loop of sort of these automated CI jobs that are looking for slop, identifying slop, kind of compiling a list of golden principles for what it means for us to write good code, finding divergences from it in the code base, proposing PRs.

到了那个时候，我们就发现，哦，这些代理已经能按照我们的期望来写代码了。

And at that point we were able to see, oh, the agents are able to write code the way we expect.

我们只需要建立合适的循环和系统，让它自己就能完成这个反馈循环。

We just kind of need to put the right loops and systems in place for it to kind of close that feedback cycle for itself.

那这大概是通过上下文或者技能之类的给它提供的吧。

And presumably that was given to it by context or skills or something like that.

在技能出现之前。

Before skills were skills.

我们

We

那时候还没有技能。

had That was a time before skills.

对，我知道。

Yeah, let's I I know.

也就八个月前吧。

Only what, eight months ago at this point.

我们当时用了个很简单的办法。

We kind of did a very cheap hack.

我们就在GitHub上开了一个issue，工程师和智能体都在上面留言，讨论好的软件开发的基本原则。

We kind of had just a GitHub issue where engineers and agents would leave comments around sort of underlying principles of good software development.

比如怎么正确地写React快照测试。

How to properly make a React snapshot test.

写可靠的网络代码意味着什么。

What it means to write reliable network code.

怎么考虑构建一个lint，最后你会得到一个有100到150条评论的大问题。

How to think about structuring a lint and you end up with this big issue with 100, 150 comments on it.

这基本上就是代理在代码库中爬行、对一系列违规行为进行排序的种子。

And that was sort of the seed for what it meant for the agent to go sort of spider through the code base, stack rank a set of violations Mhmm.

提出拉取请求。

Propose PRs.

然后一个有趣的事情是，我们有一个值班人员，他的工作是监督所有代理生成的东西，那些无头的东西，因为我们还不太信任它。

And then one interesting thing there is we had like an on call whose job it was to supervise all the agent produced stuff, the headless stuff, because we we didn't quite trust it yet.

我们需要有人盯着它来建立这种信心。

We needed eyes on it in order to build that confidence.

人类给出赞或踩，决定合并与否，留下审查反馈，这意味着下次那个反草率循环启动时，它会查看自己生成的所有拉取请求。

Human with thumbs up, thumbs down, merge or not, leave review feedback, which means the next time that anti sloppification loop spins up, it looks at all the PRs it had produced.

它吸收这些人类反馈，与上次运行时的会话日志进行比较，然后说：我犯了哪些错误？

It ingests that human feedback, compares it with the session logs from the last time it ran, says, which mistakes did I make?

我漏掉了哪些优先级？

Which priorities did I miss?

我是不是生成了不对齐的代码？为什么？

Did I produce misaligned code and why?

让我想想以后怎么避免这种情况，写点 markdown，存到 GitHub 运行的产物里。

Let me figure out how not to do that going forward, produce some markdown, save it as an artifact in the GitHub run.

这样下次运行时，我就有了从真实人工反馈中学到的额外上下文。

So the next time I run, I have this extra context that I have learned from the actual human feedback in the And

这是正确的做法吗？

is that the correct way?

你会推荐别人这样设置他们的代理，让它们几乎像这样迭代吗？

Is that the way you would recommend others set up their agents to almost like iterate through?

所以这几乎就像先写代码，然后在事后做这些审计或审查，然后提出 PR，你几乎是在护栏之外操作？

So it's almost like build the code and then do these almost like audits or reviews in post and then raise PRs where you're almost, you know, you're outside of the guardrails?

还是说你现在会考虑在代码生成时就把它加进代理里？

Or would you today look at adding that into the agent at code creation time?

两者都有。

It's a mix of both.

两者都有。

It's a mix of both.

我们确实认为这种异步循环是高效工作的关键部分。

We do think sort of these asynchronous loops are a big part of what it means to be effective.

早期版本的Aardvark安全编码也是这么运作的，对吧？

Early versions of codec security via Aardvark also worked in the same way, right?

也就是说，我的代码合并到主分支之后，才进行安全审查。

Where I'm getting a security review of my code after it has ended up in main.

可能会发现漏洞，然后生成补丁。

It is maybe finding a way in which it is vulnerable, producing a patch.

我们提交一个PR，审查通过后合并。

We put up a PR, we review it, and it merges.

我觉得这种流程非常好，因为它允许错误进入主分支，这样我们就能从中学习。

And I think this sort of flow is really, really nice because it allows mistakes to actually make it to main, which means they can be learned from.

每次我们找到问题，第一反应就是赶紧修掉。

Anytime we are able to come up with a correction, one, we wanna fix it.

同时，那些反复出现的模式，我们会通过代理、循环、蒸馏这些手段来识别。

But also repeating patterns sort of thing, which we identify with agents and loops and distillation and all of that.

这些正是我们想提前引入到流程里的东西。

That's the stuff we wanna pull forward earlier into the pipeline.

那当你推送有错误或者想修正的内容时，现在是不是没那么担心推到生产环境了，因为你知道代理能很快审查并修复？

And when you do push with mistakes or things that you wanna correct, are you less fearful of pushing those mistakes to production these days because you know how fast those can be reviewed and fixed by an agent?

这确实是个程度问题，对吧？

It's definitely a spectrum, right?

我得说，这个项目我们并没有做持续部署。

I will say that for this project, we didn't have continuous deployment, right?

我们做的是原生应用。

We were building a native app.

我们手动切发布分支，然后做冒烟测试。

We are manually cutting a release branch and doing smoke testing.

最初的冒烟测试全靠人工，我们看看时间都花在哪儿了，然后试着让代理来做一部分。

That smoke testing is initially all by humans where we look at where we're spending the time and try and get agents to do some of it.

我觉得在这个领域，对发布流程保持一定程度的人工监督是很重要的。

It is important, think, to have some level of human supervision over the release process in this world.

但如果我再戴上技术组长的帽子，对吧？

But if I were to kind of like put my group tech lead hat on again, right?

有些错误是可以犯的。

Some mistakes are able to be made.

比如后果不严重，而且就像我希望那些越来越资深的工程师能从自己的错误中学习一样。

Like the consequences of them are low and much like I want engineers who are becoming more senior to learn from their own mistakes.

我也希望看到代理犯的错误，这样我们作为他们的监督者，也能想办法让他们学习。

I also kind of want want to see mistakes that the agents make so that we, as their supervisors, can figure out how to get them to learn as well.

因为感觉在这种场景下，债务和修复的成本很低，因为就是，我们会做个审计，然后……

Because it feels like in this kind of scenario, debt and fix is cheap because it's about, well, we'll run an audit and we'll...

只要你有那些护栏和规则，比如这些是我们的最佳实践，我就可以放一些不太理想的东西进去，而且不需要花六个月等人工来审查这段代码。

As long as you have those guardrails and rules in place to say, these are some of our best practices, I can put something in which isn't ideal and no, it's not gonna take six months for a human to come and review this code.

可以在更常规的时间范围内完成。

Can be done in a more regular timeframe.

在这个代码生产如此廉价的新世界里，我实际上认为，至少在与代理合作时，你可以把DevOps的左移理念颠倒过来。

Like with this sort of like new world where the production of code is so cheap, I actually think that you can, at least working with agents flip sort of like the DevOps shift left thing onto its head

有点。

a bit.

我能做的最便宜的事情就是直接修改我的提示词。

Where the cheapest thing I can do is literally produce a change to my prompt.

无论是那些我写进GitHub问题里的黄金原则，还是我和队友分享的提示词。

Whether that is like one of those golden principles that I'm eating into that GitHub issue or, you know, a prompt that I socialize with my teammates.

这是我能做的最便宜的事情，用来修复生成的代码。

That's the cheapest thing I can do in order to fix the code that is produced.

而且，如果我们发现错误有点系统性，我可以考虑把它进一步左移。

And, you know, if we find that, like, the mistakes are a little bit more systematic, I can kind of think about pulling that further left.

我可以添加一些文档，或者让一个审查代理根据这些文档来评判每个PR。

I can add some docs or I can operationalize a review agent that is going to judge every PR according to those docs.

或者我可以进一步左移，通过编写确定性的测试来提前发现这种行为，越早越好。

Or I can do more shifting left by making deterministic tests that suss out this behavior and on and on earlier in the process.

但我想从花费最少努力就能纠正错误行为的地方开始，因为这些是神奇的推理器，大多数时候它们只需要我很少的努力就能做得很好。

But the cheapest amount of effort I can spend to shake out that misbehavior is kind of where I wanna start because these things are magic reasoners that most of the time they do pretty well with very little effort on my part.

这非常有趣。

It's super interesting.

我几乎想跳过几个问题，因为我想问你一个问题，当你……

And I'd almost love to jump ahead a couple of questions because I kinda wanna ask the question when you...

你刚才说的那种方式，就是你在那边对提示做一点小改动，然后实现和其他的东西就能跟着变。

The way the way you talk about that whereby, you know, you make minor changes over here to a prompt and then, you know, implementations and things can just change.

这让我觉得，当我们谈论持久的东西时，重点不在于代码，而在于我们的提示、我们的上下文，还有我们的功能需求。

And it kinda makes me feel that when we talk about the thing that persists, it's less about the code, and it's more about our prompts and our context and our and our, you know, functional requirements.

这让我想到一个叫Symphony的东西，它是一个幽灵库，因为它是规格说明。

And it brings me to something called Symphony, which is a ghost library because it's spec.

它完全围绕规格和上下文，而不是实现或代码。

It's all about specs and context versus implementation or code.

我觉得这么一想，代码就变成了一个用完即弃的东西。

And I guess when think about that, the code just becomes a throwaway artifact.

实现只是今天存在，明天可能就变了。

The implementation is just, it's there today, it might change tomorrow.

但真正持久的东西，我们真正在乎并且应该花最多时间的，就像你说的，把那些评审提升到功能需求层面，其实是规格和提示词。

But the thing that persists, the thing that we care about and we should put most time in, like you say, lifting up those reviews more to the functional requirements and things like that is the specification and the prompts.

这是未来吗？

Is this a future?

当我们考虑规格和代码之间的关系时，今天这个关系变化有多大？

Is this like when we think about the relationship between the specs and the code, how much is that changing today?

所以

So

这挺有意思的，传统上在规格驱动开发中，对吧，你先有规格，然后随着代码的产生，你不断细化它，让它更具体，以找出规格中隐藏的决策和模糊之处。

it's kind of funny where you think traditionally in spec driven development, right, you start with a spec and then you kind of refine it as the code is produced sort of thing to make it more concrete to suss out those hidden decisions and ambiguity in the spec.

对于像 Symphony 这样的东西，对于我们产生的所有这些链接，对于关键用户旅程、产品功能、点击流程和界面的文档，我实际上发现先写代码要容易得多。

And for things like Symphony, for all these links that we're producing, for things like documentation on critical user journeys or product features or click flows and screens, I've actually found it is much easier to produce the code first.

让代码先摆出一个初步的框架，展示可能的样子，然后团队一起讨论完善，最后从中提炼出规格说明。

To have that code sort of like present a straw man for what the world could look like, engage with that as a team to refine it, and then distill the spec out of that.

因为产出的东西，不管是代码、电子表格还是谷歌文档，对吧？

Because sort of produced artifacts, whether they are code, a spreadsheet, a Google doc, right?

这些东西信息密度很高，包含了我们做出并认可为好的成果时所做的各种决策。

These things are very information dense in terms of the decisions that were made to go into producing something that we have accepted as good.

所以对于Symphony来说，我们最终发布的东西其实是一份规格说明，但我们最初是从一个用TypeScript写的、放在我们单仓库里的一个初步实现开始的，当我们觉得它不错、解决了我们想解决的问题时，我们就开始从中整理出一份规格说明，分享给全世界。

So for Symphony, the thing that we shipped is very much a spec, but the thing that we started with was kind of a vibe implementation of Symphony in TypeScript in our mono repo, where when we thought it was good and it was solving the problems that we wanted it to, we worked on producing a spec out of that to share with the world.

这个过程非常酷，我们有一个三阶段的流程。

And it was this very, very cool process where we had this three phase pipeline.

我们一开始用的是现有的Symfony，然后交给Codex，让它生成一份能够复现这个系统的规格说明Markdown文件。

We started with Symfony as it was, and we gave it to Codex to say, produce a spec markdown file that is able to reproduce the system.

最后我们得到了一份规格说明。

We end up with a spec.

然后把它交给一个新的代理，这个代理看不到原始代码，只拿到规格说明。

Give that to a new agent that says, without access to the original, it says, you have a spec.

请实现这个系统。

Please implement this system.

你知道，它会运行一段时间，带着目标去生成这个东西。

You know, it goes for a while with goal in order to produce this thing.

当它说完成时，我们把那个系统、规格说明和原始实现交给第三个代理，让它当裁判，说我们正在尝试复刻原版 Symphony。

When it says it's done, we give that system, the spec, and the original implementation to a third agent who's gonna act as the judge to say, we are attempting to replicate the original version of Symphony.

这是给代理用来生成的规格说明，这是他们最终做出来的东西。

This was the spec that was given to an agent to produce it, and this is what they came up with.

派生产物和原始系统之间有哪些不一致的地方，并提出对规格说明的修改建议。对。

Where are there misalignments in the derived artifact versus the original system and propose changes to the spec Yeah.

为了让下一次尝试做得更好。

In order to make the next attempt better.

那是一个非常消耗令牌的过程，但你知道，在令牌工厂工作嘛。

That's like a very token intensive process, but, you know, working at the token factory.

我猜你能拿到折扣。

I suspect you get a discount.

但是

But

最终你会得到一个非常精细的规格说明，它能够可靠地按照书面内容生成系统，同时仍然为这个幽灵库的使用者留出空间和模糊性，让他们能够根据实际的业务环境、实际的代码库、问题跟踪系统以及其他类似的东西进行调整。我觉得这挺酷的，因为这意味着我们最终会以非常紧凑、高细节的方式规定那些真正重要的部分，也就是业务逻辑关键部分，同时仍然保留灵活性，让人们可以根据自己的需要去演进和修改。

what you end up with is this really refined spec that is reliably able to produce the system as it is written while still providing space and ambiguity for consumers of this ghost library to adapt it to their actual business context, their actual repository and tool chain and issue tracker and all these other sorts of things, which I think is pretty cool because it means that we end up specifying in very tight, high detail the parts that actually matter, the business logic critical parts while still leaving the flexibility for folks to evolve and change it as they see fit.

我特别喜欢的是你刚才描述的那个过程。

And what I love about that is the process that you just described.

而且我觉得我们需要从软件工程多年来的经验中学习，并将其应用到智能体开发中。

And I think we need to learn from everything that we've done in software engineering for many, many years going into agentic development.

但你刚才描述的那个过程是一个非常纯粹的快速原型。

But the process that you just described is a very pure prototype.

就是快速复制……

It's us rapidly replicate...

快速开发我们的应用。

Rapidly developing our application.

这样我们就能获得必要的经验，从而能够说，哦，这个有效，那个不行。

So we make the learnings we need to be able to say, oh, this worked, that didn't.

其实，我更喜欢这样，或者咱们改一下界面，这最终会打乱很多流程。

Actually, I'll prefer it to be like this, or let's change the UI, which ultimately, you know, breaks the number of flows.

我们学到了这些经验，但不同的是，因为现在开发成本很低，我们可以说，我直接把这个扔掉就行了。

We make those learnings, but the difference is because development is cheap now, we are able to say, I can just throw this away.

我不会舍不得用那个，但我会试着把学到的一切和功能需求等等，都提炼到那份规格说明里。

I'm not tempted to use that, but I'm gonna try and distill everything that I have learned and the functional requirements, etcetera, into that spec.

我喜欢那种可迭代的，就像这样反复迭代，直到……

And I love that iterable kind of like that iterations through to make...

直到真正……

To to to really...

你知道，我见过实现和规格之间的差距，这样真的很好，能真正填补这些差距，确保规格说明涵盖了我们需要的一切。

You know, I've seen implementation spec gaps, and this is this is really good to be able to really fill that in to make sure that the spec captures everything that we need.

这个过程并不是什么新鲜事。

It's not like this process is new.

比如，想想我职业生涯中其他领域的情况，比如和数据科学合作，他们想先在Jupyter笔记本里拼凑一个模型来验证想法。

Like, if I think about other areas in my career where I've seen this, right, like partnering with data science, they wanna cobble together a model in a Jupyter notebook to prove out the idea first.

一旦他们有了自己确信的东西，就会交给工程团队去产品化，真正部署到MapReduce管道里，实现规模化运行。

And once they get something that they are confident in, then it sort of gets handed off to engineering in order to productionize it, to actually deploy it into the MapReduce pipeline to operate at scale sort of thing.

如果我们想的是在Figma里快速做设计原型，然后交给别人去产品化。

If we're thinking about, you know, doing rapid prototype of a design like in Figma and then handing it off in order to be productionized.

这也是同样的场景。

Also the same sort of context.

但现在，因为像你说的，代码生产成本很低，我们真的可以在生产系统里做这件事。

But now, because like you said, production of code is cheap, we can actually do that in our production systems.

在我们正在构建的这个应用里，我们实现这个想法的一个巧妙方式是，在开发阶段，它是一个Electron应用，我们有一个窗口界面，代理可以启动它，这基本上能给它提供我们正在发布的原生渲染画布，以及我们完整的设计系统和组件库，这意味着它实际上可以直接在我们部署的界面上原型化新屏幕，生成截图，交给我们的设计师，这样我们就能非常非常快速地验证我们的愿景是否与现实相符。

One neat way that we made that idea real in this app we are building is in dev, it's an electron app, we had a window screen that the agent could launch, which would basically give it the native rendering canvas that we're shipping as well as our full design system and component library, which means it's actually able to prototype new screens directly in the surface that we're deploying, produce screenshots of them, give them to our designer, and now it's a super, super tight loop on does our vision match the reality?

我们能否在我们将要发布的界面上实现这种体验？

Are we able to produce this experience in the surface that we're gonna ship?

这真是太不可思议了，是的。

Which is just wild and super, Yeah.

太棒了。

Super