构建RL环境的GitHub：Prime Intellect的Will Brown与Johannes Hagemann

如果数据是瓶颈，如果真正拥有专业知识是瓶颈，那么，你是希望历史上最聪明的人在你的公司工作，还是希望一个在那里工作了三十年的人？

If data is the bottleneck, if having the real expertise is the bottleneck, like, would you rather have the smartest person in history work at your company or someone who's been there for thirty years?

有时候你确实需要那个在那里工作了三十年的人。

Sometimes you really want the person who's been there for thirty years.

很多专业知识来自于对问题的深刻理解，以及长时间与问题互动。

There's a lot of expertise that comes from really understanding a problem deeply and interact with it over a long time.

而这正是训练中发生的事情，几乎不可能在简短的提示中复制。

And this is really what happens in training that is almost impossible to replicate in a short prompt.

你确实希望机构知识能够随着时间积累，最佳实践能够随着时间积累。

You really want the ability for institutional knowledge to compound over time, for best practices to compound over time.

这就是机构和公司变得强大和成功的方式——他们站在过去成就的肩膀上，而不是每天重新开始。

And this is how institutions and companies, grow to be really powerful and successful is they stand on the shoulders of what they've done before rather than kind of resetting every day.

我们希望任何公司都能做到这一点。

And we want to have this be accessible to any company that wants to do this.

我认为这就是我们思考如何应对的方式，尤其是随着软件变得更容易操作，编程的门槛变得更低。

I think that's how we've thought about approaching it, especially as software becomes easier for people to manipulate, as the barrier to entry for coding becomes easier.

我们看到AI研究领域也在发生同样的事情。

We see the same happening for AI research.

威尔和约翰内斯在Prime Intellect工作，这是目前AI领域最酷的Neo Labs之一。

Will and Johannes work at Prime Intellect, which is one of the coolest Neo Labs in AI right now.

你们的使命是让前沿实验室的训练对所有人都开放，我觉得这是一个非常崇高的使命。

Your mission is to make Frontier Lab training accessible to everyone, which I think is a very noble mission.

你们有非常强的品味，对开发者体验有很好的直觉，知道开发者真正关心什么。

You have really, really strong taste and just developer feel and just understanding how to, you know, that intuition for what developers care about.

你们推出的强化学习环境中心，真的非常与众不同，大家都很兴奋。

And then what you all launched with the Reinforcement Learning Environments Hub was like really, really differentiated and people were very excited about.

所以今天我很期待和你们聊很多话题，包括后训练、强化学习、智能体框架、你们的平台、RL中心，还有关于后训练和RL未来走向的大问题。

And so I'm excited to chat about many topics with you all today, post training, reinforcement learning, agent harnesses, your platform, the RL Hub, and then big picture questions on what's coming next in post training and RL.

这样可以吗？

Does that work?

听起来不错。

Sounds great.

当然。

Absolutely.

也许我们可以从这儿开始说起，你们是领先的研究实验室之一，帮助客户对他们的智能体进行后训练。

Maybe to get started, you are one of the leading research labs enabling customers to post train their agents.

你能告诉我那是什么吗？

Can you tell me about what is that?

那是什么意思？

What does that mean?

好的，当然可以。

Yeah, for sure.

我很乐意回答这个问题，并简要介绍一下我们平台的功能以及我们在Prime Intellect的研究。

Happy to take that one and give a bit of a high level overview of what our platform does as well as our research at Prime Intellect.

就像我们一开始提到的，我们努力让前沿基础设施对任何初创公司、企业，还有NeoLab这样的机构开放。

As we already mentioned at the beginning, we try to make frontier infrastructure available to any, like, startup enterprise and, yeah, NeoLab as well.

基本上，这些基础设施目前被锁在大实验室的高墙之内，基本上没有人能真正接触到它们。

And, yeah, basically, the infrastructure that, yeah, is currently locked behind the walls of the big labs where, yeah, nobody really has access to them.

而且，我们确实是从计算层和计算编排层开始，然后一路向上，一直到完整的后训练堆栈。

And, yeah, we really start from, like, the compute layer and the compute orchestration layer and go all the way up then to the entire full post training stack.

所以，从大规模强化学习所需的训练框架，到我们环境中心那种更社区化的环境，再到其他实际需要的部分，比如用于安全代码执行的沙箱，以及作为环境中心一部分的评估，所有这些都包括在内。

So everything from, like, the training frameworks that are needed to do large scale reinforcement learning to, yeah, the environments with, yeah, a bit of a more community approach of our Environment Hub to other pieces that are actually needed to do this like sandboxes for secure code execution and evaluations as part of our Environment Hub as well.

从某种意义上说，就是把它作为一个端到端的产品来提供。

And to offer this as an end to end product in a sense.

那么，为什么会有这样的直觉，为什么要追求这个使命，让所有人都能使用这些基础设施？

And what's the intuition for, for why even why pursue that mission statement of making all that infrastructure available to everyone?

是的，原因有很多。

Yeah, there's a lot of reasons.

我认为其中一个，也是我们非常热衷的，就是开放科学，这是人类前进的一种方式，历史上很多重大的科学发现都是我们讨论并作为世界共同基础来构建的。但更实际地说，模型定制也有很大的价值，我们看到的那些成功的应用，都是把AI用于特定的事情，用于某个智能体，某个工作流程，同时你还需要能够大规模地、以成本效益高的性能来交付。

Think one is, and I think something that we are very passionate about is just like open science as a way that humanity moves forward, where like a lot of the big scientific discoveries historically have been things that we, we talk about and as a world we can kind of build on top of, but kind of more practically speaking as well, there's a lot of value in model customization where we are like the winning applications are using AI for a specific thing, for some agent, for some workflow, where you also want to be able to deliver this at scale and with cost effective performance.

所以，真正要端到端地优化这些系统，关键是要能够直接访问模型权重，这样你就可以把模型塑造成最适合你问题的模型，而不是一个现成的模型，只能在提示词层面进行调整。

And so really the way to kind of really optimize these systems end to end is to be able to have access to the model weights directly, where you can then craft the model to be the best model for your problem rather than some model off the shelf where the crafting happens just in a prompt.

所以，这实际上就是让你能在比提示词层面更深的层次上进行定制。

And so it's, it's really just allowing a deeper layer of customization than what you can do at the prompt level.

那么你对未来的愿景是，每家公司都会自己预训练模型、自己后训练模型、自己微调他们的……

And then is your vision for the future then that, you know, every company will be pre training their own models, post training their own models, fine tuning their...

你觉得未来会是什么样子？

Like, what do you think the future holds?

我们确实认为每家公司都会成为AI公司，而且我们认为大多数AI公司都会想拥有自己的AI研究实验室。

We definitely think that every company will be an AI company, and we think most AI companies will want to have an AI research lab.

研究可以有很多不同的形式。

And research can look like many different things.

可以是预训练，特别是如果你所在的领域不只是需要文本进文本出，而是需要更定制化的东西。

It can look like pre training, especially if you're in a domain where you maybe don't just want text in, text out, if you want something more, bespoke.

在很多情况下，会是后训练，无论是智能体还是其他形式，针对特定任务和工作流来定制模型。

In a lot of cases, will be like post training agentic or otherwise, kind of focused models for specific tasks and workflows.

我觉得这已经快到了可以投入生产并且成本效益高的阶段，你可以真正让人们在规模化的情况下，针对他们想解决的合适类型的问题，非常实际地做到这一点。

And I think that's getting to the point where it's productionizable and cost effective, where you can actually make this very practical for people to do at scale for kind of the right shape of problems that people want to solve.

太棒了。

Awesome.

那你能简单说说你们的平台是做什么的吗？

And then can you say at a high level what your platform does?

我们有一个全栈研究平台，叫Lab。Lab的核心理念是让全世界任何想从事这类研究的人，都能拥有前沿研究实验室内部才能做到的能力。

So we have a full stack research platform called Lab, and Lab is about giving everybody the ability to do the things that a frontier research lab can do internally, but for anyone in the world who wants to do this kind of research.

而且，Lab非常强调一个概念，就是“环境”。

And, a big focus of lab is the notion of an environment.

我觉得很多人可能在强化学习的语境下听说过“环境”这个词。

And so I think a lot of people have heard the term environment in the context of reinforcement learning.

对我们来说，环境确实是个重点，也就是说，强化学习环境把做强化学习所需的一切都封装起来，让模型可以通过试错来改进，但它的用途其实比强化学习更广泛。

And that's a big focus of it for us, which is that an RL environment is encapsulating the things you need to do to do reinforcement learning, where you can have a model improve via trial and error, but it's also more general beyond just reinforcement learning.

如果你之前没听说过环境，其实它本质上就相当于大家讨论新模型发布时提到的那些评估基准。

And I think if you haven't heard of an environment before, it's essentially the same thing as the evals that get reported when people talk about new model releases.

比如SWE bench、AMI、Terminal bench，这些都是环境的例子，它们包含一个任务数据集、一个供模型运行的测试框架，还有一个叫评分标准或奖励函数的东西，负责给输出质量打分。

So SWE bench and AMI and Terminal bench, these are all examples of environments where there's a dataset of tasks, there's a harness for the model to be in, and there's something called a rubric or reward function, which is responsible for grading the quality of the outputs.

所以，你离线用作评估集（也就是测试集）的东西，在强化学习里可以当作训练集来用，这样就能以交互的方式提升模型性能。

And so the same thing you'd use as an eval offline as your kind of your test set, you can use this in reinforcement learning as your train set, and this is a way to, improve model performance, interactively.

所以我们的平台真正让用户能够在他们的工作流程中使用环境，用于后训练、评估、合成数据以及强化学习。

And so our platform is really enabling people to use environments in their workflows for post training, for evaluation, for synthetic data, for reinforcement learning.

同时，它也非常注重作为一个社区平台，就像GitHub那样。因为这些东西是新的、复杂的，我们希望人们有很多可以借鉴的构建模块和大量示例，并且希望它是协作性的，让人们有理由展示使用模型或不同工具在工作流程中作为环境的方式，从而允许其他人在这些环境中后训练模型，以这种方式分享提升不同任务性能的能力。

And it's also very much focused on as a community platform in the same way as things like GitHub are, where this stuff is new, it's complicated, and we want people to have lots of building blocks they can draw from and lots of examples and for it to be collaborative and for people to have reasons to kind of show off ways of using, models or different tools in workflows as environments to allow other people to post train models, in those environments to kind of have this way of kind of sharing the ability to improve performance across different tasks.

我认为总体思路是给更多公司提供目前只有大型实验室才拥有的实际能力和优势，也就是说，这种产品模型优化循环，他们可以针对自己的特定产品优化模型。

I think the general idea is to give more companies the actual ability and advantage that's currently only the big labs have in the sense of, like, this product model optimization loop where they, yeah, can optimize their models for their specific product in a sense.

我们把它看作是一个东西，这就是为什么像ChatGPT是由OpenAI创建的，或者像Claude Code是由Anthropic创建的。

And we see it as a, yeah, thing where, yeah, that's the kind of reason why, like, a Chatchabeet was created by OpenAI or, like, a ClotCode was created by Anthropic.

他们实际上有能力针对他们的特定脚手架优化模型，让他们的模型比他们的产品表现好得多。

They actually have the capabilities to optimize models for their specific scaffolds in a sense and have their models work way better than their products.

所以这些产品也越来越流行，就像Claude Code现在变得非常流行一样。

And so more popular are those kind of products also become in a sense like a cloud code becoming extremely popular right now.

大型实验室自然没有太多动力让它对其他编码初创公司更好用。

The the big labs have, yeah, naturally less of an incentive to actually make it work better for, like, other coding startups in a sense.

我们的想法是给他们工具，让他们拥有自己的模型产品优化循环。

And the idea there is to to give them the tools to have, like, their own, like, model product optimization loop.

对，我觉得在这方面他们是早期采用者。

And, yeah, I think they're early adopters on that front.

我觉得，嗯，这方面我总会举的一个好例子就是红杉资本。他们很早就意识到了这一点，在我看来，他们构建了自己的Composer One模型，进行了大规模的后训练，并且真正优化了一个以Cursor本身为环境的模型。

I think, yeah, one great example always in this case that I always give is Sequoia, for example, that, yeah, realized that in my opinion quite early on, they built their own, like, Composer one model where they did, like, large scale post training in a sense and, yeah, really optimized a model where the environment was actually cursor itself.

所以，这给了他们所有你在Cursor里能用的工具，并且能在Cursor内部优化模型。

So, yeah, it gives them all the tools that, like, you have in Cursor in a sense as well and, yeah, optimize a model inside of Cursor.

而且，我们相信会有更多初创公司会走这条路，一方面优化他们现有的产品，另一方面也能构建出全新的产品，这些产品在目前没有这种产品模型优化循环的情况下是不可能实现的。

And, yeah, we believe there's a lot of more startups that, yeah, will go this direction to, yeah, on the one side optimize their current products in a sense, but also, build completely new products that are really not possible right now without having this product model optimization loop.

太棒了。

Awesome.

那你能说说你提到的那个有趣的观点吗？

And can we say a word on you said something interesting.

你说环境就是评估。

You said environments are just evals.

我们能深入探讨一下这个说法吗？

Can we dissect that statement?

在我看来，环境就是状态。

In my head, an environment is state.

它是对世界状态的描述，你通过它观察自己采取的行动，观察世界状态如何变化，从而更新你的世界模型。

It's a description of world state through which you observe what actions you take, you observe how world state changes, and therefore you update your word model.

这跟评估不一样，评估就像是，你应该在这组问题上得到这个答案。

That to me is distinct from an eval, which is like, you should have gotten this answer on this set of questions.

那你能帮我融合这两个概念吗？

And so can you help me merge those two realities?

当然可以。

Sure.

我觉得，有一种评估版本，大概是一两年前的情况，那时候很多评估都是问答式的，就像一大串问题。

So I think there's, a version of eval that's kind of where we were maybe a year or two ago, where a lot of evals were like question and answer, And it's like this big bang of questions.

然后可能还有另一种环境的概念，人们提到老式强化学习，比如玩雅达利游戏，那更多是关于这种长期运行的状态交互循环。

Then maybe there's other notions of environment that people think about when they talk about like an old school RL with like Atari that is much more about like this kind of long running state interaction loop.

我觉得现在的情况是，两者合二为一了，尤其是你想做大规模训练的那些环境，它们确实有这种复杂的状态。

I think where we are now is it's both in the same thing where especially the, the environments that you want to do large scale training on, they do have this complex state.

它们可能是在模拟一个网页应用。

They maybe are simulating a web app.

那是一个功能齐全的编码平台，有代理在里面执行任务。

It's a full fledged kind of coding platform where you have an agent doing these things.

但在那些原始的强化学习游戏中，总是有奖励的。

But in those, in the original RL games, there was always a reward.

有一个目标。

There's a goal.

所以，这个问题的目标不仅仅是让系统跑一遍然后由人来主观评判，而是确实有东西可以衡量进展和表现。

And so the the notion of there being a goal of this problem where it's not just running through some system and a human is gonna kind of vibe check it, there actually is something that can measure progress and performance.

而且……

And that...

这就是我说的“评估”的意思，就是有一个系统从某个状态开始，与系统、环境、测试框架、代理——随便你怎么称呼——进行交互。

That's kind of what I mean by it's uneval, in that there is a system that it starts at some state, it interacts with the system, the environment, the harness, the, the agent, whatever you want to call it.

但总有一个目标，并且有办法衡量它做得好不好。

But there is some goal, and there's a way to measure whether it's doing well or not.

好的。

Okay.

明白了。

Got it.

然后，你提到Cursor作为前沿工作和强化学习的一个很好的例子，这两者之间有什么区别吗？

And then is there a difference between you mentioned kind of cursor as a great example of somebody doing really great frontier work and reinforcement learning.

有没有一个好的思考方式，来决定什么时候应该构建强化学习环境，什么时候应该让你的实际应用和应用状态成为环境？

Is there a good way to think about when should you be kind of constructing RL environments versus when should your actual application and your application states be the environment, so to speak?

是的，我觉得肯定有理由两者都要，特别是比如，假设你在训练一个模型，让它成为一个非常优秀的Rust编码模型。

Yeah, I think there's definitely reasons where you want both, especially like, let's say you're training a model to be a really good Rust coding model.

在这种情况下，你可能希望它对很多不同的应用都表现良好，这些应用会有不同的环境，专注于某个领域任务。

Here, you might want it to be good for lots of different applications where you'd have different environments that are focused on some like domain task.

也许你是一家公司，想要一个模型，它非常擅长调用你的工具或使用你的特定领域语言，然后你可以把它作为服务提供给围绕它构建的人。

Maybe you're a company that wants a model that's going to be really good at calling your tools or using your specific, domain language that, and then you can provide as a service to people who are building around that.

还有一些情况，产品本身就是一个面向最终用户的用户界面，用户与代理交互，在这种情况下，让产品直接成为环境可能最有意义。我认为，那些可能想要这样做的公司，也是那些关心他们使用的是Claude、GPT-5还是Gemini，或者有能力选择模型，并有内部系统来评估某个系统提示是否好，或者更换模型端点是否好，或者使用模型的迷你版本是否是更好的成本性能权衡的公司。

Then there's also ones where the product very much is like a user interface for an end user where the user's interacting with an agent, in which case it might make the most sense for the product to very directly become the environment where I think the, the companies who might want to be doing this are the same who care about whether they're using Claude or GPT-five or Gemini or the ability to choose models and have internal systems to evaluate whether a certain system prompt is good or whether changing out a model endpoint is good or whether using the mini version of a model is a better, cost performance trade off.

很多人已经在那些高级代理公司里搭建的基础设施，其实和做强化学习用的是同一套。

The infrastructure to do that is that a lot of people have already been building at these kind of advanced agent companies is the same infrastructure you use to do reinforcement learning.

所以我觉得我们最终落到了一个挺方便的局面：提升模型能力最有效的训练方法，和很多人一直在锻炼的肌肉是同一回事，只是他们没做训练那部分。

And so I think that's kind of this, kind of convenient world we ended up in where the training paradigm that makes the most sense for improving model capabilities is the same sort of thing that a lot of people have been building up the muscle for just without doing the training piece.

所以这就是为什么我们认为强化学习对人们来说是个很有用的工具，可以把它作为系统优化工具箱里的一个选项。

And so that's kind of where we see RL being a very useful tool for people to then have this as an option in their toolkit for system optimization.

明白了。

Got it.

好的。

Okay.

我们来谈谈代理框架，它们是如何与强化学习结合的。

Let's talk about agent harnesses as they buy RL.

我觉得框架是当下的热门话题，尤其是你提到的Claude Co。

I feel like harnesses is like the theme of the moment, especially with you mentioned Claude Co.

得到了这么多关注。

Getting so much love.

我觉得他们在工程上做得特别出色的一个方面就是那个框架。

I think one of the things that they do exceptional engineering around is the harness.

框架和强化学习，这两者是互不相关的，还是互相排斥的？

Harness and RL, are those things orthogonal, mutually exclusive?

它们之间是什么关系？

Like how do they relate?

它们肯定是有关系的。

They definitely relate.

我觉得，怎么说呢……

I think the way...

我把框架看作是环境的一部分。

I think of a harness is like a piece of the environment.

所以，有一些……

And so, there's some...

对于任何评估或环境任务，都有一些输入，然后会发生一系列事情，最后会有一些输出状态，这个状态会被用来评分。

For any eval or environment task, there's some input, a bunch of stuff is going to happen, then there's some output state, which is then going to be graded.

所以这整个中间部分，无论是与模拟器交互，还是与其他代理或物理世界模拟交互，这就是环境，而 harness 在很大程度上是其中的一部分，它连接了模型与系统其他部分的交互方式。

And so this whole intermediate piece, whether it's, interacting with some simulator or interacting with, some another agent or physical world sim, this is the environment and the harness is a very much a piece of that where it, couples how the the model, interacts with any other pieces of the system.

而且我认为取决于应用领域，比如对于编码代理，我们有一个相当清晰的定义，你可以说 harness 就是 CLI 编码代理，而终端就是环境。

And I think depending on the application area, like, I think for coding agents, have a pretty clear, like, definition of what's the, you could say the harness is the CLI coding agent and the the terminal is the environment.

但这并不一定适用于所有不同类型的代理。

But this isn't necessarily gonna be universal across, like, all different types of agents.

在某些情况下，harness 就是系统提示和一些工具。

In some cases, it's a system prompt and some tools is the harness.

在某些情况下，它会生成子代理，而子代理也有自己的 harness。

In some cases, it's something that is going to be spawning sub agents and the sub agents also have their own harnesses.

所以这里有很多复杂性，但我认为我们的思考方式是，harness 会不断演变。

And so there's a lot of complexity, but I think the way we've thought about it is like harnesses are going keep evolving.

将会出现一场寒武纪大爆发，人们想以各种方式使用模型，而我们希望对 harness 能做什么采取一个相当通用的方法。

There's going to be this Cambrian explosion of ways people want to use models, and we want to take a pretty a general approach in defining what you could do with a harness.

所以我们真正在思考的，以及为什么我们使用环境这个术语作为抽象，是因为代理太窄了，harness 也太窄了，你可以在环境中完成所有这些事情，但环境作为一个整体抽象，允许任何系统与模型交互都在范围内。

And so what we are really thinking about and why we use the term environment as the abstraction is like agent is too narrow, harness is like kind of too narrow, and you can do all of these things within an environment, but the environment, as a abstraction on the whole, allows BIS, any sort of system, model interaction, is in scope.

然后你觉得……

And then do you think then...

你觉得所有公司都应该用环境来对模型进行后训练吗？

Do you think all companies should be post training their models with environments?

有没有一些特定的……

Are there specific kind of...

哪些情况是绝对需要用环境的，而哪些情况可以用其他方法进行后训练？

You know, where the bullseye, like, absolutely need to be using environments versus, like, you could be post training with a different method

当然。

Sure.

还是说你就应该直接提示你的模型？

Versus you should just be prompting your thing.

我觉得环境是你可以用来做所有这些事情的工具。

I mean, I think environments are tools you can use to do all of these things.

所以我觉得，当我谈到超越强化学习的环境时，它之所以是一个有用的抽象概念，部分原因在于它不把你绑定到强化学习上。

And so I think that's when I talk about kind of environments beyond RL, I think part of the reason why it's a useful abstraction is because it doesn't tie you to RL.

比如说，我想有一个小模型，想让它从大模型蒸馏出来。

Let's say I want to have a small model, and I want it to be distilled from a big model.

做法是，你把大模型放进环境里，让它运行很多次。

The way you can do this is you take the big model, you plug it in your environment, you let it run a bunch of times.

现在你有了所有这些来自相同交互的数据，可以用同一个评分器在最后筛选出最好的例子，然后在此基础上做SFT微调。

Now you have all this data that You comes from the same interaction can use the same grader at the end to filter for the best examples and then do SFT fine tuning on that.

你可以用环境来做提示优化。

You could do prompt optimization with an environment.

你可以用环境对不同模型进行A/B测试，就像你用评估集那样。

You could AB test different models with an environment, just as you would with an eval.

所以我们真的认为，每个AI公司都应该优化他们的AI系统。

And so we really, I think the idea is that every AI company should be optimizing their AI systems.

我觉得这个观点没那么有争议。

I think that's a less, like, controversial take.

好的。

Okay.

也许还有另一种方式来理解这个问题。

And maybe there's another way to frame it.

比如说，评估几乎就是你的智能体在你预期客户会面对的那组环境中的表现。

Like, the eval is almost how your agent performs in the set of environments that you expect your customers to face.

我觉得这取决于你想怎么称呼它，就像在传统机器学习术语里，你不想过拟合测试集。

I think it, it depends on what you want to call, like, I mean, in kind of traditional machine learning terms, you don't want to overfit to the test set.

所以某种程度上，我们已经接受了会用测试集或评估来衡量，然后让这种反馈回到模型里。

And so in some ways, we kind of are already accepting that we're gonna be, like, using the test set or the eval to measure, to have that kind of filter back into the model.

所以这……

And so it...

我觉得要区分什么是评估、什么是环境，其实有点棘手。

I think it's a little tricky to distinguish even like what is the eval, what is the environment.

我们基本上把它们看作是一回事。

We kind of think of them as one and the same.

我们非常宽泛地使用“环境”这个词，而评估就是一种用于衡量性能但不用于训练的环境。

And we use the environment term very generically, where an eval is a type of environment that's used for measuring performance but not training on.

这大概就是很多人做评估时的想法，他们真正在做的是找到一种衡量当前表现的方法，然后不断迭代。

And that's kind of how we see a lot of people thinking about when they're doing evals, what they really mean they're doing is they have some way of measuring current performance and they're iterating on it.

从某种意义上说，强化学习就是这种迭代的大规模应用，你自动调整模型一点、调整提示一点，然后通过这种方式朝着某个目标逐步优化。

And in some ways, RL is this iteration applied at scale where you're automatizing the process of changing the model a little bit, changing the prompt a little bit, and having this be the way that you can kind of hill climb on some goal.

再问一个问题。

Maybe another question.

你觉得强化学习在功能上等同于后训练吗？

Do you see reinforcement learning as synonymous, functionally with post training?

也就是说，如果你在做后训练，你是在做强化学习，还是在做其他事情？

Meaning like if you're post training models, are you doing reinforcement learning or are you doing other

肯定有很多——我觉得强化学习现在是个大热门，在很多情况下，实际上，如果你在做大规模模型，强化学习会占据你大部分的时间、精力和计算资源。

There's definitely a lot of thi- I think reinforcement learning is like the big thing now where it's like, in many cases, practically speaking, the, if you're doing a large scale model, RL will be where you spend the most of your time and focus and compute.

但这并不是你唯一要做的事。

But it's not the only thing you want to do.

从初始模型到你想部署的系统，整个过程中有很多环节需要处理。

There's a lot of things involved in the pull process of going from some initial model to the system you want to deploy.

这可以是提示调优。

This can be prompt tuning.

这可以是监督微调。

This can be SFT.

这可以是在线蒸馏。

This can be online distillation.

有很多算法都属于这个范畴，强化学习是其中最重要的一个，但周围还有很多其他方法。

There's a number of algorithms that all kind of are under this umbrella where RL is like the big one in the middle, but there's a lot of stuff around it.

实际上，把这个工具包展示给人们，让他们有各种可以调整的参数，这才是解锁的关键。

Really, think exposing this toolkit to people and letting them have all these knobs they can play with is the way to unlock

最佳价值。

best values.

是不是发现，真正聪明的AI研究人员才知道如何在实践中实现这些？

Are finding that it's the, you know, really smart AI researchers that actually know how to make this stuff happen in practice?

这是否符合你让AI开发民主化的目标？一般的财富500强公司知道如何使用这个平台并从中获得价值吗？

Is it, you know, towards your goal of democratizing AI development for all, you know, does your average Fortune 500 know how to use the platform and get value out of it?

我觉得大多数公司都有这样的人。

I think most companies have people who can.

我觉得任何一家财富500强公司，很可能都有一支AI工程师团队，他们能跟上最新的工具，擅长使用Cloud Code，对模型和提示词有很多见解，这些人肯定能做到。

I think any Fortune 500 any Fortune 500 company will likely have a team of AI engineers who are capable at following the, like, latest tools, who are good at using Cloud Code, who, have a lot of opinions about models and prompting, and those people certainly can do this.

这大概就是我们眼中的目标客户群。

That's kind of the audience where we see as the target customer for this.

特别是如果你给他们合适的工具去实际操作的话。

Especially if you give them the right tools to actually do it.

比如，也许那些大公司里真的没有人能调试GPU集群来启动这样的运行，或者缺少其他必要的组件，而我们的工具就是让这一切变得更简单，让他们能轻松进行大规模的、带工具使用和代码执行的智能体强化学习，把整个基础设施都抽象掉。

Like, maybe some of those large companies don't really have anybody in there who can, like, debug your GPU cluster in a sense to actually kick off such a run or, yeah, other components that are needed in there to, like, actually just make it easier in a sense to, yeah, do large scale, like, agentic reinforcement learning with tool use, with code execution, and pieces like this to, yeah, just abstract away the the entire infrastructure for them.

明白了。

Got it.

太好了。

Awesome.

说到这个，你们有没有特别喜欢的客户案例想分享一下？

Actually, on that note, do you guys have any favorite customer stories that you wanna share?

我们最喜欢的客户之一，我想特别提一下，就是RCI和Neolab，他们专注于前沿开放模型，一直在和我们合作，基本上涵盖了整个技术栈。

One of our favorite customers that I would like to, yeah, point out in a sense is RCI and Neolab working on, like, frontier open models have been working with us on, yeah, the entire stack in a sense.

我们一直在讨论强化学习。

We've been talking a lot about reinforcement learning.

不过，我们过去也做过大规模的预训练。

But, yeah, we've been, yeah, also doing a large pre training in the past.

这基本上也是我们历史的由来。

That's basically where our history is coming from in a sense as well.

所以，我和他们一起训练过一些最大的混合专家模型，并且能够将它们开源，同时也在后训练方面与他们合作。

So, yeah, I've been training with them, some of the largest, yeah, mixture of expert models and actually able to, yeah, open source them as well and, yeah, as well on the on the post training side with them as well.

威尔，你想再多分享一些吗？

Maybe, Will, do you wanna share some more?

当然。

Sure.

他们是我们非常紧密的合作伙伴，我们曾经……

So they're a very close collaborator of ours that we've...

我觉得……

I think...

大概都是老朋友了，但我们也……

Probably all been friends for a long time, but also, like, we've been...

我觉得他们是我们……

I think they've been a way where we've had the...

他们有很多基础设施方面的需求。

They've had a lot of things that they want infra for.

就是……

It's...

这促使我们构建了很多东西，从计算编排到后训练、预训练再到推理，基本上就是为了……

That's been a way that we've been able to kind of, force us to build out a lot of the pieces, both from compute orchestration to, post training to pre training to inference, around kind of just making the...

成为一个前沿实验室所需的一切。

Everything that is needed to kind of be a frontier lab.

对。

Yep.

而且我觉得他们在开放使命这方面和我们非常一致。

And I think they're very aligned with us in the kind of the openness mission.

而且我觉得……

And I think...

但我觉得他们有点专注于……

But I think they're kind of focused...

他们更针对企业和终端用户，他们会更直接地与客户合作，提供端到端的某种成果交付。而我觉得我们的定位是，我们真正专注于基础设施层的开发者体验，以及我们如何让这些工具触达更多人，让从想法到可部署模型的流程变得尽可能无缝、快速和高效。

They are more targeted at, like, enterprises and the end user, where they are kind of gonna work more directly, with customers in terms of like end to end, kind of, delivery of a certain artifact where I think where we come in is we are, we, are really focused on the developer experience at the infra layer and the ways that we can make, put these tools into more people's hands where the process of going from idea to deployable model can become as, seamless and kind of quick as possible and efficient as possible.

太棒了。

Awesome.

那还有其他你特别喜欢的客户故事吗？比如那些专门使用Environments产品的用户？

And then any other favorite customer stories maybe on people that are using the Environments product specifically?

我们确实非常关注研究社区，比如很多研究生在用，很多学生和刚进入职业生涯、正在学习这些东西的人也在用。

So, we are definitely really focused on like, the research community and like some of this is like a lot of grad students use it, a lot of like students and people who are getting, early, in like their career learning how to do this stuff are using it.

但也有很多专注于特定领域的实验室，比如说你正在创办一个医学AI实验室。

But also a lot of labs who are focused on a very specific domain where, let's say you're starting a medical AI lab.

我们和医学AI领域的很多团队合作紧密，他们想创建更多的基准测试，来了解我们的模型在医学能力方面表现如何，无论是诊断、患者互动、医学文献问答，还是针对某些医学任务的智能搜索。

We work closely with a number of groups in the medical AI space where they want to create more, both benchmarks to understand how good our models at medical capabilities, both in terms of diagnosis or patient interactions or question answer about medical literature, or agentic search over certain medical tasks.

所以，比如SOFANT和OpenMed，就是我们密切合作的两个团队，他们的重点是……

And so, like, SOFANT, OpenMed being two that we work closely with, where the focus there is to...

他们真的很在意赢得医学专业人士的信任。

They really care about, like, earning trust from the medical professionals.

所以对他们来说，他们真正想专注于自己的领域，而不是那些让很多人头疼的通用医学和LLM基础设施。

And so for them, they really want to focus on the domain and not as much on kind of the generic medical, the LLM infra that is kind of like a headache for a lot of people.

所以对他们来说，能够拥有这样一个平台来创建评估、展示评估结果，并利用这些评估来提升模型能力，然后可以将模型部署在医院本地或终端用户本地，这种定制化能力以及端到端的信任和对任务数据来源的理解，或者直接定制模型的能力，都非常关键。

And so for them, like being able to kind of have this platform for creating evaluations, for showing them off, for being able to use them to then improve model capabilities that could then be deployed locally in a hospital or, deployed locally for some end user where the ability to have this customization and have this kind of end to end trust and understanding of the data providence for tasks at hand or the ability to kind of customize models very directly is very key.

你们有没有客户是更偏向于那种老派的Atari风格学习环境的使用方式？

Do you have any customers that are using you for more of like what you call the old school kind of Atari style, you know, learning from your environment type?

你是说……

By...

你是指……

Do you mean like...

所以当我想到Atari时，我想到的是非LLM的东西。

So when I think of Atari, think of like non LLMs.

所以我们肯定专注于LLM和看起来像LLM的基础模型。

And so we definitely are focused on LLMs and, foundation models that look like LLMs.

确实有很多研究人员使用我们的平台来做那些更像……的事情。

There's definitely a lot of researchers who, use our platform for, these things that are more like...

我们平台上有些例子更像游戏。

There's some examples we have on the platform that are much more like games.

我有一个很有趣的例子，我经常用它做演示，就是《纽约时报》的Wordle游戏，它最终成了人们学习的好入门环境，因为……

So, I mean, I have one fun one that I use as like a demo a lot is the game Wordle from the New York Times, where this kind of ended up just being the, like, a great hello world environment for people because it's...

基础设施非常简单，但表达力很强，你可以很快上手，而且当你给模型奖励让它做得更好时，你会看到它学会思考这个游戏的时刻。

The infra is really simple, but it's very expressive where you can kind of get a feel for it and you get this moment of seeing a model learn to think about the game as you give it rewards for doing better.

而且你可以用非常小的模型来做。

And you can you can do it with a really tiny model too.

它正好处于一个难度适中的位置，你可以用几块GPU在一小时内完成这些运行，并看到模型真正学会如何玩得更好。

Like, it's it's in this sweet spot of difficulty where you can do these runs on, like, a couple GPUs in an hour, and see a model actually learn how to get better at the game.

然后……

And...

我觉得那些更像玩具的游戏例子，通常就是人们用来学习如何构建强化学习环境的。

I would say like the more toyish game examples, yeah, usually the ones that people are going for for actually learning how to build those reinforcement learning environments.

而且，这也是大家大量使用环境中心的原因，因为我们围绕它建立了所有基础设施，方便你在自己的环境中进行测试。

And, yeah, that's also what people are heavily using the Environment Hub for in a sense just because we have all this infrastructure built around it to, yeah, be able to actually test in your environments.

通常他们就是这样开始的，然后后来再去构建更复杂的环境。

And, yeah, that's usually how they start out and then, yeah, go to actually building more complex environments later on.

我还想特别提一下另一群人，他们在中心上构建了一些更复杂的环境，就是我们的强化学习驻留项目的成员。

Another group I would love to give a shout out to in a sense that have been building some of the more complex environments on the Hub are people part of our reinforcement learning residency where we have like a group.

我们最初只有大概八到十个人。

We initially started out with like eight to 10 people.

现在组里大概有十四到十六个人了。

I think 14 to 16 are now in the group.

里面有研究生，也有全职工作的人，他们兼职构建强化学习环境，同时也在环境中心上做新颖的研究。

You have people, grad students as well as, yeah, people working full time that, yeah, are part time, like building reinforcement learning environments as well as doing novel research on top of the Environment Hub.

而且，这些人确实构建了各种领域里非常棒的环境，从可验证的软件工程（比如Lean）到物理环境，再到一些网络安全环境。

And, yeah, those folks have, yeah, built yeah, amazing, yeah, environments in, like, all kinds of verticals in a sense, everything from, like, verifiable software engineering in Lean to make a physics environment to some cybersecurity environments.

所以……

So...

然后，我们当然也给他们提供了工具，让他们现在能在这些环境里实际进行训练。

And then, yeah, also we give them the tools obviously now to actually do the training in those as well.

太棒了。

Yeah, awesome.

你能帮我解释一下吗？

Can you help demystify for me?

我听说一些基础模型公司每个在这些环境上要花费数百万美元。

I've heard that some of the foundation model companies are spending millions of dollars each on some of these environments.

而且，你刚才提到了网络安全。

And so, like, you mentioned cybersecurity at the end.

那么，构建一个网络安全环境需要做些什么呢？

Like, what goes into constructing a cybersecurity environment?

其实，我们住院医师项目里有人专门做这些。

So, there's someone in our, residency program who actually does lot of the stuff.

而且，网络安全领域其实有很多工具，比如那些夺旗游戏，就是某个系统里藏着一个隐藏的漏洞。

And so like, there actually is a lot of like tooling in the cybersecurity world around like these capture the flag games where there's, there's some system that has some hidden bug in it.

这原本是给程序员设计的挑战，他们会搞个小黑客马拉松，试着在某个系统里找出那个漏洞。

And this is like a challenge that pro originally these are built for programmers or programmers will have like a little hackathon where they try to go find the bug in some system.

但这些挑战也可以用在语言模型上，变成一个完整的软件环境，里面有终端，智能体就在终端里，还能用工具执行bash命令。

But you can adopt these challenges to, LMs as well, where it then is a a full software environment where it's a terminal, where the agent lives in the terminal and has access to tools for doing bash commands.

可能用的是Cloud Code或者其他什么智能体框架的封装，但就是在装满文件的终端里，能跟这些文件交互。

Maybe it's, using Cloud Code or some other, wrapper for an agent harness, but it's in a terminal full of files and can interact with these files.

然后到某个时候，它……

And then at some point, the...

它会标记自己完成了整个流程，然后你就可以用其他软件或代码来评估环境的状态。

It marks that it's finished with the rollout, and then you can grade the state of the environment using other, pieces of software, other code that executes.

但我们确实喜欢……

But we do like...

我们其实有很多合作的人是在数据和环境这个领域的，我们发现很多人对用强化学习来评估数据质量很感兴趣，因为通过测量模型在一组任务、一组规则、一组环境上训练后的表现，你能发现环境里的问题，因为在强化学习里会出现一些情况，比如如果你的环境有后门，模型就能利用它，有点像钻系统的空子。

We actually, have a lot of people we work with who are in the data and environment space where we've found that, there's a lot of, interest in using reinforcement learning as a way to evaluate data quality, where, the ability to measure what happens when you train a model on a set of tasks, set of rubrics, set of, environments, allows you to understand bugs in the environments because, there are issues that come up in reinforcement learning where maybe if your environment has a backdoor, a model can exploit this and kind of game the game the system.

所以我觉得很多人有兴趣用强化学习，从想法到环境，最后进入前沿实验室，比如下一代GPT或Claude模型的基础训练，这个过程中有很多人想用RL来帮忙。

And so I think there's a lot of interest in people using RL to, like, in the pipeline from, like, idea to environment that ends up in some frontier labs, like, foundational training run for the next GPT or Claude model.

这里面有很多审查工作要做。

There's a of vetting that goes into this.

做这些，比如小规模或者中等规模的运行，比如说在一个环境里，能让你真正去探查，看看问题可能出在哪里。

Doing these, like, smaller or, like medium scale runs in, let's say one environment allows you to really poke and see where the problems might be.

好的。

Okay.

非常有意思。

Super interesting.

如果我们再进一步拿网络安全环境举例，顺便说一句，我对网络安全并没有特别的偏爱。

And if we take the cybersecurity environment example one step further, by the way, have no particular affection for cybersecurity.

我只是喜欢脑子里有一个具体的例子。

That's something that I love having a specific example in my head.

我能想象你怎么构造一个简单的例子。

I could see how you could construct a toy example.

这些“夺旗”的例子，我想它们应该是简单的例子。

These Capture the Fly examples, I would imagine they're toy examples.

我想它们看起来和真实公司的企业网络环境完全不一样，那里有各种网络安全产品。

I would imagine they look nothing like the actual corporate network environment of a real company with all of its cybersecurity products.

所以我想，你怎么构造一个真正的……

So I guess how do you construct an actual...

人们能构造的东西，真的能扩展到模仿和反映现实世界的复杂性吗？

Does what people can construct, does it actually scale up towards imitating and reflecting the complexities of the real world?

就像机器人领域里的“现实差距”那样？

Almost like the, you know, the robotics sense of real gap?

在构建这些环境时，是否存在“现实差距”？解决方案是不是就在真实的企业安全环境上训练？

Is there a seems real gap in crafting these environments and is a solution just to like train on like real kind of corporate security environments?

所以我觉得，在真正的复杂性方面，障碍其实没那么大。

So I would say there's less of a barrier in terms of the actual complexity.

原则上，这些可以做得非常复杂。

Like the, in principle, these can be as complex as you want.

只要是电脑上能有的东西都行。

It's anything that could be on a computer.

所以，只要想想电脑或电脑网络上存在的任何东西，都可以作为环境。

So just think of anything that's on a computer or network of computers as potentially an environment.

真正成为瓶颈的，很多时候是模拟器的成本，比如

What really kind of, becomes the bottleneck in many ways is like cost of the simulator, where

对。

Yep.

我觉得很多重点在于找到巧妙的方法来模拟系统的正确部分，比如说，你想让一个代理去……

I think there's a lot of focus on identifying clever ways to kind of mock the right piece of the system where, let's say you have an agent that you want to...

比如，互联网。

Like, let's say the Internet.

比如，你想让一个代理做很多网络搜索。

Like, let's say you wanna have an agent that, like, does a lot of, like, web search.

有些情况下，你其实想做真实的网络搜索。

In some cases, you actually wanna do the real web search.

有些情况下，你想找到一些方法，设计任务时其实不需要完整的东西。

In some cases, you want to find ways where you can design tasks where you actually don't need the full thing.

这有点像那种地图游戏，有一个你想探索的世界，可能有一整张地图，有些部分是暗的，但当你走过去，光照亮，你就能看到这部分了。

It's like this kind of, I don't know, I think of like the map games where there's this world you want to explore, where there might be this whole map and like, some of it's like dark, but as you walk around, like, the light shines, then you now see this part of it.

所以如果你知道在某个任务中地图的哪些部分可能会被探索，这就能让你决定哪些部分重要，哪些不重要。

And so if you know which pieces of this map might actually be explored on a given task, then this kind of allows you to decide which pieces are important or not.

所以举个例子，有一个叫Tattoo Bench的基准测试，在评估领域非常流行，它涉及的是客户服务代理和数据库。

And so like, one example, like there's some bench- there's a benchmark called Tattoo Bench that is very popular in the, the eval world where it's about customer service agents involving a database.

数据库这种东西，在真实系统中，你可能需要完整的数据库，但如果你知道代理在任务中只会问某些类型的问题，你就不需要包含数百万条记录或数千或数亿个不同示例的完整数据库，你可以用一个模拟数据库，它更便宜，可能是在内存中运行的软件，只包含代理预期或任务范围内预期的东西。

And so the database is something where like, in a real system, you might need the full database, but if you know that the agent is only going to ask certain types of questions for a task, you don't need a full database with millions of records or thousands or billions of, like different examples, you can have kind of a mock database that is, a cheaper to run, maybe in memory piece of software that, only has the things that are expected by the agent or expected in the scope for a given task.

所以识别系统中哪些部分的复杂性是过度的，是任务设计过程的一部分，以实现效率。

And so identifying, like, the pieces of the system that where the complexity is kind of overkill, is part of this task design process to enable efficiency.

那你的平台能帮助做到这一点吗？

And does your platform help with that?

听起来好像，如果创建环境是训练这些系统的瓶颈，那么能够高效地创建这些环境，就像点亮地图上需要亮的部分一样，这几乎是一种核心的平台能力。

Like, almost seems like, you know, if creating environments is the bottleneck to training these systems, then, like, being able to efficiently create these environments where, you know, you're lighting up the part of the map that needs to be lit lit up is like a core kind of almost platform competency.

你们在这方面有帮助吗？

Do you guys assist with that?

确实，这是我们设计一切时都会考虑的方式，而且我们在和不同人合作时，会深入很多不同的方向，专注于不同的任务。

Mean, it's definitely a way that we think about the design of everything, and we kind of go down a lot of these different, like, rabbit holes as we have to focus on different tasks when working with different people.

比如，编码代理就是一个例子，在处理沙盒、终端状态、确保良好的快照以及各种协议来与不同的代理框架交互时，会出现很多复杂性。

And so, like, coding agents maybe is one example where, like, there's a lot of complexity that comes up when working with, like, sandboxes and terminal states and ensuring that you have, like, good snapshotting and all of these things and protocols to interact with different agent harnesses.

所以我们围绕这个做了很多工作，但我们也知道，随着人们在这些不同领域深入，复杂性的空间会无限增长。

And so we've built a lot around that, but it's also the sort of thing that we kind of know that the space of complexity is going to grow arbitrarily over time as people start getting more, deep into these different domains.

我觉得我们试图设计一切时都留了很多余地，比如我们有空间去构建功能，就像有基础层的通用环境，然后你可以说，我想要一个编码代理环境。

And I think we've tried to design everything in a way where we we keep a lot of doors open, where, like, we have room to build features that's kind of like there's there's base layers of, generic environments, then you can go, I I want a coding agent environment.

我想要一个带有全局沙盒的编码代理环境，或者每个 rollout 一个沙盒，或者有很多不同的分支可以走。

I want a coding agent environment with a sandbox that's global across the run, or I want one per rollout, or there's lots of these different branches you can go down.

所以我们预料到会有很多这样的分支。

And so we kind of have anticipated, like, there will be a lot of these branches.

有些我们已经为此构建好了。

There's some that we've built for.

有些我觉得我们准备好了，等需要的时候再构建。

There's some that I think we're kind of ready to build for when we need to.

而且我们也经常思考如何打造良好的开发者体验，比如说，就想想文档或者给智能体用的技能文件。

And we also, like, think a lot about how do you make a good developer experience where, let's say, think about just documentation or skill files for agents.

人们在构建这些的时候会使用编码智能体。

People are going to be using coding agents when they're building these.

所以，在做这类研究时，会积累很多机构知识，无论是针对某个具体项目，还是作为一个研究团队进行大规模训练运行。

And so, there's a lot of institutional knowledge that gets built up when you're doing this kind of research, both for like a specific project or as a research team doing large scale training runs.

而能够把这些信息呈现出来，有些直接在产品里，有些体现在我们设计库的方式上，有些在文档或展示给智能体的技能文件中。

And being able to surface this information, some of it is directly in the product, some of it is in the way we design the libraries, some of it is in documentation or skill files that get shown to agents.

这类事情我们设计成可以随着时间演进的，就像研究文献在演进，不同复杂智能体的最佳实践变得更加清晰一样。

And this is the sort of thing where we we've designed it to be something that can evolve over time as like the research literature evolves, as the, best practices for different types of complex agents become more clear.

我猜你们肯定读过萨顿的《从...学习》吧？

I assume you guys have read Sutton's, Learning from...

什么是经验时代？

What is Age of Experience?

经验时代，是的。

Age Experience, yeah.

Scale AI时代的数据标注。

Scale AI era data labeling.

你觉得构建环境是不是那个的自然继承者？

Do you think that constructing environments is sort of the natural successor to that?

看起来确实已经是这样了，很多大实验室的重点似乎已经转向了……

I mean, very much seems like it kind of already is, where it does seem like a lot of the focus from the major labs has shifted to...

他们仍然在使用大量的人类数据。

They're still using a lot of human data.

所以人类数据似乎不会消失，因为创建这些环境本质上是为了那些模型还不够擅长的东西，这意味着人类在某些方面比模型更强。

And so human data doesn't seem to be going away because creating these environments kind of by definition is for things that models aren't good enough at yet, which means the humans are are better than the models in some capacity.

因此，识别哪些信息是人类最能独特地帮助模型提升技能的关键，我认为这才是真正要瞄准的重点，也就是如何建立从真正拥有专业知识的人类到模型的信息流，以及什么才算做得好，对吧。

And so identifying which pieces of information the human can most uniquely assist the model in improving its skill on, I think is really the key to target, which is how do you create this information flow from the human who really has the expert knowledge on how to do some sort of thing, what does a job well done look like Mhmm.

然后把它输入到模型里。

And get it into the model.

而且看起来强化学习是目前最有效的方式，就是给任务配上评分提示，用语言模型作为裁判，再加上一个衡量任务成功标准的评分细则，这种模式正在很多场景中兴起，而人类在其中……

And it does seem to be that RL is the most effective way to do this right now, where having tasks with, prompts to grade, with an LM judge, a rubric, for what success looks like on a task is kind of the paradigm that's emerging for a lot of these cases where a human...

有时候是奖励模型，但最直接、最直观的版本是，有一组关于“是”或“否”的问题，比如“这个任务在语言模型的回答中完成了吗？”

Sometimes it's the reward reward model, but kind of the most direct visceral version of it is, there's a set of questions about yes, no, was this done in the LM's answer?

然后这就变成了奖励分数。

And that turns into the reward score.

所以这需要大量的人类数据。

And so that requires a lot of human data.

好的。

Okay.

太棒了。

Awesome.

为什么要为环境创建一个中心？

Why create a hub for environments?

我觉得这个中心的想法，是因为看到很多不同的开源仓库，里面都有那些环境的实现，多得都快溢出来了。

I think the hub idea started by seeing a lot of different, like, open source repos out there that had, like, overflowing implementations of all those environments in a sense.

而且，总的来说，我们其实已经……

And, yeah, in general, like, we already...

其实，在我们开始做环境中心之前，Will 就已经创建了一个很好的验证器框架，里面就有不同环境的例子。

Well, Will already created a nice verifiers framework before we even started the environment hub, right, where you had different examples for environments in there.

这确实是一种让整个流程更加标准化的方法。

And it was very much, yeah, an approach to, like, standardize the whole process even more.

而且，除了分享这些环境和提供一个开源平台之外，更重要的是，你可以围绕它们建立很多基础设施，这是你光把它们上传到 GitHub 仓库之类的地方做不到的。

And beyond that, just beyond just, like, sharing those environments, right, and having, like, an open source platform for it, it's also about having, like, a place where you can build a lot of, yeah, infrastructure around them, which you can't really do if you just upload them to, like, a GitHub repo or something like this.

所以，有一个合适的评估系统，和你的环境集成在一起，让你能立刻在所有前沿模型上测试它们，这是人们大量使用环境中心的一个功能。

So, yeah, having a proper evaluations, like, integrated with your environments so you can, like, immediately test them across all frontier models is one of the features, yeah, people are heavily using the Environment Hub then for.

你还可以非常轻松地把这些环境安装到不同的训练器里。

You make it extremely easy to install one of those environments inside of, yeah, a different trainer.

所以，我们当然有自己的大规模训练器，就是 Prime RL，我们一直在为这个做大量优化，但在开源方面，我们也很开放，愿意和很多不同的训练器集成，因为大家在训练器方面也有不同的需求。

So, yeah, we obviously have our own, like, large scale trainer with with Prime RL which we've been heavily optimizing for this but yeah, we are very open there on the open source side to integrate with a bunch of different trainers because people have different needs on the trainer side as well.

这就是环境中心的核心想法。

And that's the whole idea of the environment up.

那社区的反应怎么样？

And what has the community behavior been?

比如，你看到有人去分叉或者修改这些环境吗？

Like, do you see people forking, modifying these environments?

你看到他们把自己的一些东西公开出来，然后把自己的秘密保留，不分享回去吗？

Do you see them putting something eightytwenty out into the public domain and then we're gonna keep our secrets for ourself and, you know, not share that back.

社区的行为是什么样的？

Like, what's the community behavior?

确实有很多和我们合作的人希望保持他们的环境私有，这可以理解。

I mean, there's definitely a lot of people we work with who wanna keep their environments private, as you understand.

但对他们来说，作为中心的价值在于，他们可以在那些已知的、公开的环境上做消融实验。

But the value for them of it being a hub is that they can do ablations on ones that are kind of known to be...

他们可以把自己的私有环境与类似的公开任务环境进行比较。

They compare can their private one versus some public one that might be on a similar type of task.

或者对于评估来说，拥有各种流行基准的混合且统一的实现非常有价值，这样如果你在进行训练，就可以插入一个已知的评估，作为监控训练进度的方法。

Or for evals, there's a lot of value in having kind of, mixed and, uniform implementations of popular benchmarks in a way that if you're doing a training run, you can plug in some known eval, as a way to monitor the progress of your run.

所以也许你在自己的环境中表现不错。

And so maybe you're doing well in your environment.

你可以看看，你的环境是否也能泛化到其他任务上。

You can see, does your environment also generalize to other tasks?

因此，拥有所有这些其他任务确实能帮助人们不仅理解自己的任务，还能理解他们可能希望模型擅长的其他事情。

And so having all of these other tasks available is really a helpful way for people to be able to, understand not just their own tasks, but other things they might want the model to be good at as well.

太棒了。

Awesome.

今天Hub上最流行的环境有哪些？

What are the most popular environments on Hub today?

是的，我认为往往是那些我们在文档中用作示例的环境，在软件领域，自然是最流行的那些。

Yeah, I think it tends to be the ones that are these kind of, ones, the ones that we use as the examples in the documentation that naturally in software tends to be the most popular ones.

所以是Wordle那些吗？

So the Wordle stuff?

Wordle那个确实挺受欢迎的，但我觉得有一个我们关注度很高的，而且我觉得被很多人分支、复刻、改成不同版本的那个，是我们叫Wiki search的，就是在维基百科页面上做搜索。

The Wordle's one's popular, but I think one that, we see a lot of interest around and one where I think there has been the most, degree of people kind of branching and forking and turning it into different versions is one that we call Wiki search, which is doing search over Wikipedia pages.

但它的设计初衷是作为一个模板，你可以更广泛地用于智能体搜索。

But it's designed to be this template you could use for agentic search more broadly.

所以……

So the...

有很多应用场景，人们希望智能体能够搜索他们的内部文档，或者搜索特定类型信息的文档。

There's a lot of applications where people want agents that know how to search over their internal, documents or documents for a specific type of information.

有了这种模板，你只需要替换文档。

And having this kind of template for if you just, you just have to swap out the documents.

然后你就有了一个现成的环境，其他部分都已经设置好了。

And now you have this environment ready to go where the rest of it is already kind of set up.

这种模式往往能带来很多价值，让人们能够引入他们想要进行智能搜索的其他类型的文档。

That tends to be the sort of thing where we see a lot of value in people being able to, bring other types of documents that they'd want to do a genetic search over.

明白了。

Got it.