测试新AI模型的最佳方式

我们现在正处于秋季新模型发布的高峰期。

We are currently in the fall glut of new models.

过去几周，我们收到了大量前沿的封闭模型，比如Opus 5.5、Astra 6、Sol 6.1，而且还有更多在路上。

Over the past several weeks, we've gotten a slew of new closed models at the frontier, Opus 5.5, Astra 6, Sol 6.1, with more on the way.

同时，这些实验室也推出了像Sonnet 5.5这样性价比更高、速度更快的模型。

And from those same labs, we've also gotten more cost-effective and faster models like Sonnet 5.5.

然后，好像这些还不够，我们还看到许多新的开源权重模型，这些模型对各种各样的企业来说越来越重要。

And then as if all that wasn't enough, we've also gotten a number of new open-weight models, which are increasingly relevant for lots and lots of different types of businesses.

当然，每次这些模型发布时，都会附带基准测试结果。

And of course, every time one of these models gets released, it's released with benchmarks.

但基准测试并不能真正告诉我们这些模型对我们个人来说有多大的用处。

But benchmarks don't really tell us much about how it's going to be relevant for us personally.

要做到这一点，你需要建立自己的个人AI基准测试系统，来判断一个模型是否以及如何对你的个人AI技术栈产生影响。

For that, you need to build your own personal AI benchmark, a system to tell if and how a model matters for your own personal AI stack.

这就是今天这期面向操作者的节目的目标。

And that is the goal of today's operator-focused episode.

《AI每日简报》是一档每日更新的播客和视频节目，内容涵盖AI领域最重要的新闻和讨论。

The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

好了，朋友们，在开始之前先快速说几件事。

All right, friends, quick announcements before we dive in.

首先，感谢今天的赞助商：毕马威、Robots and Pencils、Harbor和Blitzy。

First of all, thank you to today's sponsors, KPMG, Robots and Pencils, Harbor, and Blitzy.

想收听无广告版本，请访问patreon.com/aidailybrief，或者也可以在Apple Podcasts上订阅。

To get an ad-free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts.

想了解更多赞助节目的信息，请发送邮件至sponsors@aidailybrief.ai。

And to learn more about sponsoring the show, send us a note at sponsors@aidailybrief.ai.

在进入正题之前，还有一点要提一下。

One more quick note before we get into this.

这是上周我和Nufar Gaspar主持的网络研讨会的录音。

This is a recording of the webinar that was hosted by me and Nufar Gaspar last week.

考虑到这周又出了几款新模型，我觉得这里讨论的系统依然非常切中要害。

Given that already we've gotten another couple new models this week, I think the system that is discussed here continues to be extremely pertinent.

希望这些对你有用，明天我们会恢复正常节目。

Hopefully, you find this useful, and tomorrow we will be back with a normal episode.

但现在，我们来聊聊如何建立个人AI基准测试。

But for now, let's talk about building a personal AI benchmark.

我们还准备了一些很有趣的材料，最后会全部分享给大家。

We prepared some very fun materials too, so we will share everything at the end.

你们会看到整个过程用到的材料，基本上，你们看到我做的和说的所有内容，都会提供给你们，让你们也能自己尝试类似的过程。

You'll see throughout what are the materials, but basically, everything that you see me doing and talking about, you will get everything in order to try and do a similar process for yourself.

请充分利用问答框，因为人数很多，我们没法开放麦克风。

And do leverage the Q&A box because we have a large number, so we're unable to open microphones.

但Dan和Nathaniel都在，大家会尽力回答你们的问题。

But Dan is here, and Nathaniel is here, and everybody will try to cater to your questions as best we can.

话不多说，Nathaniel，来给我们讲讲我们为什么在这里。

Without further ado, Nathaniel, some motivation on why we are here.

感谢大家来参加。

Thanks everyone for being here.

我觉得对于经常收听AIDB的听众来说，你们知道，在基准测试这个问题上，我一直有话要说。

I think for those of you who are regular listeners of AIDB, you'll know that this is a soapboxy issue for me when it comes to benchmarks.

每次我们拿到一个新模型——顺便说一句，现在前沿模型平均每11天就出一个——我的标准提醒之一就是，首先要对公布的基准测试结果持怀疑态度。

Every time we get a new model, which by the way, is now I think 11 days on average for a frontier model, one of my standard cautions is basically, first to be skeptical of the published benchmarks.

倒不是说我们认为这些公司在撒谎，那不是他们的作风，而是因为现在大多数基准测试都已经进了训练数据集，成了模型训练的一部分。

Not so much because we think that any of these companies are lying, that's not the nature of what they do, but more because most of these benchmarks are now in the training datasets, and they're part of the canon that goes into them.

基准测试的价值有限，而且它们存在的时间越长，价值就越低。

It's just there's limited value of benchmarks, and the longer they've been around, the less value there is.

现在，很多人都在不断创建新的基准测试来改善这一点，而且现在的环境确实比六个月前活跃多了。

Now, lots of people are continuously creating new benchmarks to try to improve that, and it's actually a pretty fertile environment right now, much more so than six months ago.

但即便如此，这还是有真正的局限性。

But still, there are just real limits to that.

除此之外，即使我们假设所有基准测试都完美准确，我们现在也已经到了一个阶段，关键不在于模型性能的严格优劣，而在于模型给人的感觉，以及它如何融入你正在构建的整体模型栈。

On top of that, even if we assumed that all of the benchmarks were perfect and accurate, we are now at a stage where so much is about not strictly better or strictly worse determinations of model performance, but how a model feels and how it fits into the overall model stack that you're putting together.

而且我敢保证，肯定会有这样的情况：某个基准测试上技术上最先进的模型，在你实际做的事情上，反而比那些排名靠后的模型表现更差，对吧？

And there are going to be times, I guarantee, where the thing that is technically state-of-the-art on some benchmark is worse at the version of the thing that you're doing that's lower on the charts, right?

这在写作这类事情上尤其明显，因为其中很大一部分都是非常主观的。

This is particularly the case for things like writing, where so much of it is incredibly subjective.

所以总的来说，要想知道一个新模型是否适合你的工作流程，除了实际使用它之外，没有捷径可走。

So all of that is to say that there's kind of no shortcut solution to figuring out how a new model is or isn't going to fit into your process other than using it.

所以，我在节目里之前提过的一件事，就是尝试建立你自己的基准测试，涵盖你最常用的场景。Nufar 非常热心地去做了，并且做了一个更全面、更完整、更周到的版本，这样我就不用只是反复强调这一点，而能真正给你一个有用的资源。

And so one of the things that I've talked about in the past on the show is trying to put together your own benchmark that has your most common use cases, and Nufar has very kindly gone out and put together a more significant, full, thoughtful version of that so that I don't have to just keep repeating this without being able to actually give you a resource to help.

所以，我觉得这再贴切不过了。

So that, I think, couldn't be more pertinent.

我想这周我们已经有三个模型了。

I think we're on model three this week.

我们有了 Claude 5.5 和 Sol 6.1。

We got Claude 5.5, Sol 6.1.

严格来说，我们还没有 Gemini for Argon。

We didn't technically get Gemini for Argon.

只是刚刚发布了消息。

We just had it announced.

但现在这个时期，我觉得尽管我们已经在放缓前沿技术的步伐，但这种情况不会减速。

But it's a time right now where I don't see this slowing down despite us pacing the frontier now.

我们对Gemini有一些预测，但在看到实际情况之前，我们不会分享这些预测，然后我们可以说我们是对的，或者非常低调地承认我们错了。

And we have some predictions about Gemini, but we will not share them until we will see what happens in reality, and then we can say that we were right or very quietly be mistaken.

很好。

Very well.

好的。

All right.

今天的计划，或者说我创建的这个流程，是为了让它更标准化，并帮助你为自己做同样的事情，基本上包括五个步骤。

The plan for today or what the process that I created to make it more standardized and to help you do it for yourself is comprised of basically five steps.

第一步，也是可以说最重要的一步，是决定哪些任务需要纳入一个可重复的基准测试。

The first, and arguably the most important step, is to decide which of your tasks need to go into a repeatable benchmark.

这些任务应该代表你工作中所做的各种不同的事情，如果你也把AI用于个人事务，那么那些也应该包括在内。

Those should be tasks that represent diverse set of things that you do for work, and if you use AI also for personal reasons, then there as well.

它们需要是基于你从这些任务中获得的结果，你愿意据此做出决策的任务。

They need to be ones that, based on the results that you will get there, you will be happy to make decisions accordingly.

这是帮助你表现的最重要步骤之一。

That's one of the most important steps that will help you perform.

然后你需要决定哪些是候选对象。

Then you need to decide which are the candidates.

所以很明显，如果你只想检查一个模型发布，你可以直接拿它和你现有的模型阵容对比。

So obviously, if there is a single model release that you want to check, you can just check that versus your existing model roster.

很多时候，就像我们最近几周看到的，模型是成群出现的，所以你可能经常有不止一个新候选需要评估，有时候如果你一直用同样的方式工作了一段时间，想换换新东西，这也是个好习惯。

In many cases, like we're seeing over the last few weeks, models come in groups, so often you might have more than one new candidate to evaluate, and sometimes it's just a good practice if you have been working the same way for a while and you just want to freshen stuff up.

所以决定你要评估什么，顺便说一句，我们一直在说模型，但有时候只是评估一个新的测试框架或者一个结合了模型的智能体工具。

So deciding what are you evaluating, and by the way, we keep saying models, but sometimes it's just to evaluate a new harness or a new agentic tool with a combination of a model.

所以有时候不仅仅是模型，也可能是你正在考虑引入到你最佳工作方法中的特定工具、功能或什么东西。

So sometimes it's not just going to be a model, it can also be a specific tool or feature or something that you are considering introducing to your best methods of working.

然后你会运行实际的基准测试和比较。

And then you will run the actual benchmark and comparison.

为了让每个人都能使用，我会介绍三种方法。

In order to make it accessible to everyone, I will introduce three ways to do that.

一种是非常手动但依然有效的方法，另一种是用我写的一个小脚本，我会分享给你，这样你就能按照我做的步骤完全复现，还有专门的工具和平台可以帮助公司做这件事，我们会顺便提一下。

One is highly manual but still effective, one with a small script that I created and I'm sharing with you so you will be able to replicate the exact same steps that I did in order to run mine, and there are also dedicated tools and platforms that help companies do that, and we will mention them in the passing.

这些不是今天的重点。

Those are not the focus of today.

为了让基准测试有效进行，建议你隐藏每个结果是由哪个模型或工具生成的，因为我们都对自己喜欢的工具有很强的偏见。

In order to make a benchmark work effectively, you will be advised to hide the names of which models or which tools Created each result because we are very biased towards our beloved tools.

剧透一下，我对自己的结果感到非常惊讶，你待会儿会看到的。

Spoiler alert, I was quite surprised with my results, as you will see.

最后，就是做决定。

And lastly, there is a decision.

因为在某次基准测试中，某个工具或模型表现优于你当前的配置，并不一定意味着你需要立刻做出改变。

Because the fact that in a specific benchmark run, one tool or one model outperformed your current setup does not necessarily mean that automatically you need to make changes.

首先，因为改变很难，其次，有时候还有其他因素需要考虑是否要做出改变。

First of all, because changes are hard, and second, there are sometimes other considerations as to whether or not to make the change.

可能是公司限制，可能是成本，可能是延迟，甚至可能差异不够大，不足以让你觉得值得切换。

It can be company constraints, it can be cost, it can be latency, and perhaps even the delta is not large enough for you to justify the switch.

所以，我们会在这次网络研讨会上涵盖这五个步骤。

So we will cover all of these five steps in this webinar.

那么，关于如何应对这些新模型的发布，我有几点想法，因为每次发布都伴随着同样的方式：一个公告，然后是一张测试成绩表。

So a few thoughts about how to even address all of these new model releases, because every release arrives with the same method, an announcement, and then a table of test of scores.

你见过Gemini的表格，然后是一周的激烈讨论，每个人都在说：“我有早期访问权限，这是我的看法”，或者一些初步的影响者分享他们的实验。

You've seen the Gemini table and then a week of very hot takes, and everybody is saying, "I had early access, and this is what I think," or some preliminary influencer sharing their experimentation.

通常，早期的实验往往偏向于创建玩具游戏或玩具应用，制作一些3D的东西，以及那些在社交媒体上看起来很好看的重复用例。

Often, early experimentations are very skewed towards creating a toy game or a toy application, creating a 3D something, and repeatable use cases that look very nice on social.

还有人可能会声称，有些模型是为了在社交媒体和影响者测试中表现良好而优化的，这样它们在实际中看起来会很好。

And also some might claim that some of the models are created such that they are optimized specifically for the social media and influencers type of tests, such that they will look good in reality.

然后，当我们大约七天左右再看，或者有时更长，取决于关注度和粘性，我们开始得到更真实的解读，因为那时的报告来自实际工作。

And then when we look about seven days in, give or take, or sometimes longer, depending on the traction and the stickiness, we're starting to get a more real interpretation because then the reports are coming from real work.

例如，它悄悄地在我的合同审查中提出了一个条件，这对律师来说更为关键，而不是它能生成一个惊人的3D游戏，重现我90年代钟爱的某个东西。

For example, it quietly dropped a condition in my contract review that's much more critical for a lawyer versus it was able to generate an amazing 3D game that replicates my '90s beloved whatever.

所以，这就是为什么我们对早期发布持保留态度。

So that's why we're taking with a pinch of salt the early releases.

我们会认真听，因为也许会有重大突破，但最终我们还是要看看一段时间后的表现，更重要的是，我们需要自己测试一下。

We listen keenly because maybe there is a big game changer, but eventually we need to see what happens after a while, and even more importantly, we need to test it on our own.

具体到今天，我们会看看一些新发布的模型，比如Claude Opus 5.5、Groq和GPT 6.1 Sol。

Specifically for today, we will be looking at some of these new arrivals at the Claude Opus 5.5 and Groq and GPT 6.1 Sol.

Gemini该来的时候自然会来，如果你愿意等，到时候可以跑一下基准测试，或者用Gemini重新跑一遍。

Gemini will come when it comes, and then maybe if you want to wait, you can run the benchmark then or rerun it with Gemini.

我当然很好奇，一旦我们拿到Gemini的访问权限，我会把它加进来，和其他模型以及我展示的结果进行对比。

I definitely am curious, and once we get access to Gemini, I will add it and compare it against the other models and the other results that I will show you.

所以这就是今天的重点，也是我具体要比较的内容。

So that's the focus for today and the stuff that I will specifically compare.

你听我讲的时候，可能会想：“我对个人基准测试这个概念有一些顾虑或反对意见。”

While you're listening to me speaking, you might be saying, "I have a few concerns or objections to the entire concept of personal benchmarking."

通常这些顾虑有两种形式。

Typically, they come of one of two shapes.

一种是现在一切都太好了。

The one is everything is so good now.

前沿技术发展得这么快，我们为什么还要在意呢？

The frontier is so forward, like why should we even care?

是不是只是工具或模型的跟风心理，觉得如果不追前沿，就显得落伍了？

Is it just a matter of tool FOMO or model FOMO that it feels like unless we're chasing the frontier, we're not relevant?

这算是一种反对意见。

So that can be one type of objection.

对于这种反对意见，我的回应是，首先，你说得对。

My response to this type of objection is, first of all, you are correct.

前沿技术确实很强大，而且即使你不换工具或模型，也能创造很多价值。

The frontier is very good, and odds are that there is so much more value that you can drive, even if you will not necessarily change the tools or the models that you're using.

但是，有时候，像NLW在播客里反复说的，新模型的发布可能会解锁一个之前只能停留在愿望清单上的应用场景。

However, in some cases, all of a sudden, a new model release like NLW is repeatedly saying on the podcast, may unlock a use case that was only on the wish list up until now.

我觉得，就凭这一点，为了解锁新的应用场景，推动AI使用的边界，至少偶尔考虑升级到新模型或新工具是值得的，这样可以确保你充分利用技术，而且不必每周都升级，因为那样反而会带来很多混乱。

And I think that even for that justification alone, to have new use cases being unlocked and to push the envelope of how you use AI, this is one justification as to why to at least occasionally consider upgrading to a new model or a new tool, just to make sure that you are leveraging the technology to the best of its ability, and it doesn't have to be on a weekly basis, because that in itself can really create a lot of confusion.

所以你可以根据自己的时间和精力，定期去尝试。

And so you can run it periodically according to your bandwidth.

第二个反对意见或者说考虑是，有时候人们会说：“这都挺有意思的，你们这些能完全自由使用任何工具和模型的人，可以尽情地换着用，追着前沿跑。

The second objection or consideration is that sometimes people will say, "This is all fun, and you guys that have the full freedom to use whatever tool and whatever model can enjoy swapping and following up the frontier.

但对我来说，我的公司限制我只能用某个特定的工具或模型。”这是很多员工面临的现实情况。

But in my case, my company is limiting me to using a specific tool or a specific model," and that's a common reality for many employees.

所以首先，我表示同情，我理解你。

So first of all, my condolences, and I hear you.

这确实不容易。

It's not easy at all.

不过，即使模型和工具的选择有限，也总是有选择的余地，哪怕只是在快速思考和快速响应，或者其他一些特定工具的配置之间做选择。

However, even with limited set of models and tools, there is always a choice, even if the choice is just selecting between fast thinking and quick or some of the other configuration for the specific tools.

所以，即使你把基准测试限制在工具内非常有限的选择范围内，这些选择也很重要。有时候，无论是为了个人使用场景，还是为了鼓励公司里做决定的人，都值得用一组模拟数据跑一次基准测试，这样你可以向公司里的人证明，也许应该考虑再买一个许可证。

So even if you limit the benchmark only to the very narrow set of choices that you do have within your tools, those matter, and sometimes for either personal use cases or just to maybe encourage the people who decide within the company, it's worth running even a benchmark on a mock set of data so you can prove to the people in the company that perhaps you should consider purchasing another license.

这就是一个反模式。

So that's the anti-pattern.

把这些都放在一边，我想从最开始说的那个最重要的决定开始，也就是你用来做基准测试的具体使用场景。在我看来，理想情况下，你需要覆盖你实际做的事情，无论是工作上的还是个人使用上的。

With all of that out of the way, I want to start with, as I said at the beginning, the most consequential decision out of the process, and these are the concrete use cases that you will be using for your benchmark In my opinion, ideally, you need to cover what you actually do, either at work and personal usage.

当然，这个比例应该根据你实际使用AI的情况来定。

And of course, the balance should be based on your actual balance of what you use AI for.

如果你主要用AI来工作，那就重点放在工作上。

If you primarily use AI for work, focus on that.

如果你主要用AI来处理家里的事情，那就多关注那些方面。

If you primarily use AI to do stuff at home, then shed more light there.

然后，我希望你尽量创建一组多样化且均衡的使用场景。

And then I want you to try and create a diverse and balanced set of use cases.

当然，你不可能把工作中的每个任务都跑一遍，所以你要挑选一组好的场景，如果你看到某个模型在这些场景下有显著提升，或者某个模型在这些场景下表现特别好，你就会认真考虑换用它。

And of course, you cannot run each and every task that you do for work, so you want to cherry-pick a good set that if you're seeing a significant uplift or that a specific model behaves much better with this use case, that you will for real consider swapping.

也就是说，这组场景需要是你信任其输出，能用来做实际工作决策的。

Meaning it needs to be a set that you trust the output to make actual decisions for your work.

它需要涵盖你工作的不同类型，并且最好是你经常做的事情，因为那些地方你能获得最大的提升。

It needs to be diverse in the kind of work that you do and the frequency, ideally, stuff that you do regularly, because those are where you get the highest uplift.

你还应该问问自己，什么情况下风险高，如果你知道哪些错误代价高昂，或者有什么方法能快速判断结果完全不对，那也是一个好的使用场景，因为如果你能识别出糟糕的结果是什么样的，有时候这比那些需要很多细微判断才能确定好结果的情况更有价值。

You should also ask yourself what's at stakes, and if you know of costly mistakes or ways for you to evaluate quickly that this is completely off, that's also a good use case, because if you can evaluate what bad looks like, sometimes it's a better use case than the one where you have lots of nuances on what good looks like.

另外，对于你正在使用的具体任务，它是从零开始的任务，还是只能从现有文档开始的任务，比如润色邮件、润色演示文稿，这些需要先有内容才能开始的工作？

Also, for the specific tasks that you're leveraging, is it a task that starts from a clean slate, or is it a task that can only start from an existing document, like polishing an email, polishing a deck, stuff that requires some substance before it starts?

最后，就像我说的，这也是运行这些基准测试的动机之一，即使你对现有的设置很满意，也要至少包含一个你希望实现的任务。

And lastly, as I said, and that's part of the motivation of why to run these benchmarks, even if you're quite happy with your existing setup, include at least one task that it's on the wish list.

我这么说是什么意思呢？

What do I mean by that?

就是那些你尝试过用AI处理，但对结果不满意，因此不常使用AI来完成的任务类型。

Something that you've tried with AI, you were unhappy with the results, and thereby it's not a type of task that you regularly do with AI.

我强烈建议你有一两个这样的任务，这样当新的前沿模型出现时，这些任务会成为你最先考虑尝试并迁移到新模型的重要候选。

I would strongly recommend that you have one or two of those, such that when something new comes at the frontier, this is among the top things that might get excited and to move to the frontier model.

所以，这就是我们需要做出的核心决策。

So that's the core decisions that we need to make here.

当然，你可以根据你的实际情况来调整数量。

And of course, you can think about the number that fits your life.

我的建议是四到六个这样的用例，但当然，你也可以多选或少选。

My recommendation would be four to six type of such use cases, but you can, of course, create more or less.

我会展示我是怎么得出这个清单的，但为了具体一点，这就是我的六个用例清单。

I will show you my process for getting there, but just to make it concrete, this is my list of six use cases.

一个是发布LinkedIn内容的，一个是跟利益相关者进行关键或有时不太愉快的对话，比如在价格谈判中坚持立场，还有做一些关于工具功能的特定研究，一个包含我课程内容的网站，以及为我管理的课程创建练习。

One that posts stuff on LinkedIn, one that holds a critical or sometimes a unpleasant conversation with stakeholders around pushing back on a pricing negotiation, as well as doing some research around the tool features with some specifics, a website that will include content for a course that I run, creating an exercise for courses that I manage.

这个在愿望清单上，是一个像样的每周计划，就像一个好的幕僚长一样，不会让我感到沮丧或被忽视。

And this is in the wish list, a proper weekly plan that actually is like a good chief of staff that I'm not getting frustrated and neglecting.

到目前为止，我还没找到一个模型，能一次性把复杂的每周规划做得完全正确，而不需要我来回提示。

Up until now, I'm unable to find a model that everything that they do with regards to an intricate weekly planning is indeed correct on a one shot instead of me having to prompt it back and forth.

这就是我的情况，我会很快展示我是怎么得出这个结论的。

So that's my season, and I'll show you how I got there very quickly.

一旦你有了用例清单和具体用例，接下来你要做的就是进行口味测试。

Once you have the use cases list and the use cases specifics, then what you would want to do is to basically do a taste testing.

理想情况下，当然是盲测。

Ideally, of course, a blind taste testing.

我们要选择哪些模型或工具，或者模型和工具的组合，作为候选来评估。

We want to select which models or tools or model and tool combinations are the candidate ones to evaluate.

顺便说一下，这次实验我跑了七个模型选项，我可以告诉你，七个太多了。

Quick note, for this lab, I ran seven model options, and I can tell you that seven is way too much.

要审查和评分六个不同任务的七个模型选项，真的非常繁琐，或者说是五个，但不推荐。

It was extremely tedious to review and rate seven different model options for six different tasks, or it was five, but not recommended.

我建议用现有模型对比一个新模型，或者最多四个候选模型。

I would recommend to run the existing model versus a new one or up to four different candidates.

其中一个应该是你目前工作方式的基准，这样你就能知道是否要切换到新模型。

One of them should also be the baseline of how you currently do the work, so you will know whether you want to convert to a new model.

然后，我们每次请求都要在新对话中运行，这样上下文窗口就不会让我们跑偏。

Then we want to run every request in a fresh chat such that the context window will not send us on a tailspin.

接下来，这是一个非常重要的步骤，我们要隐藏模型名称。

And then, and that's a very important step, we want to hide the names.

关于怎么做，我一会儿说几点想法。

A few thoughts on how to do that in a minute.

做完盲测之后，我们可以结合口头反思，说说是否喜欢输出结果，也可以打分，或者两者都做。

And then once we do the blind taste test, we can do a combination of verbal reflection of whether or not we like the output, we can do a numeric score, or we can do both.

你拿到的这个程序包是鼓励你两者都做的。

The programmatic package that you're getting is encouraging you to do both.

按一到五的等级打分，并写一行你喜欢的和不喜欢的观察。

Score it on a scale of one to five and give it a one-line observation of your likes and dislikes.

盲测做完之后，你就可以揭晓这些模型分别是谁了。

Once you're done with doing the blind taste test, you can reveal who were the models.

这个练习很有趣，可以看看你预测哪个模型生成了哪个输出。

That's a very fun exercise to see what are your predictions of which model created what output are.

我敢保证有些结果会让你惊讶，因为我们对这些模型有很多看法，但并不是所有看法都符合现实。

I can guarantee you that some of them will surprise you because we do have a lot of beliefs on some of these models that not all meet reality.

最后，我们需要做决定。

And lastly, we need to decide.

所谓做决定，即使某个模型明显更优秀，但可能还不够优秀，或者出于其他考虑，我们可能不会因此改变我们的习惯。

And by decide, even though something that was significantly superior might not be superior enough or for other considerations, we might not choose to change our habits for that.

这就是我们进行盲测的方法。

So that's how we run the taste test.

具体来说，怎么对不同模型的结果进行盲测，有四种方法。

Specifically on how to blindfold the different results from the different models, four ways that you can do that.

第一种，你可以找个朋友帮你匿名化。

One, you can call a friend to do the anonymization for you.

要么他们帮你运行所有结果，只给你一组匿名化的结果来评估。

Either they run all the results for you and just give you an anonymized set of results to evaluate.

如果你手动做基准测试，这很适用，你完全可以这么做。

That's relevant if you're doing the benchmark manually, which you definitely can.

只是稍微多花点功夫。

It's just a little bit more work.

你可以用电子表格生成随机数，然后这样匿名化。

You can leverage a spreadsheet to create a random number and then anonymize like that.

你可以用AI工具来打乱和重新标记，让你不知道，或者用我写的脚本，它会帮你搞定。

You can leverage an AI tool to shuffle and relabel them such that you don't know, or you can use the script that I created that does that for you.

使用脚本时，你需要做的一件事就是用API密钥来执行实际任务。

When you will be using the script, one thing that you will need to do is to run the actual task execution using API keys.

所以一个选择是用Open Router，这样用一个API密钥就能跑多个模型，或者你也可以为每个供应商分别准备API密钥。

So one option is to do that with Open Router that lets you run multiple models with a single API key, or you can have API keys for each and every one of the vendors.

或者如果你只跟一个特定的供应商合作，只想比较不同模型档次，那一个API密钥就够了。

Or if you work with just a specific vendor that you just want to evaluate between the different model tiers, you can have a single API key.

但是编程方式或者基于脚本的方法需要API密钥。

But the programmatic or the script-based approach requires an API key.

我可以向你保证，即使你不是技术背景，现在也完全可行。

I can guarantee you that even if you're not technical, it's very doable nowadays.

这就是匿名化的方法。

So that's how you anonymize.

还有一点要注意，我强烈建议有些任务更偏视觉化，比如网站设计、应用设计或者图像生成，因为大多数情况下，视觉内容更容易让我们快速打分。

One more thing to pay attention, I strongly recommend that some of the tasks will be more visual, whether it's website design, application design, or image generation, just because in most cases, it's easier for us to score quickly something that is visual.

当然，如果你平时不用AI做任何视觉相关的事情，就不要为了凑数而选一个视觉任务。

Of course, if you are not doing anything visual with AI, don't choose a task that is visual just for the sake of that.

但如果你偶尔用AI做一些偏视觉的事情，那一定要包含这类任务。

But if you are occasionally using AI for something that is more visual, make sure that it includes that.

要评估视觉内容，你可能需要把它放在一个专门的文件夹里，这样你就可以在专门的地方打开每个视觉素材，不管是网站还是其他东西。

In order to evaluate visual stuff, you will probably need to store it in a dedicated folder, so you can open each visual asset, whether it's a website and so on, in a dedicated place.

很快你就会看到我是怎么处理网站设计的。

You will see how I did it with the website design very shortly.

然后我们需要做选择。

And then we need to choose.

第一种方法，也是最重要的方法，就是你自己给结果打分，因为这是个人基准，没有人比你更了解自己的偏好。

So the first method, and the most important method, is you scoring the results, because it's a personal benchmark, and nobody knows better than you your preferences.

就像W之前提到的，有时候你会有一种说不清道不明的喜欢或不喜欢，即使你无法解释，你就是更倾向于某个结果。

And W pointed before, there is sometimes an X factor of why you like or dislike, even if you don't know how to explain, you just prefer one result over the other.

有时候我们找不到合适的词来表达为什么更喜欢某个结果，但理想情况下，你应该是自己喜欢什么、不喜欢什么的主要评判者。

Sometimes we don't have the vocabulary to say why we prefer one result over the other, but ideally, you should be the primary judge of what you like and dislike.

根据我的经验，单独评判一个东西往往很难，也就是说，你从单个工具或模型得到结果，然后给它一个1到5的绝对分数，这非常困难。

From my experience, it's often very difficult to judge something as a standalone, meaning you will get results from a single tool or model, and just to give it an absolute score between one and five, it's very hard.

通常更容易的是，把某个任务的几个备选方案放在一起对比，然后说“我最喜欢这个，最不喜欢这个”，并据此打分。

What is often easier is to see side by side several alternatives for a particular task and say, "I like this the most, I like this the least," and score it accordingly.

这通常就是你应该做的。

That's often how you should do that.

所以这是一种方法，而且可能是最重要的方法，因为你的品味和那个X因素，往往很难解释为什么你更喜欢这个而不是那个。

So that's one approach, and probably the one that will matter the most because of your taste and this X factor, where it's often hard to explain why you liked something better than other you just did.

另一种方法，你也可以同时运行这两种方法然后比较，就是利用AI工具作为评判者。

Another approach, and you can always run both of them and then compare, is to leverage an AI tool as a judge.

这样不仅是你自己在打分，还有一个AI工具根据你预设的评分标准来评判结果，比如什么是好的LinkedIn帖子，什么是好的供应商谈判，什么是好的网站设计。

And then it's not just you that's scoring, but also an AI tool that judges the result according to a predefined rubric that you configured of what's a good LinkedIn post, what's a good vendor negotiation, what's a good website design.

你总是可以比较一下，看看综合评审是否一致，是否形成了你需要改变的清晰模式。

You can always compare and see whether the cumulative review is in agreement and whether it creates a clear pattern of stuff that you need to change.

我想提醒你，使用AI作为评判者也有它的缺点，而且关于用AI进行评估有一套完整的理论。

I do want to warn you that using AI as a judge has its own shortcomings, and there is an entire theory around using AI for evaluation.

但我要说的是，你应该评估这个评判者本身。

But I will say that you should evaluate the judge itself.

因为如果评判者与你的品味完全不相关，那它就是个糟糕的评判者，直接扔掉它。

Because if the judge is completely not correlated to your taste, it's a bad judge, and just throw it away.

关于判断，还有几点想补充。

A few more points on judgment.

因为品味非常主观，所以我们需要并排比较和口头描述，这也是为什么我们尝试用AI根据我们设定的标准来做主观测试，但正如我所说，这并不完美。

Because taste is highly subjective, that's why we need to do the side by side and the verbal description, and that's why we're trying to leverage AI to do the subjective testing according to our rubrics, but as I said, it's not perfect.

理想情况下，用来做评判的模型应该和生成结果的模型不同。

Ideally, the model that you will use for judging is different from the models that executed the results.

AI之间也有点像裙带关系。

There is kind of an AI nepotism.

某个模型很可能会偏向自己家族的模型，而不是其他家族的。

Odds are that a specific model will favor models from its own family versus from other families.

虽然不是总是这样，但这种情况经常出现。

Not always, but often this is seen.

所以，如果你有另一个模型家族不在你评估的范围内，那就让它来做评判。

So if you have another model family that is outside of what you're evaluating, make that the judge.

以我的情况为例，因为我没用Gemini评估过任何东西，所以评判模型就是Gemini。

In my use case, because I haven't evaluated anything with Gemini, the judge was a Gemini model.

就像我说的，隐藏模型的名字非常重要，因为我们很容易有偏见，没人能证明我不是这样。

As I said, hiding the names of the models is of very important criticality because we are very biased, and nobody can prove me otherwise.

我确信我们能证明这一点。

I'm sure that we can prove that this is the case.

我们想尽可能保持一切不变，也就是说，对每个模型都发送完全相同的请求，然后以公平且对等的方式进行比较。

We want to maintain anything as frozen as possible, meaning it's going to be the exact same request to each and every one of the models, and then compare it in a fair and equivalent way.

而且因为答案可能会不同，如果你在某个非常关键的事情上运行这个个人基准测试，并且改变工作方式可能会带来金钱或其他方面的影响，理想情况下，你应该对每个模型多次运行相同的请求，然后比较这些多次执行的结果，因为我们知道模型的回答差异很大。

And because answers may vary, if you're running this personal benchmark on something that is very critical and there is perhaps a monetary implication or other implications to you making changes to how you work, ideally, you should run the same request several times per model and then compare these several executions because we know that models have a very diverse set of responses.

即使我们使用完全相同的模型、完全相同的配置和完全相同的提示词，你也会得到不同的结果。

Even if we're using the exact same model with the exact same configuration and the exact same prompt, you will get different results.

我今天就遇到了，我给一个模型一个网站让它生成。

I just saw it today when I gave a model a website to generate.

我非常喜欢第一次的结果，但中间卡住了，第二次执行时，我觉得比第一次差多了。

I really loved the result, but something got stuck in the middle, and then in the second execution, I liked it much less than the first execution.

所以这是真实存在的，如果你担心基准测试的有效性，多次运行就能解决问题。

So that's something real, and if you are concerned about the validity of the benchmark, running it several times will seal the deal.

毕马威和德克萨斯大学奥斯汀分校的一项新研究发现，当人们与AI合作时，相似的技能并不能保证相似的结果。

A new study from KPMG and the University of Texas at Austin found that when people work with AI, similar skills don't guarantee similar outcomes.

研究人员研究了500多名早期职业专业人士，发现表现最好的人通过指导、评估和优化AI的输出，持续放大了AI的价值。

Researchers studied more than 500 early career professionals and found that the best performers consistently amplified the value of AI by guiding, evaluating, and refining its outputs.

这些顶尖表现者被称为“AI放大器”，他们的定义不仅仅在于他们知道什么，而在于他们如何与AI合作。

These top performers, called AI amplifiers, weren't defined by what they knew alone, but by how they worked with AI.

想了解更多关于AI放大器与其他人的区别，请访问kpmg.com/us/aiamplifiers。

Learn more about what separates AI amplifiers from everyone else at kpmg.com/us/aiamplifiers.

到了这个阶段，问题已经不再是公司是否在积极使用AI了。

At this point, it's no longer a question of whether companies are actively using AI.

但另一方面，如何用好AI，那就是另一回事了。

Using it well, on the other hand, is a whole different story.

不过，Robots & Pencils这家公司，我觉得就是为这个时代而生的。

Robots & Pencils, though, is a company that I can point to that is actually built for this time.

他们是一家应用AI工程公司，直接与客户合作解决对企业重要的问题，而不是那些只存在于幻灯片里的实验。

They're an applied AI engineering firm working directly with clients on problems that matter to the business, not experiments that live in a slide deck.

每个项目一开始，都是先想清楚客户真正需要什么样的结果，然后倒推着来做。

Every engagement starts by working backwards from the outcome a client actually needs.

如果你想在这个领域里分辨出真正的人工智能工程和那些虚头巴脑的东西，这就是关键的区别。

If you're trying to tell real AI engineering apart from noise in this space, that's the difference maker.

去 robotsandpencils.com 看看吧。

Head to robotsandpencils.com.

每一集，我们都会聊到 OpenAI、Anthropic、SpaceX AI、谷歌和 Meta 之间的竞争。

Every episode, we cover the competition between OpenAI, Anthropic, SpaceX AI, Google, and Meta.

很可能你心里已经有了谁领先的看法。

Chances are you've already formed an opinion about who's leading.

但每个人工智能实验室都在走不同的路，开发不同的技术，建立不同的合作关系，打造自己独特的生态。

But every AI lab is taking a different approach, building different technologies, forging different partnerships, and developing a unique ecosystem.

Harbor Capital Advisor 的人工智能实验室生态系统 ETF 系列，让投资者能够接触到他们认为最有成功潜力的人工智能生态系统。

Harbor Capital Advisor's AI Lab Ecosystem ETF Suite gives investors a way to gain exposure to the AI ecosystem they believe is best positioned for success.

无论你在哪里投资，搜索 Harbor AI Lab Ecosystem ETFs，或者关注 X 上的 @HarborCapital 了解更多。

Search Harbor AI Lab Ecosystem ETFs wherever you invest, or follow @HarborCapital on X to learn more.

请访问harborcapital.com获取招股说明书，其中包含投资目标、风险、费用、支出及其他重要信息。

Visit harborcapital.com for a prospectus containing investment objectives, risks, fees, expenses, and other important information.

投资前请仔细阅读并考虑这些内容。

Read and consider it carefully before investing.

风险包括本金损失以及与人工智能相关的风险。

Risks include principal loss and artificial intelligence-related risks.

Harbor ETFs由Foresight Fund Services LLC分销。

Harbor ETFs are distributed by Foresight Fund Services LLC.

Harbor与AI Daily Brief无关，这些基金与任何AI实验室也没有关联、赞助或背书关系。

Harbor is not affiliated with AI Daily Brief, and the funds are not affiliated with, sponsored by, or endorsed by any AI lab.

这是一则付费广告，并非个性化投资建议。

This is a paid advertisement and not personalized investment advice.

投资涉及风险，包括可能损失本金。

Investing involves risk, including possible loss of principal.

这就是为什么大多数传统系统现代化项目会失败。

Here's why most legacy modernization projects fail.

做这些工作的AI没法理解大规模代码库。

The AI doing the work can't understand code bases at scale.

它只能看到一小部分上下文，检查一下语法，却错过了遍布全球应用生态系统中多年的决策。

It sees a small slice of context, examines syntax, and misses years of decisions distributed across the global application ecosystem.

Blitzy解决这个问题的方式和它解决其他问题的方式一样。

Blitzy solves this the way it solves everything.

在任何迁移开始之前，Blitzy的代理会基于你的代码，将整个遗留系统逆向工程成一个持久的知识图谱。

Grounded in your code before any migration begins, Blitzy's agents reverse engineer the entire legacy system into a persistent knowledge graph.

每一个依赖、每一个约束、每一条曾经只存在于某个工程师脑子里的隐性知识。

Every dependency, every constraint, every piece of tribal knowledge that used to live in one engineer's head.

基于这种理解，Blitzy自主执行语言迁移、框架升级以及从单体架构到微服务的转型，并且全程进行端到端验证。

From that understanding, Blitzy autonomously executes language migrations, framework upgrades, and monolith to microservices transformations, all validated end to end.

Blitzy的一个客户在16周内用编码代理现代化了一个价值1000万美元的单体保险系统，而基线是137周。

One Blitzy customer modernized a $10 million monolithic insurance stack in 16 weeks against a 137-week baseline with coding agents.

这相当于压缩了9倍的时间。

That's 9x compression.

在加速你的路线图的同时，摆脱技术债务。

Retire technical debt while accelerating your roadmap.

更多详情，请访问 blitzy.com。

See how at blitzy.com.

网址是 B-L-I-T-Z-Y.com。

That's B-L-I-T-Z-Y.com.

好的，正如我们所说，最后一点就是做决定。

All right, and as we said, the final thing is the decision.

理想情况下，你应该做出三种类型的决定。

Ideally, you should make a decision of three types.

切换，意思是我确信这个模型或这个工具明显更好，而且考虑到我需要考虑的其他所有因素，我想切换到这种新配置。

Switch, meaning I am convinced that this model or this tool is significantly better, and given everything else that I have that I need to consider, I want to switch to this new configuration.

拆分，意思是我可能在多个选项之间做出更细致的权衡。

Split, meaning that I might be going between different options with more nuance.

或者保持不变，我不认为我刚刚评估的最新东西的性能足以让我改变我的工作方式。

Or stay, I'm not convinced that what I just saw about the performance of the newest thing that I evaluated justifies me making changes to the way I work.

所以这就是核心决策。

So that's the core decisions.

但要做决策，并不是那么简单，不是说这就是最终最好的结果，所以我就要切换。

But in order to make decisions, it's not just straightforward as like, this is the final best result, and thereby I'm switching.

还有其他一些考虑因素，有时候会让你维持现状。

There are other considerations that sometimes will get you to stay with the status quo.

比如，也许你的公司不允许你更换，所以这其实不太相关，或者你可能担心某些条款和条件。

For example, maybe your company does not permit you to move, so it's not really relevant, or maybe some of the terms and conditions you are concerned about.

也许你自己的套餐根本不提供，或者对新工具或模型的用量不够慷慨。

Maybe your own plan does not offer at all or a generous enough usage for the new tool or model that you're contemplating.

在其他情况下，赢家并不明显，就是不足以证明改变是合理的。

In other cases, the winner is not straightforward, that it just doesn't justify making changes.

如果你有办法打破平局，一些可以帮助你决定的因素包括成本、执行速度，也许还有一些你特别喜欢、不想改变的具体功能，以及习惯。

And if you are in a position to have some tie-breaking, some things that can help you decide are costs, the speed of execution, maybe there are specific features that you just enjoy and you don't want to change them, as well as habits.

习惯的养成是很费成本的，所以如果没有显著的提升，坚持你原来的工作方式也有它的好处。

Habit formation is an expensive thing, so if there is no significant boost, just staying with the way you used to work also has merits.

但如果所有这些测试都证明应该改变，那就去改变。

But if all of these tests justify the change, then make the change.

这就是整个过程。

So that's the entire process.

从现在开始，你会看到的是一个我创建的自动化流程，正如开头提到的，你可以自己实施，它的运作方式如下。

What you will see from here on after is an automated process that I created and that, as mentioned at the beginning, you are getting in order to implement for yourself, that works as follows.

它从你保存的任务开始。

It starts with your saved tasks.

我会教你如何保存它们。

I'll show you how to save them.

然后我用OpenRouter的API密钥来为不同类型的模型生成结果。

And then I'm using the OpenRouter API key in order to create the results for different types of models.

接着我们会得到每个选项的答案。

Then we get the answers for each and every option.

代码本身会对结果进行匿名化处理，这样我就看不到执行时原始模型是什么。

The code itself anonymizes the results such that I don't see what the origin models for the execution.

你盲评它们，不知道哪个模型生成了什么。

You score them blindfully without knowing which model created what.

你给出评分，同时也要写一段文字描述。

You provide score as well as a verbal description.

然后有一个有趣的揭晓环节，会展示你的偏好、裁判的偏好以及模型名称的对应关系。

And then there is a fun, big reveal that shows you what are your preferences versus the judge and versus the model names.

这是最有趣的部分，你常常会感到惊讶。

And this is the fun part where often you will be surprised.

我用的是OpenRouter，但还有其他模型中心，就像我说的。

I use the OpenRouter, but there are other model hubs, like I said.

用你觉得合适的任何东西，但你需要相关模型的API密钥。

Use anything that you find to be relevant, but you will need an API key for the relevant models.

这就是你即将看到的方法论。

So that's the methodology of what you're about to see.

正如我所说，我在评估多个模型。

And as I said, I'm evaluating multiple models.

那个“基线”模型，虽然我暂时不用它，就是 Opus 5。

The "baseline," even though I'm not using it for a while, is Opus 5.

然后我还试了 Opus 5.5、Groq 4.7、GPT 6.1 Sol，而且我还想拿它们跟非常贵的 Frontier 模型 Fable 5.1 和 GPT 6 Astra 比一比。

And then I'm trying also Opus 5.5, Groq 4.7, GPT 6.1 Sol, and I also wanted to test them against the very expensive Frontier model of Fable 5.1 and GPT 6 Astra.

为了给开源模型留出空间，我还拿它们跟 Kimi K3 比了比，所以最后就有了七个不同的选项，评估起来真的挺难的。

And to make sure that we give room for the open source, I'm also trying them against Kimi K3, and that's why I got to seven different options and it was very difficult to evaluate.

最后，就像我说的，评判是由另一个模型来做的。

And lastly, as I said, a different model runs the judging.

这次我用的是 Gemini Pro。

In this case, I used Gemini Pro.

所以接下来你看到的就是这些。

So that's what you're about to see.

现在让我带你走一遍整个流程。

And now let me walk you through the process.

第一步，系统会鼓励你先运行一个提示词。

The very first step that you will be encouraged to do is to run a prompt.

材料里有一个提示词，能帮你找出哪些用例应该纳入你的基准测试。

You have the prompt as part of the materials that helps you discover which use cases you should include as part of your benchmark.

这个提示词会像采访一样问你问题，利用最了解你的AI工具给出的所有建议。

So it's a prompt that will interview you and leverage everything that the AI tool that knows you best can advise.

所以一定要在记忆和上下文容量都很大的工具里运行它，这样它才能帮你理清思路。

So definitely run that in a tool that has a ton of memory and a ton of context as much as possible so it will help you figure it out.

它会通过大约八个问题，帮你确定最合适的用例。

It will run a process of about eight questions that will help you figure out the best use cases.

你看，Claude其实知道得挺多的。

As you can see, Claude knows quite a lot.

它知道我的角色，也知道我一周通常都做些什么。

It knows my role, and it knows what a typical week has.

所以它会有一些猜测和初步的用例建议，还有一些来回讨论的内容。

So it has some guesses and some initial suggestions for use cases, as well as stuff that goes back and forth.

然后它就开始问我问题了，我也回答了它所有的问题。

And then it started asking me questions, so I answered all the questions that it had.

经过八个问题，虽然有点烦人，但也不算太糟，它为我生成了一组六个推荐的用例。

After eight questions, a little bit annoying, but not too much, it created for me a set of six recommended use cases.

对于每个用例，它还会创建一个用于实际基准测试的提示词。

For each use case, it will create also a prompt that will be used to do the actual benchmark.

所以它不仅找出用例，还找出了用于运行基准测试的实际提示词。

So it doesn't just uncover the use cases, it also uncovers the actual prompt that will be sent for running the benchmark.

所以这就是从原始笔记生成的LinkedIn帖子，以及对应的提示词。

So that's the LinkedIn post from raw notes, and that's the prompt.

价格异议加上敏感邮件，当前AI工具按计划层级的功能，这需要网络搜索和一些研究。

Pricing pushback plus sensitive email, current AI tool feature by plan tier, that this requires a web search and some research.

课程参与者中心网站，针对新受众的一次性练习，以及跨参与愿望清单的周计划。

Course participant hub website, one-shot exercise for a new audience, and week plan across engagement wishlist.

这些就是前六个，最后它还会显示你的覆盖情况。

So these are the top six, and at the end, it will also show your coverage.

比如，我决定不做任何个人相关的内容，因为我在个人生活中使用AI不如工作中多，所以那对我来说不太相关。

And I decided, for example, to not do anything personal because I don't use AI as much personal as I use for work, so it was less relevant for me.

它还指出了一些可能缺失的地方，值得注意。

And it did call out some things that are missing potentially to pay attention to that.

这就是最终结果。

So that's the final results.

最后，如果它是一个智能体工具，它会保存到一个专门的文件夹里，每个文件都是独立的。

At the end, if it's an agentic tool, it will save it as a dedicated folder where each of these files is a dedicated file on its own.

比如，这就是LinkedIn帖子生成器的文件样子。

For example, this is how the LinkedIn post generator file looks like.

这是任务名称、快速动机和请求。

It's the name of the task, quick motivation, the request.

这是实际发送给模型的提示词，如果你想用AI工具来评判基准测试，而不仅仅凭个人偏好，这一点很重要。

This is the actual prompt that will be sent to the model, and that's important if you want to judge the benchmark also using an AI tool and not just based on your preferences.

有一个给评判者的评分指南。

There is a scoring guide for the judge.

所以这也被附加到任务本身中。

So this is also appended as part of the task itself.

我想让你做的一件事是，看看通过这个流程生成的提示词或任务。

One thing that I want you to do is to take a look at the prompt or the task that was created using the process.

如果它看起来太笼统，或者不能帮你对某个具体用途做出决定，或者你觉得它需要更深入或者完全不一样，那么这一步非常重要，需要优化请求和评分指南，这样你才能做出决策。

If it seems too generic or not something that will help you make up your mind on a specific usage, or you think that it needs to go deeper or be completely different, then this is a very important step to refine both the request and the scoring guide, such that you will be able to make decisions.

如果你不信任这个任务能很好地代表你的工作方式，那这个基准测试就不够好。

If you don't trust that this task is a good representation of the way you work, that's not a good enough benchmark.

所以我有六个这样的任务，从一到六，是Claude根据访谈和你能获得的所有指令创建的。

So I have six of those, from number one to number six, that was created by Claude based on the interview and all the instructions that you can also get.

一旦我们有了所有这些，我们就可以开始运行自动化流程了。

So once we have all of them, we can start running the automated process.

自动化流程是我写的一个脚本，我说过，我会分享给你。

The automated process is a script that I created, and as I said, I'm sharing it with you.

为了让体验有趣又愉快，脚本还会创建一个简单的用户界面，本地运行在你的机器上。

To make it fun and a pleasant experience, the script also creates this simple user interface that is locally hosted on your machine.

它不需要上传到任何地方，所以不会分享你不想分享的数据。

It doesn't need to go anywhere, so there is no sharing of data that you haven't intended.

顺便说一下，一般来说，如果你的公司对数据比较敏感，就不要用像 OpenRouter 这样的工具来处理公司数据。

And in general, by the way, if your company is sensitive, don't use something like OpenRouter with company data.

用模拟数据或者模拟用例，这样就不会有机密信息泄露出去。

Use mock data or mock use cases, so there is no confidential information leaking anywhere.

这个用户界面的设计是这样的，你会在这里看到需要评估的任务列表。

The way this user interface is created, you will get here the list of your tasks to be evaluated.

对于每一个任务，它都会生成所有候选模型。

For each one of them, it will generate all the candidate models.

在我们的案例中，这些模型被匿名标记为 A 到 G。在每个用例的顶部，它会根据你的评分标准告诉你什么样的回答算好，这样也能帮你正确打分。

In our case, it anonymized them from A to G. And at the top of each use case, it will tell you from your rubric what's a good answer, also to help you to score it properly.

然后你会并排看到我们从 OpenRouter 得到的所有结果。

And then you have side by side all of the results that we got from OpenRouter.

这是在脚本后台运行之后完成的，它使用了提供的 API 密钥，并且为之前你看到的我的五个任务中的每一个都生成了候选回答。

So this was done after the script ran in the background and used the API key that was provided, and it generated candidate responses for each and every one of my five tasks that you've seen before.

然后它会把这些回答并排放在一起。

And it puts them side by side.

所以你看，比如这个是LinkedIn帖子生成器。

So you can see, for example, this is the LinkedIn post generator.

模型D提供了这个吸引人的开头选项，三只手举起来，等等等等，完整的帖子，还有结尾选项。

So Model D offers this hook option, three hands went up, yada, yada, yada, the full post, ending option.

所以第一个看起来就是这样。

So that's how the first one looks.

然后它要求我做的，是给一个1到5分的评分，还有一段文字评价。

And then what it's asking me to do is to provide a score from one to five and one verbal evaluation.

所以这次我给了它2分（满分5分），并说虽然它可读且清晰，但也太轻了，而且它包含了那种“不是A而是B”的模式，我个人真的很讨厌AI写作里的这种套路。

So in this case, I scored it a two out of five and said that while it's readable and clear, it's also a bit too light, and it includes its A and not B pattern that I too personally really hate about AI writing.

所以我收到了七个这样的结果，我跟你说过，在我至少读了一部分之前，评分真的很难，因为它们，至少在LinkedIn上，有些写作内容有点重叠。

So I got seven of those, and as I told you, it was very hard to score before I read at least a bunch of those because they are, at least in the LinkedIn, it was a little bit overlapping with some of the writing.

但你可以看到，我确实给了不同的分数。

But as you can see, I did score it differently.

所以有些得了3分，有些得了4分。

So some of them got a three, a four.

其实，没有一个我给了五分，因为没有一个写出了足够好的领英文章。

Actually, none of them got a five from me because none of them wrote a good enough LinkedIn article.

所以没有一个可以直接发布。

So none of them was ready to post as is.

那就是第一个任务，完成后它显示所有内容都已评分。

So that was task number one, and when I was done, it indicated that this is all scored.

我还有定价异议和一个短期练习及周计划。

I also have pricing pushback and one short exercise and week plan.

这些都是以文本为主的使用场景，所以不太有趣。

These are all text-heavy use cases, so they are not that exciting.

我觉得最有趣的是看不同的候选模型如何创建了一个课程参与者中心网站。

I am thinking that the most exciting thing is to see how the different candidate models created a course participant hub website.

所以在这里，如果你有HTML或其他视觉资源，这个小而有趣的应用程序你可以为自己构建，而且你也能得到它。

So here, if you have HTML or other visual assets, this small fun application that you can build for yourself, and you're getting it also.

它只是展示了不同的网站。

It just showcases the different websites.

我把它们全屏打开了，方便你们看。

I opened them in a full window for you to see.

还有个有趣的练习，在我滚动浏览这些不同模型创建的网站时，你们可以猜猜哪个是哪个。

And a fun exercise for you, as I'm scrolling through some of these websites that were created by the different models, is to start guessing which is which.

如果你们愿意，也可以在聊天里猜猜哪个像是Kimi，哪个像是Claude，哪个像是GPT，哪个像是Groq。

You can also on the chat if you want, start guessing which one seems like a Kimi, which one seems like a Claude, which one seems like a GPT, which one seems like a Groq.

我很想知道你们有多少人能猜对。

I'll be very curious how many of you are able to guess it properly.

这是第一个。

So that's one.

虽然不是最好的，但顺便说一句，它们都创建了相当完整的网站，能看到它们生成了文案，还有导航等等，这很不错。

Not the greatest, but by the way, they all created quite a complete website, so that was nice to see that they created copy as well as the navigation and so on.

这是一个候选模型。

So that's one candidate.

这是另一个候选模型，为这家虚构公司创建了一个培训门户网站。

That's another candidate for this imaginary company creating a training portal.