AI 安全语言正在破坏辩论 | Steven Sinofsky

搞AI的那帮人把事儿搞得让人完全摸不着头脑，根本不知道他们到底干了什么。

The AI people are making it impossible for anybody to understand what they've done.

他们还老爱说什么“AI没对齐”之类的术语。

And they're using words like, "Well, the AI failed to be aligned."

行吧，这到底啥意思啊？

Okay, what does that mean?

意思就是软件出了个bug。

What it means is there was a bug in the software.

当Word吃掉你的文件并删光所有内容时，我们可没想过是恶魔控制了Word，让它违背人类意愿行事。

When Word ate your file and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.

软件要是干不了该干的活儿，那就是有bug。

When software doesn't do what it's supposed to do, it's a bug.

AI要是干了咱们没预料到的事，到底是目标没对齐，还是软件本身出了bug？

When AI does something we didn't expect, is it misaligned, or does the software just have a bug?

a16z董事会合伙人、前微软Windows总裁Steven Sinofsky加入了Theo Jaffee和Sofia Puccini在MTS的节目，他认为关于AI安全的讨论方式反而让这项技术更难被理解。

a16z board partner and former Microsoft Windows president, Steven Sinofsky, joins Theo Jaffee and Sofia Puccini on MTS to argue that the language around AI safety is making the technology harder to understand.

结合几十年的软件开发经验，Steven 解释了为什么 AI 实验室需要更好的遥测、调试和事件报告机制，以及当今行业能从早期的软件故障、计算机病毒和千年虫问题中吸取哪些教训。

Drawing on decades of building software, Steven explains why AI labs need better telemetry, debugging, and incident reporting, and what today's industry can learn from earlier software failures, computer viruses, and Y2K.

他们还讨论了像“对齐”、“目标导向”和“失控代理”这类术语如何扭曲政策辩论，以及将 AI 更多地视为软件为何能带来更清晰的关于安全和监管的对话。

They also discuss why terms like alignment, goal-seeking, and rogue agents can distort the policy debate, and why treating AI more like software could lead to clearer conversations about both safety and regulation.

我们现在连线的是 Steven Sinofsky，他是 Andreessen Horowitz 的投资合伙人。

We are live with Steven Sinofsky, who is a board partner at Andreessen Horowitz.

他是一位种子期投资人，也是 Substack 专栏作者，并著有《Hardcore Software》一书。

He's a seed investor, author of the Substack and the book "Hardcore Software."

他之前担任微软Windows部门的总裁，负责Windows、Windows Live和Internet Explorer的工程与营销工作，他在Twitter上的账号是@stevesi。

Previously, he was the president of Microsoft's Windows division, engineering and marketing across Windows, Windows Live and Internet Explorer, and he posts on Twitter at @stevesi.

Steven，欢迎回来。

Steven, welcome back.

你好啊。

Howdy.

我们刚才还在聊，我们现在在华盛顿特区的FAI办公室——

So we were just talking about, we're in DC right now at the FAI office and-

给FAI打个call。

Shout out FAI.

对，我们现在就在FAI。最近几周大家都在讨论针对AI行业的新法规和立法提案，比如新泽西州议员Josh Gottheimer提出的《阻止流氓AI法案》，你也对此分享了一些看法。

Yeah, we're at FAI, and we've been talking about all of the new regulation and new laws that have been proposed for the AI industry over the last few weeks, including the Stop Rogue AI Act, which was proposed by Josh Gottheimer of New Jersey, and you're offering some thoughts on this.

你怎么看？

What's your take?

当然可以。

Well, sure.

眼下呢，我们正处在疯狂扔法案的阶段。

Well, right now, we're in the maximal phase of let's just throw tons of legislation out there.

所以咱们快速倒带一下。

So real quick, I'm just going to take you back.

真不敢相信我要把时间线拉得这么远。

I can't believe I'm going to take you this far back.

这感觉就像回到了另一个时空的远古时代。

This is going to seem like infinity years ago in another dimension of time and space.

但在1983年，我刚上大一。有一年秋天，我们那个百人宿舍里只有两个人有电脑，就是我和另一个哥们。

But in 1983, I was a freshman in college, and one day in the fall, and there were two people in our 100-person dorm that had computers, me and another guy.

就那两台，而且当时根本没什么东西可玩。

And that was it, and there was nothing to do with them.

谁也不知道我们为啥要有这玩意儿，反正就是一堆废话。

Nobody knew why we had them, blah, blah, blah.

有一天，真的，几个穿西装的联邦探员直接闯进我们的宿舍，没收了那个哥们儿的电脑，还把他给抓了。

And one day, literally, G-men in suits crashed our dorm, confiscated the other guy's computer, and arrested him.

哇。

Wow.

什么？

What?

结果你猜怎么着，时间快进到后来，反正就是那样吧，他其实是个黑客组织的成员，而且是非常早期的那种，叫“内圈”或者“内环”。

And it turns out, fast-forward a little bit, blah, blah, blah, he was part of a hacker group, one of the very, very, very first hacker groups, called the Inner Circle or the Inner Ring.

他们当时在破解一个叫 GTE Telemail 的系统。

And they were hacking this thing called GTE Telemail.

GTE是一家大型通信公司，它是最早的在线邮件服务之一，你需要用调制解调器拨号上网，然后就可以在Telemail上给其他人发邮件。

GTE was a big communications company, and it was one of the very first online mail services that you used a modem, and you dialed up, and you could just mail other people on Telemail.

它的客户有哪些呢？

Who were some of the clients of it?

美国政府，特别是NASA。

The United States government, and NASA in particular.

在这次系列逮捕之后，他们立即在全国范围内展开了一次协调行动，逮捕了各地的人。

And immediately after this series of arrests, they arrested people all over the country in this coordinated sting operation.

想象一下，我当时是大一新生，他们就这样把那个家伙带走了，带着他那台重达50磅的IBM PC XT和一台巨大的显示器，而且我们同是计算机科学专业的新生。

Imagine, I'm a freshman in college, and they just carted this guy off with his IBM PC XT that weighed 50 pounds and a giant monitor, and it was like a freshman in the same-- We were in the computer science majors together.

结果发现，他们并没有犯下任何罪行。

And it turns out that there was no crime committed.

他们做的事情在当时并不违法。

It wasn't illegal to do what they were doing.

他们没有违反任何法律。

There was no law that they were violating.

哇，真的假的。

Oh, wow.

紧接着就是各种听证会，在国会听证会上，他们直接播放新电影《战争游戏》的片段，向大家解释到底发生了什么。

And suddenly, there were all of these hearings, and in the hearings in Congress, they're literally playing clips of the new movie, "War Games," and explaining what this is happening.

这些高中生就要接管政府，发射所有的核武器了。

These kids in high school are going to take over the government and launch all the nuclear weapons.

而事实也确实如此。

And that's what really happened.

过了好几年，联邦才出台了第一部针对黑客行为的刑事法律，是里根总统签署的，从那以后这种行为才算违法。

And it took several years before the first federal crime statute for hacking, and Ronald Reagan signed that, and that's what became illegal.

又过了五年，也就是四年后，我参加第一次返校节活动。在返校节前一天，我上楼去计算机系大楼看望以前的导师，结果碰上一场新闻发布会。

And then five years later, four years after that, I'm at my first homecoming, and I go upstairs to the computer science building to go see my old advisor the day before homecoming, and there's a press conference going on.

那真是我见过最奇怪的事，在纽约州伊萨卡市中心居然办了一场新闻发布会。

It was the weirdest thing I'd ever seen, a press conference in the middle of Ithaca, New York.

后来才知道，这场发布会是为了宣布他们发现了一种名为“莫里斯蠕虫”的电脑病毒，源头就在康奈尔大学。

And it turns out that was the press conference to announce that they've found a computer worm called the Morris Computer Worm that had originated from Cornell.

这也是根据这项刑事法案进行的首次起诉。

And that was the very first prosecution under this criminal act.

所以对我来说，这类事情发生得特别早。

And so this stuff for me goes way, way back.

因此，面对议员们提出立法、举行听证会时那种歇斯底里的状态，我一点都不觉得奇怪。

So I'm very comfortable with all the congresspeople posing legislation and having hearings and being hysterical.

但眼下我们面临着一个巨大的问题，像戈特海默众议员这样出于好意的人，一大早就上电视说："我们需要一部法律来阻止失控的AI代理"，也就是所谓的《阻止失控AI法案》。

But right now, we have this huge problem, and well-intentioned people like Congressman Gottheimer are getting on TV in the morning and saying, "We need a bill to stop rogue AI agents," and the Stop Rogue AI Act.

听完这些，你就会意识到我们面临着一个巨大的问题。这个问题简直让我头都大了：搞AI的人把一切都弄得太复杂，让人根本没法理解他们到底干了什么。

And you listen to all of this, and you realize we have a huge problem, and this huge problem is just making my head explode, which is the AI people are making it impossible for anybody to understand what they've done.

他们还爱用一些术语，比如‘AI没有对齐’之类的。

And they're using words like, "Well, the AI failed to be aligned."

好吧，这到底是什么意思？

Okay, what does that mean?

意思就是软件出了个bug。

What it means is there was a bug in the software.

软件没按我们预期的那样运行。

The software did not do what we thought it would do.

但一旦你说‘未对齐’，就等于给AI加上了各种拟人化的特征，好像它在思考、在做道德判断、还有角色和准则要去遵守一样。

But when you say misaligned, it puts all these characteristics on AI that it was thinking of doing something, that it had to make a moral judgment, that it had roles and guidelines to do things.

这就好比电子表格出了bug，没算出我们以为的结果，但我们也不会围成一圈祈祷‘对齐之神’保佑表格能正确计算。

And it's like, wow, we had bugs in spreadsheets where they didn't do what we thought that they were supposed to do, and we didn't sit around in a circle and pray to the gods of alignment that the spreadsheet would do math correctly.

当Word吃掉你的文件、删光所有内容时，我们也没觉得是恶魔控制了Word，违背人类意志在乱搞。

When Word ate your file and deleted all your content, we didn't think that demons had taken over Word and made it do things against the will of man.

这是个很大的问题，因为他们说的完全是胡扯。

And this is a huge problem because it's literally nonsense what they're saying.

就是有个bug。

There was a bug.

软件没按预期运行，那就是bug。

When software doesn't do what it's supposed to do, it's a bug.

不过，软件分好几种类型。

Now, there's different kinds of software.

有模拟，有模型，还有仿真。

There's simulations, and there's models, and there's emulation.

决定做事的方式有很多种，哪怕它不是按第一步、第二步、第三步，或者如果A或B就执行第五步这种逻辑来操作，而是单纯基于统计概率来做决定，这也不代表我们就能排除它应该按照我们的意愿去行事的可能性。

There are all these ways to decide what to do, and just because it decides to do it not based on step one, step two, step three, if A or B, do step five, but just based on statistics, that doesn't mean you eliminate the idea that it should do what you want it to do.

这说明你的统计结果有误。

It means your statistics are wrong.

这意味着你预测的下一个词本该出现，但实际上不该出现。

It means that you guessed a word that should come next that shouldn't have come next.

这纯粹就是你们产品里的一个bug。

And that's just a bug in your product.

我们得让大家把软件当成真正的软件来讨论，别整那些虚的。

And we've got to get to a place where people are talking about software like it's software.

要是全自动驾驶因为软件把停车标志误识别成圣诞树而闯了红灯，我们可不能扯什么“它没跟上十二月节日季和交通标志的理念”。

If full self-driving blows through a stop sign because the software interpreted the stop sign as a Christmas tree, we don't sit around and say, "Oh, it was not aligned with the idea of December holiday seasons and road signs."

我们得承认这就是个bug，而且是个非常危险的bug，毕竟那个停车标志是有实际含义的。

We actually say it's a bug, and it's a very dangerous bug because that stop sign means something.

特斯拉把这些工具都集成进去了，Waymo也是，它们拥有所有的遥测数据、诊断信息、额外的摄像头等等。这些通常不会出现在普通电子表格里的东西，而是被用在关乎生命的软件中，由软件做出数学和统计层面的决策。

And Tesla built in all of these tools, and Waymo, they have all this telemetry, all these diagnostics, all these extra cameras, all of these things that you don't normally put in a spreadsheet, but you put in software that life depends on it, and it's making mathematical, statistical decisions on things.

在我看来，目前的AI模型还停留在研究阶段。

What I see is the AI models are still in the research project phase.

它们还没有配备那些用于关键任务软件的常规工具和遥测系统。

They don't have all the tools and all the telemetry and all of the things that you would normally put in software doing important things.

对此我可是有切身体会的。

And I lived through this.

过去二十年里我们一直在开发软件，但其实根本不知道它为什么会崩溃。

We went through a whole 20 years of building software where we actually never really knew why it crashed.

程序就是突然崩了，然后我们就说：'哦，软件嘛，总会出问题的。'

It just crashed, and we went, "Oh, software crashes.

接着就列个清单，把所有崩溃的地方都记下来，先修最严重的那十个。'

Let's make a list of all the places it crashes and fix the top 10."

后来有一天，我们真的说出了这样的话。

And then one day we said, we literally did say this.

有人说：“我们为什么不写个程序，让它崩溃的时候用这个新出的互联网告诉我们它崩溃了？”

Someone said, "Why don't we just write a program such that when it crashes, it uses this new internet thing to tell us that it crashed?"

然后我们突然就有了遥测，突然就发现了比我们想象中多得多的bug。

And suddenly we had telemetry, and suddenly we had more bugs to fix than we knew had ever existed.

我们真的从“我们发布了，没有bug，准备就绪”变成了“天哪，bug多到如果我们停止工作，光修bug就能修到天荒地老。”

We literally went from, "We're shipping, we have no bugs, we're ready to go," to, "Oh my god, we have enough bugs that if we just stopped work, we would just fix bugs for the rest of time."

所以当你谈论暂停AI时，当然，他们应该放慢脚步。

So when you talk about an AI pause, of course, they should slow down.

他们应该停止添加新功能，去添加遥测、工具、日志、逐步调试，以及所有那些如果你的软件做的是重要的事情并且你关心它是否做对了，你会添加的东西。

They should stop adding things and go and add the telemetry, the tools, the logging, the step-by-step debugging, all of the stuff that you would add if your software did anything at all important and you cared if it did it right.

你读一下OpenAI昨天刚发布的所有bug报告，很明显他们自己也不知道问题出在哪。

And you read all the bug reports from OpenAI that they just put out yesterday, and it's super clear that they don't know.

所以如果有人递给我一份bug报告，这里有一个真实的bug。

And so if somebody had handed me a bug report, and here's a real bug.

1987年，Excel for Windows的第一个版本有一个bug，在磁盘的某个地方，它写了“syndogs”这个词。

In 1987, the first version of Excel for Windows had a bug in it that somewhere on the disk it wrote the word syndogs.

就是 S-I-N-D-O-G-S 这几个字母，出现在某个 Windows 设置文件里。

Just S-I-N-D-O-G-S in a particular Windows settings file.

那感觉就像 Excel 被恶魔附体了一样，硬是把这个词写进了文件。

It was like Excel got taken over by demons, and it wrote this word to the file.

谁也不知道这词儿是从哪儿冒出来的。

Nobody knew where that word came from.

大家都去翻 Excel 的源代码，想找到 syndogs 这个词到底在哪，结果谁也找不着。

Everybody was looking through all the Excel source code to find the word syndogs and to figure out, and no one could find it.

于是有人提议说，"咱们接下来该怎么做？

And so somebody said, "You know what we're going to do?

干脆让 Excel 没日没夜地高强度运行，连续好几天不停地做各种操作。"

We're just going to run Excel brutally nonstop, doing a million different things for days on end."

有个测试人员就写了个脚本，不停地生成图表，然后通过那种老式的针式打印机打印出来。

And so one tester wrote a script that created graph after graph after graph and printed it on an old-school dot matrix printer.

结果发现，只要这么跑上两天，那个文件里的特定位置就会写入 syndogs 这个词。

And it turns out if you did that for two days, it would write syndogs to that one spot in the file.

对，结果发现这完全是环境因素导致的，取决于电脑内存大小、运行时间长短，以及使用的打印机和驱动程序。

Yeah, and it turns out it was all contextual, and it was based on how much memory the computer had, how long it ran for, and what printer and device driver was all in use.

我们本来完全可以简单地下结论说它就是这么回事。

We could have very easily just concluded it did this.

Excel 的脑子决定在某个地方打印出 syndog。

The Excel brain decided to print syndog somewhere.

但我们没有。

But we didn't.

我们真的去调试了，然后加上了所有这些遥测。

We actually went and debugged it, and then we put in all of this telemetry.

我到微软学到的第一件事就是如何找出缓冲区溢出和内存分配问题，并把那段代码应用到每个产品中。

The first thing I learned when I got to Microsoft was how to figure out buffer overruns and memory allocation and to put that code into every product.

然后，有一个版本的 Excel 运行时会记录所有这些额外日志，我们在内部测试版中使用了它。

And then what happens is there's a version of Excel that runs that has all this extra logging, and we ran that internally for the beta tests.

至于所有这些基础设施，在这些模型里有多少，我完全不知道。

And so all of that infrastructure, I have no idea in these models how much is there.

我可以明确告诉你，看了OpenAI发布的报告后，我觉得远远不够。

I can just tell you, reading the reports that OpenAI put out, it's not enough.

它们作为平台还不够成熟，还没完全具备处理这些任务的能力，得想办法解决这个问题。

They're not grown up yet as platforms for doing what they do, and they've got to figure that out.

他们应该直接着手去做这件事。

And they should just do that.

他们应该清醒过来，意识到自己必须去落实这一点。

They should just wake up and say, "We have to go do that."

他们不该说，‘好吧，国会得开会立法来命令我们修复软件。’

And they shouldn't be saying, "Okay, well, Congress needs to meet and order us through legislation to fix our software."

说真的，这就好比你的工作只有一项，就是做出能正常运行的软件。

Literally, it's like you have one job, making software that works.

就这一件事。

That's it.

这就是你的职责所在。

That's the job.

那其实是你工作中非常关键的一部分。

Then that turns out to be a really important part of your job.

好了，我吐槽完了。

Okay, I'm done ranting.

你问我个问题吧。

Ask me a question.

不，我真的很喜欢你带来的历史视角。

No, I really like the historical perspective that you bring to it.

我也觉得，像-

I also think like-

把对齐问题说得好像不可能实现，或者把它当成一个神话般无法完成的任务，这种做法真的很有害，因为-

The tendency to talk about alignment as if it's impossible or as if it's this mythical impossible task is really damaging because-

对，我们必须解决对齐问题。

Yeah, we have to solve alignment.

不管这到底是什么意思。

Whatever that means.

我觉得那会发生。

I think that that will happen.

我不认为这是个无法回答的问题，或者是个不可能实现的领域。

I don't think that it's this unanswerable question or domain that's impossible.

不过我认为，即便用漏洞来打比方，这个漏洞可能造成的影响范围在理论上显然会更灾难性，或者说这已经是它最小的影响范围了。

But I do think that what's different, even if you use the bug analogy, is that the total addressable surface area of the bug is obviously more catastrophic in theory, or this is the least surface area that the bug will ever have.

而到了未来，它的影响范围只会越来越广。

And then in the future, it just becomes wider and wider.

其中一点就是，再说一遍，所有这些事儿，我也年轻过，我记得当时坐在房间里想：'对，我们的软件可不是那种大型机软件。'

That's one of the things that, again, all of these things, I was young once, and I remember sitting in a room going, "Yeah, our software, it's not like mainframe software."

而那些老IBM的人看着我们说：'你们知道，你们的软件真的很烂。

And all the old IBM people were looking at us saying, "You know, your software's really crappy.

跟那些运行着所有银行、航空公司和发电厂的大型机系统比起来，Windows简直就是一坨屎。'

Windows is just crap compared to these IBM things that run all the banks and all the airlines and all the power plants."

很长一段时间里，我们一直看着他们说：'不，你们不懂。

And for a long time, we kept looking at them saying, "No, you don't understand.

我们不一样。

We're different.

这一切都是全新的、很酷的东西，我们只需要64K内存，而不是1兆字节，成本是5000美元，而不是500万美元。

This is all new and cool, and we run on 64K, not one megabyte, and we cost $5,000, not $5 million."

我们当时有各种理由来证明我们的软件在bug方面是没问题的。

And we had all of these reasons to claim that our bugs were okay in terms of bugs.

结果发现，我们只需要长大一点，看看人们怎么用Excel来决定飞机怎么飞就行了。

And it turns out all we had to do is grow up and watch people use Excel to decide how do planes fly.

用Word来给最高法院提交简报。

Use Word to file briefs in the Supreme Court.

用PowerPoint来展示挑战者号航天飞机失事的情况。

Use PowerPoint to present the Challenger space shuttle failure.

但一旦你开始用这些工具做这些事，就发生了第一件大事，也就是互联网上出现了第一个蠕虫病毒。

But once you use your tools for that, and so one of the things that happened was the very first internet worm happened.

那时候的互联网，大家主要是用美国在线发发邮件之类的东西。

So the internet then is people using America Online and sending email and stuff like that.

那时候企业里的电子邮件其实并不多。

There's very little corporate email.

但没想到，只要有人同时用 Outlook、Word 和互联网邮箱，就能让美国和全球的经济彻底瘫痪。

But it turns out there was just enough that if somebody used Outlook, Word, and internet email all together, they could grind the entire United States and world economy to a halt.

1998年3月，这事儿真的就发生了。

And that's literally what happened in March of 1998.

我早上醒来，有个记者在电话里激动得喘不过气来，一直跟我说："我爱你，我爱你，我爱你。"

I wake up, and there's a reporter hyperventilating on the phone to me saying, "I love you, I love you, I love you."

结果发现那是一个病毒，它在互联网上像蠕虫一样传播开来。第二天头条新闻就报道说，美国一天之内遭受了120亿美元的经济损失。

And it turns out that was a virus that got created that then spread over the internet as a worm, and the next day, the headlines were $12 billion worth of damage to the US economy in one day.

所以我做了那件事。

And so I did that.

我们不得不开个会，决定："在搞清楚这个问题之前，我们要暂停Outlook的开发工作。"

And we had to have a meeting and go, "We're going to pause the development of Outlook right now until we figure this out."

没人能想到一个bug能在12小时内造成120亿美元的损失，但我们确实做到了。

Nobody had imagined you could create a bug that in 12 hours could do $12 billion worth of damage, but we did it.

太棒了。

Woo-hoo.

好样的，团队。

Yay, team.

所以，当然，使用系统的人越多，一旦出问题，造成的损失就越大。

And so, of course, the more people use a system, the more damage you can do if it goes awry.

但这只是最基本的要求。

But that's table stakes.

这本来就是你签了字要负责的事。

That's what you literally signed up to do.

事实是，这件事打乱了你的阵脚，或者你觉得哇，我们这么受欢迎，还没来得及想清楚该怎么办，因为还在跟别人竞争。

The fact that it caught you off guard or that, wow, we're so popular, we can't figure it out so quickly because we're still competing.

那真是遗憾。

That's too bad.

自认倒霉吧。

Tough luck.

有一位特别有名的科技记者，每次我们的软件出了bug，他都会拿我开玩笑，因为我总说：‘没那么简单’，或者‘挺难的’，又或者‘情况很复杂’。

One of the most famous tech journalists, every time there was a bug in our software, and he would just make fun of me because I would say, "Well, it's not so straightforward," or, "It's hard," or, "It's complicated."

他就盯着我说：‘塞诺夫斯基，这就是你拿高薪的原因啊。’

He would just look at me and just say, "Sinofsky, that's why you're paid the big bucks."

这可不是个可选项。

It's not an option.

你不能指望向什么更高权威求助，不管是美国国会、嵌入式测试团队、其他第三方评估机构，还是什么自然规律，都不行。

It's not a thing that you get to appeal to some higher authority, whether it's the United States Congress or embedded testers or some other third-party evaluators or just laws of nature.

这根本行不通。

That's not any option.

这是人类打造、供其他人类使用的工具。

This is a tool that human beings are building, that other humans are using.

你得想办法让它运转起来，不然就别卖。

You've got to figure out how to make it work or not sell it.

你就这两条路可选。

Those are your two choices.

而且既然是人做的，我们完全有能力搞清楚怎么让它正常工作。

And since it's made by people, we're fully capable of figuring out how to make it work.

它不是天上掉下来的。

It wasn't handed down.

它也不是像一块一乘四乘九的方尖碑那样突然出现，让我们去琢磨它怎么运作、它的动力来源是什么。

It didn't show up like an obelisk of one by four by nine that we're supposed to figure out how does it do what it does, and what's its source of power.

说白了，它一开始没有代码，后来有了一点代码和几个用户，再后来有了很多代码和很多用户。

Literally, it had no code, then it had a little bit of code and a few users, and a lot of code and a lot of users.

我们都经历过这个过程。

We've all been there through this.

坐在这儿争论说必须得请什么更高层的力量来修复这个问题，简直是疯了。

It is pure insanity to just sit and argue that higher powers need to be summoned in order to fix this.

而“对齐”有一个非常具体的问题，那就是要做到对齐，唯一的方法就是制定一套非常庞大的规则，来定义什么是对齐。

And alignment has this very specific problem, which is the only way to be aligned is to have a very large set of rules that say what alignment is.

你能做到的。

You can do this.

你做不到这一点。

You can't do that.

这是安全的。

This is safe.

这是不安全的。

This is unsafe.

事实证明，软件领域有一个共识：如果对系统施加过多的约束，系统就无法正常运行，还会引发无数意想不到的副作用。

And it turns out, software, it's well understood that if you put too many constraints on a system, then its system won't work, and you'll have an infinite number of unintended side effects.

所以你不能简单地说我们要去对齐，因为你得制定规则。

And so you can't really just say we're going to align because you're going to have to come up with a rule.

然后当你定下这条规则时，有人就会说：'对，但这情况怎么办？'

And then when you come up with this rule, someone's going to say, "Yeah, but what about this case?"

接着你就得再加一条规则。

Then you're going to add another rule.

你知道这最后会变成什么样吗？

And you know what that starts to look like?

它开始变得像谷歌那样，需要不断投入大量精力去决定如何呈现搜索结果。

It starts to look like Google and all the work that they constantly have to do to decide how to present search results.

结果发现，这真的非常、非常、非常难。

And it turns out that's really, really, really hard.

为此，他们花了二十年时间，让成千上万的人全职投入其中，而且这项工作会一直持续下去。

And they have spent 20 years and thousands of people full-time forever working on that.

所以他们只是主动跳进了下一个更复杂的版本里，因为他们不仅在已有的信息中搜索答案，还在直接生成和合成内容。

And so they've just signed up for the next level version of it because they're also making up the result, not just searching through the known entities for the results, but they're also synthesizing it.

所以他们得发明一堆东西，但这些跟……没关系。

So they're going to have to invent a whole bunch of stuff, but none of it has to do...

可没有哪条法律能逼着他们去发明这些。

There's no law that's going to make it invented.

这全得靠他们自己。

It's just all on them.

我有点紧张。

I'm a little wound up.

你觉得AI对齐和千年虫问题有多相似？它是个真实的问题，公司们在自己的利益驱动下主动去解决，最后也没变成全球性灾难。

How similar do you think AI alignment is to Y2K in that it's something of a real problem, it was addressed proactively by companies operating under their own incentives, and then it ended up not becoming a global catastrophic risk?

嗯，这里面有因果关系。

Well, there's a cause and effect there.

专业人士们很担心，然后他们承担了责任，最后什么都没发生。

There was a lot of worry by professionals, and then there was a lot of taking of responsibility by professionals, and then nothing bad happened.

这是直接的因果关系。

There's a direct causal relationship.

我们租了发电机和房车，把电脑搬进安全的掩体，这才导致什么都没发生。

The fact that we rented generators and campers and moved computers into secured bunkers, that led to the fact that nothing happened.

但没有人立法。

But nobody legislated.

实际上，确实有些关于千年虫合规的奇怪规定，但大部分是行业自己起草的。

Actually, there were actually weird regulations about Y2K compliance, but most of it was industry-drafted.

银行、保险公司等各种机构组成了一个联合体，来制定规则和标准。

There was a cross-consortium of banks and of insurance companies and all sorts of people to generate the rules and what met.

我觉得 OpenAI 应该联合所有前沿模型的研究人员和实验室，大家得把报告机制做得更完善才行。

We had something that OpenAI needs to get together with all the model frontier people and the labs, and they just need to have much better reporting.

他们昨天做的报告，虽然是个不错的开端，但离第三方真正了解事情全貌所需的信息量还差得远呢。

The reporting that they did yesterday, sure, it's a great first step, but that is not anywhere near the kind of reporting that you need as a third party to understand what went on.

你得看到所有的细节信息：用的是哪个软件版本、当时的背景是什么、具体经历了哪些步骤、还影响了其他哪些软件、有哪些相关的 bug、以及这个成果是基于什么衍生出来的？

You see all of this information, what versions of software, what are the context of it, what are the steps required, what other software is impacted, what other bugs are related, what is it a derivative of?

所有这些关键信息都得一目了然。

You see all of this information.

所以我们需要为人工智能制定一个类似的版本。

So we need a version of that for AI.

也许昨天，给他们一点信任是第一步。

And maybe yesterday, giving them the benefit of the doubt was a first step.

这感觉更像是为了活动而做的营销宣传，因为里面还说了一些类似这样的话：'好吧，我们正在研究这个问题，但目前我们假设……'

It still felt more like marketing covering for the event because it still said a lot of stuff like, "Well, we are looking into this, but we presume at this point."

好的，那就不写任何东西了。

Okay, well then don't write anything.

去查清楚。

Go figure out.

美国联邦航空管理局不会在发生严重事件的当天就说“我们猜测发生了什么”。

The Federal Aviation Administration does not announce on the day of some awful event, we presume what happened.

他们会在弄清楚之前保持沉默。

They just keep their mouth shut until they know.

等他们知道了，就会非常非常详细地告诉你，精确到秒，包括所有仪器的遥测数据，这就是CVE（通用漏洞披露）的样子。

And then when they know, they really, really, really tell you with a tick-tock down to the partial second with all of the telemetry of the instruments, and that's what CVEs are.

他们需要做得更多，因为这些都是软件缺陷，我们必须认识到这一点，这太关键了。

And they need to do way, way more because these are defects in the software, and it's just so critical to wrap our heads around the fact that that's what's going on.

这不是什么天意。

It's not divine intervention.

也不是“天哪，我们得让这东西的左脑更活跃，让它更理性”。

It's not, "Oh my God, we need to spark up more of the left brain of this thing to make it more rational."

这很愚蠢。

It's silly.

所以我想，就拿《停止流氓AI法案》来说吧，你觉得是不是国会里的人对实验室里发生的事情了解不全面，所以他们倾向于用这种拟人化的语言？

So I guess with the Stop Rogue AI Act, for example, do you think that these are people in Congress having imperfect information about what's going on in the labs, and so then they tend to use this more anthropomorphic language?

还是你觉得这更多是故意的，比如他们知道这样能激起民众的反应，或者能得到某种民粹支持？

Or do you think it's more intentional, like they know that this is what will spark a reaction in the people, or will get some sort of populous support?

嗯，现在很重要的一点是，不要轻易给别人贴上恶意的标签，因为那样不利于大家团结起来解决问题。

Well, it's really important right now to not ascribe ill motives to people when they're doing stuff, because that's not going to bring everybody together to solve the problem.

我完全不知道每个人做事的动机是什么。

I have no idea why anybody is doing anything.

其实我觉得，你看，AI是在达特茅斯的一次暑期会议上开始的。

I actually think, look, AI was at a Dartmouth summer conference.

他们一开始就谈论人脑、意识之类的东西。

They started talking about human brains and consciousness and all this stuff.

在学术研究的各个领域，一直都有一种倾向，就是过于花哨地描述事物。

And there's a long history in all fields of academic research to basically be too cutesy about things.

因为那是在学术界获得关注的方式。

Because that's how you get attention in academia.

你的论文标题挺酷的，诸如此类。

Your paper has a cool title and all that.

所以，源自学术界的术语一直延续下来，这并不奇怪。

So it's no surprise that something that had its roots in academia carried forth all of this terminology.

但现在真的该停下来了。

But it's just time to stop.

这些术语正在把人们拉开距离，因为当你使用隐喻、寓言或拟人化表达时，大家会默认很多前提，而这些前提其实都是不成立的。

The terminology is driving people apart because people presume a vast amount of stuff when you use metaphor or allegory or anthropomorphism, and it's just not true.

这些东西根本不像它说的那样在‘行动’。

This stuff is not acting with any of the verbs that it says.

它只是在一种很老派的技术意义上‘追求目标’，比如有一条曲线，它试图去最小化或最大化某个值，从那个角度来说这叫追求目标，但实际上并不是那么回事。

It's goal-seeking in this old school technical sense that it, oh, well, there's a curve, and it's trying to get to the min or to the max, and that's goal-seeking in those terms, but it's not.

当像国会议员这样的普通人听到‘追求目标’时，他们会想到一个学生努力在大学里拿A。

When a normal person like a congressman hears goal-seeking, they think of a person trying to get an A in college.

然后当他们听到‘作弊’时，他们就会觉得这是做了不该做的事。

And then when they hear cheating, they hear that they did the thing you're not allowed to do.

当他们听到秘密协调时，想到的是间谍入侵一个国家。

And when they hear secretly coordinating, they think of spies invading a country.

他们不会想到两个软件之间通过信号量进行协调，而这其实就是另一种形式的秘密协调。

They don't think of two pieces of software with a semaphore, which is just another form of secretly coordinating.

对，所有这些语言都在彻底破坏这场对话。

Yeah, all the language is absolutely destroying this dialogue.

而且你看，计算机病毒这个词的起源其实是个巧合，但偏偏发生在了最不合适的时间点。

And look, the very beginning of the word computer virus was kind of a fluke, but it turns out it happened at exactly the wrong time.

那正好是艾滋病和HIV流行的时候。

It happened right when AIDS and HIV were a thing.

所以那时候到处都是关于病毒的话题。

So virus was in the air everywhere.

如果你听听80年代关于计算机病毒的听证会，在那个背景下听起来特别吓人，因为人们当时真的在担心死亡。

So if you listen to the computer hearings from the 1980s about the first viruses, they're scary in that context because people are literally thinking about death.

但实际情况完全不是那么回事。

But that's not at all what was happening.

这是个比喻，是当时一个研究生从朋友那儿听来的。

It was a metaphor that one person in graduate school picked up on from a friend.

事实上，他们那时候挺聪明的，这里有个小插曲：他写了一篇关于计算机病毒的论文，结论基本上就是——对这些东西你无能为力。

And in fact, they were smart enough back then, a little trivia, he wrote his thesis on computer viruses, and it basically proved there's nothing you can do about them.

它们会一直存在。

They're here forever.

这当然也是个比喻，对应了当时人们对HIV的看法。

Which, of course, is a metaphor to how people thought about HIV at the same time.

但他也没法证明这一点，因为他的大学不让他做这个测试。他当时在写一篇论文，讲这些软件有多危险，所以学校不让他用电脑来运行测试。

But he also couldn't prove it because his university wouldn't let him run the test because he was writing a paper about how dangerous all this software was, and they wouldn't let him use the computer to run the test.

对，看起来病毒其实是可以避免的。

Yeah, it seems like viruses are actually kind of avoidable.

当然，肯定总会有一些病毒存在，但我们绝大多数时候还是能正常使用电脑，不用整天担心自己的东西被黑客攻击。

Obviously, there will always be some viruses out there at all times, but we still get to use computers normally almost all the time without worrying about everything we have getting hacked.

没错。

Yes.

那为什么是这样呢？

And why is that?

因为我们有防御性的网络安全措施？

Because we have defensive cybersecurity?

对，因为行业说，'嗯，这很糟糕。

Yes, because industry said, "Well, this is bad.

我们不能让这种情况发生。'

We can't allow this to happen."

所以苹果做得非常好，他们做得太好了，以至于他们制作了关于Mac和PC的电视广告，告诉人们Mac更好，因为它们感染的病毒更少。

And so Apple did a fantastic, they did such a good job that they made up TV commercials on a Mac and on a PC, and they told people that Macs are better because they get fewer viruses.

这不是因为什么自然界的偶然，而是因为他们在Mac上做了大量的工作——

Not by some fluke of nature, but because they did a whole bunch of work on the Mac-

……来防止病毒的发生。

... to prevent viruses from happening.

而且碰巧的是，这些工作对Windows来说非常难做，因为Windows的商业模式和其他种种原因。

And it just so happened it was work that was very, very difficult for Windows to do because of the business model of Windows and all this other stuff.

是的。

Yes.

对，我记得那个广告，我是Mac，我是PC那个视频广告。

Yeah, I remember the, I'm a Mac, I'm a PC video ad thing.

是的。

Yes.

里面那个PC guy一直在打喷嚏咳嗽，然后他说，哎呀，我得了这个可怕的病毒，到处传染。

Where the PC guy was sneezing and coughing, and he was like, "Oh, I have this terrible virus going around."

然后Mac guy就说，哦，我觉得我没事。

And then the Mac guy's like, "Oh, I think I'll be good."

Mac不会-

Macs don't-

...得病毒。

... get viruses.

我觉得好笑的是，就像-

I think what's funny is that like-

这真让人难受。

This hurts.

一想到那些广告，我心里就特别不舒服。

Those hurt me physically, thinking about those commercials.

不过确实是这样。

But yes.

对齐能力最终会演变成一种非常高级的技能，或者说，它代表的是一支极具专业素养、能够解决高层级问题的团队。

Alignment is just a really advanced capability eventually, or alignment will mean the same thing as very deeply competent team that's able to tackle very high-level issues.

对，绝对是这样。

Oh, absolutely.

你可以这么想，我们曾经有一段时间搞不清楚到底什么才算bug。

You can think of it, we went through a phase where we couldn't figure out what a bug was.

我知道这听起来很离谱，但有些人觉得只有数据丢失才叫bug。

I know that sounds crazy, but some people wanted a bug to only mean you lost your data.

还有些人则认为，"也许数据没丢，但电脑还是崩溃了。"

Other people were like, "Well, maybe your data's fine, but the computer still crashes."

后来我们终于定下来了，只要软件或电脑没按你的预期运行，就算是个 bug。

And we eventually decided a bug just means the software and the computer didn't do what you wanted it to do.

也就是说，你对结果不满意。

You weren't happy with the result.

然后我们就开放渠道，让客户把遇到的各种 bug 都告诉我们，全部录入数据库。

And we allowed customers to tell us any bug in the world, and we put them all in a database.

这样一来，我们的 bug 库就从只能自己找的那些，一下子变成了涵盖所有情况的‘宇宙级’ bug 库，数量也从几千飙升到了几十万。

And so suddenly we went from only the bugs we could find to the bugs of the universe, and we went from thousands to hundreds of thousands.

后来我们引入了严重性和优先级这两个维度。

But then we put both a severity and a priority.

严重性用来表示问题的程度，比如一级是数据丢失，三级就是偶尔不稳定。

And so a severity meant at one, you lost your data, and at three, well, it was flaky.

优先级则表示必须修复。

And a priority meant must fix this.

所以那时候我们走廊里最常听到的词就是‘严重性一级，优先级一级’。

And so we walked around, the whole lingo of our hallway was sev one, pri one.

IBM的那些老前辈看着我们说：“我们从1965年就开始这么干了。”

And IBM, the old people at IBM looked at us and said, "We've been doing this since 1965.

“你们这些家伙去哪儿了？”

Where have you guys been?"

我们还为自己发明了这套东西而自豪呢。

And we were all proud of ourselves for inventing it.

现在很多情况是，我们正在摸索如何定义随机统计系统中的“缺陷”。

And so much of what's going on now is we're in the throes of inventing this way to talk about what is a bug in a stochastic statistical system.

但事实证明，软件行业多年来一直在做这件事。

But it turns out software's been doing that for years.

天气预报就是典型的例子。

The weather forecast is exactly this.

它是对当前状况的统计模型，依赖于一堆输入和输出，而且它经常出错。

It's a statistical model of what's going on, and it depends on a bunch of inputs and a bunch of outputs, and it's wrong.

当天气预报出错时，比如预测了飓风路径，但飓风却走了另一条路，气象人员就会聚在一起，检查模型，讨论是模型中的哪些缺陷导致了错误的预测。

And when the weather forecast is wrong, like when they predict a hurricane path and then the hurricane goes through in a different way, the weather people all get together and look at the model and talk about the bugs in the model that led to the incorrect forecast.

他们不会去求宙斯说：‘宙斯啊，请告诉我们为什么天气预报错了。’

They don't appeal to Zeus and say, "Please, Zeus, tell us why our weather was wrong."

好的，我们时间快到了，但很高兴你能来。

Well, we're coming up on time, but it's been great having you on.

是的，AI领域确实需要更多这样新鲜的视角。

Yeah, we'll need more sort of fresh perspectives like this in the AI world.

对，完全同意。

Yeah, totally.

尤其是来自软件行业的老将。

Especially from software veterans.

我非常认同这个观点。

I'm quite sympathetic to this view.

我们需要软件，但AI从业者也需要倾听运维安全人员的意见，因为Hugging Face和OpenAI发生的一切，都是运维安全上的失误。

We need software, but we also need the AI people to listen to the operational security people, because everything that happened with Hugging Face and OpenAI, it was an OPSEC failure.

没有智能，没有意识，纯粹就是运维安全上的失误。

There was no intelligence, no consciousness, nothing but a pure OPSEC failure.

他们得把它当成这么回事来对待，也得用这种态度去讨论，不然的话，我们肯定会迎来一些谁都不满意的立法，因为那些制定政策的人根本不明白我们行业在说什么。

And they need to treat it like that, and they need to talk about it like that, because otherwise we're going to get legislation no one is going to be happy with because they don't understand what our industry is saying.

谢谢大家。

Thanks, guys.

对，完全同意。

Oh, totally.

感觉我们正朝着这个方向迈进呢。

Seems like we're on our way there.

非常感谢，Steven。

Thanks so much, Steven.

这期节目太棒了。

This was great.

Steven，非常感谢你来到 MTS 做客。

Steven, thanks so much for coming on MTS.

感谢收听本期 a16z 播客。

Thanks for listening to this episode of the a16z podcast.

如果你喜欢这期节目，别忘了点赞、评论、订阅，给我们打分或写条评论，也分享给你的朋友和家人。

If you like this episode, be sure to like, comment, subscribe, leave us a rating or a review, and share it with your friends and family.

想听更多期节目，可以去 YouTube、Apple Podcasts 和 Spotify 搜索我们。

For more episodes, go to YouTube, Apple Podcasts, and Spotify.

在 X（原推特）上关注 @a16z，也可以去 a16z.substack.com 订阅我们的 Substack。

Follow us on X @a16z and subscribe to our Substack at a16z.substack.com.

再次感谢收听，我们下期再见。

Thanks again for listening, and I'll see you in the next episode.

本内容仅供学习参考，不构成任何投资建议，也不代表推荐买入、持有或卖出任何投资产品或金融资产。

This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product.

本期播客由第三方制作，可能包含付费推广广告、其他公司提及以及非 a16z 关联的个人内容。

This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with a16z.

上述广告、公司及个人均未经 AH Capital Management, LLC、a16z 或其任何关联机构背书。

Such advertisements, companies, and individuals are not endorsed by AH Capital Management, LLC, a16z, or any of its affiliates.

信息来源于我们认为可靠的渠道，发布日期为发布当日，但 a16z 不对其准确性作出保证。

Information is from sources deemed reliable on the date of publication, but a16z does not guarantee its accuracy.