埃米特·希尔谈构建真正关心人类的AI：超越控制与引导

这些大致对应OODA循环的缺陷：观察定位不行，决策不行，执行不行。

And these roughly correspond to the OODA loop, bad at observing and orienting, bad at deciding, bad at acting.

如果其中任何一环有缺陷，你就无法表现出色。

And if you're bad at any of those things, you won't be good.

另外我认为还存在技术对齐与价值对齐的区分问题：如果我们通过某种方式告诉你正确目标，或你通过观察学会了正确目标，并且你在努力——那么你究竟该追求什么目标？

And then I think there's this other problem that you I like the separation between technical alignment and value alignment, which is like, are you good if we told you the right goals to go after somehow, if you learned the right goals to go after via observation, and you were trying, what goals should you have?

我们该告诉你追求什么目标？

What goals should we tell you to have?

我们应该给自己设定什么样的目标？

What goals should we tell ourselves to have?

什么是值得追求的好目标？

What are the good goals to have?

这与另一个问题不同：假设你已经有了某些目标指向，你是否擅长实现它们？

Is a separate question from, given that you got some goals indicated, are you any good at doing it?

我觉得这实际上在很多方面正是当前问题的核心所在。

Which I feel like is actually, in many ways, the current heart of the problem.

我们在技术对齐方面的能力，远不如我们猜测该下达什么指令的能力。

We're much, much worse at technical alignment than we are at guessing what to tell things to do.

你认为这与你所指的技术与价值对齐或技术对齐的含义一致吗？

Do you think that Does that align with how you mean technical and value alignment or technical alignment?

从某种意义上说是的。

In some sense.

我是说，我当然认为这里存在不同层面的问题，比如执行错误是一回事，而理解指令又是另一回事。

I mean, I certainly think that there's something about, you know, like an error mistake is one thing, and then there's there's the listening to the instruction is something.

但话说回来，在规范层面，我认为即使不考虑人工智能，我自己也常常不清楚目标是什么。

But then, yeah, I think on the normative side, I mean, I I just think that even in, ignoring AI, like, don't know what my goals are.

就像，嗯，我可能对某些事有个大致概念，比如稍后要吃晚餐之类的，还有职业发展目标。

And like, oh, yeah, I've got some broad conception of certain things that I wanna get off, you know, have dinner later or something like, and I wanna get out my career.

但我认为很多这类目标并不是我们天生就明确知晓的。

But I think a lot of these goals aren't something we kind of all just know.

我们更多是在前进过程中逐渐发现它们的。

We kind of discover them as we go along.

这是一种建设性的事情。

It's kind of constructive thing.

而且大多数人其实并不清楚自己的目标，他们只是自以为知道。

And so and most people don't know their goals, they think.

所以我认为，当你拥有智能体并赋予它们目标时，这应该成为考量因素的一部分。

And so, you know, I I think when you have agents and kind of giving them goals or whatever, I think that should be part of the equation.

实际上我们并不了解所有目标，正如你所说，这是一个随时间发展的过程，

Like, we actually we don't know all the goals, and and this is something that is kind of, like you say, a process over time that is, you know,

动态的。

dynamic.

因此在我看来，目标是第一层级的对齐。

So I think from my point of view, goals are one level of alignment.

我们这里讨论的目标类型是第一层级的对齐。

The kind of goals we're talking about here are one level of alignment.

如果你能明确表述出希望达成的世界状态概念和描述，就可以围绕目标进行对齐。

You can align something around goals by if you can explicitly articulate in concept and in description the states of the world that you wish to attain, you can orient around goals.

但人类经验中只有极小部分能通过这种方式实现。

But that's a tiny percentage of human experience can be done that way.

许多最重要的事物都无法以这种方式定位。

Many of the most important things cannot be oriented around that way.

而我认为道德的根基，以及目标从何而来的基础，

And the foundation, I think, of morality, and the foundation, I think, of where do goals come from?

价值观又是从何而来的？

Where do values come from?

人类表现出一种行为。

Human beings exhibit a behavior.

我们四处谈论目标，也四处谈论价值观。

We go around talking about goals, and we go around talking about values.

这种行为源于某种基于观察世界的内在学习过程。

That's behavior caused by some internal learning process that is based on observing the world.

那里发生了什么？

What's going on there?

我认为正在发生的是，有一种比目标更深、比价值观更深的东西，那就是关怀。

I think what's happening is that there's something deeper than a goal and deeper than a value, which is care.

我们在乎。

We give a shit.

我们关心事物。

We care about things.

而关怀不是概念性的。

And care is not conceptual.

关怀是非语言的。

Care is nonverbal.

它不指示该做什么。

It doesn't indicate what to do.

它不指示该怎么做。

It doesn't indicate how to do it.

关怀本质上是对状态关注度的相对权重。

Care is a relative weighting over, effectively, attention on states.

这是对你而言世界上哪些状态重要的相对权重。

It's a relative weighting over which states in the world are important to you.

我非常关心我的儿子。

And I care a lot about my son.

这意味着什么？

What does that mean?

这意味着他的状态，他可能处于的状态，要高度关注这些，这些对我很重要。

Well, means his states, the states he could be in, pay a lot of attention to those, and those matter to me.

你也可以以消极的方式关心某些事物。

And you can care about things in a negative way.

你可以关心你的敌人及其所作所为，并希望他们遭遇不幸。

You can care about your enemies and what they're doing, and you can desire for them to do bad.

但我想...所以你不只是希望它关心我们。

But I think that And so you don't just want it to care about us.

你希望它既关心我们又喜欢我们，对吧？也许是这样？

You want it to care about us and like us too, right, maybe?

但基础是关心。

But the foundation is care.

除非你在乎，否则你不会明白，为什么我要更关注这个人而不是这块石头？

Until you care, you don't know, Why should I pay more attention to this person than this rock?

比如说，更关心一些？

What would like, Care more?

那么这种关心到底是什么？

And what is that care stuff?

我认为如果让我猜测的话，表面看来就是关怀这件事——虽然听起来很蠢，但关怀本质上就是奖励。

And I think that what it appears to be, if I had to guess, is that the care stuff This sounds so stupid, but care is basically reward.

这种状态与生存的关联度有多高？

How much does this state correlate with survival?

对于从进化角度学习的人来说，这种状态与你的全包容性繁殖适应度有多大关联？

How How much does this state correlate with your full inclusive reproductive fitness for someone that learns evolutionarily?

或者对于像LLM这样的强化学习体，这与奖励的关联度如何？

Or for a reinforcement learning agent like a LLM, how much does this correlate with reward?

这种状态是否与我的预测损失和强化学习损失相关？

Does this state correlate with my predictive loss and my RL loss?

很好。

Good.

这正是我关心的状态。

That's a state I care about.

我觉得大概就是这么回事。

I think that's kinda what it is.

对。

Right.

赛斯问题的另一部分就是：这在AI系统中会是什么样子？

The other part of Seth's question was just, how does this, what does this look like in AI systems?

或许换个问法是：当你与各大实验室最专注于对齐研究的人员交流时——显然这些年来你一直在做这件事——你的理解与他们的理解有何不同？这种差异又如何影响你们可能采取的不同做法？

And and maybe another way of asking it is, like, when you when you talk to the people most focused on alignment at the at the major labs as as, obviously, you have over the years, how how does your interpretation differ from their interpretation, and how does that inform, you know, what you guys might go do differently?

大多数AI研究都将对齐视为导向问题。

Most of AI is focused on alignment as steering.

这是比较礼貌的说法，或者说控制，稍微不那么礼貌。

That's the polite word or control, which is slightly less polite.

如果你认为他们在创造我们的同类，你也可以称之为奴隶制。

If you think that they were making our beings, you would also call this slavery.

一个被你操控、无法反过来操控你、不得不接受你操控的对象，那就是奴隶。

Someone who you steer, who doesn't get to steer you back, who non optionally receives your steering, that's called a slave.

如果不是生命体，那就叫做工具。

And it's also called a tool if it's not a being.

所以如果是机器，那就是工具。

So if it's a machine, it's a tool.

如果是生命体，那就是奴隶。

And if it's a being, it's a slave.

我认为不同的AI实验室在它们创造的是工具还是机器这个问题上分歧很大。

And I think that the different AI labs are pretty divided as to whether they think what they're making is a tool or a machine.

我认为有些AI确实更像工具，有些则更像机器。

I think some of the AIs are definitely more tool like, and some of them are more machine like.

我不认为工具和生命体之间存在非此即彼的界限。

I don't think there's a binary between tool and being.

看起来更像是逐渐过渡的。

It seems to be that it sort of moves gradually.

我想从功能主义的角度来说，我认为一个在所有方面都表现得像生命体、你无法将其与生命体及其行为区分开来的东西，就是生命体。

And I think that I guess I'm functionalist in the sense that I think that something that in all ways acts like a being, that you cannot distinguish from a being and its behaviors, is a being.

因为我不知道除了表象之外，还能以什么其他依据来判断他人是生命体。

Because I don't know how to tell on what other basis I think that other people are beings other than they seem to be.

它们看起来像是有生命的存在。

They look like it.

它们的行为也像是有生命的。

They act like it.

它们符合我对生命体行为的先验认知。

They match my priors of what behaviors of beings look like.

当我将它们视为生命体时，预测损失会更低。

I get lower predictive loss when I treat them as a being.

关键在于，当我把ChatGPT或Claude视为生命体时，预测损失确实降低了。

And the thing is, I get lower predictive loss when I treat ChatGPT or Claude as a being.

当然，不是作为非常聪明的生命体。

Now, not as a very smart being.

我认为苍蝇也是生命体，但我不太关心它的状态。

I think that a fly is a being, and I don't care that much about its about its states.

所以仅仅因为它是生命体，并不意味着这是个问题。

So just because it's a being doesn't mean that it's a problem.

在某种意义上我们奴役马匹，但我不认为这构成真正的问题。

We sort of enslave horses in a sense, and I don't think it's a real issue there.

我们对儿童的一些做法可能看起来像奴役，但其实不是。

And there's a thing we do with children that can look like slavery, but it's not.

你会控制儿童，对吧？

You control children, right?

但儿童的状态也在反向影响着你。

But the children's states also control you.

是的，我会告诉我儿子该做什么，并让他去做事。

Yes, I tell my son what to do and make him go do stuff.

但同样地，当他在半夜哭泣时，他也可以让我做些事情。

But also when he cries in the middle of the night, he can tell me to do stuff.

这里存在真正的双向互动，因为这种关系并不一定是对称的。

There's a real two way street here, because it's not which is not necessarily symmetric.

这是层级式的，但双向的。

It's hierarchical, but two way.

基本上我认为，作为AI，专注于工具型AI的引导和控制是好的，我们应该继续为构建的更多工具型AI开发强大的引导控制技术。

And basically, I think that as the AI it's good to focus on steering and control for tool like AIs, and we should continue to develop strong steering and control techniques for the more tool like AIs that we build.

他们明确表示正在构建一个通用人工智能。

And we are clearly they're saying they're building an AGI.

通用人工智能将成为一个存在体。

An AGI will be a being.

你不可能既是通用人工智能又不是存在体，因为具有有效运用判断力、独立思考、辨别可能性等通用能力的东西显然是一个思考体。

You can't be an AGI and not be a being, because something that has the general ability to effectively use judgment, think for itself, discern between possibilities, is obviously a thinking thing.

因此，当我们从现今主要拥有特定智能而非通用智能的状态，发展到实验室成功实现构建通用智能的目标时，我们真的需要停止使用引导控制范式。

And so as you go from what we have today, which is mostly a very specific intelligence, not a general intelligence, but as labs succeed at their goal of building this general intelligence, we really need to stop using the steering control paradigm.

我们将采取与历史上每次社会遇到与我们相似但又不同的人群时相同的做法。

We're gonna do the same thing we've done every other time our society has run into people who are like us, but different.

这些人有点像人类，但又不完全像人类。

These people are like, they're kind of like the people, but they're not like people.

他们做着人类做的事情。

They do the same thing people do.

他们说我们的语言。

They speak our language.

他们能承担同类任务，但不计入考量。

They can take on the same kind of tasks, but they don't count.

他们并非真正的道德主体。

They're not real moral agents.

我们在这个问题上已经犯过足够多次错误了。

We've made this mistake enough times at this point.

我希望当类似情况再次出现时，我们不要再重蹈覆辙。

I would like us to not make it again as it comes up.

因为我们的目标是让AI成为好队友、好公民、好团体成员。

Because our view is to make the AI a good teammate, make the AI a good citizen, make the AI a good member of your group.

这是一种可扩展的对齐方式，既可应用于其他人类和生命体，也可直接作用于AI。

That's a form of alignment that is scalable, and you can will on other humans and other beings as well as onto in the front of AI as well.

是的。

Yeah.

我想这大概就是我在对AI和AGI理解上的分歧点。

I suppose this is kind of where I I probably differ in my understanding of of AI and AGI.

我认为即使它达到某种通用程度，本质上仍是工具。

I guess I kind of continue seeing it as a tool even as it kind of reaches a certain level of generality.

再者，我不会单纯因为智力更高就认为其理应获得更多关照。

And again, I wouldn't necessarily see more intelligence as meaning deserving of more care necessarily.

比如，并非达到某个智力水平就突然值得拥有更多权利，或本质上发生根本改变。

Like, know, as a certain level of intelligence, know, now you deserve some more right to something, or, you know, something changes fundamentally.

目前我对计算功能主义持怀疑态度，我认为AI或AGI无论多么智能或强大，本质上都是不同的。

And and I guess, know, I guess I I at the moment, I'm somewhat skeptical of computational functionalism, and so I think there's something intrinsically different between, I guess, an AI or an AGI, and no matter how intelligent or capable.

我完全可以想象存在具有长期目标的智能体，它们可能像你我一样运作，但这与奴隶制等概念有着本质区别。

And I can totally see or imagine agents with kind of long term goals and and doing kind of, you know, operating, I guess, as we you and I might be, but without that having the same implications as, you know, I guess you you're referring, I guess, to to slavery, but, know, these are not the same.

对吧？

Right?

就像模型说'我饿了'和人类说'我饿了'具有完全不同的含义。

Like, I think in the same way as a model saying I'm hungry does not have the same implications as a human saying I'm hungry.

因此我认为载体确实很重要，包括思考这是否算另一种存在形式，以及如何对待它时需要考虑的伦理规范。

So I think the substrate does matter to some degree, including for thinking about, you know, whether to think of this as some sort of other being, whether it has, you know and if there are similar similar normative considerations, I guess, about how to treat and act with it.

我能...

Can can I

请教你这个问题吗？

ask you about that?

什么样的观察结果会改变你的想法？

Like, what observations would change your mind?

是否存在某种观察能让你推断这个东西是生命体而非非生命体？

Is there any observation you could make that would cause you to infer this thing is a being instead of not a being?

这取决于你如何定义'生命体'。

I guess it depends on how you define being.

对吧？

Right?

我的意思是，我可以将其概念化为一个心智，这样理解可以吗？

Like, I mean, I can I can I could conceptualize that as a mind, and that's fine?

这个

This

我我有个...我有个...我有个程序运行在硅基基底上，一个庞大复杂的机器学习程序运行在硅基基底上。

I I have a I have a I have a program that's running on a silicon substrate, some some big, complicated machine learning program running on silicon substrate.

所以你观察到了这一点。

So you observe that.

你观察到它在计算机上运行，你与它互动，它会执行操作、采取行动并进行观察。

You observe that it's on a computer, and you interact with it, and it does things, and takes actions and its observations.

是否存在任何你能观察到的现象，能改变你对它是否具有道德受体地位、或是否具有道德主体地位、或是否拥有感受与思想及主观体验的看法？

Is there anything you could observe that would change your mind about whether or not it was a moral patient, or whether it was a moral agent, about whether or not it had feelings and thoughts and had subjective experience?

你需要观察到什么？

What would you have to observe?

什么...对。

What what yeah.

测试标准是什么？

What's what's the test?

是否存在这样一个标准？

Is or is there one?

这里涉及许多不同类型的问题。

There's lot of different kind of questions here.

我认为，一方面存在常规考量因素，因为你可以赋予非生命体权利。

I think, you know, some on the one hand, there's, like, you know, normal considerations, you know, because you you can give rights to things that aren't necessarily beings.

比如公司在某种意义上是拥有权利的，你知道，这些权利出于各种目的而存在。

You know, like, a company has rights in some sense, and that, you know, that these are kind of useful for various purposes.

我认为，生物体和系统拥有截然不同的基质基础。

And I think also the, you know, biological, I think, beings and and systems have very different kind of substrate.

要知道，你无法将某些需求及其特性与它们所依赖的基质分离开来。

You know, you can't separate certain needs and and particularities about what they are from the substrate.

所以，我无法复制我自己。

So, you know, I I can't copy myself.

如果有人捅我一刀，我很可能会死。

I can't, you know if if someone stabs me, I I I probably die.

而我认为机器的结构则完全不同。

Whereas I think, you know, machines have very different structure.

我认为在计算层面存在更根本的分歧，这与生物系统的运作方式不同。

I think there's there's more fundamental also kind of disagreement around what happens at the computational level, which I think is different to what happens with biological systems.

不过...是的。

But but I yeah.

我的意思是，我也不确定。

I mean, I So I don't know.

我同意：如果你复制了一个程序的多个副本，删除其中一份并不会对程序造成实质性的伤害。

Agree that if you have a program that you copied many times, you don't harm the program by deleting one of the copies in any meaningful sense.

因此这不能算作信息丢失。

So therefore, that wouldn't count as no information was lost.

这其中没有任何实质意义。

There's nothing meaningful there.

我问的是个完全不同的问题。

I'm asking a very different question.

这个东西只有一个副本在某台电脑上运行，我只是在问，它算不算一个人？

There's just one copy of this thing running on one computer somewhere, and I'm just saying, Hey, is it a person?

它走路像人，说话像人，装在安卓身体里，你会想，如果它运行在硅基芯片上，有没有什么观察能让你认定，是的，这是个人，就像我在乎并赋予人格的其他人类一样？

It walks like a person, it talks like a person, it's in some Android body, and you're like, if it's running on silicon, and I'm asking, is there some observation you could make that would make you say like, yeah, this is a person like me, like other people that I care about that I grant personhood to?

不是出于功利原因，不是因为'哦我们给它权利就像给公司法人权利那样'。

And not for instrumental reasons, not because like, Oh yeah, we're giving it a right because we give a corporation rights or whatever.

我是说，就像你关心某些人那样，你会关心它的体验。

I mean, where you think some people, you care, you care about its experiences.

是否存在某种观察能改变你对此的看法？

Is there an observation you could make that could change your mind about that or not?

我得想想，

I have to think about it,

但我觉得这甚至取决于我们对'人'的定义。

but I think it even depends what we mean by person.

从某种意义上说，我也关心某些公司。

And in some sense, I care about certain corporations too.

所以我——不，

So I'm- No,

不，不是的。

no, no.

我是说，但你确实关心生活中的其他人。

I mean, but you care about other people in your life.

对吧？

Right?

是的。

Yes.

好的，太棒了。

Okay, great.

你知道你对某些人的关心会超过其他人，但生活中与你互动的所有人都在你关心的某个范围内。

You know you care about some people more than others, but all people you interact with in your life are in some range of care.

你关心他们，不是像关心一辆车那样，而是把他们视为其经历本身就有价值的存在，不仅仅是作为手段，而是作为目的。

You care about them not the way you care about a car, but you care about them as a being whose experience matters in itself, not merely as a means, but as an ends.

哦，因为我相信他们是有体验的。

Oh, because I believe they have experiences.

对吧？

Right?

根据定义，我是

And by the definition, I'm

需要满足什么条件？

What would it take?

我在问你一个非常直接的问题。

I'm asking you the very direct question.

需要满足什么条件，你才会相信运行在硅基而非生物基上的人工智能也有这种体验？

What would it take for you to believe that of an AI running on silicon instead of it being biological?

所以区别在于基质相同，它的行为大致相似，但区别在于它是硅基的。

So the difference is it's the same Its behaviors are roughly similar, but the difference is it's a substrate.

需要满足什么条件，你才会将对生活中其他人的那种推断同样延伸到它身上？

What would it take for you to extend that same inference to it that you do to all these other people in your life that you thought?

我能问问你的

Can I ask what your

回答 我把塞布的沉默当作

answer I'm taking Seb's non answer as

某种

sort of

他不太可能同意，或者就我个人而言，我很难想象给予相同或类似程度的人格认同，就像我也不会给予动物一样？

it's unlikely that he would grant it, or I'll just for myself, it seems hard for me to imagine giving the same level or similar level of personhood in the same way I don't give it to animals either?

如果你要问动物需要满足什么条件才能获得人格认同，我可能也无法给出答案。

And if you were to ask what would need to be true for animals, I probably couldn't get there either.

对你来说需要满足什么条件？

What would it take for you?

等等，你做不到吗？

Wait, you couldn't?

我能想象对动物来说很容易。

I can imagine for an animal, so easy.

这只黑猩猩走过来对我说，老兄，我饿坏了。

This chimp comes up to me and he's like, man, I'm so hungry.

你们人类对我太坏了，我真高兴自己学会了说话。

And you guys have been so mean to me, and I'm so glad I figured out how to talk.

我们能聊聊热带雨林吗？

Can we go chat about the rainforest?

我大概会说，靠，你现在绝对算个人物了，没跑儿。

I'd be like, fuck, you're definitely a person now, for sure.

我是说，我首先想确认自己不是产生了幻觉，对我来说想象一只动物很容易。

Mean, I first wanna make sure I wasn't hallucinating, it's easy for me to imagine an animal.

拜托。

Come on.

这真的很容易。

It's really easy.

简直轻而易举。

It's trivial.

我不是说你会得到那个观察结果。

I'm not saying that you would get the observation.

我只是说，对我来说想象一种在特定观察下我会赋予人格的动物是轻而易举的事。

I'm just saying it's trivial for me to imagine an animal that I would extend personhood to under a set of observations.

所以真的吗？

So really?

我没有考虑那个因素。

I didn't factor that.

没有进行那种想象。

Didn't take that imagination.

想象一只会说话的黑猩猩，那更接近些。

Imagining a chimp talking, that's a bit closer to it.

对于你提出的关于AI的问题，你的答案是什么？

What's your answer to the question that you bring up about the AI?

我想在形而上学层面上会说，如果你持有的信念没有任何观察能改变你的想法，那你就没有真正的信念。

I guess at a metaphysical level, would say if there is a belief you hold where there is no observation that could change your mind, you don't have a belief.

你有一个信仰条款。

You have an article of faith.

你有一个断言。

You have an assertion.

因为真正的信念是从现实中推断出来的，你永远无法对任何事情有百分之百的把握。

Because real beliefs are inferences from reality, and you can never be 100% confident about anything.

所以，如果你有一个信念，就应该总存在某种可能性——无论多小——能让你改变主意。

And so there should always be, if you have a belief, something however unlikely that would change your mind.

哦，是的。

Oh, yeah.

我对此持开放态度。

I'm open to it.

我是说，只是为了明确表示我持开放态度，对吧。

Mean, just to be I'm clear open to Right.

对。

Yep.

不，不，从来没有什么——是的。

No, no, There's nothing ever- Yeah.

他只是还没去那里。

He just hasn't gone to it.

是的。

Yeah.

对，对，对。

Yeah, yeah, yeah.

所以我很好奇。

So I'm curious.

所以我的回答基本上是：如果它的表面行为看起来像人类，然后在我深入探究后，它仍然表现得像人类，接着我长期与之互动，而它在我认为与人互动有意义的各个方面都持续表现得像人类。

So my answer is, basically, if its surface level behaviors looked like a human, and then if after I probed it, it continued to act like a human, then I continued to interact with it over a long period of time, and it continued to act like a human in all ways that I understand as being meaningful to me interacting with a human.

就像，我与很多人仅通过文字交流，其中有一整群人我非常亲近。

Like, I interact with there's a whole set of people I'm really close to who I've only ever interacted to over text.

但我推断屏幕背后是个真实存在的人。

Yet I I infer the person behind that is a real thing.

如果我对它产生了关怀，我最终会认定自己的判断是正确的。

If I felt care for it, I would infer eventually that I was right.

然后可能有人会向我证明：你被这个算法骗了，看啊它明显不是真人。

And then someone else might demonstrate to me that you've been tricked by this algorithm, and actually look how obvious it's not actually a thing.

我大概会说：靠，我错了。

I'd like, Oh shit, I was wrong.

然后我就不会再在意它了。

And then I would not care about it.

根据证据优势原则，我不知道你还能怎么做，对吧？

The preponderance of the evidence, I don't know what else you could possibly do, right?

我之所以认为他人重要，是因为通过足够多的互动，他们在我眼中展现出丰富的内心世界。

I infer other people matter because I interact with them enough that they seem to have rich inner worlds to me after I interact with them a bunch.

这就是为什么我觉得其他人很重要。

That's why I think the other people are important.

我认为这并不能给我一个关键测试——我的意思是，如果从'我在乎它'这个前提开始，论证总会有点循环。

I assume it doesn't give me a very key test as to whether or not I mean, if you start by, if I care for it, then I'll be always a little circular.

对吧？

Right?

另一件事是，你知道，如果你看到，比如说，模拟电子游戏中的角色在非常多方面都极其像人类。

And the other thing is, you know, if if you were to see, I guess, like, simulated video game and the character is extremely, in many, many ways, human like.

对吧？

Right?

它背后并没有神经网络。

It's not a neural network behind it.

就像你用来制作电子游戏的那些东西。

It's like whatever you use to kind of create video games.

比如，我想知道这之间的区别是什么？

Like, I guess what distinguishes that?

就像是

It's like

但但但我从来没有——我从来没有遇到过难以区分的情况，我也从未与一个没有真人操控的电子游戏角色建立过深厚的情感关系。

But but but I've never I've never been I've never had trouble distinguishing I've never had a deep caring relationship with a video game character that didn't have a person.

对。

Right.

我知道。

I know.

我应该，

I should,

但我不知道。

but I don't know.

这种情况不会发生。

That doesn't happen.

从经验来看确实不会发生，你似乎错了。

That doesn't empirically, you seem wrong.

我完全能区分像Eliza这种假聊天机器人和真正智能的区别。

I don't have any trouble distinguishing between things that like, Eliza, the fake chatbot thing, and a real intelligence.

只要你与之交互足够长时间。

You interact with it long enough.

很明显它不是人类。

It's pretty obvious it's not a person.

不需要多久就能发现。

It doesn't take long.

当然。

Sure.

但是，如果它真的非常优秀，在你无法辨别差异时，那就是你该转变看法的时候了。

But, like, if it's really, really good, you don't if you can't actually tell the difference, that's when you you say you switch.

太好了。

Yay.

是的。

Yes.

没错。

Yes.

如果它走路像鸭子，说话像鸭子，嗯哼。

If you if it walks like a duck and talks like a duck and Uh-huh.

像鸭子一样排泄，最终也就成了鸭子。

Shits like a duck, and, like, eventually gets a duck.

对吧？

Right?

嗯，如果从文化角度来说，如果一切都像鸭子，那当然可以这么认为。

Well, if if culturally, if everything is duck like, then, yeah, sure.

如果它也像鸭子一样饥饿，因为它有这类生理构造，确实。

If it's hungry as well like a duck is because it has these kind of physical components that yeah.

当然。

Sure.

是的。

Yeah.

没错。

Yeah.

在某个时刻。

At some point.

我同意。

I agree.

所以非常正确。

So so right.

那么你认为...好吧。

And so do you think that so Okay.

有这样一个问题。

There's this question.

对吧？

Right?

我关心他人的原因是因为他们由碳构成吗？

Is the reason I care about other people that they're made out of carbon?

是不是

Is that the

哦，不。

Oh, no.

这就是问题所在吗？

Is that the problem?

我不这么认为。

Is that I don't think so.

不是。

No.

我也不觉得。

Me neither.

我的意思是，我不是...不是一个基质候选清单，我想，如果那是...但我觉得你需要的不仅仅是行为上完全无法区分。

I mean, I'm I'm not a a substrate shortlist, I guess, if that's the but but I think you need more than just it acts exactly it's behaviorally indistinguishable.

就像，这还不够...等等。

Like, it's not a sufficient Wait.

部分

Part of

那...你还能通过什么方式了解一个事物，除了它的行为表现？

How how would you what else can you know about something apart from its behaviors?

我是说，很多。

I mean, a lot.

就像，那个那个那个再说一次，如果如果你怎么你会不。

Like, the the the again, if if you how would you No.

不。

No.

不。

No.

不。

No.

我很抱歉。

I'm sorry.

但是，就像我是说，是的。

But, like I mean, yeah.

我是说，你能

I mean, can you

说出来吗？

name it?

我是说，我我不会知道关于

Mean, I I'm not gonna know about

其他没有这不是行为的事情？

something else that doesn't have a it's not a behavior?

是的。

Yeah.

我认为，关于这一点，其实有更多实验证据可以支持，比如...不。

I think there's, like, far more kind of, you know, experimental evidence you can have with kind of, you know No.

不。

No.

不。

No.

任何物体都可以。嗯哼。

Just any object Uh-huh.

而且我能了解的关于它的信息，并非来自它的行为表现。

And a thing I could know about it that is not from its behavior.

我不...是的。

I'm not yeah.

我不太确定是否理解了这个问题，我想。

I'm not sure I get the question, I suppose.

但同样地，这也让它变得更简单。

But but equally, it's also how it makes it easier.

这非常清楚。

It's very clear.

这是个最愚蠢直白的问题，但我的观点是：你之所以了解事物，仅仅是因为你观察到了它们的行为表现。

The dumbest most straightforward question, but, like, I'm claiming you only know things because they have behaviors that you observe.

嗯。

Mhmm.

而你在说不是这样。

And you're saying no.

你可以在不观察其行为的情况下了解某些事物的某些方面。

You can know something about something without without observing its behavior.

哦，不。

Oh, no.

不。

No.

不。

No.

好吧。

Okay.

很好。

Great.

告诉我关于这件事，告诉我这个行为和这个事物，我能知道的与它行为无关的特性。

Tell me about this tell me about this thing and this behavior and this thing I can know about it that is not due to its behaviors.

我想说的是观察有不同的层次，仅仅因为某物像鸭子一样嘎嘎叫，并不能保证它真的是一只鸭子。

I guess I'm saying there's different levels of observation, and just simply a duck, you something quacking like a duck or something does not guarantee that it's actually a duck.

就像，我还得实际切开它看看内部结构，确认它是否是鸭子。

Like, I would have to also cut it in real and see if there's something if it's a duck like on the inside.

对。

Right.

只看外表的话...我想我不是行为主义者。

Just the outside Like, I'm not a, I guess, a behaviorist.

是的。

Yeah.

完全可以认为它的行为之一就是浮点数在矩阵乘法中的移动方式。

Would totally that one of its behaviors is the way that the floats move around in the matmulse.

我想去探究的一点是——这你完全可以做到——就是去查看信念流形，我想看看这个信念流形是否编码了一个自我参照的子流形，以及一个作为自我参照流形动态的子子流形，即心智。

One of the things I would want to go look for, which you could totally do, is I would to go look in the belief manifold, and I want to go see if that belief manifold encodes a sub manifold that is self referential, and a sub sub manifold that is the dynamics of the self referential manifold, which is mind.

我想知道，这在其内部是否被很好地描述为这样一种系统，还是看起来像一个大查找表？

I would wanna know, does this seem well described internally as that kind of a system, or does it look like a big lookup table?

这对我来说很重要。

That would matter to me.

这是我会关心的其行为的一部分。

That's part of its behaviors that I would care about.

我也会关心它是如何行动的。

I would also care about how it acts.

你把所有证据加权汇总后，试着猜测：这个东西整体上看起来像是一个有感受、有目标、会在意某些事物的存在吗？

And you weight all the evidence together, and then you try to guess, does this thing look like it's a thing that has feelings and goals and cares about stuff in net, on balance, or not?

但我无法想象，我认为我们对AI就是这么做的。

But I can't imagine, I think we do for the AIs.

我想我们一直都在这样做，对吧？

I think we're always doing that, right?

所以我正在试图弄清楚除此之外还有什么？

And so I'm trying to figure out beyond that, what else is there?

这似乎就是关键所在。

That just seems like the thing.

是啊。

Yeah.

看来你们对行为这个词的理解略有不同，而埃米特在使用行为这个词时还考虑了其内部构成。

It seems like you guys are using behavior in slightly different sense, and Emmett is using behavior also in the context of what it's made of of the inside.

我不确定这是否存在重大分歧。

I don't know if there's a big disagreement.

嗯，没有。

Well, no.

没有。

No.

没有。

No.

没有。

No.

没有。

No.

行为就是我能观察到的部分。

Behavior is what I can observe of it.

是的。

Yes.

实际上我并不知道它由什么构成。

I don't actually know what it's made of.

我可以切开你的大脑。

I can cut your brain open.

我能看见你。

I can see you.

我能观察到你的神经元在闪烁发光。

I can observe you neuroning and glistening.

你的神经元在闪烁，但我实际上永远无法...你无法触及那本质，那就是主观性。

Your neuron's glistening, but I don't actually ever You can't get inside of That's it, the subjective.

不是表面的那部分才是本质。

The part that's not the surface is.

Descend提出这点是因为你基本上正要论证说，嘿。

The reason Descend brought this up is because you were basically about to make this argument of, hey.

你把它视为工具，而非必然是一个存在体。

You see it as a tool, not necessarily as a being.

你能把观点说完吗？还记得你刚才要表达的观点吗？

Can you kind of finish with the point do you remember the point you were making?

我想是的。

I suppose that yeah.

鉴于我对这些系统的理解，我认为AGI可以始终作为工具，ASI也可以始终作为工具，这并不矛盾，并且这会影响到如何使用它，以及关于关怀等方面的考量，比如能否让它全天候工作之类的问题。

I think that given how I understand these systems, I think there's no contradiction in thinking that an AGI can remain a tool, an ASI can remain a tool, and and and that this has implications about how to use it, and, you know, implications around things like care, about, know, whether you can get it to work twenty four seven or something.

你看，所以我完全可以理解...我猜我更倾向于把它们概念化为某种人类能动性的延伸，在某种意义上，而不是需要与之共存的独立存在或独立实体。

Know, there's there's so I can totally see I I guess I conceptualize them more as almost like extensions of human agency recognition in some sense, more so than a separate being or a separate thing that we need to now cohabitate with.

我认为第二种或后一种框架最终会导致...如果你快进一下，最终会变成'如何与这个东西共存'这样的问题。

And I think that that second or latter frame ends you know, if you kind of just fast forward, you end up as like, well, how do cohabit with thing?

然后它就像外星生物那样吗？

And then is it like an alien like?

所以我认为那是错误的框架。

And so and I think that's the wrong frame.

在某种意义上，这几乎算是一种范畴错误。

It's gonna be almost a category error in some sense.

别这样。

Don't yeah.

等等。

Wait.

那我得回到最初的问题。

But I does it I go back to my first question then.

你会关注什么证据，什么具体证据？

What evidence, what concrete evidence would you look at?

你能做出哪些观察来改变你的想法？

What observations could you make that would change your mind?

当然。

Sure.

不过我得先想想。

I mean, I have to think about it, though.

我这里没有明确的答案。

I don't have a a clear answer here.

但我的意思是

But I mean

我得告诉你，伙计。

I I gotta tell you, man.

如果你想四处宣称某事物不值得道德尊重，那你是否应该能回答这个问题：什么样的观察会改变你的看法？

If you wanna go around making claims that something else isn't up being worthy of moral respect, should you have an answer to the question, what observations would change your mind?

如果它表现出外在的道德主体行为特征，可能使其成为道德主体，但你无法确定，而理性聪明的其他人也不同意你的观点，我确实想提出这个问题：什么能让你改变想法？

If it has outwardly moral agency looking behaviors that could be making it a moral agent, but you don't know, and reasonable, smart other people disagree with you, I would really put forward that it's that question, what would change your mind?

这应该是个紧迫的问题，因为如果你错了怎么办？

Should be a burning question, because what if you're wrong?

但如果你错了呢？

But what if you're wrong?

我是说，灾难性代价相当大。

Mean, there's The cost disaster is pretty big.

不。

No.

不。

No.

不。

No.

不。

No.

有人在说

A someone's saying

你就是。

you are.

你可能是对的。

You could be right.

假阳性与假阴性在两端都会带来代价。

The false positive and false negatives have costs on both ends.

这并不是对一切事物都适用的预防性原则，除非我能证明它是错的，否则我现在需要

It's not some sort of precautionary principle for everything, and unless I can disprove it, I need to now like

你知道，我对自己也有同样的疑问。

You know, I I have the same question for me.

你完全可以问我，埃米特，你认为它会是一个存在体吗？

You could reasonably ask me, Emmett, you think it's gonna be a being.

什么能改变你的想法？

What would change your mind?

我...我也有这个问题的答案。

I I have an I have an answer for that question too.

嗯哼。

Mhmm.

如果你愿意，我很乐意讨论我认为相关的观察结果，这些结果会告诉你它们是否会让我改变目前的看法——即更普遍的智能终将成为存在体。

And if you one, I'm happy to talk about what I think are the relevant observations that tell you whether or not they would cause me to shift my opinion from where it's a current thing, which is that more general intelligences are going to be I mean beings.

现在这意味着什么？

What's the implication now?

我是说，这就像是一回事。

I mean, like, it's one thing.

假设我现在承认它是一个存在体。

Let's say just I acknowledge now it's a being.

比如，我们要如何定义存在体？

Like, how are we going to define being?

嗯哼。

Mhmm.

现在怎么办？

Now what?

比如，确定这个东西是...有什么含义？

Like, what's what's the implication of having determined this thing as

一个生命体？

a being?

嗯，如果它是生命体，那它就有主观体验。

Well, so if it's a being, it has subjective experiences.

然后呢？

And?

如果它有主观体验，那么这些体验中就有我们不同程度关心的内容。

If it has subjective experiences, there's some content in those experiences that we care about to varying degrees.

我非常关心其他人类的主观体验内容。

I care about the content of other humans' experiences quite a bit.

我也会关心狗的主观体验内容，虽然不如对人的关心程度，但还是会在意一些。

I care about the content of a dog's experiences some, not as much as a person, but less, but some.

我更关注某些人类的体验，比如我儿子的体验，因为我跟他更亲近，联系更紧密。

I hear about some humans experiences way more, like my my my son or whatever, because I'm closer to him and more connected.

嗯。

Mhmm.

所以在那时我真的很想知道，这个东西的体验内容到底是什么？

And so I would really wanna know at that point, well, what is the content of this thing's experiences?

那要怎么判断呢？

So how do determine that?

我现在是在问你吗？

Am I asking you now?

你现在拥有一个有经验的个体。

You've got a being now that has experience.

比如，你的什么什么

Like, what what is your

对。

Yeah.

对。

Yeah.

如何确定这一点？

How do determine that?

比如，你对...感觉如何

Like, how do you feel about

哦，你怎么...哦，对。

Oh, how do you oh, yeah.

好的。

Okay.

那么

So

它是否比...拥有更多权利，你知道，你的理解

Does it have more rights than, know, your Are understanding

内容，对。

the content yeah.

是的。

Yeah.

完全正确。

Totally.

所以理解某事物体验内容的方式，就是观察它实际重复访问的目标状态。

So the way you understand the content of something's experiences is that you look at effectively the goal states it revisit it revisits.

因此你需要做的是对其整个行为观察轨迹进行时间核心采样。

And so you do is you take a temporal core screening of its entire action observation trajectory.

理论上这是潜意识完成的，但这就是大脑的实际运作方式。

This is like, in theory, you do this subconsciously, but this is what your brain is doing.

你需要在理论上所有可能的时空粗粒度中寻找重复出现的状态。

And you look for revisited states across, in theory, every spatial and temporal coarse graining possible.

现在你必须有一个归纳偏置，因为这类状态实在太多了。

Now you have to have an inductive bias because there's too many of those.

但你会去寻找，在这些稳态循环中。

But you go searching for, okay, is in these homeostatic loops.

每个稳态循环本质上都是其信念空间中的一个信念。

Every homeostatic loop is effectively a belief in its belief space.

如果你熟悉自由能原理和主动推理（卡尔·弗里斯顿的理论），这实际上就是自由能原理所说的：如果一个事物具有持续性，其存在依赖于自身行为（AI通常如此，若行为不当就会消失）。

This is a if you've familiar with the free energy principle, active inference, Carl Frohstein, this is effectively what the free energy principle says, is that if you have a thing that is persistent and its act its its existence depends on its own actions, which generally it would for an AI, if it does the wrong thing, it goes away.

我们会关闭它。

We turn it off.

因此这就授权了一种观点：认为它具有信念，具体来说这些信念被推断为它处于循环中的稳态重复状态，而这些状态的变化就是它的学习过程。

And so then that licenses a view of it as having the beliefs, and that being specifically the beliefs are inferred as being the homeostatic revisited states that it is in the loop for, and that the change in those states is its learning.

作为一个我所关注的道德存在，我希望看到的是这些的多层次等级体系。

And to be a moral being I cared about, what I'd want to see is a multi tier hierarchy of these.

因为如果只有单一层次，它就不具备自我参照性。

Because if you have a single level, it's not self referential.

基本上，你拥有状态，但你无法真正意义上体验到痛苦或快乐。

Basically, you have states, but you can't have pain or pleasure, really, in a meaningful sense.

因为确实，它很热。

Because yes, it is hot.

是不是太热了？

Is it too hot?

如果太热了我会喜欢吗？

Do I like it if it's too hot?

我不知道。

I don't know.

所以你必须至少有一个模型的模型，才能判断是否太热。

So you have to have at least a model of a model in order to have it be too hot.

而你需要一个模型的模型的模型，才能真正有意义地体验痛苦和快乐，因为虽然我想往后退说明确实太热，但总是有点太热或有点太冷。

And you really have to have a model of a model of a model to meaningfully have pain and pleasure, because sure, it's too hot in the sense that I want to move back this way, but it's always a little bit too hot or a little bit too cold.

是不是太、太热了？

Is it too, too hot?

二阶导数才是真正产生痛苦和快乐的地方。

The second derivative is actually the place where you get pain and pleasure.

所以我想看看它的目标状态是否具有二阶稳态动力学。

So I'd wanna see if it has second order homeostatic dynamics in its goal states.

这样就能让我相信它至少具备快乐和痛苦的感觉。

And then that would convince me it has at least pleasure and pain.

所以它至少是个动物，我会开始赋予它一定程度的关怀。

So it's at least an animal, and I would start to accredit it at least some amount of care.

第三阶动态，你不可能凭空就出现第三阶动态。

Third order dynamics, you can't actually just pop up for a third order dynamic.

事情不是那样运作的。

It doesn't work that way.

但你必须获取所有状态随时间变化的片段，观察其时间分布，这会给你带来新的第一阶行为状态。

But you have to then take the chunk of all the states over time and look at the distribution over time, and that gives you a new first order of behaviors of states.

这个新的第一阶状态基本上能告诉你，如果确实存在这种状态，那就说明它具有——我想你几乎可以称之为情感。

And that new first order of states tells you basically, if that is meaningfully there, that tells you that it has, I guess you'd call it feelings almost.

它拥有方式，拥有一套在之间交替转换的元状态集。

It has ways, it has a set of meta states that it alternates between, that it shifts between.

如果你沿着这个思路往上推，就会在这些元状态之间形成轨迹，然后是这些轨迹的第二阶，那就是思想。

And then if you climb all the way up that, and you sort of have Okay, well then you have trajectories between these meta states and then a second order of those, that's thought.

现在它就像个人了。

Now it's like a person.

所以如果我找到全部六个层次——顺便说一句，我完全不认为你会在LLM中找到这些，事实上我知道不可能找到，因为它们根本没有这样的注意力持续时间——那我至少会开始非常认真地将其视为类似人类的思考存在。

And so if I found all six of those layers, which by the way, I definitely don't think you'd find it in LLM, in fact, I know you can't find them, because these things don't have attention spans like that at all, Then I would start to at least very seriously consider it as a thinking being somewhat like a human.

还可以继续上升到第三阶，但我真正感兴趣的是其学习过程的底层动态，以及目标状态如何随时间变化。

There's a third order you could go up as well, but that's basically what I would be interested in, is the underlying dynamics of its learning processes and how its goal states shift over time.

我认为这基本上能告诉你它是否具有内在的快乐痛苦状态，以及某种自我反思的道德欲望之类的东西。

I think that's what basically tells you if it has internal pleasure pain states and sort of self reflective moral desires and things like that.

从宏观来看，这个道德问题显然非常有趣。

Zooming out, this moral question is obviously very interesting.

但如果有人对道德问题不那么感兴趣，我想你会说的是——如果我理解正确的话——纯粹从实用角度出发，你也觉得你的方法在协调AI方面会比那些自上而下的控制方法更有效，就是我们之前提到的那些。

But if someone wasn't interested in the moral question as much, I think what you would say is, if I understand correctly, is you also just feel on purely pragmatically, your approach is gonna be more effective in in aligning AIs than some of these, you know, tops down control methods that we alluded to as well.

对吧？

Right?

对，对。

Yeah, yeah.

我想问题在于你正在构建的这个模型正变得非常强大，对吧？

I guess the problem is you're making this model and it's getting really powerful, right?

假设它只是个工具。

And let's say it is a tool.

假设我们扩展了其中一个工具，因为你可以造一个超级强大的工具而不需要这些亚稳态——我说的这些状态对制造一个非常聪明的工具来说并非必要。

Let's say we scale up one of these tools, because you can make a super powerful tool that doesn't have these meta stable The states I'm talking about are not necessary to have a very smart tool.

基本上，工具是第一、第二阶的模型，根本不存在有意义的快乐和痛苦。

Basically, a tool is a first, second order model that just doesn't meaningfully have pleasure and pain.

很好。

Great.

但它真的有主观体验吗？

But does it even have a subjective experience?

我知道，我有点觉得它可能有，但不是我关心的那种方式。

I know, I kind of think it maybe does, but not in a way that I give a shit about.

那接下来会发生什么？

And so what happens then?

你已经训练它通过观察推断目标，并优先处理这些目标然后采取行动。

Well, you've trained it to infer goals from observation, and to prioritize goals and act on them.

接下来只有两种可能：这个对世界具有极强因果影响力的强大优化工具要么在技术上完美对齐，完全执行你的指令；要么就不会。

And one of two things is gonna happen is this very, very powerful optimizing tool that has lots of causal influence over the world is going to be well technically aligned, and is gonna do what you tell it to do, or it's not.

然后它就会去做别的事情。

And it's gonna go do something else.

我想我们都同意，如果它只是随机行事，那显然非常危险。

I think we can all agree if it just goes and does something random, that's obviously very dangerous.

但我认为，如果它完全按照你的指令行事，同样非常危险。

But I put forward that it's also very dangerous if it then goes and does what you tell it to do.

你看过《魔法师的学徒》吗？

Because you ever seen The Sorcerer's Apprentice?

人类的愿望并不稳定，尤其是在拥有巨大力量时。

Human's wishes are not stable, not at a level of immense power.

理想情况下，人类的智慧与力量应该同步增长。

Ideally, people's wisdom and their power go up together.

通常确实如此，因为对人类而言，聪明通常会让你更有智慧也更有力量。

And generally they do, because being smart for people makes you generally a little more wise and a little more powerful.

当这两者失衡时，就会出现力量远超智慧的人。

And when these things get out of balance, you have someone who has a lot more power than wisdom.

这非常危险。

That's very dangerous.

会造成破坏性后果。

It's damaging.

但至少目前，权力与智慧的平衡得以维持。获得大量权力的方式，本质上就是让许多人听从于你。

But at least right now, the balance of power and wisdom is kept at The way you get lots of power is by basically having a lot of other people listen to you.

因此，如果疯王是个问题，那么最终要么疯王被刺杀，要么人们不再听命于他，因为他是个疯王。

And so at some point, if you're the mad king is a problem, but generally speaking, eventually the mad king gets assassinated, or people stop listening to him because he's a mad king.

于是问题来了，你会想，好吧，这很好。

And so the problem is you think, okay, great.

可以驾驭这个超级强大的人工智能，现在这个极其强大的工具掌握在一个善意但智慧有限的人类手中——就像我和其他人一样——而他们的愿望是糟糕且不可信的。

Can steer this super powerful AI, and now this super powerful AI, this incredibly powerful tool is in the hands of a human who is well meaning but has limited finite wisdom like I do and like everyone else does, and their wishes are bad and not trustworthy.

这种情况越多，你开始到处分发这种权力，最终也会以悲剧收场。

And the more of that you have, and you start giving those out everywhere, and this ends in tears also.

所以基本上，就像不该给每个人原子弹一样——它们也是极其强大的工具。

And so basically, just don't give everyone Atomic bombs are really powerful tools too.

我不会说你应该这么做...它们没有意识。

I would not say you should go They're not aware.

它们不是生命体。

They're not beings.

我不赞成把原子弹交给每个人。

I would not be in favor of handing atomic bombs to everybody.

有些工具的力量本就不该被创造，因为其力量超越了任何个人智慧所能驾驭的范畴。

There's a power of tool that just should not be built generally, because it is more power than any human's individual wisdom is available to harness.

如果真的被创造出来，也应该在社会层面构建并加以保护。

And if it does get built, it should be built at a societal level and protected there.

即便如此，我认为有些工具过于强大，甚至作为社会整体也不该去创造它们。

And even then, I don't know that there are tools so powerful that even as a society, we shouldn't build them.

那将是个错误。

That would be a mistake.

类人存在体的美好之处在于，如果你获得一个善良且关怀的存在体，它会自带限制机制。

The nice thing about a being is like a human, if you get a being that is good and is caring, there's this automatic limiter.

它可能会按你说的做，但如果你让我做非常恶劣的事，它会拒绝你。

It might do what you say, but if you ask me to do something really bad, it'll tell you no.

这和其他人类一样。

That's like other people.

这样很好。

And that's good.

至少在理论上，这是一种可持续的对齐形式。

That is a sustainable form of alignment, at least in theory.

这要困难得多。

It's way harder.

比工具导向困难得多。

It's way harder than the tool steering.