Is AI Reasoning Right for the Wrong Reasons?

现在我们每个人跟AI的接触程度都不一样。

We all have varying levels of interaction with AI today.

这是躲不开的。

It's unavoidable.

可能是打个客服电话，或者，怎么说呢，一次走心的长谈，也可能是为了工作做的很深入的研究。

It might be a customer service call or, I don't know, a long heartfelt chat or some very strategic work related research.

而且我们对AI给我们的回应，看法也各不相同。

And we all have varying opinions on what we get back from them.

有人觉得，嗯，还不错，就是有点小瑕疵；有人觉得这东西对我的生活一点用都没有；还有人会觉得，哇，这真的让我大开眼界。

It might be, oh, it's pretty good except or this serves no purpose in my life or wow, that really blew my hair back.

就连那些做AI、研究AI或者报道AI的人，对这些模型到底在做什么，也有各种各样的看法。

And even among people who make or study or report on AI, there's a wide range of takes on what these models are actually doing.

我想我们都同意，聊天机器人就是在海量训练数据的基础上，预测一段连贯对话中的下一个词。

We can all agree, I think, that chatbots are predicting the next word in a coherent conversation based on huge piles of training data.

可能就这么简单。

It's possible that that's all it is.

有用，但也就是个统计上的魔术把戏。

Useful, but a statistical magic trick.

很多人也觉得，这些远程的、专有的、保密的系统，其实在做的不止这些。

Lots of people also feel like these remote, proprietary, secretive systems are doing something more than that.

推理、思考、创造，甚至感受。

Reasoning, thinking, creating, even feeling.

也有可能它们在做的是完全不同的另一件事，一件我们还没有合适的词去描述的事。

It's also possible that they're doing something else entirely, something that we don't quite have the words for.

欢迎收听《Quanta播客》，我们在这里探索基础科学和数学的前沿。

Welcome to The Quanta Podcast, where we explore the frontiers of fundamental science and math.

我是Samir Patel，《Quanta杂志》的主编。

I'm Samir Patel, editor in chief of Quanta Magazine.

现在，AI公司内外的人都在用各种方法，去测试大型语言模型内部到底发生了什么，以及它们为什么能完成像解决复杂多步问题这样令人惊讶的事情。

There are all sorts of ways that folks inside and outside of AI companies, are testing to see what's happening inside LLMs and why they're able to do such surprising things like solve complex, multistep problems.

《Quanta》的常驻撰稿人John Pavlus已经报道AI十年了。

Regular Quanta contributor John Pavlus has been reporting on AI for a decade.

最近在我们名为《Qualia》的系列文章中，他探讨了一件一直困扰他的事情。

And in a recent essay in our series called Qualia, he explored something that's been gnawing at him.

这不仅仅是关于AI是否具备推理能力的问题，还在于过去几年里，我们讨论这个问题的方式发生了巨大变化，以及我们是否真的能分辨出来，或者这到底重不重要。

It's not just the idea of whether AIs are capable of reasoning, but also the fact that the way we talk about that question has changed dramatically in the last couple of years, and whether we can even tell or whether that even matters.

今天约翰和我们一起聊聊AI推理的话题。

John's with us today to talk about AI reasoning.

欢迎回到节目，约翰。

Welcome back to the show, John.

你好。

Hi.

谢谢。

Thanks.

很高兴能回来。

And it's good to be back.

那么，约翰，你的核心观点是什么？

So, John, what's the big idea?

这篇文章对我来说，最开始其实是源于一个大问题，基本上就是：AI的推理能力怎么可能是胡扯，但同时又确实存在呢？

With this one, it it kind of for me started more with like a big question, which is basically like, how can AI reasoning be BS and not at the same time?

因为两方面的证据都有。

Because there is evidence for both.

这真的可能吗？

Like, is that even possible?

这怎么说得通呢？

How would that work?

我写这篇文章时，跟很多人聊过，最后拼凑出来的想法是：是的，它确实可以既是胡扯又确实存在，但这需要你非常努力地对抗自己那些看似显而易见的直觉。

And the kind of idea that I scraped together from talking to people for this essay was that, yes, it is possible for it to be BS and not at the same time, but it requires some pretty hard paddling against your own intuitions about what seems to obviously be the case.

当我们谈论AI推理，以及一般意义上的推理时，我们到底指的是什么？

When we talk about AI reasoning, and we talk about reasoning in general, what do we mean?

AI在做什么，我们可能认为那是推理，而我们人类用推理来解决问题或做决定时，又是在做什么？两者有什么区别？

What is it that an AI is doing that is reasoning that we might be looking for versus what, say, we do when we use reason to solve a problem or make a decision?

推理就是通过一系列逻辑上相互衔接的中间步骤，得出一个合理的结论。

It means arriving at a sound conclusion by linking together some intermediate steps that logically follow from one another.

举个高中课本里的经典例子：苏格拉底是人。

So classic example from high school, Socrates is a man.

所有人都会死。

All men are mortal.

所以，苏格拉底也会死。

Therefore, Socrates is mortal.

这就是一环扣一环的逻辑推理，最后得出一个可靠的结论。

That's a chain of steps that link together logically that arrive at a sound conclusion.

这就是推理。十岁以上的孩子大概都能理解。

That's reasoning Anybody over the age of 10 probably can understand that.

对。

Right.

我们平时思考的时候，其实也是这么做的。

And we, you know, we do that with our thoughts.

是的。

Yeah.

我们用思维来为事物找理由。

We use our thoughts to make reasons for things.

所以这就叫推理。

That's why it's called reasoning.

AI的推理版本有点好笑，因为它比那个更简单，但也更模糊。

The AI version is kind of funny because it's like simpler than that, but also murkier than that.

而且，关于AI推理是什么，取决于你问谁。

And it depends on sort of who you ask as far as what the AI version of reasoning is.

但有些人会说，AI的推理和刚才我说的人类推理本质上是一回事。

But some people will say, the AI version is the same thing literally as what I just said the human reasoning was.

对。

Right.

另一些人会说，推理模型更像是在模仿人类推理的样子，但背后是些奇怪的统计。

Others will say, it's more like the reasoning model is mimicking the appearance of what I described as a human would do with some funky statistics under the hood.

但先抛开这些不谈，AI推理模型实际在做什么，说白了，“AI推理”这个词其实是个术语，或者说是个简称，指的是模型生成一连串合成的自我提示，来引导自己预测出更准确的答案，以应对更复杂或需要分步解答的问题。

But scraping all that aside for a second, the ground truth of what functionally is happening with an AI reasoning model is that word AI reasoning is like really a term of art or like a shorthand for what the model is doing, which is generating a stream of synthetic self prompts to itself that induce it to predict more accurate answers to some more complex or step by step questions.

好的。

Okay.

所以这个想法是，为了得到复杂问题的答案，因为我们的提问方式，它能够把问题分解成中间步骤，然后自己跟自己对话，从而更接近我们想要的复杂答案。

So the idea there being that in order to get to the answer to a complex problem, because of the way that we're asking it, it's able to kind of break that down into intermediate steps and talk to itself in order to get closer to the complex answer that we're looking for.

看起来确实是这样。

That is sure what it looks like it's doing.

好的。

Okay.

你知道吗？

You know?

但是，是的，我觉得所有用过ChatGPT的人都知道这是怎么回事。

But, yeah, like, I would say, everybody who's used ChatGPT knows how this works.

你给它一个提示，它就会回应你，然后说点什么。

You prompt it, it takes a turn, and it says something back to you.

嗯。

Mhmm.

现在几乎所有AI推理模型都会这么做，就是自己生成一些文本，然后把这些文本再喂回它那个小小的文本处理器的顶部。

What the AI reasoning models do, which is pretty much all of them now at this point, it generates some text just by itself that it feeds back into the top of its little text wood chipper.

然后它会自己多跑几轮，琢磨琢磨，最后才给出它预测的那个最终答案。

And then it kinda takes a few turns on its own to kinda crunch around before it lands on that last thing it predicts for you as its answer.

而那些制造AI模型的公司，把这个过程叫做思维链，或者说是“思考”或“推理”。

And the companies who make the AI models, they call that process a chain of thought or quote unquote thinking or reasoning.

对。

Right.

所以我在开头提到过，你从AI还没普及、还没到我们能够毫不羞愧地称之为人工智能的时候就开始报道AI了。

So I mentioned in the beginning that you've been reporting on AI since before it became widely accessible and before it became something that we would, I guess, without shame, call artificial intelligence.

你还记得你第一次遇到“AI是否真的在那样思考”这个问题是什么时候吗？

Do you recall when you first encountered this question of whether AI was doing that level of thinking?

因为我觉得这个问题并不是一直都有意义的。

Because I don't think it always made sense to ask that question.

我记得很清楚。

I pretty clearly remember.

那大概是2024年秋天，差不多九月、十月、十一月的时候，正好是这些推理模型刚开始出现的时候。

It was about the fall, maybe September, October, November 2024, which was right when these reasoning models first started appearing.

第一个模型叫o one。

The very first one was called o one.

你知道，是OpenAI出的。

It came, you know, from OpenAI.

很快大家就开始争论，这到底是不是只是个营销术语？

And pretty quickly, there was debate about, you know, is this just a marketing term?

到底是怎么回事？

What is going on?

咱们直接来试试看。

Let's poke right at it.

用起来看看。

Let's use it.

看看它有什么厉害的地方。

Let's see what's cool about it.

我开始关注这件事，是因为梅兰妮·米切尔，她是圣塔菲研究所一位知名的AI研究员，她还写了一份关于这个领域动态的很好的通讯，她至少有一期专门讨论了她所谓的“AI推理之争”。

And I started paying attention to this when Melanie Mitchell, a well known AI researcher who's at the Santa Fe Institute, who also has this really good newsletter about the goings on in this world, she did at least one issue about what she called the AI reasoning debate.

有趣的是，从一开始，几乎所有人，从记者到研究人员，都习惯性地在“AI推理”这个词上打引号，或者说是加个“所谓的”。

What's funny is like, right out of the gate, almost everybody from journalists to researchers, the default was to kind of like put air quotes or scare quotes around AI reasoning.

对。

Yeah.

因为这是新东西，第一个模型来自OpenAI。

Because it was new, the first one came out of OpenAI.

它跟之前的普通大语言模型不一样。

It was just like, this thing is different than the ones that were before, like just regular LLMs.

这就像在阿司匹林或洗衣液瓶子上加上“加强版”这样的字眼。

It was kind of like slapping the words like extra strength on a bottle of aspirin or a detergent.

所以你就想给它加上引号。

So you wanted to kind of put air quotes around it.

加引号就能让你回避一个问题：这到底是人类定义的推理，还是模型内部发生的某种模拟，或者完全是另一种东西。

The air quotes around it let you duck the question of whether that was the human definition of reasoning, what was happening under the hood, or whether it was a simulation of that, or whether it was something else entirely.

对。

Yeah.

这个说法太到位了。

That's an excellent way to put it.

你把那个o1模型描述成第一个，或者首批推理模型之一。

You described that o one model as the first, or one of the first, reasoning models.

为了让我们明确区别，我们经常提到LLM，也就是大型语言模型。

And just so we're clear on the distinction, we refer often to LMs, large language models.

你现在说的是LRM，大型推理模型。

You're talking about an LRM, a large reasoning model.

那具体到实际应用中，我们怎么区分这两者呢？

Just when the rubber hits the road, how are we distinguishing those two things?

对。

Yeah.

就像我说的，LRM和LLM在功能上的根本区别，归根结底在于所谓的“思维链”行为，这是大型推理模型训练出来的。

The ground truth, like I said, about what just kind of functionally distinguishes an LRM from an LLM, all boils down to this so called chain of thought behavior, which is trained into the large reasoning models.

它们被训练来生成这些自我提示，也就是在它停下来回复你之前，自己进行的那些步骤。

They're trained to generate these self prompts, those turns it makes by itself before it kind of stops and gets back to you.

这些被称为思维链。

And those are called chains of thought.

更中性的技术术语是中间令牌。

The more neutral kind of technical term for them is intermediate tokens.

大型推理模型在给出答案之前，会生成一堆中间令牌。

And an LRM makes a bunch of intermediate tokens before it stops with its answer.

起初，大型语言模型并不明确这样做，但现在它们会吗？

At first, LLMs weren't explicitly doing that, but are they doing it now?

比如，在实践中，大型推理模型和大型语言模型之间有没有我们知道的区别？

Like, is there a distinction between LRM, LLM in practice that we know?

我确信在内部是有区别的。

I'm sure there is behind closed doors.

但对我们来说，今天有区别吗？

But, like, is there a distinction for us today?

难道LLM现在不也在这么做吗？

Aren't LLMs doing this anyway or not?

一开始，2022年、2023年的时候，它们完全不是这样的。

In the beginning, twenty twenty two, 2023, they totally were not doing this.

而且整个思维链的概念，我在文章里也说了，可以说是研究人员既发现又设计出来的。

And the whole concept of, like, a a chain of thought was, you know, I say in the piece, it was, like, part discovered, part devised, you know, by researchers.

他们算是偶然发现了这个现象：如果你在给LLM的提示里手动提供一个逐步推理的例子，

They sort of stumbled upon this fact that, like, if you manually, in your prompt to an LLM, here's an example of a step by step chain of reasoning.

对。

Right.

然后让它照着做，或者你甚至可以做一些非常有名的事情。

Do likewise, you know, or you could even do something this was very famous.

你甚至可以在提示里直接说“一步一步思考”。

You could even do something in the prompt, just say, think step by step.

那就算是一种手动提供这种类似中间令牌行为的方式，而且看起来还挺管用的。

That would be a sort of manual way of providing this intermediate token esque kind of behavior that seemed to work.

社区里的人看到了，然后他们把它自动化集成到了模型里。

People in the community saw it, and then they automated it in to to the models.

那些就是大型推理模型。

And those are large reasoning models.

真有意思，你可以告诉它“努力一点”或者“这很重要”或者“慢慢来”，这些在推理模型范式之外，实际上往往能提高你得到的结果的质量，这真是让人摸不着头脑。

It's so funny that you can tell it things like try really hard or this is important or take your time, and that will actually outside of the reasoning model paradigm, that will actually improve often the quality of the result you get from it, which is just kind of baffling.

你之前提到，当这种推理刚开始被明确地构建到一些模型里时，人们给“推理”加上了引号。

Now you said at the beginning when this reasoning was first starting to get explicitly built into some of the models that people were putting the square quotes around reasoning.

对。

Mhmm.

对。

Mhmm.

在过去几年里，你观察到这场讨论有什么变化？这是否与模型的具体变化或我们对模型的期望有关？如果这个问题合适的话。

What have you observed change in the last couple of years about the way this conversation is proceeding, and does it track with a specific change in the models or what we expect out of the models, if that's a relevant question?

这个问题完全合适。

That's totally relevant question.

我先从你问的那个问题开始，就是什么变了。

I'll start with the where you started, which was what changed.

你知道吗，我这两年来一直想深入写写大型推理模型，基本上从它们刚出现的时候就想写了，当时我第一次看到梅兰妮·米切尔对它们有点不以为然。

You know, I've been wanting to write in-depth about large reasoning models for a couple years, basically since they first appeared, you know, when I first saw Melanie Mitchell giving a bit of side eye toward them.

但后来因为一些无聊的原因没写，不过，我过去两年看到的变化是，首先，那些所谓的引号完全被磨掉了。

And so I didn't for blah blah boring reasons, but, like, what I've saw evolve over the past two years is first first, those air quotes just fully got sanded off.

你知道吗？

You know?

我记不清那具体是多久发生的，但感觉挺快的。

I I can't I can't remember how quickly that happened, but it was it felt relatively quickly.

对。

You know.

我觉得这在一定程度上是因为方便，或者说懒得较真了，你懂的。

And I I imagine that was partly out of convenience slash exhaustion, you know.

只有

There's only

或者就是大家一致同意，当我们说推理的时候，其实不太确定具体指什么。

Or just universal agreement that we're not quite sure when we say reasoning.

我们就直接这么说吧。

We're just gonna say it.

我们不会去

We're not gonna

纠结，我们就直接说。

worry We're just about say it.

你总不能一直给这个词加引号，反复加来加去，听起来像个学究，对吧？

There's only so many times you can bracket the term that way over and over again without sounding like a pedant, you know?

但同时也越来越清楚，它们这种类似加强版阿司匹林的作用，在很多方面确实是真的，你知道吗？

But also, it became clearer that this sort of, like, extra strength aspirin aspect of them was real in many ways, you know?

它确实有效。

It works.

这是米切尔在我为这篇文章采访她时，最先告诉我的事情之一——它确实有帮助。

That was one of the first things that Mitchell told me when I interviewed her for this piece is it helps.

所以人们看到了这一点。

So people saw that.

这也是一个变化。

That was one change too.

在过去的六到十二个月里，我注意到这些模型的声誉又上了一个台阶。

In the past six to twelve months, I noticed another step change in the reputation of these models.

就像突然之间，大家觉得，好吧。

There's, like, this sudden jump to kind of like, okay.

这些现在不只是被宣传为加强版之类的，对。

These are now not just being marketed as extra strength or whatever Mhmm.

而且被当作正经的研究工具来接受，就像粒子对撞机或电子显微镜一样，是那种真正的、高端的工具。

But also being embraced as kind of legitimate research grade tools as sophisticated and quote, unquote, like real deal as like a particle collider or an electron microscope.

在我看来，好像突然之间，我们不再有疑问了。

It appeared to me like all of a sudden we're like, there's no question anymore.

没有人会质疑这些是正经工具了。

Nobody would bat an eye that these are considered legitimate tools.

顺便说一句，这最后六个月真是疯狂。

This last six months, by the way, has been crazy.

我刚从国际数学家大会回来，那是每四年举办一次的会议。

I just came back from the International Congress of Mathematicians, which is a meeting every four years.

所以他们已经有一段时间没像这样聚在一起讨论了。

So it's been a while since they've talked together like this.

现在这成了大家关注的焦点，因为它确实在产出前沿水平的数学成果，这些成果甚至可能发表，而我觉得六个月前数学家们根本没想到会这样。

And this is on everybody's mind because it's legitimately now producing results that are frontier level mathematics results that are potentially publishable, which I don't even think six months ago mathematicians really saw coming.

而且我觉得这更印证了你说的观点，就是不管我们从推理角度要求它做什么，现在似乎都在以很高的水平实现，不管它是不是真的在推理。

And I think that's more evidence of what you're saying, which is this idea that, like, whatever we're asking of it from a reasoning perspective is now, whether it's actually reasoning or not, seeming to happen at a very high level.

是的。

Yes.

但凡事都有两面性，这也是为什么我有点想，天哪，我得写点什么来谈谈这个。

But there's like a yin to that yang, which is kind of the reason why I was like, oh god, I gotta write about this some way

现在。

now.

好的。

Okay.

就是说，是的，这种巨大的进步，像陶哲轩这样的超级数学家都说，我在用这个。

Which is like, yes, this this step change and kind of like people like Terence Tau, the mega mathematician are like, I use this.

你知道吗？

You know?

我用它做了工作，还在国际数学奥林匹克竞赛中拿了金牌，还有那些新颖的证明。

I did work with it and it winning the gold medals in the International Mathematical Olympiad and the novel proofs.

就在这一切开始发生并且确实发生的同时，背后却伴随着对这些模型底层机制的严肃技术批评。

At the same time that that was all starting to happen and then definitely happening, it was happening against this backdrop of, like, serious technical critiques of the underlying mechanisms of these models.

而且不是那种零星的、一两个的批评，而是像稳定的脉搏一样，不断有这种负面结果出来。

And not just kinda like, ping, bang, you know, a handful, one or two, was like a steady pulse, you know, of these, like, negative results coming out.

最出名的一个，去年夏天真的火了，就是苹果机器学习团队那篇叫《思考的幻觉》的论文。

The most famous one that actually went viral last summer was this paper called The Illusion of Thinking from Apple's machine learning team.

那篇论文非常尖锐。

That was very spicy.

你看，标题里就写着呢。

I mean, it's right there in the title.

我写这篇文章的初衷，就是那种让人抓狂的矛盾感——一方面有些事在发生，另一方面又有别的事在发生，两者似乎互相否定。

What made me want to get into something here in this essay was, like, this frustrating schizophrenia about things are happening, but also other things are happening that seem to negate each other.

你明白吗？

You know?

就像我在文章里说的，我当时既生气又好奇。

I was, as I say in the piece, kind of pissed off and curious Right.

搞不清到底哪个方向才是对的。

About like which way is up.

是的。

Yeah.

你知道，怎么能把这些东西调和起来，而不是被迫选边站，比如我就只看我喜欢的那部分，忽略我不喜欢的那部分。

You know, how is it possible to reconcile these things and not just be forced to take a side, be like, I'll just choose to pay attention to that part that I like and ignore the part that I don't like.

你现在是说，无论是模型提供方的专家，还是整个社区和学术界的专家，都在试图解析中间令牌的影响，以及思维链到底在执行什么。

Now you're saying that experts both within the the model providers and from the the general community and from the academic community are trying to unpack the impact of the intermediate token, what a chain of thought is actually executing.

这些就是过去一两年里测试过的内容，他们发现了什么关于这个运作方式的结果？

Like, these are the things that have been tested over the last year a couple of years, year or two, and what are they finding about how this is working?

因为说实话，如果这个模型运作得很好，而且它所说的思维链确实在遵循并产生它试图达到的复杂答案，那么这次对话就会有点不同了。

Because to be honest, if it was working great, and the chain of thought that is in the model that it's saying it's doing is actually following and resulting in the complex answer that it's trying to achieve, then this conversation would be a little bit different.

对吧？

Right?

因为那并不是他们实际看到的情况。

Because that's not necessarily what they're seeing.

对吧？

Correct?

对。

Yeah.

在2025年底，不幸的是，我没能联系到做这项研究的人，因为，唉，他们现在在做AI创业公司，没回我的电话。

There was at the end of twenty twenty five and, unfortunately, I couldn't talk to the people who produced this research because big sigh they're doing an AI startup now, and they didn't return my calls.

但那是一篇综述文章。

But it was a a survey article.

那篇文章的标题大概是“大型推理模型的失败状态”之类的。

It had a title like failure states in larger reasoning models.

我看了看，心里想，天哪。

And I I looked at it, and I was like, holy cow.

里面列了得有几百个。

There's, like, hundreds.

都发在GitHub上了。

They're posted on GitHub.

而且，这就像是一个持续的脉搏。

Again, it's like a steady pulse.

并不是说只有少数边缘人在关注，试图把那些大模型拉下马。

It's not kind of like, oh, there's some fringe outliers paying attention and trying to knock down the giants here.

这些系统存在一些脆弱性，我们根本不知道问题出在哪里。

There's brittleness to these systems that we don't understand where it's coming from.

它们出错或崩溃的方式，跟人类推理时出错的方式完全不一样。

Like, the way it breaks or falls down isn't the way it would if it was reasoning like us.

这引出了一个很有意思的点，就是这些材料有多保密。

That brings up a really interesting point about how secretive some of this material is.

这可能是最让人沮丧的地方之一，因为涉及太多我们不知道的事情，因为这是专有的，而且涉及大量资金。

This is maybe one of the frustrating things about it is that there's so much we don't know about what's happening because it's proprietary, and there's lots and lots of money involved.

所以，对于每一个我们看到或讨论的好的结果，大概背后都有一堆失败的情况，那些时候并没有那样成功，但我们永远不会知道。

So for every great outcome we see or or that we see discussed, presumably, there is a swamp of failures of times that didn't work like that way, but we're never gonna know.

对吧？

Right?

对。

Yeah.

除非，你知道，前沿实验室有巨大的财务激励去完全公开他们的秘密，但咱们还是别抱太大希望了。

Unless, you know, there becomes some massive financial incentive for Frontier Labs to sort of throw their kimonos open, which let's not hold our breath.

不过，我可以具体说说这些批评都是些什么。

But, you know, I I could get into some specifics about what some of these critiques are.

它们主要分两类。

They fall into two lanes.

一个叫忠实性，另一个叫因果性，或者说因果作用。

One's called faithfulness, and one is called like causality, or like a causal role.

忠实性指的是，中间的那些标记是否准确地反映了产生答案的过程？

Faithfulness means are the intermediate tokens accurately representing the process that produced the answer?

就是说，你看那些文字，那些记录，比如思维链的内容，你会想，这算不算是一种报告，或者内心独白，反映了计算上是怎么得出这个正确答案的？

Like, you look at the words, the receipts, you know, of like the chain of thought, and you're like, is that like a report, an inner monologue of what computationally produced this correct answer?

这就有点微妙了，因为他们发现，中间标记，或者说思维链的推理痕迹，可能是不忠实的，也就是说，它们和背后实际发生的事情是脱节的。

This gets slippery because what they discovered is that intermediate tokens slash chains of thought reasoning traces can be unfaithful, which means they're pulled apart from what was happening under the hood.

有一项研究训练了一个推理模型去完成一个正式的任务。

One study trained a reasoning model to do a formal task.

然后他们做的，因为这是一个开源模型，他们可以直接打开它的内部，他们把所有的推理痕迹都替换成了错误的版本，就好像把原本应该产生答案的内心独白改写成了错误的内容。

And then what they did, because this was an open model and they could just pop the hood on it, they replaced all of its reasoning traces with incorrect versions as if they rewrote the kind of like inner monologue that was supposed to produce the answer with wrong stuff.

对。

Right.

就像苏格拉底是一条狗。

Like Socrates is a dog.

所有人都会死。

All men are mortal.

所以，苏格拉底也会死。

Therefore, Socrates is mortal.

这好像不太对劲。

That doesn't quite add up.

这个比喻太妙了。

That is an awesome way to put it.

对。

Yes.

就是这样。

That.

这确实很奇怪。

It was very strange.

比如，你那样做，模型仍然能得出正确的结果，却声称它通过这条明显错误的思维链得到了答案。

Like, you do that, the model could still produce the end result that was correct while claiming that the way that it got there was this clearly incorrect chain of thought.

哇。

Wow.

好的。

Okay.

所以这就不忠实了。

So that's unfaithful.

而且这篇论文里还有相反的情况。

And it also works the other way in this paper.

即使你只给它正确的推理过程，它仍然可能给出正确答案，但在回答过程中还会产生另一条错误的思维链。

Even if you give it only reasoning traces that are right, it can still produce a right answer and also produce another wrong chain of thought on the way to answer.

所以，所有这些都让“默认忠实”这个观点受到了质疑。

So, like, all of this really calls the whole it's faithful by default take into question.

对。

Right.

所以这就是关于忠实性的部分。

So that's the faithfulness part.

另一个奇怪的部分是因果部分，这部分在文章里，我就想，好吧，行吧，你知道，它就是个机器。

The other part, which is also strange, is the causal part, which is kind of in the piece, I'm like, okay, well, fine, you know, it is a machine.

谁在乎这些思维链的语言内容是否可疑呢。

Who cares maybe if like the linguistic content of these chains of thought is like dubious or not.

那些标记还是存在的，你知道，在机器里驱动它做事。

Those tokens are still there, you know, like in the machine making it do things.

也许这就像往弹球机里塞硬币，硬币让弹球机亮起来并运作，这种因果关系是一样的。

Maybe that is just happening in some way that is causal in the same way that pushing a quarter into a pinball machine is causal to the pinball machine lighting up and doing something.

对。

Right.

也许没那么简单。

Not maybe so much.

这项研究发现，在一系列前沿开放模型中，包括DeepSeq r one，他们发现大约30%到60%的中间标记是所谓的装饰性标记，基本上就是假的，与输出没有因果关系。

This one study found that on a collection of frontier open models, one of which was DeepSeq r one, they discovered that, like, 30 to 60% of these intermediate tokens were what they called decorative, which basically means fake, not causally related to the output.

你知道吗？

You know?

就是说，你可以把它们去掉，它还是会输出同样的结果。

Like, you could take them out and it would still produce the same output.

对。

Yes.

而他们正是这么做的。

And that's exactly what they did.

他们有一次去掉了差不多一半，模型的性能也没怎么波动。

They removed, like, half of them at one point, and the model's performance didn't really flutter that much.

所以首席研究员就说，这让我们不得不质疑，那些思维链到底在多大程度上真正决定了模型的行为。

So the lead researcher was like, that calls into question, you know, how much we should interpret those chains of thought as being causal to what the model actually did.

哦，然后最绝的是这个，这个稍微老一点，但就是那种让你歪着头想‘等等，什么？’的感觉。

Oh, and then the best one, this one's a little older, but it's just it snaps the kind of like like you cock your head and be like, wait, what?

你知道，这一切中有一个方面，就是2024年那篇叫《让我们一点一点地思考》的论文。

You know, aspect of all of this is this paper from 2024 called Let's Think Dot by Dot.

那几乎就像是一个核选项。

It was almost like a nuclear option.

他们把所谓的思维链标记换成了无意义的填充标记。

They replaced the chains of thought tokens with meaningless filler tokens, they called them.

那些标记就是点。

They were dots.

就像省略号，点点点。

They were like ellipses, dot dot dot.

但模型依然能完成推理任务。

And they could still get the reasoning model to do its thing.

所以基本上，思维链确实在起作用，但它表面上在做的事情，其实并不明显。

So it's basically like the chain of thought is doing something, but like what it seems to obviously be doing is just not obvious.

如果你能把一串所谓的思考过程，换成相当于反复说‘废话废话废话’的东西，而模型依然能正常工作，那这里面肯定有问题。

If you can take a a ticker tape of so called thoughts and just replace them with the equivalent of going blah blah blah blah over and over again and it still works, then something funny is going on.

在很多情况下，它们能正确完成复杂的多步骤过程。

In a lot of cases, they're producing the correct answer to a complex multistage process.

如果你让一个人解释他是怎么得出那个结论的，理想情况下，你会得到一个忠实的思维链。

If you were to ask a person to reason out how they got to that outcome, you would get, ideally, a a faithful chain of thought.

如果你从中去掉一个逻辑步骤，它就会崩溃。

And if you took a logical step out of that, it would break down.

没错。

Correct.

但他们确实能产出看起来非常像是经过推理的东西。

But they're able to produce this thing that really, really looks like reasoning went into it.

但每次你去仔细检查那些推理痕迹，那些看似能解释他们如何得出结论的痕迹，它们其实不一定和结果有任何关系。这就引出一个问题：如果思维链并不是他们得出答案的方式，那他们到底在做什么？

But every time you poke and prod at the reasoning traces that would suggest that's how they got there, they don't necessarily need to be related at all, which raises the question, what the heck are they doing actually if the chain of thought is not how they're arriving at that outcome?

是的。

Yes.

而且烦人的是，情况并不像你刚才说的那么明确。

And annoyingly, it's not even as clear cut as you just said.

有足够的实证证据表明，存在一些奇怪的失败状态，这些状态似乎质疑了所谓的思维链心智模型。嗯。

There's enough just enough empirical evidence to show these bizarre sort of like failure states that kind of impugn the supposed chain of thought mental model Mhmm.

你知道，值得认真对待。

To be, you know, worth taking seriously.

但因为所有模型都不一样，开源模型和闭源模型的表现不同，闭源模型你没法像开源模型那样去检查。

But because all the models are different, open models behave differently than closed models, closed models, you can't inspect, you know, in the same way that the the open models are.

而且闭源模型是更强大的那些。

And they're the more powerful ones, the closed models.

没错。

Exactly.

它们也是最强大、最有能力、最好的。

They're also the most powerful, capable, best.

对。

Right.

这就悬在一个模糊的灰色地带。

It is hanging in this kind of cloudy gray area.

我们有没有关于到底发生了什么的理论？

Do we have theories about what is happening?

有没有其他方式来理解模型内部的情况，还是说我们就像在很多其他情况下一样，只能试着去解读模型内部到底发生了什么？

Are there other ways to think about what's happening in the models, or are we just like we are in so many instances when it comes to trying to interpret what's going on inside a model?

我们是不是就束手无策了？

Are we just at a loss?

是的。

Yeah.

所以，有一位研究员，我是说，有很多研究员对这个问题感兴趣，并且在做这种实证探测。

So there is one researcher I mean, I said there's many researchers who are interested in this and doing this kind of empirical probing.

有一位我深入聊过，因为他过去几年一直在坚持不懈地研究这个。

There's one I talked to in-depth because he has been studying this kind of relentlessly for the past few years.

他叫苏巴劳·坎巴帕蒂，来自亚利桑那州立大学。

His name is Subarau Kambapati from Arizona State University.

他曾是人工智能促进协会的前任主席。

He's a former president of the Association for the Advancement of Artificial Intelligence.

他有AI规划的背景。

He's got a background in AI planning.

他非常关心一个推理模型是否真的在推理。

He really cares about knowing whether or not a reasoning model is reasoning.

好的。

Okay.

他给了我目前最接近的一个替代性工作假设，这个假设的出发点是，你知道，问题出在训练上。

And he has the closest thing that I turned up for a working hypothesis for an alternative here, and that alternative starts from this point of, you know, it's the training stupid.

大型推理模型其实就是LLM的加强版。

Like, large reasoning models are just extra strength versions of LLMs.

我们知道LLM是怎么训练的。

We know how LLMs are trained.

它们被训练成预测下一个词，大型推理模型也一样。

They're trained to be next token predictors, even the large reasoning models are.

归根结底就是这样。

That's what it boils down to.

所以如果从这个角度出发，他的工作假设就是：这既不是数据库，也不是任何意义上的确定性计算器。

And so if you start there, his kind of working hypothesis is like, this is not a database, this is not a deterministic calculator in any kind of way.

它基本上就是一个类似模糊记忆的系统。

It is basically a fuzzy memory esque system.

它是在做预测。

It makes predictions.

所以，大型推理模型因为有了这些思维链的强化训练，能做到的事情，就像他说的那些特别生动的比喻一样，它们就像是在自言自语。

And so what a large reasoning model can do because of its extra strength training with these chains of thought is they are akin to he's got these really spicy ways, these analogies of talking about it.

他管这叫“嘀咕”。

He calls it mumbling.

它就是在干这个。

That's what it's doing.

它不是在想，而是在嘀咕，就像我们有时候想回忆什么，不是去推理，而是脑子里乱转，想把那个记忆给“碰”出来。

It's not thinking, it's mumbling in an analogous way to maybe what you and I might do when we're trying to not reason something out, but just bonk loose something we wanna remember in our minds.

比如，我想不起那个演员的名字。

Like, you know, I'm trying to remember that actor's name.

是叫吉米·杰罗尼莫吗？

Is it, you know, Jiminy Geronimo?

还是杰克·乔金诺克？

Is it Jake Jorkinork?

你知道吗？

You know?

哦，是杰克·吉伦哈尔。

Oh, it's Jake Gyllenhaal.

你知道吗？

You know?

对，就是这样。

Like Yeah.

是的。

Yeah.

是的。

Yeah.

他称之为近似检索。

Like, calls it approximate retrieval.

好的。

Okay.

而思维链是自我提示的一个额外增强，就是那些中间令牌。

And the chain of thought is a extra strength add on to the self prompts, the intermediate tokens.

中间步骤本质上是在缩小或聚焦它在训练数据中寻找的内容，以预测下一个令牌，这样结果会更好。

The intermediate steps are essentially narrowing or focusing what it's looking at in its training data to predict the next token, and that results in a better outcome.

对。

Yeah.

我觉得他可能会对“缩小”或“聚焦”这种说法有意见，因为我问过他，我说，哦，那是不是有点像在搜索树里缩小选项？

I I think he might take issue with, like, narrowing or focusing because I asked him I I was like, oh, so is it kind of like narrowing options in a search tree?

他说，不是。

He's like, no.

不是。

No.

不是。

No.

那不像搜索。

It's not like search.

这就是他提出那个记忆唤起类比的原因。

That's where he came up with this memory jogging analogy.

你知道，对。

You know, Right.

而且他会第一个说这个案例还没有定论。

And he would be the first to say that this case is not closed.

嗯。

Mhmm.

但这可以解释那种，我称之为大型推理模型中那种“胡扯”和“不胡扯”的叠加态，就是说，如果中间标记的含义几乎无关紧要，或者说只是偶然的，就像我说“Jake Geronimo”而不是“Jake Gyllenhaal”一样，那并不是推理。

But it would kind of account for this, I call it like a superposition of the BS ness and the not with large reasoning models, which is like, if it barely matters or it's, let's just say, incidental, the meaning of the intermediate tokens, just the way that me saying like Jake Geronimo before Jake Gyllenhaal, it doesn't it's not reasoning.

那只是我说的一句话。

It's just like a thing I said.

你知道，这种假设可以解释大型推理模型中那些奇怪的行为。

You know, this kind of hypothesis could account for those weird behaviors in a large reasoning model.

这有助于我回忆某些事情，但我不是说我就是个LLM。

It helps if I'm trying to remember something, and I'm not saying I'm an LLM.

这只是一个类比。

This is an analogy.

但是，比如说，如果你想记住某件事，把一些相关的词在脑子里串起来，可能会对你有帮助。

But, like, if you're trying to remember something, it can help to sort of, like, string out words to yourself that are related to what you're trying to get to.

对。

Yeah.

这不是必须的，你知道吗？

It's not required, you know?

所以他说，这是他目前最好的工作假设，用来解释这些大型推理模型内部可能发生的事情，考虑到我们知道它们的训练方式以及它们底层的目标。

So he's saying that is his best working hypothesis for what is more likely to be the case of what is going on inside these large reasoning models given that we know how they are trained and what their sort of objective is under the hood.

这里的想法是，可能发生的过程与人类思维有一些不同的类比，但试图把它归入某个与我们的思维方式相关的类别是错误的，因为它的训练方式与我们通常认为的学习是不同的。

The idea here being that the process that could be taking place has some different kind of analogies to human thought, but that it's a mistake to try and put it into a certain bucket that is related to the way our own minds work because the way that it's trained is something different than what we would think of as as learning, for example.

对。

Yeah.

他喜欢用简称“Rao”，因为说起来更方便，所以我就这么称呼它。

He likes to use the nickname Rao because it's easier to say, so that's how I'll talk about it.

Rao喜欢说的是，他并不否认大型推理模型更准确，或者可以更准确。

What Rao likes to say is he's not denying that, like, large reasoning models are more accurate or can be more accurate.

他并不是说，那些已经出来的证明，比如OpenAI关于单位距离问题的那个，是骗人的。

He's not saying, oh, these proofs that have come out, like the one from OpenAI about the unit distance problem, those are a sham.

它们不是骗人的。

They're not a sham.

它们已经经过验证了。

They've been verified.

他只是说，为了调和这些基本事实，比如它们确实有效，同时它们也会以奇怪的方式失败。

He's just saying like, in order to reconcile the brute facts here of like they work plus also they fail in weird ways.

这些事实需要以某种方式结合起来，而不是简单地挥手忽略其中任何一方。

Those need to be put together in some way that's not just hand waving away, you know, one side or the other.

而这就是他试图做的事情。

And this is his attempt to do that.

这里有一个重要的部分，那就是这个假设也能解释另一个基本事实，即大型推理模型在编码和数学等领域取得这种跨越式进步的地方，这些领域被称为可验证领域。

And there's an important part of this, which is like this hypothesis also would account for, again, the brute fact that like the places or the domains where large reasoning models are making this like step change progress, you know, in coding and mathematics, Those are called verifiable domains.

这意味着，思维链的最终状态，也就是答案或结果，可以被客观地验证其正确与否。

And what that means is the end state of, like, the chain of thought, the answer, the result, can be objectively verified as to its correctness or incorrectness.

这不是模糊的。

It's not fuzzy.

要么对，要么错。

It's either one or the other.

代码能跑就是能跑，不能跑就是不能跑。

Code runs or it doesn't.

证明要么成立，要么不成立。

A proof is sound or it is not.

你可以有一条真正的思维链，最终得出一个正确且可验证的结果。

You can have a real chain of thought that lands in a correct verifiable output.

这些都可以成为大型推理模型的训练素材。

That can all become training information for a large reasoning model.

所以你可以不断加入越来越复杂的训练示例，展示什么样的推理链能得出正确的答案。

And so you can just shove in all of these increasingly more sophisticated training examples of here's what a chain of reasoning looks like that leads to an answer that is right.