菲利普·泰特洛克谈超级预测

欢迎收听《经济对话》，这是经济学与自由图书馆的一部分。

Welcome to Econ Talk, part of the Library of Economics and Liberty.

我是主持人拉斯·罗伯茨，来自斯坦福大学胡佛研究所。

I'm your host, Russ Roberts, of Stanford University's Hoover Institution.

我们的网站是econtalk.org，您可以在此订阅、评论本期播客，并找到与今天对话相关的链接及其他信息。

Our website is econtalk.org, where you can subscribe, comment on this podcast, and find links and other information related to today's conversation.

您还可以浏览我们的档案库，收听自2006年以来的每一期节目。

You'll also find our archives, where you can listen to every episode we've ever done going back to 2006.

我们的电子邮箱是mail@econtalk.org。

Our email address is mail@econtalk.org.

我们期待您的来信。

We'd love to hear from you.

今天是2015年11月30日，我的嘉宾是宾夕法尼亚大学沃顿商学院及文理学院的安纳伯格大学教授菲利普·泰特洛克。

Today is 11/30/2015, and my guest is Philip Tetlock, the Annenberg University professor affiliated with the Wharton School and the School of Arts and Sciences at the University of Pennsylvania.

他与丹·加德纳合著的《预测的艺术与科学》是本期节目的主题。

He is the author, along with Dan Gardner, of The Art and Science of Prediction, which is the subject of today's episode.

菲利普，欢迎来到Econ Talk。

Philip, welcome to Econ Talk.

嗯，谢谢。

Well, thank you.

你在书中一开始就提出了许多批评，或者说贯穿全书，我认为你对那些评论家有很多批评。

So you start with a lot of criticisms or throughout the book, I'd say you have a lot of criticisms of pundits.

他们中有些人拥有博士学位，有些是记者，还有些只是所谓的预测专家。

Some of those have PhDs and some of them are journalists and some are just so called experts who make predictions.

但事实证明，当需要判断预测是否准确时，你很难真正追究他们的责任？

But it turns out a lot of those, you can't really hold their feet to the fire when it comes time to judge whether predictions are accurate or not?

他们是否擅长预测呢？

Are they good forecasters or not?

为什么会这样？

And why is that?

在我们日常生活中，人们声称某事会发生并将其刊登在报纸上，这其中的挑战是什么？

What's the challenge with our sort of day to day world where people claim that something's gonna happen and print it in the newspaper?

嗯，你提到的那些评论家，可能是指像托马斯·弗里德曼或尼尔·弗格森这样的人，左派或右派都有。

Well, the pundits of whom you think whom you say were critical, probably thinking of people like Tom Friedman or Niall Ferguson, people on the left or people on the right.

我们研究了各种类型。

We identify all sorts.

他们都相当一致。

They're all pretty uniform.

他们都是相当一致的非常聪明的人。

They're they're pretty uniformly very smart people.

他们都非常善于表达。

They're they're very articulate.

他们知识非常渊博。

They're very knowledgeable.

他们可能会对世界政治和经济做出许多看似极具洞察力的观察。

They offer may make many observations about world politics and economics that, seem very insightful.

然而，要评估他们对可能未来的判断、选择不同政策路径的后果是否正确极其困难，因为他们几乎完全依赖我们称之为模糊措辞预测的方法。

It is extremely difficult, however, to gauge the degree to which their assessments of, possible futures, the consequences of going down one policy path or another are correct or incorrect because they rely almost exclusively on what we call vague verbiage forecasting.

他们不会说某事发生的概率是20%或80%。

They don't say that they're the 20% likelihood of something happening or an 80% likelihood of something happening.

他们会说类似'2016年全球通缩是一个明显可能性'这样的话。

They say things like, well, it's a distinct possibility that there'll be a global deflation in 2016.

当你问人们'明显可能性'是什么意思时，根据听众当时的心情，这个概率可能在20%到80%之间。

And when you ask people what distinct possibility could mean, it could mean anything from about 20% to 80% probability depending on the mood they're in when they're listening.

我并不是要暗示你在批评他们（虽然你有时确实会），但你是在批评我们这种文化——把模糊的预言当真，然后让对立阵营的人玩'抓把柄'游戏。

And I I didn't mean to suggest you're critical of them, although you sometimes are, but you're you're critical of the of our culture that takes these vague pronouncements and then there's a gotcha game that gets played by people on the other side.

但当然，他们总有办法脱身，因为那些含糊其辞的话里通常都留有退路。

But of course, there's always a way to weasel out of it because there's usually some hedging in that in that verbiage.

对吧？

Correct?

嗯，确实如此。

Well, that that that's right.

如果你处在一个'指责游戏'的文化中——每当你的明确概率判断看起来可能出错时就会被人揪住不放——那么退回到模糊措辞中其实是相当理性的选择。

If if you if you exist in a blame game culture in which people are going to pounce on you whenever you make an explicit probability judgment that that appears to be on the wrong side of maybe, it's pretty rational to retreat into vague verbiage.

我们在书中提到了一位杰出的记者——《纽约时报》的戴维·莱昂哈特，他创办了数据新闻专栏'The Upshot'。

So we talk in the book about a brilliant journalist, the New York Times journalist, David Leonhardt, created the Upshot, a quantitative column in New York Times.

他在2011年还是2012年写过一篇文章，当时最高法院以5比4的微弱优势维持了奥巴马医改法案。

And he wrote a piece back in, I guess, was 2011 or 2012, and the Supreme Court narrowly upheld Obamacare by a five four margin.

而预测市场此前认为该法案被推翻的概率高达75%。

And the prediction markets had been putting a 75% probability on the law being overturned.

据我所知，戴维·莱昂哈特对预测市场并无成见，但他最终认定预测市场这次判断失误。

And David Leonhardt, who doesn't have any grudge against prediction markets as far as I know, concluded that the prediction markets got it wrong.

这对预测市场来说是个严厉的批评，毕竟它们多年来在数百个议题上做出过数百次预测，总体表现并不差。

Now that's that's a harsh judgment on the prediction markets because they make hundreds of predictions on hundreds of different issues over years, and they're not bad.

当它们预测某事件有75%发生概率时，实际发生率确实接近75%——这意味着有25%的情况不会发生。

When they say there's a 75% likelihood of something happening, it's pretty close to a 75 likelihood, which means that 25% of the time it doesn't happen.

如果你每次都要指责一个校准良好的预测系统仅仅因为结果落在'可能'的错误一侧，那你最终将失去所有可靠的预测工具。

So if you're gonna throw it a very well calibrated forecasting system every time it's on the wrong side of maybe, you're not gonna have any well calibrated forecasting systems at your disposal.

我认为这其实暴露了第二个问题：即便你量化了预测概率，本质上仍允许结果存在不确定性。

I would say that's a second problem really, which is that even when you do quantify your prediction, by definition, you're allowing the possibility that it doesn't happen.

那么问题在于，你如何评估做出这种预测之人的准确性或判断力？

And then the question is, how do you assess the accuracy or judgment of the person who makes a statement like that?

是的，正是如此。

Yes, exactly.

这需要理解概率，需要一些耐心和意愿去长期追踪记录。

And that then that requires some understanding of probability and some willing some patience and some willingness to look at track records over time.

那就从你的个人记录开始吧。

So let's begin with your particular track record.

你在这个领域做了大量研究，关于预测是否可能、准确性如何、专家是否擅长预测这些问题。

You've done a lot of research in this area, this question of whether prediction is possible, how accurate is it, are experts good at forecasting.

谈谈你的背景。

Talk about your background.

我们稍后会讨论你书中的核心赛事，但我想先了解你的研究历程、过去的发现以及人们的反应。

We're gonna get to the tournament that's at the heart of your book, but I wanna start with your research history and what what you found in the past and how people reacted to it.

嗯，我想你这是在委婉问我到底有多老吧？

Well, I guess that's another way of asking you just exactly how old must I be?

因为我从事长期预测竞赛研究已经很久了。

Because I've I've been doing longitudinal forecasting tournaments for a long time.

那我们就直截了当地说吧。

So let let's just put on the table.

我今年61岁，这个研究是在我获得加州大学伯克利分校终身教职后不久开始的，那时我刚过30岁。

I'm I'm I'm 61 years old, and I got started at at this after right after I got tenure at the University of California Berkeley, and I I was I was a little little more than 30 years old.

那是1984年。

It was 1984.

当时苏联还存在。

And the Soviet Union still existed.

戈尔巴乔夫还没当上苏联共产党总书记。

Korbetschev had yet to become general secretary of the communist part of the Soviet Union.

我们在八十年代中期进行了初步试点研究，当时鹰派和鸽派正在争论应对苏联的最佳方式。

And we we did our initial pilot studies back in the mid nineteen eighties when people were or hawks and doves were arguing about the best ways of dealing with the Soviet Union.

而现在当我们在伊朗核问题或俄乌冲突问题上争论不休时，我们仍在进行预测竞赛研究。

And now we're doing forecasting tournaments as hawks and doves are arguing about the best ways of dealing with the Iranian nuclear program or for that matter, if we're dealing with Russia and The Ukraine.

所以我们断断续续进行预测竞赛已有三十多年了。

So we've been running forecasting tournaments off and on for thirty plus years.

第一轮大型预测竞赛在八十年代末九十年代初进行，相关成果发表在2005年出版的《专家政治判断》一书中。

The first big set of forecasting tournaments were done in the late eighties and the early nineties and were reported on a book expert political judgment that came out in 2005.

第二轮预测竞赛规模更大，涉及数千名预测者、超百万份预测报告，由美国情报机构赞助。

And the second wave of forecasting tournaments were much larger involving many thousands of forecasters, a million plus forecasts, and were sponsored by the US intelligence community.

这些竞赛从2011年持续到2015年。

And they ran from 2011 to 2015.

事实上，它们现在仍在进行中。

And in fact, they're still running.

如果读者有兴趣报名参加正在进行的预测竞赛，可以考虑访问gjopen.com网站。

So if your readers are interested in signing up for an ongoing forecasting tournament, they should consider visiting the website at gjopen.com.

回到您在苏联解体前做的早期研究，那些研究有哪些主要实证发现？

Going back to the earlier work that you did and before the fall of the Soviet Union, what were some of the main empirical takeaways from that work?

一个重要发现是，自由派和保守派提出的政策建议大相径庭，他们对不同政策路径可能产生的结果也有截然不同的条件性预测。

Well, one big takeaway was that liberals and conservatives had very different policy prescriptions, and they had very different conditional forecasts about what would happen if you went down one policy path or another.

而且没有人真正接近预测戈尔巴乔夫现象。

And that nobody really came close to predicting the Gorbachev phenomenon.

同样，也没有人真正接近预测后来苏联的解体。

Nobody for that matter came really close to predicting the disintegration of the Soviet Union later on.

但事后每个人都似乎有一套解释，要么邀功，要么推诿责任。

But everyone after the fact seemed to have an explanation that I I either appropriated credit or deflected blame.

我相信这与他们的世界观是一致的。

And that was consistent with their worldview, I'm sure.

并且与他们先前的世界观完美契合。

And meshed perfectly with their prior worldview.

所以这就像我们处于一个结果无关紧要的学习情境中。

So so it it was as though we are in an outcome irrelevant learning situation.

实际上发生了什么并不重要。

It didn't really matter what happened.

人们总能处于绝佳位置，将发生的事解释为与他们先前的观点一致。

People would were would would be in an excellent position to interpret what happened as consistent with their prior views.

预测竞赛的理念是让人们更容易记住他们过去无知的状态。

The idea of forecasting tournaments was to make it easier for people to remember their past states of ignorance.

嗯，这算是题外话，但它确实是对人性的绝妙洞察。

Well, this is an aside of sorts, but it's just a wonderful insight into human nature.

这也是Econ Talk节目中的一个主题，当你回顾并让人们给出他们记忆中的概率时，比如苏联解体，他们是怎么说的？

And it's a theme here at Econ Talk, which is when you went back and asked people to give their, what they remember as their probability of say the Soviet Union falling, What what did they say?

嗯，他们肯定认为自己当时给苏联解体的概率比实际要高。

Well, they they they certainly thought they assigned a higher probability to the dissolution of the Soviet Union than they did.

还有少数人当初给出了极低的概率，却记得自己当时预测的概率超过50%。

And there were a few people who assigned really very low probabilities who remember being on assigning higher than a 50% probability.

所以人们事后确实夸大了那些概率。

So people really pumped up those probabilities retrospectively.

心理学家称之为后见之明偏差或'我早就知道'效应，我们在苏联预测竞赛中看到了大量这种现象。

So the psychologist called out the hindsight bias or the I knew it all along effect, and we we saw that in spades in the the Soviet Forecasting Tournament.

是啊。

Yeah.

我认为这是我们所有人都会做的一件极其重要的事。

I I think that's an incredibly important thing that we all tend to do.

我们往往以为自己比实际更有远见。

We tend to think we had much more vision than we actually had.

而且我们通常不会把这些事情写下来。

And we usually don't write those things down.

你碰巧记录下了其中一些。

You happen to have written some of them down.

所以当他们拿出原始预测时确实很尴尬。

So that was awkward that they actually had their original forecast.

但对大多数人来说，'我早就知道'效应是个更严重的问题，因为我们没有记录下来。

But most of us, the I knew it all along problem is a bigger problem for most of us because we don't write it down.

事实上我们的记忆确实会出现偏差。

Well, we truly remember it differently.

即使你认为桌子对面的人知道正确答案，你仍然会记错。

Even if you think the person on the other side of the table knows what the correct answer is, you still tend to misremember it.

是啊。

Yeah.

所以最近这场锦标赛相当引人注目。

So this more recent tournament was rather remarkable.

给我们讲讲参赛者的背景、你在其中的角色、赛事是如何组织的，以及参赛者们竞争回答的一些问题示例。

Give us the background of who competed and your role in it and how it was set up and what some of the questions, for example, were that people were competing on.

好的。

Sure.

这是我和我妻子兼研究搭档芭芭拉·梅勒斯共同完成的工作，当时我们还在加州大学伯克利分校任教，直到2010年左右才转去宾夕法尼亚大学。

This was work I did jointly with my wife, research collaborator, Barb Mellors, we were faculty then at the University of California, Berkeley, and we didn't leave for University of Pennsylvania until about 2010.

2009年底我们在伯克利时，国家情报总监办公室的三位工作人员拜访了我们。

But we were visited by three people from the Office of the Director of National Intelligence when we were at Berkeley, I guess, late in 2009.

其中至少有两人对将我的早期研究中运用的技术——即用超政治判断来评估情报分析师判断准确性——应用于美国情报界这个想法非常热衷。

And at least two of them were quite enthusiastic about the idea of The US intelligence community using some of the techniques that were employed in my earlier work, extra political judgment for keeping score on the accuracy of intelligence analyst judgments.

这就是后来被称为IARPA预测锦标赛的核心构想。

And that was the core idea behind the what became known as the IARPA forecasting tournaments.

IARPA是国家情报总监办公室下属的研发部门，作为统筹机构管辖着包括中情局、国防情报局等所有情报机构。

IARPA is the research and development branch of the office of the director of national intelligence, which is the umbrella organization over all intelligence agencies like CIA and DIA and and so forth.

总共涵盖16个情报机构。

And all all 16 of them.

其核心理念是举办一场竞赛，让顶尖高校和咨询公司竞标大型合同，组建团队专门对美国情报界认定的国家安全相关未来事件做出最精准的概率预测。

And the the the idea would be would be they they would have a competition and major universities and consulting operations would would apply for large contracts to assemble teams whose purpose would be to assign the most realistic probability estimates to possible futures that The US community US intelligence community deemed to be of national security relevance.

这些预测最终会转化为具体问题。

So those those turned out to be questioned.

问题涵盖领域极其广泛，从中日东海冲突、希腊退出欧元区、西班牙国债收益率差，到俄罗斯与爱沙尼亚、乌克兰、格鲁吉亚的关系。

I know everything from Sino Japanese clashes in the East China Sea to Greece leaving the Eurozone and Spanish bond yield spreads to Russian relations with Estonia, Ukraine, Georgia.

当然还包括中东冲突、埃博拉疫情、H5N1禽流感等各类重大议题。

Of course, conflicts in The Middle East, Ebola, h five n one issues, just enormous range of issues.

四年间累计发布了500多个预测问题。

A 500 plus questions over about four years.

每个研究团队的目标都是开发出最优的概率预测方法论。

And the goal would be of each of the research operations would would be to come up with the best possible ways of assigning probability estimates.

他们当时对每个人的学术背景都进行了严格筛查，确保所有人都是正规学者。

Now they they they screened everybody for their academic bona fides, so they wanna make sure that everybody was legit.

他们可没用通灵板之类的东西。

They weren't using Ouija boards or anything like that.

但除此之外，美国情报界只关心谁能对这些极其多样化的问题做出最精确的概率预测。

But beyond that, they they now the US intelligence community was simply interested in who could generate the most accurate probability estimates for these extremely diverse questions.

他们并不在乎我们采用心理学方法、统计学方法还是综合方法。

And they didn't really care whether we took a more psychological approach or more statistical approach or composite approach.

他们唯一在乎的就是准确性，仅此而已。

What they what they cared about was accuracy, and that was it.

准确、准确、再准确。

Accuracy, accuracy, accuracy.

于是我们——我妻子芭芭和我组建了这个名为'优质判断小组'的团队，这是个汇集了杰出学者的跨学科联盟。

So we our our group, Barb Barb my my wife and I put together this group called the good judgment group, which is an interdisciplinary consortium of of of wonderful scholars.

我们着手招募优秀预测者，并努力为他们提供最佳的概率推理培训与原则指导。

And we went out about we we we tried to recruit good forecasters who and we tried to give them the best possible training and principles of good probabilistic reasoning.

我们将其中一些人组建成团队，并指导他们如何高效协作。

And we assembled some of them into teams, and we gave them guidance on how teams can work effectively together.

我们还让部分人参与预测市场，想看看预测市场的效果如何。

And we put some of them into prediction markets, and we just wanted to see how well prediction markets would work.

我们尝试了许多不同的方法进行实验。

We did we experimented with a lot of different approaches.

我们还有非常优秀的统计学家，他们尝试了各种从群体中提炼智慧的方法。

We also have really good statisticians who experimented with different ways of distilling wisdom from crowds.

因此我们的方法极具实验性。

So our approach was very experimental.

我认为其他一些方法同样具有实验性质。

I think some of the other approaches were experimental as well.

但我们的实验成果比他们的更为出色。

But our experiments worked out better than their experiments.

因此在前两年的比赛中，我们以相当显著的优势获胜，优势如此明显以至于美国情报界决定将剩余资金集中投入一个大型项目——即'优质判断计划'，该项目可以吸纳其他团队中最优秀的研究人员。

So we won the tournament by pretty resounding margins in the first two years, sufficiently resounding that US intelligence community decided to funnel the remaining money into one big group, which would be the good judgment project, which could hire some of the best researchers from other teams.

谁谁

Who who

你在和谁竞争？

are you competing against?

最初我们是在这里与不同的竞争基准进行较量。

Well, we originally we were competing different different competitive benchmarks here.

最初，我们的竞争对手是其他获得政府合同的机构，比如麻省理工学院、密歇根大学、乔治梅森大学等。

Originally, we were competing against the other institutions that received contracts from the government like oh gosh, MIT and University of Michigan and George Mason University, places like that.

后来我们还与自己运营的一个名为Inkling的公司运行的预测市场竞争，同时也与美国情报分析师自己生成的概率估计进行内部基准比较，尽管这部分是机密的，因为美国情报分析师的身份本身就是机密。

Then later and we were competing against a prediction market that we ourselves were running by a firm known as Inkling and also against internal benchmarks, US intelligence analysts themselves generating probability estimates and competing against them, although that was classified because, of course, The US intelligence analysts were classified.

但《华盛顿邮报》的大卫·伊格内修斯在第二年末或第三年时泄露了部分信息。

But David Ignatius at the Washington Post leaked some of that information in I think at the end of the second year or third year.

但两年后，你的团队彻底击败了所有人。

But after two years, you your team trounced everybody.

之后又发生了什么？

And then what happened going forward after that?

嗯，我们得以吸收其他团队的资源，因为政府显然通过暂停对其他团队的资助节省了大量资金。

Well, we were able to absorb resources from the other teams because the government was obviously saving a lot of money by suspending the funding of the other teams.

因此我们能够整合部分资源，从而更有力地与剩余的其他基准线竞争。

So we were able to consolidate some resources, and we were able to compete all the more aggressively against the the other the remaining benchmarks.

对我们而言，关键的外部基准是由Inkling运营的预测市场，以及美国政府内部更机密的那个。

Key benchmarks for us to be were an external benchmark, the prediction market run by Inkling and the more confidential one inside the US government.

你刚才提到——其实我要引用书里一句很应景的话，我特别喜欢，是来自早期医师盖伦的。

Now you mentioned, and this is just Well, actually I'm gonna read a quote from the book, I loved, which is relevant, which is from Galen, the early physician.

盖伦大致生活在什么年代？

And what time period did Galen live roughly?

他是公元二世纪的人物。

He is a second century after Christ.

大约两千年前左右。

It was about roughly 2,000 ago.

好吧，我以为他生活的年代要更晚些。

Okay, I thought he was later than that.

他在很久以前就写下了这些，你引用了以下内容说他并不热衷于实验，你写道：盖伦从不为结果所困扰，这些结果证实他是对的，无论证据在不如大师智慧的人看来多么模棱两可。

So he wrote a long time ago and you write the following that he wasn't into experiments and you wrote the following that he here's the quote: Galen was untroubled by outcome confirmed he was right, no matter how equivocal the evidence might look to someone less wise than the master.

这是盖伦的原话：所有服用此疗法的人都会在短时间内康复，除了那些未被治愈的人，他们都死了。

Here's Galen's Galen quote: All who drink of this treatment recover in a short time, except those whom it does not help who all die.

因此很明显，它只在无法治愈的病例中失效。

It is obvious therefore that it fails only in incurable cases.

还有什么比这更好的呢？

So what could be better than that?

我是说，这太了不起了。

I mean, that's phenomenal.

这让我想起，我想这就是你引用那句话的用意，即便是专家给某个事件发生的概率赋值为63.7%，无论最终是否发生。

I was reminded, I think that's where you apply the quote, of even the pundit who puts a numerical value on a certain event happening is a sixty three point seven percent chance that this will happen whether it happens or not.

如果确实发生了，他就会说：看吧，我早说过有63.7%的概率。

If it does happen, he says, See, I told you it was 63.7.

如果没有发生，他又可以说：我早说过有36.3%的概率不会发生。

And if it doesn't happen, he can say, Well, I said there was a 36.3% chance that it wouldn't happen.

所以当它没发生时，我依然是对的。

And so when it didn't happen, I'm still right.

那么问题就变成了，当你说你碾压了其他团队时，必须有一种评估概率的方法。

So the question then becomes, when you say you trounced the other teams, there has to be a way to evaluate probabilities.

在书中你提出了Briar评分。

And in the book you present the Briar score.

试着给我们讲讲你是如何衡量预测成功的。

So try to give us the flavor of how you measured success in prediction.

哦，这个观点很棒。

Oh, that's an excellent point.

确实无法衡量单个事件的概率判断准确性，除非预测者鲁莽地将概率设为零而事件发生了，或者设为1.0而事件没发生。

It really isn't possible to measure the accuracy of a probability judgment of an individual event Unless the person, the forecaster is rash enough to assign this probability of zero and it happens, or a probability of one point zero and it doesn't happen.

否则预测者总能辩称发生了小概率事件。

Otherwise, the forecaster can always argue that something improbable happened.

所以评估单个事件的准确性是不可能的，除了那些极端情况。

So assessing the accuracy of individual events is impossible, except in those limiting cases.

但可以通过评估多个事件和多个时间段的准确性来实现。

But it is possible to assess the accuracy across many events and many time periods.

因此，在世界政治中，良好的判断意味着你比其他人更擅长在众多事件和多个时间段内，为实际发生的事件分配比未发生事件更高的概率。

So good judgment in world politics means you're better than other people at assigning higher probabilities to things that happen than the things that don't happen across many events, many time periods.

举个例子来说，我们来讨论一下。

So the example would be, let's talk about it.

我们举一个具体的例子。

Let's take a particular example.

我们将尝试预测希腊退出欧元区的概率。

We're gonna try to forecast the probability of Grace leaving the Eurozone.

我说概率是0.51，而你说——抱歉，你说的是0.15。

So I say it's 0.51 and you say, excuse me, say it's point one five.

不，我们再来一次。

No, let's go again.

我说的是0.15。

I did point one five.

好的，请继续。

Yeah, okay, go ahead.

我选0.49，因为我认为可能性不大。

I'll go point four nine because I think it's not likely.

选择这个数字是因为它低于0.5。

Going because it's below point five.

我说0.49而你说0.1，结果这事没发生。

I say point four nine and you say point one and it doesn't happen.

所以结论是你判断得比我更准确。

So the argument is that you did a better job than I did.

你并不能确定这一点

You don't know that for sure

不，你不能。

No, you don't.

关于英国脱欧这件事。

With respect to Brexit.

没错。

That's correct.

你确实通过IARPA竞赛中提出的全部问题，从概率角度了解这一点。

You do know it probabilistically across the full range of questions posed in the IARPA tournament.

就你多年来一直预测0.49，而我预测0.1却未发生的情况而言，你可能会忍不住得出结论——即便关于希腊退欧问题，我的预测也更接近事实。

Now insofar as you've been predicting point four nine consistently over several years, and I've been predicting point one and it doesn't happen, you might be tempted to draw the conclusion even with respect to Grexit that I've been closer to the truth.

你可能会这么想。

You might be.

所以让我感到困扰的是这种评估良好判断力的设定方式。

So one of the things I found troubling about the setup and the way of assessing good judgment.

你的著作让人深思的一点是：评估某人是否具备良好判断力竟如此困难。

One of the things your book makes one ponder is just how hard it is to assess whether someone has good judgment.

这完全正确。

That's absolutely true.

我完全赞同。

I couldn't agree more.

这是一个非常难以操作化的概念。

It's a very difficult concept to operationalize.

是啊。

Yeah.

所以这种方式，尽管...让我们以这个案例为例，假设有10件事，我倾向于预测0.45的概率而你预测0.1，结果这些事都没发生——这样我们俩都算对了，而且我们都认为概率低于50%

So this particular way, even though, so let's take this case, let's say there's 10 things where I tended to predict point four five and you predicted point one and none of them happened so that we're both right and that we both thought it was below a half.

发生的可能性更低。

It was less likely.

但你比我更正确，因为什么？

But you were more right than I was because what?

这就是我想让你回应的点。

And here's what I want you to respond to.

在我看来，你可以辩解说，你只是比我更有信心。

It seems to me, you could argue, you just had more confidence than I did.

你在选择数字时更具策略性。

You were more strategic in how you picked your number.

你对实际概率并没有更准确的了解。

You didn't have any more accurate knowledge of the actual probability.

那么，你需要抛多少次硬币才能确定声称硬币有偏差的人比声称硬币接近平衡的人更接近正确？

Well, how many how many times did you have to flip that coin before you decided that the person who claims the coin is biased is closer to correct than the person who claims the coin is very close to equilibrium?

嗯，这是个很有挑战性的问题。

Well, that's a challenging question.

我在读这本书的时候想到了Legg Mason的比尔·米勒。

I thought while I was reading the book, I thought of Bill Miller of Legg Mason.

比尔·米勒连续至少十五年，可能更久，都跑赢了标普500指数。

So Bill Miller beat the S and P 500, I think for at least fifteen years in a row, maybe more.

很多人因此断定他一定是个天才，因为他能跑赢标普500。

And a lot of people concluded he had to be a genius because, well, he beat the S and P 500.

一年不算什么，但连续十五年，这概率太低了。

One year, not so impressive, but fifteen years, that's so unlikely.

但我们当然知道这并不能证明他是天才。

But of course, we know that that doesn't prove he's a genius.

这甚至不能证明他聪明。

It doesn't even prove he's smart.

可能仅仅意味着他运气好。

It might merely mean he was lucky.

在成千上万的共同基金经理中，恰好是他连续十五年跑赢了标普500指数。

Out of the thousands and tens of thousands of managers of mutual funds, he was the one who happened to beat the S and P five hundred fifteen years in a row.

我们知道只要有足够长的时间和足够多的经理人，这种情况就必然会发生。

And we know that over enough time and enough managers that's going to happen.

因此我们无法据此推断他未来的能力。

And so we know nothing about his ability going forward.

事实上，在他的连胜纪录被打破后，他的表现并不突出。

And in fact, he didn't do particularly well after his streak was broken.

他变笨了吗？

Did he get less smart?

他变得过于自信了吗？

Did he get overconfident?

我们无从得知。

We have no way of knowing.

因此，尽管我在书中发现许多有助于深思未来展望的有益内容，但这项根本的衡量技术在我看来仍是个挑战。

So I find myself, even though I found many things in the book that are useful in thinking thoughtfully about looking into the future, the fundamental measurement technique strikes me as a challenge.

对此你怎么看？

What do you say to that?

我认为这是个极好的问题。

I think that is a great question.

这是个非常非常深刻的问题。

It's a really, really deep question.

金融界人士当然会争论是否存在所谓的好判断力。

People in finance argue, of course, about whether there is such a thing as good judgment.

如果你坚信有效市场假说，自然会对此持高度怀疑态度。

If you're a really strong believer in the efficient markets hypothesis, you're going to be very skeptical.

只要投掷硬币的次数足够多，总会有几枚硬币连续出现76次、60次、70次甚至80次正面朝上的情况。

If you toss enough coins enough times, a few of them are bound to wind up heads 76 sixty, seventy, 80 times in a row.

你可以一直这样持续下去。

You can just keep keep doing that.

也有怀疑论者认为，比尔·米勒、沃伦·巴菲特或乔治·索罗斯不过是那些幸运的连续掷硬币序列之一，然后我们就把他们奉为天才。

And there are skeptics who argue that Bill Miller or for that matter Warren Buffett or George Soros were just one of those lucky sequences of of coin flips, and then we anoint them geniuses.

我们对超级预测者可能只是超级幸运的可能性非常敏感。

We are very sensitive to the possibility that super forecasters could be super lucky.

我们也始终持开放态度，认为任何特定的超级预测者可能只是超级幸运。

And we're always open to the possibility that any given super forecaster has been super lucky.

我们一直在寻找均值回归的模式。

We're always looking for patterns of regression toward the mean.

任务中的偶然性越大，均值回归效应就越明显。

The more chance there is in a task, the greater the regression toward the mean effect.

这正是我们持续关注的现象。

And that's just something we're continually looking for.

根据我们观察到的效应均值回归情况，我们最佳估计是IARPA赞助的地缘政治预测锦标赛中技能与运气比例约为七比三，这意味着技能占很大成分，运气也有显著影响。

Our best estimates are that the geopolitical forecasting tournament sponsored by IARPA had about a seventy thirty skill lock ratio based on the regression to toward the mean of of effects that we were observing, which means there's a big element of skill and there's a significant element of luck.

基于其他因素，比如我们引入的实验性干预措施能可靠地提高预测准确性。

And it it and based on other factors, like, we we introduce experimental manipulations that reliably improve accuracy.

如果完全是随机噪声，就不可能做到这一点。

If it were pure noise, it would wouldn't be possible to do that.

所以如果我们面对的是完全不可预测的极端噪声，就不可能开发出能提高准确性的训练模块或团队协作机制。

So it wouldn't be possible to develop training modules or teaming mechanisms that improve accuracy if we were dealing with a radically noisy deep and invariable.

但事实上我们能做到。

It is possible to do that.

因此，来自预测者个体差异的证据和实验证据等多方面证据表明，我们面对的并非完全不可预测的现象。

So various converging lines of evidence, both individual difference evidence among the forecasters and experimental evidence suggests that we're not dealing with a radically indeterminate phenomenon here.

确实存在良好的判断力这种东西，但运气成分也起着重要作用。

There is such a thing as good judgment, but there is certainly a significant element of luck as well.

当你阅读沃伦·巴菲特或查理·芒格对市场的合伙人分析时，面临的挑战之一是他们实在太聪明了。

And one of the challenges when you read Warren Buffett or Charlie Munger's partners analysis of the market, they're really smart.

他们充满了真知灼见，对吧？

They're full of interesting insights, right?

所以这强化了你的观点，也许并非运气使然。

So it reinforces your view that maybe it's not luck.

当然，挑战在于你无法确定那些特定见解是否真的重要。

The challenge, of course, is that you don't know whether those particular insights really matter.

确实如此。

That's true.

在宇宙中

In the universe

那些重要的事物里。

of things that matter.

就是

That's

就是这样。

That's that.

是啊。

Yeah.

我们在这个问题上完全达成一致。

We in complete agreement on this subject.

让我们以书中的一个例子来说明，我觉得这个例子非常有启发性，它展示了在某些预测问题和评估问题中技能的作用。比如你举的例子：有一个姓伦泽蒂的家庭，他们是独生子女家庭。

Let's take an example from the book, which I found really illuminating, which is an example of how there is a role for skill, at least in some forecasting problems and some estimation problems, which is you give the example of you're told that there's a family, their last name is Renzetti, they have an only child.

他们养宠物的概率有多大？

What are the odds that they have a pet?

然后讨论如何更深入地思考这个问题，而不是简单地说'我不知道'，或者更糟的是'哦，他们是独生子女家庭，这很重要'。

And talk about how you might think about that more thoughtfully than just saying, well, I don't know, or worse, Well, they have an only child, that's important.

而你提到的内外视角的区分让我觉得特别有启发性。

And you the inside outside distinction I found very illuminating.

嗯，这是书中关于超级预测者与普通预测者区别的更大讨论的一部分，超级预测者的特点就是倾向于从外部视角出发，逐步深入分析。

Well, it it it's part of a more general discussion in the book about what distinguishes super forecasters from regular forecasters, and that is the tendency of the super forecasters to start with the outside view and gradually work in.

所以无论是估算芝加哥有多少钢琴调音师，还是某个家庭是否养宠物，或是某个非洲独裁者能否再掌权一年这类例子，你都会先从一个初步估计开始。

So you would start with your you you would start your initial estimate whether it's, you know, trying to estimate number of piano tuners in Chicago or whether a particular family has a pet or whether a particular African dictator is likely to survive in power another year, all those kinds of examples.

你会先问：基本的存活率是多少？

You would start by saying, well, what's the base rate of survival?

宠物的基准概率是多少？

What's the base rate of pets?

另一个例子是这个非洲独裁者问题。

What what and so the another example is this African dictator problem.

我们可能会问你一个问题：y国的x独裁者是否有可能再掌权一年。

You you we might ask you a question about whether dictator x in country y is likely to survive in power for another year.

你可能会耸耸肩说：我对这个国家都知之甚少，更别说这个独裁者了。

And you might shrug and say, you know, I barely heard of the country, less still the dictator.

但你还是知道一些事情的。

But you do know a couple of things.

你知道的比你想象的要多。

You know more than you think you know.

其中之一就是，如果一个独裁者已经掌权一两年以上，那么他再掌权一年的可能性非常高。

And one of them is that once a day, if a dictator has been in power more than a year or two, the likelihood of the dictator being in power another year is very high.

这个概率高达90%以上。

It's a 9090% plus.

即便你对这位独裁者或其国家一无所知，你依然可以说：我知道当某人在一个国家建立了权力基础后，要推翻他们是很困难的。

You could even though you know nothing about the dictator or the country, you can say, well, I know that when someone has established a power base within a country, it's difficult to dislodge them.

既然如此，你会基于这个简单可验证的统计事实，以一个高概率值作为预测的起点。

Now, if if so you you would start your you would start your estimation process with a high probability because of that fact, just a simple demonstrable statistical fact.

然后你会说：现在我最好做些调研，了解一下这个人和他的国家。

And then you would say, well, now I better do a little bit of research and find out a little bit about this guy in his country.

如果你发现这个人已经91岁高龄且患有晚期前列腺癌，你可能需要调整预测概率。

If you discover that this particular person is 91 years old and has advanced prostate cancer, you might wanna modify your probability.

如果你发现首都郊区正在发生武装冲突，你可能需要调整预测概率。

If you discover there are there there's fighting in the suburbs of the capital, you might wanna modify your probability.

这体现了超级预测者独特的工作风格——在深入研究复杂的历史细节前，他们会尽可能获取初始的统计优势。

So these are captures part of the distinctive working style of the super forecasters is that they try to get as much initial statistical leverage on the problem as they can before they delve into the messy historical details.

我认为我们都认同循证医学、循证预测的理念，你的著作无疑证明了数据和统计学能帮助我们更好地预判重要事件。

And I think all of us like the idea of evidence based medicine, evidence based forecasting, your book is certainly a tribute to the potential for data and statistics to help improve our ability to anticipate events that are important.

我想难点在于选择哪些证据以及如何整合其他因素。

I guess the challenge is which evidence and how we incorporate the other factors.

我是说，你讲述了很多关于不同预测者（其中许多人只是所谓的业余爱好者）的有趣故事，这很美妙。

I mean, one of the you tell a lot of really interesting stories of the way the different forecasters, many of whom are just quote amateurs, which is beautiful.

他们不像我或其他人那样背负着博士学位，那些人也总试图预测事物。

They're not burdened by the PhD that I have and that others have who tend to try to predict things.

所以你谈了很多关于他们如何权衡证据的内容。

So you talk a lot about how they weigh evidence.

我在理智上接受这些结果时感到挑战的部分主要有两个问题。

Part I find intellectually challenging in accepting these results is sort of two issues.

首先，我想明确一点，你组建的这些业余团队与专家合作，并通过整合人员形成团队，同时提供如何协作和避免群体思维的指导——这是书中非常重要且非常有趣的部分，我认为对任何人来说都非常实用。

One is, and I want to make it clear, these amateur teams that you put together along with experts and the aggregation of folks into teams with advice on how to work together and how to avoid groupthink, which is a large part of the book, very, very interesting and very useful, I think, to anybody.

在所有这些例子中，他们都占据主导地位。

In all of these examples, they dominate.

他们并非仅仅比其他团队高出三个百分点。

It's not like they do three percentage points better than the others.

我只是想澄清这一点，明白吗？

I just wanna make that clear, right?

他们的表现确实远超许多受过更高教育的人以及所谓的专家。

They really did a lot better than just some of the more educated folk and the so called experts.

对吗？

Correct?

嗯，当你把所有因素综合起来看，在锦标赛中，这种累积优势确实让普通选手望尘莫及。

Well, when you throw everything together, the cumulative advantage does get to be quite staggering over the ordinary folks in the tournament.

确实如此。

That's true.

但你刚才讨论的是不同的组成部分。

But you were talking about different components here.

拥有天赋并让合适的人上车当然大有裨益。

It it it certainly helps to have talent and to get the right people on the bus.

超级预测者之间的个体差异表明他们绝非普通人。

So individual differences among super super forecasters are not just regular people.

他们在某些可量化的方面确实与众不同。

They are different in certain measurable ways.

他们在流体智力测试中得分更高。

They score higher measures of fluid intelligence.

他们的政治知识更丰富。

They're more politically knowledgeable.

他们的思想更开放。

They're more open minded.

但最重要的是，我不认为他们比普通人具备所有这些优势，这点我已经验证过了。

But most important, I don't think I think they have all those advantages over regular folks, and I've done that.

不。

No.

这些因素确实重要。

And those matter.

但我不认为他们比专业情报分析师更有优势。

But I don't think they have those advantages over professional intelligence analysts.

我不认为他们有更高的流体智力，肯定也没有更丰富的知识储备。

I don't think they have greater fluid intelligence, greater think definitely don't have greater knowledge.

而且我并不认为他们真的更加思想开明，尽管他们确实相当开放。

And I don't really think they're more even be more open minded, although they are pretty open minded.

我认为真正将超级预测者与情报界经验丰富的专业人士区分开来的，是他们能够超越后者——这在我看来是所有基准中最具挑战性的。

I think what really distinguishes the super forecasters from the seasoned professionals in the intelligence community whom they were able to outperform, and that was really, I thought, the most difficult of all the benchmarks.

我认为真正的区别在于，他们相信主观概率评估是一种可以且值得培养的技能。

I think what really distinguishes them is that they believe that subjective probability estimation is a skill that can be cultivated and is worth cultivating.

我想许多资深分析师，就像许多资深评论员一样，当他们看到诸如'希腊退出欧元区的可能性有多大'或'普京试图吞并更多乌克兰领土的概率是多少'这类问题时，

I think many of the sophisticated analysts, like many of the sophisticated pundits, when they when they see a question like, well, how likely is is Greece to leave the Eurozone or how likely is is Putin to try to annex more Ukrainian territory?

他们会耸耸肩，

They'll they'll shrug it.

然后说：'听着，

They'll say, look.

这是个独特的历史事件，

This is a unique historical event.

我们不可能给这种情况分配概率。'

There's no way we can assign a probability to this.

你本该在统计101课程里就学过这个。

You should have learned this in statistics one zero one.

你可以在扑克这类游戏中学会做出精确的概率判断。

You do you can you can make probab you can learn to make refined probability judgments in poker and things like that.

在扑克中你能学会区分60%和40%的下注概率，因为扑克是在明确抽样空间中的重复博弈，而频率统计正是统计101课程所教授的内容。

You can learn to distinguish sixty, forty bets from forty, sixty bets in poker because in poker, you're have you have repeated play in a well defined sampling universe, and the the the frequentest statistics everybody learns is that one zero one apply.

那些统计方法在这里根本不适用。

Those statistics just don't apply here.

所以你正在进行一种伪精确的练习。

So you're engaging in an exercise in pseudo precision.

所以你面对的是智商极高的人群。

So you've got people with really high IQ.

这听起来似乎有可能。

That strikes you as possible.

高智商人群确实会说出这类聪明话。

Really high IQ saying really smart things like this.

这阻碍了他们探索如何做得更好的可能性，而我认为IR支持者已经证明了这是可能的。

And it blocks them from exploring the potential of learning to do it better, which I think is what the IR proponent proved is possible.

现在，我想试着接受这个批评。

Now, I wanna try to take that criticism.

我会换一种方式表达。

I'll phrase it a little differently.

如果你问我，让我们来预测一下。

If you asked me, let's predict.

有一场橄榄球比赛。

There's a football game.

我们是在周一录制的这段内容。

We're recording this on a Monday.

今晚有一场橄榄球比赛。

There's a football game tonight.

是克利夫兰布朗队对阵巴尔的摩乌鸦队。

It's the Cleveland Browns against the Baltimore Ravens.

如果我没记错的话，这场比赛应该不太精彩。

If I remember correctly, it's not a very interesting game.

而我们想试着计算巴尔的摩队获胜的概率。

And we want to try to figure out the probability that Baltimore is going to win.

我觉得他们可能是被看好的一方。

I think they're probably favored.

对吧？

Okay?

所以他们理应获胜，但我们知道也可能爆冷。

So they're supposed to win, but we know that they might not.

不过我们想知道具体的概率是多少。

So we'd like to know, though, what the probability is.

解决这个问题有很多种方法。

Now, there are many ways to go about this question.

你书中那些人的做法是采用基准率——或者说方法之一——就像我们刚才讨论的那样采用基准率。

The way the people in your book go about it is they take a base rate, or this is one of the ways, they would take a base rate like we talked about a minute ago.

那些已经执政x年的独裁者中，有多少人还能再执政一年？

How many dictators who've been in office x years are in offices an additional year?

或者以宠物为例，你确实提到过，但书中讨论的是拥有宠物的家庭比例。

Or in the case of the pet example, you did mention it, but in the book you talk about what's the proportion of households that have pets.

这将是一个很好的起点。

That would be a great starting place.

你从这个开始，然后深入挖掘，尝试发现更多信息。

You start with that and then you dig deeper and you try to find out more stuff.

首先，基准率很难确定，因为这是周一晚上弱势球队的基准率吗？

It's first of all, it's really hard to know what the base rate is because is it the base rate of underdogs on a Monday night?

还是连续输掉两场比赛的球队的基准率？

Is the base rates of teams that have lost two games in a row?

于是人们开始尝试——这在足球领域可行，但要分析希腊退出欧元区就困难得多——他们试图系统性地积累统计证据，进行多元回归分析。

So what then people start to do and they can do it in football, it's harder to a lot harder to do with Greece exiting the Euro is they try to accumulate statistical evidence that, you know, in a systematic way, they run multivariate regressions.

他们在这方面相当擅长，因为就足球的本质而言，我们很擅长缩小范围。

And they're pretty good at that because for the nature of football, we're pretty good at narrowing down.