00:00:01
everybody wants to see into the future whether it's to find out whether or not
00:00:05
we should take a certain job figure out what's going to happen with a particular
00:00:08
relationship or maybe we just want to know if it's going to rain on Sunday but
00:00:12
despite all this interest in forecasting most actual forecasts aren't very good
00:00:15
and even and moreover they're also not very well analyzed in the new book super
00:00:20
forecasting the art and science of prediction Wharton Professor Philip tetlock looks into what makes people
00:00:25
good forecasters and suggests how you can incorporate some of those techniques into your own life Philip thanks for
00:00:31
being with us thank you my pleasure now thanks to people like Tom Friedman and Nate Silver and I think also maybe to
00:00:37
the rise of Big Data there seems to be a lot of interest in forecasting so I was
00:00:41
really surprised to learn from your book that despite all this interest in forecasting and maybe interest in people
00:00:46
who have fashioned themselves as forecasters forecasting itself's not very well studied it's not very well
00:00:51
analyzed at all I think that's fair to say and it's pretty threatening to keep
00:00:57
score of your forecasting accuracy you imagine you're a big shot pundit what incentive would you have to submit
00:01:06
to a forecasting tournament in which you had to play on a level playing field against ordinary human beings and the
00:01:12
answer is not much because the best possible outcome you could obtain is to tie it you're expected to win so the
00:01:18
best outcome is a tie and there's a good chance our research suggests that you're
00:01:23
not you're not going to win now the book is acted was actually decades in the
00:01:26
making I mean your research into this has stretched back for years and you actually start the book and in some ways
00:01:32
this all started with a dart throwing chimp could you tell us a little bit about that story and I know you said in
00:01:38
the book that people don't exactly get the right takeaways out of that study
00:01:41
but I thought it was interesting to sort of show what happens when we try and test forecasting sure well in our early
00:01:49
work which which I'm going to reveal how old I am goes back into the mid 1980s
00:01:54
we did use the metaphor of the dart throwing chimp to capture the idea baseline for performance which is how
00:02:01
much better can you do than chance if you had a system that we did that was just generating forecasts by chance how
00:02:08
about would you do relative to that and that actually is a baseline that some people can't beat now they can't beat it
00:02:18
for a lot of reasons sometimes the environment is just hopelessly difficult ranked if you were trying to bet on the
00:02:23
roulette wheel in Las Vegas you're not going to be able to do any better than a
00:02:27
dart throwing chimp but people sometimes fail to beat the dart throwing chimp even in environments where there are
00:02:33
predictable regularities that could be picked up if you were being a student off now in the book you point out that a
00:02:40
lot of people took away from that study that all predictions are bad that forecasting is bad but in fact what you
00:02:46
were pointing out really is that there's actually limits on predictability but
00:02:50
it's not all bad it's just that it there are limits on it that's right you don't want to be too
00:02:56
hard on people because some environments really are there's a lot of irreducible
00:03:00
uncertainty in some environments it's very difficult to to bring it down below
00:03:04
a certain point so it's unfair to portray people as as being dumb in some sense if they're failing to do something
00:03:13
that's impossible of course we don't know what's impossible until we try until we try and
00:03:19
earnest you net you don't discover what we call the optimal forecasting frontier
00:03:23
you don't discover how good it's possible to become in a particular forecasting environment until you run
00:03:29
forecasting tournaments competitive tournaments you plug in your best techniques for maximizing accuracy and
00:03:34
you see how good you can get and this essentially what we did in the forecasting tournaments with the US
00:03:38
government wasn't sponsored by the intelligence community the intelligence Advanced Research Projects Agency
00:03:44
we these are forecasting tournaments that run between 2011 and 2015 Volvo tens of thousands of forecasters trying
00:03:52
to predict about 500 questions posed by the intelligence community over that period of time and he found that some
00:03:59
people could do quite a bit better than the dart throwing chimp and they could meet some more demanding baselines as
00:04:04
well now one of the interesting things in the book is I mean you talked a little bit about where you found these
00:04:09
forecasters who are part of your study which is called the good judgment project can you talk a little bit about
00:04:13
where you recruited these people from and then also how these tournaments actually took place like what exactly
00:04:19
were they called on to do and how did they do well we were very opportunistic we recruited forecasters by advertising
00:04:28
through professional societies by advertising through blogs a number of high-profile bloggers helped us to
00:04:35
recruit forecasters people like Tyler Cowen Nate Silver various people were helpful in recruiting forecasters plus
00:04:42
we knew quite a few people from the earlier work that I'd done an expert political judgement so we were able to
00:04:46
gather initially a group of several thousand and we were able to build on that in subsequent years the questions I
00:04:55
you know I have to be careful about making big generalizations about how good or bad people are as forecasters as
00:05:03
I mentioned before you can make people look really bad if you want to you can pose just intractably difficult
00:05:09
questions or you can make people look really good you can you can pose questions that aren't all that hard and
00:05:16
so you you want to be wary of research that does cherry-picking and and and there are some aspects that some aspects
00:05:25
of that and some in some of the literature but we were looking for was a process of generating questions that
00:05:32
wasn't rigged one way or the other and the method we came up with was generating questions through the US
00:05:40
intelligence community there were questions that people inside the US intelligence community felt would be
00:05:46
of national security interest and relevance and reasonably representative of the types of tasks that intelligence
00:05:53
analysts are asked to do so these were questions typically they asked people to see out into the future
00:06:00
several months occasionally a bit longer occasionally shorter and we scored the accuracy of their
00:06:09
judgments over time we didn't have people make judgments one way or the other it wasn't yes or now we had people
00:06:16
make judgments on what's called a probability scale ranging from 0 to 1 and we carefully computed accuracy over time
00:06:25
we identified some people who are really good at it we called them super forecasters and but then they were later
00:06:30
assembled into super teams of super forecasters and they you know dominated the tournament tournament essentially
00:06:37
over the next over the next four years but we we did a number of other experiments as well looking for
00:06:42
techniques that could could be used to improve accuracy and we found some and now these super forecasters they really
00:06:48
came from all walks of life but one of the things you point out in the book is that what makes a good forecaster is
00:06:54
really how you think and can you talk a little bit about what do you mean by that and what are some of the unifying
00:06:59
characteristics of super forecasters well when you ask people in the political world who has good judgment
00:07:07
the answer typically is people who think like me so liberals tend to think that liberals have good judgment and if you
00:07:13
have good forecasting judgment and conservatives tend to think that they're better at it it turns out to be the case
00:07:19
that that pork a good forecast forecasting accuracy is not very closely associated with with ideology there's a
00:07:26
slight tendency for people who are the super forecasters to be more moderate and less ideological but there there are
00:07:34
lots of super forecasters who have strong opinions oh it distinguishes super forecasters is their ability to
00:07:39
put aside their opinions at least temporarily and just focus on accuracy and that's that's a very demanding
00:07:46
exercise for people now other things you mentioned I don't like at the end of the
00:07:50
book you have ten commandments for super forecasters and so I'm wanting now with
00:07:54
the super forecasters are there ways to make even super forecasters better are there conditions or environments to make
00:08:00
them I guess super super forecaster super super forecasters well event eventually you're gonna reach a
00:08:08
point where you're not going to be able to get any better because the as I've
00:08:11
mentioned it's some environment the environment itself has some degree of irreducible uncertainty so no matter how
00:08:15
good you are you're not gonna do a very good job probably predicting what the
00:08:20
value of Google is going to be next week on the on the New York Stock Exchange there so there are some things that are
00:08:26
very difficult to do and it's not clear that the super forecast even using super forecasters is going to
00:08:34
let you make appreciable headway on that but there are many things that are quite
00:08:38
doable that we previously didn't think we're doable and you there's a lot of
00:08:43
room for improving the accuracy of probability judgments on those things those are things like predicting weather
00:08:50
international conflicts or going to escalate or deescalate whether certain treaties are going to be signed or
00:08:56
approved by legislative legislators well what but whether Greece is going to leave the eurozone so there are a lot of
00:09:03
problems that you know have relevance the financial markets have relevance to business decisions where there is
00:09:08
potential for improving probability judgment where we have shown that experimentally now in the IR /
00:09:13
tournament where people typically don't don't do that people typically rely on
00:09:18
vague verbiage forecasts they people say well I think it's possible or this could
00:09:22
happen this might happen it's likely those are terms that you know are informative at some level but they're
00:09:29
not all that informative if I say that something could happen for example you know Greece could leave the eurozone by
00:09:34
the end of 2017 what does that mean it could mean I think there's a probability
00:09:38
of 1 percent or 99 percent you know we could be hit by an asteroid tomorrow the Sun could rise tomorrow yeah I know
00:09:47
there though there it's it's it's a very elastic word so the asking people to make crude
00:09:56
quantitative judgments which become progressively more refined over time is a very good way to both keep score and
00:10:03
get better at it now I find it I find it really ironic in the book that you know
00:10:09
we have all of these people in the world who have sort of fashioned themselves as
00:10:12
professional forecasters I mean pundits on TV they're really television personalities media personalities and
00:10:18
it's almost some and it seems like to do that job it's almost kind of a cult of
00:10:23
personality you don't want to be proven wrong you would never admit that you're
00:10:26
wrong you're just gonna sort of keep kicking it down the road and say no it's
00:10:28
going to happen but what you find in the book is that super forecasters one of the things that unites all of them
00:10:34
despite coming from all these walks of life is that there a lot they're willing
00:10:38
to be proven wrong or live they are open to the idea they're willing to sort of
00:10:41
look at things look at evidence and retread and pivot and so I found that interesting it was - it was kind of
00:10:47
ironic and I know you tell a story in the book about foxes vs. hedgehogs that was very interesting
00:10:52
right well yeah but the Fox Hedgehog metaphor is is drawn out of a surviving fragment
00:11:00
of poetry from the Greek warrior poor Killa cos 2,500 years ago scholars have puzzled over it for over
00:11:07
the centuries it runs something like this and of course I don't know ancient
00:11:11
Greek so I'm taking it on faith this is what it actually says the the Fox knows many things but the
00:11:19
Hedgehog knows one big thing now you can think of hedgehogs in in debates over political and economic issues as
00:11:33
people who have a big ideological vision Tom Friedman might be animated by a vision of a globalization the world is
00:11:40
flat libertarians are animated by the vision that free Mart there are free market
00:11:45
solutions for the vast majority of problems that beset us there are people on the left to see the
00:11:52
need for major state intervention to address various inequities there there are there are environmentalists who
00:12:01
think that we're on the cusp of an apocalypse of some sort and so you have people who are animated by a vision and
00:12:11
their forecasts are informed largely by that vision and whereas the Foxes tend to be more eclectic they kind of pick
00:12:19
and choose their ideas for a variety of schools of thought they might be a little bit environmentalist and a little
00:12:23
bit libertarian or there might be a little bit socialist and a little bit hawkish on certain national security
00:12:30
issues there they they blend things in unusual ways it's and they're harder to
00:12:36
classify politically now in the early work we found that the Foxes were more COO or more eclectic in their style of
00:12:45
thinking we're better forecasters than the hedgehogs and in the later work we found something
00:12:52
similar we found that people who scored high on psychological measures of active
00:12:56
open-mindedness and need for cognition those people who score high on those personality variables tended to do quite
00:13:03
a bit better as forecasters now it seemed to me then that we would really want is for more foxes to be kind of
00:13:10
these really prominent forecasters or these people that we're seeing on television that we're seeing in the news
00:13:15
but you know but then at the same time and although seems like their personalities are just not
00:13:20
necessarily what we think of like we're at odds with what we think of when we
00:13:24
think of a leader when we think of someone who's always putting themselves out there so how does that what does
00:13:28
that mean for trying to get more accurate forecasters I mean how do we get people to be lists to listen to the
00:13:33
Foxes when they might not be the sexiest or the most prominent or the most the people that we want to look at all the
00:13:38
time or listen to all the time that's right so what do we do that is a bit of
00:13:43
a dilemma imagine you are a producer for a major major television show and you have a
00:13:51
choice between someone who's going to come on the air and tell you that something decisive and bold and
00:13:59
interesting is going to sit eurozone is going to melt down the next two years or
00:14:03
Chinese economy is going to melt down or there's gonna be a jihadist coup and
00:14:07
Saudi Arabia he's got he's got a good big interesting story to tell and the
00:14:12
person knows quite a bit and can mobilize a lot of reasons to support the doom stir prediction say on eurozone or
00:14:19
China or Saudi Arabia or a boom stir prediction for that matter it doesn't matter but the person is charismatic and
00:14:24
forceful and can generate a lot of reasons why he or she is right as opposed to someone who comes on and says
00:14:32
yeah there's some danger eurozone's gonna melt down but on the other hand
00:14:37
there are these countervailing forces and on balance problem nothing dramatic is probably gonna happen in the next
00:14:42
year or so but it's possible that this could work so I'm who make what makes
00:14:47
better television to ask the question is to answer it so there is a tent there is
00:14:52
a preference for hedgehogs in part because hedgehogs generate better sound bites and people who generate better
00:14:59
sound bites generate better media ratings and that is what people get promoted on in the in the media business
00:15:06
so there is a bit of a perverse inverse relationship between how between having the skills that go into being a good
00:15:14
forecaster and having the skills that going go into being a fact of media presence now how does this come into
00:15:23
play though I mean sort of in the world that maybe those of us who are not hiring forecasters in a regular basis
00:15:27
don't so if I'm a company or the government and I'm trying to assemble a
00:15:32
team of good forecasters or get good accurate or accurate as possible forecasts what can I take from this book
00:15:38
I mean in terms of finding these people and employing these people and is that happening now or is it the same as what
00:15:45
maybe what we're seeing on CNN our Fox News all right well I think a lot of organizations in both the private and
00:15:50
the public sector are taking an increasing interest in forecasting tournaments and using them as methods of
00:15:56
keeping score on how accurate their their forecasting methods are they're people and their methods and their teams
00:16:02
there's a growing interest also in using forecasting tournaments to identify
00:16:06
people who are better at it and you know develop your own core of super forecasters there's growing interest in
00:16:13
exploring methods of training people to be better forecasters so I think this is
00:16:17
going on I probably shouldn't mention the names of any organizations who have adopted
00:16:22
this right now but a US intelligence community obviously has taken an active interest in this area and I and some
00:16:28
private sector organizations have as well now it was interesting to me that um so you were discussing a lot of the
00:16:34
book talks about a little bit about framing so it's not just about finding people who can make good forecasts but
00:16:39
it's also a lot about you know finding the right questions finding the right
00:16:42
ray to frame the problem breaking down a big problem into smaller little clusters
00:16:46
I mean if there's so much that goes into forecasting other than the actual forecasts that comes out of it and I
00:16:52
wonder that do people leave are people thinking enough about those things as well in addition to finding people who
00:16:57
give good forecasts I see that as one of the big objectives of the next generation of forecasting tournaments to
00:17:03
focus on generating not just good answers but good questions in the book we talked
00:17:09
about the parable of Tom Friedman and Bill flack Tom Friedman is of course a famous New York Times columnist Pulitzer
00:17:15
Prize winner who regular at Davos in the White House circulates in networks of power bill
00:17:24
flack is an anonymous retired hydrologist in Nebraska who also is a super forecaster we know a huge
00:17:31
amount about Bill flocks forecasting track record because he answered a very large number of questions in the course
00:17:36
of the tournament and demonstrated he could do so effectively but we know virtually nothing about Tom Friedman's
00:17:41
forecasting track record notwithstanding that he's written a great deal over the
00:17:45
last 35 years and he's a music he's a powerful analyst and a writer and and he
00:17:50
does many things very well but there's no way really to reconstruct with any
00:17:54
degree of certainty reasonable certainty how good a forecaster he is and Tom Friedman has
00:18:00
detractors he has admirers his his his his admirers might say well he was right about that it was a bad idea to expand
00:18:07
NATO eastward because it would provoke nationalist backlash in Russia or he was or he was wrong about Iraq because he
00:18:14
supported the 2003 invasion people have a lot of opinions about those things now
00:18:24
here's what we we did a careful analysis of Tom Friedman's columns and one of the
00:18:29
things we noticed is even though it's very difficult to discern whether or not
00:18:33
he's a good forecaster going back after the fact it is possible to detect we have some
00:18:40
really good questions he said he's a pretty darn good question generator and
00:18:44
we've actually begun to insert some of his his ideas for questions there they tend
00:18:51
to be rather open-ended we've managed to translate some of them into future forecasting tournaments so let me give
00:18:58
you an example from the past that illustrates the tension between being a super question generator and a super
00:19:04
forecaster so in 2008 2002 early 2003 before the Iraq invasion Tom Friedman wrote what I thought was a really quite
00:19:11
brilliant column on on Iraq in which he posed the following question we could really cut to the essence of a war what
00:19:18
key issue in deciding whether to go into a rocky yes is Iraq the way it is today because
00:19:25
Saddam Hussein is the way he is or a Saddam Hussein the way he is because Iraq is the way it is the chicken and
00:19:33
the egg and what would happen if you if you took away Saddam Hussein with the country disintegrate into a war of all
00:19:41
against all or would it move to boredom going to become a Jeffersonian liberal democracy in the next 15 or 20 years
00:19:49
now that maybe I'm not done maybe not quite that fast but in the in that direction things things would move in
00:19:56
that would move in that direction so Tom Friedman didn't know the answer to that question
00:20:05
many people think he made a big mistake in supporting the invasion of Iraq in 2003 but he was shrewd enough to pose
00:20:12
the right question and if we've been running forecasting tournaments in late
00:20:16
2002 early 2003 that would have been something we would have wanted very much to include in in in in in that exercise
00:20:24
and so I think the way the right way to think about Tom Friedman and Bill flack is that you know is that they they're
00:20:36
complementary and that Tom Friedman's greatest contribution to forecasting tournaments may well be and his
00:20:44
perspicacity and generating incisive questions he may be a good forecaster too but we
00:20:52
just don't know that yet right but we need in order but good forecasting we need the Tom Friedman's in the world and
00:20:57
the bill flacks so it to me it would seem it's just a question of trying to get them together in the right ways and
00:21:02
the right permutations to get the best better and to get better prediction that's where we come around in the book
00:21:07
it's not really Tom versus bill it's Tom and Bill it should be symbiotic right
00:21:12
and now to get a little back to I mentioned big data at the beginning now in the age of supercomputers a machine
00:21:16
learning and we're saying we can enter this into anything and get these answers
00:21:20
back to us I mean what do you think how do you think the role of human forecasting is going to change I mean
00:21:25
how do these how do using computers using data how does that complement or even compete with human forecasting well
00:21:32
in the book we conducted an interview with David Ferrucci who was when he was an IBM scientist he was responsible for
00:21:40
developing a famous computer program known as Watson which defeated the best human jeopardy players and we asked him
00:21:50
a number of questions about his views about the human machine forecasting and one question one law one line of
00:21:59
questioning was particularly interesting I think it was we it was it was very clear to him and
00:22:04
that it would be possible for a system like Watson to answer the following question reasonably readily
00:22:12
which two Russian leaders traded jobs in the last five years that question what Watson could search his historical
00:22:20
database it could figure it out reframe the question as will those Russian same Russian leaders change jobs
00:22:26
in the next five years would Watson have any capacity to answer a question like that and and his answer was no and the
00:22:35
question well how difficult would it be to reconfigure Watson so that it could answer a question like that and his
00:22:40
answer was massively difficult it would not be something that would be easy to accomplish any any time in the near in
00:22:46
the near future I think that's probably true I'm not an expert in that area but he obviously is
00:22:55
but when I think about what would be required but what's required to do the sorts of things that super forecasters
00:23:02
collectively do the amount of guesswork but the amount of informed guess work that goes into
00:23:08
constructing a forecast a reasonable forecast it's difficult for me to imagine existing AI artificial
00:23:15
intelligence systems doing that in the near term so now if reading the book I mean most people luckily will probably
00:23:24
not be asked to answer big questions about Iraq big questions about Korea or any some of the other things that you
00:23:29
talk about in the book but if someone is reading the book and just to become a better forecaster about their daily
00:23:34
lives I mean what do you hope that people take away from that to kind of apply it to the everyday to think
00:23:39
they're going through whether it's jobs relationships or even rain on Sunday
00:23:43
right right well I think a lot of people spend quite a bit of money on advice about the future that probably
00:23:56
isn't worth the amount of money they're spending on it and they don't really
00:24:00
know they have no way of knowing that because they have no way of knowing the track record to the people whose advice
00:24:05
they're seeking the most the most the best example that is probably in the domain of Finance where a lot of money
00:24:11
changes hands it's directed to people who claim to have some ability to predict the course of financial markets
00:24:18
that is an extraordinarily difficult thing to do I'm not saying it's impossible or that nobody can do it with
00:24:25
any better than the dark throwing chimp but it's a very difficult thing to do so
00:24:30
I think if people were more skeptical about the people to whom they turn for advice about possible futures I think
00:24:40
finance would be a case in point but I think more generally they should be very skeptical of the pundits they read and
00:24:45
the claims that politicians and other people make about the future as well it's very common for people to make bold
00:24:52
claims about the future and offer no evidence for their track records I would say it's almost universal
00:24:58
[Music] and so I guess if someone's making a bold claim is that where we should is
00:25:05
that the point where we should become suspicious I guess well the bolder the claim the more the burden of proof
00:25:12
should fall on the person to demonstrate that he or she has a good track record and it seems to me like it's often more
00:25:18
the bolder the claim the less likely someone's gonna question that person sometimes well that's a great point
00:25:24
that's a point of a human psychology as we take our cues about whether somebody
00:25:28
knows what he or she is talking about from how confident he or she seems to be and the more confident the more likely
00:25:33
you're going to be able to blunderbuss your way through the conversation so that's that's a that's a problem and
00:25:39
it suggests that people need to think a little bit more carefully when they make
00:25:43
appraisals of competence and not rely as quite as heavily as they do and what we
00:25:49
call the confidence heuristic it is it is true that confidence is is somewhat correlated with accuracy but it's it's
00:25:55
also possible for manipulative human beings to use that heuristic and turn us into money pumps well thank you so much
00:26:03
for being here we appreciate it okay thank you [Music] you