00:00:01
everybody wants to see into the future
00:00:03
whether it's to find out whether or not
00:00:05
we should take a certain job figure out
00:00:07
what's going to happen with a particular
00:00:08
relationship or maybe we just want to
00:00:10
know if it's going to rain on Sunday but
00:00:12
despite all this interest in forecasting
00:00:13
most actual forecasts aren't very good
00:00:15
and even and moreover they're also not
00:00:18
very well analyzed in the new book super
00:00:20
forecasting the art and science of
00:00:22
prediction Wharton Professor Philip
00:00:23
tetlock looks into what makes people
00:00:25
good forecasters and suggests how you
00:00:27
can incorporate some of those techniques
00:00:29
into your own life Philip thanks for
00:00:31
being with us thank you my pleasure now
00:00:32
thanks to people like Tom Friedman and
00:00:35
Nate Silver and I think also maybe to
00:00:37
the rise of Big Data there seems to be a
00:00:38
lot of interest in forecasting so I was
00:00:41
really surprised to learn from your book
00:00:43
that despite all this interest in
00:00:44
forecasting and maybe interest in people
00:00:46
who have fashioned themselves as
00:00:47
forecasters forecasting itself's not
00:00:50
very well studied it's not very well
00:00:51
analyzed at all I think that's fair to
00:00:54
say and it's pretty threatening to keep
00:00:57
score of your forecasting accuracy
00:00:59
you imagine you're a big shot pundit
00:01:02
what incentive would you have to submit
00:01:06
to a forecasting tournament in which you
00:01:08
had to play on a level playing field
00:01:09
against ordinary human beings and the
00:01:12
answer is not much because the best
00:01:13
possible outcome you could obtain is to
00:01:15
tie it you're expected to win so the
00:01:18
best outcome is a tie and there's a good
00:01:21
chance our research suggests that you're
00:01:23
not you're not going to win now the book
00:01:25
is acted was actually decades in the
00:01:26
making I mean your research into this
00:01:28
has stretched back for years and you
00:01:30
actually start the book and in some ways
00:01:32
this all started with a dart throwing
00:01:34
chimp could you tell us a little bit
00:01:36
about that story and I know you said in
00:01:38
the book that people don't exactly get
00:01:39
the right takeaways out of that study
00:01:41
but I thought it was interesting to sort
00:01:43
of show what happens when we try and
00:01:46
test forecasting sure well in our early
00:01:49
work which which I'm going to reveal how
00:01:51
old I am goes back into the mid 1980s
00:01:54
we did use the metaphor of the dart
00:01:57
throwing chimp to capture the idea
00:01:59
baseline for performance which is how
00:02:01
much better can you do than chance if
00:02:03
you had a system that we did that was
00:02:05
just generating forecasts by chance how
00:02:08
about would you do relative to that
00:02:11
and
00:02:12
that actually is a baseline that some
00:02:14
people can't beat now they can't beat it
00:02:18
for a lot of reasons sometimes the
00:02:20
environment is just hopelessly difficult
00:02:21
ranked if you were trying to bet on the
00:02:23
roulette wheel in Las Vegas you're not
00:02:25
going to be able to do any better than a
00:02:27
dart throwing chimp but people sometimes
00:02:29
fail to beat the dart throwing chimp
00:02:31
even in environments where there are
00:02:33
predictable regularities that could be
00:02:35
picked up if you were being a student
00:02:37
off now in the book you point out that a
00:02:40
lot of people took away from that study
00:02:41
that all predictions are bad that
00:02:43
forecasting is bad but in fact what you
00:02:46
were pointing out really is that there's
00:02:48
actually limits on predictability but
00:02:50
it's not all bad it's just that it there
00:02:52
are limits on it
00:02:53
that's right you don't want to be too
00:02:56
hard on people because some environments
00:02:58
really are there's a lot of irreducible
00:03:00
uncertainty in some environments it's
00:03:01
very difficult to to bring it down below
00:03:04
a certain point so it's unfair to
00:03:07
portray people as as being dumb in some
00:03:11
sense if they're failing to do something
00:03:13
that's impossible
00:03:14
of course we don't know what's
00:03:16
impossible until we try until we try and
00:03:19
earnest you net you don't discover what
00:03:21
we call the optimal forecasting frontier
00:03:23
you don't discover how good it's
00:03:25
possible to become in a particular
00:03:26
forecasting environment until you run
00:03:29
forecasting tournaments competitive
00:03:30
tournaments you plug in your best
00:03:32
techniques for maximizing accuracy and
00:03:34
you see how good you can get and this
00:03:36
essentially what we did in the
00:03:37
forecasting tournaments with the US
00:03:38
government wasn't sponsored by the
00:03:40
intelligence community the intelligence
00:03:42
Advanced Research Projects Agency
00:03:44
we these are forecasting tournaments
00:03:47
that run between 2011 and 2015 Volvo
00:03:50
tens of thousands of forecasters trying
00:03:52
to predict about 500 questions posed by
00:03:54
the intelligence community over that
00:03:56
period of time and he found that some
00:03:59
people could do quite a bit better than
00:04:01
the dart throwing chimp and they could
00:04:03
meet some more demanding baselines as
00:04:04
well now one of the interesting things
00:04:06
in the book is I mean you talked a
00:04:07
little bit about where you found these
00:04:09
forecasters who are part of your study
00:04:11
which is called the good judgment
00:04:12
project can you talk a little bit about
00:04:13
where you recruited these people from
00:04:15
and then also how these tournaments
00:04:17
actually took place like what exactly
00:04:19
were they called on to do and how did
00:04:21
they do well we were very opportunistic
00:04:25
we recruited forecasters by advertising
00:04:28
through professional societies by
00:04:30
advertising through blogs a number of
00:04:33
high-profile bloggers helped us to
00:04:35
recruit forecasters people like Tyler
00:04:36
Cowen Nate Silver various people were
00:04:39
helpful in recruiting forecasters plus
00:04:42
we knew quite a few people from the
00:04:43
earlier work that I'd done an expert
00:04:44
political judgement so we were able to
00:04:46
gather initially a group of several
00:04:48
thousand and we were able to build on
00:04:51
that in subsequent years the questions I
00:04:55
you know I have to be careful about
00:04:57
making big generalizations about how
00:04:59
good or bad people are as forecasters as
00:05:03
I mentioned before you can make people
00:05:04
look really bad if you want to you can
00:05:07
pose just intractably difficult
00:05:09
questions or you can make people look
00:05:11
really good you can you can pose
00:05:13
questions that aren't all that hard and
00:05:16
so you you want to be wary of research
00:05:20
that does cherry-picking and and and
00:05:22
there are some aspects that some aspects
00:05:25
of that and some in some of the
00:05:26
literature but we were looking for was a
00:05:29
process of generating questions that
00:05:32
wasn't rigged one way or the other and
00:05:35
the method we came up with was
00:05:37
generating questions through the US
00:05:40
intelligence community there were
00:05:41
questions that people inside the US
00:05:43
intelligence community felt would be
00:05:46
of national security interest and
00:05:49
relevance and reasonably representative
00:05:50
of the types of tasks that intelligence
00:05:53
analysts are asked to do
00:05:55
so these were questions typically they
00:05:58
asked people to see out into the future
00:06:00
several months
00:06:02
occasionally a bit longer occasionally
00:06:05
shorter and
00:06:06
we scored the accuracy of their
00:06:09
judgments over time we didn't have
00:06:12
people make judgments one way or the
00:06:13
other it wasn't yes or now we had people
00:06:16
make judgments on what's called a
00:06:17
probability scale ranging from 0 to 1
00:06:20
and
00:06:21
we carefully computed accuracy over time
00:06:25
we identified some people who are really
00:06:26
good at it we called them super
00:06:28
forecasters and but then they were later
00:06:30
assembled into super teams of super
00:06:32
forecasters and they you know dominated
00:06:35
the tournament tournament essentially
00:06:37
over the next over the next four years
00:06:39
but we we did a number of other
00:06:41
experiments as well looking for
00:06:42
techniques that could could be used to
00:06:44
improve accuracy and we found some and
00:06:46
now these super forecasters they really
00:06:48
came from all walks of life but one of
00:06:51
the things you point out in the book is
00:06:52
that what makes a good forecaster is
00:06:54
really how you think and can you talk a
00:06:56
little bit about what do you mean by
00:06:58
that and what are some of the unifying
00:06:59
characteristics of super forecasters
00:07:02
well when you ask people in the
00:07:05
political world who has good judgment
00:07:07
the answer typically is people who think
00:07:09
like me so liberals tend to think that
00:07:11
liberals have good judgment and if you
00:07:13
have good forecasting judgment and
00:07:15
conservatives tend to think that they're
00:07:16
better at it it turns out to be the case
00:07:19
that that pork a good forecast
00:07:21
forecasting accuracy is not very closely
00:07:23
associated with with ideology there's a
00:07:26
slight tendency for people who are the
00:07:28
super forecasters to be more moderate
00:07:31
and less ideological but there there are
00:07:34
lots of super forecasters who have
00:07:36
strong opinions oh it distinguishes
00:07:37
super forecasters is their ability to
00:07:39
put aside their opinions at least
00:07:41
temporarily and just focus on accuracy
00:07:43
and that's that's a very demanding
00:07:46
exercise for people now other things you
00:07:49
mentioned I don't like at the end of the
00:07:50
book you have ten commandments for super
00:07:52
forecasters and so I'm wanting now with
00:07:54
the super forecasters are there ways to
00:07:56
make even super forecasters better are
00:07:58
there conditions or environments to make
00:08:00
them I guess super super forecaster
00:08:02
super super forecasters well
00:08:05
event eventually you're gonna reach a
00:08:08
point where you're not going to be able
00:08:09
to get any better because the as I've
00:08:11
mentioned it's some environment the
00:08:12
environment itself has some degree of
00:08:13
irreducible uncertainty so no matter how
00:08:15
good you are you're not gonna do a very
00:08:18
good job probably predicting what the
00:08:20
value of Google is going to be next week
00:08:21
on the on the New York Stock Exchange
00:08:24
there so there are some things that are
00:08:26
very difficult to do and
00:08:29
it's not clear that the super forecast
00:08:31
even using super forecasters is going to
00:08:34
let you make appreciable headway on that
00:08:36
but there are many things that are quite
00:08:38
doable that we previously didn't think
00:08:40
we're doable and you there's a lot of
00:08:43
room for improving the accuracy of
00:08:45
probability judgments on those things
00:08:47
those are things like predicting weather
00:08:50
international conflicts or going to
00:08:52
escalate or deescalate whether certain
00:08:54
treaties are going to be signed or
00:08:56
approved by legislative legislators well
00:09:00
what but whether Greece is going to
00:09:01
leave the eurozone so there are a lot of
00:09:03
problems that you know have relevance
00:09:05
the financial markets have relevance to
00:09:06
business decisions where there is
00:09:08
potential for improving probability
00:09:10
judgment where we have shown that
00:09:11
experimentally now in the IR /
00:09:13
tournament where people typically don't
00:09:16
don't do that people typically rely on
00:09:18
vague verbiage forecasts they people say
00:09:21
well I think it's possible or this could
00:09:22
happen this might happen it's likely
00:09:24
those are terms that you know are
00:09:27
informative at some level but they're
00:09:29
not all that informative if I say that
00:09:31
something could happen for example you
00:09:33
know Greece could leave the eurozone by
00:09:34
the end of 2017 what does that mean it
00:09:36
could mean I think there's a probability
00:09:38
of 1 percent or 99 percent you know we
00:09:42
could be hit by an asteroid tomorrow
00:09:44
the Sun could rise tomorrow yeah I know
00:09:47
there though there it's it's it's a very
00:09:49
elastic word so
00:09:52
the asking people to make crude
00:09:56
quantitative judgments which become
00:09:58
progressively more refined over time is
00:10:01
a very good way to both keep score and
00:10:03
get better at it now I find it I find it
00:10:07
really ironic in the book that you know
00:10:09
we have all of these people in the world
00:10:10
who have sort of fashioned themselves as
00:10:12
professional forecasters I mean pundits
00:10:14
on TV they're really television
00:10:16
personalities media personalities and
00:10:18
it's almost some and it seems like to do
00:10:21
that job it's almost kind of a cult of
00:10:23
personality you don't want to be proven
00:10:24
wrong you would never admit that you're
00:10:26
wrong you're just gonna sort of keep
00:10:27
kicking it down the road and say no it's
00:10:28
going to happen but what you find in the
00:10:31
book is that super forecasters one of
00:10:32
the things that unites all of them
00:10:34
despite coming from all these walks of
00:10:35
life is that there a lot they're willing
00:10:38
to be proven wrong or live they are open
00:10:40
to the idea they're willing to sort of
00:10:41
look at things look at evidence and
00:10:43
retread and pivot and so I found that
00:10:45
interesting it was - it was kind of
00:10:47
ironic and I know you tell a story in
00:10:49
the book about foxes vs. hedgehogs that
00:10:51
was very interesting
00:10:52
right
00:10:54
well yeah but the Fox Hedgehog metaphor
00:10:57
is is drawn out of a surviving fragment
00:11:00
of poetry from the Greek warrior poor
00:11:02
Killa cos 2,500 years ago
00:11:05
scholars have puzzled over it for over
00:11:07
the centuries it runs something like
00:11:09
this and of course I don't know ancient
00:11:11
Greek so I'm taking it on faith this is
00:11:13
what it actually says
00:11:15
the
00:11:16
the Fox knows many things but the
00:11:19
Hedgehog knows one big thing
00:11:23
now
00:11:26
you can think of hedgehogs in in debates
00:11:30
over political and economic issues as
00:11:33
people who have a big ideological vision
00:11:36
Tom Friedman might be animated by a
00:11:38
vision of a globalization the world is
00:11:40
flat
00:11:41
libertarians are animated by the vision
00:11:43
that free Mart there are free market
00:11:45
solutions for the vast majority of
00:11:47
problems that beset us
00:11:49
there are people on the left to see the
00:11:52
need for major state intervention to
00:11:56
address various inequities there there
00:11:59
are there are environmentalists who
00:12:01
think that we're on the cusp of an
00:12:04
apocalypse of some sort and so you have
00:12:08
people who are animated by a vision and
00:12:11
their forecasts are informed largely by
00:12:14
that vision and whereas the Foxes tend
00:12:17
to be more eclectic they kind of pick
00:12:19
and choose their ideas for a variety of
00:12:21
schools of thought they might be a
00:12:22
little bit environmentalist and a little
00:12:23
bit libertarian or there might be a
00:12:25
little bit socialist and a little bit
00:12:28
hawkish on certain national security
00:12:30
issues there they they blend things in
00:12:33
unusual ways it's and they're harder to
00:12:36
classify politically now in the early
00:12:40
work we found that the Foxes were more
00:12:43
COO or more eclectic in their style of
00:12:45
thinking we're better forecasters than
00:12:47
the hedgehogs and
00:12:49
in the later work we found something
00:12:52
similar we found that people who scored
00:12:54
high on psychological measures of active
00:12:56
open-mindedness and need for cognition
00:12:58
those people who score high on those
00:13:01
personality variables tended to do quite
00:13:03
a bit better as forecasters now it
00:13:06
seemed to me then that we would really
00:13:07
want is for more foxes to be kind of
00:13:10
these really prominent forecasters or
00:13:12
these people that we're seeing on
00:13:13
television that we're seeing in the news
00:13:15
but you know but then at the same time
00:13:17
and although seems like their
00:13:18
personalities are just not
00:13:20
necessarily what we think of like we're
00:13:22
at odds with what we think of when we
00:13:24
think of a leader when we think of
00:13:25
someone who's always putting themselves
00:13:26
out there so how does that what does
00:13:28
that mean for trying to get more
00:13:29
accurate forecasters I mean how do we
00:13:31
get people to be lists to listen to the
00:13:33
Foxes when they might not be the sexiest
00:13:35
or the most prominent or the most the
00:13:37
people that we want to look at all the
00:13:38
time or listen to all the time that's
00:13:40
right so what do we do that is a bit of
00:13:43
a dilemma
00:13:45
imagine you are a producer for a major
00:13:49
major television show and you have a
00:13:51
choice between someone who's going to
00:13:53
come on the air and tell you that
00:13:56
something decisive and bold and
00:13:59
interesting is going to sit eurozone is
00:14:00
going to melt down the next two years or
00:14:03
Chinese economy is going to melt down or
00:14:05
there's gonna be a jihadist coup and
00:14:07
Saudi Arabia he's got he's got a good
00:14:09
big interesting story to tell and the
00:14:12
person knows quite a bit and can
00:14:14
mobilize a lot of reasons to support the
00:14:16
doom stir prediction say on eurozone or
00:14:19
China or Saudi Arabia or a boom stir
00:14:21
prediction for that matter it doesn't
00:14:22
matter but the person is charismatic and
00:14:24
forceful and can generate a lot of
00:14:27
reasons why he or she is right as
00:14:30
opposed to someone who comes on and says
00:14:32
yeah there's some danger eurozone's
00:14:36
gonna melt down but on the other hand
00:14:37
there are these countervailing forces
00:14:38
and on balance problem nothing dramatic
00:14:41
is probably gonna happen in the next
00:14:42
year or so but it's possible that this
00:14:44
could work so I'm who make what makes
00:14:47
better television to ask the question is
00:14:49
to answer it so there is a tent there is
00:14:52
a preference for hedgehogs in part
00:14:55
because hedgehogs generate better sound
00:14:57
bites and people who generate better
00:14:59
sound bites generate better media
00:15:01
ratings and that is what people get
00:15:03
promoted on in the in the media business
00:15:06
so there is a bit of a perverse inverse
00:15:09
relationship between how between having
00:15:13
the skills that go into being a good
00:15:14
forecaster and having the skills that
00:15:17
going go into being a fact of media
00:15:19
presence now how does this come into
00:15:23
play though I mean sort of in the world
00:15:24
that maybe those of us who are not
00:15:26
hiring forecasters in a regular basis
00:15:27
don't so if I'm a company or the
00:15:29
government and I'm trying to assemble a
00:15:32
team of good forecasters or get good
00:15:34
accurate or accurate as possible
00:15:36
forecasts what can I take from this book
00:15:38
I mean in terms of finding these people
00:15:40
and employing these people and is that
00:15:41
happening now or is it the same as what
00:15:45
maybe what we're seeing on CNN our Fox
00:15:46
News all right well I think a lot of
00:15:49
organizations in both the private and
00:15:50
the public sector are taking an
00:15:52
increasing interest in forecasting
00:15:53
tournaments and using them as methods of
00:15:56
keeping score on how accurate their
00:15:58
their forecasting methods are they're
00:16:00
people and their methods and their teams
00:16:02
there's a growing interest also in using
00:16:04
forecasting tournaments to identify
00:16:06
people who are better at it and you know
00:16:08
develop your own core of super
00:16:10
forecasters there's growing interest in
00:16:13
exploring methods of training people to
00:16:15
be better forecasters so I think this is
00:16:17
going on
00:16:18
I probably shouldn't mention the names
00:16:20
of any organizations who have adopted
00:16:22
this right now but a US intelligence
00:16:23
community obviously has taken an active
00:16:25
interest in this area and I and some
00:16:28
private sector organizations have as
00:16:29
well now it was interesting to me that
00:16:32
um so you were discussing a lot of the
00:16:34
book talks about a little bit about
00:16:35
framing so it's not just about finding
00:16:37
people who can make good forecasts but
00:16:39
it's also a lot about you know finding
00:16:41
the right questions finding the right
00:16:42
ray to frame the problem breaking down a
00:16:44
big problem into smaller little clusters
00:16:46
I mean if there's so much that goes into
00:16:47
forecasting other than the actual
00:16:50
forecasts that comes out of it and I
00:16:52
wonder that do people leave are people
00:16:53
thinking enough about those things as
00:16:55
well in addition to finding people who
00:16:57
give good forecasts I see that as one of
00:16:58
the big objectives of the next
00:17:01
generation of forecasting tournaments to
00:17:03
focus on
00:17:04
generating not just good answers but
00:17:06
good questions in the book we talked
00:17:09
about the parable of Tom Friedman and
00:17:12
Bill flack Tom Friedman is of course a
00:17:13
famous New York Times columnist Pulitzer
00:17:15
Prize winner who
00:17:18
regular at Davos in the White House
00:17:20
circulates in networks of power bill
00:17:24
flack is an anonymous
00:17:26
retired hydrologist in Nebraska who also
00:17:29
is a super forecaster we know a huge
00:17:31
amount about Bill flocks forecasting
00:17:33
track record because he answered a very
00:17:35
large number of questions in the course
00:17:36
of the tournament and demonstrated he
00:17:38
could do so effectively but we know
00:17:40
virtually nothing about Tom Friedman's
00:17:41
forecasting track record notwithstanding
00:17:43
that he's written a great deal over the
00:17:45
last 35 years and he's a music he's a
00:17:48
powerful analyst and a writer and and he
00:17:50
does many things very well but there's
00:17:52
no way really to reconstruct with any
00:17:54
degree of certainty
00:17:56
reasonable certainty how good a
00:17:58
forecaster he is and Tom Friedman has
00:18:00
detractors he has admirers his his his
00:18:03
his admirers might say well he was right
00:18:05
about that it was a bad idea to expand
00:18:07
NATO eastward because it would provoke
00:18:08
nationalist backlash in Russia or he was
00:18:11
or he was wrong about Iraq because he
00:18:14
supported the 2003 invasion
00:18:16
people have a lot of opinions about
00:18:18
those things now
00:18:24
here's what we we did a careful analysis
00:18:27
of Tom Friedman's columns and one of the
00:18:29
things we noticed is even though it's
00:18:31
very difficult to discern whether or not
00:18:33
he's a good forecaster going back after
00:18:36
the fact
00:18:37
it is possible to detect we have some
00:18:40
really good questions he said he's a
00:18:42
pretty darn good question generator and
00:18:44
we've actually begun to insert some of
00:18:46
his
00:18:47
his ideas for questions there they tend
00:18:51
to be rather open-ended we've managed to
00:18:53
translate some of them into future
00:18:55
forecasting tournaments so let me give
00:18:58
you an example from the past that
00:19:00
illustrates the tension between being a
00:19:02
super question generator and a super
00:19:04
forecaster so in 2008 2002 early 2003
00:19:07
before the Iraq invasion Tom Friedman
00:19:09
wrote what I thought was a really quite
00:19:11
brilliant column on on Iraq in which he
00:19:14
posed the following question we could
00:19:16
really cut to the essence of a war what
00:19:18
key issue in deciding whether to go into
00:19:20
a rocky yes
00:19:22
is Iraq the way it is today because
00:19:25
Saddam Hussein is the way he is or a
00:19:27
Saddam Hussein the way he is because
00:19:30
Iraq is the way it is the chicken and
00:19:33
the egg and what would happen if you if
00:19:36
you took away Saddam Hussein with the
00:19:39
country disintegrate into a war of all
00:19:41
against all or would it move to boredom
00:19:44
going to become a Jeffersonian liberal
00:19:46
democracy in the next 15 or 20 years
00:19:49
now that maybe I'm not done maybe not
00:19:52
quite that fast but in the in that
00:19:54
direction things things would move in
00:19:56
that would move in that direction so
00:20:01
Tom Friedman didn't know the answer to
00:20:04
that question
00:20:05
many people think he made a big mistake
00:20:07
in supporting the invasion of Iraq in
00:20:09
2003 but he was shrewd enough to pose
00:20:12
the right question and if we've been
00:20:15
running forecasting tournaments in late
00:20:16
2002 early 2003 that would have been
00:20:18
something we would have wanted very much
00:20:21
to include in in in in in that exercise
00:20:24
and
00:20:28
so I think the way the right way to
00:20:31
think about Tom Friedman and Bill flack
00:20:32
is that you know is that they they're
00:20:36
complementary and that Tom Friedman's
00:20:39
greatest contribution to forecasting
00:20:41
tournaments may well be and his
00:20:44
perspicacity and generating
00:20:47
incisive questions
00:20:49
he may be a good forecaster too but we
00:20:52
just don't know that yet right but we
00:20:54
need in order but good forecasting we
00:20:56
need the Tom Friedman's in the world and
00:20:57
the bill flacks so it to me it would
00:20:59
seem it's just a question of trying to
00:21:00
get them together in the right ways and
00:21:02
the right permutations to get the best
00:21:04
better and to get better prediction
00:21:06
that's where we come around in the book
00:21:07
it's not really Tom versus bill it's Tom
00:21:09
and Bill it should be symbiotic right
00:21:12
and now to get a little back to I
00:21:13
mentioned big data at the beginning now
00:21:15
in the age of supercomputers a machine
00:21:16
learning and we're saying we can enter
00:21:18
this into anything and get these answers
00:21:20
back to us I mean what do you think how
00:21:22
do you think the role of human
00:21:23
forecasting is going to change I mean
00:21:25
how do these how do using computers
00:21:27
using data how does that complement or
00:21:29
even compete with human forecasting well
00:21:32
in the book we conducted an interview
00:21:34
with David Ferrucci who was when he was
00:21:38
an IBM scientist he was responsible for
00:21:40
developing a famous computer program
00:21:43
known as Watson which defeated the best
00:21:46
human jeopardy players and we asked him
00:21:50
a number of questions about his views
00:21:51
about the human machine forecasting and
00:21:55
one question one law one line of
00:21:59
questioning was particularly interesting
00:22:00
I think it was
00:22:01
we it was it was very clear to him and
00:22:04
that it would be possible for
00:22:07
a system like Watson to answer the
00:22:09
following question reasonably readily
00:22:12
which two Russian leaders traded jobs in
00:22:14
the last five years that question what
00:22:18
Watson could search his historical
00:22:20
database it could figure it out
00:22:23
reframe the question as will those
00:22:25
Russian same Russian leaders change jobs
00:22:26
in the next five years would Watson have
00:22:29
any capacity to answer a question like
00:22:31
that and and his answer was no and the
00:22:35
question well how difficult would it be
00:22:36
to reconfigure Watson so that it could
00:22:38
answer a question like that and his
00:22:40
answer was massively difficult it would
00:22:43
not be something that would be easy to
00:22:44
accomplish any any time in the near in
00:22:46
the near future I
00:22:49
think that's probably true I'm not an
00:22:52
expert in that area but he obviously is
00:22:55
but when I think about what would be
00:22:58
required but what's required to do the
00:23:00
sorts of things that super forecasters
00:23:02
collectively do
00:23:04
the amount of guesswork but the amount
00:23:07
of informed guess work that goes into
00:23:08
constructing a forecast a reasonable
00:23:10
forecast it's difficult for me to
00:23:13
imagine existing AI artificial
00:23:15
intelligence systems doing that in the
00:23:19
near term so now if reading the book I
00:23:22
mean most people luckily will probably
00:23:24
not be asked to answer big questions
00:23:25
about Iraq big questions about Korea or
00:23:28
any some of the other things that you
00:23:29
talk about in the book but if someone is
00:23:30
reading the book and just to become a
00:23:32
better forecaster about their daily
00:23:34
lives I mean what do you hope that
00:23:36
people take away from that to kind of
00:23:38
apply it to the everyday to think
00:23:39
they're going through whether it's jobs
00:23:40
relationships or even rain on Sunday
00:23:43
right right well I think a lot of people
00:23:48
spend quite a bit of money on
00:23:52
advice about the future that probably
00:23:56
isn't worth the amount of money they're
00:23:58
spending on it and they don't really
00:24:00
know they have no way of knowing that
00:24:02
because they have no way of knowing the
00:24:03
track record to the people whose advice
00:24:05
they're seeking the most the most the
00:24:07
best example that is probably in the
00:24:09
domain of Finance where a lot of money
00:24:11
changes hands it's directed to people
00:24:14
who claim to have some ability to
00:24:16
predict the course of financial markets
00:24:18
that is an extraordinarily difficult
00:24:21
thing to do I'm not saying it's
00:24:23
impossible or that nobody can do it with
00:24:25
any better than the dark throwing chimp
00:24:28
but it's a very difficult thing to do so
00:24:30
I think if people were more skeptical
00:24:33
about the people to whom they turn for
00:24:36
advice about possible futures I think
00:24:40
finance would be a case in point but I
00:24:42
think more generally they should be very
00:24:43
skeptical of the pundits they read and
00:24:45
the claims that politicians and other
00:24:47
people make about the future as well
00:24:50
it's very common for people to make bold
00:24:52
claims about the future and offer no
00:24:54
evidence for their track records I would
00:24:56
say it's almost universal
00:24:58
[Music]
00:25:01
and so I guess if someone's making a
00:25:03
bold claim is that where we should is
00:25:05
that the point where we should become
00:25:06
suspicious I guess well the bolder the
00:25:09
claim the more the burden of proof
00:25:12
should fall on the person to demonstrate
00:25:14
that he or she has a good track record
00:25:16
and it seems to me like it's often more
00:25:18
the bolder the claim the less likely
00:25:20
someone's gonna question that person
00:25:22
sometimes well that's a great point
00:25:24
that's a point of a human psychology as
00:25:26
we take our cues about whether somebody
00:25:28
knows what he or she is talking about
00:25:29
from how confident he or she seems to be
00:25:31
and the more confident the more likely
00:25:33
you're going to be able to blunderbuss
00:25:35
your way through the conversation
00:25:37
so that's that's a that's a problem and
00:25:39
it suggests that people need to think a
00:25:42
little bit more carefully when they make
00:25:43
appraisals of competence and not rely as
00:25:47
quite as heavily as they do and what we
00:25:49
call the confidence heuristic it is it
00:25:51
is true that confidence is is somewhat
00:25:53
correlated with accuracy but it's it's
00:25:55
also possible for manipulative human
00:25:57
beings to use that heuristic and turn us
00:26:01
into money pumps well thank you so much
00:26:03
for being here we appreciate it okay
00:26:05
thank you
00:26:09
[Music]
00:26:21
you
