00:00:08
I first got introduced to the
00:00:12
forecasting
00:00:13
by a long time
00:00:15
at Dartmouth Ben Valentino who is a
00:00:17
political scientist and he alerted me to
00:00:21
an opportunity to forecast with the good
00:00:26
judgment project and this was right
00:00:28
around the time when I was leaving
00:00:30
Baghdad Iraq after my second tour there
00:00:34
and I was looking for some way of doing
00:00:39
forecasting because I had been something
00:00:41
that I've been interested in for for a
00:00:43
while and that's why been introduced me
00:00:45
to this idea and you know I the timing
00:00:50
worked out as well I was able to come
00:00:52
back to the United States and that was
00:00:54
right around the time the the first year
00:00:56
of the tournament kicked off and I just
00:00:58
went from there well in my work on
00:01:01
security a big part of the question is
00:01:04
whose security might be in danger in the
00:01:08
coming years so secure this field of
00:01:11
security studies inherently is about
00:01:13
forecasting making predictions about how
00:01:15
actors might be vulnerable which
00:01:18
strategies might help them and so forth
00:01:20
so I got interested in the good judgment
00:01:22
project because I wanted to build these
00:01:24
skills I've always had the interest and
00:01:26
when I first started my undergrad I
00:01:29
started a bunch of things I was one of
00:01:31
these constantly changing your majors
00:01:32
people and I changed between applied
00:01:35
physics is where I started and then I
00:01:37
went to International Studies but then
00:01:39
somehow transitioned to anthropology and
00:01:42
archaeology but I always kept the
00:01:44
interest up in International Studies so
00:01:46
it kind of went from wanting to study
00:01:48
current culture as Moe was going on to
00:01:49
studying past cultures and what was
00:01:52
going on there and that the constant you
00:01:57
know urge to learn more about all
00:01:58
societies kept me interested in
00:02:00
forecasting in what was going on in
00:02:02
different countries and as I said well
00:02:05
switching all the time I went to quite a
00:02:07
few different universities in quite a
00:02:09
few different countries I was at school
00:02:11
in the Stars beginning Utah followed by
00:02:15
Vermont and then Norway then back to
00:02:18
Utah and I finished my doctorate in
00:02:19
Scotland where I was there for seven
00:02:22
years and then I went to Ireland for ten
00:02:24
years so you know constantly moving in
00:02:26
different cultures different societies
00:02:27
different political system
00:02:29
it was always something of great
00:02:30
interest to me my professional life has
00:02:32
involved forecasting because that's what
00:02:35
I do or have done to help other people
00:02:38
make better decisions hopefully but
00:02:41
several years ago I attended a Penn
00:02:45
engaging Minds event in New York City
00:02:48
and Phil tetlock and Bob MELAS were
00:02:51
speaking about the good judgment project
00:02:54
which was very intriguing to me and so I
00:02:58
inquired about joining as a forecaster
00:03:00
and several months later when positions
00:03:04
opened up I became a forecaster I think
00:03:08
I am one of those people who can look at
00:03:12
things a lot of different ways and I'm
00:03:14
not the most adept at it among the
00:03:17
forecasters but that's definitely my
00:03:20
type and so I was a good fit but I was
00:03:24
also driven because I'm acutely aware of
00:03:27
the trouble the world is in and I think
00:03:30
the the as much as they have a dark side
00:03:35
us organizations like I are and the
00:03:39
intelligence community in general that
00:03:41
need all the help they can get we are
00:03:45
basically the good guys and so so there
00:03:52
was plenty of you know driving force
00:03:54
behind my interest that goes back to
00:03:57
when I was a younger man during the
00:04:01
first Gulf War in 1991 I was I'm
00:04:05
interested in the military military
00:04:06
history and a lot of people and pundits
00:04:09
were making predictions about the
00:04:12
outcome the manner in which the war
00:04:15
would be waged and decided as well as
00:04:19
casualty counts and how long it would
00:04:21
take and basically everyone was wrong on
00:04:25
the duration and the and the
00:04:29
destructiveness well the cat and the
00:04:30
number of casualties suffered by
00:04:31
coalition forces and that was
00:04:35
interesting to me because there was one
00:04:37
historian I'd like to say trevor Dupree
00:04:40
you got it right and I was
00:04:42
interested in how he got it right like
00:04:45
and evidently he had a system in just a
00:04:50
few years later during the 1994 midterm
00:04:54
elections when Bill Clinton lost in a
00:04:58
landslide to the Republicans the
00:05:00
Republicans took the Congress for the
00:05:03
first time in 40 years after that defeat
00:05:06
basically every pundit on Earth was
00:05:09
saying that Bill Clinton is a one-term
00:05:11
president and being a staunch Democrat
00:05:14
and a Clinton supporter I was
00:05:15
traumatized by that possibility and
00:05:17
wondered whether they knew they whether
00:05:20
that they knew what they were talking
00:05:21
about and at about that time I found a
00:05:25
book written by another historian Alan
00:05:27
Jay Lippman
00:05:28
um about predicting you know who would
00:05:31
become the next president and evidently
00:05:34
he had a system where you didn't need to
00:05:36
use polls you just look at history and
00:05:39
using his method I realized that
00:05:41
actually Bill Clinton was in a pretty
00:05:43
good shape his his chances weren't
00:05:47
guaranteed but you know even a year
00:05:48
before the election I had a pretty good
00:05:52
inkling that he had a good chance of
00:05:54
winning re-election and this was against
00:05:57
what all the pundits were saying so
00:05:59
based on those two events there were
00:06:00
probably other events as well that's why
00:06:03
I became interested in forecasting
00:06:05
basically the fact that everyone you
00:06:07
know got thing big things wrong and I
00:06:09
want to know why and some people got big
00:06:12
things right and I wanted to know why
00:06:14
and whether I could do as well as I
00:06:17
could
00:06:17
so ever since then I was I started you
00:06:20
know I'm my own forecasting you know the
00:06:22
outcomes of Wars and elections with
00:06:25
mixed results but you know so but over
00:06:28
20 years so that's how that's how I
00:06:31
became interested
00:06:32
I'm politically very interested in
00:06:36
what's happening and I used to read the
00:06:38
538 blog that was done by nate silver he
00:06:42
advertised for the good judgment project
00:06:44
i don't know where i saw the notice that
00:06:48
you guys were putting together this
00:06:50
program it was on one of the standard
00:06:53
outlets
00:06:53
maybe Wall Street Journal maybe
00:06:55
Bloomberg
00:06:56
but I read an article that you were
00:06:59
putting together this forecasting
00:07:01
tournament and that you were trying to
00:07:03
find out what it is makes a good
00:07:05
forecaster and I feel like I owe the
00:07:10
system for so many years I've been doing
00:07:14
research and I've relied on other people
00:07:16
to contribute to that research and they
00:07:18
have selflessly and so I'm a chronic
00:07:21
volunteer as well when somebody says I
00:07:24
need your help to do my research I
00:07:26
volunteer for instance the New York
00:07:28
Blood Center contacted me and said
00:07:30
you're not allowed to give blood but
00:07:32
we've got this research program and
00:07:35
blood donation will you come in and
00:07:36
donate a pint of blood every month so we
00:07:39
can see what happens so I did that just
00:07:42
to pay back the system so I'm paying
00:07:45
back the system by being in the good
00:07:47
judgment project
00:07:51
I think that
00:07:54
intellectual curiosity and just a
00:07:56
willingness to put in the work and to
00:08:00
think hard about specific problems and
00:08:02
try to be as objective as possible is
00:08:05
what allowed me to make more accurate
00:08:07
forecasts and others within the pool I
00:08:10
wouldn't necessarily say that you know
00:08:14
what we did or what I did as an
00:08:16
individual is the best possible at
00:08:18
forecasting because they're you know can
00:08:21
be so many more improvements I I think
00:08:23
in terms of how much you know about a
00:08:26
specific subject you can always improve
00:08:28
your level of knowledge and I think you
00:08:30
can always improve you know how you're
00:08:33
you're specifically viewing a situation
00:08:35
how that situation could potentially
00:08:37
evolve so having a greater understanding
00:08:39
of how humans interact with each other
00:08:41
relate to each other the political
00:08:44
contours of specific situations and
00:08:46
events those are all things that you can
00:08:48
gain greater knowledge on and get better
00:08:49
at forecasting in so yeah yeah being
00:08:53
open-minded and being open-minded enough
00:08:55
to realize that you need to remain
00:08:57
open-minded I think are some of the keys
00:08:59
to being you know for for my own
00:09:03
forecasting performance well I think
00:09:06
there were a couple of factors one was
00:09:08
that I am in the middle of my career now
00:09:10
and I've spent a lot of time studying
00:09:11
exactly the kind of issues that the good
00:09:14
judgment project was focusing on and so
00:09:18
I was familiar with trade issues I was
00:09:20
familiar with national security issues
00:09:23
the rise and fall of governments that
00:09:26
kind of thing so it really was right in
00:09:28
my issue area also I I use a theory that
00:09:33
can be well applied in this kind of
00:09:35
situation it's called structural realist
00:09:37
theory and it gives a sense of big
00:09:43
recurring outcomes and international
00:09:45
relations the rise and fall of great
00:09:47
powers the tendency for military
00:09:50
conflict to occur competition and trade
00:09:54
and so forth and so I had a kind of
00:09:56
broad understanding of international
00:09:58
affairs that I could apply to particular
00:10:01
situations and the beauty of this theory
00:10:03
was that it was it's really pre open
00:10:06
states have a lot of choices about
00:10:07
how they act and so I wasn't falling
00:10:12
into the trap of thinking that something
00:10:14
was inevitable but instead was
00:10:16
investigating which things were more
00:10:18
likely which things were less likely so
00:10:20
this theory was really conducive to
00:10:23
probabilistic forecasting and my
00:10:26
background knowledge was really helpful
00:10:28
in having a big set of information to
00:10:31
draw from and also knowledge about where
00:10:33
to find things where to find information
00:10:36
that would be helpful in what I do for
00:10:39
my day job in maritime archaeology it's
00:10:41
a very similar thing we take the outside
00:10:44
view we have a very disparate
00:10:45
information we have maybe a few
00:10:48
artifacts maybe not even maybe some
00:10:50
historical sources maybe someone
00:10:53
stumbled upon something and just have
00:10:54
rumors historical studies ethnographic
00:10:57
studies and we have to take all this
00:10:59
information and try to piece it together
00:11:00
to come up with a solution to or
00:11:03
explanation of what we're finding and
00:11:06
it's very similar in forecasting I kind
00:11:08
of approach it the same way where I'll
00:11:10
take very disparate information I won't
00:11:14
take the big huge pictures that are
00:11:16
thrust in your face I'll take you
00:11:18
outside of use and use some of the same
00:11:19
tools that I use at work I'm a huge
00:11:21
proponent of using GIS mapping of
00:11:24
different issues to try to to get like
00:11:26
say it's a question that has a
00:11:27
geographical perspective I'll make my
00:11:30
own GIS program and basically you know
00:11:34
put a model together that has all the
00:11:36
contributory information from different
00:11:38
press sources different mapping where
00:11:41
different roads are communication routes
00:11:42
transportation routes and it's the same
00:11:44
kind of thing I would do in archaeology
00:11:46
I just apply it to the future and
00:11:48
forecasting as opposed to applying it to
00:11:50
the past I don't trust my gut so usually
00:11:53
my first impression is right about half
00:11:56
the time so in order to do well in a
00:11:59
forecasting tournament you have to dig
00:12:01
deeper into the sources if you're
00:12:04
predicting elections for example you
00:12:07
know I'm find out you know what happened
00:12:09
in the past you know which parties were
00:12:14
likely to win and under what
00:12:15
circumstances
00:12:17
do your research about the electoral
00:12:20
system sometimes you know the electoral
00:12:23
system could lead to a counterintuitive
00:12:25
result you know a party could do very
00:12:28
well in the polls and still lose the
00:12:29
election so it's just really a matter of
00:12:33
doing your homework trying to understand
00:12:36
the situation the history behind the
00:12:42
event and and really working hard at it
00:12:47
and I would say a second reason would be
00:12:52
my sense of history so I think that
00:12:55
comes in very handy because most people
00:12:59
when they would forecast whatever event
00:13:01
it is would really look to the recent
00:13:03
past draw on their own memories as well
00:13:06
as maybe you know a Google search of you
00:13:09
know the recent news events but I think
00:13:12
having a sense of history that stretches
00:13:14
back far can give one a sense of mmm
00:13:22
possibilities which you know the recent
00:13:24
past might not indicate I think that I
00:13:27
make connections for instance I couldn't
00:13:33
figure out why Putin was in Syria none
00:13:35
of the explanations that people gave the
00:13:38
Talking Heads made any sense to me
00:13:40
Syria does not a join Russia the Ukraine
00:13:44
does Syria is of no strategic importance
00:13:48
to Putin
00:13:49
why is he in Syria there was just no
00:13:52
reason all of this business about how he
00:13:55
couldn't let one of his henchmen down or
00:13:57
else all the other henchmen would feel
00:13:58
insecure just didn't buy that I wanted
00:14:02
to make sense of that and all of a
00:14:03
sudden I remembered something and that
00:14:06
is that Putin is building up the Russian
00:14:10
fleet in the Mediterranean but they
00:14:13
don't have a home port in the
00:14:15
Mediterranean so they have to go all the
00:14:17
way back to Russia and the Black Sea to
00:14:20
get a home port well where is Putin
00:14:23
landing his troops next to the port of
00:14:26
Latakia which is a major port on the
00:14:29
Mediterranean
00:14:31
perhaps what Putin wants is a deal with
00:14:35
Assad for a 50-year lease on the port of
00:14:38
Latakia now that to me makes sense that
00:14:42
would explain why Putin is in Syria it
00:14:44
accomplishes a strategic thing for him
00:14:46
it gives him a port for his
00:14:48
Mediterranean fleet so I think that's
00:14:52
probably part of the reason why I was
00:14:55
successful I feel uncomfortable saying
00:14:58
successful because I don't feel
00:15:00
successful you tell me I was successful
00:15:02
I don't feel successful that's part of
00:15:05
the modus operandi that I go through
00:15:07
it's like I see disparate things and I
00:15:10
try to make sense of them for instance
00:15:12
another thing when the third Kim took
00:15:16
over North Korea all of a sudden there
00:15:20
was a huge round of executions including
00:15:22
his own uncle he blew his uncle up with
00:15:25
it
00:15:25
aircraft anti-aircraft gunner things
00:15:29
were then fairly quiet and then another
00:15:31
big round of executions I couldn't
00:15:35
figure out why he was executing all
00:15:37
these people I contacted somebody who
00:15:41
was an expert in North Korean relations
00:15:42
and I said why all the executions he
00:15:44
said well they've been more than you
00:15:45
know they just not publicized have been
00:15:47
two big rounds I thought what's going on
00:15:50
here and then I saw something that said
00:15:53
Kim has not yet traveled outside of
00:15:56
North Korea not even to Beijing to visit
00:15:59
his sponsors we don't know when Kim's
00:16:02
going to travel outside of North Korea
00:16:05
all of a sudden you put the two together
00:16:07
perhaps the guy's afraid that if he
00:16:09
leaves the country he won't be able to
00:16:11
come back that there'll be a coup in his
00:16:14
absence and he's taking care of
00:16:16
everybody who might be involved in such
00:16:18
a coup and in fact he's now due to go to
00:16:21
Moscow I think in December so again
00:16:24
trying to make sense out of something
00:16:26
just looking around at where how things
00:16:31
might fit together to make things make
00:16:33
more sense than they would otherwise the
00:16:35
tournament taught me that and in a very
00:16:39
humbling way that I knew a lot less
00:16:41
about the world than I thought I knew
00:16:43
and so that to begin with sparked my
00:16:47
thirst for knowledge and information it
00:16:51
also gave me a perspective about
00:16:53
risk-taking because in my first year I
00:16:55
was not a super forecaster I had to earn
00:16:58
that status and I was involved in in a
00:17:01
marketplace which involved essentially
00:17:04
trading and I've learned a lot about
00:17:07
risk-taking and how much I was either
00:17:11
risk-averse or overly overly risky
00:17:15
learning about things from a perspective
00:17:18
of other information has been very
00:17:22
important but once I became a super
00:17:25
forecaster in particular what was
00:17:27
helpful to me was a perspective of
00:17:29
others the sharing of information and
00:17:32
the ability to test your own beliefs and
00:17:36
hypotheses for bias by taking into
00:17:39
account what others have said or
00:17:42
analysed as well as information that
00:17:45
contradicts your own has been very
00:17:47
valuable to me
00:17:48
I was very steady I I answered as many
00:17:52
questions as I could
00:17:54
I was consistent I updated I didn't let
00:17:57
anything get forgotten and I approached
00:18:01
it like I was taking a test at school
00:18:04
come in and maybe I didn't even study so
00:18:07
but hey it's it's it's not an essay test
00:18:11
so the choices are very clear it's it's
00:18:14
it's it's numbers and and I would read
00:18:20
the question carefully and analyze
00:18:24
whether whether this would hinge on on a
00:18:26
single factor or are many many factors
00:18:29
if it was something that I had a clear
00:18:33
gut feeling right off the bat then I
00:18:37
would decide how much to back off from
00:18:41
that extreme and if it was something
00:18:43
where I was really undecided at the
00:18:46
50/50 mark I would force myself to a
00:18:50
choice because 5050 is is a is never
00:18:53
quite the right answer one of my hobbies
00:18:56
is oil painting
00:18:57
so and I and also when I've been taking
00:19:00
the test they tell me I'm visuospatial
00:19:03
so I I think that may have made a
00:19:06
difference getting the big picture and
00:19:10
knowing when to zoom in to the details
00:19:13
you know backing off and then zooming in
00:19:17
and not getting distracted by the wrong
00:19:20
details so I've learned in painting that
00:19:23
you have a big canvas you know you make
00:19:27
broad strokes but somehow sometimes just
00:19:30
the tiny nuance here are there changes
00:19:33
the way the whole painting looks and I
00:19:37
think that's true in world events too
00:19:39
and why you need a lot of different
00:19:41
perspectives so that I love my news
00:19:46
sources I have my favorite ones but when
00:19:48
you get a variety and when you hear
00:19:51
feedback from other forecasters it
00:19:53
really helps it's painfully clear that
00:19:56
the more the wiser and more thoughtful
00:20:03
and rational you are the better you're
00:20:05
going to do and so I'm afraid you know
00:20:08
that it's kind of the same people who
00:20:13
would do better a lot of different
00:20:15
things that require mental effort are
00:20:17
the ones that rise to the top in
00:20:20
forecasting
00:20:21
I think what's special about forecasting
00:20:25
is that if you're a really natural
00:20:30
researcher that can get you quite far
00:20:34
and I just completely admire and am
00:20:45
grateful to those of us who are so good
00:20:49
at it and who have you know far-flung
00:20:52
connections to decision centres in the
00:20:58
world it's I'm sort of a free rider in
00:21:04
that way
00:21:05
so there are two things that are really
00:21:07
important and
00:21:09
so I come from one side of that I'm a
00:21:12
I'm a person who was a natural for
00:21:16
computer programming for example but not
00:21:19
a natural for writing or researching
00:21:26
okay well along with not trusting my own
00:21:28
gut feelings my first impressions I
00:21:31
don't trust the gut feelings and of
00:21:33
experts and pundits I try to when ever
00:21:38
possible to discover understand what
00:21:42
assumptions underlie their forecasts
00:21:45
sometimes they're explicit about this
00:21:47
and other times they are and they don't
00:21:49
really they'll say you know this event
00:21:51
is likely but not really explained and
00:21:53
if I keep doing my research and find
00:21:56
other pundits you say a certain event is
00:21:58
likely without you know much explanation
00:22:00
at all then that's a signal for me to
00:22:03
really do some deep research and and try
00:22:07
to even if I agree with that assumption
00:22:10
you know sometimes I'll just try to play
00:22:12
the devil's advocate and say well let's
00:22:15
pretend that this silly outcome that's
00:22:17
unlikely to happen you know actually
00:22:19
could happen and kind of play that out
00:22:22
and sometimes you know I'll go back to
00:22:24
my original estimate
00:22:25
yeah that's silly it's not gonna happen
00:22:27
but you know often is the case that the
00:22:30
silly outcome is becomes the more likely
00:22:33
one so that's that's one thing I do and
00:22:39
others I try to discover or to find out
00:22:43
you know who has the power in a certain
00:22:45
situation who who is the decider in the
00:22:50
words of george w bush and that's one
00:22:52
that keeps coming back all the time and
00:22:56
also you know in every forecast you want
00:22:59
to you know write down a list or in your
00:23:03
head you know of all the constraints
00:23:05
that would pretend prevent a certain
00:23:07
event from occurring or slow it down you
00:23:12
know make an event you know and a likely
00:23:15
event take longer to occur than you know
00:23:17
what was expected before so those are
00:23:21
just a few things the first one was
00:23:23
there there's almost no but I'm I tell
00:23:26
you there's no nothing is certain that's
00:23:29
not entirely sure there's I'm sure
00:23:30
there's certain things that are served
00:23:31
but for the most part nothing is certain
00:23:34
the other thing was don't trust one
00:23:37
source of information
00:23:39
but then don't discard all sources just
00:23:41
because you think it's not valid when
00:23:44
you're whenever you're reading a source
00:23:45
of information if you have just one
00:23:46
single source and he's the only one
00:23:48
saying it I didn't
00:23:51
sometimes I wouldn't dismiss it but I
00:23:53
would not trust it I'd look for
00:23:55
something that would you know contribute
00:23:57
to it or just help verify it and that
00:24:00
can even be one of these sources where
00:24:02
you know you see it on the internet you
00:24:03
think wow that's really dubious these
00:24:04
guys are you know pretty crazy but
00:24:07
somewhere in there if you read through
00:24:08
the whole thing there might be one
00:24:09
sentence which collaborates what's going
00:24:12
on in that other source that was a
00:24:13
single source and I was like okay now
00:24:14
I've got two independent sources so and
00:24:17
you can start to build on that so one of
00:24:20
the things I would ever say is don't
00:24:21
rely on the one don't discard sources
00:24:24
just because you may think they're a bit
00:24:25
out there and there are quite a few
00:24:28
sources that are out there I think that
00:24:30
maybe the big thing is just to be aware
00:24:33
of what you know what you can be certain
00:24:35
of and what you're not sure of to know
00:24:40
the difference between subjective and
00:24:42
objective knowledge what things can be
00:24:45
measured and what things are a matter of
00:24:47
opinion look at history look at patterns
00:24:51
look at base rates and then figure out
00:24:54
whether your circumstance is similar or
00:24:57
different
00:24:59
I mean what's what was interesting for
00:25:02
me is that most of the things I was
00:25:04
forecasting on I knew nothing about
00:25:06
going in finance questions for instance
00:25:09
I I never pay attention to finance but
00:25:15
when you're forecasting on world
00:25:17
currencies you can always find graphs
00:25:19
historical charts and I look at them and
00:25:22
I see the patterns and I say why is it
00:25:25
different now you know are we operating
00:25:28
between the same extremes or will it be
00:25:30
new and usually I've gone with stay with
00:25:34
that stay with the trends so I didn't
00:25:36
let myself get intimidated by any
00:25:39
questions and I tried everything and as
00:25:42
bill mentioned some of the things where
00:25:44
I
00:25:44
wellactually was most hesitant on topics
00:25:48
that I might have knowledge on say on
00:25:50
Ebola
00:25:51
because suddenly I dunno I'm actually
00:25:54
more self-conscious about things where I
00:25:57
might miss something that I should know
00:25:59
one easy to follow
00:26:01
tip is do it every day and every
00:26:07
question that you're forecasting have a
00:26:08
look every day to see if anything has
00:26:10
changed you what point things do change
00:26:17
it significantly tends to come as a
00:26:20
surprise and if you missed that day well
00:26:22
you blew that forecast for that day the
00:26:28
my approach is to try to think through
00:26:32
causal mechanisms and and try to think
00:26:37
of all the influential factors that I
00:26:38
can think of and and try to visualize
00:26:41
scenarios and and balance probabilities
00:26:44
among scenarios things like that this is
00:26:47
a little different than what I think
00:26:49
most of us are led to do by the training
00:26:53
we've had which is it's not that the
00:26:56
training doesn't cover probability and
00:26:57
that's critical but it's to look at the
00:27:03
categories of the questions we're
00:27:06
forecasting and try to find parallels in
00:27:09
the past and look for rates of
00:27:12
occurrence and try to use those rates I
00:27:16
I think that's helpful as a starting
00:27:20
place but it's not going to get you a
00:27:23
very good score if that's all you do and
00:27:27
I think the difference between you know
00:27:35
sort of phoning it in and and getting a
00:27:39
good score is understanding the
00:27:42
situation as well as you possibly can
00:27:43
and that's the what Phil has Phil
00:27:47
tetlock has called the inside view as
00:27:49
opposed to the outside view he
00:27:51
emphasizes the people neglect the
00:27:53
outside view but I think it's the inside
00:27:56
view that that really matters to get you
00:28:02
exceptional forecast that that actually
00:28:05
could add to give real value anyway
00:28:07
that's the inside view well the first
00:28:10
thing was to really explore my biases to
00:28:14
realize that I often would have a first
00:28:17
impulse about a question and I didn't
00:28:20
want to just go with that
00:28:21
sometimes it was right but I didn't want
00:28:23
to just go on impulse and secondly I
00:28:27
really learned to think about what my
00:28:31
strengths were to focus on questions
00:28:33
that I thought were in my area of
00:28:34
strength and in particular to focus on
00:28:38
questions that I thought were were being
00:28:41
answered wrong or you know where the the
00:28:43
general consensus was different from
00:28:46
what I thought it was because there I
00:28:48
could make the most difference in
00:28:50
shifting opinion or contributing
00:28:53
knowledge that might sway the crowd and
00:28:55
make the group as a whole more accurate
00:28:57
so I think those things were really
00:28:59
important a good friend of mine also
00:29:02
named Barbara Barbara McClintock advised
00:29:05
me over and over in our conversations
00:29:07
that I should always question my
00:29:11
fundamental assumptions my underlying
00:29:13
assumptions and I think that's something
00:29:16
that's been very important to me was
00:29:18
Barbara's advice for instance a
00:29:20
spacecraft just flew past Jupiter and
00:29:24
took a whole bunch of high-resolution
00:29:26
pictures of Jupiter I'm sorry Pluto it
00:29:29
took a whole bunch of high-resolution
00:29:31
pictures of Pluto and son of a gun this
00:29:34
4.5 billion year old planet is
00:29:37
geologically active and everybody is
00:29:40
absolutely floored the solar system is
00:29:44
4.5 billion years old the planets are
00:29:46
not supposed to be geologically active
00:29:48
when they're the size of Pluto why is
00:29:51
this so everybody's trying to figure out
00:29:53
why 4.5 billion year old Pluto is by
00:29:56
geologically active they're talking
00:29:59
about more missions to Pluto half
00:30:02
hundred million dollar missions half
00:30:04
billion dollar missions I'm certain that
00:30:07
there are all sorts of young planetary
00:30:09
scientists who are trying to figure out
00:30:10
how they can start their careers by
00:30:13
figuring out and making a name for
00:30:15
themselves
00:30:16
Pluto is geologically active well what's
00:30:20
the underlying assumption here the
00:30:22
underlying assumption is that Pluto's
00:30:23
4.5 billion years old
00:30:25
what happens if Pluto isn't 4.5 billion
00:30:28
years old what happens with Pluto's a
00:30:30
good deal younger then in fact you
00:30:33
expect it to be geologically active and
00:30:35
there's no inconsistency here at all the
00:30:39
inconsistency is that your fundamental
00:30:41
assumption that Pluto is 4.5 billion
00:30:44
years old was incorrect so I'd advise
00:30:46
those young planetary scientists who are
00:30:49
thinking about trying to explain how a
00:30:51
4.5 billion year old celestial body
00:30:54
could be geologically active to question
00:30:57
whether or not it's really 4.5 billion
00:30:59
years old there's another something else
00:31:05
that gets a lot more attention and
00:31:06
that's it ran for the last four to five
00:31:09
years the entire economy of the world
00:31:11
has been an uproar because all of the
00:31:14
major nations have slapped an embargo on
00:31:17
dealing with Iran because Iran is trying
00:31:19
to build a nuclear weapon and they had
00:31:23
to negotiate long and hard with Iran to
00:31:27
negotiate Iran away from a posture where
00:31:30
they could be capable of building the
00:31:32
nuclear weapon that they were trying to
00:31:34
build and this brought us a lot closer
00:31:38
to allies like Saudi Arabia to whom we
00:31:41
perhaps should not be close and it's
00:31:44
caused no end to trouble throughout the
00:31:47
entire world particularly the Middle
00:31:48
East which is troubled enough well
00:31:50
what's the unda been underlying
00:31:52
assumption here that Iran wants to build
00:31:55
a nuclear weapon they kept saying that
00:31:57
they didn't want to build a nuclear
00:31:59
weapon
00:32:00
why didn't we believe them well we have
00:32:03
x y&z reasons why we didn't believe them
00:32:06
but again it's a fundamental assumption
00:32:08
the entire foreign policy in the Middle
00:32:11
East of nearly the entire world was
00:32:14
dictated by the underlying assumption
00:32:17
that Iran wanted to build a nuclear
00:32:19
device I personally wouldn't want to
00:32:22
build a nuclear device I can't see why
00:32:24
Iran would want a nuclear device nuclear
00:32:26
bombs are so 20th century
00:32:29
we've moved on I happen to think Iran
00:32:32
didn't particularly want to build a
00:32:34
nuclear bomb
00:32:35
but everybody's fundamental assumption
00:32:38
was that it did and I would go out on a
00:32:43
limb and I'd say that even in spite of
00:32:45
all the acrimony that we've had over
00:32:48
Iran that it's quite possible that in 20
00:32:51
years Iran will be our closest ally in
00:32:54
the Middle East and the reason for that
00:32:57
is that Iran is the largest Persian
00:33:00
nation on earth the United States is the
00:33:02
second-largest Persian nation on earth
00:33:04
and Canada is the fourth largest Persian
00:33:06
nation on earth there's a natural
00:33:08
affinity there and as soon as that
00:33:10
embargo is lifted the Persians are not
00:33:13
going to start doing deals with the
00:33:14
British or the Chinese or the Russians
00:33:16
they're going to do business with their
00:33:18
Persian American friends so I see
00:33:21
business between Iran Persia and the
00:33:25
United States particularly the Persian
00:33:27
community building very quickly and I
00:33:29
think that could lead to good political
00:33:30
relations as well that would be a major
00:33:34
casualty of misunderstanding Iran's
00:33:38
intentions with nuclear weapons the
00:33:41
process evolved over time for me and one
00:33:45
of the things that is important to me is
00:33:47
to use historical information but not to
00:33:50
be over alliant on it if there is the
00:33:54
ability to compare against similar
00:33:56
situations in the past I'm very
00:34:00
cognizant of using that information but
00:34:02
only to the extent that it is truly
00:34:05
relevant to the current circumstances
00:34:09
beyond that frequently adjusting my own
00:34:14
forecasts based upon new information and
00:34:17
new perspectives of others is extremely
00:34:20
valuable it's very easy to take a
00:34:22
position and sit back and say that's it
00:34:26
however the real world is a very active
00:34:29
one and often things change and
00:34:33
forecasts should change based upon the
00:34:35
events and
00:34:37
the happenings of the world and I found
00:34:39
it's extremely useful to pay a lot of
00:34:41
attention to those events and frequently
00:34:46
re-examine my own forecasts and see if
00:34:49
they need to be updated so I really had
00:34:51
three general rules when I made
00:34:56
forecasts especially when I made my
00:34:59
first forecast on a new question or a
00:35:01
situation I'd never seen before so the
00:35:04
first thing I would say is to be a good
00:35:07
forecasters
00:35:08
you have to be you have to hold strong
00:35:11
opinions weekly and what I mean by that
00:35:14
is that you do all of your homework
00:35:17
upfront and you really present a good
00:35:20
case for why you feel a certain way on a
00:35:22
specific question and defending your
00:35:25
probability probability estimate at that
00:35:27
point but when you recognize that
00:35:30
somebody else has more information or
00:35:32
better information than you having the
00:35:36
ability to change your mind swiftly to
00:35:39
that new forecast I think that was one
00:35:42
of the keys to why I was able to you
00:35:45
know improve my performance over time I
00:35:48
think many people really get stuck on
00:35:51
their initial forecasts and they fall in
00:35:53
love with it and wind up becoming going
00:35:56
into a defensive Crouch when it comes to
00:35:58
defending their opinion whereas I was on
00:36:02
my team sort of notorious for being able
00:36:04
to switch from one very hard position
00:36:09
all the way to the opposite when I saw
00:36:11
new information and come up that led me
00:36:13
to do that
00:36:14
the second thing is to be willing to put
00:36:19
in the homework and to put in the effort
00:36:21
to understand a situation go beyond the
00:36:25
Wikipedia page and to go beyond the
00:36:28
initial couple of news articles and to
00:36:30
really try to understand you know what
00:36:32
is it what is it about Boko Haram and
00:36:34
their composition that you know is
00:36:37
allowing them to make the kinds of
00:36:39
military gains that their that we're
00:36:41
seeing and how fragile or stable are
00:36:43
those military gains compared to other
00:36:45
like organizations and to you know not
00:36:49
just to say Oh
00:36:50
Boko Haram looks like every other
00:36:52
organization that I've encountered to
00:36:54
this point but to say to really try to
00:36:56
understand what's going on there and to
00:36:58
contextualize that appropriately and I
00:37:02
think the the third thing is to have in
00:37:07
your mind the kinds of psychological and
00:37:10
cognitive biases that you could
00:37:11
potentially fall prey to and to be able
00:37:15
to overcome those by you know listening
00:37:19
to your teammates by self critique and
00:37:22
to recognize that you know whatever
00:37:27
forecasts and argument that you're
00:37:31
putting forward now you should be able
00:37:33
to alter it or change it or improve it
00:37:36
in light of new information so you know
00:37:39
keeping in mind the cognitive biases
00:37:41
that really impact and influence and
00:37:45
reduce our accuracy and to consciously
00:37:49
try to devise them
00:37:54
I think the tournament was great
00:37:57
you need to practice so you need to have
00:38:01
a lot of questions you need to have
00:38:03
feedback you need to know where where
00:38:06
you're good and where you're messing up
00:38:09
if you have a team if you have other
00:38:11
people that you're in tournament either
00:38:14
in competition or in collaboration
00:38:15
either way you listen to them you get
00:38:18
feedback and you will you will improve I
00:38:22
think that one of the things that would
00:38:25
be extremely useful would be to
00:38:26
participate in good judgments open
00:38:28
forecasting competition because you can
00:38:31
test your own abilities and learn about
00:38:34
the things that that both limit your
00:38:37
ability to forecast and make you a
00:38:40
better forecaster and through that
00:38:44
process become somebody who is more
00:38:49
adept at forecasting the good judgment
00:38:55
incorporated itself has put together
00:38:58
various training methods which have been
00:39:01
very useful to me as a super forecaster
00:39:03
and I'm sure to others but as an
00:39:06
individual just forecasting and and
00:39:11
using that forecasting experience can be
00:39:14
a very valuable thing to learn about
00:39:17
yourself and the process of forecasting
00:39:20
what and and what you do well and what
00:39:23
you do less well and then to seek ways
00:39:25
of improving upon those weaknesses I
00:39:28
think you could really do two things to
00:39:31
become a good forecaster and one one is
00:39:37
to read and not just read say Phil's
00:39:42
book super forecasting or Danny
00:39:44
Kahneman's book Thinking Fast and Slow
00:39:46
which I think many of the forecasters
00:39:48
who excelled in torment have read and
00:39:51
done I also think that you know reading
00:39:55
say histories like the guns of August or
00:39:59
books about bureaucracy by James Q
00:40:03
Wilson and others really improves your
00:40:06
your knowledge base
00:40:08
and gives you a better context for
00:40:10
looking at situations and being able to
00:40:13
figure out whether there are any
00:40:14
commonalities or patterns there that you
00:40:17
can pick up on and that could
00:40:18
potentially apply to future situations
00:40:20
that look like that so being well-read
00:40:23
is I think a contributor to forecasting
00:40:28
success the second thing is to just to
00:40:31
practice don't get hung up on wrong
00:40:35
forecasts that you make initially you
00:40:37
know don't don't get overly you know
00:40:39
sensitive to those results but to
00:40:44
approach forecasting as something that
00:40:46
you can improve on and have that growth
00:40:49
mindset when it comes to making better
00:40:53
forecasts and so from there you're able
00:40:56
to say okay I start at some baseline
00:40:58
level I got some feedback tells me I
00:41:00
should improve in X Y & Z and to really
00:41:02
make a concerted effort to do that
00:41:03
through practice well one of the things
00:41:05
I would recommend would be to write down
00:41:07
a list of all the things that would that
00:41:12
you should be on the lookout for that
00:41:14
might cause you to change your mind so
00:41:17
after you've done your homework you've
00:41:19
thought deeply about you know the event
00:41:21
and the likelihood of of it occurring
00:41:25
write down a list of of things that you
00:41:30
know would happen leading up to the
00:41:33
event or signals markers whatever that
00:41:38
would cause you to change your mind and
00:41:40
and the reason why I say write it down
00:41:42
is because it's because you know it's
00:41:48
very hard to change your mind and if you
00:41:50
write it down it might be help you you
00:41:54
know not try to to fudge things well you
00:41:58
know if it's if it's just in the back of
00:42:00
your head you know you mM it becomes a
00:42:03
little bit more difficult to to go
00:42:05
against you know your original original
00:42:11
conclusion because I mean you put a lot
00:42:13
of work into this and you don't want to
00:42:15
go back on it and
00:42:17
there's an actual tendency not to want
00:42:20
to you know waffle or be too wishy-washy
00:42:23
but you know having it down on paper
00:42:24
will just show you well I'm not being
00:42:26
wishy-washy you know events have changed
00:42:30
things have changed and maybe my initial
00:42:35
conclusion was wrong I would say the
00:42:38
third thing would be to play devil's
00:42:40
advocate
00:42:42
after immediately after you've you know
00:42:44
come to a conclusion and made a forecast
00:42:47
seek out information that would prove
00:42:49
you wrong you know if anything you know
00:42:54
just to cover all your bases
00:42:56
I'd say my fourth piece of advice would
00:42:58
be to seek a wide variety of sources
00:43:03
even weak and biased sources because
00:43:07
sometimes you can find interest in
00:43:08
information in very flawed sources but
00:43:12
that'll take some work on your part to
00:43:14
determine you know which part of these
00:43:19
you know particular reports are worth
00:43:21
you know holding on to and which part
00:43:23
should be discarded it's not an easy
00:43:25
task but don't just you know casually
00:43:29
disregard even even bad sources because
00:43:32
you never know from four years as a as a
00:43:35
forecaster in the gjp tournament I can
00:43:39
say that another piece of advice I would
00:43:42
give is to assume that things would take
00:43:45
longer than your initial impression
00:43:48
that's a rule of thumb because it
00:43:51
happens consistently time and time again
00:43:53
that whatever it is a diplomatic
00:43:57
initiative or or or what-have-you
00:44:02
events often will take a lot longer than
00:44:05
then what what you think you know they
00:44:08
will take or what
00:44:09
officials you know the timetable of of
00:44:12
officials
00:44:14
and and that's just been my experience
00:44:18
from doing this for a long time final
00:44:21
piece of advice would be embrace
00:44:24
uncertainty don't get too hung up on
00:44:26
getting things correct you're never
00:44:29
gonna get everything correct you're
00:44:31
going to get a lot of things wrong I
00:44:32
would focus more on assigning viable or
00:44:40
realistic probabilities to events just
00:44:43
think of it in terms of you know if I
00:44:45
assign a 70% probability to a particular
00:44:48
event you know I'm gonna be wrong
00:44:50
three out of ten times you know four so
00:44:56
if you if you work on being well
00:45:00
calibrated instead of trying to get
00:45:02
everything right I think that's probably
00:45:04
a more fruitful endeavor well every year
00:45:06
in the tournament I kept a running list
00:45:10
of what questions I'd answered when I
00:45:12
first answered them how I Hearst
00:45:14
answered them and it was really helpful
00:45:16
to see how my opinion changed as new
00:45:18
information came along so I was you know
00:45:21
kind of keeping myself honest about what
00:45:23
was my original opinion and what was
00:45:25
making me change my mind and I think
00:45:28
that was really helpful because I
00:45:29
learned a lot about what kind of pits I
00:45:32
might fall into you know if I get over
00:45:34
optimistic about suddenly there's a lot
00:45:36
of news on a certain issue I might have
00:45:39
a tendency to think that something was
00:45:40
more likely to happen whereas maybe
00:45:42
wasn't really more likely to happen so
00:45:44
keeping track of my predictions was very
00:45:48
helpful I thought it was really
00:45:49
important that I was at forecasting
00:45:51
anonymously at first it made it more I
00:45:55
made me more confident about making
00:45:58
predictions without worrying about their
00:46:00
effect on my reputation and after all
00:46:03
you know I was in this to get better
00:46:06
about forecasting so I wanted to have
00:46:08
the ability to take to make bets or you
00:46:12
know answer questions in a way that I
00:46:13
thought was correct without worrying
00:46:15
about how that might be seen somewhere
00:46:17
else and that really built my confidence
00:46:19
and actually through being right then
00:46:22
frequently too it built up my confidence
00:46:25
to the point that I was willing to
00:46:27
who I was and then got to experience the
00:46:30
whole other dynamic of people knowing I
00:46:33
was an expert wondering what I thought
00:46:35
about things and then having to be
00:46:36
cautious about not relying too much on
00:46:38
my expert knowledge but still thinking
00:46:40
about the wisdom of the crowd so I think
00:46:42
the anonymity is really important and
00:46:45
also realizing that the magic of this
00:46:48
whole study is in revealing that
00:46:51
everybody has important information and
00:46:54
at different times different people's
00:46:56
information will be really valid and
00:46:58
especially important and and finding
00:47:01
that right balance between being
00:47:02
confident in your own views and and
00:47:03
being open to the views of others become
00:47:07
comfortable with researching problems
00:47:11
have a great interest in what you're
00:47:12
forecasting don't hesitate to dig right
00:47:17
in to you know study something new just
00:47:23
get into it
00:47:27
don't don't neglect if you have any bent
00:47:31
at all for statistics and probability
00:47:33
bring it into play you know but that's
00:47:35
kind of obvious anyone who has that
00:47:38
choice I guess actually that the this
00:47:45
might come up in your last question I
00:47:47
think you forget but part of the muscle
00:47:52
you want is the muscle you want to
00:47:54
strengthen intuitively is the muscle for
00:47:59
balancing scenarios against each other
00:48:02
and sort of averaging out probabilities
00:48:05
intuitively weighting them properly
00:48:08
things like that balancing things
00:48:11
against each other intuitively that are
00:48:12
things that are pointing in opposite
00:48:14
directions this is this is something you
00:48:17
get better at and feel easier about by
00:48:19
practicing I recommend that when you
00:48:21
come to a new field about which you know
00:48:23
nothing that you browse the literature
00:48:27
talk to people if you can attend
00:48:31
lectures if you can and listen for the
00:48:34
totem words the words which either
00:48:36
consciously or unconsciously
00:48:39
people in the field
00:48:41
give special meeting and I find that if
00:48:44
you can figure out what those totem
00:48:46
words are and then go away and figure
00:48:49
out what they mean you'll find out at
00:48:53
least one of the things that's very
00:48:55
important to people in the field that
00:48:58
they don't say is important to them
00:49:00
necessarily but they give it importance
00:49:02
by continually using these totem words
00:49:04
or continually emphasizing them for
00:49:08
instance last week I went to Grand
00:49:09
Rounds at our Regional Hospital and
00:49:12
there was a cardiologist giving a talk
00:49:14
and he was giving a talk on the
00:49:17
pharmaceutical management of cardiac
00:49:20
patients who had circulatory problems
00:49:23
problems with blockages problems with
00:49:26
leakages he would go through a series of
00:49:31
example patients this patient comes in
00:49:34
with these symptoms what do you do
00:49:36
we immediately stent this patient how do
00:49:39
we drug this patient for the stent we're
00:49:42
gonna use this compound and we're gonna
00:49:43
use this compound but we're gonna stay
00:49:45
away from that compound because it can
00:49:47
cause bleeding and we don't want
00:49:49
bleeding now this patient has got a
00:49:52
partially occluded artery and is in a
00:49:55
lot of pain we're going to give this
00:49:57
patient warfarin because we need to
00:50:00
dissolve that clot and prevent it from
00:50:02
growing however we're not going to give
00:50:05
this patient these other two drugs
00:50:07
because they can lead to bleeding even
00:50:10
worse than warfarin can lead to bleeding
00:50:13
we give combinations of these drugs
00:50:16
because it reduces bleeding bleeding
00:50:19
this bleeding that what I came away from
00:50:22
not knowing really that very much about
00:50:25
how to treat cardiac patients was that
00:50:28
if we can generalize from just this one
00:50:30
talk what's most important is to keep
00:50:34
the patient from bleeding no matter what
00:50:36
you do and so if I was given a new drug
00:50:40
and asked to predict whether or not it
00:50:42
was going to be a success the first
00:50:45
thing I'd look at its whether it causes
00:50:48
uncontrolled bleeding or not what the
00:50:49
likelihood is how it can be combined
00:50:52
with other things all from a standpoint
00:50:55
of does it cause bleeding now that
00:50:57
doesn't give you the answer to how that
00:50:59
new drug is going to do but it gives you
00:51:01
a lead-in to it so that you can then
00:51:04
begin to understand how things are going
00:51:05
on
00:51:06
so I'd say look for the totem words when
00:51:08
you're reading an article look for the
00:51:11
words that the author of the article
00:51:12
seems to come back to over and over and
00:51:15
then start by figuring out what those
00:51:17
words mean trust yourself
00:51:20
but doubt yourself as well and always be
00:51:23
prepared to be devil's advocate to
00:51:25
yourself if you're working within a team
00:51:27
that's even better because most
00:51:30
forecasters do not have a problem being
00:51:31
devil's advocate or would call red
00:51:33
teaming and keep an open mind don't go
00:51:37
into something thinking you know exactly
00:51:39
what's going on it never seems to work
00:51:42
out that way just when you think you
00:51:44
know exactly what's going on it changes
00:51:46
no the main thing is just you know be
00:51:50
open to change in dr. tetlock spooky
00:51:52
mentions that we're in perpetual beta
00:51:54
that is a great way of thinking of it is
00:51:56
always be willing to change and learn
00:51:58
and just you know go with it a lot of
00:52:02
people get really kind of stuck in the
00:52:03
mud on certain ideas and that's that's
00:52:05
the quickest way to get it wrong
00:52:11
I've been very insecure all my life for
00:52:14
a variety of reasons and I never really
00:52:18
thought very much of my own opinions I
00:52:21
always figured well is the world crazy
00:52:23
or is it me and the answer was I must be
00:52:25
crazy
00:52:27
I think the good judgment project has
00:52:30
shown me that occasionally I'm right and
00:52:33
other people are wrong which is not
00:52:34
something that I expected going in I
00:52:36
expected to be wrong all the time and
00:52:38
curiously and I've seen this in Phil's
00:52:42
book and I've heard it from other people
00:52:43
the things about which I was most wrong
00:52:46
with the things I was supposed to know
00:52:48
something about namely the medical
00:52:50
questions so I think the good judgment
00:52:52
project has given me a little bit of
00:52:53
confidence in myself that I didn't have
00:52:55
before I went into it it's been huge and
00:53:00
for first off you get exposed to a lot
00:53:06
of wonderful people whom you tend to
00:53:09
maintain some degree of contact with
00:53:11
I've had a couple of of my teammates
00:53:17
visit me at my house one who's from
00:53:20
South Africa the other ones from Spain
00:53:22
it's true they were on Washington for
00:53:23
other reasons but that's an example and
00:53:27
I I'm glad to stay in touch with a lot
00:53:32
of the fork in fact there there have
00:53:34
been others as well from the DC area
00:53:37
anyways I'm glad to stay in touch with
00:53:39
these new friends that's that's see
00:53:43
that's a big thing and just as big is
00:53:49
there's nothing like reality focusing on
00:53:52
reality and how it works out and you
00:53:54
tried to see the future and maybe it
00:53:58
worked and maybe it didn't and you you
00:54:02
sort of mentally record it's there's
00:54:05
feedback going on that's it's extremely
00:54:08
good feedback for turning you into a
00:54:10
more rational person it's it makes you a
00:54:14
more balanced person a more rational
00:54:17
person
00:54:19
easier to live with perhaps I don't know
00:54:22
but it's all good
00:54:24
it's been very rewarding and this
00:54:29
doesn't mean that the forecaster
00:54:30
shouldn't also be rewarded in other ways
00:54:32
but this myself is rewarding and I'm and
00:54:36
I'm sure that the others feel much much
00:54:38
the same I think being a forecaster and
00:54:42
being a researcher are similar and
00:54:45
different I think that one of the great
00:54:49
things about being a forecaster is that
00:54:52
we really weren't constrained by any of
00:54:54
the kind of customs or sociology around
00:54:59
forecasting because we were the ones
00:55:01
coming up with that and so we were able
00:55:04
to explore and be very free in how we
00:55:08
approach forecasting whereas I think
00:55:11
research while you have some freedom and
00:55:15
hypotheses that you are tackling a lot
00:55:18
of the methodology and the structure
00:55:21
around writing papers and peer review
00:55:25
and submissions is very much rigid and
00:55:29
you must follow the the specific process
00:55:32
so I think that that highlights a major
00:55:35
difference in forecasting versus doing
00:55:41
the research and that the research seem
00:55:43
to be much more process oriented
00:55:46
although obviously there there is an
00:55:48
outcome component to it whereas when
00:55:50
when we were doing strictly forecasting
00:55:53
really people only cared that we were
00:55:56
able to come up with the right answers
00:55:58
rather than you know how exactly we we
00:56:01
got there so yeah they're different
00:56:05
they're different animals I use
00:56:07
different words when I talk to myself
00:56:09
now I've always been very analytic but
00:56:13
now I I start talking those percentages
00:56:16
to myself and and that's different
00:56:19
my wife and I have two different
00:56:21
philosophies of investment she has been
00:56:25
involved over the years as a partner and
00:56:27
an investment firm that does alternative
00:56:29
investments which
00:56:32
are allegedly less risky but by my
00:56:34
lights carry a risk that is sometimes
00:56:37
more than what I like I on the other
00:56:39
hand am much more risk averse and
00:56:42
although less risk-averse then I was
00:56:45
when I first started participating a
00:56:49
good generative project my professional
00:56:54
career has been oriented towards
00:56:56
reducing the risk that decision makers
00:56:59
have and perhaps I've taken that point
00:57:03
of view in my own approach to investing
00:57:06
which my wife feels is overly
00:57:09
conservative I think that that the
00:57:12
forecast thing as part of the good
00:57:15
judgment project has given me a window
00:57:17
into my own risk-taking perspective and
00:57:22
move me at least somewhat closer to my
00:57:24
wife's point of view and and I think
00:57:29
that that's a very useful thing in real
00:57:31
life as well as you know from the
00:57:34
standpoint of participating as a super
00:57:36
forecasting so in my work
00:57:40
this experience has been extremely
00:57:43
important I am now working on an article
00:57:46
about the rise of China that I would not
00:57:50
have written if it had not been from my
00:57:52
experience in the good judgment project
00:57:54
by having to constantly update my
00:57:57
knowledge about China's economic
00:57:58
capabilities and its military
00:58:00
capabilities in order to do well in this
00:58:02
game I kept up in a much finer green
00:58:05
level with things that were happening in
00:58:08
China and that have changed my forecast
00:58:13
about how international relations will
00:58:15
work over the next couple of decades so
00:58:18
I'm an international relations scholar
00:58:19
one of the most important things in the
00:58:21
international relations field right now
00:58:23
is this question of of does the United
00:58:26
States have peer competitors is China
00:58:28
one yet and this experience helped me
00:58:32
see that it is so it's really
00:58:34
dramatically changed my career and the
00:58:36
process of going through the
00:58:39
confidence-building process
00:58:41
of you know learning how to articulate
00:58:43
forecasts learning how to be how to
00:58:49
express them in probabilistic terms how
00:58:51
to be confident even if I'm not
00:58:52
completely sure has made me willing to
00:58:55
take the risk of going out on this
00:58:57
really controversial subject I'm not
00:58:59
sure that they have all they probably
00:59:01
should having practice at forecasting
00:59:06
geopolitical events maybe I should start
00:59:08
practicing forecasting events related to
00:59:11
my own personal life maybe that would
00:59:13
help me avoid being unpleasantly
00:59:16
surprised by my things but so I'm not
00:59:20
really sure how it's affected my
00:59:21
personal life other than having nice
00:59:23
dinnertime conversations with families
00:59:25
and friends like I said I go back and
00:59:27
forth between the different you know
00:59:29
venues of inquiry in archaeology and in
00:59:31
forecasting
00:59:32
so there has been times where one of the
00:59:34
things that forecasting told me is it's
00:59:36
great if you have a model but all
00:59:38
they're going to be a better model
00:59:39
always be a better model you might have
00:59:42
one that's showing you exactly what you
00:59:43
want but you know if you tweak it a
00:59:46
little bit or you say you find a new
00:59:47
dataset can always be that little bit
00:59:49
better and I've started to apply that to
00:59:51
some shipwreck hunts that I've been
00:59:52
doing where we have a model on how to
00:59:54
locate shipwrecks in areas of high
00:59:56
probability but it can always be better
00:59:59
so I've actually started tweaking some
01:00:00
of the models that I use at work to add
01:00:02
additional data sets maybe pulling
01:00:04
things from multidisciplinary studies
01:00:06
from geomorphology marine geomorphology
01:00:10
riverbed hydrological processes going
01:00:13
into shipwreck site formation and just
01:00:16
basically you know constantly tweaking
01:00:17
the models there's something the other
01:00:19
thing that I've tried to do as much as I
01:00:21
can is I have two young sons who I'm
01:00:25
trying to teach them how to approach
01:00:27
different issues you know not not so
01:00:29
much to get him in trouble at school for
01:00:30
questioning the teacher all the time
01:00:31
which is very possible but when they
01:00:34
answer a question say well is that it I
01:00:37
spoke to my son about a tree growing in
01:00:39
a field I said well how much do you
01:00:40
think that tree is gonna grow next year
01:00:42
and he just said oh well this I said
01:00:43
well I should say that he's like well
01:00:45
because it just looks like a well did
01:00:46
you think about how much was it gonna
01:00:48
rain next year how much fertilizer is
01:00:50
the farmer going to put in that field
01:00:51
without trees in the middle how often is
01:00:53
he going to plow it what
01:00:55
you know there's so many issues that
01:00:56
surround one issue that when you hear
01:00:58
one question there's usually you know 20
01:01:00
30 however many more questions that can
01:01:03
help you answer that question so I've
01:01:05
been trying to push it on them to kind
01:01:07
of think that way
01:01:25
you
