00:00:08
I first got introduced to the forecasting by a long time at Dartmouth Ben Valentino who is a
00:00:17
political scientist and he alerted me to an opportunity to forecast with the good
00:00:26
judgment project and this was right around the time when I was leaving Baghdad Iraq after my second tour there
00:00:34
and I was looking for some way of doing forecasting because I had been something
00:00:41
that I've been interested in for for a while and that's why been introduced me
00:00:45
to this idea and you know I the timing worked out as well I was able to come back to the United States and that was
00:00:54
right around the time the the first year of the tournament kicked off and I just
00:00:58
went from there well in my work on security a big part of the question is whose security might be in danger in the
00:01:08
coming years so secure this field of security studies inherently is about forecasting making predictions about how
00:01:15
actors might be vulnerable which strategies might help them and so forth so I got interested in the good judgment
00:01:22
project because I wanted to build these skills I've always had the interest and
00:01:26
when I first started my undergrad I started a bunch of things I was one of these constantly changing your majors
00:01:32
people and I changed between applied physics is where I started and then I went to International Studies but then
00:01:39
somehow transitioned to anthropology and archaeology but I always kept the interest up in International Studies so
00:01:46
it kind of went from wanting to study current culture as Moe was going on to studying past cultures and what was
00:01:52
going on there and that the constant you know urge to learn more about all societies kept me interested in
00:02:00
forecasting in what was going on in different countries and as I said well switching all the time I went to quite a
00:02:07
few different universities in quite a few different countries I was at school in the Stars beginning Utah followed by
00:02:15
Vermont and then Norway then back to Utah and I finished my doctorate in Scotland where I was there for seven
00:02:22
years and then I went to Ireland for ten years so you know constantly moving in different cultures different societies
00:02:27
different political system it was always something of great interest to me my professional life has
00:02:32
involved forecasting because that's what I do or have done to help other people
00:02:38
make better decisions hopefully but several years ago I attended a Penn engaging Minds event in New York City
00:02:48
and Phil tetlock and Bob MELAS were speaking about the good judgment project which was very intriguing to me and so I
00:02:58
inquired about joining as a forecaster and several months later when positions opened up I became a forecaster I think
00:03:08
I am one of those people who can look at things a lot of different ways and I'm
00:03:14
not the most adept at it among the forecasters but that's definitely my type and so I was a good fit but I was
00:03:24
also driven because I'm acutely aware of the trouble the world is in and I think
00:03:30
the the as much as they have a dark side us organizations like I are and the intelligence community in general that
00:03:41
need all the help they can get we are basically the good guys and so so there was plenty of you know driving force
00:03:54
behind my interest that goes back to when I was a younger man during the first Gulf War in 1991 I was I'm
00:04:05
interested in the military military history and a lot of people and pundits were making predictions about the
00:04:12
outcome the manner in which the war would be waged and decided as well as casualty counts and how long it would
00:04:21
take and basically everyone was wrong on the duration and the and the destructiveness well the cat and the
00:04:30
number of casualties suffered by coalition forces and that was interesting to me because there was one
00:04:37
historian I'd like to say trevor Dupree you got it right and I was interested in how he got it right like
00:04:45
and evidently he had a system in just a few years later during the 1994 midterm elections when Bill Clinton lost in a
00:04:58
landslide to the Republicans the Republicans took the Congress for the first time in 40 years after that defeat
00:05:06
basically every pundit on Earth was saying that Bill Clinton is a one-term president and being a staunch Democrat
00:05:14
and a Clinton supporter I was traumatized by that possibility and wondered whether they knew they whether
00:05:20
that they knew what they were talking about and at about that time I found a book written by another historian Alan
00:05:27
Jay Lippman um about predicting you know who would become the next president and evidently
00:05:34
he had a system where you didn't need to use polls you just look at history and
00:05:39
using his method I realized that actually Bill Clinton was in a pretty good shape his his chances weren't
00:05:47
guaranteed but you know even a year before the election I had a pretty good inkling that he had a good chance of
00:05:54
winning re-election and this was against what all the pundits were saying so based on those two events there were
00:06:00
probably other events as well that's why I became interested in forecasting basically the fact that everyone you
00:06:07
know got thing big things wrong and I want to know why and some people got big things right and I wanted to know why
00:06:14
and whether I could do as well as I could so ever since then I was I started you
00:06:20
know I'm my own forecasting you know the outcomes of Wars and elections with
00:06:25
mixed results but you know so but over 20 years so that's how that's how I
00:06:31
became interested I'm politically very interested in what's happening and I used to read the
00:06:38
538 blog that was done by nate silver he advertised for the good judgment project
00:06:44
i don't know where i saw the notice that you guys were putting together this
00:06:50
program it was on one of the standard outlets maybe Wall Street Journal maybe Bloomberg
00:06:56
but I read an article that you were putting together this forecasting tournament and that you were trying to
00:07:03
find out what it is makes a good forecaster and I feel like I owe the system for so many years I've been doing
00:07:14
research and I've relied on other people to contribute to that research and they
00:07:18
have selflessly and so I'm a chronic volunteer as well when somebody says I need your help to do my research I
00:07:26
volunteer for instance the New York Blood Center contacted me and said you're not allowed to give blood but
00:07:32
we've got this research program and blood donation will you come in and donate a pint of blood every month so we
00:07:39
can see what happens so I did that just to pay back the system so I'm paying
00:07:45
back the system by being in the good judgment project I think that intellectual curiosity and just a
00:07:56
willingness to put in the work and to think hard about specific problems and try to be as objective as possible is
00:08:05
what allowed me to make more accurate forecasts and others within the pool I wouldn't necessarily say that you know
00:08:14
what we did or what I did as an individual is the best possible at forecasting because they're you know can
00:08:21
be so many more improvements I I think in terms of how much you know about a specific subject you can always improve
00:08:28
your level of knowledge and I think you can always improve you know how you're
00:08:33
you're specifically viewing a situation how that situation could potentially
00:08:37
evolve so having a greater understanding of how humans interact with each other relate to each other the political
00:08:44
contours of specific situations and events those are all things that you can gain greater knowledge on and get better
00:08:49
at forecasting in so yeah yeah being open-minded and being open-minded enough to realize that you need to remain
00:08:57
open-minded I think are some of the keys to being you know for for my own forecasting performance well I think
00:09:06
there were a couple of factors one was that I am in the middle of my career now and I've spent a lot of time studying
00:09:11
exactly the kind of issues that the good judgment project was focusing on and so
00:09:18
I was familiar with trade issues I was familiar with national security issues the rise and fall of governments that
00:09:26
kind of thing so it really was right in my issue area also I I use a theory that
00:09:33
can be well applied in this kind of situation it's called structural realist
00:09:37
theory and it gives a sense of big recurring outcomes and international relations the rise and fall of great
00:09:47
powers the tendency for military conflict to occur competition and trade and so forth and so I had a kind of
00:09:56
broad understanding of international affairs that I could apply to particular situations and the beauty of this theory
00:10:03
was that it was it's really pre open states have a lot of choices about how they act and so I wasn't falling
00:10:12
into the trap of thinking that something was inevitable but instead was investigating which things were more
00:10:18
likely which things were less likely so this theory was really conducive to probabilistic forecasting and my
00:10:26
background knowledge was really helpful in having a big set of information to draw from and also knowledge about where
00:10:33
to find things where to find information that would be helpful in what I do for my day job in maritime archaeology it's
00:10:41
a very similar thing we take the outside view we have a very disparate information we have maybe a few
00:10:48
artifacts maybe not even maybe some historical sources maybe someone stumbled upon something and just have
00:10:54
rumors historical studies ethnographic studies and we have to take all this information and try to piece it together
00:11:00
to come up with a solution to or explanation of what we're finding and it's very similar in forecasting I kind
00:11:08
of approach it the same way where I'll take very disparate information I won't
00:11:14
take the big huge pictures that are thrust in your face I'll take you outside of use and use some of the same
00:11:19
tools that I use at work I'm a huge proponent of using GIS mapping of different issues to try to to get like
00:11:26
say it's a question that has a geographical perspective I'll make my own GIS program and basically you know
00:11:34
put a model together that has all the contributory information from different press sources different mapping where
00:11:41
different roads are communication routes transportation routes and it's the same
00:11:44
kind of thing I would do in archaeology I just apply it to the future and forecasting as opposed to applying it to
00:11:50
the past I don't trust my gut so usually my first impression is right about half
00:11:56
the time so in order to do well in a forecasting tournament you have to dig deeper into the sources if you're
00:12:04
predicting elections for example you know I'm find out you know what happened
00:12:09
in the past you know which parties were likely to win and under what circumstances
00:12:17
do your research about the electoral system sometimes you know the electoral system could lead to a counterintuitive
00:12:25
result you know a party could do very well in the polls and still lose the election so it's just really a matter of
00:12:33
doing your homework trying to understand the situation the history behind the event and and really working hard at it
00:12:47
and I would say a second reason would be my sense of history so I think that comes in very handy because most people
00:12:59
when they would forecast whatever event it is would really look to the recent past draw on their own memories as well
00:13:06
as maybe you know a Google search of you know the recent news events but I think
00:13:12
having a sense of history that stretches back far can give one a sense of mmm possibilities which you know the recent
00:13:24
past might not indicate I think that I make connections for instance I couldn't
00:13:33
figure out why Putin was in Syria none of the explanations that people gave the Talking Heads made any sense to me
00:13:40
Syria does not a join Russia the Ukraine does Syria is of no strategic importance
00:13:48
to Putin why is he in Syria there was just no reason all of this business about how he
00:13:55
couldn't let one of his henchmen down or else all the other henchmen would feel
00:13:58
insecure just didn't buy that I wanted to make sense of that and all of a sudden I remembered something and that
00:14:06
is that Putin is building up the Russian fleet in the Mediterranean but they don't have a home port in the
00:14:15
Mediterranean so they have to go all the way back to Russia and the Black Sea to
00:14:20
get a home port well where is Putin landing his troops next to the port of Latakia which is a major port on the
00:14:29
Mediterranean perhaps what Putin wants is a deal with Assad for a 50-year lease on the port of
00:14:38
Latakia now that to me makes sense that would explain why Putin is in Syria it accomplishes a strategic thing for him
00:14:46
it gives him a port for his Mediterranean fleet so I think that's probably part of the reason why I was
00:14:55
successful I feel uncomfortable saying successful because I don't feel successful you tell me I was successful
00:15:02
I don't feel successful that's part of the modus operandi that I go through
00:15:07
it's like I see disparate things and I try to make sense of them for instance
00:15:12
another thing when the third Kim took over North Korea all of a sudden there was a huge round of executions including
00:15:22
his own uncle he blew his uncle up with it aircraft anti-aircraft gunner things were then fairly quiet and then another
00:15:31
big round of executions I couldn't figure out why he was executing all these people I contacted somebody who
00:15:41
was an expert in North Korean relations and I said why all the executions he said well they've been more than you
00:15:45
know they just not publicized have been two big rounds I thought what's going on
00:15:50
here and then I saw something that said Kim has not yet traveled outside of North Korea not even to Beijing to visit
00:15:59
his sponsors we don't know when Kim's going to travel outside of North Korea
00:16:05
all of a sudden you put the two together perhaps the guy's afraid that if he
00:16:09
leaves the country he won't be able to come back that there'll be a coup in his
00:16:14
absence and he's taking care of everybody who might be involved in such a coup and in fact he's now due to go to
00:16:21
Moscow I think in December so again trying to make sense out of something just looking around at where how things
00:16:31
might fit together to make things make more sense than they would otherwise the tournament taught me that and in a very
00:16:39
humbling way that I knew a lot less about the world than I thought I knew and so that to begin with sparked my
00:16:47
thirst for knowledge and information it also gave me a perspective about risk-taking because in my first year I
00:16:55
was not a super forecaster I had to earn that status and I was involved in in a marketplace which involved essentially
00:17:04
trading and I've learned a lot about risk-taking and how much I was either risk-averse or overly overly risky
00:17:15
learning about things from a perspective of other information has been very important but once I became a super
00:17:25
forecaster in particular what was helpful to me was a perspective of others the sharing of information and
00:17:32
the ability to test your own beliefs and hypotheses for bias by taking into account what others have said or
00:17:42
analysed as well as information that contradicts your own has been very valuable to me
00:17:48
I was very steady I I answered as many questions as I could I was consistent I updated I didn't let
00:17:57
anything get forgotten and I approached it like I was taking a test at school come in and maybe I didn't even study so
00:18:07
but hey it's it's it's not an essay test so the choices are very clear it's it's
00:18:14
it's it's numbers and and I would read the question carefully and analyze
00:18:24
whether whether this would hinge on on a single factor or are many many factors if it was something that I had a clear
00:18:33
gut feeling right off the bat then I would decide how much to back off from that extreme and if it was something
00:18:43
where I was really undecided at the 50/50 mark I would force myself to a choice because 5050 is is a is never
00:18:53
quite the right answer one of my hobbies is oil painting so and I and also when I've been taking
00:19:00
the test they tell me I'm visuospatial so I I think that may have made a difference getting the big picture and
00:19:10
knowing when to zoom in to the details you know backing off and then zooming in and not getting distracted by the wrong
00:19:20
details so I've learned in painting that you have a big canvas you know you make
00:19:27
broad strokes but somehow sometimes just the tiny nuance here are there changes the way the whole painting looks and I
00:19:37
think that's true in world events too and why you need a lot of different perspectives so that I love my news
00:19:46
sources I have my favorite ones but when you get a variety and when you hear feedback from other forecasters it
00:19:53
really helps it's painfully clear that the more the wiser and more thoughtful
00:20:03
and rational you are the better you're going to do and so I'm afraid you know
00:20:08
that it's kind of the same people who would do better a lot of different things that require mental effort are
00:20:17
the ones that rise to the top in forecasting I think what's special about forecasting
00:20:25
is that if you're a really natural researcher that can get you quite far and I just completely admire and am
00:20:45
grateful to those of us who are so good at it and who have you know far-flung connections to decision centres in the
00:20:58
world it's I'm sort of a free rider in that way so there are two things that are really
00:21:07
important and so I come from one side of that I'm a I'm a person who was a natural for
00:21:16
computer programming for example but not a natural for writing or researching okay well along with not trusting my own
00:21:28
gut feelings my first impressions I don't trust the gut feelings and of experts and pundits I try to when ever
00:21:38
possible to discover understand what assumptions underlie their forecasts sometimes they're explicit about this
00:21:47
and other times they are and they don't really they'll say you know this event
00:21:51
is likely but not really explained and if I keep doing my research and find other pundits you say a certain event is
00:21:58
likely without you know much explanation at all then that's a signal for me to
00:22:03
really do some deep research and and try to even if I agree with that assumption
00:22:10
you know sometimes I'll just try to play the devil's advocate and say well let's
00:22:15
pretend that this silly outcome that's unlikely to happen you know actually could happen and kind of play that out
00:22:22
and sometimes you know I'll go back to my original estimate yeah that's silly it's not gonna happen
00:22:27
but you know often is the case that the silly outcome is becomes the more likely
00:22:33
one so that's that's one thing I do and others I try to discover or to find out
00:22:43
you know who has the power in a certain situation who who is the decider in the words of george w bush and that's one
00:22:52
that keeps coming back all the time and also you know in every forecast you want
00:22:59
to you know write down a list or in your head you know of all the constraints that would pretend prevent a certain
00:23:07
event from occurring or slow it down you know make an event you know and a likely
00:23:15
event take longer to occur than you know what was expected before so those are just a few things the first one was
00:23:23
there there's almost no but I'm I tell you there's no nothing is certain that's
00:23:29
not entirely sure there's I'm sure there's certain things that are served
00:23:31
but for the most part nothing is certain the other thing was don't trust one
00:23:37
source of information but then don't discard all sources just because you think it's not valid when
00:23:44
you're whenever you're reading a source of information if you have just one
00:23:46
single source and he's the only one saying it I didn't sometimes I wouldn't dismiss it but I
00:23:53
would not trust it I'd look for something that would you know contribute to it or just help verify it and that
00:24:00
can even be one of these sources where you know you see it on the internet you think wow that's really dubious these
00:24:04
guys are you know pretty crazy but somewhere in there if you read through the whole thing there might be one
00:24:09
sentence which collaborates what's going on in that other source that was a single source and I was like okay now
00:24:14
I've got two independent sources so and you can start to build on that so one of
00:24:20
the things I would ever say is don't rely on the one don't discard sources
00:24:24
just because you may think they're a bit out there and there are quite a few
00:24:28
sources that are out there I think that maybe the big thing is just to be aware of what you know what you can be certain
00:24:35
of and what you're not sure of to know the difference between subjective and
00:24:42
objective knowledge what things can be measured and what things are a matter of opinion look at history look at patterns
00:24:51
look at base rates and then figure out whether your circumstance is similar or different
00:24:59
I mean what's what was interesting for me is that most of the things I was forecasting on I knew nothing about
00:25:06
going in finance questions for instance I I never pay attention to finance but when you're forecasting on world
00:25:17
currencies you can always find graphs historical charts and I look at them and I see the patterns and I say why is it
00:25:25
different now you know are we operating between the same extremes or will it be new and usually I've gone with stay with
00:25:34
that stay with the trends so I didn't let myself get intimidated by any questions and I tried everything and as
00:25:42
bill mentioned some of the things where I wellactually was most hesitant on topics
00:25:48
that I might have knowledge on say on Ebola because suddenly I dunno I'm actually
00:25:54
more self-conscious about things where I might miss something that I should know
00:25:59
one easy to follow tip is do it every day and every question that you're forecasting have a
00:26:08
look every day to see if anything has changed you what point things do change it significantly tends to come as a
00:26:20
surprise and if you missed that day well you blew that forecast for that day the
00:26:28
my approach is to try to think through causal mechanisms and and try to think of all the influential factors that I
00:26:38
can think of and and try to visualize scenarios and and balance probabilities among scenarios things like that this is
00:26:47
a little different than what I think most of us are led to do by the training we've had which is it's not that the
00:26:56
training doesn't cover probability and that's critical but it's to look at the
00:27:03
categories of the questions we're forecasting and try to find parallels in the past and look for rates of
00:27:12
occurrence and try to use those rates I I think that's helpful as a starting
00:27:20
place but it's not going to get you a very good score if that's all you do and
00:27:27
I think the difference between you know sort of phoning it in and and getting a good score is understanding the
00:27:42
situation as well as you possibly can and that's the what Phil has Phil tetlock has called the inside view as
00:27:49
opposed to the outside view he emphasizes the people neglect the outside view but I think it's the inside
00:27:56
view that that really matters to get you exceptional forecast that that actually
00:28:05
could add to give real value anyway that's the inside view well the first thing was to really explore my biases to
00:28:14
realize that I often would have a first impulse about a question and I didn't
00:28:20
want to just go with that sometimes it was right but I didn't want to just go on impulse and secondly I
00:28:27
really learned to think about what my strengths were to focus on questions that I thought were in my area of
00:28:34
strength and in particular to focus on questions that I thought were were being answered wrong or you know where the the
00:28:43
general consensus was different from what I thought it was because there I could make the most difference in
00:28:50
shifting opinion or contributing knowledge that might sway the crowd and make the group as a whole more accurate
00:28:57
so I think those things were really important a good friend of mine also named Barbara Barbara McClintock advised
00:29:05
me over and over in our conversations that I should always question my fundamental assumptions my underlying
00:29:13
assumptions and I think that's something that's been very important to me was
00:29:18
Barbara's advice for instance a spacecraft just flew past Jupiter and took a whole bunch of high-resolution
00:29:26
pictures of Jupiter I'm sorry Pluto it took a whole bunch of high-resolution
00:29:31
pictures of Pluto and son of a gun this 4.5 billion year old planet is geologically active and everybody is
00:29:40
absolutely floored the solar system is 4.5 billion years old the planets are not supposed to be geologically active
00:29:48
when they're the size of Pluto why is this so everybody's trying to figure out
00:29:53
why 4.5 billion year old Pluto is by geologically active they're talking about more missions to Pluto half
00:30:02
hundred million dollar missions half billion dollar missions I'm certain that
00:30:07
there are all sorts of young planetary scientists who are trying to figure out how they can start their careers by
00:30:13
figuring out and making a name for themselves Pluto is geologically active well what's
00:30:20
the underlying assumption here the underlying assumption is that Pluto's 4.5 billion years old
00:30:25
what happens if Pluto isn't 4.5 billion years old what happens with Pluto's a
00:30:30
good deal younger then in fact you expect it to be geologically active and there's no inconsistency here at all the
00:30:39
inconsistency is that your fundamental assumption that Pluto is 4.5 billion years old was incorrect so I'd advise
00:30:46
those young planetary scientists who are thinking about trying to explain how a 4.5 billion year old celestial body
00:30:54
could be geologically active to question whether or not it's really 4.5 billion
00:30:59
years old there's another something else that gets a lot more attention and that's it ran for the last four to five
00:31:09
years the entire economy of the world has been an uproar because all of the major nations have slapped an embargo on
00:31:17
dealing with Iran because Iran is trying to build a nuclear weapon and they had to negotiate long and hard with Iran to
00:31:27
negotiate Iran away from a posture where they could be capable of building the nuclear weapon that they were trying to
00:31:34
build and this brought us a lot closer to allies like Saudi Arabia to whom we perhaps should not be close and it's
00:31:44
caused no end to trouble throughout the entire world particularly the Middle East which is troubled enough well
00:31:50
what's the unda been underlying assumption here that Iran wants to build a nuclear weapon they kept saying that
00:31:57
they didn't want to build a nuclear weapon why didn't we believe them well we have
00:32:03
x y&z reasons why we didn't believe them but again it's a fundamental assumption
00:32:08
the entire foreign policy in the Middle East of nearly the entire world was dictated by the underlying assumption
00:32:17
that Iran wanted to build a nuclear device I personally wouldn't want to build a nuclear device I can't see why
00:32:24
Iran would want a nuclear device nuclear bombs are so 20th century we've moved on I happen to think Iran
00:32:32
didn't particularly want to build a nuclear bomb but everybody's fundamental assumption
00:32:38
was that it did and I would go out on a limb and I'd say that even in spite of
00:32:45
all the acrimony that we've had over Iran that it's quite possible that in 20
00:32:51
years Iran will be our closest ally in the Middle East and the reason for that is that Iran is the largest Persian
00:33:00
nation on earth the United States is the second-largest Persian nation on earth and Canada is the fourth largest Persian
00:33:06
nation on earth there's a natural affinity there and as soon as that embargo is lifted the Persians are not
00:33:13
going to start doing deals with the British or the Chinese or the Russians they're going to do business with their
00:33:18
Persian American friends so I see business between Iran Persia and the United States particularly the Persian
00:33:27
community building very quickly and I think that could lead to good political relations as well that would be a major
00:33:34
casualty of misunderstanding Iran's intentions with nuclear weapons the process evolved over time for me and one
00:33:45
of the things that is important to me is to use historical information but not to
00:33:50
be over alliant on it if there is the ability to compare against similar situations in the past I'm very
00:34:00
cognizant of using that information but only to the extent that it is truly relevant to the current circumstances
00:34:09
beyond that frequently adjusting my own forecasts based upon new information and
00:34:17
new perspectives of others is extremely valuable it's very easy to take a position and sit back and say that's it
00:34:26
however the real world is a very active one and often things change and forecasts should change based upon the
00:34:35
events and the happenings of the world and I found it's extremely useful to pay a lot of
00:34:41
attention to those events and frequently re-examine my own forecasts and see if they need to be updated so I really had
00:34:51
three general rules when I made forecasts especially when I made my first forecast on a new question or a
00:35:01
situation I'd never seen before so the first thing I would say is to be a good
00:35:07
forecasters you have to be you have to hold strong opinions weekly and what I mean by that
00:35:14
is that you do all of your homework upfront and you really present a good case for why you feel a certain way on a
00:35:22
specific question and defending your probability probability estimate at that point but when you recognize that
00:35:30
somebody else has more information or better information than you having the ability to change your mind swiftly to
00:35:39
that new forecast I think that was one of the keys to why I was able to you know improve my performance over time I
00:35:48
think many people really get stuck on their initial forecasts and they fall in love with it and wind up becoming going
00:35:56
into a defensive Crouch when it comes to defending their opinion whereas I was on
00:36:02
my team sort of notorious for being able to switch from one very hard position all the way to the opposite when I saw
00:36:11
new information and come up that led me to do that the second thing is to be willing to put
00:36:19
in the homework and to put in the effort to understand a situation go beyond the
00:36:25
Wikipedia page and to go beyond the initial couple of news articles and to really try to understand you know what
00:36:32
is it what is it about Boko Haram and their composition that you know is allowing them to make the kinds of
00:36:39
military gains that their that we're seeing and how fragile or stable are those military gains compared to other
00:36:45
like organizations and to you know not just to say Oh Boko Haram looks like every other
00:36:52
organization that I've encountered to this point but to say to really try to
00:36:56
understand what's going on there and to contextualize that appropriately and I
00:37:02
think the the third thing is to have in your mind the kinds of psychological and
00:37:10
cognitive biases that you could potentially fall prey to and to be able to overcome those by you know listening
00:37:19
to your teammates by self critique and to recognize that you know whatever forecasts and argument that you're
00:37:31
putting forward now you should be able to alter it or change it or improve it in light of new information so you know
00:37:39
keeping in mind the cognitive biases that really impact and influence and reduce our accuracy and to consciously
00:37:49
try to devise them I think the tournament was great you need to practice so you need to have
00:38:01
a lot of questions you need to have feedback you need to know where where you're good and where you're messing up
00:38:09
if you have a team if you have other people that you're in tournament either
00:38:14
in competition or in collaboration either way you listen to them you get feedback and you will you will improve I
00:38:22
think that one of the things that would be extremely useful would be to participate in good judgments open
00:38:28
forecasting competition because you can test your own abilities and learn about the things that that both limit your
00:38:37
ability to forecast and make you a better forecaster and through that process become somebody who is more
00:38:49
adept at forecasting the good judgment incorporated itself has put together various training methods which have been
00:39:01
very useful to me as a super forecaster and I'm sure to others but as an individual just forecasting and and
00:39:11
using that forecasting experience can be a very valuable thing to learn about yourself and the process of forecasting
00:39:20
what and and what you do well and what you do less well and then to seek ways of improving upon those weaknesses I
00:39:28
think you could really do two things to become a good forecaster and one one is to read and not just read say Phil's
00:39:42
book super forecasting or Danny Kahneman's book Thinking Fast and Slow which I think many of the forecasters
00:39:48
who excelled in torment have read and done I also think that you know reading say histories like the guns of August or
00:39:59
books about bureaucracy by James Q Wilson and others really improves your your knowledge base
00:40:08
and gives you a better context for looking at situations and being able to figure out whether there are any
00:40:14
commonalities or patterns there that you can pick up on and that could potentially apply to future situations
00:40:20
that look like that so being well-read is I think a contributor to forecasting success the second thing is to just to
00:40:31
practice don't get hung up on wrong forecasts that you make initially you know don't don't get overly you know
00:40:39
sensitive to those results but to approach forecasting as something that you can improve on and have that growth
00:40:49
mindset when it comes to making better forecasts and so from there you're able
00:40:56
to say okay I start at some baseline level I got some feedback tells me I should improve in X Y & Z and to really
00:41:02
make a concerted effort to do that through practice well one of the things I would recommend would be to write down
00:41:07
a list of all the things that would that you should be on the lookout for that might cause you to change your mind so
00:41:17
after you've done your homework you've thought deeply about you know the event
00:41:21
and the likelihood of of it occurring write down a list of of things that you know would happen leading up to the
00:41:33
event or signals markers whatever that would cause you to change your mind and and the reason why I say write it down
00:41:42
is because it's because you know it's very hard to change your mind and if you
00:41:50
write it down it might be help you you know not try to to fudge things well you know if it's if it's just in the back of
00:42:00
your head you know you mM it becomes a little bit more difficult to to go against you know your original original
00:42:11
conclusion because I mean you put a lot of work into this and you don't want to
00:42:15
go back on it and there's an actual tendency not to want to you know waffle or be too wishy-washy
00:42:23
but you know having it down on paper will just show you well I'm not being wishy-washy you know events have changed
00:42:30
things have changed and maybe my initial conclusion was wrong I would say the third thing would be to play devil's
00:42:40
advocate after immediately after you've you know come to a conclusion and made a forecast
00:42:47
seek out information that would prove you wrong you know if anything you know just to cover all your bases
00:42:56
I'd say my fourth piece of advice would be to seek a wide variety of sources
00:43:03
even weak and biased sources because sometimes you can find interest in information in very flawed sources but
00:43:12
that'll take some work on your part to determine you know which part of these
00:43:19
you know particular reports are worth you know holding on to and which part should be discarded it's not an easy
00:43:25
task but don't just you know casually disregard even even bad sources because
00:43:32
you never know from four years as a as a forecaster in the gjp tournament I can say that another piece of advice I would
00:43:42
give is to assume that things would take longer than your initial impression that's a rule of thumb because it
00:43:51
happens consistently time and time again that whatever it is a diplomatic initiative or or or what-have-you
00:44:02
events often will take a lot longer than then what what you think you know they will take or what
00:44:09
officials you know the timetable of of officials and and that's just been my experience
00:44:18
from doing this for a long time final piece of advice would be embrace uncertainty don't get too hung up on
00:44:26
getting things correct you're never gonna get everything correct you're going to get a lot of things wrong I
00:44:32
would focus more on assigning viable or realistic probabilities to events just think of it in terms of you know if I
00:44:45
assign a 70% probability to a particular event you know I'm gonna be wrong three out of ten times you know four so
00:44:56
if you if you work on being well calibrated instead of trying to get everything right I think that's probably
00:45:04
a more fruitful endeavor well every year in the tournament I kept a running list
00:45:10
of what questions I'd answered when I first answered them how I Hearst answered them and it was really helpful
00:45:16
to see how my opinion changed as new information came along so I was you know kind of keeping myself honest about what
00:45:23
was my original opinion and what was making me change my mind and I think that was really helpful because I
00:45:29
learned a lot about what kind of pits I might fall into you know if I get over optimistic about suddenly there's a lot
00:45:36
of news on a certain issue I might have a tendency to think that something was more likely to happen whereas maybe
00:45:42
wasn't really more likely to happen so keeping track of my predictions was very
00:45:48
helpful I thought it was really important that I was at forecasting anonymously at first it made it more I
00:45:55
made me more confident about making predictions without worrying about their effect on my reputation and after all
00:46:03
you know I was in this to get better about forecasting so I wanted to have the ability to take to make bets or you
00:46:12
know answer questions in a way that I thought was correct without worrying about how that might be seen somewhere
00:46:17
else and that really built my confidence and actually through being right then frequently too it built up my confidence
00:46:25
to the point that I was willing to who I was and then got to experience the whole other dynamic of people knowing I
00:46:33
was an expert wondering what I thought about things and then having to be cautious about not relying too much on
00:46:38
my expert knowledge but still thinking about the wisdom of the crowd so I think the anonymity is really important and
00:46:45
also realizing that the magic of this whole study is in revealing that everybody has important information and
00:46:54
at different times different people's information will be really valid and especially important and and finding
00:47:01
that right balance between being confident in your own views and and being open to the views of others become
00:47:07
comfortable with researching problems have a great interest in what you're forecasting don't hesitate to dig right
00:47:17
in to you know study something new just get into it don't don't neglect if you have any bent
00:47:31
at all for statistics and probability bring it into play you know but that's
00:47:35
kind of obvious anyone who has that choice I guess actually that the this might come up in your last question I
00:47:47
think you forget but part of the muscle you want is the muscle you want to strengthen intuitively is the muscle for
00:47:59
balancing scenarios against each other and sort of averaging out probabilities intuitively weighting them properly
00:48:08
things like that balancing things against each other intuitively that are things that are pointing in opposite
00:48:14
directions this is this is something you get better at and feel easier about by practicing I recommend that when you
00:48:21
come to a new field about which you know nothing that you browse the literature talk to people if you can attend
00:48:31
lectures if you can and listen for the totem words the words which either consciously or unconsciously
00:48:39
people in the field give special meeting and I find that if you can figure out what those totem
00:48:46
words are and then go away and figure out what they mean you'll find out at least one of the things that's very
00:48:55
important to people in the field that they don't say is important to them necessarily but they give it importance
00:49:02
by continually using these totem words or continually emphasizing them for instance last week I went to Grand
00:49:09
Rounds at our Regional Hospital and there was a cardiologist giving a talk and he was giving a talk on the
00:49:17
pharmaceutical management of cardiac patients who had circulatory problems problems with blockages problems with
00:49:26
leakages he would go through a series of example patients this patient comes in with these symptoms what do you do
00:49:36
we immediately stent this patient how do we drug this patient for the stent we're
00:49:42
gonna use this compound and we're gonna use this compound but we're gonna stay
00:49:45
away from that compound because it can cause bleeding and we don't want bleeding now this patient has got a
00:49:52
partially occluded artery and is in a lot of pain we're going to give this patient warfarin because we need to
00:50:00
dissolve that clot and prevent it from growing however we're not going to give
00:50:05
this patient these other two drugs because they can lead to bleeding even worse than warfarin can lead to bleeding
00:50:13
we give combinations of these drugs because it reduces bleeding bleeding this bleeding that what I came away from
00:50:22
not knowing really that very much about how to treat cardiac patients was that if we can generalize from just this one
00:50:30
talk what's most important is to keep the patient from bleeding no matter what
00:50:36
you do and so if I was given a new drug and asked to predict whether or not it was going to be a success the first
00:50:45
thing I'd look at its whether it causes uncontrolled bleeding or not what the
00:50:49
likelihood is how it can be combined with other things all from a standpoint of does it cause bleeding now that
00:50:57
doesn't give you the answer to how that new drug is going to do but it gives you
00:51:01
a lead-in to it so that you can then begin to understand how things are going on
00:51:06
so I'd say look for the totem words when you're reading an article look for the
00:51:11
words that the author of the article seems to come back to over and over and then start by figuring out what those
00:51:17
words mean trust yourself but doubt yourself as well and always be prepared to be devil's advocate to
00:51:25
yourself if you're working within a team that's even better because most
00:51:30
forecasters do not have a problem being devil's advocate or would call red teaming and keep an open mind don't go
00:51:37
into something thinking you know exactly what's going on it never seems to work
00:51:42
out that way just when you think you know exactly what's going on it changes
00:51:46
no the main thing is just you know be open to change in dr. tetlock spooky mentions that we're in perpetual beta
00:51:54
that is a great way of thinking of it is always be willing to change and learn and just you know go with it a lot of
00:52:02
people get really kind of stuck in the mud on certain ideas and that's that's
00:52:05
the quickest way to get it wrong I've been very insecure all my life for a variety of reasons and I never really
00:52:18
thought very much of my own opinions I always figured well is the world crazy or is it me and the answer was I must be
00:52:25
crazy I think the good judgment project has shown me that occasionally I'm right and
00:52:33
other people are wrong which is not something that I expected going in I expected to be wrong all the time and
00:52:38
curiously and I've seen this in Phil's book and I've heard it from other people
00:52:43
the things about which I was most wrong with the things I was supposed to know something about namely the medical
00:52:50
questions so I think the good judgment project has given me a little bit of confidence in myself that I didn't have
00:52:55
before I went into it it's been huge and for first off you get exposed to a lot
00:53:06
of wonderful people whom you tend to maintain some degree of contact with I've had a couple of of my teammates
00:53:17
visit me at my house one who's from South Africa the other ones from Spain it's true they were on Washington for
00:53:23
other reasons but that's an example and I I'm glad to stay in touch with a lot
00:53:32
of the fork in fact there there have been others as well from the DC area anyways I'm glad to stay in touch with
00:53:39
these new friends that's that's see that's a big thing and just as big is
00:53:49
there's nothing like reality focusing on reality and how it works out and you
00:53:54
tried to see the future and maybe it worked and maybe it didn't and you you sort of mentally record it's there's
00:54:05
feedback going on that's it's extremely good feedback for turning you into a
00:54:10
more rational person it's it makes you a more balanced person a more rational
00:54:17
person easier to live with perhaps I don't know but it's all good it's been very rewarding and this
00:54:29
doesn't mean that the forecaster shouldn't also be rewarded in other ways
00:54:32
but this myself is rewarding and I'm and I'm sure that the others feel much much
00:54:38
the same I think being a forecaster and being a researcher are similar and different I think that one of the great
00:54:49
things about being a forecaster is that we really weren't constrained by any of
00:54:54
the kind of customs or sociology around forecasting because we were the ones coming up with that and so we were able
00:55:04
to explore and be very free in how we approach forecasting whereas I think research while you have some freedom and
00:55:15
hypotheses that you are tackling a lot of the methodology and the structure around writing papers and peer review
00:55:25
and submissions is very much rigid and you must follow the the specific process so I think that that highlights a major
00:55:35
difference in forecasting versus doing the research and that the research seem to be much more process oriented
00:55:46
although obviously there there is an outcome component to it whereas when when we were doing strictly forecasting
00:55:53
really people only cared that we were able to come up with the right answers rather than you know how exactly we we
00:56:01
got there so yeah they're different they're different animals I use different words when I talk to myself
00:56:09
now I've always been very analytic but now I I start talking those percentages
00:56:16
to myself and and that's different my wife and I have two different philosophies of investment she has been
00:56:25
involved over the years as a partner and an investment firm that does alternative
00:56:29
investments which are allegedly less risky but by my lights carry a risk that is sometimes
00:56:37
more than what I like I on the other hand am much more risk averse and although less risk-averse then I was
00:56:45
when I first started participating a good generative project my professional career has been oriented towards
00:56:56
reducing the risk that decision makers have and perhaps I've taken that point
00:57:03
of view in my own approach to investing which my wife feels is overly conservative I think that that the
00:57:12
forecast thing as part of the good judgment project has given me a window into my own risk-taking perspective and
00:57:22
move me at least somewhat closer to my wife's point of view and and I think that that's a very useful thing in real
00:57:31
life as well as you know from the standpoint of participating as a super forecasting so in my work
00:57:40
this experience has been extremely important I am now working on an article about the rise of China that I would not
00:57:50
have written if it had not been from my experience in the good judgment project by having to constantly update my
00:57:57
knowledge about China's economic capabilities and its military capabilities in order to do well in this
00:58:02
game I kept up in a much finer green level with things that were happening in China and that have changed my forecast
00:58:13
about how international relations will work over the next couple of decades so I'm an international relations scholar
00:58:19
one of the most important things in the international relations field right now is this question of of does the United
00:58:26
States have peer competitors is China one yet and this experience helped me see that it is so it's really
00:58:34
dramatically changed my career and the process of going through the confidence-building process
00:58:41
of you know learning how to articulate forecasts learning how to be how to express them in probabilistic terms how
00:58:51
to be confident even if I'm not completely sure has made me willing to take the risk of going out on this
00:58:57
really controversial subject I'm not sure that they have all they probably should having practice at forecasting
00:59:06
geopolitical events maybe I should start practicing forecasting events related to
00:59:11
my own personal life maybe that would help me avoid being unpleasantly surprised by my things but so I'm not
00:59:20
really sure how it's affected my personal life other than having nice dinnertime conversations with families
00:59:25
and friends like I said I go back and forth between the different you know venues of inquiry in archaeology and in
00:59:31
forecasting so there has been times where one of the things that forecasting told me is it's
00:59:36
great if you have a model but all they're going to be a better model always be a better model you might have
00:59:42
one that's showing you exactly what you want but you know if you tweak it a little bit or you say you find a new
00:59:47
dataset can always be that little bit better and I've started to apply that to
00:59:51
some shipwreck hunts that I've been doing where we have a model on how to locate shipwrecks in areas of high
00:59:56
probability but it can always be better so I've actually started tweaking some
01:00:00
of the models that I use at work to add additional data sets maybe pulling things from multidisciplinary studies
01:00:06
from geomorphology marine geomorphology riverbed hydrological processes going into shipwreck site formation and just
01:00:16
basically you know constantly tweaking the models there's something the other
01:00:19
thing that I've tried to do as much as I can is I have two young sons who I'm
01:00:25
trying to teach them how to approach different issues you know not not so much to get him in trouble at school for
01:00:30
questioning the teacher all the time which is very possible but when they answer a question say well is that it I
01:00:37
spoke to my son about a tree growing in a field I said well how much do you think that tree is gonna grow next year
01:00:42
and he just said oh well this I said well I should say that he's like well because it just looks like a well did
01:00:46
you think about how much was it gonna rain next year how much fertilizer is the farmer going to put in that field
01:00:51
without trees in the middle how often is he going to plow it what you know there's so many issues that
01:00:56
surround one issue that when you hear one question there's usually you know 20
01:01:00
30 however many more questions that can help you answer that question so I've
01:01:05
been trying to push it on them to kind of think that way you