Superforecaster Full Video
Ask about Knowledge at Wharton
Brief Summary
Ben Valentino discusses his journey in forecasting and the importance of historical context and collaboration in improving predictions.
Key points
- The Journey to Forecasting. From military tours to academia, the speaker's path to forecasting is shaped by diverse experiences.
- The Role of Curiosity. Intellectual curiosity and willingness to learn are key to making accurate forecasts.
- The Influence of History. Understanding history is crucial for accurate forecasting, as it reveals patterns and possibilities.
- The Importance of Multiple Sources. Relying on a single source can lead to misinformation. Always seek multiple perspectives.
- Understanding Uncertainty in Forecasting. Embracing uncertainty is crucial for accurate forecasting. Focus on assigning realistic probabilities.
- Learning from Predictions. Tracking your predictions can reveal biases and improve forecasting skills.
Episode highlights
- The Journey to ForecastingFrom military tours to academia, the speaker's path to forecasting is shaped by diverse experiences.“I was looking for some way of doing forecasting...”0:30Feb 26, 2016
- The Role of CuriosityIntellectual curiosity and willingness to learn are key to making accurate forecasts.“Intellectual curiosity and just a willingness to put in the work...”7:54Feb 26, 2016
- The Influence of HistoryUnderstanding history is crucial for accurate forecasting, as it reveals patterns and possibilities.“Having a sense of history that stretches back far can give one a sense of possibilities.”13:14Feb 26, 2016
- The Importance of Multiple SourcesRelying on a single source can lead to misinformation. Always seek multiple perspectives.“Don't trust one source of information.”23:34Feb 26, 2016
- Understanding Uncertainty in ForecastingEmbracing uncertainty is crucial for accurate forecasting. Focus on assigning realistic probabilities.“Embrace uncertainty; you're never gonna get everything correct.”44:24Feb 26, 2016
- Learning from PredictionsTracking your predictions can reveal biases and improve forecasting skills.“Keep track of your predictions; it helps you learn.”45:44Feb 26, 2016
- The Importance of AnonymityAnonymity allows for a more honest exchange of ideas and information.“The anonymity is really important.”46:42Feb 26, 2016
- Balancing Confidence and OpennessFinding the right balance between confidence in your views and openness to others' is crucial.“Balancing scenarios against each other is something you get better at by practicing.”47:02Feb 26, 2016
Transcript
I first got introduced to the forecasting by a long time at Dartmouth Ben Valentino who is a political scientist and he alerted me to an opportunity to forecast with the good judgment project and this was right around the time when I was leaving Baghdad Iraq after my second tour there and I was looking for some way of doing forecasting because I had been something that I've been interested in for for a while and that's why been introduced me to this idea and you know I the timing worked out as well I was able to come back to the United States and that was right around the time the the first year
of the tournament kicked off and I just went from there well in my work on security a big part of the question is whose security might be in danger in the coming years so secure this field of security studies inherently is about forecasting making predictions about how actors might be vulnerable which strategies might help them and so forth so I got interested in the good judgment project because I wanted to build these skills I've always had the interest and when I first started my undergrad I started a bunch of things I was one of these constantly changing your majors people and I changed between applied physics is where I started and then I
went to International Studies but then somehow transitioned to anthropology and archaeology but I always kept the interest up in International Studies so it kind of went from wanting to study current culture as Moe was going on to studying past cultures and what was going on there and that the constant you know urge to learn more about all societies kept me interested in forecasting in what was going on in different countries and as I said well switching all the time I went to quite a few different universities in quite a few different countries I was at school in the Stars beginning Utah followed by Vermont and then Norway then back to Utah and I finished my doctorate in
Scotland where I was there for seven years and then I went to Ireland for ten years so you know constantly moving in different cultures different societies different political system it was always something of great interest to me my professional life has involved forecasting because that's what I do or have done to help other people make better decisions hopefully but several years ago I attended a Penn engaging Minds event in New York City and Phil tetlock and Bob MELAS were speaking about the good judgment project which was very intriguing to me and so I inquired about joining as a forecaster and several months later when positions
opened up I became a forecaster I think I am one of those people who can look at things a lot of different ways and I'm not the most adept at it among the forecasters but that's definitely my type and so I was a good fit but I was also driven because I'm acutely aware of the trouble the world is in and I think the the as much as they have a dark side us organizations like I are and the intelligence community in general that need all the help they can get we are basically the good guys and so so there was plenty of you know driving force behind my interest that goes back to when I was a younger man during the first Gulf War in 1991 I was I'm
interested in the military military history and a lot of people and pundits were making predictions about the outcome the manner in which the war would be waged and decided as well as casualty counts and how long it would take and basically everyone was wrong on the duration and the and the destructiveness well the cat and the number of casualties suffered by coalition forces and that was interesting to me because there was one historian I'd like to say trevor Dupree you got it right and I was interested in how he got it right like and evidently he had a system in just a few years later during the 1994 midterm elections when Bill Clinton lost in a
landslide to the Republicans the Republicans took the Congress for the first time in 40 years after that defeat basically every pundit on Earth was saying that Bill Clinton is a one-term president and being a staunch Democrat and a Clinton supporter I was traumatized by that possibility and wondered whether they knew they whether that they knew what they were talking about and at about that time I found a book written by another historian Alan Jay Lippman um about predicting you know who would become the next president and evidently he had a system where you didn't need to use polls you just look at history and using his method I realized that
actually Bill Clinton was in a pretty good shape his his chances weren't guaranteed but you know even a year before the election I had a pretty good inkling that he had a good chance of winning re-election and this was against what all the pundits were saying so based on those two events there were probably other events as well that's why I became interested in forecasting basically the fact that everyone you know got thing big things wrong and I want to know why and some people got big things right and I wanted to know why and whether I could do as well as I could so ever since then I was I started you know I'm my own forecasting you know the
outcomes of Wars and elections with mixed results but you know so but over 20 years so that's how that's how I became interested I'm politically very interested in what's happening and I used to read the 538 blog that was done by nate silver he advertised for the good judgment project i don't know where i saw the notice that you guys were putting together this program it was on one of the standard outlets maybe Wall Street Journal maybe Bloomberg but I read an article that you were putting together this forecasting tournament and that you were trying to find out what it is makes a good
forecaster and I feel like I owe the system for so many years I've been doing research and I've relied on other people to contribute to that research and they have selflessly and so I'm a chronic volunteer as well when somebody says I need your help to do my research I volunteer for instance the New York Blood Center contacted me and said you're not allowed to give blood but we've got this research program and blood donation will you come in and donate a pint of blood every month so we can see what happens so I did that just to pay back the system so I'm paying back the system by being in the good judgment project
I think that intellectual curiosity and just a willingness to put in the work and to think hard about specific problems and try to be as objective as possible is what allowed me to make more accurate forecasts and others within the pool I wouldn't necessarily say that you know what we did or what I did as an individual is the best possible at forecasting because they're you know can be so many more improvements I I think in terms of how much you know about a specific subject you can always improve your level of knowledge and I think you can always improve you know how you're you're specifically viewing a situation how that situation could potentially
evolve so having a greater understanding of how humans interact with each other relate to each other the political contours of specific situations and events those are all things that you can gain greater knowledge on and get better at forecasting in so yeah yeah being open-minded and being open-minded enough to realize that you need to remain open-minded I think are some of the keys to being you know for for my own forecasting performance well I think there were a couple of factors one was that I am in the middle of my career now and I've spent a lot of time studying exactly the kind of issues that the good judgment project was focusing on and so
I was familiar with trade issues I was familiar with national security issues the rise and fall of governments that kind of thing so it really was right in my issue area also I I use a theory that can be well applied in this kind of situation it's called structural realist theory and it gives a sense of big recurring outcomes and international relations the rise and fall of great powers the tendency for military conflict to occur competition and trade and so forth and so I had a kind of broad understanding of international affairs that I could apply to particular situations and the beauty of this theory
was that it was it's really pre open states have a lot of choices about how they act and so I wasn't falling into the trap of thinking that something was inevitable but instead was investigating which things were more likely which things were less likely so this theory was really conducive to probabilistic forecasting and my background knowledge was really helpful in having a big set of information to draw from and also knowledge about where to find things where to find information that would be helpful in what I do for my day job in maritime archaeology it's a very similar thing we take the outside view we have a very disparate information we have maybe a few
artifacts maybe not even maybe some historical sources maybe someone stumbled upon something and just have rumors historical studies ethnographic studies and we have to take all this information and try to piece it together to come up with a solution to or explanation of what we're finding and it's very similar in forecasting I kind of approach it the same way where I'll take very disparate information I won't take the big huge pictures that are thrust in your face I'll take you outside of use and use some of the same tools that I use at work I'm a huge proponent of using GIS mapping of different issues to try to to get like say it's a question that has a
geographical perspective I'll make my own GIS program and basically you know put a model together that has all the contributory information from different press sources different mapping where different roads are communication routes transportation routes and it's the same kind of thing I would do in archaeology I just apply it to the future and forecasting as opposed to applying it to the past I don't trust my gut so usually my first impression is right about half the time so in order to do well in a forecasting tournament you have to dig deeper into the sources if you're predicting elections for example you know I'm find out you know what happened
in the past you know which parties were likely to win and under what circumstances do your research about the electoral system sometimes you know the electoral system could lead to a counterintuitive result you know a party could do very well in the polls and still lose the election so it's just really a matter of doing your homework trying to understand the situation the history behind the event and and really working hard at it and I would say a second reason would be my sense of history so I think that comes in very handy because most people when they would forecast whatever event it is would really look to the recent past draw on their own memories as well
as maybe you know a Google search of you know the recent news events but I think having a sense of history that stretches back far can give one a sense of mmm possibilities which you know the recent past might not indicate I think that I make connections for instance I couldn't figure out why Putin was in Syria none of the explanations that people gave the Talking Heads made any sense to me Syria does not a join Russia the Ukraine does Syria is of no strategic importance to Putin why is he in Syria there was just no reason all of this business about how he couldn't let one of his henchmen down or
else all the other henchmen would feel insecure just didn't buy that I wanted to make sense of that and all of a sudden I remembered something and that is that Putin is building up the Russian fleet in the Mediterranean but they don't have a home port in the Mediterranean so they have to go all the way back to Russia and the Black Sea to get a home port well where is Putin landing his troops next to the port of Latakia which is a major port on the Mediterranean perhaps what Putin wants is a deal with Assad for a 50-year lease on the port of Latakia now that to me makes sense that would explain why Putin is in Syria it accomplishes a strategic thing for him
it gives him a port for his Mediterranean fleet so I think that's probably part of the reason why I was successful I feel uncomfortable saying successful because I don't feel successful you tell me I was successful I don't feel successful that's part of the modus operandi that I go through it's like I see disparate things and I try to make sense of them for instance another thing when the third Kim took over North Korea all of a sudden there was a huge round of executions including his own uncle he blew his uncle up with it aircraft anti-aircraft gunner things were then fairly quiet and then another big round of executions I couldn't
figure out why he was executing all these people I contacted somebody who was an expert in North Korean relations and I said why all the executions he said well they've been more than you know they just not publicized have been two big rounds I thought what's going on here and then I saw something that said Kim has not yet traveled outside of North Korea not even to Beijing to visit his sponsors we don't know when Kim's going to travel outside of North Korea all of a sudden you put the two together perhaps the guy's afraid that if he leaves the country he won't be able to come back that there'll be a coup in his absence and he's taking care of
everybody who might be involved in such a coup and in fact he's now due to go to Moscow I think in December so again trying to make sense out of something just looking around at where how things might fit together to make things make more sense than they would otherwise the tournament taught me that and in a very humbling way that I knew a lot less about the world than I thought I knew and so that to begin with sparked my thirst for knowledge and information it also gave me a perspective about risk-taking because in my first year I was not a super forecaster I had to earn that status and I was involved in in a marketplace which involved essentially
trading and I've learned a lot about risk-taking and how much I was either risk-averse or overly overly risky learning about things from a perspective of other information has been very important but once I became a super forecaster in particular what was helpful to me was a perspective of others the sharing of information and the ability to test your own beliefs and hypotheses for bias by taking into account what others have said or analysed as well as information that contradicts your own has been very valuable to me I was very steady I I answered as many questions as I could I was consistent I updated I didn't let
anything get forgotten and I approached it like I was taking a test at school come in and maybe I didn't even study so but hey it's it's it's not an essay test so the choices are very clear it's it's it's it's numbers and and I would read the question carefully and analyze whether whether this would hinge on on a single factor or are many many factors if it was something that I had a clear gut feeling right off the bat then I would decide how much to back off from that extreme and if it was something where I was really undecided at the 50/50 mark I would force myself to a choice because 5050 is is a is never quite the right answer one of my hobbies
is oil painting so and I and also when I've been taking the test they tell me I'm visuospatial so I I think that may have made a difference getting the big picture and knowing when to zoom in to the details you know backing off and then zooming in and not getting distracted by the wrong details so I've learned in painting that you have a big canvas you know you make broad strokes but somehow sometimes just the tiny nuance here are there changes the way the whole painting looks and I think that's true in world events too and why you need a lot of different perspectives so that I love my news
sources I have my favorite ones but when you get a variety and when you hear feedback from other forecasters it really helps it's painfully clear that the more the wiser and more thoughtful and rational you are the better you're going to do and so I'm afraid you know that it's kind of the same people who would do better a lot of different things that require mental effort are the ones that rise to the top in forecasting I think what's special about forecasting is that if you're a really natural researcher that can get you quite far and I just completely admire and am grateful to those of us who are so good at it and who have you know far-flung
connections to decision centres in the world it's I'm sort of a free rider in that way so there are two things that are really important and so I come from one side of that I'm a I'm a person who was a natural for computer programming for example but not a natural for writing or researching okay well along with not trusting my own gut feelings my first impressions I don't trust the gut feelings and of experts and pundits I try to when ever possible to discover understand what assumptions underlie their forecasts sometimes they're explicit about this and other times they are and they don't really they'll say you know this event
is likely but not really explained and if I keep doing my research and find other pundits you say a certain event is likely without you know much explanation at all then that's a signal for me to really do some deep research and and try to even if I agree with that assumption you know sometimes I'll just try to play the devil's advocate and say well let's pretend that this silly outcome that's unlikely to happen you know actually could happen and kind of play that out and sometimes you know I'll go back to my original estimate yeah that's silly it's not gonna happen but you know often is the case that the silly outcome is becomes the more likely
one so that's that's one thing I do and others I try to discover or to find out you know who has the power in a certain situation who who is the decider in the words of george w bush and that's one that keeps coming back all the time and also you know in every forecast you want to you know write down a list or in your head you know of all the constraints that would pretend prevent a certain event from occurring or slow it down you know make an event you know and a likely event take longer to occur than you know what was expected before so those are just a few things the first one was there there's almost no but I'm I tell
you there's no nothing is certain that's not entirely sure there's I'm sure there's certain things that are served but for the most part nothing is certain the other thing was don't trust one source of information but then don't discard all sources just because you think it's not valid when you're whenever you're reading a source of information if you have just one single source and he's the only one saying it I didn't sometimes I wouldn't dismiss it but I would not trust it I'd look for something that would you know contribute to it or just help verify it and that can even be one of these sources where you know you see it on the internet you
think wow that's really dubious these guys are you know pretty crazy but somewhere in there if you read through the whole thing there might be one sentence which collaborates what's going on in that other source that was a single source and I was like okay now I've got two independent sources so and you can start to build on that so one of the things I would ever say is don't rely on the one don't discard sources just because you may think they're a bit out there and there are quite a few sources that are out there I think that maybe the big thing is just to be aware of what you know what you can be certain of and what you're not sure of to know the difference between subjective and
objective knowledge what things can be measured and what things are a matter of opinion look at history look at patterns look at base rates and then figure out whether your circumstance is similar or different I mean what's what was interesting for me is that most of the things I was forecasting on I knew nothing about going in finance questions for instance I I never pay attention to finance but when you're forecasting on world currencies you can always find graphs historical charts and I look at them and I see the patterns and I say why is it different now you know are we operating between the same extremes or will it be new and usually I've gone with stay with
that stay with the trends so I didn't let myself get intimidated by any questions and I tried everything and as bill mentioned some of the things where I wellactually was most hesitant on topics that I might have knowledge on say on Ebola because suddenly I dunno I'm actually more self-conscious about things where I might miss something that I should know one easy to follow tip is do it every day and every question that you're forecasting have a look every day to see if anything has changed you what point things do change it significantly tends to come as a
surprise and if you missed that day well you blew that forecast for that day the my approach is to try to think through causal mechanisms and and try to think of all the influential factors that I can think of and and try to visualize scenarios and and balance probabilities among scenarios things like that this is a little different than what I think most of us are led to do by the training we've had which is it's not that the training doesn't cover probability and that's critical but it's to look at the categories of the questions we're forecasting and try to find parallels in the past and look for rates of occurrence and try to use those rates I
I think that's helpful as a starting place but it's not going to get you a very good score if that's all you do and I think the difference between you know sort of phoning it in and and getting a good score is understanding the situation as well as you possibly can and that's the what Phil has Phil tetlock has called the inside view as opposed to the outside view he emphasizes the people neglect the outside view but I think it's the inside view that that really matters to get you exceptional forecast that that actually could add to give real value anyway that's the inside view well the first thing was to really explore my biases to
realize that I often would have a first impulse about a question and I didn't want to just go with that sometimes it was right but I didn't want to just go on impulse and secondly I really learned to think about what my strengths were to focus on questions that I thought were in my area of strength and in particular to focus on questions that I thought were were being answered wrong or you know where the the general consensus was different from what I thought it was because there I could make the most difference in shifting opinion or contributing knowledge that might sway the crowd and make the group as a whole more accurate
so I think those things were really important a good friend of mine also named Barbara Barbara McClintock advised me over and over in our conversations that I should always question my fundamental assumptions my underlying assumptions and I think that's something that's been very important to me was Barbara's advice for instance a spacecraft just flew past Jupiter and took a whole bunch of high-resolution pictures of Jupiter I'm sorry Pluto it took a whole bunch of high-resolution pictures of Pluto and son of a gun this 4.5 billion year old planet is geologically active and everybody is absolutely floored the solar system is 4.5 billion years old the planets are
not supposed to be geologically active when they're the size of Pluto why is this so everybody's trying to figure out why 4.5 billion year old Pluto is by geologically active they're talking about more missions to Pluto half hundred million dollar missions half billion dollar missions I'm certain that there are all sorts of young planetary scientists who are trying to figure out how they can start their careers by figuring out and making a name for themselves Pluto is geologically active well what's the underlying assumption here the underlying assumption is that Pluto's 4.5 billion years old what happens if Pluto isn't 4.5 billion
years old what happens with Pluto's a good deal younger then in fact you expect it to be geologically active and there's no inconsistency here at all the inconsistency is that your fundamental assumption that Pluto is 4.5 billion years old was incorrect so I'd advise those young planetary scientists who are thinking about trying to explain how a 4.5 billion year old celestial body could be geologically active to question whether or not it's really 4.5 billion years old there's another something else that gets a lot more attention and that's it ran for the last four to five years the entire economy of the world has been an uproar because all of the major nations have slapped an embargo on
dealing with Iran because Iran is trying to build a nuclear weapon and they had to negotiate long and hard with Iran to negotiate Iran away from a posture where they could be capable of building the nuclear weapon that they were trying to build and this brought us a lot closer to allies like Saudi Arabia to whom we perhaps should not be close and it's caused no end to trouble throughout the entire world particularly the Middle East which is troubled enough well what's the unda been underlying assumption here that Iran wants to build a nuclear weapon they kept saying that they didn't want to build a nuclear weapon why didn't we believe them well we have
x y&z reasons why we didn't believe them but again it's a fundamental assumption the entire foreign policy in the Middle East of nearly the entire world was dictated by the underlying assumption that Iran wanted to build a nuclear device I personally wouldn't want to build a nuclear device I can't see why Iran would want a nuclear device nuclear bombs are so 20th century we've moved on I happen to think Iran didn't particularly want to build a nuclear bomb but everybody's fundamental assumption was that it did and I would go out on a limb and I'd say that even in spite of all the acrimony that we've had over
Iran that it's quite possible that in 20 years Iran will be our closest ally in the Middle East and the reason for that is that Iran is the largest Persian nation on earth the United States is the second-largest Persian nation on earth and Canada is the fourth largest Persian nation on earth there's a natural affinity there and as soon as that embargo is lifted the Persians are not going to start doing deals with the British or the Chinese or the Russians they're going to do business with their Persian American friends so I see business between Iran Persia and the United States particularly the Persian community building very quickly and I think that could lead to good political
relations as well that would be a major casualty of misunderstanding Iran's intentions with nuclear weapons the process evolved over time for me and one of the things that is important to me is to use historical information but not to be over alliant on it if there is the ability to compare against similar situations in the past I'm very cognizant of using that information but only to the extent that it is truly relevant to the current circumstances beyond that frequently adjusting my own forecasts based upon new information and new perspectives of others is extremely valuable it's very easy to take a position and sit back and say that's it
however the real world is a very active one and often things change and forecasts should change based upon the events and the happenings of the world and I found it's extremely useful to pay a lot of attention to those events and frequently re-examine my own forecasts and see if they need to be updated so I really had three general rules when I made forecasts especially when I made my first forecast on a new question or a situation I'd never seen before so the first thing I would say is to be a good forecasters you have to be you have to hold strong opinions weekly and what I mean by that is that you do all of your homework
upfront and you really present a good case for why you feel a certain way on a specific question and defending your probability probability estimate at that point but when you recognize that somebody else has more information or better information than you having the ability to change your mind swiftly to that new forecast I think that was one of the keys to why I was able to you know improve my performance over time I think many people really get stuck on their initial forecasts and they fall in love with it and wind up becoming going into a defensive Crouch when it comes to defending their opinion whereas I was on
my team sort of notorious for being able to switch from one very hard position all the way to the opposite when I saw new information and come up that led me to do that the second thing is to be willing to put in the homework and to put in the effort to understand a situation go beyond the Wikipedia page and to go beyond the initial couple of news articles and to really try to understand you know what is it what is it about Boko Haram and their composition that you know is allowing them to make the kinds of military gains that their that we're seeing and how fragile or stable are those military gains compared to other
like organizations and to you know not just to say Oh Boko Haram looks like every other organization that I've encountered to this point but to say to really try to understand what's going on there and to contextualize that appropriately and I think the the third thing is to have in your mind the kinds of psychological and cognitive biases that you could potentially fall prey to and to be able to overcome those by you know listening to your teammates by self critique and to recognize that you know whatever forecasts and argument that you're putting forward now you should be able to alter it or change it or improve it in light of new information so you know
keeping in mind the cognitive biases that really impact and influence and reduce our accuracy and to consciously try to devise them I think the tournament was great you need to practice so you need to have a lot of questions you need to have feedback you need to know where where you're good and where you're messing up if you have a team if you have other people that you're in tournament either in competition or in collaboration either way you listen to them you get feedback and you will you will improve I think that one of the things that would be extremely useful would be to participate in good judgments open forecasting competition because you can
test your own abilities and learn about the things that that both limit your ability to forecast and make you a better forecaster and through that process become somebody who is more adept at forecasting the good judgment incorporated itself has put together various training methods which have been very useful to me as a super forecaster and I'm sure to others but as an individual just forecasting and and using that forecasting experience can be a very valuable thing to learn about yourself and the process of forecasting what and and what you do well and what you do less well and then to seek ways of improving upon those weaknesses I think you could really do two things to
become a good forecaster and one one is to read and not just read say Phil's book super forecasting or Danny Kahneman's book Thinking Fast and Slow which I think many of the forecasters who excelled in torment have read and done I also think that you know reading say histories like the guns of August or books about bureaucracy by James Q Wilson and others really improves your your knowledge base and gives you a better context for looking at situations and being able to figure out whether there are any commonalities or patterns there that you can pick up on and that could potentially apply to future situations
that look like that so being well-read is I think a contributor to forecasting success the second thing is to just to practice don't get hung up on wrong forecasts that you make initially you know don't don't get overly you know sensitive to those results but to approach forecasting as something that you can improve on and have that growth mindset when it comes to making better forecasts and so from there you're able to say okay I start at some baseline level I got some feedback tells me I should improve in X Y & Z and to really make a concerted effort to do that through practice well one of the things I would recommend would be to write down
a list of all the things that would that you should be on the lookout for that might cause you to change your mind so after you've done your homework you've thought deeply about you know the event and the likelihood of of it occurring write down a list of of things that you know would happen leading up to the event or signals markers whatever that would cause you to change your mind and and the reason why I say write it down is because it's because you know it's very hard to change your mind and if you write it down it might be help you you know not try to to fudge things well you know if it's if it's just in the back of your head you know you mM it becomes a little bit more difficult to to go
against you know your original original conclusion because I mean you put a lot of work into this and you don't want to go back on it and there's an actual tendency not to want to you know waffle or be too wishy-washy but you know having it down on paper will just show you well I'm not being wishy-washy you know events have changed things have changed and maybe my initial conclusion was wrong I would say the third thing would be to play devil's advocate after immediately after you've you know come to a conclusion and made a forecast seek out information that would prove you wrong you know if anything you know just to cover all your bases
I'd say my fourth piece of advice would be to seek a wide variety of sources even weak and biased sources because sometimes you can find interest in information in very flawed sources but that'll take some work on your part to determine you know which part of these you know particular reports are worth you know holding on to and which part should be discarded it's not an easy task but don't just you know casually disregard even even bad sources because you never know from four years as a as a forecaster in the gjp tournament I can say that another piece of advice I would give is to assume that things would take longer than your initial impression that's a rule of thumb because it
happens consistently time and time again that whatever it is a diplomatic initiative or or or what-have-you events often will take a lot longer than then what what you think you know they will take or what officials you know the timetable of of officials and and that's just been my experience from doing this for a long time final piece of advice would be embrace uncertainty don't get too hung up on getting things correct you're never gonna get everything correct you're going to get a lot of things wrong I would focus more on assigning viable or realistic probabilities to events just think of it in terms of you know if I
assign a 70% probability to a particular event you know I'm gonna be wrong three out of ten times you know four so if you if you work on being well calibrated instead of trying to get everything right I think that's probably a more fruitful endeavor well every year in the tournament I kept a running list of what questions I'd answered when I first answered them how I Hearst answered them and it was really helpful to see how my opinion changed as new information came along so I was you know kind of keeping myself honest about what was my original opinion and what was making me change my mind and I think that was really helpful because I
learned a lot about what kind of pits I might fall into you know if I get over optimistic about suddenly there's a lot of news on a certain issue I might have a tendency to think that something was more likely to happen whereas maybe wasn't really more likely to happen so keeping track of my predictions was very helpful I thought it was really important that I was at forecasting anonymously at first it made it more I made me more confident about making predictions without worrying about their effect on my reputation and after all you know I was in this to get better about forecasting so I wanted to have the ability to take to make bets or you know answer questions in a way that I
thought was correct without worrying about how that might be seen somewhere else and that really built my confidence and actually through being right then frequently too it built up my confidence to the point that I was willing to who I was and then got to experience the whole other dynamic of people knowing I was an expert wondering what I thought about things and then having to be cautious about not relying too much on my expert knowledge but still thinking about the wisdom of the crowd so I think the anonymity is really important and also realizing that the magic of this whole study is in revealing that everybody has important information and
at different times different people's information will be really valid and especially important and and finding that right balance between being confident in your own views and and being open to the views of others become comfortable with researching problems have a great interest in what you're forecasting don't hesitate to dig right in to you know study something new just get into it don't don't neglect if you have any bent at all for statistics and probability bring it into play you know but that's kind of obvious anyone who has that choice I guess actually that the this might come up in your last question I think you forget but part of the muscle
you want is the muscle you want to strengthen intuitively is the muscle for balancing scenarios against each other and sort of averaging out probabilities intuitively weighting them properly things like that balancing things against each other intuitively that are things that are pointing in opposite directions this is this is something you get better at and feel easier about by practicing I recommend that when you come to a new field about which you know nothing that you browse the literature talk to people if you can attend lectures if you can and listen for the totem words the words which either consciously or unconsciously people in the field
give special meeting and I find that if you can figure out what those totem words are and then go away and figure out what they mean you'll find out at least one of the things that's very important to people in the field that they don't say is important to them necessarily but they give it importance by continually using these totem words or continually emphasizing them for instance last week I went to Grand Rounds at our Regional Hospital and there was a cardiologist giving a talk and he was giving a talk on the pharmaceutical management of cardiac patients who had circulatory problems problems with blockages problems with leakages he would go through a series of
example patients this patient comes in with these symptoms what do you do we immediately stent this patient how do we drug this patient for the stent we're gonna use this compound and we're gonna use this compound but we're gonna stay away from that compound because it can cause bleeding and we don't want bleeding now this patient has got a partially occluded artery and is in a lot of pain we're going to give this patient warfarin because we need to dissolve that clot and prevent it from growing however we're not going to give this patient these other two drugs because they can lead to bleeding even worse than warfarin can lead to bleeding
we give combinations of these drugs because it reduces bleeding bleeding this bleeding that what I came away from not knowing really that very much about how to treat cardiac patients was that if we can generalize from just this one talk what's most important is to keep the patient from bleeding no matter what you do and so if I was given a new drug and asked to predict whether or not it was going to be a success the first thing I'd look at its whether it causes uncontrolled bleeding or not what the likelihood is how it can be combined with other things all from a standpoint of does it cause bleeding now that doesn't give you the answer to how that new drug is going to do but it gives you
a lead-in to it so that you can then begin to understand how things are going on so I'd say look for the totem words when you're reading an article look for the words that the author of the article seems to come back to over and over and then start by figuring out what those words mean trust yourself but doubt yourself as well and always be prepared to be devil's advocate to yourself if you're working within a team that's even better because most forecasters do not have a problem being devil's advocate or would call red teaming and keep an open mind don't go into something thinking you know exactly what's going on it never seems to work
out that way just when you think you know exactly what's going on it changes no the main thing is just you know be open to change in dr. tetlock spooky mentions that we're in perpetual beta that is a great way of thinking of it is always be willing to change and learn and just you know go with it a lot of people get really kind of stuck in the mud on certain ideas and that's that's the quickest way to get it wrong I've been very insecure all my life for a variety of reasons and I never really thought very much of my own opinions I always figured well is the world crazy or is it me and the answer was I must be crazy I think the good judgment project has
shown me that occasionally I'm right and other people are wrong which is not something that I expected going in I expected to be wrong all the time and curiously and I've seen this in Phil's book and I've heard it from other people the things about which I was most wrong with the things I was supposed to know something about namely the medical questions so I think the good judgment project has given me a little bit of confidence in myself that I didn't have before I went into it it's been huge and for first off you get exposed to a lot of wonderful people whom you tend to maintain some degree of contact with I've had a couple of of my teammates visit me at my house one who's from
South Africa the other ones from Spain it's true they were on Washington for other reasons but that's an example and I I'm glad to stay in touch with a lot of the fork in fact there there have been others as well from the DC area anyways I'm glad to stay in touch with these new friends that's that's see that's a big thing and just as big is there's nothing like reality focusing on reality and how it works out and you tried to see the future and maybe it worked and maybe it didn't and you you sort of mentally record it's there's feedback going on that's it's extremely good feedback for turning you into a more rational person it's it makes you a more balanced person a more rational
person easier to live with perhaps I don't know but it's all good it's been very rewarding and this doesn't mean that the forecaster shouldn't also be rewarded in other ways but this myself is rewarding and I'm and I'm sure that the others feel much much the same I think being a forecaster and being a researcher are similar and different I think that one of the great things about being a forecaster is that we really weren't constrained by any of the kind of customs or sociology around forecasting because we were the ones coming up with that and so we were able to explore and be very free in how we approach forecasting whereas I think research while you have some freedom and
hypotheses that you are tackling a lot of the methodology and the structure around writing papers and peer review and submissions is very much rigid and you must follow the the specific process so I think that that highlights a major difference in forecasting versus doing the research and that the research seem to be much more process oriented although obviously there there is an outcome component to it whereas when when we were doing strictly forecasting really people only cared that we were able to come up with the right answers rather than you know how exactly we we got there so yeah they're different they're different animals I use different words when I talk to myself
now I've always been very analytic but now I I start talking those percentages to myself and and that's different my wife and I have two different philosophies of investment she has been involved over the years as a partner and an investment firm that does alternative investments which are allegedly less risky but by my lights carry a risk that is sometimes more than what I like I on the other hand am much more risk averse and although less risk-averse then I was when I first started participating a good generative project my professional career has been oriented towards reducing the risk that decision makers have and perhaps I've taken that point
of view in my own approach to investing which my wife feels is overly conservative I think that that the forecast thing as part of the good judgment project has given me a window into my own risk-taking perspective and move me at least somewhat closer to my wife's point of view and and I think that that's a very useful thing in real life as well as you know from the standpoint of participating as a super forecasting so in my work this experience has been extremely important I am now working on an article about the rise of China that I would not have written if it had not been from my experience in the good judgment project by having to constantly update my
knowledge about China's economic capabilities and its military capabilities in order to do well in this game I kept up in a much finer green level with things that were happening in China and that have changed my forecast about how international relations will work over the next couple of decades so I'm an international relations scholar one of the most important things in the international relations field right now is this question of of does the United States have peer competitors is China one yet and this experience helped me see that it is so it's really dramatically changed my career and the process of going through the confidence-building process
of you know learning how to articulate forecasts learning how to be how to express them in probabilistic terms how to be confident even if I'm not completely sure has made me willing to take the risk of going out on this really controversial subject I'm not sure that they have all they probably should having practice at forecasting geopolitical events maybe I should start practicing forecasting events related to my own personal life maybe that would help me avoid being unpleasantly surprised by my things but so I'm not really sure how it's affected my personal life other than having nice dinnertime conversations with families
and friends like I said I go back and forth between the different you know venues of inquiry in archaeology and in forecasting so there has been times where one of the things that forecasting told me is it's great if you have a model but all they're going to be a better model always be a better model you might have one that's showing you exactly what you want but you know if you tweak it a little bit or you say you find a new dataset can always be that little bit better and I've started to apply that to some shipwreck hunts that I've been doing where we have a model on how to locate shipwrecks in areas of high probability but it can always be better
so I've actually started tweaking some of the models that I use at work to add additional data sets maybe pulling things from multidisciplinary studies from geomorphology marine geomorphology riverbed hydrological processes going into shipwreck site formation and just basically you know constantly tweaking the models there's something the other thing that I've tried to do as much as I can is I have two young sons who I'm trying to teach them how to approach different issues you know not not so much to get him in trouble at school for questioning the teacher all the time which is very possible but when they answer a question say well is that it I spoke to my son about a tree growing in
a field I said well how much do you think that tree is gonna grow next year and he just said oh well this I said well I should say that he's like well because it just looks like a well did you think about how much was it gonna rain next year how much fertilizer is the farmer going to put in that field without trees in the middle how often is he going to plow it what you know there's so many issues that surround one issue that when you hear one question there's usually you know 20 30 however many more questions that can help you answer that question so I've been trying to push it on them to kind of think that way
you





