00:00:05
i've been involved in a multi-year forecasting tournament sponsored by iarpa which is the
00:00:14
research branch of the intelligence community iarpa sponsored five different university teams
00:00:21
that competed with each other to come up with the best possible ways to measure and aggregate forecasts about
00:00:32
events all over the world and they include military conflicts elections pandemics
00:00:41
refugee flows and even things like the price of commodities now what we did was to recruit thousands
00:00:49
of forecasters from blogs and professional societies and research centers and so forth and
00:00:57
had them make forecasts over a period of a year and they were given questions every two weeks they logged on to a
00:01:07
website where they made their predictions and they went back to update their forecasts as often as they wanted
00:01:15
now we didn't really know what to do to improve forecasts so we did what came naturally and that was
00:01:20
to run experiments we found that three factors did extremely well one was training people we devised a
00:01:31
one hour probability training module and and that seemed to improve predictions we put people in teams as opposed to
00:01:40
having them work individually and that improved predictions the interaction the information sharing
00:01:48
the debates about rationales were boosted accuracy more over and beyond the benefits of
00:01:59
working alone and lastly we found that tracking was a huge booster of forecasting accuracy
00:02:09
at the end of each year we took the top two percent of thousands of forecasters put them
00:02:14
together in elite groups and gave them the title of super forecasters and these people uh
00:02:23
increased their accuracy in ways in in more than we could possibly have imagined
00:02:29
they interacted more they looked for more information and the net result was amazing
00:02:37
in fact they helped us win the tournament three years in a row businesses everyone for that matter
00:02:48
relies on predictions and we know a lot more about predictions than we did prior to this tournament in the
00:03:01
in the world of business people care about whether to invest in research or expand
00:03:07
to a new market and they care about what consumers will want what extensions they'll
00:03:13
prefer when products will be ready for distribution and businesses can i think use the insights from this
00:03:24
tournament to make better predictions not everything general generalizes smoothly but
00:03:35
there are a lot of both psychological and statistical insights that we know now make predictions better
00:03:47
you know i was surprised by an awful lot of things in this tournament i was surprised that uh our training module
00:03:54
worked it's it's tough to to design a module that has any effect on judgmental accuracy
00:04:05
i was very surprised that teams worked better than independent forecasters i read the wisdom of the crowds and i
00:04:14
assumed that independent forecasters would would do better and heirs would average out but
00:04:20
the benefits of sharing information and talking about rationales outweighed the the benefits of of independence so
00:04:31
um in our case uh teams worked better and i was super surprised about the effects of super forecasters
00:04:41
it's it's like tracking kids in schools and putting the best ones together and
00:04:46
the the synergy that came from that was phenomenal yeah and i think there's several reasons
00:04:55
why our teams work so well one of them was that they worked online so it's tough to be a dominant
00:05:04
bully online people logged on whenever they felt like it and so things were sequential they weren't
00:05:14
simultaneous where i think groupthink is more likely to occur they had a lot of respect for each other
00:05:21
the the forecasters and i think that's another big part of a good recipe well i think companies can improve their
00:05:36
predictions and i think they can create their own set of super forecasters and they can do better
00:05:46
at forecasting the future um and better is really relative i mean all you have to do is be better than the
00:05:56
next guy we're not looking for perfection here and obviously we're not going to get it
00:06:01
um in the case of the u.s government policy makers face decisions that involve billions of dollars and
00:06:08
thousands of lives and in in these cases you know the stakes are so high that even a tiny little edge is
00:06:18
huge progress um the the one of the implications of of our work i think is that good
00:06:28
predictions involve both psychology and statistics it's a combination of understanding the person
00:06:34
and then understanding the the aspects of of uh statistical distributions and statistical information
00:06:45
we know now how to uh do much better at devising algorithms that aggregate multiple forecasts and we also
00:06:55
know a lot about what conditions or environments people bring out the best of of of individual
00:07:03
forecasters well every day we hear predictions we have pundits and experts and gurus
00:07:15
and specialists making predictions about what will happen in the future and many of those predictions are so
00:07:23
vaguely stated that we could never in a million years figure out how to test them
00:07:30
there are statements like there may be an increase in conflict in yemen in the next two weeks
00:07:37
or the situation in baghdad will get worse before it gets better or something like that that's
00:07:44
that's not a question or a statement that passes the clairvoyance test you could
00:07:49
you can't figure out later who's right and who's wrong now we look to these people for
00:07:58
advice we look to them for insights and we're getting really very little from those vague predictions and i think
00:08:10
the way we can do better is to keep score we've got to have questions or predictions stated
00:08:18
in such a way that they're testable and then we can find out who's right and
00:08:22
who's wrong and how we can learn to do better we are in a fortunate position of being
00:08:33
able to have an empirical basis for the claims we make we have we know what works based on
00:08:41
data based on experiments and many of the books and methods and techniques for doing better forecasting
00:08:50
are simply not tested so clearly the empirical side of things is something that is unique to our project i think
00:09:03
we've learned a lot about uh what makes things better we've learned for example that
00:09:11
survey formats with statistical algorithms combining the forecast can outperform prediction markets
00:09:20
we've learned that if you measure probabilities while people are trading in the market
00:09:26
you also ask them what's your probability that event x will occur you can do better at forecasting
00:09:34
accuracy by combining both the prices and the probabilities and these things are both surprising
00:09:40
from an economic perspective we know a bit more about individual differences that correlate with
00:09:47
forecasting accuracy so we can say a little bit more about who the best forecasters are especially
00:09:52
in this geopolitical context not surprisingly they're they tend to be smart they tend to know
00:09:59
a lot they have a lot of political uh knowledge but they are also more likely to be actively open-minded
00:10:07
thinkers they are more analytical they are more likely to take a scientific world view
00:10:16
they're more likely to take multiple perspectives on a question and use multiple reference
00:10:22
classes they're more likely to use probabilities in a more granular or nuanced fashion they're more likely to
00:10:33
say 17 and 83 percent rather than 20 and 80 percent and it turns out that extra granularity
00:10:41
has information in it if you round forecasts up to the nearest 10 20 30 40 most forecasters do worse and
00:10:54
that suggests there's there's valuable signals in that granularity not just
00:11:00
noise well uh the irb tournament will close on june 2nd in in another month but that's not the end of forecasting we
00:11:17
will be opening the good judgment project will be opening a public tournament this fall
00:11:23
and we have lots of hypotheses to test we'll need lots of volunteers and we'd
00:11:28
love to have people affiliated with wharton so if this is something that interests you
00:11:35
or you know people who would be interested go to www dot goodjudgementproject.com
00:11:44
to get more information and we'd be extremely grateful and delighted to have
00:11:49
you join us you