00:00:05
i've been involved
00:00:06
in a multi-year forecasting tournament
00:00:09
sponsored by iarpa which is the
00:00:14
research branch of the intelligence
00:00:15
community
00:00:17
iarpa sponsored five different
00:00:19
university teams
00:00:21
that competed with each other to come up
00:00:23
with the best
00:00:24
possible ways to measure
00:00:27
and aggregate forecasts about
00:00:32
events all over the world and they
00:00:34
include military conflicts
00:00:36
elections pandemics
00:00:41
refugee flows and even things like the
00:00:43
price of commodities
00:00:45
now what we did was to recruit thousands
00:00:49
of forecasters
00:00:50
from blogs and professional societies
00:00:53
and
00:00:53
research centers and so forth and
00:00:57
had them make forecasts over
00:01:01
a period of a year and they were given
00:01:04
questions
00:01:05
every two weeks they logged on to a
00:01:07
website
00:01:08
where they made their predictions and
00:01:10
they went back to update
00:01:12
their forecasts as often as they wanted
00:01:15
now we didn't really know what to do to
00:01:17
improve forecasts so
00:01:18
we did what came naturally and that was
00:01:20
to run experiments
00:01:23
we found that three factors did
00:01:26
extremely well
00:01:27
one was training people we devised a
00:01:31
one hour probability training module and
00:01:34
and that seemed to improve predictions
00:01:37
we put people in teams as opposed to
00:01:40
having them work
00:01:41
individually and that improved
00:01:43
predictions
00:01:44
the interaction the information sharing
00:01:48
the debates about rationales
00:01:52
were boosted accuracy
00:01:56
more over and beyond the benefits of
00:01:59
working alone
00:02:02
and lastly we found that tracking was
00:02:06
a huge booster of forecasting accuracy
00:02:09
at the end of each year we took the top
00:02:11
two percent
00:02:12
of thousands of forecasters put them
00:02:14
together in elite groups
00:02:16
and gave them the title of super
00:02:18
forecasters
00:02:20
and these people uh
00:02:23
increased their accuracy in ways in in
00:02:26
more than we could possibly have
00:02:28
imagined
00:02:29
they interacted more they looked for
00:02:31
more information
00:02:33
and the net result was amazing
00:02:37
in fact they helped us win the
00:02:38
tournament three years
00:02:40
in a row
00:02:46
businesses everyone for that matter
00:02:48
relies on predictions
00:02:51
and we know a lot more
00:02:54
about predictions than
00:02:58
we did prior to this tournament in the
00:03:01
in the world of business people care
00:03:04
about
00:03:05
whether to invest in research or expand
00:03:07
to a new market and they care about what
00:03:10
consumers will want what extensions
00:03:12
they'll
00:03:13
prefer when products will be ready for
00:03:17
distribution
00:03:18
and businesses can
00:03:22
i think use the insights from this
00:03:24
tournament to
00:03:26
make better predictions not everything
00:03:31
general generalizes smoothly but
00:03:35
there are a lot of both psychological
00:03:38
and statistical insights
00:03:40
that we know now make predictions better
00:03:47
you know i was surprised by an awful lot
00:03:49
of things in this tournament i was
00:03:52
surprised that uh our training module
00:03:54
worked
00:03:55
it's it's tough to to design a module
00:03:58
that
00:04:00
has any effect on judgmental accuracy
00:04:05
i was very surprised that teams worked
00:04:07
better than
00:04:08
independent forecasters
00:04:11
i read the wisdom of the crowds and i
00:04:14
assumed that
00:04:15
independent forecasters would would do
00:04:17
better and heirs would average out but
00:04:20
the benefits of sharing information
00:04:23
and talking about rationales outweighed
00:04:26
the
00:04:27
the benefits of of independence so
00:04:31
um in our case uh teams worked better
00:04:35
and i was super surprised about the
00:04:36
effects of super forecasters
00:04:41
it's it's like tracking kids in schools
00:04:44
and putting the best ones together and
00:04:46
the the synergy that came from that was
00:04:49
phenomenal
00:04:54
yeah and i think there's several reasons
00:04:55
why our teams work so well
00:04:58
one of them was that they worked
00:05:01
online so it's tough to be a dominant
00:05:04
bully
00:05:05
online people logged on
00:05:08
whenever they felt like it and so
00:05:11
things were sequential they weren't
00:05:14
simultaneous
00:05:15
where i think groupthink is more likely
00:05:16
to occur
00:05:19
they had a lot of respect for each other
00:05:21
the the
00:05:23
forecasters and i think that's another
00:05:26
big
00:05:27
part of a good recipe
00:05:33
well i think companies can improve their
00:05:36
predictions
00:05:38
and i think they can create their own
00:05:40
set of super forecasters
00:05:42
and they can do better
00:05:46
at forecasting the future um
00:05:50
and better is really relative i mean
00:05:53
all you have to do is be better than the
00:05:56
next guy
00:05:57
we're not looking for perfection here
00:05:58
and obviously we're not going to get it
00:06:01
um in the case of the u.s government
00:06:03
policy makers face decisions that
00:06:06
involve billions of dollars and
00:06:08
thousands of lives
00:06:10
and in in these cases you know the
00:06:12
stakes are so high
00:06:14
that even a tiny little edge is
00:06:18
huge progress um
00:06:22
the the one of the implications
00:06:25
of of our work i think is that good
00:06:28
predictions involve both psychology
00:06:30
and statistics it's a combination of
00:06:33
understanding the person
00:06:34
and then understanding the
00:06:37
the aspects of of uh
00:06:41
statistical distributions and
00:06:42
statistical information
00:06:45
we know now how to uh do much better
00:06:49
at devising algorithms that
00:06:53
aggregate multiple forecasts and we also
00:06:55
know a lot about what
00:06:57
conditions or environments people
00:07:00
bring out the best of of of individual
00:07:03
forecasters
00:07:09
well every day we hear predictions
00:07:13
we have pundits and experts and gurus
00:07:15
and specialists
00:07:17
making predictions about what will
00:07:19
happen in the future
00:07:21
and many of those predictions are so
00:07:23
vaguely stated
00:07:24
that we could never in a million years
00:07:27
figure out how to test them
00:07:30
there are statements like there may be
00:07:33
an
00:07:34
increase in conflict in yemen in the
00:07:36
next two weeks
00:07:37
or the situation in baghdad
00:07:40
will get worse before it gets better or
00:07:43
something like that that's
00:07:44
that's not a question or a statement
00:07:47
that
00:07:47
passes the clairvoyance test you could
00:07:49
you can't figure out later
00:07:51
who's right and who's wrong
00:07:55
now we look to these people for
00:07:58
advice we look to them for insights
00:08:02
and we're getting really very little
00:08:06
from those vague predictions and i think
00:08:10
the way we can do better is to keep
00:08:12
score
00:08:14
we've got to have questions or
00:08:17
predictions stated
00:08:18
in such a way that they're testable and
00:08:20
then we can find out who's right and
00:08:22
who's wrong
00:08:23
and how we can learn to do better
00:08:29
we are in a fortunate position of being
00:08:33
able to have
00:08:34
an empirical basis for the claims we
00:08:37
make
00:08:38
we have we know what works based on
00:08:41
data based on experiments and many of
00:08:44
the
00:08:46
books and methods and techniques for
00:08:49
doing better forecasting
00:08:50
are simply not tested so clearly
00:08:54
the empirical side of things is
00:08:57
something that
00:08:58
is unique to our project i think
00:09:03
we've learned a lot about
00:09:06
uh what makes things better we've
00:09:09
learned for example that
00:09:11
survey formats with statistical
00:09:14
algorithms combining the forecast can
00:09:17
outperform prediction markets
00:09:20
we've learned that if you measure
00:09:23
probabilities
00:09:24
while people are trading in the market
00:09:26
you also ask them
00:09:28
what's your probability that event x
00:09:31
will occur
00:09:32
you can do better at forecasting
00:09:34
accuracy by combining
00:09:36
both the prices and the probabilities
00:09:38
and these things are both surprising
00:09:40
from an economic perspective
00:09:43
we know a bit more about individual
00:09:45
differences that correlate with
00:09:47
forecasting accuracy so we can say a
00:09:49
little bit more about
00:09:50
who the best forecasters are especially
00:09:52
in this geopolitical
00:09:53
context not surprisingly they're
00:09:57
they tend to be smart they tend to know
00:09:59
a lot they have a lot of political uh
00:10:01
knowledge but they are also
00:10:05
more likely to be actively open-minded
00:10:07
thinkers
00:10:09
they are more analytical they
00:10:12
are more likely to take a scientific
00:10:15
world view
00:10:16
they're more likely to take multiple
00:10:19
perspectives
00:10:20
on a question and use multiple reference
00:10:22
classes
00:10:24
they're more likely to use probabilities
00:10:27
in a more
00:10:28
granular or
00:10:31
nuanced fashion they're more likely to
00:10:33
say 17
00:10:34
and 83 percent rather than 20 and 80
00:10:37
percent
00:10:38
and it turns out that extra granularity
00:10:41
has information
00:10:42
in it if you round forecasts up
00:10:46
to the nearest 10 20 30 40
00:10:50
most forecasters do worse and
00:10:54
that suggests there's there's valuable
00:10:56
signals
00:10:57
in that granularity not just
00:11:00
noise
00:11:06
well uh the irb tournament will close
00:11:09
on june 2nd in in another month
00:11:13
but that's not the end of forecasting we
00:11:17
will be opening
00:11:18
the good judgment project will be
00:11:20
opening a public tournament this fall
00:11:23
and we have lots of hypotheses to test
00:11:26
we'll need lots of volunteers and we'd
00:11:28
love to have people
00:11:30
affiliated with wharton so if this is
00:11:33
something that interests you
00:11:35
or you know people who would be
00:11:37
interested go to www
00:11:40
dot goodjudgementproject.com
00:11:44
to get more information and we'd be
00:11:47
extremely grateful and delighted to have
00:11:49
you join us
00:12:13
you
