00:00:02
um so this particular research involved
00:00:04
looking at the efficacy of retention
00:00:06
campaigns so many companies do retention
00:00:08
campaigns this is basically campaigns
00:00:10
which are targeted towards consumers
00:00:12
they want to keep those consumers so
00:00:13
they might give incentives for example
00:00:15
you might kind of say for your next
00:00:16
purchase you might get ten dollars off
00:00:18
so these campaigns are very common and
00:00:20
many companies think that these
00:00:21
campaigns work do they really work so we
00:00:24
had an opportunity to work with a cell
00:00:26
phone company which was trying to do
00:00:27
these retention campaigns what was great
00:00:29
for us is that they actually did a field
00:00:31
experiment for one group of customers
00:00:33
who were randomly selected they did a
00:00:35
retention campaign this was a pricing
00:00:37
plan campaign for another group
00:00:38
customers they did nothing what did we
00:00:40
find the group of customers who are
00:00:43
actually in the retention campaign
00:00:45
actually left a lot more in fact
00:00:47
staggeringly a lot more this was pretty
00:00:49
bad for the company so what's the big
00:00:51
picture do retention campaigns work
00:00:53
actually on mass they might not work
00:00:55
what we found was a more nuanced finding
00:00:57
they work but only if you're that
00:00:58
targeted so do retention campaigns but
00:01:01
do it in a targeted way
00:01:06
so the key takeaways again focus on
00:01:08
retention campaigns i think primarily as
00:01:11
many companies start thinking about
00:01:12
these campaigns the typical thing that
00:01:14
most companies do is do a mass-marketed
00:01:16
campaign send a retention package to
00:01:19
every one of their customers why because
00:01:20
it's easy to do they don't have to think
00:01:22
too much they'll send it to everyone
00:01:24
what did we find sometimes actually
00:01:26
sending campaigns to people might
00:01:28
actually make them start questioning
00:01:30
their own behavior for instance in this
00:01:33
particular case for the cell phone
00:01:34
company when they sent a retention
00:01:36
campaign which was basically about
00:01:38
looking at people's behavior their usage
00:01:39
patterns and saying look there might be
00:01:41
other plans that might be better for you
00:01:43
it made many customers question
00:01:46
whether a they were getting a good deal
00:01:48
and if they're getting a good deal here
00:01:49
why not look elsewhere so what that
00:01:51
suggests is retention campaigns must
00:01:53
often be targeted campaigns so think
00:01:55
carefully about who in your customer
00:01:57
base might be likely to leave
00:01:59
don't do it mass do it on a targeted
00:02:02
basis
00:02:06
so on the surface when you think about
00:02:08
it again going back to this idea of
00:02:09
retention campaigns most people think
00:02:11
the retention campaigns work so let me
00:02:13
give you some hard numbers what we found
00:02:15
so in the study that we did when we did
00:02:17
the retention campaign
00:02:18
and this was for a bunch of customers
00:02:20
from a telecom service we monitored
00:02:22
their behavior three months before the
00:02:23
campaign and three months after the
00:02:25
campaign
00:02:26
what we found was when when these
00:02:28
customers who were in the retention
00:02:29
campaign we looked at what happens three
00:02:31
months afterwards 10 of these customers
00:02:34
left the service as opposed to a control
00:02:36
group where no campaigns were done six
00:02:38
percent left the service so you would
00:02:40
imagine on the surface four percent is a
00:02:42
staggeringly high rate of churn
00:02:44
now should companies not do these
00:02:46
campaigns that's not what we found what
00:02:48
we found was on average yes
00:02:50
it's very hard to kind of find evidence
00:02:52
whether retention campaigns work but it
00:02:54
was very easy to find evidence for who
00:02:56
it worked for there were many sets of
00:02:58
customers which had certain
00:03:00
characteristics which are very easy for
00:03:02
firms to observe for whom these
00:03:03
campaigns work so surprising fact was on
00:03:06
average they didn't work
00:03:08
and for many companies i think this is
00:03:09
an eye opener because what they should
00:03:11
be thinking carefully about is how
00:03:13
should we customize even our retention
00:03:14
campaigns
00:03:19
so i think this is very interesting in
00:03:21
terms of customization what is uh what
00:03:23
is very good nowadays is especially many
00:03:25
companies who are data driven they have
00:03:27
a lot of information about their
00:03:28
customers so for instance in the mobile
00:03:30
phone context companies routinely gather
00:03:32
information about past consumption in
00:03:35
our case we had information for example
00:03:37
in the past three months that we had
00:03:38
been collecting you know what was the
00:03:40
level of overage which is the amount
00:03:41
that people are going consuming over the
00:03:43
number of plant minutes we had
00:03:45
information about how much variability
00:03:47
do they have you know in one month are
00:03:48
they using 100 minutes in the other
00:03:50
month are they suddenly using 200
00:03:51
minutes or two you know 500 minutes so
00:03:54
these are very observable types of
00:03:55
patterns that you can find in your own
00:03:57
customer base so many times what we
00:03:59
found was cutting up the data on
00:04:01
customers using these observable
00:04:02
patterns might be a great way to
00:04:04
customize so let me give you a specific
00:04:06
example
00:04:07
in our case what we found was that
00:04:09
people who were consuming a lot way
00:04:11
above the number of minutes that they
00:04:13
had consumers who had a huge amount of
00:04:15
variation
00:04:17
consumers who had negative trend over
00:04:18
the time that they were consuming less
00:04:20
and less
00:04:21
those were the people who are likely to
00:04:22
leave
00:04:23
and it cannot be surprising for many of
00:04:26
these customers doing the retention
00:04:27
campaign actually made them more likely
00:04:30
to leave
00:04:31
so sometimes it's best to let sleeping
00:04:33
dogs lie
00:04:38
now i think one of the interesting thing
00:04:39
that's going on nowadays especially if
00:04:41
you think about big data analytics and
00:04:42
all of that is companies are rapidly
00:04:44
experimenting
00:04:46
so many many companies out there do
00:04:48
routinely do a b testing now what did we
00:04:50
find here what we found here was it's
00:04:52
not enough to just do a b testing it's
00:04:54
important to analyze the data correctly
00:04:56
so let me give you a specific example in
00:04:58
our a b test one group was the people
00:05:00
who received recommendations the other
00:05:02
group was people who received no
00:05:03
recommendations
00:05:04
on the surface people who received
00:05:06
recommendations in that group and who
00:05:08
accepted them actually churned less
00:05:11
as compared to the control group
00:05:13
now you would imagine well then one
00:05:15
might think that the
00:05:16
retention campaign worked but what's
00:05:18
important to remember is that customers
00:05:20
decided to accept the campaign so there
00:05:22
is self-selection there so even for
00:05:25
people who are exposed to the campaign
00:05:27
and decided not to accept it they
00:05:29
actually churned a lot more so the very
00:05:31
fact that they were exposed to the
00:05:32
campaign changed their behavior so it's
00:05:35
important when companies are doing a b
00:05:37
testing they think carefully about
00:05:38
what's randomized and what's
00:05:40
self-selected by consumers glossing over
00:05:43
the fact can actually lead them to think
00:05:45
that some campaigns are successful when
00:05:47
they are not
00:05:48
and a b testing is something that we
00:05:50
believe is going to be rapidly taking
00:05:52
off in this date of in the day of data
00:05:54
analytics
00:05:55
but again doing a b testing is easy
00:05:58
analyzing it and interpreting the
00:05:59
results correctly is way more important
00:06:06
so a lot of people including some of my
00:06:07
own work has looked at pricing plans and
00:06:09
how customers choose among pricing plans
00:06:12
what's a big problem there many of these
00:06:14
things are self-selected by consumers so
00:06:16
from a company's perspective if they
00:06:18
want to look at the causal impact of
00:06:20
what happens when they give pricing
00:06:21
plans it's difficult to do so a prior
00:06:23
right because there's an aspect of
00:06:25
consumer self-selection involved so how
00:06:27
do we get around this convincing a
00:06:29
company that they should do a field
00:06:30
experiment that is the gold standard of
00:06:33
causal interpretation what we ended up
00:06:35
doing was doing a field experiment where
00:06:37
again people were given some pricing
00:06:38
plans and some people were not given any
00:06:40
recommendations so that helped us give a
00:06:43
causal interpretation which was very
00:06:45
very hard to do so just using secondary
00:06:47
data which many researchers have done
00:06:53
so i think what i'd like to continue on
00:06:55
is work more on this area of pricing
00:06:57
plans and recommendations i'm actually
00:06:58
working with a company down in austin
00:07:00
which is starting to look at energy
00:07:02
consumption and the whole idea of smart
00:07:04
meters how do we make people understand
00:07:07
that when they are they are consuming
00:07:08
electricity they are on different
00:07:09
tariffs and different tiers what we're
00:07:11
trying to do actually is to do a field
00:07:12
experiment again making things salient
00:07:14
to consumers and seeing how they might
00:07:17
change their energy consumption over
00:07:18
time or over days depending upon how
00:07:21
much they're consuming now so that's the
00:07:22
next plan
00:07:46
you
