00:00:02
um so this particular research involved looking at the efficacy of retention campaigns so many companies do retention
00:00:08
campaigns this is basically campaigns which are targeted towards consumers they want to keep those consumers so
00:00:13
they might give incentives for example you might kind of say for your next purchase you might get ten dollars off
00:00:18
so these campaigns are very common and many companies think that these campaigns work do they really work so we
00:00:24
had an opportunity to work with a cell phone company which was trying to do these retention campaigns what was great
00:00:29
for us is that they actually did a field experiment for one group of customers who were randomly selected they did a
00:00:35
retention campaign this was a pricing plan campaign for another group customers they did nothing what did we
00:00:40
find the group of customers who are actually in the retention campaign actually left a lot more in fact
00:00:47
staggeringly a lot more this was pretty bad for the company so what's the big
00:00:51
picture do retention campaigns work actually on mass they might not work what we found was a more nuanced finding
00:00:57
they work but only if you're that targeted so do retention campaigns but do it in a targeted way
00:01:06
so the key takeaways again focus on retention campaigns i think primarily as many companies start thinking about
00:01:12
these campaigns the typical thing that most companies do is do a mass-marketed campaign send a retention package to
00:01:19
every one of their customers why because it's easy to do they don't have to think
00:01:22
too much they'll send it to everyone what did we find sometimes actually sending campaigns to people might
00:01:28
actually make them start questioning their own behavior for instance in this particular case for the cell phone
00:01:34
company when they sent a retention campaign which was basically about looking at people's behavior their usage
00:01:39
patterns and saying look there might be other plans that might be better for you
00:01:43
it made many customers question whether a they were getting a good deal and if they're getting a good deal here
00:01:49
why not look elsewhere so what that suggests is retention campaigns must often be targeted campaigns so think
00:01:55
carefully about who in your customer base might be likely to leave don't do it mass do it on a targeted
00:02:02
basis so on the surface when you think about it again going back to this idea of
00:02:09
retention campaigns most people think the retention campaigns work so let me give you some hard numbers what we found
00:02:15
so in the study that we did when we did the retention campaign and this was for a bunch of customers
00:02:20
from a telecom service we monitored their behavior three months before the campaign and three months after the
00:02:25
campaign what we found was when when these customers who were in the retention campaign we looked at what happens three
00:02:31
months afterwards 10 of these customers left the service as opposed to a control
00:02:36
group where no campaigns were done six percent left the service so you would imagine on the surface four percent is a
00:02:42
staggeringly high rate of churn now should companies not do these campaigns that's not what we found what
00:02:48
we found was on average yes it's very hard to kind of find evidence whether retention campaigns work but it
00:02:54
was very easy to find evidence for who it worked for there were many sets of customers which had certain
00:03:00
characteristics which are very easy for firms to observe for whom these campaigns work so surprising fact was on
00:03:06
average they didn't work and for many companies i think this is an eye opener because what they should
00:03:11
be thinking carefully about is how should we customize even our retention campaigns
00:03:19
so i think this is very interesting in terms of customization what is uh what is very good nowadays is especially many
00:03:25
companies who are data driven they have a lot of information about their customers so for instance in the mobile
00:03:30
phone context companies routinely gather information about past consumption in our case we had information for example
00:03:37
in the past three months that we had been collecting you know what was the level of overage which is the amount
00:03:41
that people are going consuming over the number of plant minutes we had information about how much variability
00:03:47
do they have you know in one month are they using 100 minutes in the other month are they suddenly using 200
00:03:51
minutes or two you know 500 minutes so these are very observable types of patterns that you can find in your own
00:03:57
customer base so many times what we found was cutting up the data on customers using these observable
00:04:02
patterns might be a great way to customize so let me give you a specific example
00:04:07
in our case what we found was that people who were consuming a lot way above the number of minutes that they
00:04:13
had consumers who had a huge amount of variation consumers who had negative trend over
00:04:18
the time that they were consuming less and less those were the people who are likely to
00:04:22
leave and it cannot be surprising for many of these customers doing the retention
00:04:27
campaign actually made them more likely to leave so sometimes it's best to let sleeping
00:04:33
dogs lie now i think one of the interesting thing that's going on nowadays especially if
00:04:41
you think about big data analytics and all of that is companies are rapidly experimenting
00:04:46
so many many companies out there do routinely do a b testing now what did we find here what we found here was it's
00:04:52
not enough to just do a b testing it's important to analyze the data correctly
00:04:56
so let me give you a specific example in our a b test one group was the people who received recommendations the other
00:05:02
group was people who received no recommendations on the surface people who received
00:05:06
recommendations in that group and who accepted them actually churned less as compared to the control group
00:05:13
now you would imagine well then one might think that the retention campaign worked but what's
00:05:18
important to remember is that customers decided to accept the campaign so there is self-selection there so even for
00:05:25
people who are exposed to the campaign and decided not to accept it they actually churned a lot more so the very
00:05:31
fact that they were exposed to the campaign changed their behavior so it's important when companies are doing a b
00:05:37
testing they think carefully about what's randomized and what's self-selected by consumers glossing over
00:05:43
the fact can actually lead them to think that some campaigns are successful when
00:05:47
they are not and a b testing is something that we believe is going to be rapidly taking
00:05:52
off in this date of in the day of data analytics but again doing a b testing is easy
00:05:58
analyzing it and interpreting the results correctly is way more important so a lot of people including some of my
00:06:07
own work has looked at pricing plans and how customers choose among pricing plans
00:06:12
what's a big problem there many of these things are self-selected by consumers so
00:06:16
from a company's perspective if they want to look at the causal impact of what happens when they give pricing
00:06:21
plans it's difficult to do so a prior right because there's an aspect of
00:06:25
consumer self-selection involved so how do we get around this convincing a company that they should do a field
00:06:30
experiment that is the gold standard of causal interpretation what we ended up doing was doing a field experiment where
00:06:37
again people were given some pricing plans and some people were not given any recommendations so that helped us give a
00:06:43
causal interpretation which was very very hard to do so just using secondary data which many researchers have done
00:06:53
so i think what i'd like to continue on is work more on this area of pricing
00:06:57
plans and recommendations i'm actually working with a company down in austin
00:07:00
which is starting to look at energy consumption and the whole idea of smart meters how do we make people understand
00:07:07
that when they are they are consuming electricity they are on different tariffs and different tiers what we're
00:07:11
trying to do actually is to do a field experiment again making things salient to consumers and seeing how they might
00:07:17
change their energy consumption over time or over days depending upon how much they're consuming now so that's the
00:07:22
next plan you