00:00:01
our guest today is Karthik hosonaga a professor of technology digital business and marketing at Wharton and we are
00:00:09
speaking with him about his recent book titled a human's guide to machine intelligence how algorithms are shaping
00:00:16
our lives and how we can stay in control Karthik welcome to knowledge at Wharton
00:00:21
thank you so much for speaking with us today well cool thanks for having me it's
00:00:25
always a pleasure to talk to you and the knowledge is important GRU thanks so these days there's a growing buzz
00:00:31
about artificial intelligence and machine learning and quite a few books I have have come out recently on these
00:00:37
topics in all the conversations that are going on what do you think are some of the points that are being overlooked are
00:00:45
not sufficiently emphasized and how does your book seek to fill that gap yeah clearly there's a lot of buzz around AI
00:00:54
and machine learning which is a subfield of AI I think the conversation tends to
00:01:02
you know either glorify the technology or in many instances lately create a lot of fear-mongering around it and you know
00:01:15
I don't think the conversation has focused on you know what's the solution
00:01:21
how are we going to work with AI and especially in the context of making decisions and so my book is focused on
00:01:29
making decisions through intelligent algorithms and certainly we have various kinds of AI but one of the core
00:01:36
questions when it comes to AI is are we going to use AI to make decisions if so are we going to use it in a decision
00:01:43
support way are we going to have the AI make decisions autonomously if so what can go wrong what can go well and how do
00:01:51
we manage this because we know AI has a lot of potential but I think there will be some growing pains on our way there
00:01:58
and so those growing pains is what I focus on how can algorithmic decisions go wrong and how do we make sure that we
00:02:06
have control over the narrative of how technology impacts the decisions that are made for us or about us
00:02:14
I'd love to come back to the part of our decision-making an algorithmic algorithmic decision-making but I really
00:02:22
love the way you began the book with some very striking examples about chatbots and and how they interact with
00:02:29
humans I wonder if you could use that illustration to talk a little bit about how human beings interact with
00:02:37
algorithms and what some of the implications are I began the book with a description of Microsoft's experience
00:02:44
with a chat part called Xiao Bing in China it's called Xiao Bing and elsewhere in the world it's called Xiao
00:02:50
Weis this was a chat board created in the avatar of a teenage girl and it's meant to engaged in engage in fun
00:03:01
playful conversations with young adults and teenagers and this chat pot has about 40 million followers in China and
00:03:11
the reports say that roughly a quarter of those followers have said I love you to Xiao Weis so that's the kind of
00:03:18
affection and following Xiao Weis has so inspired by the success of Xiao Weis in
00:03:23
China Microsoft decided to test a similar chat bot in the US and they created a chat bot in English which
00:03:31
would engage again in fun playful conversations and targeted once again at young adults and teenagers they launched
00:03:38
it on twitter under the name Tay ta y and this chat BOTS experience was actually very different and it's it was
00:03:48
very short-lived experience as well because within an hour of launching the chat pod turned sexist racist fascist it
00:03:57
tweeted very offensively and said things like Hitler was right and Microsoft shut
00:04:04
it down within 24 hours and later that year mi t--'s Technology Review rated
00:04:09
Microsoft Day as the worst technology of the year and that incident you know opened up this question for me which was
00:04:19
how can two similar chart parts or pieces of AI built by the same company produce such
00:04:30
different results and what does that say about you know our decision to to use algorithms for a lot of our decisions in
00:04:41
our personal and professional lives and that's what helped start this exploration into use of AI the extent to
00:04:50
which AI can be predictable biased as in the case of Microsoft a and then of course how do we what does that mean in
00:04:57
terms of using these systems to make significant decisions for us so why did T experiences differ so dramatically and
00:05:05
is anything that can be done about that one of the insights that I got as I was writing this book as I was trying to
00:05:14
explain the differences in behavior of these two chatbots was actually from human psychology
00:05:22
so psychologists describe human behavior in terms of our nature and our nurture and so our nature is our genetic code
00:05:29
and nurture is our environment and so psychologists will attribute problematic issues like alcoholism partly to nature
00:05:39
and partly to nurture and when I was looking at algorithms I realized algorithms have nature and nurture as
00:05:45
well so the nature for algorithms is not genetic code but the code that the engineer actually codes in or writes and
00:05:54
that's the logic of the algorithm the nurture for the algorithm is the data from which algorithms learn and so
00:06:02
increasingly as we are moving towards machine learning we're moving from a world where engineers used to program
00:06:11
the end-to-end logic of an algorithm would actually specify what happens when any situation happens if this happens
00:06:17
you respond this way if that happens you respond a different way and so it used to be all nature because the programmer
00:06:23
gave all the very minut specifications of how the algorithm will work but as we move towards machine learning we're kind
00:06:31
of telling algorithms here's data learn from it and so nature starts to become
00:06:36
less important and nurture starts to dominate so if you look at what happened between tea and showers the
00:06:44
difference is in terms of their training data in some ways and in particular in the case of showers showers was created
00:06:51
to mimic how people converse on Tay and so it was picking up how people are talking to it and it would reflect that
00:07:00
and so there were many intentional efforts to trip day and so there was nurture there as well and part of it was
00:07:08
nature as well meaning that the court could have specified certain rules the court could have specified rules like do
00:07:16
nots say the following kinds of things or do not get into discussions of these topics and so on so it's a bit of both
00:07:22
and I think that's what in general rogue algorithmic behavior comes down to that
00:07:29
come back a little later to the whole question of what happens go rogue yeah but I wanted to chat a little bit about
00:07:39
the way in which this algorithmic decision-making itself has changed and it wasn't there was a time when
00:07:49
decision-making algorithm decision-making seemed to be almost like Amazon will tell you what books you
00:07:55
should read or Netflix will recommend which movies you should watch but because of AI algorithmic algorithmic
00:08:03
decision-making has become a lot more complex and I was wondering if you could offer some examples and what are some of
00:08:10
the implications on the choices that we make or don't make as a result of this
00:08:16
yeah algorithms pervade our lives and sometimes we see it like Amazon's recommendations and sometimes we don't
00:08:23
realize it but they have a huge impact on decisions we make on Amazon for example over 1/3 of the choices that we
00:08:31
make are influenced by algorithmic recommendations people who bought this also bought those people who viewed this
00:08:36
eventually bought that and so on on Netflix over 80% of the viewing activity is driven by algorithmic recommendations
00:08:44
they also drive decisions such as who we date and Mary when you look at applications or apps like tinder which
00:08:52
actually ABB's we're algorithms create most of the matches over there they're also at the workplace you know
00:08:59
for example you make you apply for our alone mortgage approval decisions are made by algorithms increasingly you know
00:09:07
if you apply for a job resumes screaming algorithms are deciding which ones to invite for an interview and they're
00:09:14
making life-and-death decisions as well there for example being used in the criminal justice system in courtrooms in
00:09:21
the u.s. there are algorithms that predict the likelihood that the defendant will reoffend so that judges
00:09:27
can make sentencing decisions in medicine we're moving towards personalized medicine so that two people
00:09:33
with the same symptoms might not get the same treatment it might be customized based on their DNA profile algorithms
00:09:39
again will guide the doctors on those decisions and also we're moving as AI has progressed and advanced we're moving
00:09:49
to a point where the algorithms don't merely offer decision support they can
00:09:53
function autonomously as well and driverless cars are a great example of that where we're trying to pretty much
00:10:00
say you can automate the whole process and and have algorithms function without a human making the final sort of
00:10:09
decision so as more algorithms influence or make more and more decisions is there anything like free will in the
00:10:18
world anymore well so free will is an interesting concept and for the most part I used to think of free will in a
00:10:25
very philosophical sense right and philosophers have argued we don't even have free will but that was in a very
00:10:31
different very as I said philosophical interpretation but I think we have a literal interpretation of free will now
00:10:37
in the context of algorithms which is are you making the final choice and I just said you know a third of your
00:10:43
choices are driven on Amazon by recommendations 80% of viewing activities on Netflix are driven by
00:10:50
algorithmic recommendations at YouTube seventy percent of the time people spend on YouTube is driven by algorithmic
00:10:56
recommendations so it doesn't feel like algorithms are merely recommending to us
00:11:01
what we want and we make decisions I mean think about a search on Google you take the most
00:11:06
Oh Tarek of search terms like vintage toy model trains or something like that you will still find hundreds of
00:11:14
thousands perhaps even millions of results for that search term we might see less than 0.01% of the search
00:11:21
results because rarely do we even cross page one the algorithm has decided which
00:11:25
pages we look at so yes they're making a lot of choices for us so do we have free
00:11:31
well at some level yes but mostly I'm going to say we don't have the level of
00:11:37
independent decision-making that we think we do because most of us think we get these recommendations we not
00:11:44
politely and then we do what we want but indeed the algorithms are not jiggles in
00:11:48
very interesting ways and and I don't think we have the level of independent
00:11:53
decision-making or choice that we think we do mostly that's a good thing in some
00:11:57
ways because they're saving us time so we can focus on leisure instead of wasting our time sifting through lots of
00:12:05
alternatives but sometimes we become very passive about how we use algorithms and becoming that passive about
00:12:11
algorithm use can have consequences so let's talk a little bit about those consequences okay especially unintended
00:12:18
consequences yes there's a fascinating part on your book about that and how do
00:12:23
design choices lead to unintended consequences I was wondering if you could speak about that yes so when I
00:12:31
mention unintended consequences in the book I'm referred referring to situations where you know you're trying
00:12:38
to optimize some aspect of a decision and perhaps you manage to improve that really well but then something else goes
00:12:47
wrong so an example might be that when Facebook was manually curating its trending stories through human editors
00:12:57
as an editor you will appreciate that it's real work right and they had people
00:13:02
doing that but then Facebook was accused of having a left-leaning bias that these
00:13:08
editors were choosing left-leaning stories and curating those more often so they said you know algorithms can't be
00:13:15
accused of a political bias so they used an algorithm to curate this they tested
00:13:19
it for political bias it did not have any political bias but there's something else it did which they
00:13:25
hadn't explicitly tested for which is as we know it curated fake new stories and
00:13:29
and circulated them so that's an example of unintended consequences and an algorithm design can drive that in many
00:13:38
ways you know I've done a lot of work on recommendation systems and how they
00:13:42
influence the kinds of products we consume the kinds of media we consume and I've specifically studied two kinds
00:13:49
of recommendation algorithms one kind of algorithm is based or it's like the
00:13:56
Amazon people who bought this also bought this so it's based on social curation what are the what are others
00:14:03
consuming the other kind of algorithm attempts to understand at a deep level what is it that I'm recommending and the
00:14:10
algorithm is recommending and tries to find items that are very similar to the users interest an example of that would
00:14:18
be Pandora so Pandora's music recommendations are not people who like this song also like these other songs
00:14:25
Pandora actually has very detailed information over 150 attributes for each song musical attributes like how
00:14:34
rhythmic is the song how much instrumentation is there in the music and every time you say you like a music
00:14:40
a song or you don't like a song they look at the musical qualities of the song and then they adjust their
00:14:46
recommendations based on other songs which have attributes similar to what you have now I looked at both these
00:14:51
designs and I looked at which design is more helpful in helping us find let's
00:14:58
say indie songs or very novel and niche books or movies and at the time we did the study and this was some time back
00:15:07
the conventional wisdom was that all of these algorithms help in pushing the long tail meaning these nice novel items
00:15:16
or indie songs that nobody's heard of and what I found was that these designs
00:15:20
were very different the algorithm that looks at the look said you know what others are consuming people who bought
00:15:28
this also bought this it has a popularity bias because it's trying to recommend stuff
00:15:33
that others are consuming and so it tends to lean towards popular items and so it cannot truly recommend those
00:15:40
hidden gems but an algorithm like the Pandora algorithm is it doesn't have popularity as a basis for
00:15:49
recommendations so it tends to do better and that's why what we've seen now is
00:15:52
companies like Spotify Netflix and many others have changed the design of their algorithms they've combined the two
00:15:59
approaches they've combined the the social appeal of a system that looks at what others are consuming and the
00:16:09
ability of the other design to surface hidden gems let's go now to the point
00:16:17
you were up earlier about algorithms going rogue yes why does that happen and what can be done about it yeah if we
00:16:25
look at let me point a couple example examples of algorithms doing rolled and then we'll talk about why this happens
00:16:31
so I mentioned algorithms are used in courtrooms in the u.s. in the criminal justice system in 2016 there was a
00:16:39
report or study done by ProPublica which is a nonprofit organization they looked
00:16:45
at algorithms used in courtrooms and they found that these algorithms have a race bias specifically they found that
00:16:51
these algorithms were twice as likely to falsely predict future criminality in a
00:16:59
black defendant than a white defendant and this was race bias in that algorithm late last year Reuters carried a story
00:17:07
about Amazon trying to use algorithms to screen job applications and the you know
00:17:15
again Amazon gets a million plus job applications they hire hundreds of thousands of people you know over the
00:17:21
last few years they've done that it's hard to do that manually and so you need
00:17:26
algorithms to help automate some of this but they found that the algorithms tended to have a gender bias they tended
00:17:33
to reject female applicants more often even when the qualifications were similar now Amazon ran the test and
00:17:40
concluded and realized that there are savvy companies so they decided not to roll this out there's probably many
00:17:46
other companies that using algorithms to screen resumes and they might be prone to race bias gender
00:17:52
bias and so on so I've mentioned two examples now race bias gender bias I talked about fake news on on Facebook so
00:18:01
there's many examples in terms of why they go rogue there's a couple reasons I
00:18:07
can share one is as we have gone from those old traditional algorithms where the programmer wrote up the algorithm
00:18:17
end to end and we've moved towards machine learning we have created algorithms that are usually more
00:18:24
resilient because these algorithms you know can perform much better but they're
00:18:31
prone to biases that exist in the data so if the data have biases so for example if you tell a resume screening
00:18:38
algorithm you know here's data on all those people who applied to our job and
00:18:44
here's the people we actually hired and here are the people who we promoted now
00:18:49
figure out who to invite for job interviews based on this data the algorithm will observe that in the past
00:18:55
you are rejecting more female applications or you were not promoting women at the workplace and will tend to
00:19:03
pick up that behavior so it tends to pick that up I think the other piece is that engineers in general tend to focus
00:19:10
narrowly on one or two metrics you know with a resume screening application you will tend to measure the accuracy of
00:19:18
your model and if it's highly accurate you'll roll it out but you don't
00:19:22
necessarily look at fairness and bias or in Facebook's example look at fake news and other
00:19:29
possibilities as well what are the challenges involved in autonomous algorithms making decisions on our
00:19:39
behalf well I think when you have autonomous algorithms making decision on our behalf I think one of the big
00:19:48
challenges is there is usually no human in the loop so we lose control and that is one one challenge and many studies
00:19:57
show that when we have limited control we are less likely to trust algorithms and so that
00:20:04
is one one aspect that is challenging the other piece about having autonomous algorithms is that again if there's a
00:20:11
human in the loop there's a greater chance that the user can detect certain problems and you know the likelihood
00:20:20
that problems get detected is therefore greater I'm really glad you brought up
00:20:25
the point about trusting algorithms because you tell this really fascinating story in the book about a patient who
00:20:31
gets diagnosed with definitely fever I wonder if you could share that story with our audience and and spell out some
00:20:39
of the implications for trust in algorithms and and what the implications are yeah the story that I share is that
00:20:48
of a patient walking into a doctor's office and the patient feels fine and healthy and the patient and doctor are
00:20:56
joking around and doctor eventually picks up the pathology report and he suddenly looks very serious and he's
00:21:07
says or informs the patient that I'm sorry to let you know that you have to happen early fever and the patient
00:21:16
hasn't heard of that Bernoulli fever and he asks what exactly is it and so the
00:21:20
doctor says it's a very rare disease and it's known to be fatal and so I suggest
00:21:28
that you have this tablet and it will you know reduce the chance that you will have any problems and you know he says
00:21:38
here you take this you know three times a day and and and then you go about your
00:21:42
life and I asked my readers if you know that story is something that if they were the patient they would feel
00:21:53
comfortable in that situation here's a disease you don't nothing about and
00:21:56
here's a solution you know nothing about and the doctor has given you the choice
00:22:03
and told you to go ahead but not given you very many details and with that I pose the question if an algorithm way to
00:22:13
again make this wreck that you have this rare disease and we want you to take this medication without
00:22:21
any information would you and of course Mukul you've read the book so you know Tappin le fever is not a
00:22:27
real disease I'm a fan of Sherlock Holmes I used to read it a lot as a kid and it's a disease in a one of the
00:22:35
Sherlock Holmes stories and you know that inspired me to consider this because even in the original Sherlock
00:22:43
Holmes story it turns out that the person who has definitely fever doesn't actually have it but setting that aside
00:22:50
it kind of brings up this question of transparency you know are we willing to trust decisions when we don't have
00:22:57
information about why a certain decision was made the way it was and what I highlight is sometimes we're seeking
00:23:04
more transparency from algorithms than humans but in practice lots of companies are imposing algorithmic decisions on us
00:23:12
without any information about why these decisions are being made and are we fine
00:23:18
with that and a lot of research shows that we're not fine with that research
00:23:22
for example that one PhD student then a PhD student at Stanford looked at an algorithm that would compute grades for
00:23:32
students and how did they do when they just got their score versus they got their score with an explanation and as
00:23:39
you expect when they have an explanation they trust it more then why is it that in in the real world there's a lot of
00:23:46
algorithms making decisions for us or about us and we have no transparency about those decisions and so I advocate
00:23:53
that we need a certain level of transparency with regard to for example what kinds of data were used to make the
00:24:00
decision so for example if we applied for a loan and the loan was rejected we would like to know why that was the case
00:24:07
if you applied for a job and it was rejected it would be helpful to know that the algorithm not only evaluated
00:24:15
what you submitted as part of your job application would also looked at your social media posts and so transparency
00:24:22
regarding what data was considered what were the key factors that drove a decision
00:24:27
is important but what's the end of the book you recommend an algorithmic Bill
00:24:33
of Rights what exactly is that and why is it necessary so the algorithmic Bill of Rights is a concept that I borrowed
00:24:42
from the Bill of Rights in the US Constitution and the history of the Bill of Rights is that when the founding
00:24:48
fathers were setting up or drafting the Constitution some people were worried that we're creating a very powerful
00:24:57
government here in the US and the Bill of Rights was created as a way to protect citizens now today we are in a
00:25:08
situation where there's a lot of talk about powerful tech companies you know
00:25:12
everywhere you see there's new stories big tech companies and what are we going
00:25:15
to do about them and so there's a sense that again consumers need certain protections and so the Bill of Rights is
00:25:24
targeted at that but before I talk about the Bill of Rights one aspect related to
00:25:28
the Bill of Rights I do want to address is that a lot of consumers feel that they're helpless against big tech and
00:25:35
against algorithms deployed by big tech and personally I feel that consumers do have some power and that power is in
00:25:45
terms of our knowledge our votes and our dollars you know knowledge is about you
00:25:49
know we shouldn't be passive users of technology we should be active and deliberate about it we should know how
00:25:55
it's changing decisions we are making or others are making about us and at some
00:26:00
level it seems like you know knowledge is fine but what can I do with that knowledge but I mean look at how
00:26:05
Facebook is changing their product design today and that changes you know support for encryption and so on is
00:26:11
because of push from the users and it shows that when users complain changes to happen votes there are another aspect
00:26:19
of that and votes are all about you know us being aware of which elected representatives understand the nuances
00:26:30
of algorithms and the challenges and how to regulate them and it's about voting
00:26:36
for them right and the question is how are these regulators going to protect us that's where the bill of rights comes in
00:26:42
and the Bill of Rights I propose has a few key pillars one pillar is transparency transparency with regard to
00:26:49
the data used to make decisions and with regard to the underlying decision itself
00:26:55
what were the most important factors that led to a certain decision today Europe's GDP are actually has certain
00:27:03
provisions like write to explanations and information on the data that companies are using and so I think some
00:27:09
of that transparency is needed and companies should provide that another pillar in my bill of rights is the idea
00:27:17
of some user control that we cannot be in an environment where we have no control over the technology we should
00:27:25
for example be able to with a simple instruction tell Alexa you're not listening to any conversation in the
00:27:33
house until I'd instruct you that it's allowed right there's no such provision
00:27:38
we're told that the system is not listening but then we're also hearing from others that there are instances
00:27:44
where it listens even when you're not actually saying Alexa and giving an instruction and that control is very
00:27:52
important if you look at Facebook the false news issue two years back there was no way for users to alert Facebook's
00:27:58
algorithm and say this post in my newsfeed is false news even though users were seeing false news in their newsfeed
00:28:05
even today with just two clicks you can let Facebook know that a certain news post in your feed is either offensive or
00:28:14
it has false news and that feedback is so important for the algorithm to now correct itself and previously users had
00:28:21
no way even though you're observing problems no way of informing the algorithm or helping the algorithm
00:28:27
course correct so some level of user control is also another pillar I proposed and lastly I
00:28:33
have been advocating this idea that companies should formally audit algorithms before they deploy them as
00:28:39
especially in socially consequential settings not every algorithm needs to be audited
00:28:44
but algorithms in socially consequential settings like recruiting need to go through an audit process and that audit
00:28:51
process can be done by an internal team or it could be outside but it's got to be done by a team that
00:28:56
is independent of the team that developed the algorithm and the audit process is going to be important because
00:29:03
it will help ensure that somebody's looked at things beyond say the prediction accuracy of the model they
00:29:10
have looked at things like privacy they've looked at things like bias and fairness and so that will help curb some
00:29:17
of these problems with with algorithmic decisions so I got we could keep talking
00:29:22
about this all day I've sort of come to the end of the list of questions I had
00:29:28
for you are there any time points that you would like to emphasize that I haven't asked about one of the key
00:29:37
messages I want to share with people even though I'm sharing many of the challenges with algorithms in my book is
00:29:44
that I'm not an algorithm skeptic I'm actually a believer in algorithms and I
00:29:48
don't want any listener or viewer or reader to leave this becoming wary of technology I think the message is not be
00:29:58
very but the message is engage more actively and more deliberately and influence be part of the process of
00:30:05
influencing how these technologies develop so and the reason I say that is that studies show that algorithms on
00:30:12
average are less biased than human beings so if we say we don't want algorithms we need to ask what's the
00:30:18
alternative and the alternative is biased furthermore my contention is that it is
00:30:23
easier in the long run to fix fix algorithm bias than it is to fix human bias but the challenge with algorithm
00:30:31
bias is just that that bias scales in a way human bias does not scale and what I
00:30:36
mean by that is that a prejudiced judge can impact the lives of maybe 200 300 people but an algorithm used in all the
00:30:46
courtrooms in a country or across the world can influence the lives of hundreds of thousands or even millions
00:30:51
of people similarly a bias recruiter can affect the lives of hundreds of people but a bias recruiting algorithm can
00:30:58
affect the lives of millions of people so it's the scale we worry about and that's why we need to take the issue
00:31:03
seriously but the message mostly is I think we're going to a world where these algorithms will
00:31:10
help us make better decisions and will have growing pains along the way and a few examples I mentioned I think is only
00:31:18
just the beginning we'll hear many more and we should engage actively now to
00:31:22
minimize those incidences Karthik thank you so much for speaking with knowledge at work thank you for having me it's
00:31:29
been a great pleasure for more insight from knowledge at Wharton please visit knowledge Wharton UPenn edu
00:31:42
[Music] you