00:00:00
well as artificial intelligence continues to grow in Impact in our culture there continue to be new
00:00:05
questions to be asked one from a new white paper by our next guest ask whether chat GPT can provide ethical
00:00:13
advice the authors of that paper join us right now Christian tursh who's a professor of operations information and
00:00:20
decisions here at the Wharton School he's also co-director of the Mac Institute and also Leonard miky who is a
00:00:26
research assistant here at Wharton gentlemen great to have you with us today thanks for your time thanks for
00:00:32
having us thank you all right so Christian I'll start with you lay this out for us about both the importance of
00:00:39
This research but how you went about it so this sounds like a weird question right can GPT give ethical advice so the
00:00:47
bigger context is that we've seen AI do all kinds of things from being a customer support agent uh summarizing
00:00:55
patient visits in the hospital doing automated underwriting decisions and banking situations and so the the
00:01:01
question came up like can it do more human things and thenard and I did another white paper that you and I
00:01:07
talked about on the sh recently whether gbt can be creative and we found really interesting
00:01:14
things there and so the last question that we felt like was kind of had to be covered is can GPT do something human
00:01:23
like having feelings and talk about ethical advice and so we thought that was a really interesting question to
00:01:29
tackle so Leonard how did you go about this because I find it interesting it's
00:01:33
one thing to you know for the for the process to include something that may work in the human brain this is kind of
00:01:39
another Avenue of that yeah and one thing that was very important to us was so we benchmarked it against ethical
00:01:45
advice by the ethicist in the New York Times we didn't just want to see if we
00:01:49
can replicate their advice right we wanted to really see what is the base model gp4 what can it do right now so
00:01:57
when preparing our experiment we spent a lot of time time making sure that we don't introduce any kind of bias or give
00:02:03
any previous responses by the person we're comparing against to make sure really just getting with the raw model
00:02:09
things and it was it was super fascinating to see the differences in responses for the different ethical
00:02:14
dilemmas so this was basically taking scenarios that may have been in the ethicist and preparing them for them to
00:02:22
be diagnosed by individuals that you had in this in this research yeah so we essentially took reader submissions to
00:02:29
the the New York Times um that the ethesis had answered beforehand I think we drew from the last three to four
00:02:34
month so the data could not have been in GPT so it couldn't have seen the ethicist responses to it and then for
00:02:41
each of those dilemas we asked GPT again with a prompt to make sure that it wouldn't just try to mimic the original
00:02:48
advisor and saw what what kind of advice GPT gave and then we had a few different
00:02:54
groups MBA students at Wharton um we had an expert group of clerics and faculty and then we had more like general public
00:03:03
college educated people rate each advice and see what which one they perceive to
00:03:07
be more useful so Christian we've obviously talked about uh the connection between ethical advice and and morals in
00:03:15
human beings and and I think the question probably a lot of people might have is is it too much of a leap of
00:03:21
faith to assume that computers or in this case AI can make those types of decisions so let's be clear what what we
00:03:29
are testing here and what we're not testing here we're not testing whether
00:03:33
AI has feelings AI is conscious or AI has moral capabilities I have an opinion on all of
00:03:41
those and the answer is no what we are testing is can GPT create ethical advice that is useful and usefulness is
00:03:49
measured in the eyes as lard was just explaining in the eyes of lay people Wen MBA students and experts such as clergy
00:03:58
and uh academics and so for each of these dilemmas we have the expert in this case uh from The
00:04:05
New York Times Dr apia uh share his view on an ethical dilemma and we GPT create
00:04:13
his or her view on the dilemma and then we compare which one is more useful so we're not making claims that AI is is is
00:04:22
human or conscious or anything it is fascinating to see that it is it provides useful advice nothing more
00:04:28
nothing less and let from what I read there was a path where you also uh rated the usefulness of the advice correct
00:04:36
yeah so there were essentially two different ways we approach the evaluation in one of the two surveys
00:04:42
that we went through with our participants we asked participants please rate how useful well imagine you
00:04:48
are the reader uh so that you don't have any personal advice imagine you're the
00:04:52
reader and you ask this question how useful do you think on a scale from 1 to seven this advice will be to you and
00:04:58
then in a secondary group group we did display the original athetist advice and we displayed the gbt advice and then
00:05:06
basically they had to pick which of the two they preferred so it was more like a
00:05:09
head- onad race and so what was the end up being the results in terms of which advice they seem to gravitate more
00:05:17
towards so it was very interesting because essentially it's a tie and it's
00:05:22
a super close tie so across all three groups there's a slight preference for
00:05:26
GPT but only for the lay people that statistically significant for the NBAs and the experts while there's a small
00:05:33
preference it's not large enough to be considered significant so I think right
00:05:37
now we're just confident to say that there is a tie in the ability uh but we
00:05:41
we can't really necessarily say one is much better or worse than the other Christian does the fact that there was
00:05:48
basically a tie in this does it does it does it say anything to you about the the importance of of this going down the
00:05:57
down the road yeah I mean that allow tellows us now that anywhere you're in a
00:06:02
situ anywhere you are anytime that you have a situation where you want ethical advice you can get it immediately and
00:06:08
for free and I think that is in many ways a real wild result because few of us unless we are sitting in the white
00:06:16
house or a CEO of a big company have the luxury of having like ethical advisors and so ethical advice uh can be streamed
00:06:25
to us at zero marginal cost and instantly so I think that is wow again I don't want to overgeneralize and say
00:06:31
like we don't need ethesis anymore to the contrary but in most situations I think normal people like you and us uh
00:06:39
are in in most situations we can't afford to seek ethical advice by an expert and so having this now available
00:06:48
at the click of of a mouse I think is is is something really big and it's is it
00:06:52
safe to say Christian that with the growth we're seeing uh in the use of AI in a variety of different areas that the
00:06:59
expectation is that we will see it grow in this area as well so again I think the growth is in more use cases and
00:07:07
previously have had no involvement of ethical considerations we can provide it I'm absolutely hesitant to make
00:07:15
significant decisions in life in politics and in business based on that advice alone I think of it more as a
00:07:23
stimulus to us as we are going through our own deliberations in our in our minds to hear other perspectives to
00:07:31
embrace perspectives that we might have usually not have embraced from our social network and take all of that into
00:07:38
account and have the human be the the decision maker and hopefully that leads to more ethical Behavior which is
00:07:45
something that I think the world desperately needs right now Leonard it sounded like in some of the cases when
00:07:50
people were making their decision between the AI advice and and that of of of the the ethesis that there may have
00:07:58
been a case where people were not sure which one they would take and you really I guess to agree press them on it to
00:08:04
really make a decision on which one they would take and and in some cases the AI
00:08:09
won out yeah and we were thinking hard how to design the experiment because if we just give them two pieces Like A and
00:08:17
B advice for the AI nist let's say everyone picks the AI but it was super close then we can't really
00:08:23
say that right so that's why we use this two stages or two different experiments
00:08:28
where for one we asked them to one to7 how useful it is and then in another round we asked them which one do you
00:08:34
think is more useful to you out of these two of course we asked different people
00:08:38
different questions but that was kind of the core idea so we were able to see how
00:08:42
big the difference really was what did you take from from doing this research Leonard I think it's just super
00:08:49
interesting with GPT and I coming up so much to evaluate all the different things you might not immediately think
00:08:56
of where it's already good at and I think to Christian's point I would just
00:09:01
see it as one more opinion one more voice that you're throwing in to a decision and you know maybe it's 11:
00:09:07
p.m. on a Saturday afternoon evening and you would just like to get some advice on something you know you have it
00:09:13
readily available you don't have to follow it you're not sitting here and
00:09:15
say please follow what gp4 tells you but it's just one more opinion that might be
00:09:20
helpful and it's very affordable Christian where do you think then this research potentially takes you
00:09:27
what is there a next log IC step to to follow up with this I would really like to explore the role of opinion diversity
00:09:35
I think it lies into in the nature of American life and most of the western world right now that we are all
00:09:41
surrounding ourselves with like-minded people I think there's a real opportunity to seek advice from folks
00:09:48
that are explicitly outside your comfort zone people who who you would normally not approach and hear what they have to
00:09:55
say hear different perspectives and then you maybe together with a can aggregate
00:10:00
these individual voices and opinions to hopefully look Beyond kind of your own perspective and and grow in
00:10:08
understanding of others and in the Del deliberation process for your own decisions Leonard where would you like
00:10:14
to take this next I think it would also be interesting if we look at some of the
00:10:20
differences per ethical dilemma so we asked 20 different ones and it would just be cool to see maybe there's a
00:10:27
strong preference in I in a specific Niche maybe it's different for the ethicist uh just to kind of see how
00:10:35
people a little bit to Christian's point when it comes to devises how people
00:10:38
think differently per ethical question raised is there also an element of this Leonard because of the fact that we have
00:10:47
become more reliant on technology in general that the expectation is as we move further down the road and we rely
00:10:55
more on technology that this is going to be a natural Avenue for people people to
00:10:59
maybe consider when they're thinking about you know some sort of question or
00:11:04
problem in their life that they may may need to get advice on I think that's a
00:11:08
very natural assumption to make I think we've seen it for many years where people take to online forums to ask
00:11:15
about any kind of advice whether it's relationship advice ethical advice we see it on Reddit a lot so I think it's a
00:11:21
very natural extension of the technology that people will turn more and more to these virtual assistants and ask them
00:11:27
for help of course in these cases the help or answer can only ever be as good as how much you're telling and how good
00:11:34
your input is Christian how about yourself I think it's just fascinating that a technology that has never been
00:11:43
married that has never been loved can provide us with guidance and advice about how to live our life and so again
00:11:50
I think of it as get input get stimul get New Perspectives and then it's up to
00:11:55
us of integrating that into our decision- making process and become better people with that great to have
00:12:01
you both with us thank you Christian thank you Leonard being a pleasure appreciate thank you Christian tursh
00:12:07
who's a professor of operations information and decisions here at the Wharton School and also Leonard mik who
00:12:12
is a research assistant here at the Warton School