00:00:00
well as artificial intelligence
00:00:01
continues to grow in Impact in our
00:00:03
culture there continue to be new
00:00:05
questions to be asked one from a new
00:00:08
white paper by our next guest ask
00:00:10
whether chat GPT can provide ethical
00:00:13
advice the authors of that paper join us
00:00:16
right now Christian tursh who's a
00:00:18
professor of operations information and
00:00:20
decisions here at the Wharton School
00:00:22
he's also co-director of the Mac
00:00:23
Institute and also Leonard miky who is a
00:00:26
research assistant here at Wharton
00:00:29
gentlemen great to have you with us
00:00:30
today thanks for your time thanks for
00:00:32
having us thank you all right so
00:00:34
Christian I'll start with you lay this
00:00:36
out for us about both the importance of
00:00:39
This research but how you went about it
00:00:42
so this sounds like a weird question
00:00:44
right can GPT give ethical advice so the
00:00:47
bigger context is that we've seen AI do
00:00:49
all kinds of things from being a
00:00:51
customer support agent uh summarizing
00:00:55
patient visits in the hospital doing
00:00:57
automated underwriting decisions and
00:00:58
banking situations and so the the
00:01:01
question came up like can it do more
00:01:03
human things and thenard and I did
00:01:05
another white paper that you and I
00:01:07
talked about on the sh recently whether
00:01:10
gbt can be
00:01:12
creative and we found really interesting
00:01:14
things there and so the last question
00:01:16
that we felt like was kind of had to be
00:01:19
covered is can GPT do something human
00:01:23
like having feelings and talk about
00:01:25
ethical advice and so we thought that
00:01:27
was a really interesting question to
00:01:29
tackle so Leonard how did you go about
00:01:31
this because I find it interesting it's
00:01:33
one thing to you know for the for the
00:01:35
process to include something that may
00:01:37
work in the human brain this is kind of
00:01:39
another Avenue of that yeah and one
00:01:41
thing that was very important to us was
00:01:43
so we benchmarked it against ethical
00:01:45
advice by the ethicist in the New York
00:01:47
Times we didn't just want to see if we
00:01:49
can replicate their advice right we
00:01:51
wanted to really see what is the base
00:01:54
model gp4 what can it do right now so
00:01:57
when preparing our experiment we spent a
00:01:59
lot of time time making sure that we
00:02:00
don't introduce any kind of bias or give
00:02:03
any previous responses by the person
00:02:05
we're comparing against to make sure
00:02:07
really just getting with the raw model
00:02:09
things and it was it was super
00:02:10
fascinating to see the differences in
00:02:13
responses for the different ethical
00:02:14
dilemmas so this was basically taking
00:02:17
scenarios that may have been in the
00:02:19
ethicist and preparing them for them to
00:02:22
be diagnosed by individuals that you had
00:02:25
in this in this research yeah so we
00:02:27
essentially took reader submissions to
00:02:29
the the New York Times um that the
00:02:31
ethesis had answered beforehand I think
00:02:33
we drew from the last three to four
00:02:34
month so the data could not have been in
00:02:36
GPT so it couldn't have seen the
00:02:38
ethicist responses to it and then for
00:02:41
each of those dilemas we asked GPT again
00:02:44
with a prompt to make sure that it
00:02:45
wouldn't just try to mimic the original
00:02:48
advisor and saw what what kind of advice
00:02:51
GPT gave and then we had a few different
00:02:54
groups MBA students at Wharton um we had
00:02:57
an expert group of clerics and faculty
00:03:01
and then we had more like general public
00:03:03
college educated people rate each advice
00:03:05
and see what which one they perceive to
00:03:07
be more useful so Christian we've
00:03:09
obviously talked about uh the connection
00:03:12
between ethical advice and and morals in
00:03:15
human beings and and I think the
00:03:18
question probably a lot of people might
00:03:19
have is is it too much of a leap of
00:03:21
faith to assume that computers or in
00:03:24
this case AI can make those types of
00:03:27
decisions so let's be clear what what we
00:03:29
are testing here and what we're not
00:03:31
testing here we're not testing whether
00:03:33
AI has feelings AI is conscious or AI
00:03:37
has moral
00:03:39
capabilities I have an opinion on all of
00:03:41
those and the answer is no what we are
00:03:43
testing is can GPT create ethical advice
00:03:47
that is useful and usefulness is
00:03:49
measured in the eyes as lard was just
00:03:52
explaining in the eyes of lay people Wen
00:03:55
MBA students and experts such as clergy
00:03:58
and uh academics
00:04:00
and so for each of these dilemmas we
00:04:03
have the expert in this case uh from The
00:04:05
New York Times Dr apia uh share his view
00:04:10
on an ethical dilemma and we GPT create
00:04:13
his or her view on the dilemma and then
00:04:16
we compare which one is more useful so
00:04:18
we're not making claims that AI is is is
00:04:22
human or conscious or anything it is
00:04:24
fascinating to see that it is it
00:04:26
provides useful advice nothing more
00:04:28
nothing less and let from what I read
00:04:30
there was a path where you also uh rated
00:04:33
the usefulness of the advice correct
00:04:36
yeah so there were essentially two
00:04:37
different ways we approach the
00:04:39
evaluation in one of the two surveys
00:04:42
that we went through with our
00:04:43
participants we asked participants
00:04:45
please rate how useful well imagine you
00:04:48
are the reader uh so that you don't have
00:04:50
any personal advice imagine you're the
00:04:52
reader and you ask this question how
00:04:53
useful do you think on a scale from 1 to
00:04:55
seven this advice will be to you and
00:04:58
then in a secondary group group we did
00:05:01
display the original athetist advice and
00:05:03
we displayed the gbt advice and then
00:05:06
basically they had to pick which of the
00:05:07
two they preferred so it was more like a
00:05:09
head- onad race and so what was the end
00:05:13
up being the results in terms of which
00:05:15
advice they seem to gravitate more
00:05:17
towards so it was very interesting
00:05:20
because essentially it's a tie and it's
00:05:22
a super close tie so across all three
00:05:24
groups there's a slight preference for
00:05:26
GPT but only for the lay people that
00:05:28
statistically significant for the NBAs
00:05:31
and the experts while there's a small
00:05:33
preference it's not large enough to be
00:05:35
considered significant so I think right
00:05:37
now we're just confident to say that
00:05:39
there is a tie in the ability uh but we
00:05:41
we can't really necessarily say one is
00:05:43
much better or worse than the other
00:05:45
Christian does the fact that there was
00:05:48
basically a tie in this does it does it
00:05:51
does it say anything to you about the
00:05:54
the importance of of this going down the
00:05:57
down the road yeah I mean that allow
00:05:59
tellows us now that anywhere you're in a
00:06:02
situ anywhere you are anytime that you
00:06:03
have a situation where you want ethical
00:06:06
advice you can get it immediately and
00:06:08
for free and I think that is in many
00:06:11
ways a real wild result because few of
00:06:14
us unless we are sitting in the white
00:06:16
house or a CEO of a big company have the
00:06:18
luxury of having like ethical advisors
00:06:21
and so ethical advice uh can be streamed
00:06:25
to us at zero marginal cost and
00:06:27
instantly so I think that is wow again I
00:06:29
don't want to overgeneralize and say
00:06:31
like we don't need ethesis anymore to
00:06:33
the contrary but in most situations I
00:06:36
think normal people like you and us uh
00:06:39
are in in most situations we can't
00:06:41
afford to seek ethical advice by an
00:06:44
expert and so having this now available
00:06:48
at the click of of a mouse I think is is
00:06:50
is something really big and it's is it
00:06:52
safe to say Christian that with the
00:06:55
growth we're seeing uh in the use of AI
00:06:57
in a variety of different areas that the
00:06:59
expectation is that we will see it grow
00:07:02
in this area as well so again I think
00:07:05
the growth is in more use cases and
00:07:07
previously have had no involvement of
00:07:09
ethical considerations we can provide it
00:07:13
I'm absolutely hesitant to make
00:07:15
significant decisions in life in
00:07:17
politics and in business based on that
00:07:20
advice alone I think of it more as a
00:07:23
stimulus to us as we are going through
00:07:26
our own deliberations in our in our
00:07:28
minds to hear other perspectives to
00:07:31
embrace perspectives that we might have
00:07:34
usually not have embraced from our
00:07:35
social network and take all of that into
00:07:38
account and have the human be the the
00:07:40
decision maker and hopefully that leads
00:07:43
to more ethical Behavior which is
00:07:45
something that I think the world
00:07:46
desperately needs right now Leonard it
00:07:48
sounded like in some of the cases when
00:07:50
people were making their decision
00:07:52
between the AI advice and and that of of
00:07:56
of the the ethesis that there may have
00:07:58
been a case where people were not sure
00:08:00
which one they would take and you really
00:08:02
I guess to agree press them on it to
00:08:04
really make a decision on which one they
00:08:06
would take and and in some cases the AI
00:08:09
won out yeah and we were thinking hard
00:08:12
how to design the experiment because if
00:08:14
we just give them two pieces Like A and
00:08:17
B advice for the AI
00:08:18
nist let's say everyone picks the AI but
00:08:21
it was super close then we can't really
00:08:23
say that right so that's why we use this
00:08:25
two stages or two different experiments
00:08:28
where for one we asked them to one to7
00:08:30
how useful it is and then in another
00:08:32
round we asked them which one do you
00:08:34
think is more useful to you out of these
00:08:35
two of course we asked different people
00:08:38
different questions but that was kind of
00:08:39
the core idea so we were able to see how
00:08:42
big the difference really was what did
00:08:44
you take from from doing this research
00:08:47
Leonard I think it's just super
00:08:49
interesting with GPT and I coming up so
00:08:52
much to evaluate all the different
00:08:54
things you might not immediately think
00:08:56
of where it's already good at and I
00:08:58
think to Christian's point I would just
00:09:01
see it as one more opinion one more
00:09:03
voice that you're throwing in to a
00:09:05
decision and you know maybe it's 11:
00:09:07
p.m. on a Saturday afternoon evening and
00:09:10
you would just like to get some advice
00:09:12
on something you know you have it
00:09:13
readily available you don't have to
00:09:14
follow it you're not sitting here and
00:09:15
say please follow what gp4 tells you but
00:09:18
it's just one more opinion that might be
00:09:20
helpful and it's very
00:09:22
affordable Christian where do you think
00:09:25
then this research potentially takes you
00:09:27
what is there a next log IC step to to
00:09:30
follow up with this I would really like
00:09:32
to explore the role of opinion diversity
00:09:35
I think it lies into in the nature of
00:09:38
American life and most of the western
00:09:40
world right now that we are all
00:09:41
surrounding ourselves with like-minded
00:09:43
people I think there's a real
00:09:45
opportunity to seek advice from folks
00:09:48
that are explicitly outside your comfort
00:09:51
zone people who who you would normally
00:09:53
not approach and hear what they have to
00:09:55
say hear different perspectives and then
00:09:58
you maybe together with a can aggregate
00:10:00
these individual voices and opinions to
00:10:02
hopefully look Beyond kind of your own
00:10:05
perspective and and grow in
00:10:08
understanding of others and in the Del
00:10:11
deliberation process for your own
00:10:12
decisions Leonard where would you like
00:10:14
to take this
00:10:15
next I think it would also be
00:10:17
interesting if we look at some of the
00:10:20
differences per ethical dilemma so we
00:10:23
asked 20 different ones and it would
00:10:25
just be cool to see maybe there's a
00:10:27
strong preference in I in a specific
00:10:30
Niche maybe it's different for the
00:10:31
ethicist uh just to kind of see how
00:10:35
people a little bit to Christian's point
00:10:37
when it comes to devises how people
00:10:38
think differently per ethical question
00:10:41
raised is there also an element of this
00:10:44
Leonard because of the fact that we have
00:10:47
become more reliant on technology in
00:10:50
general that the expectation is as we
00:10:52
move further down the road and we rely
00:10:55
more on technology that this is going to
00:10:57
be a natural Avenue for people people to
00:10:59
maybe consider when they're thinking
00:11:01
about you know some sort of question or
00:11:04
problem in their life that they may may
00:11:05
need to get advice on I think that's a
00:11:08
very natural assumption to make I think
00:11:11
we've seen it for many years where
00:11:13
people take to online forums to ask
00:11:15
about any kind of advice whether it's
00:11:17
relationship advice ethical advice we
00:11:19
see it on Reddit a lot so I think it's a
00:11:21
very natural extension of the technology
00:11:23
that people will turn more and more to
00:11:24
these virtual assistants and ask them
00:11:27
for help of course in these cases the
00:11:30
help or answer can only ever be as good
00:11:32
as how much you're telling and how good
00:11:34
your input
00:11:36
is Christian how about
00:11:38
yourself I think it's just fascinating
00:11:41
that a technology that has never been
00:11:43
married that has never been loved can
00:11:45
provide us with guidance and advice
00:11:47
about how to live our life and so again
00:11:50
I think of it as get input get stimul
00:11:53
get New Perspectives and then it's up to
00:11:55
us of integrating that into our
00:11:57
decision- making process and become
00:11:59
better people with that great to have
00:12:01
you both with us thank you Christian
00:12:03
thank you Leonard being a pleasure
00:12:05
appreciate thank you Christian tursh
00:12:07
who's a professor of operations
00:12:08
information and decisions here at the
00:12:10
Wharton School and also Leonard mik who
00:12:12
is a research assistant here at the
00:12:14
Warton
00:12:20
School
