00:00:01
this podcast is brought to you by
00:00:03
knowled of
00:00:04
[Music]
00:00:15
Warton hello my name is Stephanie Cy and
00:00:18
I'm an assistant professor of management
00:00:19
at the Wharton School of the University
00:00:21
of Pennsylvania and I'm delighted to
00:00:23
welcome you to today's episode of the
00:00:25
knowledge at War and leading diversity
00:00:27
at work podcast series which is focus
00:00:29
focused on responsible and fair air AI
00:00:32
in the workplace and the marketplace
00:00:35
joining me today are two very special
00:00:38
guests first we have Dr broer Turner who
00:00:41
is an assistant professor of marketing
00:00:43
at Virginia Tech Pamplin School of
00:00:45
Business and a visiting fellow at
00:00:47
Harvard Business school's Institute for
00:00:49
business and Global Society he also runs
00:00:52
the technology race and Prejudice lab or
00:00:55
the Trap lab where he and his fellow
00:00:57
Trappers are pushing the boundaries on
00:01:00
understanding how race and racism
00:01:01
underly many consumer and managerial
00:01:04
decisions his main research area focuses
00:01:07
on the intersection of marketing
00:01:08
technology racism and emotion next we
00:01:12
have Dr kareim Gan who is the founder of
00:01:16
RA AI audit an AI governance and
00:01:19
research consultancy with 16 years of
00:01:22
experience in ethics and governance
00:01:24
spanning both industry and Academia he
00:01:27
recently served as the founding user
00:01:29
experience researcher on meta's
00:01:31
responsible AI team leading AI fairness
00:01:35
user research at the company he has
00:01:37
helped meta our AI team scale its
00:01:40
products across the company and bolster
00:01:42
the responsible adoption of AI Dr ganana
00:01:46
holds a PhD in management from the
00:01:48
University of Virginia's Darden School
00:01:49
of Business specializing in business
00:01:52
ethics and organizational behavior and I
00:01:54
just realized we've got a Virginia
00:01:56
connection uh across the two of you here
00:01:59
who would have who would have thought
00:02:00
that that would happen but in any case
00:02:02
welcome bro and Kareem I am so honored
00:02:05
to have you with me today for a
00:02:06
conversation on responsible and fair AI
00:02:10
in the workplace and in the marketplace
00:02:12
so we're hoping to cover you know a lot
00:02:14
of interesting ground here today and as
00:02:16
I as talked to brck and cre about this
00:02:19
earlier I like to think of myself as a
00:02:21
as a novice on these topics so today I'm
00:02:24
going to represent the average uh
00:02:26
consumer um and worker who doesn't have
00:02:28
a lot of experience talking about um AI
00:02:32
other than what I read about and hear
00:02:34
about um on mainstream media and from my
00:02:37
colleagues so so we're hoping today that
00:02:39
um this conversation will be helpful to
00:02:41
others of you who are just like me who
00:02:43
are trying to figure out how relevant uh
00:02:46
a conversation about responsible and
00:02:48
fair AI is to us as workers but also as
00:02:52
consumers so let's get started talking
00:02:55
about broadly about responsible and fair
00:02:58
Ai and so project I'm I'm going to go to
00:03:00
you first and what I'm hoping you can do
00:03:02
is is share a bit about how and why you
00:03:05
became involved in this topic and
00:03:07
summarize for us some of your current
00:03:09
work related to this topic yeah so I got
00:03:13
involved in this topic area uh honestly
00:03:17
more than a decade ago when I was
00:03:18
teaching high school math uh at a public
00:03:22
high school in Atlanta Georgia where I
00:03:23
taught like
00:03:25
90% uh black students and what I wanted
00:03:29
to do while I was there was figure out
00:03:31
ways to make those kids' lives better
00:03:34
fairer more Equitable uh and
00:03:36
simultaneously I used to teach linear
00:03:39
regression and students asked me Mr turn
00:03:41
am I ever going to use this and now I
00:03:43
have an answer the answer is now that uh
00:03:47
you know so I've moved into this space
00:03:49
where I'm thinking a lot about how race
00:03:52
and racism
00:03:53
underly uh Market Systems and Technology
00:03:57
being a system that matters and our
00:04:01
research group the technology race and
00:04:03
Prejudice lab is doing active research
00:04:05
now on what levers can be moved to lead
00:04:09
to uh more equality in systems to lead
00:04:13
to better outcomes in technology uh this
00:04:16
has extended some into some uh company
00:04:18
advisory work where uh we're trying to
00:04:21
get company consider that moving uh
00:04:24
communities earlier into the development
00:04:26
process leads to better uh Products that
00:04:29
come out out and then on the public
00:04:30
knowledge side uh we're writing white
00:04:33
papers on uh how different uh identity
00:04:37
groups may be impacted by gender of AI
00:04:40
for instance so we just uh finished a
00:04:43
report on for Hispanic herited Munch
00:04:45
Hispanic herit Heritage Month thank you
00:04:48
on how uh in uh how folks with Hispanic
00:04:53
Origins and background should be
00:04:54
thinking about gender of AI how do they
00:04:57
fit into this interlan system of data
00:04:59
the classification code okay great thank
00:05:03
you so much uh so Kareem same question
00:05:05
to you can you share a bit about how and
00:05:07
why you became involved in this topic
00:05:09
around responsible and fair Ai and
00:05:12
summarize some of your current work on
00:05:14
this topic yeah absolutely firstly thank
00:05:17
you so much for inviting me I'm excited
00:05:19
about this dialogue with yourself and
00:05:21
brri and uh love the Virginia connection
00:05:24
Virginia is well represented here today
00:05:27
um so i' I've been involved in ethics
00:05:28
and governance
00:05:30
um for quite some time you know prior to
00:05:32
pursuing my PhD so I'm not surprised
00:05:35
that I ended up in kind of AI governance
00:05:38
uh what I'm a little bit surprised about
00:05:39
is actually post pursuing my PhD going
00:05:42
back to uh industry initially when I
00:05:44
started off my PhD program I had no
00:05:48
intentions uh to go back to industry but
00:05:50
in my last year of my PhD which was my
00:05:53
sixth year I had received offers
00:05:55
basically from Academia and Industry and
00:05:58
uh I went for back to Industry so um
00:06:02
firstly as someone who belongs to a
00:06:04
minority group um I was very passionate
00:06:07
and still am about having a front seat
00:06:09
to these conversations that are
00:06:10
happening in Tech and uh helping shape
00:06:14
the uh the technology using you know the
00:06:18
latest and greatest research but also
00:06:20
interacting with product teams to be
00:06:22
able to do that policy and legal and so
00:06:26
forth um the other reason that I chose
00:06:28
to go back was
00:06:30
you know as I was thinking through my
00:06:32
impact to be honest uh joining a company
00:06:35
like meta's responsible AI team where
00:06:38
the company has over three three billion
00:06:40
users I think you know it's very
00:06:42
difficult to kind of argue with the
00:06:44
extent of impact that you can do uh when
00:06:47
you uh you have three billion users and
00:06:49
any kind of like product changes that
00:06:51
you make can impact really a lot of
00:06:53
people worldwide and and lastly honestly
00:06:56
uh one of the concerns that I've had
00:06:59
about uh Academia was you know being
00:07:02
stuck in a small college town somewhere
00:07:05
uh where there's not very much diversity
00:07:08
and uh that would not be appropriate for
00:07:11
the upbringing of my kids and so that
00:07:13
was a a a very uh strong concern uh not
00:07:17
having much of an autonomy about where I
00:07:19
live or where I work um so it drove me
00:07:22
back to actually go to um to
00:07:26
Industry now as far as my research
00:07:30
uh much of what I do for clients is
00:07:32
obviously confidential but on the public
00:07:34
side of things I've been testing out uh
00:07:36
some popular AI image generators um and
00:07:39
I'm finding a few patterns so uh most of
00:07:42
the images uh that are being generated
00:07:44
from uh the uh at least the AI
00:07:48
generators uh that I have tested tend to
00:07:51
be of people from the white race uh
00:07:53
people of Co color are uh greatly
00:07:56
overlooked uh these models tend to
00:07:58
associate
00:08:00
uh a professional headshot or a
00:08:01
professional dress code or a
00:08:03
professional hairstyle with those of uh
00:08:06
white people uh women are
00:08:10
underrepresented uh and uh when
00:08:12
presented as professionals they're often
00:08:14
limited to gendered occupations such as
00:08:17
teachers nurses graphic designers and
00:08:20
much less likely to produ to appear as
00:08:22
CEOs for example or medical doctors or
00:08:25
lawyers um images also tend to spew on
00:08:28
the younger side of things and Senior
00:08:31
managers for example are associated with
00:08:33
having gray hair which is a sign of
00:08:36
aism um and finally you know these
00:08:40
models tend to portray certain
00:08:42
populations in demeaning ways for
00:08:44
example uh if if you ask Del Bing to
00:08:48
portray Turks uh it will uh provide you
00:08:51
with a picture of a Stern turkey uh a
00:08:55
Stern turkey dressed in a turban right
00:08:58
um whereas if you're ask it to produce
00:09:00
for example an image of an American or a
00:09:02
French person or something like that
00:09:04
will do a much better job right um they
00:09:07
might also refuse to produce images of
00:09:09
certain populations For No Good Reason
00:09:12
such as if you ask it to produce um an
00:09:14
image of a Muslim or a Jew uh while
00:09:17
presenting followers of other religions
00:09:19
so definitely this technology is
00:09:21
extremely powerful uh I am not a
00:09:25
doomsayer per se as it uh relating to
00:09:28
the technology but I
00:09:30
believe uh that you know we have to do
00:09:32
our prudent and due diligence in order
00:09:35
for us to be able to direct the
00:09:36
trajectory of this technology in ways um
00:09:39
such that we're maximizing the benefits
00:09:41
while minimizing the harms my my
00:09:44
favorite example of uh that gender of AI
00:09:47
uh art is when people do prompts for
00:09:51
Jesus in the temple flipping the table
00:09:53
that they literally get Jesus doing like
00:09:56
gymnastics doing a backflip over a table
00:09:59
uh so
00:10:01
yeah text text has no meaning it's it's
00:10:03
just funny absolutely so we're gonna
00:10:06
definitely dive into there's a lot of to
00:10:08
unpack there and certainly the
00:10:09
implications of this I mean I think of
00:10:11
where I sit I can automatically connect
00:10:13
those dots but I think as we begin to
00:10:15
talk about some of the challenges it'll
00:10:16
be important for people to understand
00:10:17
what could possibly go wrong when you
00:10:20
you're not represented in how these
00:10:22
models are being uh trained I I think to
00:10:26
me that's an obvious answer but I don't
00:10:27
think that's obvious to everybody what
00:10:29
happens when you lack representation in
00:10:32
in the data amongst the data or you're
00:10:34
misrepresented right who you are and the
00:10:36
the the groups that you're part of are
00:10:38
misrepresented in in how the model is
00:10:40
being trained so we're going to talk
00:10:41
about that in a minute but let me just
00:10:43
and probably this is part of the answer
00:10:44
to your question here Kareem um let's
00:10:47
let's start with business and employers
00:10:49
first right and and I can imagine I have
00:10:52
some sense of this from my own work that
00:10:54
businesses and employers um are likely
00:10:57
and can be struggling to understand
00:11:00
their conversation not just in a role
00:11:02
about AI use but fair and responsible AI
00:11:06
use and so obviously without you know
00:11:09
saying anything confidential can you
00:11:11
talk generally about how you through
00:11:13
your work are helping businesses
00:11:15
employers make sense of their roles and
00:11:18
responsibilities on this
00:11:19
topic yeah I think it's important to be
00:11:22
clear that um companies are accountable
00:11:24
for how their AI systems operate and
00:11:27
that responsibility can be offloaded
00:11:30
right it can be considered an
00:11:31
externality of some sort they have
00:11:33
obligations to their customers and it's
00:11:35
just a matter of time uh before
00:11:38
legislation comes into effect in this
00:11:40
space you know obviously we have right
00:11:42
now the uh EU AI act which we're waiting
00:11:46
it's just a matter of time before
00:11:48
enforcement happens and there's a lot of
00:11:50
talk uh about legislation uh Congress
00:11:53
and so forth here in the United States
00:11:57
um and honestly if come companies want
00:11:59
to build successful products in this AI
00:12:01
era era they must invest in responsible
00:12:04
AI infrastructure right um doing so
00:12:07
requires a commitment from the board of
00:12:09
directors from senior leadership but ALS
00:12:12
ultimately what it does is it pays off
00:12:15
uh in terms of earning customer trust
00:12:18
and confidence right it it just doesn't
00:12:20
cut it for you to actually have an AI
00:12:22
strategy uh without thinking through the
00:12:26
the trust layer or the responsible layer
00:12:29
um you know issues of like fairness and
00:12:31
privacy and um robustness and and so
00:12:35
forth right because if if you're a
00:12:38
product manager for example you're
00:12:39
trying to produce the best products no
00:12:41
one really wants to deal with a company
00:12:43
whose products are reaching you know
00:12:46
their privacy uh you know their their
00:12:50
privacy obligations and stuff like that
00:12:52
like if you're using personal data uh of
00:12:55
your users to train your models or if
00:12:58
your products don't work for certain
00:12:59
segments of the population or exposes
00:13:01
your users to specific types of harms
00:13:04
you know cutting Corners like that
00:13:06
really does a disservice uh to your
00:13:09
stakeholders of course uh but it also
00:13:12
exposes you to legal and reputational uh
00:13:16
risk and it's just a bad way of doing
00:13:19
business it's just a matter of time
00:13:21
before you know uh better companies are
00:13:25
able to gain more market share because
00:13:27
they're taking this more um seriously
00:13:30
and investing in setting up the
00:13:32
infrastructure yeah you know a lot of
00:13:34
what I hear you saying is you know
00:13:37
sometimes it's easy to think that ethics
00:13:41
and responsibility questions of
00:13:42
responsibility isn't the job of a
00:13:45
corporation but you take a strong stance
00:13:47
and certainly through your own
00:13:48
dissertation work and your own
00:13:49
scholarship is that these are things
00:13:51
that are not nice to haves um there must
00:13:54
haves and and you know while it's it
00:13:57
shouldn't always come down to this we do
00:14:01
have a system called the legal system
00:14:03
which is set in place to help adjudicate
00:14:06
these issues when it does feel like it's
00:14:08
a great area even though a gray area
00:14:10
excuse me even might be black and white
00:14:12
with respect to businesses and their
00:14:14
responsibility to ensure that they're
00:14:16
not inducing harm uh through their
00:14:18
activities brri I mean certainly I know
00:14:21
that you do work with companies as well
00:14:23
I I would also like you to put on your
00:14:25
researcher and educator hat um and help
00:14:28
us to understand like the nature of this
00:14:32
conversation from a scientific
00:14:34
perspective and also in the work that
00:14:36
you're doing you know as a professor as
00:14:38
part of your fellowship I'm actually
00:14:41
thinking a lot about chat GPT these days
00:14:44
certainly because as an educator this is
00:14:46
sort of the one this is probably the
00:14:48
most I know about this conversation
00:14:49
around Should students be able to use
00:14:52
GPT to complete their assignments and it
00:14:54
seems to me like it's such a polarizing
00:14:56
sets of issues um so I am just just
00:14:58
curious as you think about the the
00:15:00
places where you set project um how
00:15:03
you're thinking about the issue around
00:15:06
businesses and employers and Ed
00:15:08
education's role in this conversation so
00:15:11
I think a couple of thoughts right so
00:15:13
first uh to piggy back on Kareem when I
00:15:17
talk to both students and businesses I
00:15:19
think of myself as an educator no matter
00:15:21
where I am right and I have the same
00:15:23
thing I tell them uh which is if you get
00:15:27
this wrong right you don't include uh a
00:15:31
wide SWA of human beings in the creation
00:15:33
of your technology
00:15:35
products when you fail because it's not
00:15:38
an if but when you fail you lose money
00:15:41
because you spent all this money on
00:15:42
development without considering uh the
00:15:45
human beings at the end and when the
00:15:47
product gets released the product fails
00:15:49
again because people don't use it right
00:15:53
so as a really simple example uh those
00:15:56
automatic uh faucets where you put put
00:15:58
your hand under it did not
00:16:01
include uh like darker melanated people
00:16:04
in the data set in the training set for
00:16:07
when they were testing out this
00:16:08
automatic hand washer and so while it
00:16:10
sailed sold fine in North America when
00:16:13
it went further out into uh places
00:16:17
closer to the Equator where people were
00:16:18
Browner regardless of uh ethnic origin
00:16:21
they didn't sell any because the product
00:16:24
did not work they would install it and
00:16:26
people put their hand under the thing
00:16:28
and I know if you're a black or a brown
00:16:30
person you have done this at the airport
00:16:32
bathroom been upset right but if your
00:16:35
country is majority melanated then no
00:16:39
one bought them and so have they
00:16:41
included those folks earlier in the
00:16:44
development process like all that money
00:16:47
they spent on development uh would have
00:16:49
been worth it because they could have
00:16:50
sold uh more products and so when I'm
00:16:53
talking to my students and Business
00:16:56
Leaders who in some ways are my students
00:16:58
uh uh I go look this is why we do it
00:17:01
like I'm I'm going to speak to the same
00:17:03
incentives that you have you want to
00:17:04
make more money you want to keep your
00:17:06
job uh you want to do right by your
00:17:09
shareholders then you need to move uh
00:17:12
the process or move the people earlier
00:17:14
into the development of this process now
00:17:17
in terms of this question around chat
00:17:20
GPT and uh the way that uh people are
00:17:24
using this inside and outside the
00:17:26
classroom uh we developed the triap lab
00:17:29
this these three questions call the 3D
00:17:31
model of Equitable Tech adoption right
00:17:34
and I'm going to share those questions
00:17:36
with you and this will think help you
00:17:38
understand help people understand what
00:17:40
do I do with any Tech that comes into my
00:17:42
business or into my classroom those
00:17:44
three questions question one what does
00:17:47
this product actually do not what do you
00:17:50
want it to do not what it might do in
00:17:52
the future not what it could do if we
00:17:55
had uh a billion more hours of Compu
00:17:58
like what does it actually do today
00:18:01
second question is who does this product
00:18:04
disempower all right every technology
00:18:07
will increase power for some and
00:18:08
decrease power for others so ask
00:18:10
yourself who does this disempower and
00:18:13
then the third question is what is the
00:18:16
daily use of this product not the edge
00:18:19
case don't tell me that you know we're
00:18:22
going to uh I give you a per example for
00:18:27
Hispanic Heritage Month there is uh some
00:18:29
question that maybe we should include in
00:18:31
facial recognition whether or not uh
00:18:35
this person is Hispanic or this person
00:18:37
is uh Latino and the sales pitch for
00:18:42
this is that this will help us when we
00:18:43
do the census to identify who is
00:18:46
Hispanic or Latino so we have a better
00:18:48
count right now I can get into like the
00:18:50
research and on this and why this is
00:18:52
problematic or some of the philosophy of
00:18:54
like how this goes arve but let's talk
00:18:56
about that case they're saying we're
00:18:58
going to use this for the census the
00:19:01
census happens every 10 years all right
00:19:04
there is no government service that's
00:19:06
going to buy this expensive technology
00:19:09
and then only use it once every 10 years
00:19:12
so then what's the most likely daily use
00:19:16
case right of a technology that uh like
00:19:21
says this person Hispanic or not right
00:19:23
are they going to install it at Borders
00:19:26
probably uh would they install where you
00:19:29
hand over your uh passport probably
00:19:32
right and is that problematic definitely
00:19:34
right and so this is how I'm thinking
00:19:36
about these things uh and in the case of
00:19:38
chat GPT your students should ask
00:19:40
themselves what does this actually do
00:19:43
right is this writing papers or is it
00:19:47
just creating plausible sounding
00:19:49
sentences right and if it's just
00:19:51
creating plausible sounding sentences o
00:19:53
that could get real bad real fast uh if
00:19:56
that plausibility has no relationship to
00:19:59
accuracy all right so you know feel free
00:20:02
to use it and feel free to fail that's
00:20:04
on you
00:20:05
uh that's come down to it so you're
00:20:08
doing a great job both of you already of
00:20:10
raising some of these key challenges but
00:20:12
I'd like to explore these more and
00:20:15
certainly you know we don't have time
00:20:17
for tap tens so we'll do and I'm sure
00:20:19
that there are 10 on each of your lists
00:20:21
but let's think about this in terms of
00:20:23
the top two or three issues from where
00:20:25
you sit so project let's go back to you
00:20:27
you can those that you've already talked
00:20:29
about or you know invite others into the
00:20:31
conversation what would you say from
00:20:33
where you sit are the top two to three
00:20:36
issues with respect to responsible and
00:20:38
fair AI in the workplace and the
00:20:42
marketplace uh two things uh the number
00:20:47
one issue is that the human beings that
00:20:52
end up using the AI or being users of
00:20:56
machine learning or being users of this
00:20:58
generative AI aren't necessarily the
00:21:01
people that are included in the data all
00:21:04
right aren't necessarily the people that
00:21:05
are included in the classification of
00:21:07
the data and definitely aren't the
00:21:09
people that are setting the rules and
00:21:11
doing the coding of the data and so the
00:21:13
most pressing uh issue is to get those
00:21:17
people into those spaces right to get
00:21:21
representative data to get
00:21:23
representative uh classification of that
00:21:25
data to get representativeness in the
00:21:28
actual coders that decide the rules for
00:21:31
what we see and then that way the stuff
00:21:33
that actually comes out will be closer
00:21:36
to fair and Equitable because those
00:21:38
people would be in the room so that's
00:21:41
one the second thing is that all of my
00:21:43
research uh is around this space is
00:21:47
really to
00:21:48
demystify this black box there is
00:21:51
nothing magic all right going on inside
00:21:54
of these statistical models they are
00:21:56
statistical models and if you learned in
00:21:59
high school y equal MX plus b then you
00:22:04
like have the building block that you
00:22:07
need to understand how these systems
00:22:09
work uh we can talk about this if you
00:22:12
ever come hang out on the Trap lab but
00:22:14
trust me if you know y equals MX plus b
00:22:17
all right then you too can start to
00:22:20
understand that it's not magic going on
00:22:22
inside of these systems but just a bunch
00:22:25
of opinions uh around this commodify
00:22:28
human labor and the data the
00:22:29
classification and the code and it's
00:22:32
interesting you know we've sort of
00:22:33
hinted around sort of like language here
00:22:36
um in this case I think I find myself
00:22:39
personally feeling like it does sound
00:22:42
like it's magic because the the language
00:22:44
of AI and technology that we're using to
00:22:47
talk about these things sounds very
00:22:49
foreign to me and so I think that is
00:22:51
what gives it its Mystique is if I can
00:22:54
talk about it in a way that doesn't
00:22:56
coincide with lay ter terminology that
00:22:58
regular people use then it sounds like
00:23:01
it's inaccessible to me right it's like
00:23:03
it's a is that a power move I don't know
00:23:06
but language is powerful and language
00:23:09
can be used as a way to exclude and in
00:23:12
this case you know I think what you're
00:23:13
helping us to understand brck is you
00:23:16
many of you because you learned y equals
00:23:18
MX plus b you know that's a language
00:23:21
that you spoke at some point and to the
00:23:23
extent that we give people a grounding
00:23:25
for understanding this new technology
00:23:28
through something that they already
00:23:30
understand language that they already
00:23:31
possess then the I think the Mystique
00:23:34
arounds it disappear the power
00:23:36
differential between the creators of the
00:23:39
technology and the consumers and the
00:23:40
employees who were trying to figure out
00:23:42
what does this mean for them that starts
00:23:44
to be reduced as well is that sort of
00:23:46
along some of the same lines R of of
00:23:48
what you're suggesting that's exactly it
00:23:50
right so like if you hear the term oh
00:23:52
chat gbd says they have a billion
00:23:55
parameter model right you'll go oh are
00:23:58
big words billion parameter what does
00:24:00
that mean let's explain it right let's
00:24:02
break it down real quick so let's go
00:24:04
back to this yals MX plus b b is call it
00:24:08
our Y intercept some error term we're
00:24:09
not going to worry about that we're just
00:24:11
going to focus on the yals MX Y is an
00:24:14
output every computer every machine
00:24:17
right they're all uh based on the
00:24:19
touring machine nothing really changes
00:24:21
they can only do what they've been told
00:24:22
to do why is going to be the output that
00:24:24
comes out all right that could be
00:24:26
plausible sounding sentences
00:24:28
that could be art of Jesus flipping over
00:24:30
a table whatever all right and then M
00:24:33
and X is what matters X is inputs so if
00:24:37
I'm deciding I'm building a system
00:24:39
that's going decide whether or not uh
00:24:42
someone gets parole for instance those
00:24:44
X's might be ZIP code right problematic
00:24:47
those X's might be past criminal history
00:24:49
those X's might be height right those X
00:24:52
could be anything then the m is what
00:24:55
matters right the m is a essentially the
00:24:58
slope of that line we learned this in uh
00:25:02
uh 10th grade but that slope is an
00:25:04
opinion it is some developers opinion or
00:25:07
if it's unsupervised it's still some
00:25:09
developers opinion on how much that X
00:25:13
matters how much does it matter that I
00:25:15
live in the zip code how much does it
00:25:17
matter that you know I'm six foot6 right
00:25:20
how much does it matter that my name is
00:25:21
broaderick and then like that opinion
00:25:25
gets added in so each one of those mxs
00:25:27
we can call that
00:25:28
a parameter so if it's a billion things
00:25:30
has a billion opinions but they all get
00:25:32
filtered out and come out to this output
00:25:35
why that's it we have now learned
00:25:37
machine learning congratulations
00:25:39
everybody right like Pat yourselves on
00:25:41
the back if you learned why goes MX plus
00:25:43
b you too uh can start to understand
00:25:47
what's going on these systems you can do
00:25:49
like Kareem or myself and run audits
00:25:51
where you basically test these systems
00:25:54
uh with a bunch of stimuli to see what
00:25:57
comes out to explain uh this thing is
00:26:00
maybe leading to inequality because of
00:26:02
uh this weirdness right and so let's
00:26:05
just demystify the whole thing it's not
00:26:07
I mean it's complex let me not poo poo
00:26:09
my computer science folks out there but
00:26:13
uh we can share a language that inside
00:26:16
of you know this increasing complexity
00:26:19
comes down to a pretty simple building
00:26:21
block and you know the building block
00:26:22
already if you made it through high
00:26:24
school and our Wharton grads or NBAs and
00:26:28
undergrads I know uh that you learned y
00:26:31
equals MX plus b I have to tell you it's
00:26:33
been a long time since I learned why
00:26:36
equals MX plus b I'm not sure my high
00:26:38
school teacher was as great as you were
00:26:40
explaining that but I feel like I have
00:26:41
in the last three minutes a much better
00:26:44
understanding of what it is that people
00:26:46
are trying to suggest is sophisticated
00:26:48
um which often I think sometimes feels
00:26:51
inaccessible but I think you made me
00:26:53
believe and I'm sure others believe that
00:26:55
they too can understand sort of like
00:26:57
what the big deal here is and assess for
00:26:59
themselves is it a big deal or is it
00:27:01
just more uh of what we already know
00:27:04
Kareem let me turn to you and let's get
00:27:06
your top two to three issues I don't
00:27:08
know how much they overlap I know
00:27:09
earlier you spoke about you know issues
00:27:12
around representation in the data not
00:27:14
sure if that's one of your top two or
00:27:16
three issues if you have others can you
00:27:17
share with us um from your Vantage Point
00:27:21
what are some of the key
00:27:22
challenges yeah so first thing I agree
00:27:24
with h bradrick in that you know
00:27:26
transparency is a is a major issue right
00:27:28
the Splat boox problem and trying to
00:27:31
basically understand uh what companies
00:27:33
are doing are companies being
00:27:35
transparent about how they're training
00:27:37
these models what data sets they're
00:27:39
using are they explaining to their users
00:27:41
what is happening behind the scenes
00:27:43
obviously there's an optimal level of
00:27:45
transparency as well like transparency
00:27:47
after specific um you know uh particular
00:27:51
level might get too in depth to the
00:27:53
extent that the uh average user might
00:27:56
zone out right it's irrelevant in
00:27:57
information right you don't want to
00:27:58
inundate your user with with uh too much
00:28:02
technical details that they really zone
00:28:04
out um so for me the the first pressing
00:28:07
challenge is addressing uh unfairness in
00:28:09
AI systems right so if you're training
00:28:12
data like Rodrick has mentioned is
00:28:14
missing certain populations or if your
00:28:16
data is Mis mislabeled obviously that
00:28:18
can give rise to bias and can have
00:28:20
adverse effects on um you know certain
00:28:24
segments of the population uh this
00:28:26
particularly becomes uh you know
00:28:29
problematic in issues of like Health
00:28:32
Care
00:28:33
employment uh you know uh criminal
00:28:36
justice where these decisions are
00:28:39
consequential for people right um so if
00:28:43
obviously if these uh these issues of
00:28:45
bias are not left unaddressed uh they
00:28:47
can perpetuate uh unfairness in society
00:28:50
at a very high rate right we're not just
00:28:53
talking about your prototypical kind of
00:28:57
BU we're talking at like an exponential
00:29:00
rate with these automated decision
00:29:01
systems which is why they um they can be
00:29:05
very dangerous right so the second
00:29:07
problem I see is what is called the
00:29:09
hallucination problem which is pretty
00:29:11
much like making up stuff right like
00:29:14
rodick had mentioned these llms learn to
00:29:16
predict the next word right or phrase in
00:29:19
a sentence so um they can misrepresent
00:29:24
facts uh they can also tell you a very
00:29:26
good story uh that has a very good
00:29:30
narrative uh that is extremely plausible
00:29:34
uh but is misleading right and I think
00:29:36
this is a particularly big problem uh
00:29:39
given that we as human beings we have
00:29:41
like an automation bias where uh we have
00:29:44
a propensity to kind of favor
00:29:45
suggestions from an automated decision
00:29:48
system uh and to ignore um contradictory
00:29:52
information that uh that you know we
00:29:55
might know right we just can't kind of
00:29:57
defer to uh this automated decision
00:30:01
system right um the third one I would
00:30:04
say is kind of like data privacy and
00:30:06
security concerns and this involves
00:30:09
things like unauthorized access uh of
00:30:12
data scraping of data that for example
00:30:15
the the the company or the llm uh might
00:30:18
not have uh access to uh you know or
00:30:22
consent to use uh things like data
00:30:25
leakage for example where
00:30:27
a large language model might um you know
00:30:31
present some information that a user had
00:30:34
used um and we have cases like that
00:30:36
coming up in in the media where Samsung
00:30:40
for example has banned uh the use of
00:30:42
chat GPT VI its uh employees because of
00:30:45
the fact that uh the the llm was uh you
00:30:49
know revealing some of these Trade
00:30:51
Secrets uh so things like malicious use
00:30:54
and manipulation attempts by nefarious
00:30:57
actors these are all very serious
00:30:59
concerns as
00:31:00
well yeah so I as as I listen to you all
00:31:04
talk about these concerns it sort of
00:31:07
raises the concerns that I've just had
00:31:10
again just as an an employee right and
00:31:12
certainly as as a consumer and and
00:31:15
certainly what came across my uh social
00:31:17
media feed recently was uh in the last
00:31:21
couple weeks there was a uh lawsuit
00:31:24
class action lawsuit uh filed in the
00:31:27
Northern District of
00:31:29
California um the plaintiffs are open Ai
00:31:33
and chat GPT and the Microsoft cor
00:31:36
Corporation um and one of the big areas
00:31:40
in included in this um lawsuit is
00:31:44
invasion of privacy and it's it's pretty
00:31:47
compelling again I'm not a lawyer no and
00:31:49
we none of us knows how this is going to
00:31:51
pan out but just looking at it to
00:31:54
understand the the lengths at which
00:31:56
people people um don't understand how
00:32:00
their data is being accessed and
00:32:01
utilized without their permission um is
00:32:06
it can be concerning and so I would
00:32:08
encourage people to it's a it was a nice
00:32:11
tutorial I would say if you will for me
00:32:12
in privacy concerns around this topic
00:32:15
but I'm going to certainly um put punt
00:32:18
back to the two of you you come and if
00:32:21
you've had a chance to like look at
00:32:23
anything in relation to this class
00:32:25
action lawsuit I'm wondering if you're
00:32:28
surprised by this lawsuit or if you just
00:32:31
thought that this was going to somebody
00:32:33
it was just a matter of time before it
00:32:34
happen any thoughts on that baric and
00:32:37
then I'll come back to you kareim uh
00:32:39
yeah so clearly it was only a matter of
00:32:42
time and we think about again what do
00:32:45
these systems actually do all right so
00:32:48
these large language models train uh on
00:32:52
previous data and training just means
00:32:54
that they take in a bunch of data so
00:32:56
they then connected uh together but
00:33:00
whose data did they take where did that
00:33:02
data come from now there is some
00:33:04
indication that you know when you're
00:33:07
training on a huge Corpus of uh data
00:33:12
that you're goingon to tend towards the
00:33:14
cheapest and freest sources right this
00:33:17
is why we get some a lot of weirdness
00:33:19
that comes out of these systems in terms
00:33:21
of bias because they're trained on the
00:33:23
free parts of the internet uh a lot of
00:33:26
the time and what is over represented in
00:33:29
the free Parts the internet
00:33:31
Stephanie the free parts are people who
00:33:35
I'm trying to think about social media
00:33:37
as an example as a source of data and
00:33:39
then I'm thinking about who the users
00:33:41
might be um I'm thinking about um not
00:33:44
sure if this is where you're going but I
00:33:46
think about all the young people who use
00:33:48
social Med put all of their
00:33:51
information resources yeah so we so we
00:33:53
have that right we have an over
00:33:54
representation of younger folks all
00:33:57
right so we're going to have weird age
00:33:59
related things that don't really exist
00:34:01
in the data the other thing that gets
00:34:03
over represented on the internet on the
00:34:05
free Parts at least is propaganda right
00:34:08
so some websites where they'll say
00:34:11
really negative things about presidents
00:34:13
for instance are free whereas if I go to
00:34:16
I don't know the New York Times I can
00:34:17
read four articles before I get to a
00:34:19
firewall and they're like no sir you're
00:34:21
done right uh and so there's going to be
00:34:23
some weirdness that's in the data from
00:34:26
that the other that's free on the
00:34:28
internet or over represented for free on
00:34:30
the internet is pornography right so if
00:34:32
I'm taking in porn and propaganda into
00:34:35
these systems right then I'm going to
00:34:37
get all types of weirdness out and the
00:34:40
only way to improve these things is to
00:34:43
get access to uh you know data that has
00:34:48
closer to accuracy that has time spin
00:34:50
into it you want to read one of our
00:34:52
research articles Stephanie how much
00:34:53
does it cost if you're like not
00:34:54
affiliate with school it's like
00:34:55
thousands of dollars right and so if
00:34:58
you're one of these companies and you
00:35:00
want to improve your data how do you do
00:35:02
it do you spend thousands of dollars per
00:35:05
article from Stephanie Cur or you steal
00:35:07
it right is that like a a rhetorical
00:35:12
question you can't say but I'll answer
00:35:15
the question the answer is you're
00:35:17
probably you're probably not gonna spend
00:35:19
thousands of dollars per article per
00:35:22
Professor If instead you can steal it
00:35:25
right and this goes for any of we'll
00:35:27
call it better data uh because I need a
00:35:30
bunch of it to get to some version of
00:35:33
it's not accurate but a better uh
00:35:36
distribution of data and so yeah of
00:35:38
course they're getting sued because to
00:35:41
make the system better they had to take
00:35:43
this stuff in and that we do have
00:35:46
intellectual property laws somebody's
00:35:48
going to have to pay or they're gonna
00:35:49
have to change the law Kareem do you
00:35:52
want to chime in on this conversation
00:35:53
before we move to the how do we fix it
00:35:56
yeah absolutely I I think um you know
00:35:59
lawsuits are just going to mushroom from
00:36:01
here onwards uh it's just a matter of
00:36:03
time honestly you know every day we're
00:36:06
hearing about another lawsuit uh you
00:36:09
know whether it's relating to privacy or
00:36:11
discrimination or um other aspects um so
00:36:16
to me I'm I'm not too surprised I think
00:36:19
uh these companies are you know
00:36:22
obviously I I can't speak on on their
00:36:24
behalf but you know we have a responsib
00:36:26
ility to make sure that we're training
00:36:28
our llms on reliable data sources uh we
00:36:32
need to kind of fine-tune these llms we
00:36:34
need to have like some kind of a
00:36:36
factchecking mechanism to ensure uh you
00:36:39
know if they're producing uh garbage
00:36:41
that you know they're they're being kind
00:36:44
of they're receiving feedback and and
00:36:45
they're improving we have to have human
00:36:48
reviewers in the loop uh that can play a
00:36:51
role in in correcting like kind of the
00:36:54
the trajectory of of these llms um you
00:36:59
know perhaps even including like
00:37:01
confidence scores to allow users to kind
00:37:03
of understand Engage The reliability of
00:37:06
the information that they're getting
00:37:08
right and uh and even to uh provide
00:37:12
users with a mechanism to provide
00:37:15
feedback for the llm as to whether you
00:37:18
know the response that I got was
00:37:20
terrible or not so that uh they're
00:37:23
taking in another signal from their
00:37:25
users so I think in conjunction all of
00:37:28
these different measures uh hopefully
00:37:30
over the course of time and as long as
00:37:32
there's you know strong Buy in from
00:37:35
leadership to kind of improve these
00:37:37
models I think um companies can do a
00:37:40
better job at uh protecting the privacy
00:37:43
of people but also ensuring that you
00:37:45
know whatever gets propagated uh in
00:37:47
terms of outputs is more reliable okay
00:37:51
uh any other suggestions Solutions um
00:37:54
around that or the other challenges that
00:37:57
you and brck brought up today um in like
00:38:01
we think about like one or two key
00:38:03
things that can be done uh for various
00:38:06
audiences and I think I'm thinking of
00:38:08
lots of audiences I'm thinking of um you
00:38:11
know employers and institutions I'm
00:38:14
thinking of consumers and employees as
00:38:20
well yeah there's I let kareim go first
00:38:22
on that one yeah I think there's the
00:38:25
grass is so green um there is so much
00:38:29
that can be done in this space um so
00:38:32
firstly there needs to be uh legislation
00:38:34
there needs to be comprehensive AI
00:38:36
legislation and enforcement to protect
00:38:38
public interests uh not sufficient for
00:38:41
companies to have voluntary commitments
00:38:44
that's good but it uh it's not
00:38:46
sufficient second thing is you know
00:38:48
there needs to be um when we're thinking
00:38:50
about data protection laws uh they need
00:38:53
to be strengthened to include you know
00:38:55
AI systems and Ure that um the Privacy
00:38:58
rights of users and so forth uh you know
00:39:02
we spoke about transparency and that
00:39:04
companies uh need to disclose how AI
00:39:07
systems have been built what data
00:39:09
sources they've been using and to kind
00:39:12
of um to take a crack at the blackbox
00:39:15
problem uh stakeholder engagement is
00:39:17
really important you know as we're
00:39:19
thinking through diverse audiences and
00:39:22
data uh sources uh you need to engage
00:39:25
with different stakeholders
00:39:27
I always say when you're working in this
00:39:29
space you really need to get it like be
00:39:31
cross functional because there's a lot
00:39:34
of barbwire in this space you know
00:39:35
whether it is you know privacy legal
00:39:38
regulatory you're working with ethicists
00:39:40
product managers you're working with uh
00:39:42
data scientists researchers uh you know
00:39:45
Eng it is extremely cross functional
00:39:48
space and it has to be that way because
00:39:51
it is a soot technical problem that uh
00:39:54
takes all the great minds to be able to
00:39:56
attack it from different ways you need
00:39:58
to have diverse teams for example so
00:40:00
that you have representation people can
00:40:03
identify and rectify bias effectively
00:40:05
right training data needs to be more
00:40:07
inclusive make sure you know like
00:40:09
bradrick was saying who's missing out
00:40:11
from this data like who are we not
00:40:13
seeing you know I always say that um
00:40:15
this technology is not neutral um this
00:40:18
technology is already has a point of
00:40:21
view and its point of view is what it
00:40:24
has been trained on right so if it has
00:40:28
been missing people of color for example
00:40:30
then that's its point of view right it's
00:40:33
not like it's coming as a blank slate no
00:40:36
it is already has a point of view and so
00:40:39
uh that's a little bit of mythbusting so
00:40:42
I mean we need to kind of like take
00:40:45
inclusive data and inclusive uh
00:40:48
inclusivity into consideration uh
00:40:51
throughout the product development life
00:40:53
cycle right we need to have like audits
00:40:56
that are done on uh these these models
00:41:00
and and ensure that you know we have uh
00:41:03
results and that can kind of feed into
00:41:06
how we can improve uh human oversight
00:41:08
for example I mentioned this earlier
00:41:10
that there needs to be a human in the
00:41:12
loop to be able to kind of uh you know
00:41:15
spot and uh improve the trajectory of uh
00:41:19
outcomes I mean there's so much that can
00:41:21
be done I can go on and on and on
00:41:25
but start
00:41:27
yeah so much brck your turn what would
00:41:30
you say um are some key specific things
00:41:33
that can be done so I'm going to break
00:41:35
this down into uh different market
00:41:37
segments because that's how my brain is
00:41:39
wired so we're going to talk about what
00:41:41
can companies do what can consumers do
00:41:43
what can researchers do and then what
00:41:46
can your students do all right so first
00:41:49
for companies consider that 3D model
00:41:53
that I laid out earlier before you roll
00:41:54
out a product what is this product do
00:41:56
who does it disempower uh and what uh is
00:42:00
its daily use all right if you need help
00:42:04
answering these questions then call RA
00:42:07
audits uh you know Kore will pick up the
00:42:09
phone uh or you can holler at the folks
00:42:12
over here in the Trap lab right and we
00:42:13
can help you think through some of these
00:42:15
things uh as we're trying to make this
00:42:18
technology better and more Equitable all
00:42:20
right two consumers what can you do do
00:42:25
not accept that It's Magic there is
00:42:28
nothing magical about these systems
00:42:31
these systems are just people and their
00:42:33
opinions of people if you learned yals
00:42:37
MX plus b then you have learned the
00:42:39
building blocks of every machine
00:42:42
Learning System all right so if there is
00:42:45
some weird outcome that comes out when
00:42:47
you're on Facebook or Twitter or you
00:42:50
notice some weirdness on your uh when
00:42:53
you you put out an application for a
00:42:55
loan
00:42:56
trust your gut all right it is weird and
00:42:59
say something because again the only way
00:43:00
to improve these things as well is uh uh
00:43:04
for them to update their model and it's
00:43:06
you you are right that something's wrong
00:43:09
all right it is not magic is not your
00:43:11
fault is them uh don't accept magic for
00:43:16
researchers if you are interested in
00:43:18
working on these topics and in this
00:43:20
space go to jointhe trap.com and come
00:43:23
hang out with us uh we meet online once
00:43:27
a week over Zoom Wednesdays at 1 pm uh
00:43:31
and then finally for students this is
00:43:34
going to be a weird one uh because in
00:43:37
some ways this rise of uh generative Ai
00:43:43
and writing and art uh makes people
00:43:46
believe that we are like this close to
00:43:50
uh having AI that can write beautiful
00:43:53
books or make beautiful art uh and I'm
00:43:57
going to challenge that all right we
00:43:59
they also told us we were this close to
00:44:01
having fully self-driving cars uh within
00:44:04
a year 10 years ago and they said this
00:44:06
every year and it's not happening
00:44:08
anytime soon all right I'm going to say
00:44:10
the same thing is the case for art and
00:44:13
writing and uh
00:44:16
creativity I think that they will be a
00:44:18
premium uh on people that can actually
00:44:21
Express themselves clearly and
00:44:24
accurately and honestly and so if you
00:44:27
are currently a student uh anywhere if
00:44:29
you're in a b school if you're in a
00:44:31
college if you're in high school and
00:44:33
you're listening to this I don't know
00:44:34
why you're listening to a Wharton
00:44:35
podcast in high school but maybe you
00:44:37
know you're one of those kids uh get
00:44:40
really into building your uh toolbox of
00:44:45
creativity right get really into uh
00:44:49
creative writing right get really into
00:44:51
making more art because there will be a
00:44:54
value on the ACT actual human element
00:44:57
that comes out of this if uh what
00:45:00
everyone else is doing is just plugging
00:45:02
in some chatbot that gives you uh
00:45:05
plausible sentences that aren't that
00:45:07
good if you can write better sentences
00:45:10
you'll
00:45:11
win thank you and and I'm just going to
00:45:13
tell you project we actually do have
00:45:15
knowledge at war in high school so your
00:45:18
high school learners are going to learn
00:45:20
a lot from what you all share today as
00:45:23
much as your you know fully experienced
00:45:25
senior leaders will um as well uh so so
00:45:29
broject and Kem this has been fantastic
00:45:31
I feel so much more knowledgeable as
00:45:34
somebody who believes that I know a lot
00:45:36
about a lot of things I I know that I
00:45:38
don't know a lot about this topic and so
00:45:40
the past amount of time 45 minutes that
00:45:42
we've been chatting have been amazing
00:45:44
and just I think helping me to feel more
00:45:46
empowered um as a researcher a consumer
00:45:49
and a worker around sort of what exactly
00:45:52
is happening and and how I need to pay
00:45:54
attention um to to what is being shared
00:45:58
um so I want to thank you uh so much for
00:46:00
sharing your insights and your expertise
00:46:02
with us uh and to all the uh leading
00:46:04
diversity at work podcast listeners we
00:46:07
truly appreciate you for being here uh
00:46:09
so that's all for now thanks to our
00:46:12
audience for joining us and listening to
00:46:14
this episode of the knowledge at Warton
00:46:15
leading diversity atw work podcast
00:46:17
series goodbye for
00:46:19
now for more insight from knowledge at
00:46:22
Wharton please visit knowledge. won.
00:46:26
[Music]
00:46:34
EDU
