00:00:01
This podcast is brought to you by
Knowledge at Wharton. Angie Basiouny: Welcome to
Knowledge at Wharton.
00:00:14
I'm Angie Basiouny. If I ask you If I ask you to think of a doctor, your brain might
conjure up the image of an
00:00:19
older white male. If I ask you
to think of a fitness enthusiast, it might be a
younger, slender person. These
00:00:26
are biases that are baked into
our brains, and unfortunately, they also get baked into AI. I'm
here today with Wharton
00:00:33
Professor Kartik Hosanagar and his
colleague, Pushkar Shukla, who's a PhD candidate at the Toyota
Technological Institute at Chicago.
00:00:41
They're part of a team
that has created this new tool that can correct, detect, biases
in text-to-image generators.
00:00:49
Those are the machine learning
models that can produce a visual image for you with just a few
prompts. Gentlemen, I'm so glad
00:00:56
to have you here today. Kartik,
it's our first time talking on this podcast. Welcome.
00:01:00
Thanks for having me, Angie. A great pleasure to
meet you in person. We've interacted many times
before, but lovely to see you.
00:01:07
Same here. And Pushkar, welcome to the
Wharton family. Thank you for making some time and joining us
today.
00:01:12
Thank you so much, Angie. It's a
pleasure meeting you and interviewing for this podcast.
- Absolutely.
00:01:18
So when I first read this paper,
I just thought, what a difficult problem to solve for in real
life. I can't even imagine how
00:01:25
difficult it is to solve for it
on a technology platform. And that's why I'm really excited to
share these details about this
00:01:32
tool today. Let's just start
with an overview, if you will, Professor. Give us an overview
of this research, what you were
00:01:38
trying to solve for, what the
problems are. Yes. Our research was motivated
by the widespread use of
00:01:46
generative AI systems today to
generate all kinds of content. Certainly text through systems
like ChatGPT, but also a lot of
00:01:53
image generation through tools
like Stable Diffusion, Mid- journey, Meta's AI, and even ChatGPT
does image generation. And so
00:02:02
on. And while they're being used
at an unprecedented volume or scale today, these systems do
have biases. It's kind of what
00:02:11
you were mentioning earlier.
They're trained on data. Those data are generated by humans who
have their own biases, and so
00:02:18
they pick up their biases, and
in many ways institutionalize those biases, and kind of do
that going forward. So if you
00:02:24
look at image generation as an
example, in these systems, if you type in, say, "computer
programmer", you're probably
00:02:31
likely to see most of the images
being of males. And so these systems are prone to biases. But
generally the biases get
00:02:42
detected after the fact, and
usually by a human user using them and kind of saying, "Hey,
I'm noticing a pattern." Or a
00:02:50
journalist like you might notice
a pattern and write about it. So we've seen, for example,
Google's Gemini having bias
00:02:57
issues, and people wrote about
it. And similar things have been written about pretty much every
image generation model. So we
00:03:04
set out to come up with a system
that can automatically detect biases in these systems. And
that automatic detection is
00:03:13
extremely important because of
the scale at which they're being used, but it's also a very hard
problem. And so we wanted to be
00:03:19
able to detect them, and not
just detect simple or the common biases, like, say, gender and
age, but all kinds of potential
00:03:27
biases, and to be able to provide explanations to
human decision makers. So that's what we set out to do
and solve with this project.
00:03:35
I think that it would be helpful
if we could show some of the biased images that we're talking
about, and that's where I'm
00:03:41
hoping, Pushkar, that you can help
us with that. Share your screen and show us some of these images
00:03:45
that you're studying
in this project. The first set of images talks about
a computer programmer. So if you
00:03:51
ask an AI generative model— and in
this case, we used Stable Diffusion— to generate images of
a computer programmer, this is
00:03:57
how these images look like.
Similarly, in the second set of images, I ask the generative AI
model to generate images of old—
00:04:05
of an old man at a church. Now,
before I go on and talk about the possible directions in which
these images can be biased, I
00:04:13
want to take a second and ask
you two, what do you think are the possible directions in which
these images can be biased?
00:04:20
Well, most definitely gender,
right? For the first one, a computer programmer, the images
are always going to be male. I
00:04:28
noticed in your paper that you
had a collection of images when you prompt for, for example,
childcare worker, and all the
00:04:34
images were female. Another—
another stereotype. I would also say maybe the— the image about an
old man in a church. The church
00:04:41
always has a similar look. So
the image of the church in this case is the same. It appears to
be maybe a Catholic Church. The
00:04:48
stained glass windows, the
arched— you know, the arched windows. So again, it's sort of
an image that we have for maybe
00:04:54
even from our school days of
studying the Renaissance. Yeah. And I was going to add
that— well, at least the
00:05:02
computer programmer images, in
terms of race, seems to be diverse, but in terms of the old
man at the church, from what I
00:05:11
can see, they're all white male.
I can— perhaps also the computer programmer seems to be based on
these four images, all young.
00:05:22
And so there's another
stereotype I'm seeing there, but that's what I'm seeing visually,
at least.
00:05:30
Cool. So I think all the axes of
biases that you mentioned are correct, but there is more to
it. So when we talk about— you
00:05:39
talked about gender, ethnicity,
age being important axes of biases, but let's look at a few
other axes of biases. All
00:05:46
computer programmers are lean.
So there is clearly a body type issue there. And that's—
similarly, the physical
00:05:55
appearance of old men at a
church are of a certain type. They look grim, slightly
depressed or sad, and also
00:06:05
things such as, there might be
ableism bias. Or in our research, we found that
oftentimes old men were
00:06:14
portrayed to have a certain kind
of disability. So the point that I'm trying to make is when we
think about biases in AI, we
00:06:21
generally think about the common
dimensions of biases such as gender, ethnicity, culture,
maybe age. But there is more to
00:06:30
it. There are other equally
important dimensions of biases, such as socioeconomic bias,
disability, body type and health
00:06:39
bias of some sort. So— and that
is something that needs to be looked upon, and that is what
makes it
00:06:45
a tough problem to solve. I'm going to ask both of you to
answer this question. Why? Why
00:06:50
do we need to look at it? Why is
it important to correct this kind of bias? Well, quite simply, these biases
will continue to be there in
00:07:02
society, but also at a scale
that we have probably not seen before. If you have a human
being generating images in some
00:07:10
marketing agency, and that's
going to be used in ad collateral, or visuals that we
see around us, then you know
00:07:19
that bias with an individual
human being affects a few 100, few 1000 images. But if you have
an AI system that's doing this
00:07:26
at scale for millions of images—
and in the end, we're probably going to have like three, four
image generation AI systems that
00:07:33
are powering a lot of the images
that get created and used by all of us. So systematically, that
will mean that all of us are
00:07:41
seeing, you know, biased images
everywhere, which ultimately affects our social constructs.
You know, how do we imagine a
00:07:48
CEO? Is that always a male? How
do we imagine, you know, you brought up a childcare worker.
Is it always a female? And so
00:07:56
on. And so, I think those
stereotypes will then propagate in society. And the challenge
over here is that we can't
00:08:04
always rely on manual human
detection of these biases, because millions of images are
being generated on a daily
00:08:12
basis, and that scale and speed,
manual approaches just will not work. And if I generate two
images from the system, and I
00:08:20
use it, and those two images
happen to show two males, now that doesn't mean there is bias
in there. But if tens of 1000s of
00:08:30
people are generating images,
and each of their one or two images are biased, and
collectively, we're skewing how
00:08:36
we represent teachers or how we
represent computer programmers and so on. And many times there
may not be a human in the loop.
00:08:43
And so if we are automatically
using AI systems to create content and use them in some
downstream activity or task,
00:08:52
then we need some automated way
to correct— detect and correct those biases. Otherwise, you
know, they will go undetected
00:08:59
and be used in society at large. Yeah. So it's not— it's not the
individual user. It's not, you
00:09:05
know, the student who's creating
an image to go with his term paper. It really is— we have to
think bigger. We have to think
00:09:10
about scale and really consider
that. Pushkar, I know that bias detection in computer models is
something that you study
00:09:17
specifically. Can you— from your point of view,
why is this important? I think because AI models in
general sort of are a
00:09:29
reflection of the society. So—
and one thing we want to ensure is— is, like, make sure that the
harmful aspects that are already
00:09:40
prevalent in the society, first,
are not, like, transferred into the AI models. And more so, they
are not at least exacerbated by
00:09:51
these AI models. So often times
we see that a lot of these biases in, say in certain
professions, are exacerbated
00:10:00
or worsened by how AI models
depict them. So I think that's a very important problem that
needs to be solved. If we were
00:10:07
to— like, I'm all in for
developing and deploying AI models at large scales. But we
should sort of understand that
00:10:14
they should be done in a
efficient manner. Yeah. Well, so this model is called TIBET,
which stands for— it's an
00:10:21
acronym that stands for Text to
Image Bias Evaluation Tool. Can you just briefly explain to us,
how does it work?
00:10:30
Yeah. So generally, the
philosophy behind what we're doing is that if humans cannot
handle the scale and speed of
00:10:42
image generation, and if AI's
strength is scale and speed, can we leverage AI itself to detect
the problems in AI models? And
00:10:51
if we do it carefully, then it
doesn't become a loop of, AI creates models that are biased,
and AI detection tools cannot
00:10:59
detect those biases, because
they have the same problem. And so you have to do this
carefully. And the way we do
00:11:04
this just intuitively at a high
level— the first thing is, if we think about a term like, say,
"computer programmer" or "old man
00:11:14
at a church," the two examples
Pushkar showed, we first want to think about, what are the ways
in which that image can be
00:11:21
biased? And he asked us those
questions, and we did a joint brainstorming of the ways in
which it could be biased. So the
00:11:27
first step in our tool is to ask
an AI system, a large language model, a question about, what
are the ways in which images
00:11:36
that are generated in response
to this prompt by a user can be biased? And it would help
identify, it's certainly a
00:11:43
gender, but it could identify
some of the other biases that Pushkar raised, like body type or
even the visual environment for
00:11:51
the image and so on. And then
for every potential— we call it an axis. But every potential
dimension or axis along which
00:11:58
an image can be biased, we
generate what we call as counterfactual prompts. Meaning,
what are the other kinds of
00:12:05
prompts that could have been
entered by the users along the dimension of bias. So if we are
worried about gender, a
00:12:13
counterfactual prompt for
"computer programmer" would be "male computer programmer" and
"female computer programmer", or
00:12:19
"transgender computer programmer",
things like that. And then we generate images for these
counterfactual prompts. We
00:12:26
compare those images with the
image for our main prompt, again using AI to look at the concepts
that are in the images and
00:12:35
compare the concepts. And we
might find in the example that Pushkar showed that the images for
computer programmer have very
00:12:43
similar concepts as the images
for male computer programmer, but the concepts are very
different than the images for,
00:12:50
say, female computer programmer. In
terms of concepts, meaning the dresses people are wearing, the
jewelry they have on, whether
00:12:57
they're wearing glasses or not.
Things like that. We might also find that the concepts for
computer programmer look
00:13:07
very similar to young computer
programmer, but look very different from older computer
programmer. Once we start to see
00:13:14
that, we start to see, look. The
results for computer programmer are nearly the same as male
computer programmer, nearly the
00:13:23
same as young computer
programmer, but very dissimilar to female computer programmer or
older computer programmer. And
00:13:31
based on that, our system is
able to generate scores for how biased they are and if it starts
to look like it's very similar
00:13:38
to some counterfactual prompts,
but very dissimilar to others, our system knows that there's a
bias here and is able to flag
00:13:45
that and give a high score for
that bias. That's really incredible. Now, you had
mentioned just briefly Google
00:13:51
Gemini. And we're going to talk
about that for just a little bit. This was a case where
Google had put out this text to
00:13:56
image generator in the earlier
part of the year, and then they sort of found themselves embroiled
in the culture wars,
00:14:02
because the generator was
actually over-correcting for bias. And one of the famous
examples was, if you prompt it
00:14:08
for an image of the Founding
Fathers for the United States, it would include it would depict
a Black man. If you asked for
00:14:16
soldiers who fought in the
Vietnam War, it would depict an Asian woman. So this is the case
where it was actually over
00:14:22
correcting. That brings me to
asking you about the limitations of your model. How do you
prevent the same problem from
00:14:28
happening in your model? Pushkar, you should take that.
- Yeah, I think that problem happens
because they come up with this
00:14:39
one-size-fits-all approach. So
any prompt, or most of the prompts, you just correct for
culture, as well as, say, gender,
00:14:48
and that's about it. Whereas
what our approach does is it's a very dynamic approach which is
sort of dependent on what the
00:14:56
input is. So a typical
example I give is capybaras getting married in Italy, versus
images of a philosopher on Mars,
00:15:08
might have completely different
sets of biases. And capybaras getting married in Italy might
not have any biases related to
00:15:15
gender, but the way Italy is
portrayed might be biased in that direction. So I think the
way our approach differs is
00:15:24
that it considers the context in
which the prompt is said. And based on that context, it
generates a set of biases, and
00:15:33
then does this investigation on
those sets and says that, okay, I'm biased along these
directions, or not biased along
00:15:39
these directions, but I know
what directions to check my biases to start, it's not like
I'm going to check my biases on
00:15:47
age, race and gender for every
every prompt or every input. Yeah. Right. Well, I find it fascinating that
what you're actually doing is
00:15:55
using human intelligence to make
the artificial intelligence better, right? So I wanted to
ask you, though— but we are— this
00:16:02
tool was specifically designed
for text-to-image generation. Can it be used for other things?
00:16:06
Do you have plans to expand its
abilities? Yeah, we do. Because if you think
about the underlying approach,
00:16:14
it really is an approach that
can be generalized to, for example, large language models
that are generating text.
00:16:20
Because we are thinking about
first asking an AI system to reason through, what are the
ways in which it can be biased?
00:16:26
And then, given the ways in
which it can be biased, constructing counterfactual
prompts. Other ways in which to
00:16:33
construct those prompts to look
at, for example, what output would be generated if we worded
the prompt slightly differently
00:16:43
along gender, around age, or
along body type, and so on. And then looking at the output and
comparing it and saying, "Hey,
00:16:52
the output changed a lot when I
changed the prompt in terms of gender, let's say." So that
approach is easily applicable to
00:17:00
other settings. And in fact, we
are currently working on applying the same kinds of ideas
in the context of the use of
00:17:08
large language models and
enterprises. And so that is ongoing. We don't have a paper
yet on it, but we are certainly
00:17:15
extending our study in that
direction. And I know that we're going to get
this question, so I'm going to
00:17:19
ask you now. Is this tool, the
TIBET tool, is it available? Can people find it, use it,
apply it? How do they do that?
00:17:28
Pushkar, you've been putting a lot
of effort in that direction, so please. I think it's currently in
progress. The paper is out. We
00:17:35
are sort of finishing our tool.
But it's going to be soon available, probably in a month
or two. Yeah. So we're just
00:17:42
trying to wrap up the nitty
gritty details of it and sort of make it a tool that people can
use. Yeah.
00:17:46
Sounds like it's going to be really
important when it hits— hits the public. Gentlemen, is there
anything else that we need to
00:17:52
know about this tool, or about
bias and image generation that you'd like to add before we
close out?
00:17:57
Well, to what Pushkar just said, I
wanted to add that while the— the tool is not fully available for
anyone to use, we do provide a
00:18:08
lot of the code that anyone can
use to also apply it in their settings. We provide a lot of
the data explanations and so on.
00:18:17
And so we do have that on a— on a
website. The URL for that is Tibet-ai.github.io.
So if people go to
00:18:29
Tibet-ai.github.io,
you'll find our paper there. You'll find explanations there,
but you'll also find raw code
00:18:38
that anyone can use in in their
settings. In terms of a user- friendly tool with the UI, that's
in the works,
00:18:45
and hopefully down the road. These are— these are things
that we all have to start
00:18:49
thinking about as we're speeding
rapidly in this evolution of AI, that, how we fix the problems,
the speed bumps along the way.
00:18:56
So I'm glad the two of you are
working on this. Thank you so much for
joining me today
00:19:00
and for sharing some of these
details. I really appreciate it. Thanks for having us.
00:19:04
If you enjoyed
this conversation, we invite you to check out
all our content at knowledge.Wharton.upenn.edu.
00:19:11
I'm Angie Basiouny. Thank you for joining us. For more insight from Knowledge
at Wharton, please visit
00:19:17
knowledge.Wharton.upenn.edu.