00:00:01
This podcast is brought to you by
Knowledge at Wharton.
00:00:13
Angie Basiouny: Welcome to
Knowledge at Wharton.
00:00:14
I'm Angie Basiouny. If I ask you
00:00:16
If I ask you to think of a doctor,
00:00:18
your brain might
conjure up the image of an
00:00:19
older white male. If I ask you
to think of a fitness
00:00:22
enthusiast, it might be a
younger, slender person. These
00:00:26
are biases that are baked into
our brains, and unfortunately,
00:00:29
they also get baked into AI. I'm
here today with Wharton
00:00:33
Professor Kartik Hosanagar and his
colleague, Pushkar Shukla, who's
00:00:37
a PhD candidate at the Toyota
Technological Institute at Chicago.
00:00:41
They're part of a team
that has created this new tool
00:00:44
that can correct, detect, biases
in text-to-image generators.
00:00:49
Those are the machine learning
models that can produce a visual
00:00:52
image for you with just a few
prompts. Gentlemen, I'm so glad
00:00:56
to have you here today. Kartik,
it's our first time talking on
00:00:58
this podcast. Welcome.
00:01:00
Thanks for having me, Angie.
00:01:03
A great pleasure to
meet you in person.
00:01:04
We've interacted many times
before, but lovely to see you.
00:01:07
Same here. And Pushkar,
00:01:08
welcome to the
Wharton family. Thank you for
00:01:10
making some time and joining us
today.
00:01:12
Thank you so much, Angie. It's a
pleasure meeting you and
00:01:15
interviewing for this podcast.
- Absolutely.
00:01:18
So when I first read this paper,
I just thought, what a difficult
00:01:21
problem to solve for in real
life. I can't even imagine how
00:01:25
difficult it is to solve for it
on a technology platform. And
00:01:29
that's why I'm really excited to
share these details about this
00:01:32
tool today. Let's just start
with an overview, if you will,
00:01:35
Professor. Give us an overview
of this research, what you were
00:01:38
trying to solve for, what the
problems are.
00:01:41
Yes. Our research was motivated
by the widespread use of
00:01:46
generative AI systems today to
generate all kinds of content.
00:01:49
Certainly text through systems
like ChatGPT, but also a lot of
00:01:53
image generation through tools
like Stable Diffusion, Mid-
00:01:57
journey, Meta's AI, and even ChatGPT
does image generation. And so
00:02:02
on. And while they're being used
at an unprecedented volume or
00:02:07
scale today, these systems do
have biases. It's kind of what
00:02:11
you were mentioning earlier.
They're trained on data. Those
00:02:14
data are generated by humans who
have their own biases, and so
00:02:18
they pick up their biases, and
in many ways institutionalize
00:02:21
those biases, and kind of do
that going forward. So if you
00:02:24
look at image generation as an
example, in these systems, if you
00:02:28
type in, say, "computer
programmer", you're probably
00:02:31
likely to see most of the images
being of males. And so these
00:02:36
systems are prone to biases. But
generally the biases get
00:02:42
detected after the fact, and
usually by a human user using
00:02:47
them and kind of saying, "Hey,
I'm noticing a pattern." Or a
00:02:50
journalist like you might notice
a pattern and write about it. So
00:02:53
we've seen, for example,
Google's Gemini having bias
00:02:57
issues, and people wrote about
it. And similar things have been
00:03:00
written about pretty much every
image generation model. So we
00:03:04
set out to come up with a system
that can automatically detect
00:03:09
biases in these systems. And
that automatic detection is
00:03:13
extremely important because of
the scale at which they're being
00:03:15
used, but it's also a very hard
problem. And so we wanted to be
00:03:19
able to detect them, and not
just detect simple or the common
00:03:23
biases, like, say, gender and
age, but all kinds of potential
00:03:27
biases, and to be able to
00:03:29
provide explanations to
human decision makers.
00:03:31
So that's what we set out to do
and solve with this project.
00:03:35
I think that it would be helpful
if we could show some of the
00:03:38
biased images that we're talking
about, and that's where I'm
00:03:41
hoping, Pushkar, that you can help
us with that. Share your screen
00:03:43
and show us some of these images
00:03:45
that you're studying
in this project.
00:03:47
The first set of images talks about
a computer programmer. So if you
00:03:51
ask an AI generative model— and in
this case, we used Stable
00:03:54
Diffusion— to generate images of
a computer programmer, this is
00:03:57
how these images look like.
Similarly, in the second set of
00:04:01
images, I ask the generative AI
model to generate images of old—
00:04:05
of an old man at a church. Now,
before I go on and talk about
00:04:09
the possible directions in which
these images can be biased, I
00:04:13
want to take a second and ask
you two, what do you think are the
00:04:16
possible directions in which
these images can be biased?
00:04:20
Well, most definitely gender,
right? For the first one, a
00:04:24
computer programmer, the images
are always going to be male. I
00:04:28
noticed in your paper that you
had a collection of images when
00:04:31
you prompt for, for example,
childcare worker, and all the
00:04:34
images were female. Another—
another stereotype. I would also
00:04:38
say maybe the— the image about an
old man in a church. The church
00:04:41
always has a similar look. So
the image of the church in this
00:04:44
case is the same. It appears to
be maybe a Catholic Church. The
00:04:48
stained glass windows, the
arched— you know, the arched
00:04:50
windows. So again, it's sort of
an image that we have for maybe
00:04:54
even from our school days of
studying the Renaissance.
00:04:59
Yeah. And I was going to add
that— well, at least the
00:05:02
computer programmer images, in
terms of race, seems to be
00:05:06
diverse, but in terms of the old
man at the church, from what I
00:05:11
can see, they're all white male.
I can— perhaps also the computer
00:05:17
programmer seems to be based on
these four images, all young.
00:05:22
And so there's another
stereotype I'm seeing there, but
00:05:25
that's what I'm seeing visually,
at least.
00:05:30
Cool. So I think all the axes of
biases that you mentioned are
00:05:34
correct, but there is more to
it. So when we talk about— you
00:05:39
talked about gender, ethnicity,
age being important axes of
00:05:43
biases, but let's look at a few
other axes of biases. All
00:05:46
computer programmers are lean.
So there is clearly a body type
00:05:52
issue there. And that's—
similarly, the physical
00:05:55
appearance of old men at a
church are of a certain type.
00:05:59
They look grim, slightly
depressed or sad, and also
00:06:05
things such as, there might be
ableism bias. Or in our
00:06:10
research, we found that
oftentimes old men were
00:06:14
portrayed to have a certain kind
of disability. So the point that
00:06:17
I'm trying to make is when we
think about biases in AI, we
00:06:21
generally think about the common
dimensions of biases such as
00:06:25
gender, ethnicity, culture,
maybe age. But there is more to
00:06:30
it. There are other equally
important dimensions of biases,
00:06:33
such as socioeconomic bias,
disability, body type and health
00:06:39
bias of some sort. So— and that
is something that needs to be
00:06:42
looked upon, and that is what
makes it
00:06:45
a tough problem to solve.
00:06:47
I'm going to ask both of you to
answer this question. Why? Why
00:06:50
do we need to look at it? Why is
it important to correct this
00:06:53
kind of bias?
00:06:55
Well, quite simply, these biases
will continue to be there in
00:07:02
society, but also at a scale
that we have probably not seen
00:07:06
before. If you have a human
being generating images in some
00:07:10
marketing agency, and that's
going to be used in ad
00:07:14
collateral, or visuals that we
see around us, then you know
00:07:19
that bias with an individual
human being affects a few 100,
00:07:22
few 1000 images. But if you have
an AI system that's doing this
00:07:26
at scale for millions of images—
and in the end, we're probably
00:07:30
going to have like three, four
image generation AI systems that
00:07:33
are powering a lot of the images
that get created and used by all
00:07:37
of us. So systematically, that
will mean that all of us are
00:07:41
seeing, you know, biased images
everywhere, which ultimately
00:07:45
affects our social constructs.
You know, how do we imagine a
00:07:48
CEO? Is that always a male? How
do we imagine, you know, you
00:07:53
brought up a childcare worker.
Is it always a female? And so
00:07:56
on. And so, I think those
stereotypes will then propagate
00:08:00
in society. And the challenge
over here is that we can't
00:08:04
always rely on manual human
detection of these biases,
00:08:08
because millions of images are
being generated on a daily
00:08:12
basis, and that scale and speed,
manual approaches just will not
00:08:15
work. And if I generate two
images from the system, and I
00:08:20
use it, and those two images
happen to show two males, now
00:08:25
that doesn't mean there is bias
in there. But if tens of 1000s of
00:08:30
people are generating images,
and each of their one or two
00:08:33
images are biased, and
collectively, we're skewing how
00:08:36
we represent teachers or how we
represent computer programmers
00:08:39
and so on. And many times there
may not be a human in the loop.
00:08:43
And so if we are automatically
using AI systems to create
00:08:48
content and use them in some
downstream activity or task,
00:08:52
then we need some automated way
to correct— detect and correct
00:08:56
those biases. Otherwise, you
know, they will go undetected
00:08:59
and be used in society at large.
00:09:02
Yeah. So it's not— it's not the
individual user. It's not, you
00:09:05
know, the student who's creating
an image to go with his term
00:09:07
paper. It really is— we have to
think bigger. We have to think
00:09:10
about scale and really consider
that. Pushkar, I know that bias
00:09:14
detection in computer models is
something that you study
00:09:17
specifically. Can you—
00:09:19
from your point of view,
why is this important?
00:09:23
I think because AI models in
general sort of are a
00:09:29
reflection of the society. So—
and one thing we want to ensure
00:09:35
is— is, like, make sure that the
harmful aspects that are already
00:09:40
prevalent in the society, first,
are not, like, transferred into
00:09:46
the AI models. And more so, they
are not at least exacerbated by
00:09:51
these AI models. So often times
we see that a lot of these
00:09:55
biases in, say in certain
professions, are exacerbated
00:10:00
or worsened by how AI models
depict them. So I think that's a
00:10:03
very important problem that
needs to be solved. If we were
00:10:07
to— like, I'm all in for
developing and deploying AI
00:10:10
models at large scales. But we
should sort of understand that
00:10:14
they should be done in a
efficient manner. Yeah.
00:10:18
Well, so this model is called TIBET,
which stands for— it's an
00:10:21
acronym that stands for Text to
Image Bias Evaluation Tool. Can
00:10:25
you just briefly explain to us,
how does it work?
00:10:30
Yeah. So generally, the
philosophy behind what we're
00:10:35
doing is that if humans cannot
handle the scale and speed of
00:10:42
image generation, and if AI's
strength is scale and speed, can
00:10:46
we leverage AI itself to detect
the problems in AI models? And
00:10:51
if we do it carefully, then it
doesn't become a loop of, AI
00:10:56
creates models that are biased,
and AI detection tools cannot
00:10:59
detect those biases, because
they have the same problem. And
00:11:02
so you have to do this
carefully. And the way we do
00:11:04
this just intuitively at a high
level— the first thing is, if we
00:11:10
think about a term like, say,
"computer programmer" or "old man
00:11:14
at a church," the two examples
Pushkar showed, we first want to
00:11:18
think about, what are the ways
in which that image can be
00:11:21
biased? And he asked us those
questions, and we did a joint
00:11:24
brainstorming of the ways in
which it could be biased. So the
00:11:27
first step in our tool is to ask
an AI system, a large language
00:11:30
model, a question about, what
are the ways in which images
00:11:36
that are generated in response
to this prompt by a user can be
00:11:39
biased? And it would help
identify, it's certainly a
00:11:43
gender, but it could identify
some of the other biases that
00:11:45
Pushkar raised, like body type or
even the visual environment for
00:11:51
the image and so on. And then
for every potential— we call it
00:11:55
an axis. But every potential
dimension or axis along which
00:11:58
an image can be biased, we
generate what we call as
00:12:01
counterfactual prompts. Meaning,
what are the other kinds of
00:12:05
prompts that could have been
entered by the users along the
00:12:09
dimension of bias. So if we are
worried about gender, a
00:12:13
counterfactual prompt for
"computer programmer" would be
00:12:15
"male computer programmer" and
"female computer programmer", or
00:12:19
"transgender computer programmer",
things like that. And then we
00:12:22
generate images for these
counterfactual prompts. We
00:12:26
compare those images with the
image for our main prompt, again
00:12:31
using AI to look at the concepts
that are in the images and
00:12:35
compare the concepts. And we
might find in the example that
00:12:38
Pushkar showed that the images for
computer programmer have very
00:12:43
similar concepts as the images
for male computer programmer,
00:12:47
but the concepts are very
different than the images for,
00:12:50
say, female computer programmer. In
terms of concepts, meaning the
00:12:54
dresses people are wearing, the
jewelry they have on, whether
00:12:57
they're wearing glasses or not.
Things like that. We might also
00:13:01
find that the concepts for
computer programmer look
00:13:07
very similar to young computer
programmer, but look very
00:13:10
different from older computer
programmer. Once we start to see
00:13:14
that, we start to see, look. The
results for computer programmer
00:13:18
are nearly the same as male
computer programmer, nearly the
00:13:23
same as young computer
programmer, but very dissimilar
00:13:26
to female computer programmer or
older computer programmer. And
00:13:31
based on that, our system is
able to generate scores for how
00:13:34
biased they are and if it starts
to look like it's very similar
00:13:38
to some counterfactual prompts,
but very dissimilar to others,
00:13:42
our system knows that there's a
bias here and is able to flag
00:13:45
that and give a high score for
that bias. That's
00:13:48
really incredible. Now, you had
mentioned just briefly Google
00:13:51
Gemini. And we're going to talk
about that for just a little
00:13:53
bit. This was a case where
Google had put out this text to
00:13:56
image generator in the earlier
part of the year, and then they
00:13:59
sort of found themselves embroiled
in the culture wars,
00:14:02
because the generator was
actually over-correcting for
00:14:05
bias. And one of the famous
examples was, if you prompt it
00:14:08
for an image of the Founding
Fathers for the United States,
00:14:11
it would include it would depict
a Black man. If you asked for
00:14:16
soldiers who fought in the
Vietnam War, it would depict an
00:14:19
Asian woman. So this is the case
where it was actually over
00:14:22
correcting. That brings me to
asking you about the limitations
00:14:25
of your model. How do you
prevent the same problem from
00:14:28
happening in your model?
00:14:32
Pushkar, you should take that.
- Yeah,
00:14:34
I think that problem happens
because they come up with this
00:14:39
one-size-fits-all approach. So
any prompt, or most of the
00:14:44
prompts, you just correct for
culture, as well as, say, gender,
00:14:48
and that's about it. Whereas
what our approach does is it's a
00:14:52
very dynamic approach which is
sort of dependent on what the
00:14:56
input is. So a typical
example I give is capybaras
00:15:02
getting married in Italy, versus
images of a philosopher on Mars,
00:15:08
might have completely different
sets of biases. And capybaras
00:15:12
getting married in Italy might
not have any biases related to
00:15:15
gender, but the way Italy is
portrayed might be biased in
00:15:19
that direction. So I think the
way our approach differs is
00:15:24
that it considers the context in
which the prompt is said. And
00:15:29
based on that context, it
generates a set of biases, and
00:15:33
then does this investigation on
those sets and says that, okay,
00:15:37
I'm biased along these
directions, or not biased along
00:15:39
these directions, but I know
what directions to check my
00:15:42
biases to start, it's not like
I'm going to check my biases on
00:15:47
age, race and gender for every
every prompt or every input. Yeah.
00:15:52
Right.
00:15:53
Well, I find it fascinating that
what you're actually doing is
00:15:55
using human intelligence to make
the artificial intelligence
00:15:58
better, right? So I wanted to
ask you, though— but we are— this
00:16:02
tool was specifically designed
for text-to-image generation.
00:16:05
Can it be used for other things?
00:16:06
Do you have plans to expand its
abilities?
00:16:09
Yeah, we do. Because if you think
about the underlying approach,
00:16:14
it really is an approach that
can be generalized to, for
00:16:17
example, large language models
that are generating text.
00:16:20
Because we are thinking about
first asking an AI system to
00:16:23
reason through, what are the
ways in which it can be biased?
00:16:26
And then, given the ways in
which it can be biased,
00:16:29
constructing counterfactual
prompts. Other ways in which to
00:16:33
construct those prompts to look
at, for example, what output
00:16:39
would be generated if we worded
the prompt slightly differently
00:16:43
along gender, around age, or
along body type, and so on. And
00:16:48
then looking at the output and
comparing it and saying, "Hey,
00:16:52
the output changed a lot when I
changed the prompt in terms of
00:16:55
gender, let's say." So that
approach is easily applicable to
00:17:00
other settings. And in fact, we
are currently working on
00:17:04
applying the same kinds of ideas
in the context of the use of
00:17:08
large language models and
enterprises. And so that is
00:17:12
ongoing. We don't have a paper
yet on it, but we are certainly
00:17:15
extending our study in that
direction.
00:17:17
And I know that we're going to get
this question, so I'm going to
00:17:19
ask you now. Is this tool, the
TIBET tool, is it available?
00:17:23
Can people find it, use it,
apply it? How do they do that?
00:17:28
Pushkar, you've been putting a lot
of effort in that direction, so please.
00:17:32
I think it's currently in
progress. The paper is out. We
00:17:35
are sort of finishing our tool.
But it's going to be soon
00:17:39
available, probably in a month
or two. Yeah. So we're just
00:17:42
trying to wrap up the nitty
gritty details of it and sort of
00:17:44
make it a tool that people can
use. Yeah.
00:17:46
Sounds like it's going to be really
important when it hits— hits the
00:17:49
public. Gentlemen, is there
anything else that we need to
00:17:52
know about this tool, or about
bias and image generation that
00:17:55
you'd like to add before we
close out?
00:17:57
Well, to what Pushkar just said, I
wanted to add that while the— the
00:18:02
tool is not fully available for
anyone to use, we do provide a
00:18:08
lot of the code that anyone can
use to also apply it in their
00:18:13
settings. We provide a lot of
the data explanations and so on.
00:18:17
And so we do have that on a— on a
website. The URL for that is
00:18:22
Tibet-ai.github.io.
So if people go to
00:18:29
Tibet-ai.github.io,
you'll find our paper there.
00:18:33
You'll find explanations there,
but you'll also find raw code
00:18:38
that anyone can use in in their
settings. In terms of a user-
00:18:42
friendly tool with the UI, that's
in the works,
00:18:45
and hopefully down the road.
00:18:47
These are— these are things
that we all have to start
00:18:49
thinking about as we're speeding
rapidly in this evolution of AI,
00:18:53
that, how we fix the problems,
the speed bumps along the way.
00:18:56
So I'm glad the two of you are
working on this.
00:18:58
Thank you so much for
joining me today
00:19:00
and for sharing some of these
details. I really appreciate it.
00:19:03
Thanks for having us.
00:19:04
If you enjoyed
this conversation,
00:19:06
we invite you to check out
all our
00:19:07
content at knowledge.Wharton.upenn.edu.
00:19:11
I'm Angie Basiouny.
00:19:12
Thank you for joining us.
00:19:14
For more insight from Knowledge
at Wharton, please visit
00:19:17
knowledge.Wharton.upenn.edu.
