00:00:00
Daniel Rock: That's a really fascinating
point. I mean, you think about
00:00:02
the value of an OpenAI or
Anthropic, even a Microsoft,
00:00:05
Google— you know, whoever's
building it, Llama— like, a lot
00:00:08
of the value of those companies
is in the complementary
00:00:12
investments that their customers
are making. Or, like, the
00:00:14
ecosystem at large is making.
That's really interesting,
00:00:17
right? Like the— they get more
valuable as their customers and
00:00:21
as their consumers learn how to
integrate that toolkit. So I
00:00:25
think that's something that'll
take a little while, but they're
00:00:28
well aware of it too.
00:00:29
You know, they're trying to
make it easier to use.
00:00:31
- Welcome to <i>The Ripple Effect</i>,
00:00:33
the podcast
that takes you on a journey
00:00:35
through the minds of Wharton
faculty. I'm your host, Dan
00:00:38
Loney, and in each episode,
we'll be diving deep into the
00:00:41
inspiration behind the
groundbreaking research that
00:00:44
Wharton professors have
conducted and exploring how
00:00:47
their findings resonate with the
world today.
00:00:51
Dan Loney: Well, when we think
00:00:52
about innovation these days,
there's a good chance that
00:00:54
artificial intelligence is going
to come into the conversation.
00:00:58
Even though it's been around for
some time, it only now feels
00:01:02
like the majority of the public
at large are seeing the impact
00:01:06
of artificial intelligence in
our lives. Daniel Rock is an
00:01:10
Assistant Professor of
Operations, Information and
00:01:12
Decisions here at the Wharton
School. He and colleagues have
00:01:14
looked at how AI can impact
something like productivity.
00:01:18
Dan, great to have you here
today. Thanks for your time.
00:01:20
Great to be here. Thanks for
having me.
00:01:22
So when I bring up AI,
00:01:23
doesn't it seem like
productivity kind of is a— a
00:01:27
natural— a natural first thing
for people to think about?
00:01:30
Oh, absolutely.
00:01:32
And I think when we talk about
productivity, it's important to
00:01:35
define terms here. For
economists, productivity can be
00:01:39
a few different things. It can
be how much output per worker
00:01:41
you have, how much revenue per
unit of input, but generally,
00:01:45
all of it points to one big
idea, which is, what are the
00:01:48
number of outputs we get per
unit of input?
00:01:51
So it's not, you know, how
00:01:52
do we cut jobs, necessarily, or
reduce the resource use. It's
00:01:56
also, how do we create more? And
I think with these tools
00:01:59
empowering people to do, you
know, greater and more
00:02:02
interesting things, productivity
in the long run has to kind of
00:02:05
be positively impacted by what
we can do with them.
00:02:08
- Would you say that, though— that
there's a
00:02:09
paradox when you think of this?
00:02:11
Yeah, sure. I think here's—
here's the core thing with a
00:02:15
sufficiently transformative
technology, what economists
00:02:17
would call a general purpose
technology. That is, it's
00:02:20
pervasive, it improves over
time, and then it kind of
00:02:23
necessitates and spawns
complementary innovation. That
00:02:26
is the other stuff you need to
build to get this stuff to go.
00:02:30
So, yeah, there's a lag. It
takes a long time to build up
00:02:33
those additional assets, to
reconfigure your organization,
00:02:36
to train people to use stuff.
Over time, that's going to pay
00:02:39
off in a big way, and we're
seeing people make huge
00:02:41
investments in that. But it's
not going to be, right off the
00:02:45
bat, super powerful. Actually,
it's funny with AI, there are
00:02:48
some applications that are right
off the bat super powerful, but
00:02:51
the long run implications are
going to take a while to play
00:02:54
out, I think.
- You have four areas
00:02:56
of potential impact that you've
come up with in the work that
00:03:00
you've done, the first being
false hopes. Explain that a
00:03:03
little bit.
00:03:04
Oh yeah. So this is the
explanation for the paradox. So, why
00:03:06
it takes a while. So I already
preempted what I think is going
00:03:09
on. But yes, there is the— there
is the chance, right— this is
00:03:12
sort of a Bob Gordon view. I
don't want to put too many words
00:03:16
in his mouth, but, you know,
00:03:17
AI just isn't
that big a deal. Or you could
00:03:20
broaden this to say any
technology just isn't that big a
00:03:23
deal. And we see lots of promise
and hype, but it's just never
00:03:27
going to materialize. That's a
consistent way to view the world
00:03:31
in the early stages, if you
don't know what's going to
00:03:33
happen. But then you do have to,
you know, change tack if you see
00:03:36
the benefits start to show up. I
think with AI, we're starting to
00:03:38
see that a bit. So that's number
one.
00:03:41
The second one is mismeasurement.
00:03:44
So this is kind of—
there's some folks in Silicon
00:03:46
Valley who say this, and there's
some evidence that this might be
00:03:49
going on too. The idea here is
that the gains are real. They're
00:03:53
happening, but we're not
capturing them properly in the
00:03:55
economic statistics. And I think
this— the folks at the BLS and
00:04:01
the BEA do a really great job of
trying to measure the economy.
00:04:06
You know, where it might be
tougher to measure things is
00:04:08
like, you know, Google's free. I
asked my MBAs, would you rather
00:04:13
have, you know, search or indoor
plumbing? And you know, after
00:04:18
trying to wriggle out of that
conundrum, many of them will
00:04:21
still pick search over indoor
plumbing. I'm kind of with them
00:04:24
on that one. - That's probably a
good idea. Yeah.
00:04:26
It gets cold in Philly, but not too cold,
- Exactly.
00:04:29
So that's the second
00:04:30
one. We could be mismeasuring
things. And yes, there are some
00:04:32
cases where that may be the
case. But in general, you have
00:04:35
to make an argument for why it's
different now. What changed to
00:04:39
make us worse at measurement,
given what the economy is
00:04:41
producing? And I think that's a
tougher case. My co-author, Chad
00:04:44
Syverson at University of
Chicago, has kind of disposed
00:04:48
that argument. At least, you
know, up until 2017 or so.
00:04:52
So that's the second one. The third
one is sort of a rent
00:04:54
dissipation argument. What does
that mean? It's, the gains are
00:04:58
real, but they're accruing to a
really small proportion of
00:05:02
people in the economy. They're
taking all of the gains, and
00:05:06
nobody else is seeing anything
there. I think you could make an
00:05:09
argument that a lot of that is
still happening, but it would
00:05:12
have to be really enormous to
take away the gains from the
00:05:15
technology, given the
expectations. And then the last
00:05:18
one, which we just discussed,
restructuring and implementation
00:05:22
lags. The stuff can take a
while. We see a lot of promise,
00:05:25
but let's not confuse a clear
view for a short walk. It's
00:05:28
going to take a long time to
implement this stuff.
00:05:30
So is the expectation, then, with
where we are kind of currently,
00:05:34
that we're still going to see
innovation coming from other
00:05:37
areas to kind of complement what
is, I think a lot of people
00:05:41
believe, the core of— of what AI
is right now?
00:05:44
Yeah, absolutely. I think that's
that's already happening in a
00:05:48
lot of areas. One of the really
cool things about AI is— I don't
00:05:52
know if you you code, or if you
if you use coding assistance,
00:05:55
but the fact that these tools—
the software you need to augment
00:06:01
them, to augment the AI tools,
can be partially written with
00:06:05
AI help. So we can— we really
get in this nice, like, flywheel,
00:06:09
where you use AI to improve the
sorts of tooling you need to
00:06:13
make AI more effective.
00:06:15
That's— when I
talk with companies, I'm like,
00:06:17
"Are you guys doing this? Because
you should be."
00:06:19
It's very helpful.
00:06:20
There's an element that you talk about
in this paper, about the added
00:06:25
costs being a type of capital
that can be just part of the
00:06:31
build out, I guess, correct?
00:06:32
Yeah, absolutely. When you have
these adjustment costs or fixed
00:06:36
costs of investment, these are
things like the training or
00:06:39
intangible capital, or even the
culture around how you build an
00:06:42
organization that— that works
with machine learning or AI
00:06:46
software. Like, this is a
different type of software. It
00:06:50
creates output that's non-
deterministic, or maybe a little
00:06:54
bit fuzzier. It's not perfect
every time, it's not cookie
00:06:57
cutter. That's a mindset shift,
too. You can't expect the same
00:07:00
results as you could with— with
ordinary kind of rules-based
00:07:03
software. So you have to pay
that upfront cost, or maybe even
00:07:07
ongoing cost, to keep people in
the loop, to structure your
00:07:11
organization processes properly.
And then what you— what you get
00:07:14
out of that is, you know,
competitive advantage,
00:07:18
basically. You can do things
other companies can't, if you
00:07:20
crack that.
- Sure. - Yeah.
00:07:22
And obviously, part of
that also probably fills into,
00:07:25
with the companies that are in
the development of AI, the value
00:07:30
that those companies have— you
know, that that is a significant
00:07:33
beneficial capital component
that they have to their
00:07:36
companies, which, you know,
00:07:38
obviously is larger than other
companies.
00:07:40
Sure. That's a really fascinating
point. I mean, you think about
00:07:42
the value of an OpenAI or
Anthropic, even a Microsoft,
00:07:46
Google— you know, whoever's
building it, Llama— like, a lot
00:07:48
of the value of those companies
is in the complementary
00:07:52
investments that their customers
are making. Or, like, the
00:07:54
ecosystem at large is making.
That's really interesting,
00:07:57
right? Like the— they get more
valuable as their customers and
00:08:01
as their consumers learn how to
integrate that toolkit. So I
00:08:05
think that's— that's something
that will take a little while,
00:08:08
but they're well aware of it
too. You know, they're trying to
00:08:10
make it easier to use. In some
sense, ChatGPT is more of a UX
00:08:13
innovation. The playground
existed. You could use stuff
00:08:17
before, but ChatGPT just, like
showed people, "Hey, you can—you
00:08:21
can really engage with these
models and do something cool." I
00:08:25
massively updated how important
I thought UX was after I saw the
00:08:29
success of that app.
00:08:30
So where we are right now then,
you mentioned about some of the
00:08:34
metrics that will come into play
here. It feels like we're still
00:08:39
at a point where the development
of some of those metrics is kind
00:08:42
of either— it either hasn't
happened or it's ongoing right
00:08:45
now, and so it may be hard to
truly gauge the value or the
00:08:51
component of productivity,
especially when we don't have
00:08:54
the dynamics fully tweaked to
what we need, right?
00:08:58
Yes, I agree with that. Though
what makes AI, this wave of
00:09:03
software, a little bit easier to
do a good job there is, we're
00:09:06
using the last wave of IT to
instrument it, right? So we have
00:09:10
software to track the software.
Before, I mean, the best you
00:09:13
could do, best you could hope
for, was surveys of some kind,
00:09:16
like, say, in the early '90s. Now
we can scale up those efforts.
00:09:19
You can track sort of what some
of my colleagues called "digital
00:09:23
exhaust", right? You connect to
APIs for companies, you can see
00:09:26
how they're changing what
they're doing. You know, there's
00:09:29
little pockets. This is like
this big iceberg, and we're
00:09:32
seeing just the tip of it. But
you can use those sort of points
00:09:37
where we can see the tip of the
iceberg and how it's changing as
00:09:41
a way to gauge what's going on.
And a concrete example is
00:09:43
something— you know, something
I've done in my own work. I've
00:09:46
tracked how many people with AI
skills are being hired company
00:09:49
by company, assuming that if
you're hiring people to do this,
00:09:52
you're probably building out all
the other complements to make
00:09:55
them effective. And, you know,
that's— we're trying to measure
00:09:58
the— kind of the size of the— the
investments on that front, which
00:10:02
Which for many companies, is probably
the way you need to do it, is
00:10:04
bring on the talent before you actually
do the level of implementation
00:10:09
to get to that point, so you
have people who understand it
00:10:11
going— going— going into the
process.
00:10:14
100%. You can't get away from— from
labor markets and those
00:10:18
complementary investments. If
you do AI well, then you
00:10:21
probably did data science well
before. If you did data science
00:10:24
well, you probably did the cloud
well before. There's a whole
00:10:26
stacking of these technologies. It
actually makes AI super
00:10:29
concentrated in— in only a few
firms right now. And it's a
00:10:32
little bit, you know— a little
bit of an explanation where that
00:10:35
might be coming from. But yeah,
you're bottlenecked in three
00:10:39
potential areas. It's either
talent or data or compute right
00:10:43
now. But you mix those three
things together in the right
00:10:47
proportions, and you start to
get, you know,
00:10:49
AI internal capabilities.
00:10:51
What do you think, then, doing this
research helped you to better
00:10:55
understand about where we are
going, and that connection
00:10:59
between AI and productivity?
- Yeah.
00:11:02
So I started to think, what are
the ingredients inside of a
00:11:06
company that would generate
productivity? Like, how— where
00:11:10
does it come from? And there's a
few models that colleagues in
00:11:14
other places have put together
that I can use as sort of
00:11:16
workhorse models. There's sort
of the task-based approach.
00:11:20
There are researchers like David
Auter and Daron Acemoglu at MIT
00:11:24
who have used this heavily. The
idea is like, let's break down a
00:11:27
job into bundle of tasks and
track all those tasks. Those
00:11:30
tasks or those bundles are
changing. I think the more
00:11:33
change you see at that level,
the more of an indication you
00:11:37
have that something is different
now. So that's one early kind of
00:11:40
check you can do to see what's
going on. So I'm working on some
00:11:44
of that, looking at job postings
with a few colleagues. And then
00:11:49
there's kind of a perspective
that Tim Bresnahan at Stanford
00:11:53
as well as Joshua Gans, Avi
Goldfarb and Ajay Agrawal at
00:11:57
University of Toronto have kind
of put forth. It's the sense that
00:12:02
nobody's ever lost their job to
task-based automation. It's not
00:12:05
happening task by task,
necessarily, or change isn't
00:12:08
happening task by task. It's
happening at the system level.
00:12:11
So when we change the direction
in what's possible with a model,
00:12:16
like, I can discover new drugs
with these tools, or I can— I
00:12:21
mean, I can't. But I can make
pretty lousy images or paintings
00:12:26
that I couldn't do before. I'll
stop there with my new
00:12:29
capabilities. But like, as— as you
give people these new
00:12:32
capabilities, you redesign the
system, and that new system's got
00:12:35
different demands for people.
Different demands for capital.
00:12:38
- Right. - So let's see if, like,
there are companies making big
00:12:41
changes saying "We're going to
reconfigure this module." It's
00:12:43
kind of hard to do that if
things are moving really
00:12:45
quickly. You don't have
certainty. You don't feel like
00:12:47
you're standing on solid ground
when you do that.
00:12:49
Does it feel, like, then— I'll finish up
on this. Does it feel like
00:12:53
that where we are with AI and
the workforce right now, that
00:12:58
obviously AI is going to play a
significant role, but the human
00:13:02
component will be there. And to
a degree, maybe even the human
00:13:05
component becomes kind of even
more of a learning experience as
00:13:09
we move forward, because of how
AI is kind of
00:13:12
guiding the ship a
little bit here.
00:13:13
Yeah, I get accused of being an
optimist on this point, so I
00:13:17
strongly agree with that. But I
do think there are, of course,
00:13:19
going to be pockets where things
go better or things go worse.
00:13:23
One thing I've grown fond of
saying recently is that we can't
00:13:26
get away from labor markets. So
the— I will say, you know,
00:13:29
perhaps it's a little bit of a
stretch. I don't want to get too
00:13:32
far out of my skis here. But the
augmentation versus automation
00:13:36
debate matters at some unit of
analysis. But at the individual,
00:13:41
like, you know, worker or
manager making a decision about
00:13:44
where to deploy the technology—
you can augment someone that can
00:13:47
do the job of 20 people, and if
the company doesn't want 20
00:13:50
people to do it, then that's not
great news. For the workers, that
00:13:54
is. On the other hand, you can
automate things that people hate
00:13:56
doing and refocus their work on
to other stuff where demand
00:14:01
expands. So these are choices
that companies and managers and
00:14:05
workers can make. They're not
foregone conclusions. And I
00:14:08
think I've— I have confidence in
the— you know, the talent of
00:14:12
people out there in the world to
make good choices there, and,
00:14:15
you know, ultimately end up in a
more fulfilling sort of work
00:14:18
configuration scenario.
00:14:19
Dan, great to have
you in here today.
00:14:21
Thanks very much.
- Thanks so much
00:14:22
for having me. Great to
be here.
00:14:23
Thank you. Daniel Rock,
00:14:24
who's Assistant Professor of
Operations, Information and
00:14:27
Decisions here at the Wharton
School.
00:14:29
- Thank you for listening
00:14:30
to <i>The Ripple Effect</i>. We hope
you found this episode
00:14:33
informative and engaging. Don't
forget to subscribe and leave us
00:14:36
a review so that we can continue
to bring you the best insight
00:14:40
from the Wharton School.
