00:00:00
Daniel Rock: That's a really fascinating
point. I mean, you think about the value of an OpenAI or
Anthropic, even a Microsoft,
00:00:05
Google— you know, whoever's
building it, Llama— like, a lot of the value of those companies
is in the complementary
00:00:12
investments that their customers
are making. Or, like, the ecosystem at large is making.
That's really interesting,
00:00:17
right? Like the— they get more
valuable as their customers and as their consumers learn how to
integrate that toolkit. So I
00:00:25
think that's something that'll
take a little while, but they're well aware of it too.
00:00:29
You know, they're trying to
make it easier to use. - Welcome to <i>The Ripple Effect</i>,
00:00:33
the podcast
that takes you on a journey through the minds of Wharton
faculty. I'm your host, Dan
00:00:38
Loney, and in each episode,
we'll be diving deep into the inspiration behind the
groundbreaking research that
00:00:44
Wharton professors have
conducted and exploring how their findings resonate with the
world today.
00:00:51
Dan Loney: Well, when we think about innovation these days,
there's a good chance that
00:00:54
artificial intelligence is going
to come into the conversation. Even though it's been around for
some time, it only now feels
00:01:02
like the majority of the public
at large are seeing the impact of artificial intelligence in
our lives. Daniel Rock is an
00:01:10
Assistant Professor of
Operations, Information and Decisions here at the Wharton
School. He and colleagues have
00:01:14
looked at how AI can impact
something like productivity. Dan, great to have you here
today. Thanks for your time.
00:01:20
Great to be here. Thanks for
having me. So when I bring up AI, doesn't it seem like
productivity kind of is a— a
00:01:27
natural— a natural first thing
for people to think about? Oh, absolutely. And I think when we talk about
productivity, it's important to
00:01:35
define terms here. For
economists, productivity can be a few different things. It can
be how much output per worker
00:01:41
you have, how much revenue per
unit of input, but generally, all of it points to one big
idea, which is, what are the
00:01:48
number of outputs we get per
unit of input? So it's not, you know, how do we cut jobs, necessarily, or
reduce the resource use. It's
00:01:56
also, how do we create more? And
I think with these tools empowering people to do, you
know, greater and more
00:02:02
interesting things, productivity
in the long run has to kind of be positively impacted by what
we can do with them.
00:02:08
- Would you say that, though— that
there's a paradox when you think of this?
00:02:11
Yeah, sure. I think here's—
here's the core thing with a sufficiently transformative
technology, what economists
00:02:17
would call a general purpose
technology. That is, it's pervasive, it improves over
time, and then it kind of
00:02:23
necessitates and spawns
complementary innovation. That is the other stuff you need to
build to get this stuff to go.
00:02:30
So, yeah, there's a lag. It
takes a long time to build up those additional assets, to
reconfigure your organization,
00:02:36
to train people to use stuff.
Over time, that's going to pay off in a big way, and we're
seeing people make huge
00:02:41
investments in that. But it's
not going to be, right off the bat, super powerful. Actually,
it's funny with AI, there are
00:02:48
some applications that are right
off the bat super powerful, but the long run implications are
going to take a while to play
00:02:54
out, I think.
- You have four areas of potential impact that you've
come up with in the work that
00:03:00
you've done, the first being
false hopes. Explain that a little bit. Oh yeah. So this is the
explanation for the paradox. So, why
00:03:06
it takes a while. So I already
preempted what I think is going on. But yes, there is the— there
is the chance, right— this is
00:03:12
sort of a Bob Gordon view. I
don't want to put too many words in his mouth, but, you know,
00:03:17
AI just isn't
that big a deal. Or you could broaden this to say any
technology just isn't that big a
00:03:23
deal. And we see lots of promise
and hype, but it's just never going to materialize. That's a
consistent way to view the world
00:03:31
in the early stages, if you
don't know what's going to happen. But then you do have to,
you know, change tack if you see
00:03:36
the benefits start to show up. I
think with AI, we're starting to see that a bit. So that's number
one.
00:03:41
The second one is mismeasurement. So this is kind of—
there's some folks in Silicon
00:03:46
Valley who say this, and there's
some evidence that this might be going on too. The idea here is
that the gains are real. They're
00:03:53
happening, but we're not
capturing them properly in the economic statistics. And I think
this— the folks at the BLS and
00:04:01
the BEA do a really great job of
trying to measure the economy. You know, where it might be
tougher to measure things is
00:04:08
like, you know, Google's free. I
asked my MBAs, would you rather have, you know, search or indoor
plumbing? And you know, after
00:04:18
trying to wriggle out of that
conundrum, many of them will still pick search over indoor
plumbing. I'm kind of with them
00:04:24
on that one. - That's probably a
good idea. Yeah. It gets cold in Philly, but not too cold,
- Exactly.
00:04:29
So that's the second one. We could be mismeasuring
things. And yes, there are some
00:04:32
cases where that may be the
case. But in general, you have to make an argument for why it's
different now. What changed to
00:04:39
make us worse at measurement,
given what the economy is producing? And I think that's a
tougher case. My co-author, Chad
00:04:44
Syverson at University of
Chicago, has kind of disposed that argument. At least, you
know, up until 2017 or so.
00:04:52
So that's the second one. The third
one is sort of a rent dissipation argument. What does
that mean? It's, the gains are
00:04:58
real, but they're accruing to a
really small proportion of people in the economy. They're
taking all of the gains, and
00:05:06
nobody else is seeing anything
there. I think you could make an argument that a lot of that is
still happening, but it would
00:05:12
have to be really enormous to
take away the gains from the technology, given the
expectations. And then the last
00:05:18
one, which we just discussed,
restructuring and implementation lags. The stuff can take a
while. We see a lot of promise,
00:05:25
but let's not confuse a clear
view for a short walk. It's going to take a long time to
implement this stuff.
00:05:30
So is the expectation, then, with
where we are kind of currently, that we're still going to see
innovation coming from other
00:05:37
areas to kind of complement what
is, I think a lot of people believe, the core of— of what AI
is right now?
00:05:44
Yeah, absolutely. I think that's
that's already happening in a lot of areas. One of the really
cool things about AI is— I don't
00:05:52
know if you you code, or if you
if you use coding assistance, but the fact that these tools—
the software you need to augment
00:06:01
them, to augment the AI tools,
can be partially written with AI help. So we can— we really
get in this nice, like, flywheel,
00:06:09
where you use AI to improve the
sorts of tooling you need to make AI more effective.
00:06:15
That's— when I
talk with companies, I'm like, "Are you guys doing this? Because
you should be."
00:06:19
It's very helpful. There's an element that you talk about
in this paper, about the added
00:06:25
costs being a type of capital
that can be just part of the build out, I guess, correct?
00:06:32
Yeah, absolutely. When you have
these adjustment costs or fixed costs of investment, these are
things like the training or
00:06:39
intangible capital, or even the
culture around how you build an organization that— that works
with machine learning or AI
00:06:46
software. Like, this is a
different type of software. It creates output that's non-
deterministic, or maybe a little
00:06:54
bit fuzzier. It's not perfect
every time, it's not cookie cutter. That's a mindset shift,
too. You can't expect the same
00:07:00
results as you could with— with
ordinary kind of rules-based software. So you have to pay
that upfront cost, or maybe even
00:07:07
ongoing cost, to keep people in
the loop, to structure your organization processes properly.
And then what you— what you get
00:07:14
out of that is, you know,
competitive advantage, basically. You can do things
other companies can't, if you
00:07:20
crack that.
- Sure. - Yeah. And obviously, part of
that also probably fills into,
00:07:25
with the companies that are in
the development of AI, the value that those companies have— you
know, that that is a significant
00:07:33
beneficial capital component
that they have to their companies, which, you know,
00:07:38
obviously is larger than other
companies. Sure. That's a really fascinating
point. I mean, you think about
00:07:42
the value of an OpenAI or
Anthropic, even a Microsoft, Google— you know, whoever's
building it, Llama— like, a lot
00:07:48
of the value of those companies
is in the complementary investments that their customers
are making. Or, like, the
00:07:54
ecosystem at large is making.
That's really interesting, right? Like the— they get more
valuable as their customers and
00:08:01
as their consumers learn how to
integrate that toolkit. So I think that's— that's something
that will take a little while,
00:08:08
but they're well aware of it
too. You know, they're trying to make it easier to use. In some
sense, ChatGPT is more of a UX
00:08:13
innovation. The playground
existed. You could use stuff before, but ChatGPT just, like
showed people, "Hey, you can—you
00:08:21
can really engage with these
models and do something cool." I massively updated how important
I thought UX was after I saw the
00:08:29
success of that app. So where we are right now then,
you mentioned about some of the
00:08:34
metrics that will come into play
here. It feels like we're still at a point where the development
of some of those metrics is kind
00:08:42
of either— it either hasn't
happened or it's ongoing right now, and so it may be hard to
truly gauge the value or the
00:08:51
component of productivity,
especially when we don't have the dynamics fully tweaked to
what we need, right?
00:08:58
Yes, I agree with that. Though
what makes AI, this wave of software, a little bit easier to
do a good job there is, we're
00:09:06
using the last wave of IT to
instrument it, right? So we have software to track the software.
Before, I mean, the best you
00:09:13
could do, best you could hope
for, was surveys of some kind, like, say, in the early '90s. Now
we can scale up those efforts.
00:09:19
You can track sort of what some
of my colleagues called "digital exhaust", right? You connect to
APIs for companies, you can see
00:09:26
how they're changing what
they're doing. You know, there's little pockets. This is like
this big iceberg, and we're
00:09:32
seeing just the tip of it. But
you can use those sort of points where we can see the tip of the
iceberg and how it's changing as
00:09:41
a way to gauge what's going on.
And a concrete example is something— you know, something
I've done in my own work. I've
00:09:46
tracked how many people with AI
skills are being hired company by company, assuming that if
you're hiring people to do this,
00:09:52
you're probably building out all
the other complements to make them effective. And, you know,
that's— we're trying to measure
00:09:58
the— kind of the size of the— the
investments on that front, which Which for many companies, is probably
the way you need to do it, is
00:10:04
bring on the talent before you actually
do the level of implementation to get to that point, so you
have people who understand it
00:10:11
going— going— going into the
process. 100%. You can't get away from— from
labor markets and those
00:10:18
complementary investments. If
you do AI well, then you probably did data science well
before. If you did data science
00:10:24
well, you probably did the cloud
well before. There's a whole stacking of these technologies. It
actually makes AI super
00:10:29
concentrated in— in only a few
firms right now. And it's a little bit, you know— a little
bit of an explanation where that
00:10:35
might be coming from. But yeah,
you're bottlenecked in three potential areas. It's either
talent or data or compute right
00:10:43
now. But you mix those three
things together in the right proportions, and you start to
get, you know,
00:10:49
AI internal capabilities. What do you think, then, doing this
research helped you to better
00:10:55
understand about where we are
going, and that connection between AI and productivity?
- Yeah.
00:11:02
So I started to think, what are
the ingredients inside of a company that would generate
productivity? Like, how— where
00:11:10
does it come from? And there's a
few models that colleagues in other places have put together
that I can use as sort of
00:11:16
workhorse models. There's sort
of the task-based approach. There are researchers like David
Auter and Daron Acemoglu at MIT
00:11:24
who have used this heavily. The
idea is like, let's break down a job into bundle of tasks and
track all those tasks. Those
00:11:30
tasks or those bundles are
changing. I think the more change you see at that level,
the more of an indication you
00:11:37
have that something is different
now. So that's one early kind of check you can do to see what's
going on. So I'm working on some
00:11:44
of that, looking at job postings
with a few colleagues. And then there's kind of a perspective
that Tim Bresnahan at Stanford
00:11:53
as well as Joshua Gans, Avi
Goldfarb and Ajay Agrawal at University of Toronto have kind
of put forth. It's the sense that
00:12:02
nobody's ever lost their job to
task-based automation. It's not happening task by task,
necessarily, or change isn't
00:12:08
happening task by task. It's
happening at the system level. So when we change the direction
in what's possible with a model,
00:12:16
like, I can discover new drugs
with these tools, or I can— I mean, I can't. But I can make
pretty lousy images or paintings
00:12:26
that I couldn't do before. I'll
stop there with my new capabilities. But like, as— as you
give people these new
00:12:32
capabilities, you redesign the
system, and that new system's got different demands for people.
Different demands for capital.
00:12:38
- Right. - So let's see if, like,
there are companies making big changes saying "We're going to
reconfigure this module." It's
00:12:43
kind of hard to do that if
things are moving really quickly. You don't have
certainty. You don't feel like
00:12:47
you're standing on solid ground
when you do that. Does it feel, like, then— I'll finish up
on this. Does it feel like
00:12:53
that where we are with AI and
the workforce right now, that obviously AI is going to play a
significant role, but the human
00:13:02
component will be there. And to
a degree, maybe even the human component becomes kind of even
more of a learning experience as
00:13:09
we move forward, because of how
AI is kind of guiding the ship a
little bit here.
00:13:13
Yeah, I get accused of being an
optimist on this point, so I strongly agree with that. But I
do think there are, of course,
00:13:19
going to be pockets where things
go better or things go worse. One thing I've grown fond of
saying recently is that we can't
00:13:26
get away from labor markets. So
the— I will say, you know, perhaps it's a little bit of a
stretch. I don't want to get too
00:13:32
far out of my skis here. But the
augmentation versus automation debate matters at some unit of
analysis. But at the individual,
00:13:41
like, you know, worker or
manager making a decision about where to deploy the technology—
you can augment someone that can
00:13:47
do the job of 20 people, and if
the company doesn't want 20 people to do it, then that's not
great news. For the workers, that
00:13:54
is. On the other hand, you can
automate things that people hate doing and refocus their work on
to other stuff where demand
00:14:01
expands. So these are choices
that companies and managers and workers can make. They're not
foregone conclusions. And I
00:14:08
think I've— I have confidence in
the— you know, the talent of people out there in the world to
make good choices there, and,
00:14:15
you know, ultimately end up in a
more fulfilling sort of work configuration scenario.
00:14:19
Dan, great to have
you in here today. Thanks very much.
- Thanks so much for having me. Great to
be here.
00:14:23
Thank you. Daniel Rock, who's Assistant Professor of
Operations, Information and
00:14:27
Decisions here at the Wharton
School. - Thank you for listening to <i>The Ripple Effect</i>. We hope
you found this episode
00:14:33
informative and engaging. Don't
forget to subscribe and leave us a review so that we can continue
to bring you the best insight
00:14:40
from the Wharton School.