00:00:00
welcome to the analytics at Wharton and AI at Wharton podcast series on artificial intelligence my name is Eric
00:00:06
Brad a professor of marketing and statistics here at the Wharton School I'm also the vice dean of analytics what
00:00:12
we're doing in this series is to explore the role of artificial intelligence in
00:00:16
various aspects of business and today certainly is no exception maybe one that most people consider the most exciting
00:00:23
which is a artificial intelligence in Innovation management I'm joined by two
00:00:28
of my colleagues uh first is valer yakovich uh valer is executive director of the Mac Institute for Innovation
00:00:35
management uh the Mac Institute focuses on creating synergies between research teaching and the practice of innovation
00:00:42
management within the school so valer welcome to our podcast oh thank you glad to be here I'm also joined by my friend
00:00:48
and colleague Christian turves uh Christian is the Andrew mher professor at the Wharton School he's a professor
00:00:53
and chair of the Wharton operations informations and decisions Department here he's also co-director of the Mac
00:00:59
Institution and he also holds a faculty appointment in the Pearlman School of Medicine Christian welcome to the show
00:01:05
thanks for having us it's great to have you both so let's first start I hate it
00:01:09
when people use jargon so valer maybe I'll start with you what is innovation
00:01:15
management and what does the Mac Institute do and then we'll dive into what role AI might have to play in that
00:01:22
oh let me try also I actually can rely on Christian's favorite definition of
00:01:27
innovation I learned from my faculty called directors so basically it's about
00:01:31
matching uh customer needs with the technological solutions we have out there and what we do basically we U our
00:01:41
priority our depart kind of departure point is uh faculty research we fund it in Innovation entrepreneurship and then
00:01:49
we try to translate it into um experiential learning for students and business practice for that purpose we
00:01:55
have a course with students Project based we have corporate partners with whom we work kind of trying to identify
00:02:02
their problems and provide some kind of guidance thought leadership and so over the years we identified basically four
00:02:09
areas which are the critical for us okay it's about um opportunities and risks uh how we
00:02:18
discover them and analyze them strategy development uh organizing for Innovation
00:02:24
what kind of organizational structures teams and so on you uh set up and uh employ and finally uh value capture from
00:02:33
Innovation and so in my view today's conversation is about kind of how different types of AI in particular
00:02:40
generative AI U affect all these areas basically that's why I think I'm here
00:02:46
yeah so Christian maybe you could tell our listeners here on this this Sirius XM Wharton podcast series I I'm going to
00:02:52
use the vernacular here all hell must be breaking loose at the Mac Institute I mean if you guys are focusing on
00:02:58
innovation and I think most people would argue one of the big areas of application of these large language
00:03:05
models like Bard and chat GPT is innovation like how do you get started like how do you as a scholar think about
00:03:13
your research how do you think about as a A Center Director how do you think about as the chair of the department
00:03:18
with all the hats you wear where do you even get started and how do you think about it I would narrow it down to to
00:03:24
three dimensions of innovation that when I teach Executive Education when I teach
00:03:29
rbes students I would want to focus on right there is the initial idea and that is this combination of solution and need
00:03:36
uh that hopefully creates some form of value that is something that the student needs to be able to manage from I have
00:03:41
an idea towards launching a venture and I think W has been wonderful at doing that AI is helpful at that level because
00:03:49
a lot of the things that used to be very expensive very difficult to do such as market research such as prototyping
00:03:55
through generative II got a lot better the second thing that we want talk is about is in in bigger organization
00:04:02
established organizations there's a pipeline of ideas flowing uh they don't
00:04:07
have just one idea they have thousands of ideas and that's a process that needs
00:04:11
to be managed and AI has been as we show in some of our research been really good
00:04:16
at at fueling that process of filling that pipeline of ideas and then the third thing is that process needs to
00:04:22
have a Direction I mean most organizations unlike the unless they Venture Capital firms they have some
00:04:28
form of strategic intent and I have to help managers kind of think about possible future states of
00:04:34
the world possible disruptive threats and again CH GPT or other kind of gener of models can help me imagine a world
00:04:43
for which I would should be prepared that I myself would have want to imagine so let me ask you valy as well um
00:04:50
whether it's in general or the two specific uses that Christian mentioned which sit in my area of the world too
00:04:56
which is marketing research and prototyping um would f are firms as you're talking to them would they really
00:05:03
replace direct marketing research or creating minimal viable products and doing prototyping would they really
00:05:10
trust artificial intelligence with this crucial step of the let's call it new
00:05:16
product development process or what are you seeing out there well actually they definitely don't uh transfer decision
00:05:23
making to these large language models but what we do see they bring them in and trying to kind of um take all the
00:05:32
Corpus of knowledge they accumulated over years uh bring it into kind of in interactions with these models and try
00:05:39
to automate some parts of the process and actually we are doing the same here we the consistent uh request we have for
00:05:46
example from the medical school or engineering school when we work with them on their inventions and trying to
00:05:51
kind of uh develop Pence Innovation ecosystem uh they ask us for market research and the demand is so
00:06:00
substantial we can't meet it with the available tools and available resources
00:06:06
so what we are doing now we are trying to actually uh figure out how to use l language models in assessing let's say
00:06:13
potential uh commercialization potential of in new inventions in the School of Engineering so could maybe Christian
00:06:20
since I know you're both you've written a number of books you've written stuff
00:06:24
on Innovation and Innovation tournaments you've been an entrepreneur can you take our list ERS
00:06:29
kind of this stepbystep process imagine the Mac Institute wanted to partner with
00:06:34
pennovation to try to help think about the economic value or how to best launch projects how would you use artificial
00:06:42
intelligence to help support that process I think in any Innovation process there are two key functions that
00:06:48
is the generation of opportunities create more better and higher variance opportunities it's a very derian process
00:06:55
so we need to create variety first so let me start with that one large lot of argument against generative
00:07:03
AI is that it tends to work for the you know the the center mass of the distribution it doesn't give you very
00:07:11
good ideas in the long tail it doesn't if it's not in that trained Corpus you
00:07:16
won't see it in some so how do you think it does in that first phase yeah I know
00:07:22
I'm I'm glad that you asked so we have done a study based on MBA generated
00:07:26
ideas of which we have thousands in our database so after teaching Innovation for many years and we compare this with
00:07:32
large language models and to our surprise the large language models are actually better of creating what we call
00:07:39
High variance Innovations of polarizing Innovations of Innovations where the payoff distribution has a high variance
00:07:46
which is good because Innovations ideas have this real option flavor that if the
00:07:51
idea is bad we just cancel it we don't execute just for our listeners as measured by whom how do you measure the
00:07:57
variance like do you use humans or you ask AI engine to score the one that would be uh Magic uh so we get get to
00:08:05
your second part in the moment when I get to my second part which is a selection step uh so how do we evaluate
00:08:10
the quality of the ideas uh we do purchase intense studies uh that Standard Market Research we Rego on amk
00:08:18
prolific or other platforms we showcase textural descriptions of the ideas we asked for purchase intent probabilities
00:08:25
of large crowds which is not perfect but again best practice of what you guys in
00:08:30
marketing do very before you get to the selection piece let me ask you let's
00:08:34
imagine there's a world where people are doing what you're doing and thousands of
00:08:40
these types of let's call it generative studies are done and M Turk and all of
00:08:45
that's done won't eventually an AI engine be able to forecast the stuff that we're using humans for to evaluate
00:08:54
right now like right now we just have a data problem but if we had the data AI could do that too it's a really
00:08:59
interesting question right I mean what we've seen with humans over many many
00:09:03
years is the selection decision is always the hard one right right I mean coming up with ideas is I don't want to
00:09:09
say easy but it's it's something that humans can do and now ai can do when we
00:09:15
try to use AI to predict the quality of an idea it still struggles and again it's not too surprising in the sense
00:09:22
that humans even Venture capitalists are really having a hard time predicting the
00:09:26
odds H Val any thoughts about that about the I'll call it the idea Generation by
00:09:31
the way I love this bifurcation into the idea generation stage versus the selection phase and I as a statistician
00:09:40
I would imagine with enough data eventually and enough variation you know as we always say you need variation in X
00:09:47
to be able to have good selection models and you need to of course observe outcomes over time I would imagine that
00:09:53
if we're sitting here 5 years from now ai engines may be able to do better A
00:09:57
lot better on the selection phase than they they do now but what are your thoughts well actually I think if we
00:10:02
look a couple years back we thought AI will never be creative we always thought AI will be predictive but creativity
00:10:10
based on existing data and so on and suddenly we're surprised we find that generative AI is quite creative and but
00:10:17
if you think more about it it's not so surprising one how do we Define creativity we go back to shet famous
00:10:24
economist he said it's about Rec combination of existing ideas and because um the large language models are
00:10:30
trained on such a huge volume of information which encompasses all kinds of diverse opinions if the task is not
00:10:38
very well defined it actually does better because it can produce all kinds of opinions you can imagine all kinds of
00:10:45
customer profiles you can imagine and uh this variability becomes very helpful for a combination right that's why these
00:10:53
findings are quite uh consistent with what we know where how creativity operates and and by the way the nice
00:10:59
thing that uh chrisan is doing he's sharing a lot of his findings with the press and social media so I had read
00:11:03
that article and I'm glad that I read it no I I give myself a good grade because
00:11:07
that is what I read from the article so I'm glad that I interpreted the findings
00:11:11
appropriately um so there's the technology piece there's the Innovation piece but what about the company
00:11:19
adoption piece so what are you guys seeing in the Mac Institute part of the world are companies embracing this as
00:11:26
the next great opportunity or as or as companies thinking my God this is a threat to my business model what do you
00:11:33
see um happening out there in the world uh so we just on Wednesday evening a session in Executive Education with the
00:11:41
uh customer analytics program that you're well familiar with of course which I taught on on Monday all right
00:11:46
and so we're talking about with the participants of what is it means for their business and I think many of them
00:11:52
are struggling making sense of the technology they know it's big but they have a hard time for like what do I do
00:11:58
next Where do I get started and I think I hate to say this to a marketing Professor right but I mean you start
00:12:04
with your customer Journey you start with your customer pain points it's not
00:12:08
the right strategy to say like let's AI everything you look for the customer
00:12:12
Journey what are the customer needs where are the pain points and then identify those and think along those
00:12:18
customer Journeys where could AI be the right solution you now have a new set of
00:12:22
tools and you can go through your existing pain points that you might have known for many years and fix those plus
00:12:30
you can find through the sensing technology of AI by having it read customer reviews by interviewing it you
00:12:37
can find new pain points along the way that you might have not been aware of I see so valer um what are you seeing
00:12:43
since uh you know as the executive director of the Mac Institute a lot of your role is to interface with companies
00:12:48
you know us as faculty directors we obviously do a lot of research we also interface with companies but you're
00:12:53
really on the front lines what are you seeing today and how are companies thinking that the Mac Institute can help
00:13:00
them well uh I think uh right now that's the major disruptive technology that preoccupies manager
00:13:11
attention basically uh I mentioned briefly that we have this experiential learning piece at the Mark Institute we
00:13:17
do projects with companies this semester out of seven projects four are about generative Ai and I kind of looked at
00:13:24
them before can you tell us can you tell our listeners without giving necessarily
00:13:27
the company or something can you tell us what are the projects what they a few kind of pretty much their names their
00:13:33
titles smart Supply Chain management using generative AI uh AI Synergy strategizing and operationalizing
00:13:41
intelligent transformation it's a very general kind of topic one is directly
00:13:45
relevant to in AI in Innovation management it's intelligence software testing with a AI ml Innovations right
00:13:53
so software testing is one major part of any Innovation or testing prototype typing in general and finally evaluating
00:14:01
the potential of disrupting the mortgage title industry through AI technology what I also see I was last week in
00:14:08
Silicon Valley meeting with our corporate Partners meeting with startups some of them gaining a lot of traction
00:14:13
for example going back to Innovation management there is one company that um pretty much automated the process of
00:14:20
patent writing which is a extremely labor I saw an academic talk on that maybe about two or three months ago
00:14:28
where I I thought it was remarkable yeah I mean you can think it's very well
00:14:34
structured language right and very specific very hard to understand language if you read the patent uh but I
00:14:39
can imagine it's pretty straightforward to train um machine learning AI generative AI L language model to do it
00:14:47
and uh what I notice when companies deal with these things what they need to do they they obviously take an existing uh
00:14:54
L language model Foundation L language models they don't develop their own but
00:14:58
then they need sa safeguards uh a wall between the vendor of the model and their own knowledge base and vendors now
00:15:06
willing to provide it at the same time also a number of startups emerged that actually uh offering these companies um
00:15:15
security privacy and other tools to also not only Safeguard their own knowledge base but the knowledge base of their
00:15:21
clients that they're going to use in order to deliver value to clients so I think this privacy uh confidentiality
00:15:28
are key issues and security of these models what we see going on and uh another example of very interesting
00:15:35
creative application for Innovation management I encounter it it's a company
00:15:40
that have a huge databas base of cancer patients and uh now they're trying to
00:15:47
engage large language models to match it with uh fda's uh uh database of clinical trials
00:15:55
that are going on in order to find the right subjects for the right trial so apparently it's a huge value added
00:16:02
that can be done now at large scale using this model so Christian let me ask you on the you know I'll call it biggest
00:16:08
opportunity side and the biggest area where you don't you still don't see AI
00:16:13
being used much what are you seeing as like if you had to give a lecture tomorrow to your MBA students and say
00:16:20
this is the most sophisticated interesting value added application of AI this is what I've seen the last three
00:16:27
years and if you also had to give the same lecture and say and here's an example where I think there's an
00:16:31
opportunity but I haven't seen anything yet what would those be so in terms of
00:16:36
word works I think anything that is simple text writing and simple all the way to a new patent but it is a writing
00:16:43
task AI is amazing and that I think that n has been cracked it's only going to
00:16:48
get better I think as faculty we all get these inquiries Professor bradl could you summarize this paper for me I mean
00:16:54
don't do this right AI can do this and so let me ask you you um part of both of
00:17:00
our jobs is as Journal editors and reviewers should I ask should I no the the journals have a policy against this
00:17:09
potentially right now maybe not in the future but let's say there was no policy
00:17:13
should I take an article jam it into an AI engine ask it to give me a summary of
00:17:18
it in addition to my reading of it as a reviewer you're should not but as somebody who wants uh to stay current
00:17:26
with the literature having basic the AI give you every morning uh a two-minute summary of a paper that otherwise takes
00:17:34
an hour or two hours to read I think that would be a very healthy thing forget the ethical or moral parts of it
00:17:40
why do you say as a reviewer I should not I have I could give a reason but I'm
00:17:44
not here I'm here to interview you who cares what I think what do you think so
00:17:47
as a reviewer you have to make you have to turn over every stone to make sure that there's not the fall the flaw in
00:17:54
the paper in the methodology I think that is something which is a COR Corner that I don't think AI is ready to do
00:18:01
I've tried this AI is doing a decent job when you feed it a PDF of a paper saying
00:18:07
like look there might be some issues with endogenity or what have you some pretty generic ones I don't think it has
00:18:14
the Precision to dial in and say like in equation seven the erot term is correlated with the explanatory variable
00:18:21
I see right so I think there uh we still have to do the homework um the second type of work is the one that puzzles me
00:18:27
the most is analytical types of things right I mean so uh especially at the beginning when I gave GPT my MBA exam a
00:18:37
year ago please tell our listeners about this because this by the way I I I would
00:18:42
guess it is probably the most publicized article that has come out of Wharton in
00:18:47
the last 5 years no I'm just saying it was on every major News Network it was
00:18:51
republished everywhere so please tell people about the study what you found and then what you're still puzzled about
00:18:58
uh so over the winter break my kids and I were sitting together my kids are in college or through college who were
00:19:04
talking about GPT like everybody else probably in the world and so the question came up like that you're
00:19:09
teaching in this MBA course you think that GPT could take your exam and so we literally took my MBA exam and fed the
00:19:15
questions cut and pasted them in the prompt line and it did really well it did what I would have given it a solid B
00:19:23
if not a B+ over the subsequent months then was GP before coming out it is now well in the a range basically the type
00:19:32
of questions and my my questions are many cases so to say like 10 lines of text with some computations in there
00:19:40
find the Bott compute the inventory cost uh do some queuing analysis um GPT is amazing at that which again is
00:19:49
counterintuitive because it's a language model right it has it has no representation inside of what capacity
00:19:55
even is uh you tell it to find a a rout through uh traveling salesman problem connect uh cities in the right sequence
00:20:04
to minimize Transportation time it does a pretty decent job already right out of
00:20:09
the box what is new since is uh with GPT 4 also we have these plugins now one of
00:20:16
the set of plugins is from W from Al Alpha which is kind of the the the power side for analytics that's going to help
00:20:23
and that's going to help right but even the plain vanilla GPT out of the box has
00:20:28
gotten pretty good at doing analytics task so I imagine just as a lay person most of our listeners here would have
00:20:37
not used wol from analytics unless they nurds like you and me right I mean that's an Insider type of tool sure you
00:20:42
now have basically a user interface that lets you do sophisticated analytics that
00:20:49
lay persons can do and in make inquiries into exploring into analyzing hard mathematical problems data sets
00:20:57
operations research type of problems I think that is super exciting H so uh valer could you tell us about what role
00:21:03
do students play at the Mac Institute and um what do you think like when students I'm sure ask you all the time
00:21:12
they ask me all the time especially when around machine learning like Professor bradow what should I be studying now
00:21:18
what do you tell students what should they focus on becoming great prompt Engineers should they focus on being
00:21:24
able to take the output of chat GPT integrate it with their own beliefs and then help in decision making how do
00:21:30
you see that well basically uh we have they have a challenge now uh and Christians research showed what the
00:21:39
challenge is you have to figure out CH GPT can do better some tasks you have to figure out where you belong and there is
00:21:46
actually another uh recent study done by a large group group of researchers I on
00:21:52
them Ethan mik our colleague in management uh it was done at the Boston Consulting group where they really in
00:21:59
the randomized experiment looked what Consultants can do what can't do with generative fi and roughly speaking my
00:22:07
reading of the paper is that on exploration their productivity drastically increases with the Gen L
00:22:15
language models on problem solving more specific contextual tasks which require more Precision understanding of the
00:22:25
context and so on those who use I do worse so basically we know um Ethan actually Ethan and his colleagues talk
00:22:36
about this kind of changing Frontier where which task can be done or cannot be done by
00:22:42
geni U and it's hard to identify and it's a moving Target right but they need
00:22:48
to experiment themselves they have to innovate and reinvent their careers in some sense uh so disruptive uh this
00:22:55
technology is uh yes so Christian let me ask you um if we were sitting here 5 years from now what do you think will
00:23:04
have changed in those five years is it the language models will get better at prediction which I'm sure the answer to
00:23:11
that is yes the application areas that we haven't even thought of will get done
00:23:15
that's probably yes but what do you think are the big changes our listeners should know that is coming in the next
00:23:21
five or so years I think we have to stop thinking about the substitution game where will the NBA student be replaced
00:23:28
by GPT the amount of work that is going around in the world is not constant if we make it efficient and cheap enough
00:23:35
there's new work that is going to Bubble Up right so I just wrote as my latest
00:23:40
paper a paper on ethical advice and AI can AI give me ethical advice and we show in the paper that it is basically
00:23:47
as good as the ethicist in the New York Times on providing advice to or to readers or people who face ethical
00:23:55
dilemas so what does this mean one hypothesis it it puts the ethics advisors out of business but that's what
00:24:01
not not what I think what I think is going to happen is we're going to go to
00:24:04
a a virtual zero marginal cost ethics advisor a lot more and get more ethical advice right and so this productivity
00:24:12
gain is not putting people out of work the amount of work that we can productively serve is going to grow up
00:24:19
and so the effect on employment is highly ambivalent H yeah please uh just to add to this uh I just worked on the
00:24:27
paper with Peter Capelli and S tber yeah on kind of trying to think through this
00:24:31
organizational implications of generative Ai and um one point we are trying to make is that jobs consist of
00:24:39
multiple tasks and what we see so far some tasks indeed can be automated but then the amount of information you have
00:24:48
to process produced by generative fi is at such a scale that sometimes uh it becomes costly you need to now make
00:24:55
sense of this and the problem is that these models really the kind of explainability is still a big problem so
00:25:03
I talked to some engineers in this um uh area related to healthc care and they're
00:25:09
saying that new types of models now they believe might replace L language models
00:25:14
which actually can so to say understand things at a large more conceptual level when instead of predicting the next word
00:25:21
as these models do you kind of uh based on what you know so far you predict the next kind of the high level concept and
00:25:30
the text Will Follow from that concept this is the way we think right and so apparently there is quite a bit of
00:25:37
attraction in that area um so we'll see large language models is it the last
00:25:41
word uh or we'll have a totally quite different technology which is already out there for for example uh visual
00:25:50
images basically when you see a part of the image you reconstruct the second part completely instead of specific
00:25:56
pixels right so there is a lot of development again it's a moving Target which uh is exciting to watch these
00:26:03
developments and we need to adjust quickly and kind of explores things as they appeared well this has been our
00:26:09
episode on AI and Innovation management I'd like to thank my colleague uh Valeri
00:26:14
yakovich the executive director of the Mac Institute and my colleague uh Christian tursh the chair of the oid
00:26:20
department here at the warten school the co-director of the Mac Institute uh thank you both for joining me on this
00:26:25
analytics at Wharton AI at Warton podcast here thank you very much for having for
00:26:29
having us