AI's Impact on Innovation Management – Christian Terwiesch & Valery Yakubovich | AI in Focus Series

Knowledge at Wharton

Ask about Knowledge at Wharton

Episode Description

EPISODE OVERVIEW Wharton’s Christian Terwiesch and Valery Yakubovich join Eric Bradlow, vice dean of Analytics at Wharton, to discuss AI’s influence on innovation management, from risk assessment and strategy to organization and value capture. How can businesses use AI to ideate and better meet customer needs? This interview is part of a special 10-part series called "AI in Focus."

Topics and questions covered: 00:00 - Introductions 1:07 - Defining 'innovation management' 4:46 - Will firms trust AI to prototype? 8:31 - Can AI innovate independently? 12:42 - Mack Institute's role in assisting companies 16:05 - Assessing AI's strengths and challenges 21:00 - How can students use AI? 23:00 - Potential frontiers for AI

SERIES OVERVIEW The artificial intelligence revolution is officially upon us. Hear from host Eric Bradlow, Vice Dean of Analytics at Wharton, as he meets with Wharton faculty experts across a wide range of business domains – from healthcare and neuroscience to sports and the auto industry – to learn how AI will impact these fields moving forward, and what we can do to stay ahead of the curve.

----- The Wharton School of the University of Pennsylvania is committed to sharing its intellectual capital through Knowledge at Wharton, the school’s online business analysis journal.

Launched in 1999, Knowledge at Wharton offers free access to:

- Articles, podcasts, and videos highlighting Wharton faculty research and analysis of current business trends - Interviews with book authors from Wharton School Press - In-depth series of curated content like Wharton Executive Education’s Nano Tools for Leaders - A searchable database of more than 10,000 articles covering all aspects of business - A weekly newsletter that delivers Knowledge at Wharton insights directly to your inbox

Visit the Knowledge at Wharton homepage: https://whr.tn/3qUpSBE Subscribe to the Knowledge at Wharton YouTube channel: https://whr.tn/3DEAgAU 📂 Innovation;Technology;AI 🏷️ Research;AI in Focus

Brief Summary

AI is transforming innovation management through enhanced idea generation and market research.

Key points

  • Welcome to the AI at Wharton Podcast. Join us as we explore the role of AI in business innovation.
  • The Role of AI in Innovation Management. Discover how AI is transforming innovation management practices.
  • AI in Market Research. Exploring the potential of AI in enhancing market research processes.
  • Generative AI's Impact on Creativity. Surprisingly, generative AI is proving to be quite creative in generating ideas.
  • Challenges in AI Adoption. Companies are grappling with how to effectively implement AI in their strategies.
  • The Power of GPT. GPT performs surprisingly well on MBA exam questions, achieving a solid B grade.

Transcript

welcome to the analytics at Wharton and AI at Wharton podcast series on artificial intelligence my name is Eric Brad a professor of marketing and statistics here at the Wharton School I'm also the vice dean of analytics what we're doing in this series is to explore the role of artificial intelligence in various aspects of business and today certainly is no exception maybe one that most people consider the most exciting which is a artificial intelligence in Innovation management I'm joined by two of my colleagues uh first is valer yakovich uh valer is executive director of the Mac Institute for Innovation management uh the Mac Institute focuses on creating synergies between research

teaching and the practice of innovation management within the school so valer welcome to our podcast oh thank you glad to be here I'm also joined by my friend and colleague Christian turves uh Christian is the Andrew mher professor at the Wharton School he's a professor and chair of the Wharton operations informations and decisions Department here he's also co-director of the Mac Institution and he also holds a faculty appointment in the Pearlman School of Medicine Christian welcome to the show thanks for having us it's great to have you both so let's first start I hate it when people use jargon so valer maybe I'll start with you what is innovation management and what does the Mac

Institute do and then we'll dive into what role AI might have to play in that oh let me try also I actually can rely on Christian's favorite definition of innovation I learned from my faculty called directors so basically it's about matching uh customer needs with the technological solutions we have out there and what we do basically we U our priority our depart kind of departure point is uh faculty research we fund it in Innovation entrepreneurship and then we try to translate it into um experiential learning for students and business practice for that purpose we have a course with students Project based we have corporate partners with whom we work kind of trying to identify

their problems and provide some kind of guidance thought leadership and so over the years we identified basically four areas which are the critical for us okay it's about um opportunities and risks uh how we discover them and analyze them strategy development uh organizing for Innovation what kind of organizational structures teams and so on you uh set up and uh employ and finally uh value capture from Innovation and so in my view today's conversation is about kind of how different types of AI in particular generative AI U affect all these areas basically that's why I think I'm here yeah so Christian maybe you could tell

our listeners here on this this Sirius XM Wharton podcast series I I'm going to use the vernacular here all hell must be breaking loose at the Mac Institute I mean if you guys are focusing on innovation and I think most people would argue one of the big areas of application of these large language models like Bard and chat GPT is innovation like how do you get started like how do you as a scholar think about your research how do you think about as a A Center Director how do you think about as the chair of the department with all the hats you wear where do you even get started and how do you think about it I would narrow it down to to three dimensions of innovation that when

I teach Executive Education when I teach rbes students I would want to focus on right there is the initial idea and that is this combination of solution and need uh that hopefully creates some form of value that is something that the student needs to be able to manage from I have an idea towards launching a venture and I think W has been wonderful at doing that AI is helpful at that level because a lot of the things that used to be very expensive very difficult to do such as market research such as prototyping through generative II got a lot better the second thing that we want talk is about is in in bigger organization established organizations there's a

pipeline of ideas flowing uh they don't have just one idea they have thousands of ideas and that's a process that needs to be managed and AI has been as we show in some of our research been really good at at fueling that process of filling that pipeline of ideas and then the third thing is that process needs to have a Direction I mean most organizations unlike the unless they Venture Capital firms they have some form of strategic intent and I have to help managers kind of think about possible future states of the world possible disruptive threats and again CH GPT or other kind of gener of models can help me imagine a world

for which I would should be prepared that I myself would have want to imagine so let me ask you valy as well um whether it's in general or the two specific uses that Christian mentioned which sit in my area of the world too which is marketing research and prototyping um would f are firms as you're talking to them would they really replace direct marketing research or creating minimal viable products and doing prototyping would they really trust artificial intelligence with this crucial step of the let's call it new product development process or what are you seeing out there well actually they definitely don't uh transfer decision making to these large language models

but what we do see they bring them in and trying to kind of um take all the Corpus of knowledge they accumulated over years uh bring it into kind of in interactions with these models and try to automate some parts of the process and actually we are doing the same here we the consistent uh request we have for example from the medical school or engineering school when we work with them on their inventions and trying to kind of uh develop Pence Innovation ecosystem uh they ask us for market research and the demand is so substantial we can't meet it with the available tools and available resources so what we are doing now we are trying

to actually uh figure out how to use l language models in assessing let's say potential uh commercialization potential of in new inventions in the School of Engineering so could maybe Christian since I know you're both you've written a number of books you've written stuff on Innovation and Innovation tournaments you've been an entrepreneur can you take our list ERS kind of this stepbystep process imagine the Mac Institute wanted to partner with pennovation to try to help think about the economic value or how to best launch projects how would you use artificial intelligence to help support that process I think in any Innovation process there are two key functions that

is the generation of opportunities create more better and higher variance opportunities it's a very derian process so we need to create variety first so let me start with that one large lot of argument against generative AI is that it tends to work for the you know the the center mass of the distribution it doesn't give you very good ideas in the long tail it doesn't if it's not in that trained Corpus you won't see it in some so how do you think it does in that first phase yeah I know I'm I'm glad that you asked so we have done a study based on MBA generated ideas of which we have thousands in our database so after teaching Innovation for many years and we compare this with

large language models and to our surprise the large language models are actually better of creating what we call High variance Innovations of polarizing Innovations of Innovations where the payoff distribution has a high variance which is good because Innovations ideas have this real option flavor that if the idea is bad we just cancel it we don't execute just for our listeners as measured by whom how do you measure the variance like do you use humans or you ask AI engine to score the one that would be uh Magic uh so we get get to your second part in the moment when I get to my second part which is a selection step uh so how do we evaluate

the quality of the ideas uh we do purchase intense studies uh that Standard Market Research we Rego on amk prolific or other platforms we showcase textural descriptions of the ideas we asked for purchase intent probabilities of large crowds which is not perfect but again best practice of what you guys in marketing do very before you get to the selection piece let me ask you let's imagine there's a world where people are doing what you're doing and thousands of these types of let's call it generative studies are done and M Turk and all of that's done won't eventually an AI engine be able to forecast the stuff

that we're using humans for to evaluate right now like right now we just have a data problem but if we had the data AI could do that too it's a really interesting question right I mean what we've seen with humans over many many years is the selection decision is always the hard one right right I mean coming up with ideas is I don't want to say easy but it's it's something that humans can do and now ai can do when we try to use AI to predict the quality of an idea it still struggles and again it's not too surprising in the sense that humans even Venture capitalists are really having a hard time predicting the odds H Val any thoughts about that about

the I'll call it the idea Generation by the way I love this bifurcation into the idea generation stage versus the selection phase and I as a statistician I would imagine with enough data eventually and enough variation you know as we always say you need variation in X to be able to have good selection models and you need to of course observe outcomes over time I would imagine that if we're sitting here 5 years from now ai engines may be able to do better A lot better on the selection phase than they they do now but what are your thoughts well actually I think if we look a couple years back we thought AI

will never be creative we always thought AI will be predictive but creativity based on existing data and so on and suddenly we're surprised we find that generative AI is quite creative and but if you think more about it it's not so surprising one how do we Define creativity we go back to shet famous economist he said it's about Rec combination of existing ideas and because um the large language models are trained on such a huge volume of information which encompasses all kinds of diverse opinions if the task is not very well defined it actually does better because it can produce all kinds of opinions you can imagine all kinds of

customer profiles you can imagine and uh this variability becomes very helpful for a combination right that's why these findings are quite uh consistent with what we know where how creativity operates and and by the way the nice thing that uh chrisan is doing he's sharing a lot of his findings with the press and social media so I had read that article and I'm glad that I read it no I I give myself a good grade because that is what I read from the article so I'm glad that I interpreted the findings appropriately um so there's the technology piece there's the Innovation piece but what about the company

adoption piece so what are you guys seeing in the Mac Institute part of the world are companies embracing this as the next great opportunity or as or as companies thinking my God this is a threat to my business model what do you see um happening out there in the world uh so we just on Wednesday evening a session in Executive Education with the uh customer analytics program that you're well familiar with of course which I taught on on Monday all right and so we're talking about with the participants of what is it means for their business and I think many of them are struggling making sense of the technology they know it's big but they have a hard time for like what do I do

next Where do I get started and I think I hate to say this to a marketing Professor right but I mean you start with your customer Journey you start with your customer pain points it's not the right strategy to say like let's AI everything you look for the customer Journey what are the customer needs where are the pain points and then identify those and think along those customer Journeys where could AI be the right solution you now have a new set of tools and you can go through your existing pain points that you might have known for many years and fix those plus you can find through the sensing technology of AI by having it read customer reviews by interviewing it you

can find new pain points along the way that you might have not been aware of I see so valer um what are you seeing since uh you know as the executive director of the Mac Institute a lot of your role is to interface with companies you know us as faculty directors we obviously do a lot of research we also interface with companies but you're really on the front lines what are you seeing today and how are companies thinking that the Mac Institute can help them well uh I think uh right now that's the major disruptive technology that preoccupies manager attention basically uh I mentioned briefly that we have this experiential learning piece at the Mark Institute we

do projects with companies this semester out of seven projects four are about generative Ai and I kind of looked at them before can you tell us can you tell our listeners without giving necessarily the company or something can you tell us what are the projects what they a few kind of pretty much their names their titles smart Supply Chain management using generative AI uh AI Synergy strategizing and operationalizing intelligent transformation it's a very general kind of topic one is directly relevant to in AI in Innovation management it's intelligence software testing with a AI ml Innovations right so software testing is one major part of

any Innovation or testing prototype typing in general and finally evaluating the potential of disrupting the mortgage title industry through AI technology what I also see I was last week in Silicon Valley meeting with our corporate Partners meeting with startups some of them gaining a lot of traction for example going back to Innovation management there is one company that um pretty much automated the process of patent writing which is a extremely labor I saw an academic talk on that maybe about two or three months ago where I I thought it was remarkable yeah I mean you can think it's very well structured language right and very specific very hard to understand

language if you read the patent uh but I can imagine it's pretty straightforward to train um machine learning AI generative AI L language model to do it and uh what I notice when companies deal with these things what they need to do they they obviously take an existing uh L language model Foundation L language models they don't develop their own but then they need sa safeguards uh a wall between the vendor of the model and their own knowledge base and vendors now willing to provide it at the same time also a number of startups emerged that actually uh offering these companies um security privacy and other tools to also

not only Safeguard their own knowledge base but the knowledge base of their clients that they're going to use in order to deliver value to clients so I think this privacy uh confidentiality are key issues and security of these models what we see going on and uh another example of very interesting creative application for Innovation management I encounter it it's a company that have a huge databas base of cancer patients and uh now they're trying to engage large language models to match it with uh fda's uh uh database of clinical trials that are going on in order to find the right subjects for the right trial

so apparently it's a huge value added that can be done now at large scale using this model so Christian let me ask you on the you know I'll call it biggest opportunity side and the biggest area where you don't you still don't see AI being used much what are you seeing as like if you had to give a lecture tomorrow to your MBA students and say this is the most sophisticated interesting value added application of AI this is what I've seen the last three years and if you also had to give the same lecture and say and here's an example where I think there's an opportunity but I haven't seen anything yet what would those be so in terms of word works I think anything that is

simple text writing and simple all the way to a new patent but it is a writing task AI is amazing and that I think that n has been cracked it's only going to get better I think as faculty we all get these inquiries Professor bradl could you summarize this paper for me I mean don't do this right AI can do this and so let me ask you you um part of both of our jobs is as Journal editors and reviewers should I ask should I no the the journals have a policy against this potentially right now maybe not in the future but let's say there was no policy should I take an article jam it into an AI engine ask it to give me a summary of it in addition to my reading of it as a

reviewer you're should not but as somebody who wants uh to stay current with the literature having basic the AI give you every morning uh a two-minute summary of a paper that otherwise takes an hour or two hours to read I think that would be a very healthy thing forget the ethical or moral parts of it why do you say as a reviewer I should not I have I could give a reason but I'm not here I'm here to interview you who cares what I think what do you think so as a reviewer you have to make you have to turn over every stone to make sure that there's not the fall the flaw in the paper in the methodology I think that is something which is a COR Corner that I don't think AI is ready to do

I've tried this AI is doing a decent job when you feed it a PDF of a paper saying like look there might be some issues with endogenity or what have you some pretty generic ones I don't think it has the Precision to dial in and say like in equation seven the erot term is correlated with the explanatory variable I see right so I think there uh we still have to do the homework um the second type of work is the one that puzzles me the most is analytical types of things right I mean so uh especially at the beginning when I gave GPT my MBA exam a year ago please tell our listeners about this because this by the way I I I would

guess it is probably the most publicized article that has come out of Wharton in the last 5 years no I'm just saying it was on every major News Network it was republished everywhere so please tell people about the study what you found and then what you're still puzzled about uh so over the winter break my kids and I were sitting together my kids are in college or through college who were talking about GPT like everybody else probably in the world and so the question came up like that you're teaching in this MBA course you think that GPT could take your exam and so we literally took my MBA exam and fed the questions cut and pasted them in the

prompt line and it did really well it did what I would have given it a solid B if not a B+ over the subsequent months then was GP before coming out it is now well in the a range basically the type of questions and my my questions are many cases so to say like 10 lines of text with some computations in there find the Bott compute the inventory cost uh do some queuing analysis um GPT is amazing at that which again is counterintuitive because it's a language model right it has it has no representation inside of what capacity even is uh you tell it to find a a rout through uh traveling salesman problem connect uh cities in the right sequence to minimize Transportation time it does

a pretty decent job already right out of the box what is new since is uh with GPT 4 also we have these plugins now one of the set of plugins is from W from Al Alpha which is kind of the the the power side for analytics that's going to help and that's going to help right but even the plain vanilla GPT out of the box has gotten pretty good at doing analytics task so I imagine just as a lay person most of our listeners here would have not used wol from analytics unless they nurds like you and me right I mean that's an Insider type of tool sure you now have basically a user interface that lets you do sophisticated analytics that

lay persons can do and in make inquiries into exploring into analyzing hard mathematical problems data sets operations research type of problems I think that is super exciting H so uh valer could you tell us about what role do students play at the Mac Institute and um what do you think like when students I'm sure ask you all the time they ask me all the time especially when around machine learning like Professor bradow what should I be studying now what do you tell students what should they focus on becoming great prompt Engineers should they focus on being able to take the output of chat GPT integrate it with their own beliefs and then help in decision making how do

you see that well basically uh we have they have a challenge now uh and Christians research showed what the challenge is you have to figure out CH GPT can do better some tasks you have to figure out where you belong and there is actually another uh recent study done by a large group group of researchers I on them Ethan mik our colleague in management uh it was done at the Boston Consulting group where they really in the randomized experiment looked what Consultants can do what can't do with generative fi and roughly speaking my reading of the paper is that on exploration their productivity drastically increases with the Gen L

language models on problem solving more specific contextual tasks which require more Precision understanding of the context and so on those who use I do worse so basically we know um Ethan actually Ethan and his colleagues talk about this kind of changing Frontier where which task can be done or cannot be done by geni U and it's hard to identify and it's a moving Target right but they need to experiment themselves they have to innovate and reinvent their careers in some sense uh so disruptive uh this technology is uh yes so Christian let me ask you um if we were sitting here 5 years from now what do you think will have changed in those five years is it

the language models will get better at prediction which I'm sure the answer to that is yes the application areas that we haven't even thought of will get done that's probably yes but what do you think are the big changes our listeners should know that is coming in the next five or so years I think we have to stop thinking about the substitution game where will the NBA student be replaced by GPT the amount of work that is going around in the world is not constant if we make it efficient and cheap enough there's new work that is going to Bubble Up right so I just wrote as my latest paper a paper on ethical advice and AI can AI give me ethical advice and we show in the paper that it is basically

as good as the ethicist in the New York Times on providing advice to or to readers or people who face ethical dilemas so what does this mean one hypothesis it it puts the ethics advisors out of business but that's what not not what I think what I think is going to happen is we're going to go to a a virtual zero marginal cost ethics advisor a lot more and get more ethical advice right and so this productivity gain is not putting people out of work the amount of work that we can productively serve is going to grow up and so the effect on employment is highly ambivalent H yeah please uh just to add to this uh I just worked on the paper with Peter Capelli and S tber yeah

on kind of trying to think through this organizational implications of generative Ai and um one point we are trying to make is that jobs consist of multiple tasks and what we see so far some tasks indeed can be automated but then the amount of information you have to process produced by generative fi is at such a scale that sometimes uh it becomes costly you need to now make sense of this and the problem is that these models really the kind of explainability is still a big problem so I talked to some engineers in this um uh area related to healthc care and they're saying that new types of models now they

believe might replace L language models which actually can so to say understand things at a large more conceptual level when instead of predicting the next word as these models do you kind of uh based on what you know so far you predict the next kind of the high level concept and the text Will Follow from that concept this is the way we think right and so apparently there is quite a bit of attraction in that area um so we'll see large language models is it the last word uh or we'll have a totally quite different technology which is already out there for for example uh visual images basically when you see a part of the image you reconstruct the second

part completely instead of specific pixels right so there is a lot of development again it's a moving Target which uh is exciting to watch these developments and we need to adjust quickly and kind of explores things as they appeared well this has been our episode on AI and Innovation management I'd like to thank my colleague uh Valeri yakovich the executive director of the Mac Institute and my colleague uh Christian tursh the chair of the oid department here at the warten school the co-director of the Mac Institute uh thank you both for joining me on this analytics at Wharton AI at Warton podcast here thank you very much for having for

having us

Tension

40/ 100Low tension

Related episodes