00:00:00
welcome welcome to this edition of the
00:00:03
AI at Wharton and analytics at Wharton
00:00:05
podcast series on artificial
00:00:06
intelligence today's episode will
00:00:08
actually have a dual role while it says
00:00:10
Ai and education here our guest actually
00:00:12
has lots of expertise in Ai and
00:00:14
education and the workforce and a lot
00:00:16
more General topics as well I'm joined
00:00:18
by my faculty colleague uh Ethan mik
00:00:21
Ethan is the Ralph J Roberts
00:00:22
distinguished faculty scholar he's an
00:00:24
associate professor in our Management
00:00:26
Department he's also the academic
00:00:27
director of Wharton interactive so Ethan
00:00:30
Welcome to our podcast series I'm
00:00:31
thrilled to be here thank you well I
00:00:34
don't even know where to start because
00:00:36
I'll just say and I this is my first
00:00:38
question to you most of what I've
00:00:41
learned on AI and education will start
00:00:43
there is by watching the five-part
00:00:45
series that you and your wife created um
00:00:47
so could you tell our listeners here on
00:00:49
our Ai and education and Workforce uh
00:00:52
episode what was in those five episodes
00:00:55
like what do all of us as professors
00:00:57
need to know about Ai and education well
00:01:00
I mean there's at least three different
00:01:01
things that matter right the first thing
00:01:03
that matters is disruption uh homework
00:01:05
is over right there's not a homework
00:01:07
assignment basically anywhere that a
00:01:08
well- prompted AI can solve at this
00:01:10
point so that's a big deal and just to
00:01:12
be clear let me take them one at a time
00:01:14
there are explain to our listeners there
00:01:16
are multiple versions of even let's even
00:01:19
say chat GPT which is just one of the
00:01:21
open AI sources like some versions can
00:01:23
ingest documents some versions cannot
00:01:26
some have just a text prompt so which
00:01:29
version or versions are you referring to
00:01:31
when you say kind of home homework as we
00:01:34
know it is over okay so when you think
00:01:36
about AI you want to think about um sort
00:01:38
of all the what's called Foundation
00:01:40
models which are llama and Chachi all
00:01:43
these kind of different models but you
00:01:44
also want to think about what are called
00:01:45
Frontier models not to create more
00:01:47
confusing vocabulary but there's really
00:01:49
only three Frontier models right now
00:01:51
which are open AI chat gp4 uh which is
00:01:55
the paid version but you can also get it
00:01:56
for free through Microsoft Bing in
00:01:58
creative mode which turns out to be
00:01:59
really important for education for
00:02:00
reasons we'll talk about which is the
00:02:01
way I do right now and it's a little
00:02:03
Limited in some ways it's weird it's has
00:02:05
a personality we can talk about that
00:02:07
then there is um Google's Bard which
00:02:09
right now is powered by a underpowered
00:02:10
model called pal 2 but all the rumors
00:02:12
are that it will be upgraded to a model
00:02:14
that probably will be the first model to
00:02:15
beat gp4 in the next couple of months
00:02:18
and then finally there's a company
00:02:19
called anthropic that has a product
00:02:21
called Claude 2 so when we talk when I
00:02:23
talk about AI can do something I'm
00:02:24
almost always talking about the frontier
00:02:26
models so
00:02:27
gp4 um currently has like a separate
00:02:30
mode for vision and pictures that's all
00:02:31
being united there's already been
00:02:32
they're already rolling that out so
00:02:33
they'll be able to take in documents
00:02:35
take in images read PDFs it already can
00:02:37
it just does it a little bit uh jenily
00:02:39
right now well let me ask you a few
00:02:40
things um well all of this right now
00:02:43
there are paid and unpaid versions
00:02:45
what's your vision since you also teach
00:02:47
Innovation you teach
00:02:49
entrepreneurship are all these things
00:02:51
going to stay free if they are what's
00:02:53
their revenue model is it adverti like
00:02:55
how do you see this playing out from
00:02:57
just a from our point of view and from
00:02:59
the company point of view so right now
00:03:02
open aai is has announced that they're
00:03:04
on a run rate of $1.2 billion in revenue
00:03:06
for after less than a year after
00:03:07
releasing chat GPT most of that money
00:03:09
probably comes through their use of
00:03:10
their API which is their their um
00:03:13
businessto Business Solution right uh
00:03:15
less of it is the $20 a month we pay
00:03:17
right if you pay for gbd plus which by
00:03:19
the way if you can you should um there's
00:03:22
is almost the difference between gbd4
00:03:24
and gbt 3.5 the paid version and the
00:03:26
free version is so large as it may not
00:03:28
appear that way at first but it's big
00:03:30
enough that it is 100% worth it um
00:03:32
Microsoft releases a bunch of gp4
00:03:35
products through Bing and they're
00:03:36
planning on doing it free as far as I
00:03:37
know for the near future Google bard is
00:03:39
also releasing theirs for free both
00:03:41
because it's part of the search engine
00:03:43
fight going on so we are the benefits
00:03:45
but it also means that people in 169
00:03:46
countries around the world have access
00:03:48
to gp4 and will probably have barred
00:03:50
access as well which means the same
00:03:52
model that you get access to if you go
00:03:53
to Goldman Sachs or you go to you know
00:03:55
McKenzie or you go to Nike it's no
00:03:57
better than the model that every kid in
00:03:59
ug Gand and Sri Lanka has access to
00:04:01
which I think is really exciting and
00:04:02
interesting also really a big deal for
00:04:04
Education well you've brought me five
00:04:06
questions but let me go one at a time so
00:04:08
I'm getting excited here um obviously
00:04:10
you spent time thinking about Wharton
00:04:12
interactive so this idea of
00:04:14
democratizing Education through
00:04:16
something like this has to be really
00:04:18
thrilling and exciting to you as an
00:04:20
educator and a scholar because I know
00:04:22
that was part of your mission and still
00:04:24
is with Wharton interactive yeah so for
00:04:26
those who don't know Wharton interactive
00:04:27
is our attempt to build games and
00:04:29
simulations to teach entrepreneurship at
00:04:30
scale and Wharton's been incredibly
00:04:32
supportive we have you've been
00:04:33
incredibly supportive we had we've have
00:04:35
built these very large games we have a
00:04:36
team of people we have writers and
00:04:38
coders and you know interactive fiction
00:04:40
experts and you know once gbd4 came out
00:04:42
we just tried the little experiment we
00:04:44
tried saying like what if we just write
00:04:46
a paragraph create a simulation of a
00:04:48
negotiation give me grading on it make
00:04:50
it realistic and 80% of the way there
00:04:52
with a paragraph like gbd4 just runs a
00:04:54
simulation so we have pivoted now all of
00:04:55
our simulations are basically AIS
00:04:57
powering every we have ai watching ing
00:04:59
Ai and AIS that are instructors and AIS
00:05:01
that are mentors all interacting with
00:05:03
each other that are actually doing
00:05:04
teaching right so so is it writing the
00:05:06
code doesn't it doesn't even need to
00:05:08
write the code like it is writing the
00:05:09
code but that turns out to be secondary
00:05:11
it's writing the code it's creating the
00:05:12
images it's doing all that stuff but
00:05:14
what it really is is also the brain's
00:05:15
the operation if we do good prompting we
00:05:17
can tell it here's your goal make sure
00:05:19
that you're keeping students engaged
00:05:21
change Tone If you need to here's your
00:05:23
overall and it just does it all I'm just
00:05:27
in shock because wow so let me ask you a
00:05:30
question how does one become I know this
00:05:32
is part of your video series how does
00:05:34
one be I don't know if called an expert
00:05:36
but how does one become sophisticated is
00:05:38
a good word in prompt engineering like
00:05:40
could I do this be do I need subject
00:05:43
matter expertise to create specific
00:05:47
enough prompts or is it just by as you
00:05:49
and your wife talked about in the video
00:05:51
series you'll just learn by doing yeah
00:05:54
so it's a really good question and sort
00:05:56
of all of the above so a few things one
00:05:58
is for those who don't know prompt
00:05:59
engineering right is the idea of writing
00:06:01
really good prompts to AI it is going to
00:06:02
go away uh there's not a I talk to open
00:06:04
AI regularly talk to Microsoft talk to
00:06:06
Google nobody who's insiders thinks this
00:06:08
going to last because the AI is really
00:06:09
good at intent if you say I want to
00:06:11
write a novel fairly soon it'll just be
00:06:13
able to say like okay here's let's go
00:06:14
through the steps together it already
00:06:15
kind of does that right so it's you can
00:06:17
get 80% of the way there by just
00:06:19
interacting with the AI now there is an
00:06:21
exception if you want to encode your
00:06:22
expertise as a subject maner expert you
00:06:24
want this to do a really good you know
00:06:25
marketing analysis it will do a
00:06:27
perfectly fine job but if you en code
00:06:29
your expertise into it by saying here's
00:06:31
the angle you should take here's the
00:06:32
approach do a little bit of prompt
00:06:33
engineering you could give that prompt
00:06:35
to anyone and they'll get the benefits
00:06:36
almost of your experience or wisdom so
00:06:38
there is some value in doing that uh
00:06:40
it's pretty straightforward most of it's
00:06:42
using it 10 hours of the frontier model
00:06:44
is my minimum rule of thumb but then
00:06:46
beyond that piece there's a couple
00:06:48
simple tricks so one is you tell the AI
00:06:50
who it is you give it context so the
00:06:51
more context the better you are an
00:06:53
expert marketer and weirdly by the way
00:06:55
there is a research now suggesting that
00:06:56
when you tell the AI is an expert under
00:06:57
some circumstances it works better um
00:06:59
again it's there's a lot of strange
00:07:01
stuff about prompting the second thing
00:07:02
you want to do is provide a lot it's
00:07:04
called fuse shot you want to provide a
00:07:05
lot of examples so when you're sending
00:07:07
that you want to say here's some
00:07:07
examples of the kind of report that
00:07:09
you'll produce and the third thing you
00:07:11
want to do is have it do step-by-step
00:07:13
thinking so you want to say first do
00:07:15
this then do that then do this because
00:07:16
it kind of only knows what it writes so
00:07:18
you want to write stuff out and then go
00:07:20
back to it and then build a plan from
00:07:21
there those three things will make you a
00:07:23
better prompt engineer but it's not
00:07:25
going to be that important in the long
00:07:26
term all right so let's go back to the
00:07:28
focus of AI and education so let's talk
00:07:30
about the roles that we have as
00:07:31
Educators you already said the
00:07:34
traditional way of doing homework let's
00:07:36
start with homework is in Jeopardy so
00:07:38
given that what can we do or in my view
00:07:42
I've already I'm teaching next semester
00:07:44
I'm like use chat GPT as a matter of
00:07:47
fact if you know how to do this to solve
00:07:49
the problems I'm asking you to do that's
00:07:51
a skill set or should I be thinking
00:07:53
about this differently then I want to go
00:07:54
to standing in the classroom then I want
00:07:56
to go to other forms of assessment and
00:07:59
things that we do yeah well so the
00:08:01
problem is what I feel is everybody's
00:08:03
rushing to do a chat GPD class right and
00:08:05
then the answer is like yes chat can do
00:08:06
that I mean there are very few things
00:08:08
that um at a sort of medium level that
00:08:11
chat like at the 80th percentile that
00:08:13
chat doesn't do reasonably well right
00:08:14
now so the question is do we want all of
00:08:16
our classes to be can you use chat CP to
00:08:18
solve this problem I think we still want
00:08:20
to teach the subjects that are really
00:08:21
good at teaching we still think people
00:08:23
need to learn these things which means
00:08:24
we have to adjust right uh you can't use
00:08:26
by the way everybody should know do not
00:08:28
use any kind of um AI detector they do
00:08:31
not work they're biased against people
00:08:33
use English and second language they all
00:08:34
whole five you know that that ship is
00:08:36
sailed we cannot detect AI okay so just
00:08:38
to be clear just like when we we have
00:08:41
canvas at the University of Pennsylvania
00:08:43
there's a turn it in which is so
00:08:45
whatever version of that for AI you
00:08:48
might as well just forget it no it
00:08:49
doesn't mean but well this isn't look I
00:08:51
think you can see where my next question
00:08:52
is going let's say Eric bradow and Ethan
00:08:54
mik are both in a class and they both
00:08:57
use chat GPT for to solve some problem
00:09:01
and let's say by chance they happen to
00:09:02
both put in the same prompt will they
00:09:05
get the same exact text back and if the
00:09:08
answer is no if both of those were
00:09:10
turned in could canvases turn it in not
00:09:13
say it was AI generated but would it say
00:09:15
hey wait a second there's cheating going
00:09:17
on there because their responses are so
00:09:18
similar so really good question a few
00:09:20
things first of all they wouldn't get
00:09:21
the same answer because there's
00:09:22
Randomness built into this a temperature
00:09:24
right so there's a r there's a random
00:09:26
seed initially and then the words are a
00:09:27
little bit randomly different which IES
00:09:29
over time you're saying large language
00:09:31
models are prob I'm a statistician are
00:09:33
are they're probabilistic models which
00:09:35
means even if we put in the exact same
00:09:38
prompt we're going to get out things
00:09:39
because there's a probability of the
00:09:41
next word or the next phrase and they're
00:09:43
autoaggressive so once they head into
00:09:44
one direction or another they sort of
00:09:46
spin off in that direction further very
00:09:47
interesting so that's the first thing
00:09:49
right the second so the the second thing
00:09:51
is I have had an assignment even before
00:09:53
chat GPD came out using gpt3 where I had
00:09:55
my students cheat in class so I had them
00:09:57
write the best essay they could part
00:09:59
part of the assignment is you have to
00:09:59
prompt it at least five times by the
00:10:01
time youve prompted AI two or three or
00:10:03
four times there's no way that they seem
00:10:04
similar anymore right if you give it so
00:10:07
yes if people just pasted in the
00:10:08
question right they're not going to be
00:10:10
the same answer but they might have some
00:10:12
similarities if they do any work like
00:10:13
make this more Vivid or here's my
00:10:15
writing style that's all they need to
00:10:17
make it very different turn in will not
00:10:20
detect those things and by the way I
00:10:21
really think it's unethical to use turn
00:10:23
in right now you should be turning it
00:10:24
off right like it is it is it it will
00:10:27
falsely has a high f accusation rate and
00:10:29
also by the way even worse is to ask GPD
00:10:31
4 or chat GPD free the 3.5 free version
00:10:34
whether cly was created by AI um new
00:10:37
studies showed GPD 4 has a 95% rate of
00:10:39
just telling you that something's made
00:10:41
by AI if you paste it in and ask if it's
00:10:42
made by Ai and GPD 3.5 has like a 5%
00:10:45
rate of telling you they just randomly
00:10:46
say this is made by AI or not they have
00:10:48
no way of telling so what kind of things
00:10:51
can be you know what when you presented
00:10:54
to the Wharton faculty which was one of
00:10:56
the best most informative presentations
00:10:58
I've seen the a long time when you
00:11:00
presented to the Wharton faculty I was
00:11:02
like okay maybe at the time this was
00:11:04
true like all right so maybe it's not
00:11:06
text data but maybe I'll give exam
00:11:09
questions that have video or maybe I'll
00:11:12
give stuff that has voice because you
00:11:13
know what chat GPT can't possibly do as
00:11:16
well with that am I off base or well let
00:11:19
me just say I might be correct but it's
00:11:22
better than you think no it's it's
00:11:24
better at voice than humans are right
00:11:26
now so whisper which is the free uh G
00:11:29
built in the chat GPT app uh probably
00:11:32
illegally trained not illegally I don't
00:11:34
I don't know who's watching but trained
00:11:35
on uh on YouTube videos probably as far
00:11:37
as we can tell has better than human
00:11:39
hearing so like accents um you know
00:11:42
mixes of languages I use it all the time
00:11:44
when actually my students pitch to it um
00:11:46
I have real venture capitalist and then
00:11:47
I have the AI playing a VC the VCS think
00:11:50
that the AI does a better job than they
00:11:51
do in giving feedback um so yes it can
00:11:53
listen um they and now can see things so
00:11:56
any visual problem you just upload and
00:11:58
it will address a video or anything so
00:12:00
video still has a little bit of trouble
00:12:01
with so that you can just pull off video
00:12:03
right now um but give it give it give it
00:12:06
a few days no I mean a couple months
00:12:08
probably well you actually wrote up
00:12:09
another topic and we'll we've been
00:12:11
talking about Ai and education again I'm
00:12:13
joined by my friend and colleague Ethan
00:12:15
mik he's the Ralph J Roberts
00:12:16
distinguished faculty scholar a
00:12:18
professor in our Management Department
00:12:19
also the academic director of Wharton
00:12:21
interactive and clearly I think it's
00:12:23
fair to say one of the leading Scholars
00:12:24
on artificial intelligence both in
00:12:26
education and the workforce um could you
00:12:29
talk to us about the work that you've
00:12:30
done on AI and the workforce cuz I know
00:12:32
you're extremely proud of the work
00:12:34
you're doing and I just want to see
00:12:36
because a lot of our you know uh experts
00:12:38
have talked about it's can't be AI or
00:12:41
humans it's got to be Ai and humans I'm
00:12:43
just interested in the angle in which
00:12:44
you've studied AI in the workforce yeah
00:12:46
s okay so um one example we have a I
00:12:49
have a paper with um a whole bunch of
00:12:51
great people at Harvard including kareim
00:12:52
Lani uh frbo dequa um and people at MIT
00:12:57
Kate Kate Kellogg a whole bunch of
00:12:59
people on this project but what we did
00:13:00
was we went to BCG right one of the
00:13:02
three big Elite Consulting companies um
00:13:04
and a lot of our Boston Consulting Group
00:13:06
a lot of our students want of work there
00:13:08
a lot of alumni work there and we did an
00:13:09
experiment we uh created 20 tasks they
00:13:12
were all realistic tasks with BCG
00:13:13
they're actual tasks they use uh and we
00:13:16
gave some eight we used 8% of the global
00:13:18
Workforce which is a lot and Al So when
00:13:20
you say an experiment you mean an actual
00:13:21
experiment I mean an actual experiment
00:13:22
8% of their Global Workforce and some of
00:13:24
them got the help of gbt 4 and some did
00:13:26
not and there were a bunch of other
00:13:27
conditions the people who were given GPT
00:13:29
40 use in business house had a 40%
00:13:31
Improvement in quality no training no
00:13:33
specialization in the mod just the same
00:13:35
chat GPT all of us have access to 40%
00:13:38
increase in quality across 108
00:13:39
regressions how was quality measured
00:13:42
every way we could so we did okay
00:13:43
analytical tasks and marketing task and
00:13:45
persuasion T and all of them were graded
00:13:47
by human phds human NBAs and then we
00:13:49
also used gp4 which by the way grades
00:13:51
just as well as humans do it just is a
00:13:53
little nicer on the scores but the
00:13:54
relative scores were exactly the same
00:13:56
and then they completed tasks 26% faster
00:13:58
got 12 point Sorry 26% more tasks done
00:14:00
12.5% faster no training nothing
00:14:03
training like we only had like 5 minutes
00:14:05
of training for sub conditions others
00:14:06
for none just to put that in context
00:14:09
when steam power was put into a factory
00:14:11
in the early 1800s and improved
00:14:13
performance by 18 to 22% we've never
00:14:14
seen a 40% Improvement this is not tuned
00:14:17
this is not train this is the chat
00:14:19
interface that you're used to using um
00:14:21
so huge huge performance impacts on on
00:14:24
from just a little experiment all right
00:14:26
so usually in academic papers we have
00:14:27
some thesis hypothesis and in your case
00:14:29
you have an experiment in results what
00:14:32
did the back end of that paper look like
00:14:34
so let's imagine you're now Consulting
00:14:36
for a company BCG or you're Consulting
00:14:40
to our students undergrads mbas like
00:14:42
this should be how you think about your
00:14:44
training what did the back end of the
00:14:46
paper like what conclusions did you come
00:14:48
to as a result of this well we barely
00:14:51
there's so much else we could talk about
00:14:52
here too that are you know interesting
00:14:53
caveats on creativity and who who uses
00:14:56
what answers but let let's and and also
00:14:58
how people people work with AI right
00:14:59
that's another thing I've been doing a
00:15:00
lot of work on uh effectively but um I
00:15:03
mean the back end of the paper is really
00:15:05
the idea that like look this is a big
00:15:07
enough impact that this should be a red
00:15:09
alert everywhere in every organization
00:15:11
right you don't see these kind of
00:15:12
performance improvements a lot of people
00:15:14
are taking their time on AI they're
00:15:15
assisting that they do something like
00:15:17
you know integrate their own data with
00:15:19
the AI system we didn't have to do that
00:15:20
here like the AI the p and GPT stands
00:15:22
for pre-trained it knows a lot of stuff
00:15:24
already it's not clear that you should
00:15:26
be waiting to build a large data
00:15:27
integration use r and all these other
00:15:29
techniques when you should just probably
00:15:30
be using this and it should be a red
00:15:32
alert to figure out how to use this
00:15:34
because you know as much as we say oh
00:15:36
it's people using AI but also I mean
00:15:38
another side of this paper was there's
00:15:40
appendix C to the paper that I don't
00:15:41
always talk about because it's I don't
00:15:43
quite know what to do with it but it
00:15:44
measures uh what's called retainment how
00:15:46
much of gp4 is answer that you just use
00:15:48
as your answer and there's almost a
00:15:50
direct correlation between how much of
00:15:51
the answer you use and how successful
00:15:53
your results are I mean there's a direct
00:15:54
correlation it's almost perfect
00:15:55
correlation so basically the only way to
00:15:57
mess up was to changed chat gpt's answer
00:16:00
H and not only that the performance
00:16:02
boost was at the largest for everyone in
00:16:05
the bottom half of performance so we
00:16:06
measured prior and after performance 42%
00:16:08
boost and improvement from the bottom
00:16:10
half 18% for the top half performance it
00:16:12
leveled everybody up to like the 8th
00:16:13
percentile of BCG Consultants like I
00:16:16
don't even know what to do with that
00:16:17
that's such a big number so given as you
00:16:19
said this is more impactful than the
00:16:22
steam engine what what did BCG now do
00:16:25
with this like what's their planned I
00:16:27
mean is there any let me just say Do
00:16:29
they have any doubt that what you found
00:16:32
is generalizable like maybe here's an
00:16:34
argument I'll be a statistician for a
00:16:36
moment maybe this wasn't Al you said 8%
00:16:38
of the workforce maybe this wasn't a
00:16:40
massive sample size maybe it works for
00:16:42
these 20 tasks but not for these tasks
00:16:44
or maybe there's some sort of you know
00:16:47
maybe it helps in the short run but you
00:16:49
know what we can also train humans so
00:16:51
maybe the effectiveness is going to
00:16:52
decrease over time I'm just playing a
00:16:54
like if I were a reviewer on a paper I'm
00:16:56
just playing The Devil's Advocate what
00:16:57
would be the respon to all of this so a
00:16:59
few things there we did manage to create
00:17:01
one task the AI couldn't do and right so
00:17:04
one of the things to know about our
00:17:05
listeners here how want to hear what
00:17:06
that is so well it was hard right it was
00:17:09
a task where we had to hide data in
00:17:10
interviews and some was in spreadsheets
00:17:12
and this was before Ada the Advanced
00:17:14
Data analysis module came out but we
00:17:15
managed to find something right um and
00:17:17
it took some work and then on that task
00:17:20
you what happened was people who used AI
00:17:21
did worse because they were mistaken
00:17:23
because they took in what they was do so
00:17:24
there so part of what people are saying
00:17:26
is like well what's the border of what
00:17:27
AI does and doesn't we call that the
00:17:29
jagged Frontier like for example if you
00:17:31
ask gbd4 to write a 25-word paragraph
00:17:33
it'll have trouble doing that because it
00:17:35
doesn't see words it sees tokens but if
00:17:37
you ask it to write a sonnet it'll do an
00:17:38
amazing job of that sonnets are harder
00:17:40
for humans than 25 words you have to
00:17:42
learn the frontiers of AI going back to
00:17:44
the point we mentioned earlier if you
00:17:46
use it a lot that's how you start to
00:17:47
understand it's going to be good at this
00:17:48
task bad another task so to go back to
00:17:50
the overall kind of question right about
00:17:52
what do you do with this right is it
00:17:53
generalizable this is just one piece of
00:17:55
result there's another study out MIT
00:17:57
that got published in science that shows
00:17:59
similar size improvements in business
00:18:01
writing tasks in a completely different
00:18:02
sample there's a study um out of GitHub
00:18:05
showing the same kind of improvement for
00:18:06
programmers like the 30 to 70% number
00:18:09
just keeps coming up over and over again
00:18:11
in different samples and different
00:18:13
there's a piece on Creative work there's
00:18:14
another paper U out of Harvard looking
00:18:17
at um at you know business implications
00:18:19
uh sorry business um proposal writing
00:18:21
there's our own colleague Christian turt
00:18:24
Carl erck and and their colleagues work
00:18:26
showing Innovation so this is not like a
00:18:28
onetime thing you know this is a pretty
00:18:30
broad-based set of findings so where do
00:18:32
you think the I I like the word Jagged
00:18:35
Edge yeah where is Jagged Frontier where
00:18:38
is the jagged Frontier here I mean do
00:18:40
you even I mean you probably know more
00:18:42
than let's say you're in the one 1000
00:18:44
upper percentile of people in knowledge
00:18:46
about GPT or AI in general right now
00:18:49
where do you think that Jagged Frontier
00:18:51
is so it's hard to explain right which
00:18:53
is part of why this is as much doesn't
00:18:55
it always move out that is that's the
00:18:57
main thing is the frer is moving out
00:18:59
like I guarantee in the next month the
00:19:01
frontier is going to move out right like
00:19:03
some for some reasons I know I can't
00:19:04
talk about some reasons that are already
00:19:05
public elsewhere but like this is not
00:19:08
stopping there's no indication to me
00:19:09
that the jagged Frontier is not going to
00:19:11
keep moving forward in the next few
00:19:13
months right next year or two and I
00:19:14
think ultimately the big question is how
00:19:17
far how fast and when does it stop and I
00:19:19
will tell you the people training the AI
00:19:20
models I don't think have an answer to
00:19:21
that question so let's start with a few
00:19:23
things um how much I I saw I think it
00:19:26
was yesterday President Biden sign
00:19:28
signed a bill or something an executive
00:19:30
order on artificial intelligence and
00:19:33
some sort of safety security protection
00:19:35
could you tell our listeners what is
00:19:37
that about like what is what is the
00:19:38
policy trying to do so there there's a
00:19:41
few things in the policy right what I I
00:19:44
I I have not spent a huge amount of time
00:19:45
in the executive order but what it does
00:19:47
that I like about is there's there's
00:19:49
sort of two stages of threats from AI
00:19:50
that people are worried about one is you
00:19:52
may have hear if you read the press a
00:19:53
lot you may hear about Extinction risk
00:19:55
right what if we make AGI artificial
00:19:56
intelligence smarter than a human and
00:19:58
what does it do to us does it save us
00:20:00
does it kill us what happens if we build
00:20:01
a machine God and by the way that's like
00:20:02
the stated goal of open a AI is to build
00:20:05
AGI right that's their plan by the way
00:20:07
just so I know I I just want to be sure
00:20:09
since I'm a big movie guy I don't know
00:20:10
how much you watch movies like wasn't
00:20:12
that some part of Terminator like in
00:20:15
other words these robots became so smart
00:20:17
that in some sense they ended up
00:20:18
launching Wars and I mean the
00:20:20
schwarzeneger movie ter that is it's not
00:20:23
unrelated right no that that is one
00:20:25
example of AGI right um the people who
00:20:28
are in favor of AGI think that this will
00:20:30
save us all and you know and redeem
00:20:31
humanity and give us all eternal life
00:20:33
the people who uh don't like it think
00:20:35
it'll murder us all so I think it's
00:20:36
worth worrying about like it's something
00:20:38
that enough serious people in computer
00:20:40
science are worried about that we should
00:20:41
I'm glad that we're addressing that but
00:20:42
I think the bigger policy implications
00:20:44
in the near term for me as somebody at
00:20:46
the Wharton School is look we've got
00:20:48
something that's doing high-end creative
00:20:50
work high-end managerial work it's going
00:20:52
to get better at this like we're
00:20:53
improvising a chat interface and using
00:20:55
that to do Consulting work like that's
00:20:57
pretty crazy so that means there's going
00:20:58
to be widespread implications for work
00:21:00
widespread implications for Education as
00:21:02
we've were talking about and also one of
00:21:04
the things that really is important like
00:21:05
the part of the reason we catch
00:21:07
criminals and like Bad actors so easily
00:21:09
especially when they're not connected to
00:21:09
like a state intelligence agency is most
00:21:11
of them aren't that great right now what
00:21:13
happens if the AI makes brings everybody
00:21:15
up to 80th percentile in biological
00:21:17
engineering 80th percentile in building
00:21:19
chemical weapons that's also a concern
00:21:22
and then there's privacy concerns deep
00:21:23
fakes are perfect from this thing right
00:21:24
like so can you define what a deep fake
00:21:27
is like what does that term mean sure I
00:21:29
mean AI um content is basically
00:21:31
undetectable I have made videos of my
00:21:33
fake me talking I I my presentations I
00:21:35
always have one real picture everything
00:21:36
else I generate on my own no one could
00:21:38
tell what the real picture is right so
00:21:39
you cannot tell if I can create an actor
00:21:42
with their voice I can do this right now
00:21:43
for $150 with software that anyone can
00:21:46
use it's not even like dark web software
00:21:47
it's a company that's VC backed uh and I
00:21:49
can use 11 labs in did and create a fake
00:21:52
video of you talking right now and you
00:21:54
know it'd be pretty realistic so we have
00:21:56
this issue with this kind of deep fake
00:21:59
email it's also perfect fishing engine
00:22:01
like you shouldn't trust anything you
00:22:01
see online anymore and that's not a joke
00:22:03
like there literally is no way to know
00:22:05
like I've already talked to Banks who've
00:22:06
gotten calls uh in from the voice of
00:22:08
people who weren't actually calling them
00:22:09
demanding money from ransoms like stuff
00:22:11
is going crazy already that is ship is
00:22:13
already staled so part of the Biden
00:22:15
agenda is like how do we Watermark these
00:22:16
things not going to be possible like
00:22:18
it's just that's that's not going to
00:22:20
happen really because even though these
00:22:21
Frontier models are based in the US
00:22:23
there's a whole bunch of Open Source
00:22:24
models worldwide models that are not
00:22:26
going to have this kind of protection so
00:22:28
the the the attempt of this executive
00:22:30
order for the way I see it is both to do
00:22:32
worrying about this sort of future Ai
00:22:34
and training it but also trying to think
00:22:35
about how do we restore privacy how do
00:22:37
we restore you know and I don't know how
00:22:39
much of that's possible but the
00:22:40
regulation is probably needed so a lot
00:22:43
of our listeners are probably sitting
00:22:44
here saying I need to get started like I
00:22:47
I I haven't even started but now clearly
00:22:49
if I'm listening to Professor mik I've
00:22:51
got to start now where do you suggest
00:22:53
that someone starts like you mentioned
00:22:55
10 hours let's say that's good what what
00:22:58
should they spend their 10 hours doing
00:23:00
like where should they go you know we're
00:23:02
at Wharton so we're fortunate we have
00:23:04
the video series you sent around but I
00:23:06
mean whether it's you have materials
00:23:08
others have materials can you go I'm
00:23:09
making it up can you go to KH Academy I
00:23:11
I don't know where did where does
00:23:12
someone go to get started so a few
00:23:14
things those videos are on YouTube
00:23:15
anyone can see them if you search the
00:23:17
name of ethol you'll find it I also have
00:23:19
a substack with a whole bunch of getting
00:23:20
started guides called one useful thing
00:23:22
all which I just which I just started
00:23:24
following today excellent hopefully
00:23:26
you'll enjoy that but I mean I think the
00:23:27
B BS of this are my my principles of AI
00:23:29
my first principle of using AI is just
00:23:31
invited to everything you morally and
00:23:33
legally can use it for everything use it
00:23:35
for your job like literally you want to
00:23:36
send an email see how good the AI is
00:23:38
writing the email you have to do
00:23:39
ideation you know have it do ideation
00:23:41
you going into a meeting bring it to the
00:23:42
meeting have it record the meeting and
00:23:44
give you advice and feedback on what you
00:23:45
should do better next time just use it
00:23:47
for everything that's the only way to
00:23:48
figure out what the jagged Frontier is
00:23:49
in your field there nobody knows
00:23:52
anything right now right like as I said
00:23:53
I talk to all of the major AI companies
00:23:55
on a regular basis and like no one has
00:23:57
an instruction manual for this thing no
00:23:58
one knows whether it's going to be good
00:24:00
or bad in your subfield whoever you're
00:24:01
listening at right now so you can be the
00:24:03
world expert by just using it and seeing
00:24:05
what that is so just try it for your
00:24:07
work and then there's a bunch of
00:24:08
techniques you'll start to learn but the
00:24:09
first thing is to try it it's really
00:24:10
important though to use a Frontier Model
00:24:12
to use the most advanced model available
00:24:14
to you right now now you mentioned
00:24:15
something else about you think that I
00:24:17
think let me see if I got this right
00:24:19
that I think it was Google you I think
00:24:21
you use the term like they're coming out
00:24:23
with something new that might be better
00:24:26
than chat GPT what would better mean in
00:24:29
like when you use the word better I was
00:24:31
just intrigued by that what does it mean
00:24:33
to be better than one you know uh large
00:24:36
language model Etc being better than
00:24:37
another okay so there's a lot of
00:24:39
interesting angles to that right now all
00:24:41
the major front the two major Frontier
00:24:43
models which are open ai's model which
00:24:45
sort of Microsoft Powers it as well and
00:24:48
Google's model both are have added a
00:24:50
whole bunch of capabilities that if
00:24:51
you're not paying attention you may have
00:24:52
missed so they're all fully multimodal
00:24:54
you can ask them to create pictures they
00:24:56
they also can see the world right so
00:24:58
like not doing an image search but like
00:24:59
you literally could show a picture and
00:25:00
say how does this dress fit and it will
00:25:02
give you reasonable feedback on that or
00:25:04
how do you know what how do I undo this
00:25:05
lock or what's this passcode whatever
00:25:07
you want to do um so they're all
00:25:09
multimodal they all are going to be
00:25:10
doing voice back and forth they all
00:25:12
connect to other do can read documents
00:25:14
connect to other materials so that's
00:25:15
kind of the basics all of that is
00:25:17
connected to the large language model
00:25:18
itself which you can kind of think of as
00:25:19
the brain and so large language models
00:25:22
basically get smarter over time so if we
00:25:23
think about uh GPT 3.5 the free version
00:25:25
you're using maybe High School sophomore
00:25:27
I would say GPD 4 at its best moments is
00:25:30
is a first year grad student so part of
00:25:32
the question yeah so part of the
00:25:33
question is what is two or four times
00:25:35
better than that look like we don't know
00:25:36
yet so we could just do raw test scores
00:25:38
right we go from scoring at the you know
00:25:40
at the 5ifth perc of the bar exam
00:25:43
beating 5% of humans for free chat GPD
00:25:45
to beating 95% of humans for GPD 4 what
00:25:48
happens with the next one we we don't
00:25:50
really know but so smarts are sort of
00:25:51
the smarts of the brains behind the
00:25:53
whole thing so this is the question I've
00:25:55
asked everyone in this series uh to kind
00:25:57
of end the episode if we're sitting here
00:26:00
10 years from now and we'll make a date
00:26:02
I'm going to interview this I hope
00:26:03
interview the same group of people 10
00:26:05
years from now what are we talking about
00:26:07
do you think that has happened over the
00:26:09
previous 10 years I can only think it's
00:26:11
scenarios at this point right I mean
00:26:13
because the the only question that kind
00:26:15
of matters for this is how fast will
00:26:18
these models improve and when will they
00:26:20
hit their limits right and nobody knows
00:26:21
the answer to those questions right so
00:26:24
it's not going to be static if you think
00:26:25
you have time to wait you don't because
00:26:27
these model are advancing very rapidly
00:26:29
the question is do they stop at the 95th
00:26:31
percentile of the best humans in one
00:26:33
area like so everybody's got something
00:26:34
they're really good at that they
00:26:35
definitely beat the AI in 99th
00:26:37
percentile better than human we don't
00:26:39
know right and so to me that's the only
00:26:42
relevant question right and so nobody
00:26:44
has the answer to that so we have to
00:26:46
prepare for a scenario where okay we're
00:26:48
getting close to the top I don't I
00:26:49
haven't seen evidence of this but it's
00:26:50
entirely possible that it starts to slow
00:26:52
down then we still have at least 10 or
00:26:54
15 years of absorbing what gbd4 can do
00:26:56
because it's barely connected to
00:26:57
anything anything in the world right
00:26:58
we've got 10 years of disruption ahead
00:27:00
of us that's going to roll ahead anyway
00:27:02
if they keep getting better then we
00:27:03
start to think seriously about if not
00:27:05
AGI what does it mean that it beats
00:27:07
every human at you know writing
00:27:08
marketing copy like what do we do with
00:27:10
that right do we you know and so there's
00:27:12
a lot of open questions so I don't have
00:27:14
the easy answers but I think that there
00:27:16
is I think we're you're more likely to
00:27:17
see a transformed World in 10 years than
00:27:19
in five even if the technology stops
00:27:21
because it takes a while for systems to
00:27:22
absorb change right the the the futurist
00:27:24
rule is that everyone overestimates
00:27:27
short-term change it underestimates
00:27:28
long-term change I think 10 years we're
00:27:30
going to see a transformed World in a
00:27:31
lot of ways that are some are good some
00:27:33
are bad well I'd like to thank uh
00:27:35
Professor Ethan mik for joining me today
00:27:37
on the podcast series on AI in education
00:27:39
and the workforce uh Ethan's an
00:27:40
associate professor of management he's
00:27:42
also an academic director of whorton
00:27:43
interactive and as he said you can go to
00:27:45
your favorite engine and type in Ethan
00:27:47
mik and you can see about his substack
00:27:49
and about his videos on YouTube Ethan
00:27:51
thank you for joining me thank you for
00:27:52
having
00:27:56
me
