00:00:01
we're speaking today with Justin got
00:00:03
Schlick who leads the Machine
00:00:05
programming research team at Intel labs
00:00:08
the newly formed research group focuses
00:00:11
on the promise of machine programming
00:00:14
which is a fusion of machine learning
00:00:16
formal methods programming languages
00:00:19
compilers and computer systems Justin
00:00:22
welcome to knowledge at wharton thank
00:00:24
you so much for being with us here today
00:00:26
wonderful thank you for having me McCool
00:00:28
so I just read that description of what
00:00:31
machine programming is but I think that
00:00:34
you know given all the buzz around AI a
00:00:37
lot of people are familiar with machine
00:00:39
learning but most of them like me don't
00:00:42
have a clue what machine programming
00:00:44
means perhaps you could explain the
00:00:46
difference between the two certainly
00:00:48
yeah so at the very highest level
00:00:51
machine learning can be considered
00:00:53
within the subset of artificial
00:00:55
intelligence and there's many different
00:00:58
types of machine learning techniques one
00:01:01
of the most prominent right now are
00:01:02
these things called deep neural networks
00:01:04
and that's a lot of what people are
00:01:08
using to make the tremendous progress
00:01:09
that we're seeing over the last decade
00:01:11
the machine programming is really the
00:01:15
idea that we are trying to automate the
00:01:17
development and maintenance of software
00:01:19
so the fundamental difference between
00:01:22
the two is that with machine programming
00:01:25
you can kind of think of all of the
00:01:28
field of machine learning being a subset
00:01:30
of the field of machine programming but
00:01:34
in addition to using machine learning
00:01:36
techniques which are these approximate
00:01:38
types of solutions we'll also use other
00:01:40
things like formal program synthesis
00:01:43
techniques that will give us
00:01:45
mathematically proven correct pieces of
00:01:48
software and then between those two
00:01:51
points you can kind of think of those as
00:01:52
a spectrum you have the approximate
00:01:54
solutions here and the precise solutions
00:01:56
here and then there's a fusion of a
00:01:58
number of different ways that you can
00:02:00
combine these and every one of these
00:02:02
things essentially is a part of the
00:02:04
bigger landscape of machine programming
00:02:07
so if I understood you right machine
00:02:11
programming is when you create software
00:02:14
that that can create more software right
00:02:18
how would that happen and I wonder if
00:02:20
you could give us a couple of examples
00:02:21
just to help our audience understand
00:02:24
this yeah so the idea of creating
00:02:29
software that creates its own software
00:02:31
is really the core idea between or
00:02:34
inside of machine programming and a
00:02:37
couple examples of that is we recently
00:02:39
built a system using genetic algorithms
00:02:41
and what it does is it'll allow you to
00:02:45
take certain input output examples and
00:02:48
then by running through a number of
00:02:50
iterations we call them evolutions in
00:02:52
the genic algorithm space it will then
00:02:55
automatically synthesize the program
00:02:57
that will match the input and output you
00:03:01
do this sort of in the training phase
00:03:02
and then it will take new input-output
00:03:05
examples that it's never seen before and
00:03:07
then generate these new particular types
00:03:10
of programs so that's like one example
00:03:12
that you might have in the space of
00:03:15
machine programming and you would think
00:03:18
about the impact that machine
00:03:19
programming might have on different
00:03:21
industries which industries do you think
00:03:24
are likely to be affected most by this
00:03:26
and over what period of time yeah this
00:03:30
so this is a fantastic question and one
00:03:33
that could require a very long response
00:03:36
but I'll try to keep it slightly
00:03:37
abbreviated at the highest level one
00:03:40
could imagine that any of the industries
00:03:42
that are predominantly based in software
00:03:44
are going to benefit this benefit from
00:03:46
this tremendously there was a recent
00:03:48
survey that was done in earlier 2019
00:03:52
that showed I think we have something
00:03:56
like half a million computer scientist
00:03:58
positions that are open so these are
00:04:01
programming positions in industry that
00:04:03
we need to fill but we're only producing
00:04:05
roughly 10% of the actual programmers to
00:04:10
fill those roles so what we're having in
00:04:13
the software industry is essentially a
00:04:14
bottleneck of supply if we can start to
00:04:19
automate some of the simple tasks you
00:04:23
know reading in a file parsing data
00:04:25
helping us automate that
00:04:27
software development testing this will I
00:04:31
think tremendously accelerates the rate
00:04:33
at which software is being developed so
00:04:36
I would say that that's probably the
00:04:38
first very obvious place the other area
00:04:42
that I think is going to be impacted
00:04:44
tremendously by this are autonomous
00:04:45
systems in most of the spaces in
00:04:49
autonomous systems a core ingredient of
00:04:51
those systems is software so for example
00:04:54
if one were to think about autonomous
00:04:56
vehicles a large part of what's holding
00:04:59
us back from getting to level 4 level 5
00:05:03
autonomy which is the point where the
00:05:05
car can essentially handle all of the
00:05:07
nuance behaviors of driving in downtown
00:05:09
Philly or something of that nature a big
00:05:11
bottleneck of that is really the
00:05:15
implementation and the algorithms of the
00:05:17
machine learning systems if we can
00:05:19
automatically construct those these
00:05:22
autonomous systems will also accelerate
00:05:24
in their advancement I'd like to come
00:05:27
back in a little bit to the question
00:05:28
about autonomous vehicles and the impact
00:05:31
on the auto industry but given the fact
00:05:33
that a lot of our audience is in the
00:05:35
financial services industry I wonder if
00:05:38
we could go a little deep into that so I
00:05:41
mean I know that AI has made quite a
00:05:42
significant impact for example in areas
00:05:45
like fraud detection right do you think
00:05:48
machine programming can also have a
00:05:50
major impact there and if so what what
00:05:52
might that look like absolutely this is
00:05:56
a fantastic question and as I was
00:05:58
mentioning earlier with machine learning
00:05:59
systems their foundation is essentially
00:06:04
learning through statistical analysis in
00:06:06
some sense this makes them probabilistic
00:06:08
that they are getting us close to the
00:06:12
the right answer but there's in many
00:06:15
cases we're not guaranteed that we'll
00:06:18
have the right answer when we think
00:06:20
about machine programming we go back to
00:06:22
that spectrum of how we have
00:06:23
probabilistic to very precise solutions
00:06:26
it's my belief that for example in the
00:06:29
financial sector there are certain cases
00:06:31
where probabilistic solutions aren't
00:06:33
sufficient so one might speculate that
00:06:36
as you're doing some sort of financial
00:06:38
transaction
00:06:39
having the probability that oh i sort of
00:06:41
been rounding off the sense right
00:06:43
you know i'm close enough probably isn't
00:06:46
sufficient if those transactions are
00:06:47
happening billions of times a day in
00:06:50
that case we need a more precise
00:06:53
solution and this is one of the areas
00:06:54
that i think you could use machine
00:06:56
programming an interesting coming back
00:06:59
to the other industry that you mentioned
00:07:01
and the auto industry what kind of
00:07:03
impact do you think the machine
00:07:06
programming will have on the whole drive
00:07:09
towards autonomy you i think you started
00:07:11
talking about that in the context of
00:07:13
philadelphia yeah but i wonder if you
00:07:15
could go a little more deep into what
00:07:17
that might look like
00:07:18
for sure so as i was mentioning earlier
00:07:21
we recently built a system that is using
00:07:24
this dis genetic algorithm to
00:07:27
automatically construct programs what i
00:07:29
didn't mention is one of the pieces
00:07:31
that's part of genetic algorithm just
00:07:34
this thing called a fitness function the
00:07:36
fitness function is essentially you can
00:07:38
think of it as a way that you grade the
00:07:41
accuracy of the programs or the results
00:07:43
that the genetic algorithm is giving so
00:07:45
genic algorithm produces a result and
00:07:47
the fitness function says you get a b or
00:07:50
you get an A historically though fitness
00:07:53
functions have been written by humans
00:07:55
and not just any human really expert
00:07:58
machine learning humans oftentimes what
00:08:01
we find is the complexity of the problem
00:08:04
you're trying to solve is directly
00:08:06
related to the complexity of the fitness
00:08:09
function so one could naturally infer
00:08:11
then why would you write the fitness
00:08:13
function just solve the problem yourself
00:08:15
so we took a look at this and what we
00:08:17
did is we figured out a way using
00:08:20
machine learning that we could
00:08:21
automatically create the fitness
00:08:24
function without a human involved so now
00:08:27
going back to your question if you think
00:08:29
about this type of thing in the
00:08:30
autonomous vehicle space one of the
00:08:32
things that's holding us back is the
00:08:34
advancement of ML systems and
00:08:36
historically the advancements that we've
00:08:38
had with ml systems have been through
00:08:40
humans creating them but if we use
00:08:43
machine programming one could imagine
00:08:45
like we have with the genetic algorithm
00:08:47
solution that the machine can actually
00:08:50
start to invent
00:08:51
it's own machine learning systems that
00:08:54
will then accelerate the progress of
00:08:57
these autonomous systems so what are the
00:08:59
implications of that well one of the
00:09:01
things that I've heard about that's
00:09:04
holding back autonomous systems as you
00:09:06
are the autonomous vehicles is the fact
00:09:11
that it might be too late to too far for
00:09:17
the system to make a certain decision
00:09:19
yeah and and you because you don't want
00:09:23
to actually hit something right you you
00:09:25
you you probably need software that can
00:09:28
predict what's about to happen before it
00:09:30
actually happens is that one of the
00:09:32
issues right here absolutely right so
00:09:35
this is tremendous insight and and
00:09:37
actually we had a nurbs paper nerps is
00:09:40
one of the leading research conferences
00:09:42
in machine learning we had a nurbs paper
00:09:45
in 2018 that tried to start to address
00:09:48
this problem so historically what you're
00:09:50
describing here is the space of anomaly
00:09:53
detection
00:09:54
so the Antonidas vehicle space when we
00:09:56
think about these various behaviors we
00:09:58
think this is an anomaly and in
00:10:00
particular that's a time series anomaly
00:10:02
so for example you're trying to prevent
00:10:04
this vehicle from colliding with this
00:10:06
other vehicle or make sure it doesn't
00:10:07
hit a pedestrian and as you pointed out
00:10:09
it's too late if you already have the
00:10:12
event happened to you know detect it so
00:10:15
what we did in order to address this is
00:10:18
we recreated the mathematical foundation
00:10:22
for anomaly detection specifically for
00:10:24
time series and so with this now what
00:10:27
our hope is is that the community will
00:10:29
adopt the new mathematical foundation
00:10:31
we've created and then they can apply
00:10:33
this for time series and Omni detectors
00:10:35
which will start to address those types
00:10:37
of problems so and machine programming
00:10:39
helps at all this absolutely so in the
00:10:42
context of autonomous vehicles you could
00:10:44
think of using this mathematical
00:10:45
foundation to better predict these
00:10:47
anomalies but when you think about
00:10:49
machine programming or you think perming
00:10:51
in general many of the problems we're
00:10:53
seeing today with software is we have
00:10:56
these correctness bugs we have security
00:11:00
bugs we have these privacy violations
00:11:02
all of these things in
00:11:04
some sense are time-series in nature
00:11:06
program is really just a sequence of
00:11:09
instructions one after another so if you
00:11:11
take that mathematical foundation you
00:11:14
can also apply it in the space of
00:11:16
machine programming which is exactly
00:11:17
what we're doing well one thing I'm
00:11:21
curious about is that machine
00:11:22
programming like a lot of other areas of
00:11:25
AI has been around since the 1950s right
00:11:27
what's behind you know the sudden
00:11:31
interest in machine programming now why
00:11:35
is it picking up in such a big way and
00:11:36
why is Intel you know so interested in
00:11:40
sort of investing it in in it in a big
00:11:42
way this is a fantastic question and
00:11:46
I'll try to break it down to those two
00:11:48
pieces first address why we're seeing
00:11:51
the you know the resurgence of this
00:11:54
because it has been around since the
00:11:55
1950s and then why is Intel so
00:11:58
interested if we look at why it's taking
00:12:02
off today I would say principally
00:12:03
there's two reasons the first is I
00:12:05
believe we're at an inflection point and
00:12:07
the second is I believe my colleagues
00:12:10
and I at Intel Labs and at MIT we've
00:12:13
made an important sort of observation
00:12:15
and how to think about the future of
00:12:17
machine programming so as far as the
00:12:19
inflection point goes we believe there's
00:12:21
really or my view is really that there's
00:12:23
three things that have created this the
00:12:26
first is we have tremendous advances in
00:12:29
algorithms in machine learning and in
00:12:32
formal methods things that didn't exist
00:12:35
say 12 months ago potentially are
00:12:39
fundamental to the advancement of
00:12:41
machine programming the second is we
00:12:45
have tremendous advances in compute
00:12:46
today as the recent Turing Award winners
00:12:50
Dave Patterson and John Hennessy how are
00:12:53
pointing out is we're living in sort of
00:12:55
what they call the Golden Age of
00:12:57
computing which they refer to as they
00:13:00
call them domain-specific architectures
00:13:02
that for a long time it was really just
00:13:04
the CPU but now we're seeing based on
00:13:07
the advances were having a machine
00:13:08
learning in other areas we have these
00:13:10
accelerators that are specific to these
00:13:12
domains and so it's creating a
00:13:15
tremendous opportunity for acceleration
00:13:17
of
00:13:18
machine learning and formal methods that
00:13:20
wasn't possible and then the third piece
00:13:23
is the abundance of big and dense data
00:13:26
for example there is a repository that's
00:13:30
called github and github basically as a
00:13:33
place where people store their software
00:13:35
and what we've seen by looking at github
00:13:38
is back in 2008 I think it had roughly
00:13:43
33,000 repositories 2009 when I looked
00:13:47
at it earlier this summer I think it was
00:13:50
somewhere around over 200 million which
00:13:54
is yeah it's it's a tremendous growth
00:13:56
it's it's actually nearly a four order
00:13:58
of magnitude growth in a decade and that
00:14:02
kind of of growth and data as you
00:14:05
probably know data really drives a lot
00:14:08
of these machine learning systems so
00:14:10
this is created essentially a vehicle in
00:14:13
which we can start to explore this space
00:14:14
so that's the inflection point
00:14:17
but now goo the other point of the thing
00:14:20
that's Intel and I think MIT observed is
00:14:23
fundamentally what we're seeing is the
00:14:27
way we've historically done programming
00:14:28
we think is flawed that there's
00:14:31
essentially a blurring of the
00:14:33
programmers intention with these
00:14:36
algorithms and with the system level
00:14:38
details what we really want to do as we
00:14:41
move forward is we want the programmer
00:14:43
to just specify his or her intention
00:14:46
that you you want to create a program
00:14:48
that will tell you where the nearest
00:14:51
Starbucks is and you just say computer
00:14:53
create a program that will always notify
00:14:56
me when I'm near Starbucks and then the
00:14:58
computer handles all the details of the
00:15:00
algorithm to implement it understands
00:15:02
how to translate that to work on the
00:15:04
hardware that's on your cell phone or in
00:15:07
a data center that type of thing so that
00:15:10
those are the two pieces that we think
00:15:11
really are creating this opportunity for
00:15:14
tremendous growth in machine programming
00:15:16
now to your second point about why is
00:15:19
this interesting to Intel until
00:15:22
obviously is very interested in advances
00:15:24
in hardware and me being at Intel for
00:15:27
about a decade now one thing that I've
00:15:28
seen which is really exciting
00:15:31
is Intel really used to be just a CPU
00:15:34
company and we're not today today the
00:15:37
heterogeneous hardware landscape that we
00:15:39
have at Intel is enormous
00:15:41
we have neural network processors we
00:15:43
have neuromorphic processors we have
00:15:46
GPUs we have a variety of accelerators
00:15:49
we have FPGAs and we have a ton of CPUs
00:15:52
the problem though is programming these
00:15:56
things that we can have all this
00:15:58
tremendous harder but how can we
00:16:00
possibly expect that the average
00:16:01
developer can program this and this is
00:16:04
really why machine programming is
00:16:05
essential to Intel as Intel understands
00:16:08
that with this new heterogeneous
00:16:10
Hardware landscape that really is
00:16:12
required to advance all of the
00:16:15
technology that we're seeing
00:16:16
we need a way that it's simple enough
00:16:19
for the average programmer harness this
00:16:23
massive amount of heterogeneous compute
00:16:26
it sounds amazing the ISO since he was
00:16:30
speaking about the work that you did
00:16:31
with MIT I understand that you wrote a
00:16:34
paper called the three pillars of
00:16:36
machine programming mm-hm
00:16:38
some time ago about some of these
00:16:41
concepts and I was wondering if you
00:16:43
could share some of the main insights
00:16:45
from that and how they relate to some of
00:16:48
the things we're talking about certainly
00:16:49
yeah so back in I think it was 2017 a
00:16:53
few of us from Intel labs teamed up with
00:16:57
several people at MIT and we came up
00:17:00
with this vision of what if we did this
00:17:04
thing called machine programming what
00:17:06
would the landscape look like and the
00:17:09
main reason for this is we were seeing
00:17:12
in these research venues that people
00:17:15
were starting to explore machine
00:17:16
programming but they were a bit
00:17:19
disorganized that there wasn't structure
00:17:22
around the way we were thinking and so
00:17:25
the paper that we wrote the three
00:17:26
pillars of machine programming is
00:17:28
essentially a road map on how we want to
00:17:32
express and explore the research space
00:17:34
there are three pillars intention
00:17:37
invention and adaptation the intention
00:17:40
pillar is really what we would think of
00:17:42
as the programmer is do
00:17:44
in the future I don't really call these
00:17:46
people programmers I call them software
00:17:49
creators because at the end of the day
00:17:51
our our blue sky vision is these folks
00:17:54
won't write a single line of code they
00:17:57
will express their intention either
00:17:59
through natural language gestures visual
00:18:03
diagrams whatever is best for them and
00:18:06
and for those hardcore programmers out
00:18:08
there they can still write all the code
00:18:09
that they want so that's the intention
00:18:11
pillar the invention pillar is it then
00:18:14
takes the programmers or the software
00:18:17
creators intention and then translates
00:18:20
that into the actual software these are
00:18:22
the algorithms the data structure is
00:18:24
that type of thing once that's
00:18:25
established then that work gets handoff
00:18:28
to the adaptation pillar the adaptation
00:18:31
pillar then takes that code and then
00:18:33
figures out ok what is the software and
00:18:36
hardware ecosystem look like for this
00:18:39
particular program how do we need to
00:18:41
augment it to make it run efficiently
00:18:43
securely correctly and then in the
00:18:46
machine learning context accurately now
00:18:50
in addition to Intel I'm sure there are
00:18:53
other companies that are also working on
00:18:55
machine programming and I wonder if
00:18:57
there are any companies with whom you
00:18:59
collaborate whose work you could talk
00:19:02
about just to explore how this field is
00:19:05
evolving absolutely so yeah we have many
00:19:08
collaborators in industry as well as
00:19:10
academia and some of our industrial
00:19:12
partners that are looking into this are
00:19:14
Microsoft and Facebook so Microsoft's
00:19:18
they have a wonderful gentleman over
00:19:20
there submit Gulani who's seen in in
00:19:23
many people's eyes is one of the
00:19:24
founders of formal program synthesis and
00:19:27
he actually has developed the system
00:19:30
inside of Excel that will automatically
00:19:35
figure out what the users intent is they
00:19:37
call this flash fill so this is a good
00:19:40
concrete example of real-world evidence
00:19:42
that this is not just a research toy you
00:19:45
can actually build this into real
00:19:47
products so Microsoft is deeply
00:19:49
interested in this another company is
00:19:51
Facebook so Facebook is actually doing
00:19:53
tremendous work in this space they
00:19:55
recently published a paper about
00:19:57
a system called aroma and what aroma
00:20:00
does is essentially works along the same
00:20:02
lines of the three pillars it's
00:20:04
principally focused on trying to help
00:20:07
with the intention that a programmer has
00:20:11
an intention of trying to write some
00:20:13
code and doesn't quite know exactly how
00:20:16
to write that code
00:20:17
the aroma system then will take a little
00:20:20
bit of that code and do an analysis over
00:20:22
a very large database and then send the
00:20:25
user back is this what you meant and
00:20:27
it's sort of a human-in-the-loop machine
00:20:29
learning approximate solution that's
00:20:32
sort of good early evidence that there's
00:20:34
a lot of while while we think of the
00:20:37
space machine programming as being a
00:20:38
very long journey there's things that we
00:20:41
can be doing today in industry that
00:20:43
could be extremely valuable
00:20:45
that sounds pretty remarkable again know
00:20:49
you spoke about several companies which
00:20:52
countries do you think are making
00:20:54
progress that you find impressive in the
00:20:57
area of machine programming in general
00:21:00
in AI I've heard that China is you know
00:21:03
advancing in leaps and bounds could you
00:21:06
talk about you know what's happening in
00:21:08
other parts of the world and what are
00:21:11
some of the things that you are paying
00:21:12
attention to absolutely
00:21:14
so as you pointed out China is doing
00:21:16
tremendous things one of the things that
00:21:18
they're doing is they have a very strong
00:21:20
sort of governmental infrastructural
00:21:23
support for AI and it's my belief that
00:21:26
the u.s. also has this but maybe not to
00:21:29
the level that China does it's something
00:21:31
that I think as a country we probably
00:21:34
need to be a little bit more aggressive
00:21:36
and progressive about there's also a lot
00:21:39
of involvement and advances that are
00:21:41
happening in Europe and that's also tied
00:21:44
in both with their academic schools they
00:21:47
have very strong ml leaders and academia
00:21:50
and then also the vision that they have
00:21:53
through their governmental
00:21:53
infrastructure which European country is
00:21:56
you think are doing the most interesting
00:21:57
work that that's a great question off
00:22:01
the top of my head you know Germany is
00:22:04
actually doing some really tremendous
00:22:05
stuff as one might imagine part of that
00:22:08
has to do with the fact that
00:22:09
they've been deeply involved in
00:22:11
autonomous or vehicles and the natural
00:22:14
evolution is autonomous vehicles and
00:22:16
then the byproduct of that is deep
00:22:17
engagement in AI and machine learning no
00:22:21
D which innovations and machine
00:22:23
programming do you think most promising
00:22:26
and where do you think the next
00:22:29
breakthroughs will occur in the
00:22:31
immediate future yeah so this is this is
00:22:35
a really fascinating question and as I
00:22:37
was mentioning before there is a lot of
00:22:40
low-hanging fruit where we can make
00:22:42
advances and we can build things like
00:22:44
aroma or flash fill that are very useful
00:22:46
but there are some core challenges that
00:22:50
at least with the folks that I'm
00:22:52
interacting with that places like
00:22:54
Stanford and MIT and Google and deeply
00:22:56
within Intel labs that we don't quite
00:22:59
have the answer to the first is the
00:23:04
structural representation of intention
00:23:08
so what I mean by this is oftentimes
00:23:11
when we're writing code
00:23:13
the programmers intention is diffused
00:23:16
across of the code what we really want
00:23:19
to be able to do is understand how to
00:23:20
properly represent the user's intention
00:23:24
and we don't quite have this there's a
00:23:26
lot of advances that we've made
00:23:27
historically with things like compilers
00:23:29
and static analysis tools that create
00:23:33
different sort of graphical or tree
00:23:35
structures but when we've tried to apply
00:23:37
these in the space of machine
00:23:39
programming they don't quite fit that we
00:23:42
can sort of you know push the you know
00:23:45
square peg in the round hole but it's
00:23:46
not the right match so I didn't tell
00:23:49
we're thinking about this thing that
00:23:51
we're sort of roughly calling the
00:23:54
abstract semantic graph and the idea
00:23:57
here is this structure whatever this is
00:24:00
that we don't quite understand will be
00:24:03
some sort of graphical representation of
00:24:04
the semantics essentially the intention
00:24:07
of what the user wants once we figure
00:24:11
out how to build this thing my belief is
00:24:14
the field of machine perming will see a
00:24:16
tremendous spike of growth so a lot of
00:24:19
people are working on this I'm working
00:24:21
with collaborators
00:24:22
both in industry and then in academia
00:24:24
folks at Penn Berkeley MIT we're all
00:24:28
thinking deeply about this hopefully
00:24:32
we'll be able to figure out this this
00:24:34
abstract semantic graph soon and until
00:24:38
we do I think that we'll just sort of
00:24:40
work with the maybe not perfect
00:24:43
solutions and try to you know edge our
00:24:45
way forward so if you figure it out what
00:24:48
might some of the implications be so
00:24:51
some of the implications will be the
00:24:54
programs that we'll be able to generate
00:24:56
are likely to be orders of magnitude
00:24:58
more complex than the ones that we can
00:25:01
create today for example in the space of
00:25:04
formal program synthesis or approximate
00:25:07
solutions for machine programming we may
00:25:09
be restricted to let's say programs that
00:25:12
are up to maybe a hundred instructions
00:25:15
or less if we figure out how to build
00:25:19
this abstract semantic graph it's my
00:25:20
belief that we will move from hundreds
00:25:23
to thousands potentially millions of
00:25:26
lines of code that so the implications
00:25:29
of this thing they're enormous so when
00:25:32
any new technology comes along
00:25:34
especially a machine learning or as you
00:25:38
described machine programming very often
00:25:41
technologists have to justify these
00:25:44
investments to the CFO or to the CA CEO
00:25:47
not just in terms of this is very cool
00:25:49
technology but right this fits in with
00:25:52
the ROI of where the business wants to
00:25:54
go or order fits in with the business
00:25:57
strategy what are some of the metrics
00:25:59
that you think about in terms of what's
00:26:03
the ROI in measuring the ROI of machine
00:26:07
programming right this is a great
00:26:09
question and of course as the leader of
00:26:12
the machine programming research group
00:26:13
at Intel it's my job not only to work on
00:26:17
the research but then also justify its
00:26:19
business value as you pointed out and
00:26:23
one of the things that you you might
00:26:26
know is Intel's very interested in
00:26:28
performance but we're not just
00:26:30
interested in Hardware performance we're
00:26:33
also interested in software performance
00:26:35
so one could imagine
00:26:36
that if you have a programmer that's
00:26:38
writing code that's slow
00:26:40
they might blame the Intel processors is
00:26:43
being slow even though the problem is
00:26:44
not the processor it's actually the
00:26:46
software one of the promises of machine
00:26:50
programming and we're seeing early
00:26:51
evidence of this is that the code we can
00:26:54
generate through these automated methods
00:26:56
will be superhuman in their performance
00:26:59
correctness security so on and so forth
00:27:01
one concrete example of that is one of
00:27:04
my colleagues Andrew Adams
00:27:07
Jonathan Regan Kelly cave on photo Holly
00:27:09
and Diaz are folks from MIT Stanford and
00:27:12
Facebook that were actually I think
00:27:13
Andrew Adams just pivoted to Adobe
00:27:16
research they have built a system called
00:27:20
halide it's a programming language that
00:27:22
separates out the programmers intention
00:27:25
from the actual scheduling of that
00:27:27
intention and in their recent paper this
00:27:30
year that was published or I think in
00:27:32
June they've shown for the first time
00:27:34
that the world's foremost experts in
00:27:38
this programming language can't compete
00:27:41
with the machine that the machine is
00:27:43
producing code now that is regularly
00:27:47
more efficient and I think it's by I'm
00:27:50
gonna be I'm just gonna guess here by at
00:27:52
least 50% it might be upwards of you
00:27:55
know a hundred percent faster and this
00:27:58
is the first time in the decade that
00:28:01
they've been working in halide that
00:28:02
they've been able to achieve this but so
00:28:04
this gives us promise that if we can do
00:28:07
it in halide maybe we can generalize
00:28:10
this and start to improve the efficiency
00:28:13
of code everywhere this is really
00:28:15
important to Intel because obviously we
00:28:18
want everyone software to run as
00:28:19
efficiently as possible and we don't
00:28:21
want people to mistakenly believe that
00:28:23
our hardware slow when actually the
00:28:26
problem is somewhere else no I I I hear
00:28:29
you and it actually reminds me of a
00:28:32
broader concern that I have often heard
00:28:34
about AI which is that as a lot of
00:28:36
automation begins to take place and AI
00:28:39
systems get implemented that the impact
00:28:43
of jobs could be considerable so for
00:28:45
example but autonomous trucks
00:28:49
it has been a fear that lots of truck
00:28:53
drivers could be losing their jobs if
00:28:55
you know autonomous vehicle stopped
00:28:58
hauling you know goods across the
00:29:02
highways so do you think that there is a
00:29:05
risk that if machine programming takes
00:29:09
off that the same thing could happen to
00:29:12
software programmer jobs and that this
00:29:14
this is something that the industry
00:29:16
should be concerned about right this is
00:29:19
an excellent question and one that I'm
00:29:21
asked quite often my my honest opinion
00:29:25
is actually the inverse will happen is
00:29:28
that through machine programming we will
00:29:31
create many jobs perhaps millions or
00:29:34
tens of millions of jobs and the
00:29:36
reasoning is actually very simple right
00:29:38
now we have a global population in the
00:29:42
billions yet the programmer pool today
00:29:45
is a very small percentage I don't know
00:29:48
the exact percentage but I think that
00:29:50
it's roughly around like 1% of the
00:29:52
global population with machine
00:29:54
programming what we are are trying to
00:29:56
achieve is enable the entire global
00:30:00
population to create software for
00:30:03
example my mother she's incredible you
00:30:05
know entrepreneur she's created several
00:30:07
businesses and has just done fantastic
00:30:10
but she's not a programmer so the entire
00:30:12
world of software is closed off to her
00:30:14
and in fact this is one of the reasons
00:30:16
that I became so compelled is I see
00:30:18
someone like her who's wildly creative
00:30:20
she has some amazing ideas but because
00:30:24
software's closed off those ideas never
00:30:26
get realized and hopefully with machine
00:30:29
programming with this intentionality
00:30:30
that we're discussing earlier this will
00:30:32
this will create tens hundreds of
00:30:35
millions of jobs it'll also keep the
00:30:37
programmers that we have today employed
00:30:39
because there is work to do on building
00:30:43
these very complex systems and as we
00:30:45
expand intentionality we're going to
00:30:47
require those people those what we call
00:30:50
it Intel those ninjas to be ensuring all
00:30:53
the subsystems that are part of those
00:30:55
three pillars are advancing
00:30:57
appropriately so since we were talking
00:31:00
about these advice
00:31:01
consequences it reminded me of a
00:31:03
conversation almost 15 years ago with
00:31:06
Andy Grove the former CEO of Intel whom
00:31:10
we had interviewed in 2004 and and he
00:31:15
once said that for every metric there
00:31:17
should be another paired metric that
00:31:19
addresses the adverse consequences of
00:31:21
the first so as you were thinking about
00:31:23
some of the metrics that you would use
00:31:25
to measure the success or the ROI yeah
00:31:28
off of machine programming what would be
00:31:31
some of the adverse consequences of
00:31:33
machine programming and what metrics
00:31:36
would you use just to keep an eye to
00:31:37
make sure things don't get out of
00:31:39
control I'm really glad you asked this
00:31:42
question and on top of that before I
00:31:44
answer I just want to say I'm a huge fan
00:31:47
of Andy Grove it's wonderful to be at a
00:31:49
company with such strong legacy of
00:31:52
leadership and you know we see the
00:31:55
impact that Andy has had even today that
00:31:59
the company is really trying to follow a
00:32:01
lot of his principles I would agree with
00:32:04
you wholeheartedly and in fact in a book
00:32:06
that we wrote in 2004 that knowledge at
00:32:08
Wharton wrote he was identified as the
00:32:11
top leader among 25 leaders in the past
00:32:13
25 years oh that's fantastic
00:32:15
yeah I never got the chance to work with
00:32:17
him personally but I know people that
00:32:19
have and from everything I hear he was
00:32:22
not only a tremendous leader but also
00:32:24
tremendous technologist which is sort of
00:32:27
a very rare combination but going back
00:32:30
to your question about the adverse
00:32:32
consequences we actually talked about
00:32:34
this a little bit in our three pillars
00:32:36
paper and this is part of the reason why
00:32:39
we we wrote this paper is that what we
00:32:42
were seeing in some cases of advances is
00:32:45
there would be research for example one
00:32:48
of my colleagues that I'm very fond of
00:32:50
Alvin Chung he's a professor at Berkeley
00:32:52
and he's doing this work called verified
00:32:54
lifting and verified lifting essentially
00:32:57
uses formal program synthesis techniques
00:33:00
to lift code from one programming
00:33:02
language and then drop it down into
00:33:04
another programming language this is
00:33:06
wildly useful for legacy systems that
00:33:09
can't be maintained because we don't
00:33:11
have a programmer supply we can lift
00:33:13
that code out put it in
00:33:14
language where we have lots of
00:33:15
programmers however one of the things we
00:33:18
noticed and I've discussed this with
00:33:20
Alvin so he will probably not be shocked
00:33:23
when I mention this is that there's a
00:33:25
potential byproduct of that lifting that
00:33:29
can reduce intentionality so for example
00:33:32
his work we would say principally falls
00:33:35
in you know the invention and then the
00:33:38
adaptation based on how that code is
00:33:42
transformed the intentionality of that
00:33:45
code could be reduced for example things
00:33:48
like variable names function names
00:33:49
things that are really important to
00:33:52
programmers may not map properly to the
00:33:54
new structure so as we're making forward
00:33:58
progress in machine programming what
00:34:00
we've asked the community to do is think
00:34:02
of the context of the three pillars and
00:34:04
then try to understand are you
00:34:07
inadvertently hurting another pillar and
00:34:10
if you are clarify that so we understand
00:34:13
that this is another thing that we now
00:34:14
need to advance now since you mentioned
00:34:17
you know some of the academic
00:34:20
collaborations I was I know that you
00:34:22
spoke at Penn that did precise event
00:34:26
last week and I was wondering if we
00:34:29
could end by talking a little bit about
00:34:31
what kind of work you are planning to do
00:34:32
here at Penn oh yeah so I'm yeah very
00:34:35
delighted that you asked this question
00:34:37
and I'm really happy to be here
00:34:39
I had the honor of giving this talk at
00:34:42
the precise industry day which was over
00:34:46
sold out I think people were sitting on
00:34:48
the floor it was just very well attended
00:34:51
and rightfully so the lot of the thought
00:34:54
leaders in the space of computer science
00:34:58
and formal methods in machine learning
00:35:00
are part of the precise center recently
00:35:04
I accepted invitation to help chair the
00:35:09
technologists regroup for precise and
00:35:12
also act as their executive director for
00:35:15
artificial intelligence my role with
00:35:18
precise and with Penn actually I think
00:35:21
is twofold the first is with precise
00:35:25
they have a very strong sort of
00:35:27
technical
00:35:27
Horsham of industry collaborators and
00:35:30
what I would like to do is ensure that
00:35:32
all of the industrial partners are
00:35:34
working in a very complementary way that
00:35:38
we understand what the core challenges
00:35:39
are and that we're not working in a way
00:35:41
that's overlapping duplicating effort so
00:35:44
that's one part the other part that's
00:35:46
really important to me is right now we
00:35:49
have sort of a lack of machine
00:35:52
programming engineers and researchers in
00:35:55
fact there's very few of us which makes
00:35:58
sense because the field even though it's
00:36:00
been around since the 50s it's had
00:36:03
struggles to get to the point where it
00:36:06
is today
00:36:06
so what we're working on with Penn and
00:36:09
other academic Institute's is to start
00:36:12
to incorporate curriculum changes and
00:36:15
get our undergrads get our grad students
00:36:18
more familiar with it and then also
00:36:20
generate the new leading minds through
00:36:23
the PhD programs that are then going to
00:36:25
drive the research they're happening
00:36:27
both in academia and in the industrial
00:36:30
labs that sounds wonderful and Justin
00:36:34
thank you so much for taking the time to
00:36:36
explain all these things to us it's
00:36:38
wonderful to meet you and very happy to
00:36:41
have you here at knowledge at Wharton
00:36:42
yeah it's been my pleasure thank you so
00:36:44
much for having me for more insight from
00:36:49
knowledge at Wharton please visit
00:36:50
knowledge Wharton UPenn edu
00:36:56
[Music]
