00:00:00
The scary open secret in the AI industry right now is that it's possible that we'll end up essentially creating a new
00:00:05
species that ends up ruling the world with a 70% chance that this goes horribly wrong like human extinction.
00:00:10
That's one possibility. There's many more. It's quite chilling what you're saying.
00:00:13
>> Yeah, it's uh it gets me down sometimes. I basically told my wife like, let's not
00:00:20
have any more kids. It's too uncertain. I don't think they'll ever join the workforce.
00:00:24
Everybody should be afraid that their jobs are going to be lost. And I know this because I went to open air in 2022.
00:00:28
What I did there was forecasting as to what the next couple years might look like. And unfortunately, most of the
00:00:33
world is kind of asleep at the wheel and doesn't really realize what's going on with AI. So, I resigned.
00:00:37
>> I read it somewhere that you lost $2 million for not signing an anti-disparagement clause, meaning you
00:00:43
couldn't criticize the company. >> Yes. For reasons I'm happy to get into. But the main thing I've learned is when
00:00:47
I go talk to people at Anthropic and OpenAI about forecasting, they're like, "It's not going to take that long. You
00:00:51
need to shorten them again. get them back to 2027 or 228 because these powerful CEOs, Ariel or Sam or Elon are
00:00:57
racing each other to be in control of the most powerful AIs and are literally afraid that if the other guy gets there
00:01:03
first, he might become dictator. I mean, Anthropic is on track to be the entire economy by 2030. But none of these
00:01:09
people should be trusted with that much power. So, this is the most important thing happening in our lifetimes,
00:01:13
probably in all of history, in fact. And it's very important that it go well. So,
00:01:17
I think that there's a lot we can do to like steer things in a better direction.
00:01:19
There's loads of benefits that we could get from AI if we do it right. And if we
00:01:22
do solve the problems, then things could be absolutely amazing for everyone. >> Well, this report here in 2021, it was
00:01:28
remarkably accurate. And then you just published this one. >> Yeah. So, this is our new scenario.
00:01:32
>> So, let's go through these slowly and one at a time. >> I would be incredibly happy if all my
00:01:36
predictions turn out to be wrong. >> This is super interesting to me. My team given me this report to show me how many
00:01:43
of you that watch this show subscribe. And some of you have told us according to this that you are unsubscribed from
00:01:48
the channel randomly. So, favor to ask all of you. Please could you check right now if you've hit the subscribe button
00:01:53
if you are a regular viewer of the show and you like what we do here. We're approaching quite a significant landmark
00:01:57
on this show in terms of a subscriber number. So, if there was one simple free thing that you could do to help us, my
00:02:03
team, everyone here to keep this show free, to keep it improving year over year and week over week, it is just to
00:02:09
hit that subscribe button and to double check if you've hit it. Only thing I'll ever ask of you, do we have a deal? If
00:02:14
you do it, I'll tell you what I'll do. I'll make sure every single week, every single month, we fight harder and harder
00:02:19
and harder and harder to bring you the guests and conversations that you want to hear. I've stayed true to that
00:02:23
promise since the very beginning of the D of Coio, and I will not let you down. Please help us. Really appreciate it.
00:02:29
Let's get on with the show. >> Daniel Cocatello, at the very heart of what you do, um,
00:02:38
what is your mission and why? So what would you do if you thought that super intelligence was coming in a few years?
00:02:46
>> I guess it depends what the consequences were. Well, let's talk about it. So super intelligence, AIs that are better
00:02:54
than the best humans at everything while also being faster and cheaper. Also able
00:02:58
to operate robots that can do everything in the physical world that humans can do, but better, faster, and cheaper. If
00:03:04
that really is coming in a few years, then we need to prepare and we need to think about how to make it go well
00:03:10
instead of poorly. So that's sort of my answer is like I'm doing that to the best of my ability.
00:03:14
>> So you believe it's coming in a few years? >> Yes. >> How could you be so sure?
00:03:19
>> I spend a lot of time trying to forecast this sort of thing. My sort of median
00:03:22
estimate 50% chance is currently in 2029. Maybe it'll slip to 28 28. It's possible that it'll take significantly
00:03:29
longer, like maybe 10 years or something like that, but uh you know, for reasons
00:03:35
I'm happy to get into, seems to me like it's probably happening by the end of the decade. What's less important is the
00:03:41
the sense of how close we are. What's more important is the pace of the trends. Anthropic
00:03:48
this time last year was making something like a billion dollars a year and now they're making something like $60
00:03:54
billion a year. So that's 60x growth in one year, which is extremely impressive even for very small startups. But for a
00:04:03
company of their size, it might be the fastest growth in history. Um, we expect that rate of growth to slow down, but
00:04:12
even if it slows down quite a lot, they're still on track to be, you know, the entire economy by 2030 or so.
00:04:21
>> Why should the average person care? The high level thing is absolutely everything is going to change for the
00:04:25
whole world and including therefore for them and their families. Um could change
00:04:29
for the better, could change for the worse depending on the details of how it's done. So for example, everyone
00:04:34
could die you know um this is the classic loss of control scenario or one version of it. If we do build these
00:04:41
super intelligences and we use them to automate all the jobs and we put them in the military and we, you know, have them
00:04:49
giving advice to politicians and so forth, they will eventually have accumulated enough real world power that
00:04:55
they don't need humans anymore and they're smarter than us, they're more strategic, etc. At that point, we sort
00:05:00
of have to hope that they are virtuous, that they have, you know, the goals that
00:05:04
we wanted them to have, the values that we wanted them to have, etc. And the sort of scary open secret in the AI
00:05:11
industry right now is that right now that is kind of just a hope. It's not something that we can be at all
00:05:16
confident in. And in fact there's lots of evidence and arguments that it we're not on track to achieve that. So there's
00:05:21
lots of reason like current AIS for example will often lie uh to people or they will like you tell them to do
00:05:27
something and they go do something else and then pretend that they did it right.
00:05:30
So, it's an inherently difficult problem to make something that's super intelligent and also has the values and
00:05:36
virtues that you want it to have. And it doesn't seem like we're on track to solve that problem. Also, it seems like
00:05:41
the sort of problem that you could think you've solved when you haven't actually
00:05:44
solved it, right? Uh that's a big reason why this is scary. So, for all those reasons, it's possible that we'll end up
00:05:50
essentially creating a new species that ends up ruling the world instead of us. And then maybe we go the way of other
00:05:57
extinct species in the past that were out competed by humans. That's one possibility. There's many more. Even if
00:06:02
you're not worried about that and you think that the AI will be totally controlled, there's the question of who
00:06:07
controls the AIS, right? When there's a couple corporations that have made these
00:06:11
super intelligences and are using them to automate all the jobs. Well, that's a lot of power. You know, that's a lot of
00:06:17
money. It's a lot of political power. They'll have the best strategists, the best advisers, you know, they'll think
00:06:23
faster militarily. uh the countries that has these AIs will be able to absolutely
00:06:28
wipe the floor with all the other countries. The AIS themselves it's it's kind of a single point of failure like
00:06:33
central uh control system where you know the CEO of Enthropic Dario he coined this phrase the country of geniuses in
00:06:42
the data data center that was his phrase to describe what they're trying to build. You know I think that's a little
00:06:48
bit misleading. I think it would be more accurate to describe it as army of geniuses in the data center because it's
00:06:53
not like it's a bunch of diverse different AIs, you know, living in their different parts of the data center.
00:06:58
They're all copies of the same big model and they're owned by the company and so
00:07:04
they all follow the orders given by the company, right? People should be asking questions of like who controls this army
00:07:09
or these armies and what are they going to be doing with them? I think that we could very easily end up in a sort of
00:07:16
uh a situation where some tiny group of people are essentially oligarchs or dictators. And ironically,
00:07:24
both of these risks, the loss of control and the constitution of power are things
00:07:28
that people in the industry have been thinking about for decades. Um even before the AI industry existed, you
00:07:34
know, people thinking about AI were talking and writing about these things. And then part of the founding narrative
00:07:39
or the founding myth of deep mind and open AAI and anthropic is these problems are real. So we need to get there first
00:07:48
so that we can handle it responsibly. Those are I think the big two reasons. But then I can go on there's lots more
00:07:53
reasons as well. So one thing is you know World War II geopolitical conflict. Um if AI does in fact get incredibly
00:08:01
powerful that's going to change the balance of power between nations. That's going to disrupt a lot of things. That
00:08:06
puts us at increased risk of crisis more generally, right? Another one, what about those jobs?
00:08:12
You you're going to lose your taxi job, but not just the taxi driver, everybody pretty much. Um there might be a few
00:08:17
exceptions like people whose jobs for legal reasons are only allowed to be done by humans, but for the most part,
00:08:24
everybody should be afraid that their jobs are going to be lost even if we manage to avoid all the other problems.
00:08:28
Right? this narrative has started to emerge and I've had several interviews on this show where I've interviewed
00:08:33
people who are very very scared and anxious about AI and these are people that have worked in the industry for
00:08:37
sometimes decades. Yeah. >> Um the counternarrative coming over the hill is that this is doomerism that
00:08:43
these people are for whatever reason just trying to scare people and that they don't really understand what
00:08:47
they're talking about. How do you respond to that sort of counternarrative? And you must have seen
00:08:51
this emerging yourself especially from people who stand to benefit dare I say. >> Yeah, exactly. this counter narrative is
00:08:58
fairly recent and it's been pushed by the people who stand to benefit um from it and it's not true. Like these these
00:09:05
concerns have been around for decades since before the AI industry existed. They're actually pretty reasonable
00:09:09
concerns. Like if you take the companies at their word and imagine that they are
00:09:12
in fact going to build super intelligence. Well, it raises a lot of questions like who's going to control
00:09:17
it? Will anybody control it? What about the jobs? You know, like these are just kind of obvious implications to be
00:09:23
thinking about and worrying about. Who are you and what's your story? >> My name is Dan Coatello. Um, I currently
00:09:30
run the AI Futures Project, which is a small nonprofit that mostly focuses on forecasting the future of AI. Before
00:09:38
that, I worked at OpenAI. >> AI forecasting. >> Yeah. So think about how like you know
00:09:46
industry analysts who work for hedge funds and stuff will make these forecasts of like here's you know how
00:09:51
many cars Tesla will be selling 5 years from now or like here's what the price of electricity will be in 2 years right
00:09:58
that's forecasting I was doing that but specifically focused on AI the reason I was doing it is because it's incredibly
00:10:04
important to to see where this is all headed >> why did you go to open AI what did you
00:10:09
do there what did you observe while you were there and how did Did it change your perspective on the future of AI but
00:10:16
also I guess open AI as a company and for anybody that doesn't know OpenAI are the company that produced chatbt?
00:10:21
>> Yeah, so I went to OpenAI in 2022. Uh a large part of what I did there was more
00:10:25
forecasting. AI 2027 is a scenario that you may have heard of. I did like smaller, you know, lower effort versions
00:10:32
of them internally for just internal circulation of like here's some guesses as to what the next couple years might
00:10:37
look like. I also worked on evaluations for dangerous capabilities. So, you know, trying to measure the AI's cyber
00:10:44
abilities or persuasion abilities or situational awareness. And I also briefly was on a uh a capabilities team
00:10:52
doing reinforcement learning to create agents. AI is in fact getting uh a lot better and I can say more about why you
00:10:59
know scaling laws um deep neural nets bigger trained on more data become more efficient, more competent at those
00:11:05
things. I also became a bit more disillusioned with the AI industry. So, OpenAI, Anthropic, and
00:11:15
Deep Mind all had these sort of founding narratives of like, yes, these risks are
00:11:18
real, but we've thought about them and we're going to try to handle them responsibly. And that's why it's
00:11:23
important for us to keep doing what we're doing. And I increasingly came to think that these were rationalizations
00:11:31
to justify what they were doing rather than sort of like deeply guiding their actual behavior and that when push comes
00:11:37
to shove, they'll follow their incentives rather than do what's actually good. >> So you're inside Open AI at the time and
00:11:45
you start to believe that they're following commercial incentives versus the I guess social in or societal
00:11:52
incentives that they founded themselves on. >> Sort of. I mean, I wouldn't actually
00:11:55
describe it as commercial incentives. I think I would describe it as um power seeking incentives. So,
00:12:02
>> like it's true that the companies care a lot about making a lot of money, but
00:12:07
especially at the very top of these companies, like the leaders, they understand that this is about more
00:12:12
than just money. You know, there are these emails that came up in, you know, the the lawsuit between Musk and and um
00:12:19
OpenAI. A bunch of emails were surfaced in that lawsuit, which you can go read. And in some of them, the founders of
00:12:26
OpenAI were talking back in like 2017 about how the reason why we made OpenAI was because we were worried that Demos
00:12:34
at Google was going to become dictator with AGI. Even back then, they were this is obviously about more than just money.
00:12:39
Like these these powerful CEOs are literally afraid that if the other guy gets there first, he might become
00:12:46
dictator. And they don't trust each other. And so that's why they are racing as hard as they can so that they're the
00:12:53
ones who get there first, so to speak. >> Have you met Sam Alman? >> Yeah. >> And did did that shape your opinion of
00:13:00
his incentives or what why he's doing what he's doing? Cuz there's a lot, you know, speculated about what his
00:13:04
incentives are. I mean, his most recent narrative says uh for the good of humanity. I think that's what
00:13:09
>> Yeah. I mean, I think the main thing I've learned is don't pay attention to the narratives, you know, like uh what
00:13:14
they say to one person is just different from what they can say to some other person at the same time. And what they
00:13:19
say in public is a third thing entirely. I think you should judge people by their
00:13:22
actions, not by their words. >> And why are you no longer at OpenAI? >> Largely for the reason that I mentioned.
00:13:28
So I became gradually disillusioned with how the company was going to behave. For
00:13:32
example, when I first joined in 2022, at least the people I talked to, my colleagues at the company, there was
00:13:39
this general sense of like, of course, we wouldn't actually just build super intelligence as soon as possible. Once
00:13:45
we started getting really close, like once we started getting to AIS that could maybe automate the AI research
00:13:50
process, we would pause and figure out how to make it safe. That's cuz we're the good guys. And
00:13:56
that's obviously the safe thing you should do rather than just going full speed ahead. But we're worried about
00:14:01
other people who might not pause. You know, our competitors, Google for example. And so that's why we need to be
00:14:07
in the lead so that we have that room to do the safe stuff, right? That was sort
00:14:11
of like a thing that seemed like maybe like the median position or something among the colleagues I talked to when I
00:14:17
was there when I started, including people like Sam, you know, including the leadership. And then by the time I left,
00:14:22
I was like, "Oh man, they're really not going to do that, are they?" Like like they they've sort of, you know, partly
00:14:27
because this has become more politicized and they've become bigger and under more
00:14:31
scrutiny, people have started asking like, "Why are you doing this in the first place if it's so risky?" And so
00:14:35
they've pivoted their narrative to being more like actually it's not that risky,
00:14:38
you know? Um, and so yeah, I mean it seems like they're just going to keep going roughly as fast as they can and
00:14:46
hope that they can figure it out on the way. >> How did your time at OpenAI come to an
00:14:49
end? >> Uh, I resigned in 2024. I had a nice goodbye party. >> What were the reasons you gave for
00:14:55
quitting OpenAI? >> I thought that we were rationalizing too much and that we needed to think more
00:15:00
about what would actually be good for the world. Um, I wanted more freedom to publish.
00:15:05
So at OpenAI as it became a bigger company it became more of a normal tech company with incentives and you know a
00:15:13
PR department and things like that and so it started becoming more difficult to um to publish the sort of research that
00:15:20
I was doing for example those scenarios that I mentioned couldn't uh couldn't publish those right they're just for
00:15:25
internal use I thought that that was a shame because right now most of the world is kind of asleep at the wheel and
00:15:32
doesn't really realize what's going on with AI and doesn't really realize what's coming in the pipeline a couple
00:15:36
years from now and the companies aren't really incentivized to tell people that much about it. I mean they say some
00:15:43
vague stuff in a sort of hypy way but um you know well they didn't want me to publish the scenario for example laying
00:15:51
out like here's how things might actually look. >> I'm just I'm super curious as to what
00:15:55
it's like being in a company like that when they you know chat GBT3 is released. You were there at that time
00:16:00
right? >> Mhm. um which was a moment where I think the whole world stood up and realized
00:16:04
that this technology was powerful. >> Yeah. >> Um and the conversation really began
00:16:09
from a society level. Um the company starts growing super quickly. >> Yeah. >> Quicker than I think anybody could ever
00:16:16
have imagined. And what what was it like inside there? What did you see change um
00:16:21
over over that period of time? >> I remember one all hands meeting where Ilia said something like
00:16:26
>> Ilia being >> Ilia Sgver who was um head of research at that time. He said something like,
00:16:30
"Okay, now the world is starting to pay attention. Each of you is going to be the most popular person at every party
00:16:36
uh for the next year. Don't let it get to your head. Focus on the mission. Got to build AGI." The company grew a lot.
00:16:43
It already wasn't really feeling like a nonprofit when I joined, but it definitely didn't feel like a nonprofit
00:16:48
by the time I left. Um, lots of new people came in. Ironically, the like amount of conversation about
00:16:57
super intelligence and the implications of super intelligence arguably sort of went down over time due to this growth,
00:17:05
right? So, because the company would like double and then double again and then double again, all these new people
00:17:11
were coming in from other parts of the tech industry who hadn't really been thinking about these things and were
00:17:15
attracted by the high salaries. You lost $2 million for not signing an anti-disparagement clause, which would
00:17:22
mean you could speak, you couldn't criticize the company. >> Uh, yes. Well, so, um, I got to keep the
00:17:27
money. >> Oh, you got to keep >> So, what happened was after I had left, said my goodbyes, etc., um, I got the
00:17:34
the exit paperwork, and it included this clause that said you basically have to agree not to criticize the company
00:17:39
again. Um, and also a clause saying, you can't tell anyone about this. And so I thought that was kind of rich coming
00:17:48
from a nonprofit that's supposed to be, you know, for the benefit of all humanity. So I didn't sign it. And if
00:17:54
you don't sign, you don't get to keep your equity. So your compensation, you know, what what they pay you is a bunch
00:18:00
of money and then also a bunch of stock basically. But then they had this stuff in the contract that they get to yank
00:18:07
back your your stock if you don't sign this thing. Um, and my wife and I, you know, were uh upset about this. We
00:18:16
talked about it for like a month or two, consulted some lawyers, um, and then ultimately decided to just refuse to
00:18:21
sign, >> which would mean you lost, you would have lost $2 million. >> That's right. Which was like 80% of our
00:18:27
net worth at the time. Um, fortunately, uh, it didn't go the way we expected. It
00:18:34
blew up basically on the internet. Like when people heard that that we had done this and that we had said no, it became
00:18:40
like this huge scandal. Employees at the company started like asking questions in
00:18:44
Slack and like asking leadership like, "Wait, what? Like why are you going to take away our equity? What is this?" You
00:18:49
know, cuz a lot of people hadn't really noticed this before. It had been whispered about, but it hadn't been sort
00:18:54
of like a thing that most employees knew about. Um, and so they backtracked and they said, "Never mind. Never mind.
00:18:59
We'll change the paperwork. You can keep the equity. It's fine." And Samman came
00:19:03
out and said he was embarrassed that he didn't realize this was >> Yeah, he had no idea apparently.
00:19:08
>> You don't believe him? >> No, I think he probably knew. And if he didn't know, then people close to him
00:19:13
probably did, such as his head lawyer. >> Why did you decide not to take the $2 million?
00:19:19
>> I mean, >> most people would have. I think >> it's true. Most people would have and
00:19:23
most people did. And you know, money is nice, but like it's not the only thing. You know, sometimes it's good to take a
00:19:30
stand on principle. I I keep mentioning super intelligence. Perhaps I should say
00:19:34
more about like the the sequence of events that the companies are planning to do. So, right now they're focusing on
00:19:42
automating coding. They're taking their AIS, they're making them bigger, they're
00:19:46
training them for longer, and they're especially focusing the training on getting them to be good at autonomously
00:19:52
writing and editing code because uh that will help the companies go faster, right? Right? If they can automate the
00:19:59
code, then they can do their own work better and faster and accelerate progress. The next step, which they've
00:20:05
already begun, is to look at the rest of the research process as well. Coming up
00:20:10
with ideas, um, analyzing experiments, communicating those results, all the other parts of of the research process,
00:20:17
they're trying to figure out how to train AIs to be good at those as well, so that they can have AIs do the entire
00:20:22
thing autonomously. When you say do the entire thing, what do you mean do the entire thing?
00:20:27
>> So like Anthropic and OpenAI in particular are trying to automate themselves. Like they're trying to make
00:20:32
it the case that um they don't really need human employees anymore. Uh they just have a giant army of AIS that's
00:20:40
turnurning away doing all this autonomous research to make better AIs to train the new AIs, put them in charge
00:20:48
so they can make even better AIs and so forth. And of course, not just not all just happening internally, but also like
00:20:53
interfacing with the world, right? Like going out and talking to people, collecting the data, setting up the
00:20:57
training environments, doing the business deals, and so forth. Like they're they're trying to automate all
00:21:02
of that. The reason why they're doing this is because they're trying to get to a position where they have AIs that are
00:21:10
superhuman at everything, super intelligence, and they're trying to get there before their competitors do.
00:21:16
Needless to say, this is incredibly dangerous, I would say. You know, and in addition to being dangerous, it's a
00:21:22
power grab, right? Like, if they actually succeed at this, then they'll be sitting on top of this army of
00:21:28
superhuman AIs that will give them immense leverage over all sorts of other actors in the economy in so far as they
00:21:35
can work out something with the president and, you know, integrate it into the military or whatever, then that
00:21:40
would give the US immense hard power over all other countries, right? Obviously, nobody knows exactly when
00:21:46
this is happening, but a very disquing thing has happened over the last year to me, which is that when we published
00:21:53
2027, people were generally of the opinion that my timelines were too short and that like probably it would take more
00:22:01
than 2027 until we got to the sort of events that I was just mentioning. you know, recursive self-improvement, AI is
00:22:08
automating the whole research process, super intelligence. These these types of milestones um they happen in 2027 in AI
00:22:17
2027, >> which is this research paper you published. >> That's right. It's it's a scenario
00:22:21
forecast that sort of lays out like month by month a possible future trajectory. There was sort of like at
00:22:27
the time that we started writing it was my best guess as to what would actually happen. Obviously, there's lots of
00:22:31
uncertainty, but you know, I I I thought it's valuable to make a concrete guess just to sort of see what it might look
00:22:36
like. And at the time we were writing this, a lot of my friends in the AI industry and in nonprofits and so forth
00:22:43
that work on AI, a lot of people were saying like, "Yeah, that stuff's going to happen, but like it'll probably take
00:22:47
a couple years longer than you think." And now it's more 50/50. Especially when I go
00:22:55
talk to people at Anthropica and OpenAI, they're often like, "Yeah, no, 2027, that's basically what's going to happen,
00:23:03
just like you wrote. Why did you why did you become why did you update your timelines?" Oh, yeah. Context for this
00:23:09
is um after after writing 2027, I shifted my timelines to be a little bit more conservative. So, at the time that
00:23:16
we published, my 50% mark was in 2028, not in 2027. And then after we published progress just seemed like it was going a
00:23:24
bit slower and so I updated to 2030 which is you know it still could happen sooner could happen later 2030. Um but
00:23:32
now when I talk to people in the companies they're like it's not going to take that long. They're like oh you need
00:23:38
to shorten them again like get them back to 2027 or 2028 you know. Um so that's a
00:23:43
bit disquing. Um, again, don't know how long it's going to take, but this is the
00:23:48
stated plans of the AI companies is to do this incredibly dangerous thing, and they think that they're just a few years
00:23:53
away. So, you wrote this um report here, what 2026 looks like, and you wrote this
00:23:58
in 2021, >> and it was remarkably accurate, helped make a name for yourself amongst um
00:24:05
amongst everybody in AI. And I Which one was it that JD Vance, the vice president, read? I think it was this
00:24:10
one, wasn't it? Yeah, this one. Um and then so then you published this one AI 2027 and this was published I believe in
00:24:16
2025. >> Uh yes that's right April. >> Yeah. >> What were you forecasting in here? What
00:24:21
is what are the key things that you said in here for people that haven't read it?
00:24:24
>> The high level version of it is they automate the coding then they automate the rest of the research process. Then
00:24:30
the pace of progress accelerates dramatically. They get the super intelligence. They're working with the
00:24:34
government who specifically the president the executive branch naturally wants to control this technology and
00:24:39
otherwise wants to use it to beat China and integrate into military and so forth. By this point it's sort of doing
00:24:45
basically all the work itself. I mean it's it's super intelligent. So it's coming up with all these great ideas for
00:24:51
how to integrate itself into everything and all these new technologies it's invented and so forth and uh because of
00:24:56
the race dynamics and because of the profit motive they end up deploying it everywhere and it builds robot
00:25:00
factories. They build more robots. to build more robot factories etc. transforms the world entirely and then
00:25:06
at some point it has enough power it meaning the AIs have enough power that they don't have to pretend to to be
00:25:13
aligned anymore right um then they stop listening to orders that's the race ending of AI27
00:25:24
we also wrote a sort of different branch which is the slowdown ending which was intended to sort of illustrate the
00:25:30
concentration of power issues um that I mentioned previously. So, what if hypothetically the alignment issues get
00:25:38
sorted out sufficiently quickly? Like what if it turns out that like it's not too hard with two months of slowdown, we
00:25:44
can figure out how to make the AIS robustly do what we want um and have the values that we want them to have. So,
00:25:49
that's one possible branch. And in that branch, uh it looks pretty similar. You know, they take the jobs, beat China,
00:25:56
etc. Um, but instead of the AIS ultimately killing everyone, they create this sort of amazing utopia. But the
00:26:04
amazing utopia is whatever the people who control the AIS wanted it to be, right? And so that would be a very small
00:26:11
group of people like the president, some CEOs, etc. >> There should be a button just down below
00:26:17
here. And if it says subscribed, you're already subscribed. If it says subscriber, that means you're not yet.
00:26:23
And if you're not subscribed, please could you do us a favor and hit that button? It helps the show more than you
00:26:27
know. And according to the algorithm, you're someone that watches our show, but you haven't yet hit that button.
00:26:32
Thank you so much. >> Is there any possibility, do you think, that we never get to this thing called
00:26:36
AGI? And and how do we distinguish AGI from this term super intelligence? What's the difference?
00:26:42
>> Yeah, so the difference is that AGI is a more vague uh and weak term. So super
00:26:47
intelligence is a bit more precisely defined. It's better than the best humans at everything, faster and
00:26:51
cheaper. Um, AGI is more like it stands for artificial general intelligence, which means AI is that can do things in
00:26:57
general rather than like some specific task. >> Yeah. >> And so arguably we've already achieved
00:27:01
AGI, right? If you use cloud code or something like that, it's like it can do a lot of stuff. It's it's almost kind of
00:27:06
like a little employee that you can like have go do stuff. So it's it is quite general. It's not maximally general
00:27:12
though. It can't do everything. Whereas super intelligence by definition can do all the things that a human can do but
00:27:17
better. >> And how does this sort of overlap with robotics? Because obviously we're seeing
00:27:21
this huge robotics boom at the moment. There are some real world things that humans can still do because these AIs
00:27:26
are still stunk on my computer. >> The way people talk about this is that they basically just say we've achieved
00:27:31
super intelligence for cognitive tasks. Then you can talk about like full super intelligence that can do this physical
00:27:37
stuff. >> And are we going to get there? Are we going to get there with both? >> I think so. I mean again this is not
00:27:42
something that we can be certain about. Um you asked like is it possible we'll never get there? Yes, it's possible
00:27:46
we'll never get there. I don't think it's likely though. I think that there's nothing sort of like magical about the
00:27:52
human brain. It's, you know, um it's just a bunch of neurons. It is possible for a digital system to do similar
00:28:00
functions in the same way that like, you know, a plane can fly just like a bird.
00:28:06
Not in the same way as a bird necessarily, like it doesn't have it's not flying in the same way that a bird
00:28:10
flies, but it flies, you know. >> Um so, so it does seem like yeah, like >> seems possible.
00:28:16
>> You've written all these, you know, these research reports. You're working on another one that will be released um
00:28:20
likely on the 9th of July. You have worked inside OpenAI. You then quit OpenAI because you were concerned about
00:28:27
what was going on there and about the future of the industry. You know more than I do. Are you optimistic about the
00:28:34
future or pessimistic? Are we heading to a bad place if things don't change? Um based on everything that you know,
00:28:40
>> I think we are headed to a bad place if things don't change. Um I'm not confident in that. I would say something
00:28:45
like 70%, it's very very hard to predict of course, but yeah, it seems like the current default path is heading towards
00:28:51
a very very scary place. >> How do you contend with that personally and emotionally?
00:28:55
>> Um, it's rough. I mean, I think it it's the sort of thing that like gets me down
00:29:04
on a regular basis, but also I've been dealing with this for so many years now that I've sort of gotten used to it, if
00:29:10
that makes sense. Um, yeah. Yeah. I I'll put it this way. I would be incredibly happy if all my
00:29:19
predictions turn out to be wrong and uh an AI hits the wall, for example. >> It gets you down on a regular basis.
00:29:26
>> I used to be known as a pretty chipper and optimistic person. But um in 2020 my
00:29:32
AI timelines predictions started collapsing due to GPT3 and the scaling laws papers and um the bioankers report
00:29:40
which I can talk about if you're interested but basically some events happened in 2020 that convinced me that
00:29:45
actually this stuff was like quite plausibly coming by the end of the decade and humanity is very obviously
00:29:51
not ready for this you know in a whole bunch of different ways and so that's obviously very scary
00:29:56
>> and that's an extremely scary world because of all things you've said but but again because of this recursive
00:30:00
self-improvement where AIs can train themselves and at such point we're starting to lose hold of what's going on
00:30:06
here. I mean the AIS are already training themselves to be clear. It's more like closing the entire research
00:30:11
loop, right? Doing everything. >> Yeah. Like right right now a lot of the training data is generated by AIS. A lot
00:30:17
of the reinforcement like the grading that happens out of positive and negative reinforcement is itself done by
00:30:23
AI. >> Can you explain that in layman's terms for >> Yeah. So an important thing for
00:30:27
everybody to understand is that modern AI systems are not software in the normal sense. I mean they are
00:30:33
technically software but they're not lines of code you know it's not like some engineers at anthropic went and
00:30:40
wrote lines of code that basically says like you know when the user asks for this type of thing then go do this type
00:30:47
of thing for this many steps or whatever. There's nothing like that. Instead it's a neural net. You know
00:30:52
>> what's that? >> Well think about how the brain is a bunch of neurons connected to each
00:30:56
other. >> Yeah. >> That are firing um signals back and forth. The brain learns over time the
00:31:02
types of patterns of firing that caused success that caused a dopamine rush or various other types of feedback get
00:31:10
reinforced and fire more often. And the types of patterns that caused failure like touching a hot stove get
00:31:17
anti-reinforced, get, you know, um, destroyed so that they fire less often. And as a result of all of that, you over
00:31:24
the course of years learn to act in the world and you learn all sorts of skills and you learn world models. You learn
00:31:31
like beliefs about the world and you can sort of like mentally simulate how it's
00:31:34
going and stuff like that. So artificial neurallets are like that except artificial. So it's it starts off as a
00:31:40
giant tangled spaghetti mess of randomly generated uh artificial connections called parameters. these days there
00:31:51
might be something like 10 trillion parameters uh in the biggest AIS. So it starts off randomly generated. So, it's
00:31:59
of course completely useless. Like if you give it some input, it'll just produce gibberish as an output. But then
00:32:04
they train it. And they start with pre-training, which is where you give it a bunch of internet text, and you show
00:32:12
it the first piece of text, and you put that in as the input, and then it gives a gibberish output, and then you
00:32:20
positively or negatively reinforce it based on how accurate that output was at predicting the next piece of text. Um,
00:32:26
so it's basically playing this game of like predict the next word. >> Isn't that how it happens with babies? I
00:32:31
had a I think I had a neuroscientist tell me that babies have more neural connections um than adults. And yeah, it
00:32:37
says yeah, toddlers have twice as many neural connections as adults. And they I guess they whittle down through
00:32:43
reinforcement. >> Yep. >> We have more pathways when we're younger and just like the process of training an
00:32:48
AI, we're trained down to like remove the ones that aren't useful and build up on the ones that are.
00:32:53
>> Yeah. It's both pruning and strengthening. Okay, it seems like in humans it's actually more pruning than
00:32:57
strengthening, but it's both. Uh, and in AI it's the same thing. It's both. So the first portion of training is where
00:33:03
they train the AI to predict text, which is kind of like training it to read. Um,
00:33:08
and it a similar thing does happen in humans. So basically the the random tangle gradually takes shape and
00:33:15
gradually sort of coaleses into more useful circuitry that has stored lots of facts about the world and has stored
00:33:22
lots of skills for how to you know process information and transform it and then produce predictions. That's just
00:33:30
the first step. After they do the pre-training then they try to teach it more useful skills besides just
00:33:36
predicting text. And so, you know, by the end of the process, they've thrown lots of coding problems at it and
00:33:42
they've said like, "Here's a coding problem. Go. Here's a coding problem. Here's an environment. You have access
00:33:47
to this virtual computer. Here's like the codebase you're working with. You can write code. You can edit the code.
00:33:52
You can run the code. You can read it. You can use the internet. Go." And it does that for a while. And then based on
00:33:59
how successful it is, reinforcement happens. and they have thousands maybe millions of examples of coding problems
00:34:06
like that that they train it on and that's why they're so good at coding now. >> So what does super intelligence look
00:34:11
like in this regard? Is it just more of these connections and how would they get
00:34:14
more connections? Can you explain that to me like on >> So there's different AI models, right?
00:34:19
So there's like you know GPT3 and GPT4 and GPD 4.5 and GPD5 and GPD 5.5 and 5.6, right? Sometimes they're just the
00:34:28
same previous model but with extra training. Mhm. >> Sometimes there are a whole new model
00:34:32
that's been trained from scratch, including starting the whole pre-training process again. Over the
00:34:37
last couple years, they've done several new rounds of starting over from scratch. And typically, when they start
00:34:42
over from scratch, they make the whole thing bigger, the artificial brain much bigger.
00:34:46
>> Okay. >> Right now, they're at something like 10 trillion parameters. Back in 2020, um it
00:34:52
was more like 175 billion. >> So, we've grown like two orders of magnitude uh in six years.
00:34:58
>> Two orders of magnitude. Yeah, like two 10 x's, so 100x, right? >> So that process is continuing. Um,
00:35:06
they're also improving the algorithms themselves. So they're not literally just the same type of AI, but bigger.
00:35:12
They've also come up with all sorts of ideas for how to change the structure of the of the connections and the neurons
00:35:18
and so forth and change the like reinforcement algorithms that they're using and to
00:35:23
change the training data that they're training on. All sorts of tweaks that have made this whole thing more
00:35:28
efficient. We're literally building a brain >> basically. Yeah. As they make more
00:35:32
brains, they're getting better at making they're making them bigger and making them more efficient and so forth.
00:35:36
>> And it's literally modeled on the brain, like the way it works, right? >> It's it's certainly heavily inspired by
00:35:41
the brain, but I I shouldn't overstate the analogy. Like there's lots of differences, too. So, for example, the
00:35:46
transformer architecture um >> which is >> which is the architecture that they use
00:35:50
for for these LLMs, uh is not really recurrent. So the information sort of flows one way rather than allowing all
00:35:58
these sort of little loops on the inside. Also the the back propagation algorithm is different from the sort of
00:36:04
um learning that naturally happens in human brain. So there are some differences but yes like broadly
00:36:08
speaking uh we are sort of making artificial brains. It's kind of like for brains what like a plane is for a bird.
00:36:14
>> Mhm. That's a really good analogy. Yeah, >> that that analogy helped me think
00:36:18
through a bunch of questions people often ask about AI when they said can it be creative but actually that analogy
00:36:24
kind of helps me understand that actually that maybe that's not the question it's can it produce something
00:36:29
that you would consider to be creative because creativity is people think of it as like a process but actually it's it's
00:36:34
judged based on the output isn't it? I mean, you you can get philosophical about like, is it truly creativity that
00:36:40
they have, but you can also be like, well, I mean, just look at all the stuff they're accomplishing, you know, and uh
00:36:45
it seems like they're going to be accomplishing a lot more in the near future. >> Yeah, I do I I asked the question about
00:36:50
how this weighs on you personally because I can I can sense that you're actually personally bothered.
00:36:55
>> I mean, I think the situation is crazy. Like, first of all, it's very exciting. Like,
00:37:01
AI is really fascinating and interesting stuff. I've been following the field for
00:37:04
more than a decade now. Um, I've been part of it uh for some years and um it's really cool, really interesting and it's
00:37:11
really fun to think about what's going on inside these artificial brains and why they are the way that they are. And
00:37:17
it's really cool to see all the applications of this technology out in the world. But it really seems like
00:37:23
we're on a pretty scary path. And the more you think about it, the more worried you get. And you know, in
00:37:29
stories it always ends well, but this is real life. And I I think we have to sort
00:37:35
of stare reality in the face and tell it and realize that like it might not actually end well, you know.
00:37:41
>> Were there any recent dare I say I was going to say Eureka moments, but paradigm shifting moments where even
00:37:47
your own sort of mental model of what's going on here and how this is going to look were changed for better or for
00:37:52
worse. >> For better or for worse and probably for worse, things are kind of on track for
00:37:56
AI 2027. There are a few things that have been different. not exactly like paradigm shift differences but like
00:38:02
there have been some differences from what we expected at the time we wrote this. So the government has actually got
00:38:07
involved faster than we expected and has been more aggressive than we expected. So the export controls on mythos being
00:38:13
uh the biggest example and also threatening anthropic with uh being destroyed by the defense production act.
00:38:19
Um >> another thing that's been surprising to us is that anthropic in particular has
00:38:24
gone from second place to first place in the sort of in the race basically. Why do you think that happened? Because it
00:38:30
seemed like chat Gvt were out front and clear as it relates to OpenAI were out front and clear, but suddenly Anthropic
00:38:37
have uh lapped them. >> Yeah, I mean I guess they have um probably higher talent density um and
00:38:44
better strategy, but not by a lot, but enough to make the difference. >> Why do you think they have more talent?
00:38:52
>> Well, they don't have more compute. Like what are the inputs, right? Like they're
00:38:56
in the lead now. They used to be behind. What are the possible explanations for this? Well, it could have been that they
00:39:02
had more resources, like more compute, more money. But that's not true. They have less resources and less money,
00:39:06
right? So then I guess talent is is the next best alternative. You could maybe say strategy,
00:39:13
>> some combination of those things. Yeah. >> Something that wasn't just like the
00:39:16
amount of resources they had. >> Just like Jon Jones, where marginal improvements in your cognitive
00:39:21
performance can have a massive impact. Sometimes I podcast for 10 hours a day. Over the last couple of weeks, I've been
00:39:26
in filming for a TV show and then I have like one or two days off to get all of my work done, which means there's lots
00:39:32
of cognitive load. And so, I turned to ketones because I find myself more articulate, able to think more clearly,
00:39:37
able to work out better when I'm fueled by ketones. And so, the reason I became a co-owner of this company and the
00:39:43
reason why they now are a sponsor of this podcast is because I remember one of my team members called Cristiana, she
00:39:47
tried it once and came up to my desk and she goes, "This is the best product ever
00:39:51
made." And I think in part that's because she really cares about those cognitive benefits as I do, as John
00:39:56
Jones does, and as I think most of my listeners probably will. So, if you haven't tried these yet, all you have to
00:40:01
do is go to ketone.com/stephven and you'll also get 30% off your first subscription order. You'll get exclusive
00:40:08
Ketone IQ merch and of course, cognitive benefits that might just change your life. Much of the reason most people
00:40:16
haven't posted content or built their personal brand is because it's hard and it's timeconuming and we're all very
00:40:21
very busy. And if you've never posted something before, there's so many factors in your psychology that stop you
00:40:28
wanting to post, what people will think of you. Am I doing this right? Is the thing I'm saying absolutely stupid? All
00:40:34
of these result in paralysis, which means you don't post and your feed goes bare. I'm an investor in a company
00:40:40
called Stanto, which you've probably heard me talk about. And what they've been building is this new tool called
00:40:45
Stanley that uses AI, looks at your feed, looks at your tone of voice, looks at your history, looks at your best
00:40:50
performing posts, and tells you what you should post, makes those posts for you.
00:40:54
You can also just use it for inspiration. And sometimes what we need when we're thinking about doing a post
00:40:59
for our social media channels is inspiration. Building an audience has fundamentally changed my life, and I
00:41:05
think it could change yours, too. So, I'm inviting you to give this new tool a shot and let me know what you think. All
00:41:11
you have to do is search coach.stand stand.store now to get started. >> Uh, a friend of mine who knows some of
00:41:17
these people sat me down once upon a time in London. He's actually said this a few times to me, but I remember one
00:41:23
particular conversation where he says that some of these AI CEOs predict the probability of extinction at being I
00:41:31
think he said 7%. I don't know why I have that number in my head, but I remember it being less than 10%. And the
00:41:35
point he was making to me was that even if it was 1%, like if there was a 100 buttons on this table now
00:41:41
>> Yeah. and one of them would end the world. Would I dare press any of them? You know, um,
00:41:48
>> no, >> I wouldn't press any of them. But he made the case to me that these AI CEOs
00:41:52
are very smart and they understand super intelligence and that they think actually if there was a 100 buttons on
00:41:56
this table right now, maybe 10 of them could end the world. I've heard you say, I think it was on the the the Daily
00:42:01
Show, the interview did, you said that you think there's a 70% chance of human extinction due to AI.
00:42:06
>> I wouldn't say human extinction exactly. I would say something like 70% chance
00:42:09
that this goes horribly wrong like human extinction. But that's just one of several possibilities. But yeah,
00:42:14
basically like for example, possibly the AI take over and then don't actually kill
00:42:20
everyone. You know, maybe they do something else. Like just just because they've taken over doesn't mean they're
00:42:25
definitely going to kill us, right? They might, but they could do something else.
00:42:28
So that's what that's that's why I don't usually say like 70% chance of like actual human extinction, but 70% chance
00:42:33
of like something like AI is taking over some some sort of very big catastrophe like that that could lead to human.
00:42:39
>> I guess I got two points there, which is you've been around these CEOs. I mean,
00:42:43
you've worked for Sam Alman at OpenAI before you quit. Do you think that they think there's a chance of human
00:42:48
extinction? >> Yes. But I think that an important thing to understand is that like people sort
00:42:55
of believe what they need to believe in order to think that they're good people and that they need to keep doing what
00:43:00
they're doing. This is what rationalization is. And so I think that the tech CEOs have like genuinely
00:43:06
convinced themselves that like probably things are going to be fine and that the
00:43:10
way to make things fine is for them to keep doing what they're doing. And like they need to like make sure that like
00:43:15
you know Sam needs to make Sam's probably thinking like can't let Dario or Elon get there first. You know I know
00:43:21
Dario is thinking Sam can't get there first. Elon's thinking that like you know they they they've all probably
00:43:26
convinced themselves that like oh yeah like maybe it'll go horribly wrong but like probably it's going to be fine and
00:43:31
probably, you know, I should be the one in charge. It appears to me that Anthropic are the only ones that are
00:43:37
talking about the potential chance of extinction or a catastrophic event or um the down the real downside still. They
00:43:44
seem to be the only ones that are still publishing on it and now they're actually becoming the enemy in many
00:43:49
respects of the >> the tech industry in San Francisco. I'm watching a lot of interviews and it's
00:43:53
everyone's attacking Dario because he's saying listen things could go bad. They're calling him a doomer and
00:43:58
questioning his incentives. Even with Mythos, which is a a claude model that they started to warn the world about,
00:44:04
again, he is attacked immediately for saying that. >> Yeah. >> My question is, do you see him as being
00:44:09
slightly different from Sam in this regard? >> Yeah. I mean, it seems like Anthropic
00:44:14
and Stereo have been more willing to say and do things that are costly to their bottom line uh at least in the
00:44:22
last year or so. That's an example of it. um like I don't think that really wins them favors in the administration
00:44:28
or among their investors to say that type of thing and you know a better example is just the whole fight between
00:44:35
the department of war and anthropic was an example of them doing something that like cost them a lot of money and even
00:44:39
more importantly cost them a lot of power for something like like they could have just signed a contract you know
00:44:46
that said I really don't want to be in a situation where we're like which CEO is
00:44:50
the least bad CEO let's support that one you know like none of these people should be trusted did uh with that much
00:44:56
power basically. >> Nobody should. >> Nobody should >> regardless. >> Regardless. Yeah.
00:45:00
>> Mhm. So on this point of the buttons, you you you do believe that they think there's a credible chance of extinction.
00:45:06
>> Yeah. But they've convinced themselves that like it's probably fine and also it'll be even worse if I'm not doing it,
00:45:12
you know? Like that that's what they'll say inside the companies too. Like two people will be like, "Okay, well if we
00:45:17
stop, what about the other guys?" Like they're not going to stop, you know? >> Yeah. This is this has always been why
00:45:22
I've had this outstanding question which is how does this not go bad when human incentives seem to rule the day when you
00:45:27
look at history and all of the human incentives are saying well if you you're damned if you do i.e. you're damned if
00:45:33
you carry on developing these bigger and bigger and bigger AI brains but you're also then damned if you don't from a
00:45:38
geographical perspective because the United States will lose to that country or this company will lose to that
00:45:43
company. So when you just look at human incentives and goes how does how does if
00:45:46
just purely incentives and disincentives how does this end? Well it carries on going. seems like it. I mean there there
00:45:51
is a caveat to that which is a hopeful caveat which is that first of all if the world wakes up to all of this then there
00:45:58
can be a more serious conversation about regulation and international treaties and things like that and that can change
00:46:05
the incentives right so the government could come in and say like actually here are some rules that you all have to
00:46:11
follow and because they're rules that you all have to follow then you're not incentivized to like break them anymore
00:46:18
because you get punished if you break and everyone else is also following them too and so you know it's fine. So so
00:46:24
there is that sort of like ray of hope that like we can change the incentives if the government and especially the US
00:46:31
government but then later other countries act to to change the incentives but that's not going to
00:46:36
happen until people sort of wake up to all of this. The second thing is that even individually at some point,
00:46:43
you know, Dario or Sam or Elon might realize that like actually it's like not even in their own interest to to keep
00:46:50
racing unilaterally. And it the problem with that is it's only if it gets extremely obvious and
00:46:56
extremely dire. So like in in AI 2027 in that scenario, there's this choice point
00:47:01
that I mentioned and in one case the AI are misaligned, in the other case the AIS are aligned. At that choice point,
00:47:08
we have like one branch that depicts the the misalignment ending and one branch that depicts like they they slow down a
00:47:12
bit and solve the alignment issues. >> The instigator for that choice point is they see some evidence that their AI
00:47:18
might be misaligned and plotting against them, right? So, if you actually see that evidence,
00:47:23
>> then it's like, oh gosh, uh maybe we shouldn't put it in charge of everything
00:47:28
and let it rip, you know, because that's evidence is staring us right in the face
00:47:33
that it's it's untrustworthy, you know. But if they don't see that sort of very clear evidence, then I think they're
00:47:40
going to convince themselves that they need to keep going, you know, but maybe they will see very clear evidence like
00:47:44
that. In which case, even if we don't have regulation, they might just sort of voluntarily stop. Um, so that's the
00:47:49
second ray of hope. Like overall, I don't think that we're like definitely doomed. You know, like I said 70%, but
00:47:55
like I could see it working out pretty well as well. >> H what about uh jobs? >> Yeah.
00:48:04
So I think I think I'm excited to at some point get into the new thing which is the more optimistic positive vision.
00:48:10
>> Uh and that will have a lot to say about this because in the in the in the prediction you know in AI 2027 by the
00:48:18
time everyone loses their jobs there are worse things happening or like it's it's
00:48:22
kind of like too late by that point. Um, but yes, like once if I mean just just think about it. If the companies do
00:48:28
manage to build super intelligence, then by definition they're going to be able to take almost all the jobs or all the
00:48:34
jobs, right? Because it's better, faster, cheaper than the best humans at everything.
00:48:38
>> And that again, the timeline is by the end of sort of 2030, you reckon you think super intelligence might arrive?
00:48:43
I'm trying to think about when we could start to see job displacement in the economy.
00:48:47
>> We're already starting to see a little bit of it now, but not very much. >> Why?
00:48:50
>> Um, cuz the AI aren't good enough yet. like they they're they're they're impressive, but they're not like they're
00:48:56
not just a drop in replacement for a human worker in almost any field. >> And do you think that'll be sudden?
00:49:02
>> I think it'll be sudden because of the intelligence explosion dynamics or recursive self-improvement dynamics. So
00:49:09
you can imagine a different world where it's gradual. >> Mhm. And and this is this is maybe how
00:49:14
it is in a lot of science fiction is you know the AIS gradually get better at a bunch of things and you know they
00:49:21
gradually automate like this one industry like pharma then they automate like you know steering drones then they
00:49:27
automate like driving cars or something like that. Um but what's different about
00:49:32
the real world is that the companies have converged on this strategy of automating themselves first, you know,
00:49:39
automating the AI research process. And so if they're allowed to continue with this
00:49:45
strategy, we're not going to see like, you know, the robo taxis and like the plumber
00:49:52
robots and, you know, the lawyer AIs. We're not going to see that sort of like broad diffusion of AI into the economy
00:49:59
happening first because that's not what they're focusing on first. They're focusing on automating themselves,
00:50:05
automating their own research so that they can do everything that they're doing faster. And they want that to sort
00:50:10
of get going and get to, you know, very high levels of intelligence, very high levels of general intelligence. Um, and
00:50:18
then deploy more out into the economy, right? So by the time it's actually coming for like all these different jobs
00:50:26
they will have had fully autonomous AI research happening for months maybe years you know and that means that like
00:50:34
the AIs will be vastly superhuman at AI research and probably also vastly superhuman at lots of other things just
00:50:39
as a side effect you know if you're wondering what this looks like well we wrote about what it looks like it's sort
00:50:46
of like this this wave smashing through the economy after they do the intelligence explosion internally.
00:50:52
>> What I'm hearing there is that because the AI will be able to improve itself and train itself, it'll be getting
00:50:58
better at everything at once and then it will be released at kind of once. Is that accurate?
00:51:03
>> But it's it's not it's not even exactly that because even if it's mostly just
00:51:06
getting better at the things that it's doing like re research that'll have some spillover effects to other skills as
00:51:12
well and then when it turns to focusing on the those other skills it'll be able to do them very fast.
00:51:17
>> What jobs remain in such a scenario do you think? I think that's actually a political
00:51:22
question, not a technical question >> because >> because on a technical level, all the
00:51:26
jobs can be done by the AIS if they've reached that level. And so it's a question of what jobs are allowed for
00:51:34
them to do >> and what kind of jobs wouldn't be allowed, do you think? >> That depends on who's in charge. So
00:51:39
there'd be some sort of political conversation about like what we're going to allow and disallow.
00:51:43
>> I mean, in this scenario, the humans are still controlling them. The AIs >> depends on what you mean by control,
00:51:47
right? So there's like there's do the AIs actually have the goals and values that you want them to have and are they
00:51:53
going to robustly do that and behave as intended into the future and then there's like are they obeying your
00:51:58
orders for now? >> Are they obeying the orders is really what I'm saying. >> Yeah. So like even in AI27 in the
00:52:03
scenario where the AI take over and kill everyone, there's a period of like several years where they're still
00:52:07
obeying orders and they're, you know, taking some jobs but not other jobs and they're helping to make better weapons
00:52:14
that the US government can use to like do its arms race with China and so forth. And that's why they're able to
00:52:22
get so much power so quickly is because the governments and the corporations and
00:52:27
so forth trusts them and is deliberately deploying them into all of these positions because it thinks that things
00:52:34
are fine. But because these things are neural nets, you can't just like look inside and see what it's really
00:52:40
thinking. You can't really tell. I think this is a really important point because
00:52:44
unlike software we can look at the code and see what's going on theoretically with AI you're saying that we don't know
00:52:50
what why it's making the decisions that it's making because we can't get inside.
00:52:53
>> One note of optimism is that it doesn't necessarily have to be that way. Like
00:52:57
there's a a sub field of machine learning called mechanistic interpretability and a broader subfield
00:53:02
called interpretability more generally that's trying to solve that problem and trying to take these these trained
00:53:08
artificial neural nets and piece them apart and understand like how the information is flowing and how the
00:53:14
decisions are being made so to speak. Um the problem is just it's a very inherently hard problem. If you have 10
00:53:19
trillion connections to look at, you can look at any particular group of them and
00:53:24
be like, okay, so this is how like this particular connection works, but like how do you get a sense of the whole, you
00:53:28
know, how do you get a sense of like >> what's happening at a high level? And the answer is, well, it might be
00:53:32
impossible, but people are working on it and they are making progress and if they
00:53:37
can make enough progress, then we're in a very different and much brighter world. I think that it would be much
00:53:42
less likely for us to get into those loss of control scenarios if we could just actually see what our AIS were
00:53:48
thinking and why and how at any given time, right? >> Yeah. >> So, we would still have the other
00:53:53
problems to worry about, but at least we could mostly solve that one. >> It is pretty crazy to think that we're
00:53:57
building a technology, a brain that we don't understand. >> Yeah, it's pretty crazy. I mean, it's
00:54:01
one of those things where like >> in a movie, like a sci-fi movie, a bunch of scientists stood around this big
00:54:06
brain and they're all just like they're making it more they're feeding it. >> Yeah. Yeah,
00:54:09
>> but they don't really know what the >> Yeah, I mean it's it's kind of just like
00:54:12
obviously a dangerous thing to be doing. >> Um, but we're doing it anyway because of
00:54:16
this history of how the field has developed in the last 10 years where, you know, people were like, "Oh, wow.
00:54:21
Yeah, that's obviously dangerous. Oh, no. What if someone else did it and did a bad job of it, therefore we should do
00:54:26
it and do a good job of it." And now they're in this race where where they're racing each other. And they're also
00:54:32
under all sorts of political pressure to like pretend that it's not as bad as it
00:54:35
seems because they don't want to like anger their investors. They don't want to anger the White House.
00:54:41
>> One of the the key questions we had from our audience was which and I kind of
00:54:44
asked you this in part, but which jobs are genuinely likely to survive AI and what skills should people/ students
00:54:51
focus on over the next 10 years? That's kind of like like imagine if you were someone living in Mexico in like 1500
00:55:01
and then you hear that like the concistadors are coming. You could be asking yourself like okay
00:55:07
well what sort of job should I be switching to to like survive this transition. But like you have a lot more
00:55:13
to worry about besides that. But yes, I think I would say that like if we manage
00:55:17
to avoid the loss of control problem and we end up with humans still in charge of
00:55:22
the AI and humans can like say what the AI's goals and values are supposed to be
00:55:25
even as they become much smarter than humans and even as they run the whole economy then probably there will be
00:55:31
regulation that protects some areas and you can try to guess at what those areas
00:55:36
might be maybe stuff that's more like like like judges potentially. What about podcasters?
00:55:44
>> Be honest. >> Probably not podcasters, I think. Um, stuff like, you know, being a nanny,
00:55:51
maybe, right? Like, I think it's even if there's a robot nanny that's like really
00:55:55
really good, I think a bunch of people might prefer to have an actual human because they might be creeped out by the
00:55:59
idea of a really good robot nanny. So, you can sort of you can sort of reason like that. There's also like stuff that
00:56:05
might be legally protected, like maybe judges, for example, like are going to be legally required to be humans and not
00:56:10
robots. Some people say though there's going to be so many jobs created that we can't foresee right now like there was
00:56:14
in the industrial revolution or the internet boom or whatever. >> The problem with that is that um past
00:56:21
technological advancements have been more narrow. They've like automated some things but not everything. But we are
00:56:28
talking about a hypothetical future situation in which everything gets automated. So there isn't any new job
00:56:34
that you could do that the AI couldn't also do except if it's like protected by regulation or something. That's that's
00:56:40
that's also a thing. But so like for example, right now there's this sort of like cycle where
00:56:47
you know the AI has learned to do a certain thing like write copy or like draft code or like debug something and
00:56:56
then humans who used to do that thing switch to managing AIS or switch to doing the other stuff that the AIS can't
00:57:02
do. And that's why there's been this dynamic historically of, you know, new jobs opening up and people flooding to
00:57:09
them. But if it gets to the point where the AI can do everything that humans can
00:57:13
do and better and faster and cheaper, then whatever that new job is that you might have switched to, like the AI can
00:57:17
switch to that, too. And they'll already be better at it than you. >> Because we haven't seen widespread
00:57:22
unemployment yet in the economy, do you think people are getting a little bit complacent? Because what I'm seeing on
00:57:26
my timeline is a lot of people saying, "I told you so. I told you everything would be fine." And when you look at the
00:57:31
the US unemployment rate, currently the it's flat to slightly down. If you look at the UK, it is up. The trend is up
00:57:39
compared to last year. We're at about 5% unemployment. The US is at 4.2% unemployment.
00:57:44
>> Yeah. Basically, nobody has said that there would be mass unemployment by now.
00:57:48
Or at least we didn't say that, you know, and we were historically one of the more bullish people on AI progress
00:57:53
in AI 2027 because of the dynamics that we just described. The mass unemployment
00:57:57
doesn't happen until 2028 or 2029 after they already have super intelligence because again the companies aren't
00:58:03
trying to cause mass unemployment as step one. That's like step three after you know it's like step one automate
00:58:09
themselves. Step two have this recursive self-improvement to get to super intelligence. Step three expand out into
00:58:16
the economy and automate everything. And so this is really unfortunate from humanity's perspective because one might
00:58:23
have hoped that if there was this broad wave of automation going through the economy, people would sit up and pay
00:58:29
attention and think about where all this is headed and demand good regulations from the government. But that's not
00:58:37
actually the strategy the companies are taking. you know, they're going to be getting the super intelligence first and
00:58:40
then doing the broad wave of automation, which means that by the time they're actually doing all of that, uh, well,
00:58:47
it's already going to be moving very fast and the AI will already be very powerful.
00:58:50
>> In your 2027 report, so you wrote that in 2025, but it is called AI 2027. You
00:58:56
said that in mid 2025, we'd have the autonomous employee, which is sort of like AI agents taking instructions over
00:59:01
Slack or Teams. >> That happened. I've actually got an AI agent in my WhatsApp, which I talk to,
00:59:07
of course. I've got Clawbot exploded obviously around the world and and now um you know Claude have talked about uh
00:59:12
their new Slack integration but lots of people are using agents now and that happened I'd say for us at the we really
00:59:18
sort of caught on to it the the start of 2026 you also said by by 2026 companies
00:59:24
begin replacing entire corporate departments with AI agent subscriptions 2027 the final job AI automates the job
00:59:30
of the human AI researchers themselves and begins the machine learning research to upgrade and build the next generation
00:59:35
of AIs. >> Yeah. Yeah. So again timelines we are uncertain about how long it will take to
00:59:41
achieve these milestones in this scenario they happen at those times but by the time we had actually published
00:59:48
the scenario our timelines had shifted back a little bit specifically mine had so like my 50% mark was 2028
00:59:55
>> for that for the full automation of AI research milestone not 2027 uh and then other people on my team had
01:00:02
more like 2030 2031 things like that so I I I kind of want to Excellent. >> Maybe try to illustrate this with, you
01:00:08
know, we have like this probability distribution. It's like a smeared out probability mass and like the 50% mark
01:00:14
is this particular year, but there's like a lot of possibility that it happens >> years earlier or years later, right?
01:00:22
>> What is this AI 2040? >> So AI 257 was a best guess prediction as to how things would actually go.
01:00:28
>> Yeah. >> AI 2040 plan A is our recommendation for how things should go. So we called it AI
01:00:34
2040 because in this scenario uh they build super intelligence in 2040 instead of much sooner because they delay
01:00:42
things. >> Why do they delay things? >> To manage the risks and make sure that power is distributed equitably. They
01:00:48
basically like regulate AI development so that it still continues but at a slower more
01:00:54
reasonable pace in a more transparent and safe way and spread out over more countries and companies and as a result
01:01:01
they get to super intelligence in 2040 instead of in say 2030 and then we call it plan A because well it's our
01:01:09
recommendation like we've we've come up with a plan for what government should do and uh the scenario is an
01:01:16
illustration of what it might look like to imple ment that plan in a similar way
01:01:19
to how AI27 is kind of an an illustration of what it might look like to do what the companies are currently
01:01:25
planning to do. That makes sense. >> And is this wishful thinking or is this what you think is going to happen?
01:01:31
>> No, it's definitely not what we think is going to happen. >> It's not what you think is going to
01:01:34
happen. >> No. No. What we think is going to happen is still something more like this,
01:01:38
right? We we don't expect the world to listen to us, right? this is our recommendation, but we mean we hope that
01:01:44
that people do something like this and we think it's possible, but it's not our like prediction for what's going to
01:01:49
happen by default, you know. So, I do want to run through the plans, the potential plans and also plan A, but um
01:01:56
just to close off on how things might look after the year cuz I think I wanted to touch on robotics too and I've got
01:02:00
this graph here which talks about share of labor output. >> Yeah. >> Um which I found to be quite striking. I
01:02:06
I've been sat here wondering as an employer who employs hundreds and hundreds of people
01:02:10
>> when when all this stuff is going to happen and you know we're still hiring more people as things stand there are
01:02:16
some roles where our consideration is changing shifting considerably >> and I'd have to say that you know we're
01:02:22
probably in a phase where our teams are AI powered and they're using agents to do some of their work now
01:02:27
>> but I'm wondering as an employer like when is it when does this happen? >> Yeah, great question. So if we could
01:02:33
maybe zoom in on this a little bit. >> Yeah, we'll put it on the screen. Um, so this is in the AI 2040 plan A scenario
01:02:39
and notably in that scenario there's significant regulation introduced in 2029 that slows down the pace of AI
01:02:46
development. In the scenario, they do that sort of at the last moment. So in the scenario, if they hadn't done that,
01:02:52
then it was about to take off similar to how it does in AI 247. Um, but as you can see, like in the scenario, there's
01:02:59
still a bunch of jobs at the point that they implement it. And this gets back to
01:03:04
what I was saying earlier is that if you wait until most people have lost their jobs to regulate the AI companies,
01:03:10
that's already too late because they will probably already have super intelligent AI by then because their
01:03:16
strategy is to first get super intelligent AI and then do all that stuff. >> And I think you say that it would
01:03:20
collapse the economy and cause even more harm to suddenly regulate something that
01:03:23
all of us and all of our lives were then at that point relying on. >> Oh, but it's a risk well worth taking. I
01:03:28
mean, we it's true that right now a lot of people use AI for a lot of things, but like if we could somehow slow or
01:03:34
halt AI development now to set up a better way to do it, that would be well worth it. Um, even though there would be
01:03:40
significant costs, >> but you can't over here, right, can you at this point where AI and robotics are
01:03:45
doing most of the labor output. >> That's right. But in but in but in in this scenario in the AI 2040 plan A
01:03:50
scenario they put in the regulations in 2029 >> and then they slowly and carefully
01:03:54
develop AI in a way that avoids all the problems which we can get into in a little bit. And so eventually, yes,
01:04:00
eventually the AIs take the jobs. Eventually, basically the whole economy is run by AIs and robots, but it it
01:04:06
happens gradually over the course of the 2030s >> instead of happening in this sort of
01:04:12
crazy shock, you know, a year later, >> right? >> Because in this scenario, they don't let
01:04:18
the companies recursively self-improve and get to super intelligence as fast as possible. Instead, they regulate AI
01:04:24
development so that the core capabilities of the AIS are improving at a more reasonable pace and also in a
01:04:31
more transparent way so that the scientific community can see what's going on and help make it safe.
01:04:36
>> But it's uh I guess I noticed here that in both your scenarios eventually AI and
01:04:41
robotics do pretty much all the jobs. >> Yes. >> So you kind of side there with Elon when
01:04:46
Elon says that working will be a choice. Uh >> because I mean what happens? >> Yes. I mean if it by definition if it
01:04:56
can do all the things then it can do all the things. I think that there's a question of like should we allow there
01:05:01
to be AIS that can do all the things right? Some people think that the answer is no and we should just shut it all
01:05:07
down and prevent these types of AIs from being created in the first place. And we're actually kind of sympathetic to
01:05:14
that. We we have our Should we bring out the plans diagram? >> Yeah. >> Thanks. Yeah. So
01:05:21
our scenario is called AI 240 plan A. It's a scenario in which they slow down AI development to make super
01:05:27
intelligence happen in 2040 instead of earlier. And plan A is our recommendation. So this is sort of
01:05:31
illustrating our recommendation. But for comparison, we made like mini scenarios
01:05:35
illustrating different alternative plans which we call plan S, plan B, plan C, and plan D. Plan D is basically the same
01:05:44
thing that happens in AI27. Like the race continues. there's very little regulation. Um, you can read about that
01:05:50
in AI27. Plan C also very similar to what happens in the slowdown ending of AI27 where they solve the alignment
01:05:56
problems. So, in that ending, they like slow down a little bit, pivot more resources to AI alignment and AI safety
01:06:04
research, get lucky and succeed, and now they have aligned AIS and then they speed up again and take all the jobs and
01:06:11
beat China and all those things. Plan B is it's kind of like plan C in that well
01:06:21
basically in plan B you're uh being more aggressive towards China and you're like
01:06:26
taking actions to sabotage or cyber attack them to like keep them behind so that you have more breathing room to to
01:06:32
solve the alignment problems yourself. Plan A is our recommendation. It's uh domestic regulation and then an
01:06:39
international deal to continue building AI, but in a much better way. Plan S is shut it all down. If you want to have a
01:06:47
future where there aren't AIs running around that can do everything better and faster than humans, you kind of want
01:06:53
something like plan S. >> What What do you want? >> Plan A is our recommendation. I think
01:06:59
that I'm sympathetic to plan S, but for reasons we explain, we recommend plan A instead. And what do you think is most
01:07:05
probable if you're being honest? >> Plan D, >> which is that they just >> type of thing where they keep racing,
01:07:11
they don't really slow down significantly. Um, and uh things happen extremely fast. The diagram sort of
01:07:19
explains like roughly the reasoning behind this too. So like there's this high level thing of like do you want to
01:07:24
keep racing as fast as possible to make the AI smarter and smarter to put them in charge of more things so that we can
01:07:30
be China? You know, if you're happy with that, then you can get down into this variation of optins here. If you are
01:07:39
worried about that, well, you get to something like this. There's more different options besides these, but
01:07:44
this is kind of like the ones that we could compress onto a screen. >> Do you have children?
01:07:51
>> Yeah, I have two children. It's kind of sad. Like I think that one way or another this
01:08:00
will probably all be over by the time they're old enough to join the workforce. So I don't think they'll ever join the
01:08:06
workforce. >> When you say this will be all over by the time they join the what do you mean
01:08:10
by this will be all over? >> So these milestones that I described like AI is automating the AI research AI
01:08:17
is getting super intelligent. Um AI is then exploding out into the economy, taking the jobs, building robot
01:08:24
factories to build more robots, to build more factories, etc. GDP starting to go
01:08:29
vertical. That sort of thing is what I mean like all of those events transpiring. Maybe there's like, you
01:08:34
know, 10 20% chance or something that hits the wall and and none of this comes to pass even if you don't do anything.
01:08:43
>> How old your oldest? >> Six. >> Six. Boy, girl, >> girl, >> girl. So your daughter comes to you and
01:08:49
says, "Dad, what shall I um what shall I study in school?" >> I mean, again, like if these radical
01:08:55
transformations happen, then the world would just look completely different and what sort of jobs you set yourself up
01:09:01
for basically won't matter that much probably. I would say um that the thing to do is well a try to make it actually
01:09:09
go well. Like if you can exert any influence at all on history and how this all develops, you should be trying very
01:09:14
hard to steer the future in better directions. And then separately from that on a personal level you should
01:09:20
focus on well being a good person and doing things that are sort of good in the for their
01:09:26
own sake rather than good because they'll set you up for later employment because that later employment is going
01:09:31
to be very uncertain. Um basically >> Elon talks about this age of abundance we're heading towards
01:09:37
age of abundance. >> There'll definitely be abundance. The question is who controls the abundance
01:09:43
and what do they do with it? Right? Are the AIs controlled by anyone or are they
01:09:48
doing their own thing? And then if they are controlled by people, who controls them and what do they do? And what's the
01:09:54
sort of like political structure governing how they make those decisions? >> I think it was Jeffrey Hinton that said
01:09:59
to me, he said there's no example in nature where a more intelligent species is has less control than a less
01:10:09
intelligent species. Thus saying that we're quite arrogant to think that in a world where there's this artificial
01:10:16
brain that's a gazillion times the size of mine that I'm going to give it orders.
01:10:20
>> Yeah. I mean that that's the thing is I I think it's like that should be our default assumption is that like well
01:10:26
there's these brains. We can't see exactly what they're thinking. We're going to make them smarter than us and
01:10:31
put them in charge of everything >> and then we're going to give them bodies. >> Yeah. And then they're going to be
01:10:35
autonomously building new factors and so forth. And like how is this supposed to
01:10:38
end well again? Like isn't this just exactly like us picking a new species that's then going to out compete us when
01:10:45
it doesn't need us anymore? Like I think that is just the default trajectory. Now
01:10:50
there's a whole argument we can get into about like ways that we could get off of
01:10:52
that default trajectory. So for example, there's research into interpretability that I described previously. And if that
01:10:58
research bears fruit, then you will be able to actually see what they're thinking and then that would be an
01:11:02
excellent tool for shaping them and controlling them and making sure that they do what we want. Right? there's
01:11:07
other sorts of um AI alignment research agendas that are making progress and if enough of those agendas succeed
01:11:14
sufficiently we can avoid this problem of course also there's the regulatory side too where like part of what makes
01:11:19
this difficult is that we're building these AIs in race conditions you know like the the companies are secretive
01:11:25
about their recipes for making these AIs because it's secrets that they want to protect so that other people can't copy
01:11:31
them and so a lot of it is happening you know behind closed doors only a few people can really see the recipes that
01:11:38
they're using to train these AIs and and so forth. And then often times when the
01:11:42
AI behave in unexpected ways or even just like blatantly misaligned ways, sometimes that information doesn't
01:11:48
really flow out to the public because the companies are not really incentivized to tell everyone about how
01:11:52
they messed up and how their AI is evil. It's just not very conducive to scientific progress on these issues. If
01:11:58
the regulatory system was different, then perhaps we could be in a better situation, make faster progress. Also,
01:12:03
of course, we wouldn't be planning to put these AIs in charge of everything as fast as possible, and we wouldn't be
01:12:08
planning to like let themsel improve, you know, like the these are choices that we could not make, you know?
01:12:17
I don't speak Vietnamese, but this show can because of AI video technology from our sponsor, Hey Jen. I get messages
01:12:23
every single week from those of you listening to the D of Co all around the world, and you express how much impact
01:12:28
it's had on you and your life. And if that's true, then those conversations shouldn't only reach people in English.
01:12:33
Ken can take one recording of me and deliver it in any language while keeping my voice, timing, and expressions
01:12:40
intact. But you don't need a studio like this to make it work for you. Record 15
01:12:45
seconds of yourself and get an AI avatar that delivers studio quality video in over 175 languages. We're up to 20
01:12:53
languages now, and we're not the only ones using it. Hey Jen is already used by 30 million people, including 85% of
01:13:00
the Fortune 100. Whether you're building an audience on social media, launching an online course, or rolling out
01:13:05
training across your team, check out Heyen now. Your first three videos are totally free at heyjen.com/doac.
01:13:13
That's hygen.com/doac. See you there. Ilia was, as you said, he was one of the leaders at OpenAI and he left and he
01:13:22
started his own company now, Safe Super Intelligence. very curious name of a company, Safe Super Intelligence, after
01:13:28
leaving OpenAI. Did you ever get to work with him? >> Uh, I wasn't directly working with him.
01:13:32
I had a couple chats with him. >> Do you think he's he's genuinely concerned as well?
01:13:36
>> I think he is, but I think he's I think he's similar to these other CEOs where
01:13:43
I mean, just think about the sort of incentives that they're under, right? like they can sort of see the problem and
01:13:49
then they can be like okay but like if I don't if I stop if I quit my job and or
01:13:55
do something else that's not going to solve the problem cuz the other CEOs are going to keep going and even if all of
01:14:01
us didn't go then maybe China would keep going. So like man seems like this is just going to happen one way or another
01:14:05
whether I do anything about it or not. I guess I should be involved, you know, and like maybe I can make it go well.
01:14:11
And at any rate, like I don't want to be out in the cold while these other people
01:14:14
I don't trust are in charge of everything. So they all sort of like reason through all of this and then
01:14:18
convince themselves that like the thing to do is for them >> to build it and to do it better. And I
01:14:23
think Ilia is just the latest example of this. Elon's another example. Dario is another example.
01:14:29
>> You know, arguably OpenAI at the beginning Sam was an example. Although like Elon and Dario were at OpenAI early
01:14:34
on. So >> what do you think they should all do then? So I think what should happen is
01:14:38
some sort of international regulation or at least domestic regulation similar to
01:14:42
what we described in plan A. >> Okay. So walk me through plan A. >> Yeah. So in this scenario, AI takes
01:14:49
longer to the to get to recursive self-improvement and full automation of AI research than it does in AI 27. We
01:14:55
figured that we should try to illustrate like a range of different possibilities
01:14:58
because we do have those sort of uncertainty intervals. So we chose 2030 as the moment when full automation would
01:15:06
finally be achieved and things would really kick off. And then working backwards from that, when's the last
01:15:11
moment you could really have good regulation? 2029. So in this scenario, AI progress slows down a little bit
01:15:17
naturally and the AI companies keep keep racing, but they don't quite succeed in
01:15:22
automating uh themselves in 2027 or in 2028 or in 2029, but they're getting really close and they're going to do it
01:15:28
in 2030. And then in 2029, the government steps in and regulates them. What regulations do they do? Well, they
01:15:35
basically just shut it down temporarily. >> Can I ask um how does the elections overlay with your
01:15:42
time frames here? Because there's going to be a big election, isn't there, in 2028?
01:15:46
>> And it seems now that sentiment has really really turned against AI in in sort of in the general public and that
01:15:51
it will be one of the big ticket items on the on the ballot. >> We think that it'll be maybe the most
01:15:56
important issue in the presidential election in 2028. Um, I think a lot of people mo most people will be quite
01:16:02
concerned about where things are headed. And that's part of why we we chose to depict things the way they were doing in
01:16:08
this scenario because that helps explain why they might do this sort of regulation in 2029 is that the voters
01:16:13
have been demanding it and the presidential candidates have been promising it. >> And in this scenario and in 2027, would
01:16:18
the general public have felt the consequences of AI much more severely than they have now by then?
01:16:23
>> Yes. Although still even in 2029 in this scenario they still mostly have the jobs
01:16:28
as as depicted here. Right. So in in 2029 in this scenario lots of jobs now involve managing AI agents. You you
01:16:35
mentioned you have an AI agent right? Well in 2029 in this scenario the AI agents will be much better. Still though
01:16:41
not enough to just completely do everything. You know that was the sort of thing that would come in 2030
01:16:45
>> in this in this timeline. Again we're uncertain about timelines. Things could
01:16:49
go faster than depicted in this scenario. And in fact, I think things probably will go a bit faster than
01:16:53
depicted in this scenario, but we're uncertain. We already did the very fast timeline scenario, so now we're doing a
01:16:58
slower timeline scenario. But maybe we should talk about the high level goals. So
01:17:03
they want to have AI continue, but in a slower pace. >> Who's that? >> So they can make it safe. The
01:17:08
politicians, you know, the president and the people who voted for the president and, you know, the heads of other
01:17:13
governments and so forth. So goal one, slow things down. Um, goal two, make it more transparent so that the scientific
01:17:21
community can catch up to this stuff and make more progress and also so that we don't have to take the company's word
01:17:26
for it when they say that their systems are safe and when they say that they haven't, you know, put in any biases
01:17:31
into their systems, for example. That's a concentration of power issue. We also want to avoid a situation where there's
01:17:37
an intense concentration of power. So in addition to these the transparency and the slowdown, we actually think it's
01:17:44
actively good for there to be multiple AI companies across multiple different countries that have similar levels of
01:17:49
very advanced AI capability and for there to be like broad diffusion of AI into society rather than you know a
01:17:57
single mega project that has all the best AIs for example. And the nice thing about that is you kind of get that by
01:18:02
default if you do the first two things. If you slow it down and if you make it more transparent, then that means
01:18:07
there's breathing room for other projects to sort of catch up, right? And the transparency just like literally
01:18:13
helps them catch up because then they can like copy copy some of the ideas. And then I think the fourth thing would
01:18:18
be reversibility. So in what follows in the scenario, we are going to be building up a lot of data centers and a
01:18:24
lot of robots. We're going to be transforming the world at a in a sort of like slower pace, though still a very
01:18:30
fast pace, but slower. And if things go wrong and the deal breaks down and everyone starts racing each other again
01:18:36
to get to super intelligence as fast as possible, that would be very scary. And so the fourth principle is basically
01:18:43
build the new data centers in such a way that if everything breaks down and everyone starts racing again, the newly
01:18:49
built data centers get destroyed so that we're sort of back to square one again instead of in an even worse race where
01:18:55
there's even more AI and robust and compute everywhere. Um, so I can sort of walk you through the timeline if you're
01:19:01
interested. Sure. >> Or the president talks to China, talks to the leaders of a bunch of other
01:19:05
countries and says, "We're going to basically halt AI development until we can figure out a plan for how to do it
01:19:12
in the in ways that achieve these goals." So they basically send inspectors to each other's data centers.
01:19:18
Like Chinese inspectors come to US data centers, US inspectors go to Chinese data centers and verify that they are
01:19:23
doing inference and not training. developing new AIS that's that involves training them. But just taking existing
01:19:31
AIS and using them to serve customers that's called inference. And so the sort of like solution they come up with here
01:19:38
in this scenario is will allow them to keep doing inference but not training for now until we can get the new
01:19:44
training data center set up. So they retrofit the existing data centers to serve inference. People can still keep
01:19:51
talking to their AI agents, but they're going to stop getting better and better for like six months to a year while they
01:19:57
build the new data centers that are going to be the transparent data centers. And that's where the training
01:20:01
is going to happen. Once they get those new data centers set up in 2030, then AI
01:20:07
research continues. This is a bit spicy. We advocate for total research transparency which means that on the
01:20:12
training data centers that are training the new models they basically have to publish everything which means you get
01:20:17
to see all the details of the recipes for training these models you get to see the architecture etc. We think that's
01:20:23
sort of open science is really important for solving the alignment problem fast enough because you don't want to have
01:20:28
these sort of biased companies making the decisions about whether the AIS are safe. Um and we also think it's
01:20:34
important for just good regulations more generally because right now most of the
01:20:38
expertise in the world on AI is sort of concentrated in Silicon Valley and the the governments in particular kind of
01:20:45
are don't really understand AI that well and imagine an alternative instead of total research transparency you had like
01:20:52
an auditor system where the government says here are some rules for how to make the AI safe and then we're going to have
01:20:58
like an agency that like goes into the companies and asks them questions and tries to make sure that they're
01:21:02
following the rules that creates this sort of adversarial dynamic where the company is incentivized to like fool the
01:21:08
the regulator, you know, and and also if they if they discover some new problem that's not even on the government's
01:21:14
radar, >> they're might be incentivized to like not tell the government about it, right?
01:21:18
So, if you have the total transparency, it helps the government make better decisions faster, but
01:21:22
>> it kills that competitive advantage. >> Yes, Dropbox's not going to like this.
01:21:26
You know, OpenAI is not going to like this. This would be uh probably bad for the valuations. I don't think it would
01:21:31
kill them completely, but it means that it would commoditize more, right? So, it
01:21:36
means that there'd be like a bunch of AI companies that would catch up to the frontier. They would train AIs that are
01:21:41
like roughly similar, roughly equivalent. They could still make money by doing that and then selling their
01:21:46
AIs, but they wouldn't have a monopoly. They wouldn't have anything close to monopoly, which I think is good for
01:21:50
humanity, although it's bad for the bottom line of those particular companies. Notably, it's good for the
01:21:55
bottom line of lots of other companies. Like if you're a company that's behind and you don't you're not anthropic or
01:22:00
you're not open AI then you would love this because this helps you catch up you know or this this helps you to like um
01:22:06
capture more of the value from the chips you're selling for example or from the like downstream product that you're
01:22:11
making >> and by 2031 then you have 1ifth of all cognitive labor done by AI. >> Yeah. So what's happening here is that
01:22:19
we're imagining that the government of the United States and the government of these other countries that are involved
01:22:23
in this agreement that are sort of implementing similar regulations. Um they don't have to be exactly the same.
01:22:28
Uh but that's another thing that's nice about the transparency is that if you have this sort of transparency then if
01:22:34
two governments are implementing different regulations like if one of them is like telling their companies to
01:22:40
go slower or like banning more stuff than the other one is they can both see >> Yeah. like, oh, you're letting them do
01:22:46
that sort of thing and you're not like maybe we should let them do this too, you know? So, it helps to sort of
01:22:52
naturally equalize the regulations to some extent without having there to be a central power that just gets to make
01:22:58
regulations for everybody. >> Mhm. >> So, anyhow, we're imagining that when they when they get this transparency set
01:23:03
up, they basically agree to ban the dangerous stuff to allow the not so dangerous stuff. And there's a constant
01:23:09
ongoing conversation about like, well, what's dangerous and what's not? What should we ban? What should we allow?
01:23:13
What about this country? what about that country? That conversation evolves over
01:23:16
time, but the gist of it is, at least if they do it the way that we recommend it,
01:23:20
is that they don't do an intelligence explosion. They don't let the AI, you know, autonomously self-improve.
01:23:25
Instead, they slowly and carefully scale up the AIS that they currently have and
01:23:31
invest lots into finding ways to make them more interpretable, uh, to make them more easy to control, to understand
01:23:36
better how they work, and so forth. The result is that AI progress continues but
01:23:40
it's not quite as fast and it's much much much safer and more transparent. >> But still through these you we see job
01:23:48
disruption. >> It is continuing because they are building more data centers right like
01:23:51
this whole time they're building more and more data centers more and more chips and they're continuing to like
01:23:57
make there be a larger and larger population of AIS so to speak and that causes this huge transformation over the
01:24:04
course of the 2030s. sort of a big thing that we sort of want people to take away
01:24:07
is that even if you heavily restrict AI progress, you still get this sort of crazy
01:24:13
transformation. You know, in this scenario, they basically allow progress to continue but at a slower, more safe
01:24:18
pace here in 2030 and then as a result, it takes until 2035 to get to top expert
01:24:25
level AI. So remember, they were on track to do that in 2030, but then sort of at the last moment they stopped. But
01:24:31
because it was sort of so close to the last moment, that means that like they can sort of get there pretty soon if
01:24:36
they want to. And it's just a matter of like how long they they allow it to go, right? So they sort of they sort of slow
01:24:42
it down, spread it out, leisurely arrive at this level after 5 years. By this point, they've built up massive amounts
01:24:49
of data centers everywhere. So it's not just that the AIs are smarter and able to do all the things that humans can do,
01:24:55
but also there's a lot more of them and there's a lot of robots and so forth. So
01:24:59
by this by this point you kind of have the economy that a lot of people would have imagined with AGI where there's AIS
01:25:06
there's lots of them they're able to do all sorts of jobs there's robots there's
01:25:09
lots of them they're able to do all sorts of physical work and basically the economy is being run by these machines
01:25:15
>> so in 20 31 you you have the 1/5if of all cognitive labor done by AI in 2023 you have 60 million AIs running at 100x
01:25:24
speed in 2033 there's cash dividend to all Americans. >> Mhm. >> Um I've got to explain explain this to me.
01:25:35
>> Yeah. So, if the AIS are going to be taking people's jobs, then it's very important that people not starve to
01:25:41
death and still have money. And if companies are going to be using AI and robots to take all these jobs, then that
01:25:48
means that there needs to be some sort of taxation scheme or something to like make sure that people still have a slice
01:25:54
of that pie. M >> the pie is going to grow huge, but you still need to actually give people a
01:25:58
slice of the pie. And our proposal for how to do that, we call it the citizens dividend. Basically,
01:26:04
people have shares in a agency that sells permits to the robot companies and to the compute companies and makes
01:26:13
profit from selling those permits and then those are people have shares in that entity. It starts off small. It
01:26:19
starts off something like $25,000 per person. Uh, and then by the end it's something like $10 million per citizen
01:26:26
>> per person. >> Per person per year. >> Factoring in inflation. Like what you mean?
01:26:31
>> Factoring in inflation. >> So we're all going to be multi-millionaires. >> Yes. If this happens, which they
01:26:36
probably won't, but if it happens, this is where it go. And again, this is the thing I want to emphasize is that if you
01:26:40
get to the point where your AIs are close to being able to do all the research and then you sort of pause and
01:26:46
slow down, that means that like you still have a lot of transformation ahead of you because if you allow those AIs to
01:26:52
like still proceed slowly and like start to automate various jobs and so forth after some years they will in fact have
01:26:58
done that and they will have, you know, built huge amounts of new data centers, huge amounts of new chip fabs, huge
01:27:04
amounts of new robots, robot factories, etc. You know, we're not sure obviously how fast this will go exactly, but we've
01:27:10
thought about it a lot and we have our our guesses and this is sort of like our median guess.
01:27:13
>> What does this mean? 2037, the apocalyptic arrival of truth on Earth. >> Yeah. So, like this is the point where
01:27:20
we say they get to top expert level AI. So, it's not super intelligence in the sense that it's not like vastly smarter
01:27:26
than humans at things because they deliberately pause it at the level of top experts. So, so here they're going
01:27:32
slow. here they've just actually stopped but they've stopped at a point where the
01:27:36
AIs are just actually really good at everything. So kind of they've definitely got AI maybe they got like
01:27:41
weak super intelligence because they have so many of these AIs and because they think faster than
01:27:46
humans you know they just run much faster that's going to transform society dramatically. So we talk about some of
01:27:53
the ways in which it transforms society like this is sort of life after work we talk about what it would be like to be
01:27:58
living on your cit citizens dividend and not have a job anymore in this sort of world. Um, here we talk about all the
01:28:04
scientific changes and all the social changes that would come from all of the intellectual progress and activity that
01:28:10
would be generated by all of these AIs. So, for example, here is things like cancer cures and like you know people
01:28:18
living in apartments that were built by robots 2 years ago, >> 2036 >> providing again we stop in 2029.
01:28:25
>> Yeah. >> And providing I mean a conservative this is a conservative time frame.
01:28:30
>> Yeah. Like unfortunately, I actually think that things will happen faster than this by default and that if we
01:28:34
don't slow down, things will happen much faster than this. Once you get to the point where you've got, you know, a
01:28:39
billion AIs running day and night and they're each better than the best humans at everything. And so they're doing a
01:28:45
lot of science. They're doing a lot of talking to each other. They're doing a lot of thinking. Everyone's constantly
01:28:50
talking to their AI assistants and so forth. There's going to be a lot of scientific progress. There's going to be
01:28:54
a lot of changes to politics, to ideologies. It's going to be very disruptive and crazy and we get into
01:29:01
some of the ways in which it is uh later. Basically, >> I still not super clear on what this
01:29:06
means. The apocalyptic arrival of truth on Earth. It's just it's just because there's so many eyes AIs that are so
01:29:12
smart that they're uncovering making new discoveries in sciences. >> Let me give you an example. Lie
01:29:16
detectors. >> Yeah. >> So, that's an example of a technology that might be invented.
01:29:21
>> Yeah. >> You know, right now we don't have good lie detectors. We have very bad lie
01:29:24
detectors that like sort of work but don't don't fully work. But once you've had these top expert level AIs thinking
01:29:31
for many years at, you know, 100x human speed and there's billions of them and they have access to robot factories to
01:29:36
do research and stuff. >> They'll probably invent a ton of technologies. Maybe they'll invent on
01:29:40
lie detectors that actually work on real humans. That'll have big social effects,
01:29:44
right? Imagine a presidential candidate who's like, "Those allegations are false, and to prove them, I will go
01:29:50
under a lie detector and say that they're false." I was just thinking about the whole justice system and how
01:29:56
that would be overturned. Um, in fact, you could, you know, theoretically walk down the street and be
01:30:01
>> Yeah, >> it's both terrifying and exciting. One thing that we talk about in this in this
01:30:06
section is like the invention of lie detectors could be really bad. Like it could be that it enables a new form of
01:30:12
totalitarianism where the powerful people, you know, the CEOs and the politicians
01:30:17
>> force the people under them to go under lie detectors and say like, "Yes, I'm
01:30:20
loyal to the dear leader. I would never do anything against the cheerleader, right?
01:30:24
>> And if you're lying, then you're in. >> And then if you're lying, you get fired,
01:30:26
right? So like there's there's a ton of like very harmful uses of lie detector technology. There's also the good uses.
01:30:31
And broadly speaking, I would say the good uses are when lie detectors are used on the powerful instead of by the
01:30:37
powerful. >> What's this 2040 passing the torch to AIS? >> Yeah, great. So here they pause at the
01:30:43
top expert AAI level. And the reason why they pause is because their safety cases
01:30:48
aren't good enough for going beyond that level. Um, so in the sort of regulatory
01:30:52
systems that they set up over the course of these years, roughly speaking, the way they would work is when you're
01:30:58
making a new AI and then when you're trying to deploy the AI into something, you have to have some sort of safety
01:31:03
case explaining like what your intentions are and like why you think it's going to work the way that you want
01:31:08
it to work. And in particular, why the AI is going to like >> do as it's told, for example, and why
01:31:14
nothing super terrible is going to happen like AI takeover. It's relatively easy to make safety cases like this when
01:31:20
your AIs are still not capable of automating everything, >> but the more powerful they get, the more
01:31:26
difficult it is to actually argue that things are going to be fine because the AIS are just more capable and they can
01:31:31
they can get up to more stuff. And if you if they're actually untrustworthy, the the possible downsides are bigger.
01:31:36
So that's why they stop at this level is that they they realize that if they keep
01:31:40
going, then they might actually lose control of everything. But at the current level, they're convinced by
01:31:45
safety cases that it's fine, but they don't want to go further. So they stop there. And then what happens in 2040 is
01:31:51
they've made significant progress scientifically, including on alignment. And they figured out how to make AIs
01:31:57
that are actually aligned in a robust way >> with humans >> with humans. So they can actually trust
01:32:02
those AIs and they can allow them to become much smarter again. So that's why we call the whole thing AI 2040 because
01:32:07
in 2040 they sort of let off the brakes and allow the AIs to become significantly smarter than humans.
01:32:14
>> I guess you know this is a this is a plan and this is a hope. >> Yes. >> But in reality this is not what you
01:32:22
think probabilistically if you had to >> that's right it's important to distinguish like this is what we
01:32:26
recommend. This is what we want to happen from like this is what we actually think will happen by default.
01:32:32
Now, we do think it's possible for this to happen, but you know, that will require a lot of people to sort of wake
01:32:36
up and pay more attention and advocate for something like this to happen. So, our main scenario is mostly talking
01:32:44
about the policy choices made and the broad scale effects on society. We figured it would also be nice to
01:32:49
accompany this with a little mini scenario that describes what it would actually feel like to live through this
01:32:56
from an ordinary person's perspective. >> Okay. >> Um 2029, everyone's yelling at each
01:33:00
other. the presidents are negotiating something and they've paused AI but you still have access to the existing AI so
01:33:06
it doesn't really feel that different although it definitely is like something exciting happening 2031 they've started
01:33:11
progress again the AIS are really smart more people have lost their jobs it's like really starting to actually affect
01:33:16
things but I think still most people have their jobs but their jobs have sort of transformed so like by 2031 it's like
01:33:21
most white collar jobs involve working with AIs to a large extent or managing teams of AIS or collaborating with them
01:33:27
somehow also there are some things like robo taxis that are basically just working citizens dividend, you know,
01:33:32
ideally this would happen sooner. Like in our scenario, they kind of do things at the last minute, you know, so like a
01:33:38
lot of these policy things are like happening kind of like just in time. Obviously, we would recommend that you
01:33:43
do them sooner and and do a better job of them too, but so 2033 you start getting your your checks from your
01:33:49
dividend. So you're forecasting that there will be a citizens check that your model says it could be around 25,000 at
01:33:55
the start per person >> and then it would grow as the economy grows. >> But also as I guess job displacement
01:34:00
takes hold. They're going to need to to grow that. Check make sure you >> and that's why it's kind of the last
01:34:03
possible moment because if you waited to implement this until like 2037 then like
01:34:08
everyone would have already lost their jobs by the time that happens, right? people losing their jobs, especially if
01:34:14
it happens quickly like like we see on this sort of graph here, is going to cause lots of problems in terms of civil
01:34:21
unrest, social unrest, purpose, mental health, these kinds of things theoretically.
01:34:25
>> Yes. >> How do you think about that? >> Uh it's it's going to be rough and hopefully we can navigate that well. We
01:34:31
think that at a high level people need to have money and also people need to have power. And I think these are like
01:34:37
somewhat different things. It's like why are jobs important? Well, there's a lot
01:34:40
of reasons why jobs are important, but I think the main ones are um well, it's how people get money so they can survive
01:34:45
and get the things that they want by buying the things that they want. So, if people are going to be losing their
01:34:48
jobs, you need some other way of people getting money. And then there's also the
01:34:52
power thing, which is that right now people have political power in part due to their economic power. people can
01:34:58
threaten to go on strike for example or you know countries that are ruled by dictators can't
01:35:06
just completely you know genocide an entire subop or they can but like it's costly for them to do so because
01:35:14
then they'll have less money because that subop is contributing to their economy and contributing tax revenue and
01:35:18
so forth but if you end up in a world where actually nobody's contributing tax menu revenue except for the AI companies
01:35:24
and the robot companies then you're you the government are less incentivized to care about what you know the common
01:35:31
people think. So, so when people lose their jobs, they're not just threatened with lack of loss of income. They're
01:35:37
also threatened with loss of political power. And so, we think that it's important to like do things to push
01:35:42
against that. >> What does that look like? How do you how do people have power in such a world?
01:35:47
>> Well, in democracies at least, they still have votes. >> Okay. So I think that it's very
01:35:52
important for there to be uh regulations on the use of AI that help make the public discourse more sane and more um
01:36:04
actually giving the people what is in their interest and what they want and avoiding a sort of um opposite outcome
01:36:10
where you know the masses are easily manipulated by AI powered media for example or where everyone's talking all
01:36:20
to their AI advisers and the AI advisers are like subtly steering them away from
01:36:26
voting for the candidate that would not be what the AI companies want because the AI companies have this other
01:36:32
candidate that they like better and they're like secretly biasing their AIS to like steer people towards voting for
01:36:37
that candidate, right? So, so we want to be in a situation where um people have AIs that are actually trustworthy and
01:36:44
that are truth seeeking AIs, honest AIs and that don't have any sort of like political agendas put into them by the
01:36:51
AI companies or by the government. You know, you want to avoid a situation where the AI company where where the
01:36:55
government has issued some sort of secret order that like the AI have to be such and such a way. Yeah. The
01:37:00
Department of War dispute versus Enthropic is like a an interesting sort of foreshadowing of this, right? where
01:37:07
um Enthropic was giving their AIS to the Department of War. The Department of War
01:37:10
wanted to use them for certain things and was upset that Enthropic's AIS were like not supposed to be used for those
01:37:17
things. Uh that things in particular were domestic surveillance and uh autonomous robots.
01:37:24
>> Mhm. There's going to be a lot more issues like that coming up and you want it to be the case that like people know
01:37:29
what they're getting and that if people are like spending hours a day talking to
01:37:32
their chatbot that chatbot doesn't have political biases put into it or a secret
01:37:35
agenda or things like that and instead has been trained to like give honest true answers to things. And I think if
01:37:40
you can do that it can improve the discourse and help people to use their votes to put even better regulations and
01:37:46
even better politicians in place and so forth. it can sort of potentially bootstrap this to having something where
01:37:53
people's power is even more secure than it is today. >> A lot of the stuff we've we've covered
01:37:57
in part, so you know the wars and drones and missiles, we're already seeing this
01:38:00
around the world at the moment, which is really really interesting. Um, and we've
01:38:05
talked about robots outnumbering humans as well, which is part of this prediction. Some of the ones down here I
01:38:10
found to be really curious, which is people will be protected by AIS wherever they go.
01:38:15
>> Yeah. In this scenario, they delay the creation of super intelligence until 2040 and in fact they pause from 2035
01:38:23
but then they let it go after then and then they let the AIS become vastly super intelligent. And we think that
01:38:30
once the AIs are vastly super intelligent, the world will transform even more radically than what happens in
01:38:36
the 2030s in this scenario. So in the 2030s in this scenario, it's more like human level. You know, the AIS are not
01:38:43
they're they're doing the same sorts of things that human experts would have done. They're just doing it a bit
01:38:47
better, a bit faster, and a lot cheaper, and there's a lot more of them. And the
01:38:51
robots are still, you know, doing the same sorts of things that human workers would have done. There's just more of
01:38:55
them, and they're cheaper. And because of exponential growth, uh you start with a world that looks not that different
01:39:02
from today in 2029, and then by 2039, you end in a world that's radically transformed, where everyone's living in
01:39:07
these like fancy new apartments that were built by robots two years ago. There's like giant special economic
01:39:12
zones that are full of robots and solar panels and factories producing more robots and solar panels and factories
01:39:17
and so forth. Most of the economy is AIS and robots and people don't have jobs anymore. That sort of transformation is
01:39:23
what you get if you pause at human level. But if you go beyond the super intelligence, there's a whole another
01:39:30
transformation coming that's going to look more like magic. Think about how the technology of today would look like
01:39:35
magic to someone from 500 years ago. Mhm. >> You know, and that's without even like a
01:39:40
qualitative improvement in intelligence, right? Like the humans of today aren't like qualitatively smarter than the
01:39:45
humans from 500 years ago. It's just that we've had more time to do research >> and we have more like money and
01:39:50
resources to build, you know, prototypes and experiments and run experiments and
01:39:54
so forth. But if you had a point where there were billions and billions of AIs that were not only faster than humans,
01:40:01
but like qualitatively way way way better at everything and in particular at doing scientific research, we should
01:40:07
expect that some of the things that they develop will seem like magic to us >> and we'll just completely like we did
01:40:13
not think that was even possible. You know, people don't want to die. People don't want to be hit by cars. People
01:40:17
don't want to be like attacked by a random mass murderer. >> Cancer's gone. I mean not just cancer
01:40:23
like you know all all a lot of the stuff that happens in science fiction will probably have happened by then. So
01:40:28
things like people scanning their brains and uploading into into computers right
01:40:32
or self-replicating robots in the asteroid belt uh creating more and more satellites to
01:40:40
uh produce more and more power to produce more and more self-replicating robots and so forth.
01:40:43
>> Most people still live on Earth but the trend is to move to space. >> That's right. Yeah. So like if if you
01:40:49
end up in the situation where the entire human economy is just like a tiny drop in the bucket that is the entire economy
01:40:58
and it's just like this huge amounts of robots and AIs that are moving incredibly quickly, then what you want
01:41:05
is Earth to be mostly left as something like a preserve. You know, I think a lot
01:41:11
of people are worried about the environment being destroyed, which it totally would be if it wasn't protected.
01:41:16
And uh you know there's a lot of people who sort of like their lives as it is and don't want to be uploaded or live in
01:41:23
some crazy new future thing. And it seems to us like the reasonable solution to these issues is uh create new living
01:41:30
spaces off the planet with some of that vast economic wealth and activity that's
01:41:36
happening for the people who want that sort of thing and then that way the earth can be preserved.
01:41:41
>> Data picture here of data centers in the ocean. Uh I mean there's three images
01:41:46
there of different environments where humans might live. >> Again like our proposal was you preserve
01:41:53
like 99% of the earth uh mostly as is as historic or environmental from historic
01:41:59
or environmental reasons but then like some parts of it you designate as special economic zones where the robots
01:42:04
can go crazy and dig giant pit mines and produce factories and so forth. Um, we were thinking it would be good to build
01:42:11
the data centers on the ocean instead of um on land for a variety of reasons, although later space would be better and
01:42:19
we could see that being reasonable as well. >> What about immortality in a world of AI?
01:42:24
Um, 20 well 30 45 you say you've lived a dozen lifetimes and are immortal, passing from life to life as if by
01:42:32
reincarnation. I mean, there's a lot of billionaires at the moment that are focused on
01:42:36
longevity. I mean, Brian Johnson's said he's got this central rule, which is do not die right now.
01:42:42
>> Yeah. >> Because we're in the age of AI and it's conceivable that with super intelligence, we'll be able to choose
01:42:46
when we die. >> Yep. I think that's probably right. We don't depict that happening in this part
01:42:51
because at this part they only have, you know, human level AIs, but that's one of
01:42:56
those things that seems quite plausible that super intelligence could achieve um
01:43:01
through a variety of means. What is your hope with all of this stuff now? Why did you do this? Why did you
01:43:09
make this 2040 plan ape? >> In the like first week after we published AI 2027, it it blew up a lot
01:43:15
bigger than we expected, by the way. Like after we published AI 2027, it it blew up a lot bigger than we expected,
01:43:21
by the way. Like we actually made forecasts beforehand of like how many views it would get and stuff like that,
01:43:27
and it was like 90th percentile outcome. So like um very much not what we expected. Um, but in like the Twitter
01:43:35
storm that happened, various people were like, "Are why are you giving us all this like doom and gloom uh predictions?
01:43:42
Like, how about a more positive vision of like what you think we should do instead?" And I think that that seed
01:43:47
sort of like implanted in us and then we were like, "Yeah, that's reasonable." Like, we've sort of depicted what we
01:43:53
think the default path looks like and why we think it's pretty scary. Now maybe we should switch facts and come up
01:44:00
with some actual recommendations and then depict that as well >> even though you don't believe they're
01:44:04
pro probable. >> Yeah. I mean you can vote for a political candidate even if you aren't
01:44:07
confident that they're going to win, you know, and and you can say like here's what I think we should do even if you
01:44:13
think that people are probably not going to do it. You shouldn't say this if you
01:44:16
think it's completely unlikely. Like if you think there's no chance then like maybe you shouldn't bother. But we think
01:44:20
there's a chance. like in particular for the reasons that we described in the scenario we think that people are going
01:44:26
to wake up to the power of AI over the next few years >> because of something happens
01:44:32
>> the companies are saying that they're going to do this >> and they are kind of on track and it just
01:44:39
sort of makes sense that like if they get anywhere close to this level of AI then there's like big issues and big
01:44:46
problems and like we need to like do something about this and so I think that Even if there's not any like very
01:44:52
dramatic warning shot or something, I think that just naturally people are going to start paying more attention to
01:44:57
this and reasoning through the implications and trying to predict what's going to happen. And so naturally
01:45:02
people are going to be more interested in regulation of AI for example. And in fact
01:45:09
there's actually like there's there's actually more of this happening than we predicted.
01:45:13
>> More of what happening? >> Serious interest in reg AI regulation. So, at the time that we published AI
01:45:18
2427, the sort of like mainstream position of the tech companies and in the government was kind of like AI
01:45:24
regulation, bad idea, >> free-for-all. >> Free-for-all. Yeah. In fact, there was
01:45:28
even an attempt to um preemptively ban states from regulating AI. >> Yeah. >> You remember that? Now, it seems like
01:45:35
the conversation has changed a lot. Like now the the US government just told Enthropic they have to shut down their
01:45:41
AI because they were worried that bad actors would use it for cyber attacks. you know, the government is like waking
01:45:48
up and doing more stuff than we expected uh already. And we're actually hopeful that that trend will just continue and
01:45:55
that before it's actually too late, there will be very serious conversations happening inside the government and
01:46:00
outside the government and in the broader society about all of these issues and trying to chart a course that
01:46:07
is um avoids the loss of control and concentration of power risk that we mentioned. you um you've spent well must
01:46:14
be almost coming up to 15 years thinking about this stuff. Um if this here was a
01:46:20
button and if you press that button your plan S would occur and it would shut down every data center that is currently
01:46:29
training a frontier AI model uh for good. There would never be any other >> AI labs um working on these problems.
01:46:38
Would you press that button? I was I was about to slam it until you said for good.
01:46:43
>> Okay. >> Like I think I think if it was a sort of temporary shutdown, I would totally slam
01:46:47
that button because we are not ready to do this. You know, like civilization is not ready to have these companies
01:46:55
automate themselves and then get smarter and smarter and then have the super intelligent like no. There's a bunch of
01:46:59
reasons why that's really uh dangerous. But I would be at least hesitant to press this button if it permanently
01:47:07
foreclosed the possibility of ever doing it again. for sure. >> But but if you think that plan D is
01:47:12
probable, which is this race we're on to super intelligence, >> if I had a choice between D and S, I
01:47:17
think I would press it. >> Well, it's it comes down to what you think, right? Cuz if you think that's
01:47:21
that is what's going to happen, plan D and the only alternative. >> I didn't say this is what's going to
01:47:27
happen >> proistically. >> Yeah. Yeah. Like like I'd be like this is the most likely, maybe this is the
01:47:31
second most likely, maybe this is the third most likely. They are all possible. So with your current
01:47:35
perspective on whatever one you think is going to happen, would you press the button? I'm giving you an S, a definite
01:47:40
S or whatever you think is going to happen. >> That's tough. >> What is the scope of the shutdown? So is
01:47:50
it >> it's no one can train an AI model again, ever again. >> That's real rough because like I said,
01:47:58
there's loads of benefits that we could get from AI if we do it right. Um, >> I think I I've almost put you in the
01:48:03
position of Sam Holtman. >> Yeah. >> To some degree. >> Yeah. >> Um, let me Do you mind if I just take a
01:48:10
moment to think about this? >> I think I prefer you to think. >> Yeah. I think I would not press the button,
01:48:24
but I'm I feel very torn about it. Um, the reason why I think I would not press the button is that I still have
01:48:30
substantial hope that we can get something much better than this, something more like this. And I think
01:48:35
that basically I think that if we don't build powerful AI systems eventually, then
01:48:43
we're probably going to die as a civilization. eventually, you know, like 100 years
01:48:48
from now, 200 years from now, something like that, like nuclear war, pandemic or
01:48:51
something, you know, I I don't think human civilization right now is like super super stable. Um, and
01:49:00
so I think that basically what I was about to say was the possible benefits for posterity and for all the billions
01:49:08
and billions of people who could live in the future outweigh the like the current level of risk. But actually,
01:49:17
>> I've heard that narrative before. >> Yeah. I don't know. Like, >> yeah, like maybe maybe it's just like,
01:49:22
nope, the people right now are the people we should prioritize. People right now are in grave danger. They're
01:49:29
going to be fine for at least the next couple decades. So, never mind posterity.
01:49:36
Prioritize the people right now. Um, and people right now definitely don't want to do this lottery, I would say. Um,
01:49:46
yeah, you've really asked me a tough question. >> So, would you press the button if that
01:49:50
was the button? >> Probably not, but I would feel very torn. >> Okay. So, what I I always think about
01:49:58
the personas of like the audience that are watching and these are, you know, they're they're very curious people,
01:50:02
especially on the subject of AI as we've seen, but they they want to know like what it means for them. I think a lot of
01:50:07
them also want to know what they can do. >> Oh, yes. Yeah. What can people do? Well,
01:50:12
I think that if you either have talent or passion, you can get directly involved. There's lots of organizations
01:50:20
that are worried about these things and that are trying to do something about it, like political advocacy or technical
01:50:25
research or like building useful tools that will hopefully help people be better and stuff. But if you don't want
01:50:31
to like make any major career changes or or things like that, then I would say just pay more attention to these issues
01:50:39
and talk about them more with people. do stuff like, you know, emailing your congressman or whatever, it it doesn't
01:50:45
change things that much, but it does help. I think that especially for this particular issue, the core problem is
01:50:51
that people aren't taking it seriously yet. Like if the sorts of things that I was just saying to you for the last hour
01:50:57
or two were just like top of everybody's mind, we wouldn't even be here. Like there
01:51:03
there would already be much more significant regulation in place, you know, and not only would there be more
01:51:09
heavy regulation in place, but there would have been better regulation in place that's less, you know, less like a
01:51:15
cudgel and more like a scalpel and that's like more sensitive to what's actually bad and what's not so bad and
01:51:21
so forth. And there'd be more expert people in the government and advising the government and so forth. So, just in
01:51:26
general, like the more people wake up to these concerns and to these projections,
01:51:32
uh, I think the more likely it is that we can do good stuff before it's too late.
01:51:36
>> What about how they should vote at the polls? We've got an election coming up
01:51:40
in the United States in a couple of years time, but there's elections happening all over the world all the
01:51:43
time. >> You should ask your candidates what they think about all this AI stuff. You
01:51:47
should try to get them to like have opinions, and then you should vote for the candidates whose opinions are better
01:51:52
on this topic. This is the most important thing happening. uh in our lifetimes, probably in all of history in
01:51:58
fact and it's very important that it go well and so it's what all the all the leaders of all the countries should be
01:52:03
thinking about and making plans for. >> Isn't it such a weird thing to be alive at this moment in time? Like I was
01:52:08
thinking about all the times that I could have been born and I guess my ancestors probably thought the same, but
01:52:12
I was thinking as you were speaking I was like I think it's when you referred to it as like the final show.
01:52:16
>> Yeah. >> What was the phraseology you used? >> I said the the clim run up to the climax
01:52:21
or something. >> Yeah. I mean, what a what a crazy thing to be born in the run-up to the climax
01:52:26
where everything you're describing here is within my lifetime, conceivably. Hopefully.
01:52:30
>> Yeah. >> Um or maybe not hopefully. >> What a crazy time to be alive. >> Certainly.
01:52:36
>> I noticed that when I met asked you if you had kids, your demeanor changed quite considerably.
01:52:40
>> Well, it's Yeah. >> It's like you dropped into a different state. Obviously, that's been central to
01:52:46
the rumination that you've been experiencing. Well, it's a sad topic, right? Like when when I had kids, like
01:52:54
the reason to have kids is in large part about the future, you know, like it's not just like a cuddly thing to have
01:53:00
with you in the moment. It's because you have all these hopes and dreams about how they'll grow up and how they'll go
01:53:05
do their own thing and be their own person and stuff. And because of what's happening with AI, I think a lot of
01:53:10
those dreams are in jeopardy. >> Presumably, you still would have had kids. >> I've actually flip-flopped on this
01:53:15
occasionally. Yeah. Basically, the topline answer is I'm not sure. the my first child was had we we had her when
01:53:22
we were um in 2019 she was born 2019. >> Yeah. >> So this is before my timeline shortened
01:53:27
a lot. So at this at this point I was interested in AI I was tracking the field. I was making forecasts but I
01:53:32
didn't like actually expect it to happen soon >> you know. >> Mhm. >> And then this caused like when I when I
01:53:38
did start thinking like oh my gosh it's going to be happening like real soon um like by 2030 you know. um that caused
01:53:46
some reconsidering and so I basically told my wife like let's not have any more kids. It's too uncertain, you know.
01:53:54
But that turned out to be really hard because especially for my wife, like we already had one kid and like no
01:54:00
siblings. Um so eventually I sort of gave in and was like, "Okay, well, you know what? We already have one. It's
01:54:07
gonna be all right. Like maybe maybe the future will be good. And even if it's not like, well, we're all in the same
01:54:13
boat together. >> It's quite chilling what you're saying. It's chilling because you know more than
01:54:18
me. And if you're at home saying to your wife, listen, maybe we should pause on having more children and building a
01:54:23
family because of what's going on with AI. >> To be clear, this is Yes. I mean, yes,
01:54:27
it's very concerning. I am I am chilled. Uh this is bad. This is what I've been saying. I hope things go well. I think
01:54:35
things might go well. Um I think that there's a lot we can do to like steer things in a better direction. I mean,
01:54:39
one of those things as well, I have to say, is just speaking about it. It's I think a lot of the progress we've seen
01:54:45
with governments waking up and, you know, we've seen certain things with people booing certain people at certain
01:54:51
events. >> Yeah. >> Um is is a downstream from people like yourself actually coming on shows like
01:54:56
this and all the other podcasts and >> Yeah. >> telling us what's going on. >> Yeah.
01:55:01
>> Because elsewhere, to be fair, we're going to be gaslighted by the people that have the biggest PR machines.
01:55:05
>> Yeah. So, um, I often I think it's probably worth me saying I find myself kind of in two minds because I'm an
01:55:10
entrepreneur and I'm I'm an investor. I'm an investor in probably more than 100 companies now. And so many of those
01:55:16
companies are using AI. I invested in Grock, the inference chip company. I've invested in SpaceX which now own another
01:55:21
Grock and they're doing AI. I use AI every day in my life. I've been using it through this conversation to understand
01:55:26
different things that you've said. >> So that's one side of me which is like business builder entrepreneur who has
01:55:32
seen the benefits of AI in my own life. And then there's the other side of me. And it's funny because I think sometimes
01:55:36
people think you have to pick a camp. But through all of my life, even when I was a social media CEO and I was saying,
01:55:41
by the way, listen, I'm building a social media business, but I think there's some downsides to social media.
01:55:45
I find myself at the same moment where I'm like, I build with AI. I have AI investments. And at the same time, as a
01:55:51
civilian, I'm like, >> yeah, I mean, I think that is attention. I think that there's there's different
01:55:57
ways you can draw the line. So, and I know lots of people who draw the line in lots of different ways. So, like there's
01:56:02
some people who just like, I'm not going to use AI. I think this stuff is bad um
01:56:06
and on a bad trajectory, so I'm going to like boycott AI, right? I'm not one of those people. I use AI a lot. We all do
01:56:11
at AI features project. Um it's helpful for a lot of our work. The opposite end of the spectrum is
01:56:19
people being like, well, it seems like it's on a trajectory to happen, so the thing to do to make it go well is to
01:56:25
like get involved and accumulate power and try to like steer it from the inside. Mhm.
01:56:29
>> And so I'm going to go work at OpenAI or Anthropic and like try to like climb the
01:56:33
ranks and then like you know be someone who matters when the important decisions
01:56:37
are being made. And I know loads of people like that. That was like what I was doing when I was that wasn't what I
01:56:42
was doing exactly but like >> that was the path >> that was like that was I mean this in
01:56:45
some sense this is what the whole narrative of the companies are right like this is why they tell themselves
01:56:48
it's okay to do what they're doing is that they're worried about the other guys you know and so like all these
01:56:53
people are are deciding like we're going to like lean really hard into it. are going to like be there in the room when
01:56:59
the important decisions are being made, you know? So, there's a whole spectrum and I'm sort of like somewhere in the
01:57:03
middle. Like, I'm not at the AI companies. I'm not helping them go faster. Instead, I'm talking to the
01:57:09
broad public and trying to advocate for what I think is the my current best guess as to the way out, you know, the
01:57:16
way forward. Um, but I'm not like boycotting all the AIs. I'm I'm not like, you know, uh trying to I'm not
01:57:22
refusing to like engage with it in that way. Do you think it's too late? >> No, I don't think it's too late. If I
01:57:30
thought it was too late, I wouldn't be here. H >> Where would you be? >> With my family.
01:57:36
>> What's your closing message to the general public? If you had to have a closing statement to them,
01:57:42
>> maybe I would say that like you're going to hear a lot of things and you already
01:57:46
have been hearing a lot of things about AI and it's going to sound like science fiction, but sometimes things which
01:57:53
sound like science fiction happen in reality. And in fact, many times historically things used to be science
01:57:59
fiction have then become reality. And people need to stop thinking about what does or doesn't sound like science
01:58:05
fiction and just start thinking about like the trends and you know the actual trends that this technology is on and
01:58:13
reading and forecasting how it's going to go and then taking seriously the possibility that it could go something
01:58:18
like this and then thinking about what should be done about that. >> And where would you direct them to get
01:58:23
more information? >> You can go to a27.com to read our previous scenario. You can go to
01:58:29
a2040.com plan A to read our new proposal for what is to be done. Um, these things are not just a sci-fi
01:58:37
story. They also have lots of like explainers and links to other things. And so they're kind of like a nice
01:58:42
jumping off point to to learn about all of this stuff. M >> um if you want I could um after this is
01:58:49
over like give a reading list of like other papers and articles and >> blogs to follow and so forth
01:58:57
>> and I'll link them all below in the comment section. So if you're listening now go ahead and take a look at the
01:59:01
comment se the um description of this episode and you'll see a bunch of links which is Daniel's recommendations of
01:59:06
what you should read. You know I think it's it's just a really really great moment in time to get educated on this
01:59:11
stuff. Um humans have an inclination because of cognitive dissonance where we feel uncomfortable about something to
01:59:18
bury our heads in the sand and avoid it. But actually I think this is one such time to do the very opposite for many
01:59:24
reasons to to inform yourself so you know what actions to take but also because AI you know unavoidably is going
01:59:30
to be a huge part of all of our lives and careers. >> Yeah. Yeah. Thank you. And that that's a
01:59:34
good way to to say it. It's going to matter a lot. It's going to it's going to be everywhere soon and um we need to
01:59:41
do something about it before it's too late. And >> what about AI future project?
01:59:44
>> That's our organization. We spent a year writing AI 2027 after I left OpenAI and
01:59:48
then we spent another year writing AI 2040 plan A. >> Daniel, thank you. >> Thank you.
01:59:53
>> Thank you for all the work that you do. I can see how much you care about this
01:59:55
stuff and it's your care. It's funny. Care itself makes others feel care. and um seeing how personal this is for you
02:00:01
and seeing how much you've dedicated your life to this, but also hearing that you you basically walked away from $2
02:00:07
million to be able to speak to the public about this information is incredibly admirable and uh I I think
02:00:13
voices like yours are more important now than they've ever been on this subject.
02:00:17
So, please do keep fighting the fight that you're fighting. And that's one of information. It is of honesty and it is
02:00:22
uh of saying what what is often the quiet part out loud and doing really really smart research. I'll link
02:00:27
everything we've discussed today below. Um, and I hope we can chat again sometime soon.
02:00:31
>> Thank you. >> YouTube have this new crazy algorithm where they know exactly what video you
02:00:36
would like to watch next based on AI and all of your viewing behavior. And the algorithm says that this video is the
02:00:43
perfect video for you. It's different for everybody looking right now. Check this video out and I bet you you might
02:00:48
love