00:00:00
all right freeberg is back welcome back to the Allin podcast episode 160 something your favorite podcast in the
00:00:05
world yada yada yada with me again the chairman dictator moth po hoaa the Rainman yeah definitely David Sachs is
00:00:13
here and back from his time in the metaverse we found him somewhere out in space in the solar system in his Apple
00:00:22
goggles your favorite sultant of science David fredberg is back from the metaverse I miss you guys come home
00:00:31
thanks for having me what what did you discover when you went to Uranus in Google class sorry Apple you actually
00:00:38
use the Apple Vision Pro jcal I ordered them I ordered them and I walked by the Apple Store and I was going to go in and
00:00:44
try them and there were so many lunatics in there I was like yeah I'm not doing it but I ordered them you use you
00:00:49
actually used them what I ordered one online to be delivered and it was like delayed by a month so I went down to the
00:00:55
Apple Store and picked one up okay and my kids cannot stop using it really I went down to the Apple Store but got
00:01:02
cleaned out by the thief that stole everything so the Oakland one let your winners
00:01:10
[Music] ride and instead we open source it to the fans and they've just gone crazy
00:01:20
with [Music] it that was crazy that was crazy we'll put the video in here to the idiots who
00:01:28
are robbing apples SCE all the devices get bricked when you steal them and they all have GPS in them have you tried it
00:01:34
your mouth no I was too busy working out making love and winning oh okay got it so you were you were making sweet love
00:01:42
you were watching your portfolio go up and you were just generally winning got it got it yeah yeah so freeberg the rest
00:01:48
of us were being men in the world accomplishing stuff but but do tell us about your time in the metaverse do
00:01:54
those goggles come with a lifetime prescription of ssris you sound like one of these like Tech journalists that are
00:02:02
actually anti-tech people you guys are actually Tech journalists like it tech journalist seem gen Computing platform I
00:02:09
remember when the iPad came out and everyone poo pooed the iPad I thought it was stupid I tried to use it I couldn't
00:02:13
get any value out of it and in 2010 or 2011 when did it come out 2010 2011 we started using it with our sales team
00:02:20
selling to Farmers and we gave every sales guy an iPad and they went out in the field with 3G and they were able to
00:02:27
close sales in the field meeting with Farmers which had never been done before you usually had to get a farmer to come
00:02:31
into an office how many IP to sell the product Oh so we we had like they were selling cl.com software we had we had
00:02:39
dozens of these sales guys we gave them out to our sales agents as well the independent agents they started using
00:02:43
them and it was like a real GameChanger in how sales was done in agriculture and
00:02:48
I had never even contemplated that when I first used the iPad let's let's get to
00:02:51
Brass tax here what is the killer app what do you think in the next five years people are going to be doing with this
00:02:57
thing on a daily basis is there a daily use case I'll say a couple things one is
00:03:01
like I feel the same way I did about the iPad which is I don't know what it is today but I can tell that there's
00:03:05
something there and I'll give you an example of something I thought about first of all the AR is gamechanging okay
00:03:10
if you've used like The Meta yeah the Oculus Quest it like makes me super dizzy makes my head hurt makes my eyes
00:03:17
hurt like you're super disoriented what Apple solved is that you're like still in reality but then you get to interact
00:03:24
with these three-dimensional kind of objects in reality and it's like really well done it's definitely V1 and there's
00:03:30
going to be incredible changes in the next couple Generations but it gets rid of all that dizziness disconnected kind
00:03:35
of stuff that happens with the the full VR experience which I thought was really
00:03:39
incredible then last week and I'm sorry I missed the show we have a facility with my company in North Carolina we
00:03:46
have this giant Greenhouse facility and I was doing meetings with farmers and stuff I go to the greenhouse facility
00:03:51
and there's so much work that the greenhouse text and lab teexs are doing where they're using an iPhone and a
00:03:57
barcode scanner and a printer and they're holding all these pieces of equipment scanning the QR codes on
00:04:03
flowers taking the pollen out putting it in the next flower training each other how to do it and I was like I put this
00:04:09
apple Vision Pro on and I was like man all the the apps and all the tools that we had all these different pieces for
00:04:15
that was taking people tons of time image collection data collection could all just be done streamlined while
00:04:21
you're working you could have a report yeah yeah you have a task on the right cameras are taking images in the middle
00:04:28
QR codes are automatically scanned data is being ingested the task list is kind of you know giving folks next steps they
00:04:35
can listen to music while they're working and I realized for that job and I met with all the the team out there
00:04:39
and spent time with them and I actually did the work that they do to get a better sense for the workflow and I was
00:04:44
like man literally every aspect of this job will be massively improved and productivity will go up by 10x with
00:04:50
these goggles will it happen in the next couple weeks or months I don't know but
00:04:53
my engineering team is looking into it can we take it can we use some software can we build some software and can we
00:04:58
put this on folks to give them a better work experience to increase our productivity to do automated data
00:05:03
capture so I don't know exactly where it goes but I could start to see how this can become a more ubiquitous part of a
00:05:09
Workforce setting and not just be a video game and movie tool for consumers so I'm I'm reasonably optimistic about
00:05:15
where this goes it's definitely V1 I feel like it's the iPad days where no one's really sure where the applications
00:05:20
are but yeah yeah Enterprise applications unbelievable makes total sense and also training training right
00:05:25
assembly line Workforce warehouse workers where you're real time kind of task updates data's being ingested all
00:05:32
in real time and and by the way the other thing I'll say is training is incredible there's spatial video
00:05:37
recording on it so it looks like you're living through the experience that someone else had so you can train
00:05:42
someone how to do a difficult task and rather than have a human go spend hours training a Workforce the workforce can
00:05:48
be trained by the goggles in a way that you cannot do a two-dimensional video today so I don't know I'm I'm I'm pretty
00:05:53
optimistic very strange days right I don't know you're you're a fan of SciFi but remember Strange Days tot trath
00:05:59
what's going to happen first here are humans going to become more like robots by putting these on and do this fact
00:06:05
reer work or is Elon with Optimus and some of Humane I think is the other one there's a couple of other people
00:06:10
building General use robots figure is the other one figure yeah which one wins the day is it going to be humans having
00:06:18
eyes and you know data collection like robots or robots having appendages like humans well let me let me put two ideas
00:06:26
together and see what you think of this argument if you think about the generation of
00:06:32
human beings that have as close to any other generation before it lived in a totally immersive world I would say
00:06:42
the best representation of that are current teenagers and 20-year-old people and maybe at the upper Edge the early
00:06:50
30s people and why is that you know they've lived inside of social media their entire lives they've lived inside of
00:06:56
immersive video games their entire lives but the question is is are they better off and happier as far as we know from
00:07:03
an evolutionary perspective and I would tell you that the answer is a is a huge gaping no so if you believe that the
00:07:15
rise in depression the rise in suicide the dependency on drugs the dependency on ssris the sexual promiscuity the lack
00:07:24
of marriage the lack of kids if all of those things are in some way a correlated byproduct let's not say
00:07:33
it's causal right let's just say it's a correlated byproduct of this entire immersive almost exclusionary detached
00:07:41
world that these folks have grown up in taking that to the Limit I'm just going to put out there
00:07:47
may not be the solution to our problems and so I guess the more directed answer to your question is I would hope that
00:07:54
the ladder wins so that we take these goggles off and actually learn how to talk to each other and look each other
00:07:59
in the eyes get married and have children because I think that's actually better for the
00:08:05
world and I would probably say that it's almost better for the world than a 10 Xing of
00:08:10
productivity interesting and then you see the correlation to cancer and disease that is disproportionately
00:08:16
higher amongst these young people so I think it's at some point to ask ourselves what is structurally happening
00:08:23
in the lives of these 16 you know 15 to 31 year olds that is just so in terms of
00:08:30
outcomes and if you look at some of the environmental variables that they live in and then take some of those and take
00:08:36
them to the Limit I think that there's a reasonable argument that their lives get
00:08:40
worse before it gets better yeah I mean the amount of time you spend on social media is correlated uh with depression
00:08:46
not just social media I'm just saying just this immersive like I'm going to detach from the world and live through a
00:08:52
microphone and glasses taken to the Limit I'm not sure is the solution to these kids feeling detached lonely
00:09:00
isolated isolated yeah yeah I mean it it correlates all of these things that we're seeing in this younger generation
00:09:05
correlates with the introduction could it be a good productivity device yes do I hope it's a good productivity device
00:09:11
yes but if we try to make it the Panacea for anything and everything I think we're going to we're going to compound
00:09:18
the systemic issues that these young people have and I suspect on the margin if you were going to bet all of these
00:09:25
things that we see in these young people today will get worse as a byproduct of technology not necessarily get better so
00:09:31
if you can take a different path like Optimus or the figure AI robots where that work is done at least we have a
00:09:39
different problem probably maybe even more existential abundance but a different problem which is now how do
00:09:45
you find purpose but maybe you can find purpose through connection and the types
00:09:48
of things that humans have been bred over billions of years to actually optimize for okay saak I remember when
00:09:57
you were starting craft you fir it up like a group for VR and you got pretty heavy into it you made a couple of small
00:10:04
bets I remember I don't think it any of it worked out really you could tell me if I'm wrong here but you got in a
00:10:09
little bit early there maybe you could talk about the business case for this and has that changed because you you
00:10:14
believed I've believed a lot of folks thought hey maybe this is the time when Zuck really start you know had bought
00:10:21
Oculus and and they started putting out some good product seemed like it was a false start is this the actual starting
00:10:26
pistol and is this the start of the VR AR adoption race I don't think we're quite
00:10:32
there yet okay we've been talking about VR being a thing for over a decade yeah no more like 30 remember the Nintendo VR
00:10:42
stuff it's like always on the verge of happening I think that the big complaint about the Apple device is has a lot of
00:10:47
capability but it's still a pretty huge device to wear on your forehead it's just not really going to be comfortable
00:10:53
enough to be something that people want to use all the time h i mean there's also a question of use cases but they're
00:11:01
getting there with the use cases in any event I I do think that Apple Vision Pro is it's like I said
00:11:09
last week it's a useful prototype or proof of concept and it will get better so I'm glad they did it because I think
00:11:17
you need to start somewhere and then just keep iterating but eventually for this to I think really take off you need
00:11:24
to shrink the form factor miniaturize the technology just every version of it make it simpler lighter easier to use
00:11:33
yeah I mean eventually it'll feel like sunglasses and so that is I guess if they become like regular glasses I think
00:11:40
we all agree it becomes the next I got to tell you I feel like it's pretty damn comfortable I don't know if you guys you
00:11:45
guys haven't really used it but that's what I've heard that's the surprising online saying it's unlike any
00:11:50
other headset I've ever worn they did an incredible job designing like does it feel like ski goggles it doesn't feel
00:11:58
heavy it doesn't feel pre pressure compared to ski goggles how if you were wearing ski goggles it's less
00:12:03
constricting than ski goggles it's more comfortable it like floats on you a little bit they did a great job with
00:12:08
this cushioning device they built and the band you put on it feels very natural it's apple design right it's
00:12:13
like a really well-designed product that's unlike anything else you've ever tried I've always felt like when Apple
00:12:19
comes into the race that's the starters pistol and I think this is it because i'
00:12:22
I've heard the same thing from everybody you you have to try it it feels like different than Oculus and some of those
00:12:28
version that came out previously and they have the app ecosystem and I I would not discount
00:12:35
that when you know the ability to monitize the app ecosystem and have all the people who are already building the
00:12:40
com app the Uber app whatever notion you know all the stuff that people use and love Spotify YouTube and then poured it
00:12:48
over here fortnite whatever I think that's going to be the magic and the statistics are not lying here I mean
00:12:54
this is unbelievable they've sold already 200,000 units which doesn't seem like a lot but for a V1 that is a lot
00:13:01
and they're going to sell a half million this year it's going to be close to like
00:13:05
that's not that many well it's a couple of billion meta sells more they do yeah but you know this is
00:13:11
$4,000 this isn't 500 so to sell that many of a $4,000 device is incredible it's a proof of concept it's not like a
00:13:19
regular Apple product that is a mass Market device that tens or hundreds of millions of people are going to buy but
00:13:26
it puts them on a path yeah to where they can iterate and keep making it better see I think and this is I guess
00:13:32
what I'd ask freeberg do you compare this to buying a MacBook Pro buying an iPhone or buying the Oculus you know
00:13:39
whatever the you know $500 unit because everybody I see talking about online is comparing it to the purchase of a laptop
00:13:47
because of the desktop and you can kind of do your coding or surf the web and do
00:13:51
all that where do where do you put this is it buying a TV is it buying a laptop is it buying a smartphone what would you
00:13:55
have to have a keyboard to be really productive on it uhhuh if you're going to use it for writing purposes or coding
00:14:01
purposes so it doesn't really work with just the the headset but you could do that yeah it's definitely like buying a
00:14:07
new Computing device but people felt the same way about the iPad get go back to 2010 when the iPad came out and everyone
00:14:14
was like who's it for it's a whole new computer who's it for you already have a phone you already have a computer why do
00:14:20
you need an iPad and then they sell tens of millions a quarter now yeah so I really I as I do the math on this I was
00:14:26
just kind of doing some back of the envelope stuff I think they're going to sell 100 bill
00:14:30
of Apple Vision Pros not this version but this version plus the next version probably over the
00:14:36
next I would guess for them to get to 100 billion in sales it'll take them less than five years I think they're
00:14:42
going to run the table on everybody I think they're going to own the entire space I think everyone's underestimating
00:14:47
this as a new Computing platform and once these applications particularly in the Enterprise setting start to kick in
00:14:52
and I will say that the movie watching experience is way better than watching on a TV in your living room my kids
00:14:57
cannot stop asking me to use the goggles to watch instead of an iPad or TV because you see 3D like all Pixar
00:15:04
movies are natively 3D and so you got the Disney Plus app on there you watch a Pixar movie and you're watching in 3D
00:15:10
the kids are blown away so I think we're all going to be surprised by how this Go
00:15:14
Disney's all in on it remember when our parents told us not to sit too close to the TV now we're just strapping the
00:15:20
thing to our face yeah I I had the most Silicon Valley moment ever I go to buy a
00:15:27
cup of coffee I was going for my little walk I see blue Bott I'm like oh you know I get myself a mocha you know I
00:15:31
lost a little bit of weight I'm going to treat myself $9 for a mocha number one that in the C tilted me $9 for a it was
00:15:39
$8 and then I gave a dollar tip and then I felt cheap giving a dollar tip you know it's $88.99 for a carton of clover
00:15:45
milk all organic I mean you can make infinite lattes at home anyway where did you go
00:15:52
for your $9 mocha I was I'm in Palo Alto right now because we lost like the blue
00:15:56
bottle yeah said this I'm like $9 what am I doing you know I just I felt like buying a chocolate bar the stain dirty
00:16:04
lips left on the cup oh my God look at so you know what you're a little obsessed with my lips take it easy there
00:16:11
so anyway then there's a kid in the place wearing the goggles with the keyboard he's pounding he's getting work
00:16:19
done this kid was doing work and I tell you the trth put in the hours he was putting in the hours no one looks at
00:16:25
your laptop no one looks at your screen that's what work without anyone seeing what you're doing this kid had four
00:16:31
desktops up this guy was probably on PornHub Spotify writing code how many words did this person say to another
00:16:38
human being while you were there no zero and you know what when they're on a laptop they're the same what's the
00:16:43
difference he's he's coding nobody B it and I I think this is gonna they're gonna run the table on this I think it's
00:16:48
100 billion 100 billion sales under five years yeah I take the over I take the over what do you you got the over or the
00:16:55
under cuz even if they keep it at three grand they got to sell 30 million UNS to
00:16:59
get to 100 bil they're going to make up a lot of money on this app store too by way I you guys are right that it's going
00:17:03
to be successful in terms of Revenue what I'm asking is a more societal question is do you guys actually think
00:17:09
it's better no I don't want my kids in this all day no and I can see this becoming super predicting hey freeberg I
00:17:14
I can I buy three for your kids just have them walk around with them I have a no iPad in the house rule as well but
00:17:19
wait a minute hold on what about productivity freeer my kids aren't trying to be productive they're using it
00:17:24
to burn it's called childhood you don't have a productive childhood it's supposed to be not productive you guys
00:17:29
understand that at some point you guys will be the only six kids whose parents haven't given them this stupid thing to
00:17:36
put on their face no this is going to be Tim restricted I have a no iPad no phone
00:17:41
no like I let them use the headset good for them it's so good no no it burns their Burns their brain away Burns their
00:17:47
brain away it's terrible man I totally agree with you social interaction the loss of our ability to communicate as
00:17:54
hum it's critical and it's a fail point I do think that there where these things
00:17:59
create great unlocks I think this is an Enterprise device can you imagine giving
00:18:02
a field sales team on the farms to go there they can take off their swey headset when the sun is shining and then
00:18:08
give it to the farmer to put on and then he can put it on and feel the sweat and
00:18:12
the the headband will be wet that's not the use case it doesn't by the way it's it's a very personal device in order to
00:18:18
log in you know it does like a I scan um or you have to have like a lock in like
00:18:23
login like you do with your phone but then you got to reset the eye because it automatically sets the eye position so
00:18:28
when put on someone else's headset you got to reset the IP it's a whole thing so it's not a transferable device it's a
00:18:33
very personal Computing you know kind of thing so I don't think it's going to be
00:18:37
the same as like an iPad or a phone it's a very different kind of thing I don't know what it's going to look like yet I
00:18:40
don't know I I say next week we do the show inside of these or at least me and you freeberg will be will be does is is
00:18:45
there a zoom very funny there's um there's an a there's an avatar thing and so what it does it scans your face while
00:18:52
you're talking all four of us can see each other as the Avatar yeah all right let's do it it'll be hilarious I had a
00:18:58
moment this week in parenting I had a moment this week where I told one of my children that when I send a text message
00:19:06
I expect an immediate response otherwise I am going to cancel that child's phone and take it away and
00:19:13
then separately when they respond it has to be in structured wellth thought out perfectly formatted English and then
00:19:20
then third I said every single email I see from you interacting with your teachers or anybody else that's there to
00:19:27
help you needs to be incredibly well written and formatted and if I see garbage English I'm going to take your
00:19:33
phone away oh okay so you don't want them on their phones but they have to respond right away well they have very
00:19:39
strict rules on what they can use they're there for literally all they can do is communicate like they can use
00:19:45
iMessage but it is shocking to me that despite the lack of games that they have or whatever how poor they are in being
00:19:54
able to communicate and what little access to devices they have have already made them orders
00:20:02
of magnitude less able to communicate than frankly I was able to when I was their age and so yeah I can just imagine
00:20:08
what happens when you become even more ins sconed in something that you can cocon yourself with and not have to
00:20:13
interact with the rest I don't disagree with you I don't disagree with you not to say that it's not going to be a
00:20:18
revenue generator but I think that you could just as easily frankly instead of impacting Apple's
00:20:24
revenues you can probably go along the makers of ssris pot here comes the spread trade pot
00:20:32
Bumble and Tinder and you'll get to the same place economically all right all right here we go we got a lot on the
00:20:39
what a Great Leap Forward for Humanity I can't wait s i I just see this as the laptop replacement okay I wanted to talk
00:20:47
a little bit about what apparently is going to be the the spread trade of the last year meta is continued their
00:20:54
unbelievable run and snap dropped like 30% here's a chart for yall of snap versus meta you can take a quick look at
00:21:02
it here and just for context both companies did great during Co and Zer hit all-time highs in 2021 but they both
00:21:09
got crushed due to the ad spend pullback obviously but then meta started to get less focused on their headsets and more
00:21:16
focused on AI started doing their reduction in headcount 22% year-over-year from 86,000 to 67,000
00:21:24
the last quarter for meta and they're quarterly profits have increased to an all-time high of $14 billion that's
00:21:32
profits folks in Q4 for meta all-time high for the stock price $470 a share $1.2 trillion market cap sna down 60%
00:21:43
from its closing price on its IPO Day in 2017 let me just jump to chth before I get into more charts and everything you
00:21:49
pointed out chth and maybe you could explain to the audience just how ridiculous the voting rights were and
00:21:57
the mass massive dependence that the snap team and the executives had on stock based comp two issues for
00:22:06
youth well I mean I think I said it before I think that case studies have been written about
00:22:13
how tilted the governances in snap I think the point is that they basically have infinite to zero voting power over
00:22:20
common shareholders so there's no real feedback loop and I think that that has probably adversely affected
00:22:28
the types of people that traffic in their stock now look activists and short sellers sometimes
00:22:38
have a very bad reputation but if you steal man their side of it what they are there to do is
00:22:45
to shine a light on inefficiency and in the short seller case sometimes impropriety but it should all lead to
00:22:52
companies being better run right I think meta had this example where they had a really big Hiccup and everybody
00:23:00
including us sort of pointed out the levels of spend that they were making really
00:23:07
didn't make any sense I think we had a chart that compared the level of spend of meta second only
00:23:12
to like the spaceship program right just like Bonkers an enormous amount of money
00:23:17
and look Mark got the message he heard it loud and clear I think he got fed up with whatever was going on there and he
00:23:25
fixed it and it's in the number now I don't know snap because to be honest with you I've never taken more
00:23:32
than one second to look at that company and the reason is there's just zero ability for me to have any useful say so
00:23:40
I've never honestly looked at its performance I've never studied a single characteristic I've never trended it and
00:23:47
I think the point is that I am probably where a lot of other reasonably smart folks who could give a reasoned opinion
00:23:54
on how to make it better land and part of the reason is because there is no feedback loop that matters yeah and when
00:24:02
you know that why would you waste your time at least in other options right there are other options and and meta was
00:24:08
another one you know you can write a letter it gets picked up on CNBC and Bloomberg and whatever and all of a
00:24:15
sudden they kind of pay attention and I think and you look at Disney Nelson pelts goes and gets Ike pearlmutter
00:24:21
shares buys some more takes a Lars yeah we'll see whether that fixes itself the point is that and of these other cases
00:24:29
people are investing the time because they think that there's even a small shred of a chance that the company
00:24:35
listens but if you literally have no say you couldn't even do a proxy you couldn't vote the shares why would you
00:24:42
bother and I think that that's more of an example where maybe there is a I so I don't even know why snaap it poorly and
00:24:50
again I'm not going to really take the time because it's like why bother taking the time sack should should they unwind
00:24:55
this like no voting common shares super voting shares nonsense and and should this go away as a concept in the stock
00:25:03
market well I mean Facebook or meta has a pretty similar concept I mean I guess Zuckerberg has 60% voting control
00:25:12
whereas Evans spel has 99% so snap is more egregious the difference is that Zuckerberg is listening and Spiegel is
00:25:21
not the the reason why snap is doing poorly is not because its Revenue has erated so I looked up or let's put it
00:25:30
this way I asked chat GPT for their key metrics so assuming GPT is not hallucinating if you compare 2021 to
00:25:39
2023 their total revenue went up from 4.1 to 4.5 billion and gross profit went from call it 2.4 to 2.5 billion so not a
00:25:51
huge increase but revenue and gross profit were slightly up but if you look at operating expenses they went from 3
00:25:58
billion to 4 billion a year and that is why their operating income or operating loss went from a $700 million loss to$
00:26:07
1.4 billion loss in two years so that's the source of the problem is that they increase their operating expense by a
00:26:16
billion dollars a year from 2021 to 2023 it's pretty simp they seem like they're
00:26:22
the last ones to get the memo yeah they they were the last ones to get the memo and just just finish the point so you
00:26:27
saw that a few days ahead of this quarterly announcement where their stock got crushed they put out a press Le
00:26:34
saying they're going to cut their head count 10% it's too little it's too little too late yeah they knew right
00:26:40
they knew they had a problem so they released the the Press Le saying oh we're going to cut well you should have
00:26:46
done what Zuckerberg did you know Zuckerberg did a 20% cut last year he got serious he got lean and fit and
00:26:55
instead these guys held out did nothing then when they know that the Market's going to crush them they put out this
00:27:01
lame announcement 10% no not 10% really if you just want to get back to where you were two years ago in terms of
00:27:09
operating expense you need a 25% reduction yeah yeah but it's more than that if you look at the numbers let's
00:27:16
use operating cash flow was 165 million for SNAP for the quarter so their operations generated 165 million of
00:27:22
profit but for the entire year because they lost money in the quarters prior they generated free cash flow of only
00:27:29
$35 million so the business net in produced $35 million of incremental cash you know how stock-based comp accounting
00:27:37
works the charge happens when it vests so this is what employees are vesting during the year of 2023 employees vested
00:27:45
$1.3 billion of stock-based comp so that means new shares or options were issued
00:27:50
that on an accounting basis the options are valued using black schs and the shares are valued based on the share
00:27:55
price so they issued 1.3 billion of stock based comp so they generated 35 million of free cash and they used $1.3
00:28:02
billion to compensate employees beyond their Opex so that means that they paid employees 40 times the free cash flow
00:28:10
that was generated for shareholders during the year which is also equivalent to 10% of the Enterprise market value of
00:28:17
this company so the Enterprise value of the company is $15 billion 10% of that was issued to employees to compensate
00:28:24
them now let me give you the the the story of another city meta and by the way snap's share count because they
00:28:30
issued all the stock the number of shares outstanding increased by 4% during the year during the year meta's
00:28:37
number of shares outstanding decreased by half a percent because they used cash to go and buy back stocks so they were
00:28:43
able to reduce the shares outstanding now as you guys talked about meta cut employee count by 22% and snap cut
00:28:49
employee head counts by 3% during the year but here's the crazy difference in performance the stock-based comp expense
00:28:57
for meta during that year was about $14 billion invested that year that company generated 71 billion of operating cash
00:29:06
flow so while while snap gave employees 40 times the free cash flow meta gave employees you know about a 20% of the uh
00:29:15
of the free cash flow and then and then meow went around and they used some of that extra cash to buy back $20 billion
00:29:20
of stock so they bought back more shares than what the employees were issued that
00:29:24
that year were so it shows such a differ in looking out for shareholders so if I'm an investor and by the way meta is
00:29:32
trading at like 25 times free cash flow which is not a crazy multiple given all the new businesses that they have in llu
00:29:37
and the progression to cloud and other things that they might do if I'm looking at those two businesses as a shareholder
00:29:43
you got this guy that controls the whole stock he's giving employees a billion three of shares a year when he's only
00:29:49
making $30 million of free cash flow a year and then the other guy is issuing $4 billion of shares buying them all
00:29:56
back and he's making 70 billion of free cash flow a year I don't know it's very hard to decide which one to go after
00:30:01
Spiel brought it up in an interview I saw and a lot of the layoffs were topheavy so he got rid of a lot of the
00:30:08
top people who had these huge comp packages and then what I'm hearing from a lot of Executives is cutting these
00:30:15
highly stock comped Executives who are you know also have big cash comp cutting them putting lieutenants in charge and
00:30:23
then moving more jobs to other locations where people don't expect stop stock-based comp you know if you're in
00:30:29
India or you're in South America whatever you know stock-based comp is not like the obsession it is here so as
00:30:36
everybody optimizes these businesses I mean Facebook even why do they need 5,000 employees so they announced
00:30:43
roughly 500 job cuts out of what 5,500 employees that's crazy I mean should that company be operating with 2,000
00:30:53
employees it's good question cut the number of Twitter employees from 8,000 to 1500 when you look at the number of
00:31:00
apps that they're running and the number of products that they're running compared to meta right meta has far more
00:31:04
apps far more infrastructure meta is serving 3.2 billion daily active users snap is about 400 million So Meta is 8x
00:31:14
the users with many more applications and much more infrastructure so I think it's a it's
00:31:20
another great kind of ratio to look at the performance of these two I think you're exactly right yeah the other
00:31:26
advantage that it has is because they're so profitable they have the resources to
00:31:31
go big in AI big time which is very expensive so yeah so they are the leader you get all this option value at meta
00:31:37
which you don't get at snap there's all this infrastructure that they can leverage much like Amazon did with AWS
00:31:43
into things like Cloud AI tools for thirdparty developers third party applications and then obviously the you
00:31:51
know meta is the biggest advertising platform next to Google in the world now and there's much more that they can
00:31:57
start to do to to extend further into the the they get they did get an awesome save remember Apple screwed them and was
00:32:04
like you can track devices now and like that just took a massive hit in the ad Network and it was all those headwinds
00:32:11
they were like okay we're just going to use AI to optimize ads and supposedly the AI optimization of ads I was talking
00:32:16
to somebody on the inside they said like yeah we got it all back we gained it back we've got massive AI advertising
00:32:24
optimization going on so totally yeah that's great that Tim Cook you know kicked us in the nuts but we don't care
00:32:30
by the way that's a great point jcal it really says a lot about how meta was able to respond to that change which a
00:32:37
lot of people speculated would destroy the advertising business and the fact that they were able to engineer
00:32:41
solutions to drive advertising Revenue up to $40 billion it's just mind-blowing it's a really kind of impressive outcome
00:32:49
for the team and I think it speaks a lot to the quality of the engineers there yeah I think it's a great Point yeah
00:32:53
saaks you tweeted that you're seeing a little SAS bounce back all of a sudden that's interesting I am seeing something
00:33:00
similar last year last two years you had a ton of people cutting their SAS Bend maybe removing the number of SAS vendors
00:33:07
they had consolidating vendors uh you tweeted many public and private software companies are experiencing accelerating
00:33:12
growth after six to 7 quars of deceleration SAS recession appears to be over according to the SAS Master David
00:33:21
saxs you want to unpack this for us what do you seeing well it's still pretty early because not everyone's reported
00:33:26
but if if you looked at the big Tech Cloud performance in Q4 you could see that there's a bounce back in here this
00:33:34
is net new ARR added for AWS Azure and Google Cloud so you see here in Q4 that there's a huge increase in net and U AR
00:33:46
for the the big cloud computing platforms and then I think another Bell weather is at lassan so we're still
00:33:52
waiting to hear from HubSpot Salesforce Zoom Adobe companies like that they haven't reported yet but if you look at
00:33:59
makes jira amongst other products they're based in Australia yeah the major yeah exactly collection of SAS
00:34:05
companies right it's a collection of SAS products yeah so net new ARR would be the amount of growth in that quarter and
00:34:13
this is on a year-over-year basis so you can kind of see Q4 of 21 was the absolute Peak and then it plummeted and
00:34:23
then it actually went negative for about a year that's that's tough to be in a company with new AR going negative yeah
00:34:31
yeah that doesn't mean by the way the company's shrinking it just means that the amount of net new ARR which is the
00:34:37
amount of growth is actually smaller than that same quarter a year before yeah and then in Q4 you could see
00:34:46
there's some acceleration here that they're starting to add more they added more net new ARR I guess 33% more in Q4
00:34:53
than they did over the previous year and part of that sacks is because the comps
00:34:57
are lower and they kind of bottomed out yeah they bottomed out now they're re accelerating so you know we're starting
00:35:03
to see this in some of my board meetings as well where in 2022 everybody was missing their numbers and reforecasting
00:35:11
down and then they would miss the reforecast yeah so by 2023 the forecasts were very very
00:35:18
conservative and I would say and now I'm seeing companies beat the the sort of the lower forecast in Q4 this wasn't
00:35:27
happening earlier in the year but finally I think people are starting to beat their sort of their lower forecast
00:35:32
for Q4 that's the question that I was curious about what do you what do you actually think is happening is that
00:35:38
we've Reb baselined these businesses so now what would have looked like just a massive Miss over the last two years now
00:35:45
looks like a beat because we've just completely reset expectations is it that or is it that the economy is actually
00:35:52
expanding and we can count on some reasonable growth rates is it a combo of the two what do you think it actually is
00:36:01
yeah I mean it's definitely a new Baseline in the sense that and if you go back to 2020 or 2021 we considered good
00:36:10
growth to be you know 2 to 3x year-over year and now if it's going from 60 to 80% growth year-over year you're happy
00:36:18
so there's definitely been a lowering of expectations that being said you still see in these numbers there has been a
00:36:24
bottoming out and we're starting to now grow from this new Baseline so for example I think with
00:36:33
atlassian here we are seeing an increase in spend basically in in growth right so
00:36:39
the way a recession is typically defined is uh two quarters of negative growth right we had six to seven quarters of
00:36:46
decelerating or negative growth in SAS in Tech in SAS which is why called it the session or be the yeah it was
00:36:54
actually kind of a depression you're right but now we're seeing quarter over quarter growth so growth is
00:37:00
reaccelerating growth is higher than it was so is it going to get to where it was that probably will take some time
00:37:06
but it feels like the problems in the ecosystem work themselves out and now we're back to growth again that yeah I
00:37:12
can add psychologically because I'm on a couple of SAS boards as well and psychologically it felt like you tell me
00:37:18
if I'm right SAS saak if you saw the same thing there were two years of calling up customers and they were like
00:37:23
we're we're consolidating vendors and by the way we did a riff and so we need 20%
00:37:27
less seats so we're going to have 20% less SAS companies that we're buying from and we're going to have 20% less
00:37:35
seeds so you start putting that all together man everybody was just in psychological triage mode we cannot
00:37:41
spend money I don't want to lose my job so you're if you're a procurement person
00:37:45
you're the CTO you don't want to lose your job you don't want to have more Cuts so you're like well I can cut some
00:37:49
software costs do I get points for that and the points you would score for the last two years was cutting costs with
00:37:55
the market RI in and you you know now got a really you know efficient company you're like hey can we spend a little
00:38:01
bit on SAS to make the remaining employees even more you know productive okay maybe that's a reasonable
00:38:08
discussion and then people are playing ball in terms of negotiating prices so that's the other thing I see is like
00:38:14
people are like we we'll take your software but here's what we want to pay and then they're coming to the board and
00:38:18
saying can we do this deal would have been a million dollar deal but it's a $200,000 like yeah take the money take
00:38:23
the money let's let's be hug that customer the market is generally an escalator on the way up an elevator on
00:38:29
the way down so the recovery is going to take a long time but at least we've bottomed out and we're in recovery as
00:38:35
opposed to continuing declines yeah by the same token if you're a startup and you're not seeing Improvement in your Q4
00:38:42
sales then you no longer have a macro excuse for why you're not doing well interesting and then freeberg you added
00:38:50
you know you're like I'll I'll make my own software you said uh you know some softare software is too expensive I'll
00:38:56
put a developer on it and so how's that working out for you are you still in that mindset of like yeah maybe we just
00:39:02
build our own software yeah I mean I it's not just us I think we're seeing a lot of companies pursuing this path a
00:39:09
couple Engineers can rebuild the functionality of core applications particularly because I think if you
00:39:16
think about the business model that makes SAS so great is they could value share rather than charge the cost of an
00:39:22
engineer plus some margin the business model the equity value that comes in software if you can build something once
00:39:31
that creates $100 of value you could probably charge your customer $30 $40 for that product because it's saving
00:39:37
them 60 bucks 70 bucks and they'll make that switch to software so you know the ROI driven value share model in SAS has
00:39:46
made it incredibly valuable the problem now is that an engineer can be hired to build the replacement and so it creates
00:39:54
price compression so the SAS company can no longer capture that much value because the savings is actually less
00:40:00
than that because the Enterprise might say hey I'm going to hire someone and instead of spending 60 Grand a year on
00:40:07
your software I'm going to allocate a quarter of an engineer's time to build that software and it's going to replace
00:40:11
that that cost so I think that that's still the case so while there might be bookings there's still which are driven
00:40:18
largely by a search for efficiency gains a search for more profitability for more
00:40:22
productivity within an Enterprise there are other options for that Enterprise to
00:40:26
realize that productivity gain today and that's what's going to cause perhaps price compression and more competition
00:40:33
than has been the case but I don't think that the adoption of software is going to slow down it certainly seems to be re
00:40:40
accelerating which is great competitive right we're moving into a hyper competitive market right especially with
00:40:45
AI it's a mix of internal software it's a mix of internal software as you guys know there are very few traditional
00:40:51
non-tech Enterprises now that don't have a software team that can write code so now that so many companies have software
00:40:58
teams that write code they're all going to be asking the question should we be buying the software or should we be
00:41:02
building something internal yep it's a classic buyer build situation all right let's talk a little bit about VCS and
00:41:08
how they're investing in AI there there seems to be three camps shaping up here chamath you know one group is like and
00:41:14
the incumbents are going to win you know Microsoft Google Amazon everybody they're going to win the day so they're
00:41:19
going to wait and see then there's another group who's sitting it out because they're like hey open sour is
00:41:25
going to win meta committed to open source and collaborative platforms I've been playing with hugging face with suep
00:41:34
as well as you Chim moth and it's pretty amazing what's happening over there and
00:41:37
then a bunch are obviously placing bets right now the valuations are absurd Founders fund and inre and Hartz two
00:41:43
notable firms are approaching it differently Founders fund bought into open AI at a $29 billion valuation but
00:41:51
aside from that investment they're generally avoiding the AI deals on the other hand andreon is is betting heavily
00:41:58
character AI repet 11 Labs mistel you're also in repet Sachs so what do you think
00:42:05
is open source going to win the day you've been picks and shovels the whole way you've been talking about
00:42:09
compression maybe this isn't actually a good Market what's you're thinking as a capital allocator J I think foundational
00:42:16
models will have no economic value I think that they will be an incredibly powerful part of the
00:42:22
substrate and they will be broadly available and and entirely free wow so if you think about that any closed model
00:42:31
especially a closed model that operates on open on the open internet is not very
00:42:37
valuable and any open source model that operates on the that trains on the open internet will make that so so in that
00:42:46
world things like mistol and llama will essentially Decay the market to zero so if you if you're looking at
00:42:55
any economic value that has been captured up until today if it has been captured by having a proprietary closed
00:43:02
model trained on open data that economic value will go away and I think Google and Microsoft and
00:43:11
Facebook and Amazon and all these startups have a deep economic incentive actually to make that so so now you can
00:43:20
evaluate what that means so if you get an opened model from hugging face that's just kick out
00:43:27
where do you spend money well you're going to have to spend money to actually train it to finetune it maybe to have
00:43:35
some pretty Zippy inference and all of that means that there's a new kind of substrate that has
00:43:41
to be built which is all around the way that the tokens per second are provisioned to the apps that sit on top
00:43:48
of the model what that means is you need to go back to 2006 and7 and say okay when we first created the
00:43:55
cloud who made money and fast forward 18 years later it's the same people that are still making money so the people
00:44:03
that made money in 2006 and 7 were Amazon principally because of ec2 and S3 The Perfect Analogy of ec2 and S3 in
00:44:14
2024 is the token per second provider now there you have to double click and say okay well what does a tokens per
00:44:21
second provider need to do to make a lot of money and I think the ultimate answer
00:44:25
is you need your own proprietary Hardware so who is in a position to do that Amazon has announced that they have
00:44:32
an inference and training solution for training cerebrus has announced a pretty compelling solution Google obviously has
00:44:39
TPU then there's a handful of startups including one that I helped get off the ground in 2016 that I funded called Gro
00:44:47
all of those companies are in a position to build a tokens per second service then you have companies like together AI
00:44:53
which basically just go and take Venture money and wrap Nvidia gpus and you can debate what the
00:45:02
advantage will be there one could say well it's not really a huge advantage over time so my refined thoughts today
00:45:11
are sort of what my initial guess was when we started talking about AI a year ago which is the picks and shovels
00:45:18
providers can make a ton of money and the people that own proprietary data can make a ton of money but I think open
00:45:25
Source models will basically crush the value of models to zero economically even though the utility will go to
00:45:31
Infinity the economic value will go to zero did any of you guys see chim's interview with Jonathan
00:45:37
Ross NOP not yet you put it out right Cham you made it public you know I did it just for my subscribers but Jonathan
00:45:43
is the founder and CEO of grock the company that I just mentioned and the quick version of that story is I was I
00:45:50
would I would pour over the Google earnings results in the mid teens of 2000 cuz I was pretty actively investing
00:45:57
in a bunch of different public equities and Sundar said in a press release he mentioned that they had rolled their own
00:46:03
silicon for machine learning called TPU and I was like what is going on that Google thinks that they can actually
00:46:12
roll their own silicon what must they know that the rest of us don't know and so it took me about six or nine months
00:46:18
but through sunny I got introduced to Jonathan and then we were able to get Jonathan to leave Google and he started
00:46:26
and he Jonathan was a founder of TPU at Google and then he started grock which I
00:46:30
was able to lead that funding round in in 2016 so eight years ago anyways I did a a spaces with
00:46:39
Jonathan talking about the entire AI landscape and AI acceleration to my subscribers but it was so good I got to
00:46:46
say he is he was so impressive that we kind of like figured out a way to just play the space and
00:46:56
tape it and then we published it to everybody so it's it's on it's on my Twitter for anybody that wants to listen
00:47:00
to it it is amazing he is really impressive I was sitting on the 17 going to Santa Cruz not moving for hour and a
00:47:11
half and I listened to it so I kept me alive but I thought it was really great what you think he is great no he's great
00:47:18
some great insights and I think he's very compelling in arguing why some of the big cloud providers today that are
00:47:27
offering infrastructure for AI model training and inference are going to be challenged if
00:47:35
someone can build full stack and be and do it successfully so it was a really good interview I actually think it's
00:47:41
really worth listening to but I enjoyed it yeah thanks for putting it out there I was like
00:47:46
literally just sitting I was sitting in the car browsing Twitter and I saw your thing and I clicked on it and then I
00:47:51
just ended up listening it was a little it's a little hard actually when you do a space for your Subs you can't actually
00:47:57
just flip a switch and then release it to all of your followers so we actually had to like literally play it and then
00:48:06
just capture the audio out and then republish it but anyways despite that inconvenience if anybody's interested in
00:48:12
learning about AI Hardware he is very compelling and he's very educational so saak your thoughts on just how you're
00:48:18
approaching investing in AI if you're specifically investing in the underpinnings of AI pix and shovels yada
00:48:24
yada or if you're just looking on the application Level and it's you know you know that kind of approach well we we
00:48:30
divided the space into three categories uh one is the the models themselves the foundation models which can be either
00:48:37
open source or closed Source there's infrastructure so like jam saying it could be like model
00:48:44
training it could be Vector databases tools that developers use to create the AI stack typically inside their
00:48:52
Enterprise and then the third would be applications which can be things like co-pilots or it could be a pre AI app
00:48:59
that's using AI to kind of turbocharge its capabilities yeah most SAS would be in the application bucket and so that's
00:49:09
principally where we're focused although we do look at infrastructure plays and models however I do think there is an
00:49:15
argument for I mean really with the question of commoditization well like all the model companies just get totally
00:49:23
commoditized we really we're talking about open AI right because they're the leader so the question is can they
00:49:27
maintain their lead I do think there is an argument that open AI will stay in the lead and actually do
00:49:36
quite well and I think there's a few points there one is that if you're a consumer you just want to use the best
00:49:44
GPT you want to use Google Ex it's just like search right if Google is a little better or the perception is it's a
00:49:51
little better than Bing or the other search engines you don't win a plurality of search traffic you actually end up
00:49:58
winning it all because consumers just want the very best one so most of the tests show that open AI is still ahead
00:50:05
of the open source models and I think even people in the open source movement will tell you that open AI is call it
00:50:10
six months ahead they have no doubt that open source will get to where open AI is
00:50:15
now in six months nonetheless if open AI just maintains a little bit of a lead over open source then it could compound
00:50:26
yeah it basically win the vast vast majority of the call it consumer search or consumer GPT market so that's Point
00:50:33
number one point number two is now that open aai has these hundreds of millions of consumers using it that's a pretty
00:50:42
attractive audience for developers to want to reach and open AI has done a really good job creating a platform for
00:50:50
developers to create you know what are called custom gpts so most developers don't want to go through the hassle of
00:50:59
training a Model fine-tuning A model doing all of that work that you would have to do in the open source ecosystem
00:51:04
they just want to point chat GPT at a repository of data or documents information have it learn what it needs
00:51:15
to learn fine-tune it in that way maybe add some lightweight functionality using
00:51:19
open ai's platform to create a custom GPT that's what I think most developers want is they just want a simple stack to
00:51:27
work with and they're going to prize again Simplicity and the power of the developer tools over the theoretical
00:51:35
control they get by rolling their own models training and funing their own models in open source and so I think
00:51:41
what you're seeing now is I mean how many custom gpts have already been created on the platform it might be tens
00:51:47
of thousands I mean there's so Millions yeah so easy to create them yeah so you have a classic developer Network effect
00:51:53
where you've got open AI aggregating hundreds of millions of consumers because they perceive that chat GPT is
00:51:59
the best then you've got developers wanting to reach that audience so they build custom gpts on the open AI
00:52:05
platform that actually gives chat GPT more capability yeah and that's something that open source can't easily
00:52:11
catch up with actually actually just finish the point so yeah so it is a flywheel where you know classic
00:52:19
operating system developer Network effect where you want to use the operating system
00:52:24
that is the most programs written for it yeah and interestingly hugging face has
00:52:30
realized this and hugging face released this week their own version of gpts which is really interesting and you can
00:52:36
pick Sachs which open source project you want to use to make it so unlike gpts on
00:52:41
chat GPT we have to pick theirs on the hugging face one you could pick you know llama or whichever one you want there's
00:52:49
an account called artificial analysis that you can follow the thing to keep in mind saaks is that for any of this to be
00:52:56
true these apis need to be usable right I mean I don't know if you remember but when we were building apps even as back
00:53:02
as the late 2000s and early 2010s one of the things was there was a pretty important paper that was published by
00:53:09
Google about attention span and it would look at page load times in a cold cache
00:53:15
environment right and it basically said you have to be at like 150 milliseconds right that's like Best in Class
00:53:22
performance or faster and I remember when we read that at Facebook we went crazy so much so that at one point a
00:53:28
small team and I kind of actually launched a strip down version of Facebook to compete with Facebook if
00:53:34
there's a Nick you can probably find this article on Tech runch and we did it without telling everybody was called
00:53:38
like Facebook zero anyways the point is speed matters because in the absence of having very Snappy response you could
00:53:45
have the best model in the world but if it takes 10 20 30 seconds to basically initiate and get back data from a fetch
00:53:52
request it's an impossible thing to to do so I think one of the things that you have to keep in mind is that there are
00:53:59
these two things that need to move at the same time one is the quality of how the model is but two is the speed and
00:54:04
its responsiveness which is a function of again hardware and your ability to basically tokenize tokens per second
00:54:11
very very quickly so that developers are incentivized to not just play around in
00:54:15
a sandbox but to actually build production code and I don't think we've seen that second thing happen because
00:54:21
nobody is delivering it and that's the big thing that nobody talks about for example like AWS if you look inside of
00:54:27
how expensive it is to build an app there I've tried even when they give you credits the credits they give you aren't
00:54:32
sufficient enough to even pay for half the power and then the way that they schedule and the way that they try to
00:54:39
orchestrate you to use Hardware makes building production apps unless you are willing to spend millions and millions
00:54:45
of dollars for a very slow app unfeasible and so if you go back to a startup economy raising money here the
00:54:53
Venture investor should start asking the question well what is the speed and usability of these services that I'm
00:55:00
funding and the reason is because you could build the best experience in the world that runs on Local Host but if all
00:55:07
of a sudden you actually try to launch it as an app and the thing takes 35 and 40 seconds to generate something it's
00:55:13
DOA and I don't think enough people ask those questions or understand that that's true so this is why I think you
00:55:19
have to sort of be looking at both of these two things at the same time but this account is interesting because it
00:55:25
kind of just strips things down to the bare facts and they start to allow you as a third party to understand what you
00:55:35
can do yeah speed is just such a a critical component of this and what Google found was as you know free
00:55:41
Brokers you were there every time they lowered a certain number of milliseconds the usage went up right people did more
00:55:47
searches which makes sense if you get your results back faster yeah it was a key metric from day one at Google
00:55:53
Marissa Mayer ran all the consumer facing products at Google during this you know earlier era she was like beat
00:56:01
it into the team I mean if you guys remember one of the first the the the first kind of early feature of the
00:56:06
Google results page was the amount of time it took to load the results they'd show you how many milliseconds yeah they
00:56:11
show you that yeah they they literally put your Northstar metric exposed to the consumer which totally that must have
00:56:16
lit a fire under the asses of all the developers and server people yeah well I mean they were kind of showing off the
00:56:21
quality of the infrastructure and the way they did indexing and everything but the result
00:56:26
really played out in usage the the faster the results the more frequently you would use the search engine and the
00:56:31
more likely you were to come back and it's amazing how much consumer Behavior drifts based on milliseconds like you
00:56:38
have a few milliseconds of delay learned this right I mean if you look at the if
00:56:42
you ever see the movie the founder where they explain the McDonald's process they
00:56:45
learned it too guys look at this this is really interesting on this analysis I mean tth are you saying that you don't
00:56:51
think open AI can achieve the necessary levels of performance no I'm saying two things open AI is three different
00:56:57
businesses open AI has a closed model that's trained on the open internet I think economically it's going to be very
00:57:03
hard to sustain that unless they start buying all number of apps so that they can get some fine tunes that they
00:57:10
control that are proprietary to them so for example if open AI were to buy all of Reddit that would be a really
00:57:15
interesting development that would improve the quality of open AI in a unique and differentiated way relative
00:57:23
to where things like llama Mist will get to at the same time as well as X's Gro I
00:57:28
think they're all going to converge to the same quality in the next probably 12 to 18 months that's Point number one
00:57:35
your belief there is there's enough data in those pools that everybody reaches parody no did you guys okay Nick did you
00:57:42
so I I published this primer on a primer yeah yeah there is a slide in there Nick
00:57:47
that you can pull out but it just shows you that there is a converging in the quality of the results
00:57:55
as the number of the parameters of the model gets higher and higher and what it effectively shows you is that we are
00:58:00
already in the land of diminishing returns when models are trained on the same underlying data so if you are using
00:58:08
the open internet llama mistl open aai they're all getting to the same quality code point and they will be there within
00:58:15
the next 6 to n months so that's business number one on open AI business number two is a consumer facing app
00:58:22
called chat GPT that has a lot of legs because I think people are you know develop habits it'll be very sticky and
00:58:29
I think it'll get better and better and then the third business that they're in is selling Enterprise services to large
00:58:36
Fortune 500s in fact if you look at their open AI day what they talk about is they sell they've sold already to
00:58:42
like 94% of the Fortune 500 what does that mean I think what that actually means is they've sold a lot of test
00:58:48
environments and sandboxing but again in order to translate that into functional
00:58:53
production code that's used by Bank of America right or Boeing in production you have to have Zippy Zippy
00:59:02
fast SAS and a level of performance that no cloud provider yet has delivered none
00:59:09
nobody so Nick if you just go to that please the thing I just wanted to show you this because it's a really
00:59:14
interesting chart this is not mine this is theirs if you look at quality versus price saxs it starts to starts to show
00:59:20
you like where do you want to be you want to be in the upper left quadr in their analysis
00:59:27
right and so the point is what you can see is that a ton of different models are getting to the same place and so
00:59:35
obviously you'd want to use the model that's the cheapest or most convenient well who's going to pay for
00:59:41
that if you if you and your LPS want to pay for that the person that figures out the way
00:59:47
that it's the cheapest to give you the same answer will actually end up winning because you will run out of money and
00:59:52
they will not I don't know I mean I think that there's a lot of business problems inside companies where people
00:59:58
just want to very quickly set up their own again custom GPT without having to go
01:00:05
through the time the cost the hassle of trying to do model training or fine-tuning so let's just back up here's
01:00:12
the path that open AI is on so step one get hundreds of millions of consumers using it and getting them to view open
01:00:21
AI or chat GPT as the Google in this area right strong presumption this is just the one you go to when you have a a
01:00:30
question step two these same people these same consumers now want to use chat GPT at
01:00:36
work because there's some research they want to do so openai has just rolled out
01:00:42
um both Enterprise licenses and team work spaces so you can work collaboratively on the same queries in a
01:00:48
team work space step three is the rolling out a very easy to use Dev platform that allows devel Vel opers to
01:00:55
again create custom gpts by just pointing open AI at repositories okay and so let's say that
01:01:03
you're the customer support team and you want to create a a GPT to help customer
01:01:10
support answer cases you could basically then train chat GPT on let's say every customer support ticket and
01:01:25
email that the company has ever produced right now you could wait for the company's it department to get us act
01:01:32
together and figure out how to train an open source model on the same thing but do you really want to wait for that or
01:01:39
do you just want to get going you know and now open AI has given you the Enterprise license that you need
01:01:46
to pacify the concerns about security and privacy and all that kind of things to some degree there's always going to
01:01:52
be those super paranoid Fortune 500 companies that will insist on doing on owning everything and and doing it doing
01:02:00
it open source let me build on your example so I run a small software company during the day called hustle and
01:02:07
we saw a lot of tickets related to this specific legislation that exists whenever you're
01:02:15
texting or you're doing Auto dialing stuff called 10 DLC and so we wanted to eliminate those those tickets right so I
01:02:25
actually went and I built a GPT which was called the privacy policy generator because a lot of these trouble
01:02:31
tickets were because the Privacy policies were bad and we trained them using a handful of ones that were good
01:02:38
and a handful of ones that are bad with a bunch of rules and I train the model and it's wonderful except I can't run it
01:02:45
in production because it's not the kind of thing that is usable in that way right now it's still very difficult and
01:02:53
so all I'm saying is I'm happy to keep spending a few hundred dollars a month a few thousand bucks a month whatever it
01:02:59
is that I'm spending I don't quite exactly know and I agree with you it was very easy I think open a does an
01:03:05
excellent job of getting off the ground but what I'm also saying is that when you actually translate that into a
01:03:13
Mainline use case right where I want to now give it to my support team and say this is now a tool you can rely on it's
01:03:21
integrated into your workflow into your other tools it's integrated to how you pipe out data into Salesforce
01:03:27
or what have you it's just very hard and I'm not saying it's not going to get fixed I'm saying we're just not there
01:03:33
yet and one of the Rays in which it's not there is that there is no place I can go including open AI that actually
01:03:41
makes it fast enough to be usable in production you wrote this on open AI stack you wrote a custom GPT yeah built
01:03:48
myself yeah and you can do them on hugging face now it's going to be a lot of options in terms of integrating into
01:03:54
your workflows I think it's a really interesting point because I saw a demo somewhere where now actually I think
01:04:01
openai announced this that you can at mention a custom GPT yeah yeah Sunny showed me that this week on the Pod yeah
01:04:09
in in chat GPT you can now mention a custom GPT to kind of invoke it yeah so how it works is you would say hey I'm
01:04:16
heading to New York what flights can I get at Expedia at kayak whatever and then it gives you you know
01:04:23
the results here and you you you're kind of pulling that up just to the point about about where data advantages lie
01:04:29
and that's ultimately going to drive value I cannot I've tried to think a lot about this I cannot think about a better
01:04:38
data advantage that is orders of magnitude better than anything else YouTube YouTube
01:04:46
Say it is so here's here's the numbers I I pulled this up you guys know like gpt3
01:04:53
and three and a half were trained with a heavy waiting on common crawl which is this open sourc we talked about this
01:04:58
before Gil elbaz runs it open source crawling of the web the total amount of data in common crawl which I think it
01:05:06
counted and I could be off on this something like 40 to 60% of the waiting in GPT 3 or 35 I'm off on this probably
01:05:13
so the total amount of data in that common crawl data set is about 10 pedabytes okay based on YouTube's public statement
01:05:22
recently they're seeing about 500 hours a minute of video uploaded or 720,000 hours a day and if you assume somewhere
01:05:31
between you know just under 1080p on that video we're talking about probably one to two pedabytes of data being
01:05:39
uploaded to YouTube per day so if you assume like over time the definition of the videoos gone gotten better and the
01:05:47
amount of uploads gotten up you could probably assume that there's roughly I'm guessing there's probably somewhere
01:05:53
between 2,000 and 3,000 pedabytes of data in YouTube growing by 1 to two pedabytes per day which makes YouTube's
01:06:03
data repository 300 times larger than common crawl which makes it bigger than anything else that anyone else has and
01:06:10
here's the amazing thing about it it has video it has image it has audio it has text it has everything multi and it is
01:06:19
growing so if you were to take a a bet or build a thesis around this point that the data Advantage is going to drive
01:06:26
value creation if Google gets its act together and leverages the data repository at YouTube it is an
01:06:32
insurmountable moat that will only continue to extend because the quality of the YouTube experience and the
01:06:38
network effects continue to accumulate for them so I think it's the most valuable asset in the world today based
01:06:44
on the thesis that AI value is going to ACR to the data owner I think you're making such an important point this is
01:06:50
why the counterfactual is is true and it's actually showing up in the data and Nick
01:06:55
will show you this slide again from from the AI primer but that is why we're seeing these diminishing returns
01:07:00
freeberg in all of these third party benchmarks of these models using the same data set it's all using the same
01:07:05
data set so what we are proving is not that the underlying Hardware can't scale nor that Transformers are only efficient
01:07:12
to a point that's not what all of this convergence is showing it's that in the absence of proprietary data you're just
01:07:17
going to get to the same model quality and we're seeing a bunch of different models get to a very early Finish Line
01:07:24
which again if people like Facebook are doing for free that's much easier to underwrite because you you don't have to
01:07:31
underwrite it being a differentiator in five years but if you have a if you have a startup with Equity
01:07:38
value tied to a model I think it's very it's much more of a tenuous place to be in the absence of proprietary data
01:07:46
and everyone in the world has a camera and a microphone in their pocket and high-speed internet now from the phone
01:07:53
in their pocket and more and more people are uploading that content that that data that's being
01:07:58
generated YouTube's got this free data vacuum and they just out in the world and most of it's getting upload you well
01:08:04
it is public facing though so it's not just true for text it's also true for you know all of the image generation so
01:08:12
like if you look they can train more than just an llm on it right they can build all sorts of yeah go ahead no no
01:08:18
no I was just going to say like the version of common crawl for training these image models also exists and so to
01:08:23
your point it's like we are all operating from the same brittle very fixed small Quantum of training
01:08:31
information and so that is why I think like Facebook and Google are doing a really
01:08:38
important job by deciding that these models should be free right and then being able to so then the question that
01:08:46
just accentuates their data Advantage it does and and I think that it allows them
01:08:51
to decide how much to leak out so for example whenever like if you were using a lot of Google services like GFS big
01:08:59
table big query you know tensorflow the versions that you had access to Via gcp was always one or two
01:09:08
generations behind what the Google employees got to use right but it was still so much better than anything else
01:09:15
that we could get anywhere else that you would still build to those endpoints and
01:09:18
I think there's a similar version of this where Facebook and Google probably realize like look we'll have version
01:09:25
five running internally to optimize ads and all of this other stuff that makes our business that much better and we'll
01:09:31
expose version three to the public but version three is still trained on so much proprietary data that it's so much
01:09:36
better than version 10 of anything else that's just operating on the open internet right and and you know to your
01:09:43
point freeberg that's the outward facing stuff YouTube is a collection of things
01:09:47
people want to share what Google also has is Google Docs and Gmail things that people say say privately so they have
01:09:55
another data resource there that they can tap you know and and there'll be regulations of privacy around that but
01:10:01
maybe there's a difference there but I I honestly can't think of the quantum coming close to YouTube not even close
01:10:07
well the the thing to Jason's point which is really interesting is like you know there's a modality in AI called rag
01:10:13
where you can actually just augment with very specific training on a very specific substate of of documents to
01:10:19
improve it's like a it's like a hacked version of a fine tune but the Beautiful about that is like if you have a Google
01:10:25
workspace my entire company runs on on Google workspace in fact most of my companies do at this point to click a
01:10:32
button where all of a sudden now all of that stuff in all of my G drives all of a sudden is trainable so that the N plus
01:10:40
first employee comes in and has an agent that's tuned on every deck every model spreadsheet every document that's a huge
01:10:49
Edge totally huge Edge by the way and as a CEO if you gave me that choice I don't
01:10:55
think anybody underneath that reports to me has any right to make that decision but as a CEO I would click that button
01:11:01
instantly and I had that right as a CEO and so like that's the CEO pitch it's like look I can just give you these
01:11:06
agents that are that are like the next version of a knowledge base that we've always wanted inside of a company right
01:11:14
notion has this you know they' basically you can start asking your entire notion
01:11:18
instance questions about notion which is incredible and uh yeah you can just and
01:11:24
as a CEO you can see across everything chth because as you know with Google Docs if you're in a compliance-based
01:11:30
industry like Finance you can see everything every message every email every document and you can search the
01:11:37
security model and the data model becomes very complicated in all of that stuff like for example like how do you
01:11:43
know that this spreadsheet is actually you should learn on it but who gets to actually then have that added to
01:11:51
the subset of of answers right all all of a sudden like salaries the information gets put into the
01:11:58
training model very dangerous or subset a of a companies working on a proprietary chip design that they
01:12:03
actually like the way that Apple runs highly highly segregated teams where nobody else can know so there's all
01:12:10
kinds of complicated security and and data model and usage questions there but yeah BR new world so there's been a lot
01:12:16
of discussion real estate you you shared a video with us why don't you kick it off for us here preber what's going on
01:12:21
in commercial real estate and saxs you've got Holdings and a lot of as well so let's kick up the commercial real
01:12:26
estate challenges of the moment well I mean I think we're teeing off of Barry's comments at this event last week he and
01:12:34
I met backstage because I spoke right before him and then he gave this talk which is available on YouTube where he
01:12:41
talked about the state of the commercial real estate market and particularly he talked about the office Market just to
01:12:47
take a step back to talk about the scale of commercial real estate as an asset class in the US Nick if if you'll pull
01:12:53
up this chart the total estimated market value of commercial real estate in the US across different categories is about
01:13:01
$2 trillion with about $3 trillion being in the office Market which is specifically what he was talking about
01:13:07
he was saying that in the US we're seeing people not coming back to work and all these offices are empty and
01:13:13
we've talked a lot about these offices being written down so how significant of a problem is this so $ 20 trillion asset
01:13:19
class obviously the multif Family Market is probably not as bad as office and retail which are the most heavily
01:13:25
affected Each of which are about $3 trillion a piece the rest of these categories seem relatively
01:13:32
unscathed in comparison industrial Hospitality healthare you know those those real estate sectors are probably
01:13:39
pretty strong data Cent is obviously growing like crazy Self Storage is a great Market if you pull up the next
01:13:44
image so it turns out that of the 20 trillion do of market value there's about $6 trillion of debt so you can
01:13:52
kind of think about that 20 trillion being 6 trillion owned by the debt holders and 14 trillion by the equity
01:13:59
holders and the debt is owned roughly 50% by Banks and thrifts and this was this concern that we've been talking
01:14:08
about with higher rates is the debt on office actually going to be able to pay the debt on retail going to be able to
01:14:12
pay when half of that debt is held by Banks and thrifts that as we' talked about have such a close ratio to
01:14:20
deposits that you can actually see many banks become technically insolvent if the debt starts to default Barry's point
01:14:28
that he made was if you look at the office Market which you know is marked on everyone's books as $3 trillion of
01:14:35
market value he thinks it's probably worth closer to 1.8 trillion so there's $1.2 trillion of
01:14:42
loss in the office category and if you assume 40% of that 3 trillion is held as debt you're talking about $1.2 trillion
01:14:51
of office debt a reduction from 3 trillion to 1.8 trillion means that the equity
01:14:59
value has gone down from 1.8 trillion to 600 billion so they've lost Equity holders in office real estate have
01:15:07
probably lost 2third of their value two-thirds of their investment and who owns all of that most of that 60 plus
01:15:16
perent call it 2third of that is likely owned by private Equity Funds and other institutions where the end benefici is
01:15:23
actually Pension funds and retirement funds and so if 2third of the value has to be written off in these books and it
01:15:29
hasn't happened yet what's going to happen to all these retirement funds and this is where going back to my
01:15:34
speculation a couple months ago kind of gets Revisited if you're actually talking about a two-third right down on
01:15:39
the value in these funds most of that being Pension funds you're not going to see governments let that happen you're
01:15:45
going to see the federal government yeah it's there's going to be some action at
01:15:49
some point and it's unlikely the office Market is going to m rebound overnight if this stays the way it is who's going
01:15:57
to fill that hole for retirees and pensioners because we're not going to let that all get written down someone is
01:16:02
going to step in and say we've got to do something about this and there's going to need to be some sort of structured
01:16:07
solution to support retirees and pensioners because that's ultimately who ends up holding the bag in this massive
01:16:13
write down he didn't go all the way there in his statements he was talking more about his estimate of 3 trillion to
01:16:17
1.8 trillion and then I tried to connect the dots and what that actually means and ultimately there's going to be some
01:16:23
pain felt by retirement funds that's going to need to be dealt with somehow so SX I don't know if that if that sits
01:16:29
right with you I mean I think the big picture is right I think you're applying a lot of averages right I think in the
01:16:35
office Market in particular the typical office deal is more like onethird equity
01:16:39
and twoth thirds debt there's just a lot more leverage right so that' be Point number one which makes the situation
01:16:45
worse even worse yeah so I would say that there's a huge amount of equity that's been written off but in addition
01:16:52
to that there's a lot of debt holders who are in trouble too and that debt is is held by Regional Banks so
01:17:01
these commercial loan portfolios are significantly impaired that's what we saw with Community Bank of New York is
01:17:07
that their stock cratered when they reported higher than expected losses in their commercial real estate
01:17:14
portfolio so freeberg I think the point is just the the pain from this is not just going to be on the Equity holders
01:17:23
but also on these Banks which can't afford to lose it's not evenly distributed yeah right yeah right and we
01:17:31
saw this in San Francisco where some of these buildings have 70% debt to equity ratios and you know the the value puts
01:17:37
them in the hole and equities wiped out completely and the debt holders have to take a hit and normally you know that
01:17:42
debt is not really written off very often it's well this is why that the debt holders the debt holders don't want
01:17:48
to foreclose they don't want to get these buildings back because when they do they're going to have to write down
01:17:52
the low loan as long as the loan is still outstanding and they haven't foreclosed they can pretend that the
01:17:58
value of the building is not impaired Kick the Can down the road is the best strategy for them so it's it's called Uh
01:18:04
pretend and extend so what they'll do is they'll work out a deal with the the landlord the equity holder that the
01:18:11
equity holder would say listen I can't pay the interest so they'll just tack on the interest basically as principle at
01:18:17
the end of the loan and they'll extend out the term of the loan which would wipe out the equity at a certain point
01:18:22
yeah well what it does it allows the equity holder to stay in control own the building right because yeah the equity
01:18:30
holder can't pay make their debt payments today but they're going to postpone those debt payments till the
01:18:36
end of the of the loan and again in the meantime just kind of hope that the market could that debt at some point
01:18:42
since they have so little equity in these buildings typically just exceed the value of the of the property and
01:18:48
it's like I'm just working for the bank now and yeah why am I even putting this working because everyone kind of hopes
01:18:54
that the market will recover the value of their Equity will go up and they'll be able to make their debt payments
01:19:00
again yeah so if you're the equity if you're the equity holder you'd rather hold on and have a chance of your Equity
01:19:05
being worth something in recovery then definitely lose the building and if you're a Regional Bank you'd rather
01:19:12
blend an extend or pretend an extend as opposed to having to realize the loss right now yep and showing the market
01:19:20
that your solvency may not be as good as you thought the same thing happened with
01:19:25
government bonds remember that with svb and these other Banks they had these huge held to maturity port bond
01:19:31
portfolios y these are main mostly just uh t- bills that were worth I don't know
01:19:37
60 cents on the dollar when interest rates spiked from 0 to 5% but they didn't have to recognize that loss as
01:19:44
long as they weren't planning to sell them right and then when they had the bank run they had to sell well yeah
01:19:51
that's right so when depositors left left because they needed their money or because there was a run or because they
01:19:57
could get higher rates in a money market fund all of a sudden these Banks had to
01:20:01
sell their heal to maturity portfolios they had to recognize that loss and that's when everyone realized oh wait a
01:20:07
second they're not actually solving okay so jamat Supply demand matters in real estate we have a Tail of Two Cities here
01:20:13
on one side in real estate for commercial real estate no demand for office space which uh is and way too
01:20:20
much Supply paradoxically on the other side we have this incredible market for developers which is gosh there's not
01:20:28
enough homes I think we need 7 million more homes and the demand is off the charts for homes yeah yeah I mean I
01:20:34
think I think you're basically right it's not I keep trying to explain residential is not a great Market either
01:20:38
because interest rates have spiked up so there's not a vacancy problem multif family developers are still able to
01:20:44
lease the units they're still a able to rent the problem is their financing costs have shot through the roof so
01:20:52
again let's say you were a developer who built multif family in the last few years you took out a construction loan
01:20:58
that construction loan might have been at 3 4% yeah now you want to put long-term financing on it but if you can
01:21:05
even find debt right now because there's a credit crunch going on you may have to
01:21:08
pay 8 nine 10% yeah but at least you can find a renter you can find a renter that's true but only at a certain price
01:21:16
and let's say you underwrote that property to I don't know like a five cap like a certain yield yeah but now your
01:21:22
fin costs are much higher than you thought you might be underwater yeah meing that situation isn't as bad as
01:21:29
what's happening in why I think it's I think it's worse in some ways if you're fully if you're fully
01:21:36
rented and your building is underwater because now your debt payments are much higher than you expected then there's no
01:21:42
business model yeah but are we seeing that are we seeing tons of multi family go under can I make two points one I
01:21:48
think I think David is is Right which is that I don't know this Market very well
01:21:52
but just just as a as a bystander here's what I observe it seems that the residential Market has a
01:21:59
feature and I don't know whether it's good or bad but that feature is that you repic to market demand every year so to
01:22:08
the extent that Supply demand is changing and default rates are up or whatever that's reflected in rents and
01:22:15
you see that because rents change very quickly and most human beings are signing six-month to oneyear leases so
01:22:21
that reset happens very quickly so it can more dynamically adapt so to the extent that a market segment is impaired
01:22:28
you see the impairment quickly on the on the office side what I see is that there's been a structural Behavior
01:22:35
change in covid that has reset in every other part of the world except for the United States where there are these
01:22:43
frankly typically younger typically more Junior employees that have held many of
01:22:49
these companies hostage in a bid to return back to office office space and so we know that there is this vacancy
01:22:55
Cliff that's going to hit commercial real estate we just don't know when because there are they're in long-term
01:23:01
leases they're canceling these leases over long periods of time so the reset cycle is longer that's just my
01:23:06
observation as an outsider I don't know what that me for for prices or anything else but it just seems that at least the
01:23:12
residential Market can find a bottoming sooner because you can reset prices every year but commercial just seems
01:23:18
like a melting ice Direction correct to you SX that assessment commercial has both a demand problem and a financing
01:23:27
problem multif family just has a financing problem but it's important office we're talking about office
01:23:32
there's retail and then there's office and then there's other industrial you see in China China has 50 million
01:23:39
homes ahead of schedule 50 million additional Supply that can house 150 million people so as acute as our issues
01:23:46
are the China issue might be much much yeah seismic can let me just give you an example on the multif family side okay
01:23:53
let's say that you buy a building okay let's say you bought a building in 2021 the absolute peak of the market and you
01:24:00
could get debt at say 4% okay and you penciled out let's call it a 6% yield that with the debt you getting so let's
01:24:09
say you did 2/3 debt at 4% you could now lever up that 6% yield to 10% okay that's like sort of the math right now
01:24:20
all of a sudden and and to get there you'd have to do some added work on the property you have to Spruce it up okay
01:24:25
now it's a few years later and your short-term financing is running out and you need to refy and you've done your
01:24:32
value added work but here's the problem the overall valuations in the market have come way down so before the bank
01:24:41
was willing to give you 2third loan to value now the values come way down you may not even be able to get two-thirds
01:24:47
loan value so you're going to have to do what's called an equity in refinancing you're going to have to produce more
01:24:53
Equity you're going have to Pony up more money so instead of taking Equity out like when the deal goes well you're
01:24:57
going have to put equity in you may not have that Equity if you're the developer
01:25:01
the other thing is that your financing cost now might be 10% so now you've got negative leverage you're generating a 6%
01:25:09
yield but you're borrowing at 10% to generate that 6% yield so the debt no longer makes sense you're again you're
01:25:16
not positively leveraged you're negatively leveraged so you're not going to want to take out that debt and if you
01:25:21
do take out that debt the the building's going to be underwater it's not going to be
01:25:25
generating net operating income it's going to be generating losses so that's why even categories like multif
01:25:34
family where you don't have a vacancy problem there's strong demand yeah those properties still don't make sense if you
01:25:41
had long-term debt on your multif family if you were able to lock in that 4% loan
01:25:46
for 10 years you're fine but for all the people who are refinancing now who are coming up this year last year next year
01:25:54
they're in deep trouble and that's why there's a rolling crisis in real estate is because the debt rolls over time it's
01:26:01
not like everybody hits the wall and has to refinance at the same time well that
01:26:05
God right I mean this would be cataclysmic if if it was if everybody can you imagine if Silicon Valley and
01:26:11
San Francisco had to say here's actually the reality anybody want to actually pay
01:26:14
for this office all in the same year right that would be insane but the crisis is growing is as the leases roll
01:26:23
and those old rents that were higher the market roll off and now you have to take
01:26:27
on new leases if you can even get them it's going to be Bal at a much lower rate and as the old loans roll that were
01:26:34
at a much lower interest rate you have to get financing even if you get it at a much higher interest rate that's when
01:26:40
all of the sudden these buildings go from being basically solvent to insolvent yeah I mean Janet yellen's
01:26:47
just going to bail these folks out I mean you won't bail out the banks themselves but you'll bail out the
01:26:50
creditors obviously the people holding the bag they'll get bailed yeah that's everybody agrees Janet yelling yelling
01:26:58
our treasury secretary I don't know if she's going to be the one to do it I there's going to be congressional action
01:27:03
on this stuff yeah I mean they tend to lead it so all right for the Sultan of science David freeberg and David saaks
01:27:13
and chamar poaa the chairman dictator I am the world's greatest moderator we'll see you next time on the Allin pod
01:27:19
bye-bye bye-bye will let your winners ride we open source it to the fans and they've just gone crazy with it love
01:27:34
queen [Music] of [Music] Besties my dog taking your driveways man oh man my habiter will
01:27:51
meet we should all just get a room and just have one big huge orgy cuz they're all
01:27:55
this useless it's like this like sexual tension but they just need to release [Music]
01:28:05
somehow we need to get merch are [Music] all I'm [Music] going