00:00:01
Welcome, welcome to Wharton Moneyball, the podcast edition here on the Wharton podcast network. This is Eric Bradlo,
00:00:08
professor of marketing, statistics, and data science here at the Wharton School.
00:00:11
As everyone who's listened to us for the last 11 plus years knows, some combination of myself, Kate Massie, Audi
00:00:17
Winer, Shane Jensen, sports never stops. Data science and statistics never stops.
00:00:23
So, we're here every week on Wharton Moneyball. I've always said that one of
00:00:28
my greatest honors of doing this show again for the last 11 plus years is having people that are living, breathing
00:00:34
analytics applied to sports all day long. I just do it as a part-time job. I got the regular job doing my own
00:00:41
research, teaching other stuff. And certainly today is no exception. Um, we're joined today by the I don't even
00:00:48
know how many times Aaron's been on the air. I'm going to say 10 plus. It's got
00:00:51
to be more than that because whatever. We're joined today by Aaron Shatz. is
00:00:55
chief analytics officer of FTN Fantasy. A lot of people also know his work on ESPN. A lot of our listeners certainly
00:01:01
know him as the founder of uh Football Outsiders. A lot of people also know him as the creator of Advanced Statistics,
00:01:08
including DVOA. And something I just learned today, and we're going to talk about this later on, is he's one of the
00:01:13
members of the Associated Press that votes for the AllP Pro team and the regular season awards. So Aaron, uh
00:01:19
welcome back to uh the Wharton Podcast Network and Wharton Moneyball. Hey, thank you for having me back. Always
00:01:24
good to be here, man. >> Well, this is obviously a a great NFL season. It's The reason I'm going to say
00:01:31
that and I'd love your a kind of analytical look at it is, you know, I can't tell you who's going to go I can't
00:01:39
even tell you in my view who's going to go to the Super Bowl. I think this is
00:01:43
just a year with more uncertainty. You know, it's like I think Shane read the
00:01:47
stat last week. This is like the first time in 20 years there's been no Patrick
00:01:52
Mahomes or Tom Brady in the Super Bowl. And so I'll just ask you, let's just
00:01:57
start at the highest level. How do you see the 2025 NFL season right now? Who are the true contenders? And if you had
00:02:05
to list, you know, I always like to give the I'll call it the paro curve. like
00:02:10
how many teams would I have would you have to give me to make it like an even bet between making the playoffs and
00:02:17
between going to the Super Bowl and not? How do you view the AFC and the NFC right now?
00:02:21
>> Well, the first thing I'll say is it's a very topsyturvy year. A lot of weird
00:02:26
stuff has happened. There are a lot of teams where their record doesn't really
00:02:30
match what the underlying statistics suggest about how good they are, right? like Carolina, New England, Chicago
00:02:38
don't seem quite as good as their records. Detroit, Kansas City, Indianapolis have played better than
00:02:44
their records. Uh, some of these are teams you would have expected in the preseason, some of them are not. Uh, I
00:02:51
used my current ratings, current ratings, and ran a simulation of the whole season. And in that simulation,
00:02:58
the amount of times that Baltimore, Detroit, Cincinnati, and Kansas City all missed the playoffs was 1.1%.
00:03:06
>> Wow. >> And that's not using preseason projections. That's using how good the
00:03:10
teams have actually been this year. >> So, we always like to ask that question.
00:03:14
So, now the question I I I'll ask you, which is probably not a surprising question, is it could mean one of a
00:03:20
couple things. It could mean the thing that we all have in our simulations, which is not enough uncertainty. And if
00:03:26
the answer is there isn't enough, where is it coming from? Another possibility
00:03:30
is um 1.1% of events happen 1.1% of the time. So, you know, let's not let's not
00:03:36
go overboard that you said 1 in 20 million. So, you didn't say that. The third, of course, is your point
00:03:42
estimates of the team's strengths could also be off. It's not an uncertainty
00:03:46
issue. It's kind of >> It's an imperfect science. Yeah. >> Yeah. It's an imperfect science. So
00:03:50
which of those three explanations or all a combination of all three do you think
00:03:54
it is? >> I think it's a combination of all three. But the big thing is just that events
00:04:00
get ordered in different ways. And so teams play one game where they're like
00:04:06
outstanding the whole game and then another game where they lose by three points. And this year is filled with
00:04:11
teams where they've just been either really really good in one-score games or
00:04:18
really really bad in one-score games. The other thing that's going on is the
00:04:21
is we'll talk about this in a little bit. I think schedules being very extreme
00:04:27
because of the way certain divisions are good and bad this year. There are certain teams that have very extremely
00:04:34
easy or extremely hard schedules. Um, as far as uh the NFC is the better conference, that's the first thing. Uh,
00:04:42
I my numbers differ from market. Is the NFC the better conference on let's say
00:04:49
whether it's average or median or do you think the top end because those are
00:04:53
different statements than saying the top end of the NFC is better than the top end of the AFC or both.
00:04:59
I would say both are true, but in particular the top end of the NFC. Um, the market I have Seattle and the
00:05:10
Rams much farther ahead of the rest of the league than a lot of other people who do advanced analytics.
00:05:17
Uh, the reason for that is almost everybody else who does advanced analytics uses some form of EPA, right?
00:05:25
Expected points added with some sort of adjustment. And I don't I use my own stat which is
00:05:32
DVOA. And because of some of the ways that DVOA works and EPA works, Seattle and
00:05:39
the Rams come out a lot higher in DVOA than they do in EPA. One, some of it has to do the increasy intricacies of, you
00:05:47
know, certain amount of yards and a certain down and distance. And some of it is easy to explain stuff like I
00:05:54
downweight turnovers because they're less predictive. And Seattle has a lot of turnovers. So if you do EPA,
00:06:02
Seattle's offense is not going to come out as good as they do in DVOA because
00:06:07
I'm downweing the turnovers because less predictive. So because of that, I have
00:06:13
Seattle and the Rams at almost a 5050 shot to win the Super. >> I just want to be clear them com just to
00:06:22
be clear. Are you saying if you take the Rams and Seattle, you have the Rams in Seattle, I have the rest of the league
00:06:29
and that's a fair bet. Is that what you're saying? >> My numbers would say yes. It's a surp
00:06:34
It's surprising and I don't know whether maybe I have them a little too high and
00:06:37
no one else would have this, but yeah. I mean, everybody has right now has Seattle and the Rams as their top two.
00:06:43
>> So, just give me an or this is what I was going to ask you. whether one's EPA
00:06:46
based or DVO based. Can you give me a sense? Or even if I went on to ESPN, FBI, whatever it was, how much
00:06:53
probability would they give combined? Are you at 50 and they're at 20 or you're at 50 and they're at like 35? I I
00:06:59
always like to get a sense of the order of magnitude of these discrepies differences. I can look up ESPN while
00:07:06
we're here and tell you that for Seattle and the Rams, they have them winning the
00:07:13
Super Bowl a combined 27% of the time. >> Wow. So, you're almost twice as high.
00:07:18
So, that's we would consider >> past everybody when it comes to those two teams. Yeah.
00:07:25
>> Yeah. So, I just just for myself maybe I'm thinking about this wrong. I'm
00:07:29
pretty sure most of our listeners and this is the great thing about Wharton Moneyball is saying, "Wow, would Aaron
00:07:35
really give me even money despite like someone we both agree someone from the AFC is making the Super Bowl. I don't
00:07:42
know who it is, but someone is." And we could argue they have almost a 50% chance to win that game. So, in your
00:07:50
view, well, let me ask a question. So, there's this is the wonderful thing about winning the Super Bowl.
00:07:56
We agree two things have to happen for you to win the Super Bowl. If you're Rams or Seattle, first you have to win
00:08:01
the NFC. Then you have to go to the Super Bowl and win it. Which of those two numbers is really, really high? It
00:08:10
sounds like it might be both. Like here's something that could yield 50%. >> The combined probability is 7070. That
00:08:17
would get you to 50%. Like they're the one of the two of them has a 70% chance
00:08:22
of going in total. And if they go, they're a twothirds favorite. Roughly that get you to 50.
00:08:28
>> So my ratings are based on the idea of, you know, how efficient is a team
00:08:31
compared to average. >> Seattle and the Rams are two of the 10 best teams since 1978.
00:08:41
>> Measure. I know I have them way ahead of everybody else. I know exactly how
00:08:46
you're measuring that. It's based on a play-by-play breakdown and the success
00:08:51
on each play is compared to a baseline that's adjusted for situation and opponent.
00:08:57
And one thing is that I include special teams and a lot of people who do advanced stats don't include special
00:09:03
teams because special teams is less predictive than offense and defense. But it's not not predictive and I think
00:09:10
Seattle and Rams game that the fact that Seattle has really good special teams. So just to be clear, I compute this
00:09:17
>> I compute this metric for every play and then I sum it over all the plays.
00:09:21
>> Yeah. Yes. >> Okay. So Okay, I see. I just wanted to understand the Okay. Wow.
00:09:27
>> So just to give you an idea of where the numbers are, Seattle and the Rams are at
00:09:31
like 42 43%. The next highest team that will make the playoffs is like Jacksonville right now, which is
00:09:42
at like 20%. Like that's the gap I have between the Rams and Seattle and everybody else. I
00:09:49
don't know if I would really do the Rams and Seattle versus the Field to win the
00:09:52
whole thing. >> If you gave me the 49ers though, I might do it. If you gave me the NFC West
00:09:59
versus everyone else. >> You almost made the whole MS NFC West. >> I might I might do that.
00:10:05
>> Huh. That's very very very interesting. Um, what aspect? Well, first of all,
00:10:11
could you just tell us since you must since you're making a statement that they're in the top 10 since 1978. Can
00:10:16
you give maybe I I don't know if you have this handy, can you tell me since I'm a football historian just like you
00:10:22
are, can you tell me who else is in that list or do you have that list handy? >> The top three through this is through 15
00:10:29
games. The top three are the 2007 Patriots. >> They were pretty damn good. The 1991
00:10:36
Redskins. >> They were good. >> The 1985 Bear. >> I knew the 85 Bears were going to be in
00:10:42
there somewhere. Wow. Yeah. So, >> you also get some teams that didn't make
00:10:47
it all the way like the 2010 Patriots and the 1983 Redskins who lost to the Raiders in the Super Bowl and the
00:10:56
9549ers who got upset by the Panthers in the divisional round. And then for some
00:11:03
reason the last three years I have had a ton of teams come out as historically good in a way that hadn't happened for a
00:11:12
dozen years before them. The 2023 Ravens, the 2023 49ers, the 2024 Ravens, like come out as really historically
00:11:23
good. And I don't know why because I haven't changed what I'm doing. And yet
00:11:29
there weren't any teams that were anywhere near that high from 2011 to 2022 and then all of a sudden in the
00:11:35
last couple of years there have been these teams that are ridiculously high. H wow. Um so because if you had asked
00:11:43
most people I think I they would say that the Rams and the Seahawks are better appear to be better than others
00:11:48
this year but not I'll call them. Well, I don't think anybody would put them
00:11:51
among the historically great teams. >> And it's it's it's a little weird
00:11:56
because of the fact that I've had this going for a couple of years and those
00:12:00
teams have not won the Super Bowl. But it's not like my ratings always come out
00:12:06
with modern teams that high because from 2011 to 2022, there were no teams that high.
00:12:12
>> It's just the last three years. >> So, I'm going to ask you a question that
00:12:16
every listener here on Wharton Moneyball. And again, I'm I'm joined here by Aaron Shats. Aaron is the chief
00:12:20
analytics officer at FTN Fantasy. Everybody also knows him from his original work at Football Outsiders, his
00:12:27
work at DBA, his work with DVOA. So, let me just ask you, I'm always going to ask these kind
00:12:32
of questions because I know the answer is no to each of them, but what the hell? I'm going to answer them anyway.
00:12:37
Rams and Seahawks. Um, if I'm correct, the quarterback for the Seahawks is Sam Darnold, right?
00:12:44
>> Yes. So you have no concerns? >> No, I have concerns. >> Okay. So, so let me ask you a different
00:12:53
question. Is that brought mathematically into your model anywhere that it's Sam
00:12:59
Darnold's? >> No. And it's something I want to look at doing in the future, which is should I
00:13:04
have an additional career history uh variable for quarterback that goes past this year all the way back multiple
00:13:15
years of whether you trust a certain quarterback compared to other quarter. It's clear that Vegas does.
00:13:23
>> Absolutely. >> Because there's no other reason why they have the Bills as the favorite to come
00:13:27
out of the AFC. The Bills have not been by any metric the best team in the AFC this year. The only reason to have them
00:13:35
as the favorite to come out of the is Josh Allen. Is the idea that you have more historical trust in Josh Allen than
00:13:43
you do in Drake May or Bo Nicks or Trevor Lauren? >> So, which team in the AFC has been the
00:13:51
best team based on your Jacksonville? Is it Jacksonville? Uh, if you look over the course of the entire season, it's
00:13:57
Indianapolis. If you weight it towards more recent games, it's Jackson. Where do you have I have debate with my
00:14:05
kids about this all the time who are big analytics guys as well. Isn't I mean, they they believe
00:14:12
Houston's a fraud. Where do you guys have Where do you have Houston who I think's won at least eight straight if
00:14:18
I've got this right? >> We have them uh fifth. >> We have fifth in the NFL. a fifth in the
00:14:24
NFL. They are number two on defense, but they're only 21st on offense. >> And there is that thing where offense is
00:14:31
a little bit more predictive than defense. And that is that is true, right? So, you're like, okay, maybe
00:14:38
they're not quite that good going forward, but that defense is really, really good and the offense has played
00:14:43
better in recent week. >> Um, and okay, very interesting. So, let me ask you another question. So, you've
00:14:50
met you made a comment in the notes here about the Patriots and their schedule. Um, could you talk more not only about
00:14:57
their schedule, but maybe this is why one of the reasons why we have advanced analytics because you can't just look at
00:15:04
one loss record and so could you talk a little bit more about which teams you see where and you mentioned it a little
00:15:10
bit when you're in your opening remarks about who's one lost record or how about
00:15:14
the following. Let's say I did a by a graph of w number of wins on the x- axis
00:15:20
and whether it's Aaron Shatz's measure of team strength on the y ais. Which
00:15:25
teams would have the largest deviation from that 45 degree line? >> And a lot of these teams, anybody doing
00:15:32
advanced analytics has the dev deviation? the Panthers, the Bears, the Patriots, and on the
00:15:41
other side, the Lions, the Chiefs, and the Colts. Although obviously the Colts and Chiefs have a different quarterback
00:15:49
now than they did for most of the season, and that affects how you would predict things going forward. But as far
00:15:55
as how they've actually played during the year, the Colts and the Chiefs have
00:15:58
been better than their record. And the Lions have The Lions are going to be one of the top DVOA teams ever to miss the
00:16:04
playoff. >> Wow. So, I was going to ask you that also. Do you ever as a I'll call it a
00:16:10
face validity check. Let's imagine that every year DVOA based strength um has
00:16:18
lots and lots of outliers or deviations where the DVA DVOA based strength does not match let's call it the win loss. Do
00:16:26
you ever start to say maybe that's a deficiency in DBOA because there's of
00:16:31
course there's randomness within a given year but like across a long period of
00:16:35
time eventually you know if I look in totality it should be properly calibrated.
00:16:40
>> Yeah. I mean there's a lot of variance in sports and there's a lot of variance
00:16:45
in the NFL. It's never going to be properly calibrated but no I'm always
00:16:48
looking to improve it. I'm always looking to improve the predictive ability. I'm always looking to improve,
00:16:53
but a lot of these uh it's interesting for a lot of the teams that I just described,
00:16:58
the year-to-year DVOA is more consistent than their year-to-year win loss records.
00:17:03
>> Well, that's and that's what you would hope. >> Like Detroit, Indianapolis is an
00:17:06
exception, but Detroit was good last year. Kansas City was good last year. Carolina was not good last year. So, the
00:17:14
idea that Carolina is not as good as its record makes some sense if you think about who they were last year. And the
00:17:20
idea that the Chiefs are better than their record makes sense if you think about who they were last year,
00:17:25
>> right? Is it Am I correct in saying this is just from my I haven't looked at it
00:17:29
analytically, but I would just give an opinion that Well, I think I know, maybe I don't know. I was going to say is
00:17:35
Well, it must be from your answer. The strongest conference division, sorry, in football has to be the NFC West, right?
00:17:41
Given you just said the Rams and Seattle are historically strong. You said you would take the 49ers. I think each of
00:17:46
them has at least 11 wins at this point. Um, and so that's the strongest and the
00:17:52
NFC South is the weakest. >> Yeah. Oh, yeah. The AFC South was shockingly strong this year. The AFC
00:17:58
South, >> that was going to be my second guess. >> Really surprised. Indie Jacksonville
00:18:04
really surprised. Um, Houston, I think people thought was going to be, you know, just what they are, which is a
00:18:10
great defense with no offensive line. Um, but uh right now in my ratings for the whole season, so not weighted to
00:18:17
recent game. Uh, the Colts, the Texans, and the Jaguars go four, five, six. But with the NFC West, those two teams are
00:18:26
so far ahead and the 49ers are also good that Yeah. the NFC West is the best division. And the whole NFC South is in
00:18:32
the bottom half. >> Wow. Yeah, because I you may remember I'm a Buccaneers fan through my cousin,
00:18:39
etc. Uh what's happened the last three weeks to the Buccaneers has been embarrassing. Like I mean
00:18:45
>> they're the best team in the division. Why can't they win games? It's very
00:18:48
frustrating. >> Yeah. I mean they are the best team. That's clear. And now they have all
00:18:52
their players back. So there's no excuse for that. And they've just they lost at
00:18:57
home to the let me think if I got this right. I was at the game with the Saints so I know they lost that one and then
00:19:02
they lost to the Falcons at home and then they just lost to the Panthers on the road. So they just lost to, you
00:19:09
know, three teams in their division and they all stink. >> Yeah, it's I can't imagine. It's very
00:19:16
frustrating. Yes. >> And of course they still control their own destiny which is also the crazy
00:19:21
part. I mean they may end up going eight and nine and winning the division because the accord to you that's likely
00:19:27
to happen because uh the Panthers play Seattle this week and that's a big game
00:19:32
for Seattle. The Bucks play the Dolphins. Well, the the Bucks probably >> starting a rookie quarterback.
00:19:40
They're gonna be >> they should probably beat him. The Bucks would have to win the division at nine
00:19:45
and eight because somebody's going to have to win the last game of the season
00:19:48
between those two teams. But that may be a historically weak division winner. And
00:19:53
whoever the five seed is um might be happy they're the five seed. >> I mean, there are other divisions, the
00:19:59
2020 NFC East, the 2004 NFC West. There have been other divisions that are weaker than this, but this is pretty
00:20:07
weak. >> Now, the next thing in the notes that you helped us uh create here, which I I
00:20:12
really appreciate that, is um I assume when I asked you the question about DVOA and your it being based on playbyplay
00:20:19
um and then you kind of add it up. Um you put in the notes here the s the search for historical playby-play data
00:20:25
and the upcoming edition edition of the 1977 season. So, is it fair for me to say that what you and I completely agree
00:20:33
with this by the way that you've built most of your analytical career for the
00:20:37
NFL on having this playby-play data as opposed to just aggregated statistics and that's why
00:20:45
>> Yeah, please go ahead. >> When I got started, there were people who were like, "Yeah, we did analysis
00:20:48
based on playbyplay and there was nothing based on playbyplay in it." And I was like, "Let's do something based on
00:20:54
playbyplay." And then I started collecting old years and a lot of it's been with the help of Jeremy Snyder who
00:21:00
people know on social media as quirky research. >> Okay. >> And he has helped me and we have tracked
00:21:06
down and we keep going back and back and back. So we've got back to 1978 now. We
00:21:12
have transcribed things off of video. We have found old game books. Um and we have enough to do 1977 this off season.
00:21:22
And it is no longer just mine. I have shared it with Pro Football Reference. Pro Football Reference now has all the
00:21:30
playbyplay going back to 1978 on their site, searchable with all the playby-play logs for all the games. I'm
00:21:37
very proud of doing that. It's been I think it's a really huge addition to the
00:21:44
ability to study historic football historically. And it's great. It's so
00:21:48
much fun for me when I'm like, "Oh, yeah. The best teams to miss the playoffs, the 1979 Redskins." Like, I
00:21:56
can go back that far and say they're one of the best teams that didn't make the
00:22:00
playoffs. You know, let me ask you a question. This might seem naive, but I'm
00:22:05
sure a lot of our listeners here on Morton Moneyball are asking might be thinking the same. What do you exactly
00:22:11
mean when you say playbyplay data? Because one could dream. Let's let's forget what you have. I'm going to
00:22:16
answer what my dream might be and then you're going to tell me the data that
00:22:20
one actually would have conceptually. You could have video motion data of every player and their XY coordinates on
00:22:31
every single play and all of and I know that's not the data you have but so what
00:22:36
do you mean by playbyplay data and how do you compute kind of like an expected performance
00:22:43
given that data >> for a given play? >> All my years are normalized so that
00:22:49
average is zero. So the same goes all the way back to 78. Every year is normalized so that I'm comparing teams
00:22:56
to just that year. And what I have is basically everything that was in the playbyplay up to 2004.
00:23:06
So like is it a pass or a run? Who was the player? Who was the receiver? What was the down
00:23:15
and distance? What was uh the time is not in there for a lot of old years until the last two minutes of each half,
00:23:22
but we have the time to start each drive. And then the further back you go, the more spotty it gets as far as like
00:23:28
the direction of runs or tackles or passes defense, but we have like who had sacks and who has interceptions and all
00:23:36
that data. Uh but we have targets, right? Like no one ever had targets for the 80s before before we collected that.
00:23:44
So, not just receptions, but targets like is in this data. Um, we don't have
00:23:50
um yards after the catch and air yards. That doesn't start till 2005. >> We don't officially have whether a
00:23:58
quarterback run is a scramble, but I've gone back to a lot of video and answered
00:24:03
whether a lot of those are. Uh, we don't have a quarterback knockdowns. That
00:24:08
didn't start till 2006, I don't think. And we don't have all the kind of
00:24:13
charting data that you now get from FTN or SIS or PFF. Not we don't have any of
00:24:20
that going back before 2005. But everything that people had from 99 to 2004, we basically now have for 78 to
00:24:28
98. >> H I I can imagine that's ext and then can you give us a sense of how just you
00:24:35
know maybe give me an example of how you score a play. So, how is a play score? >> Well, like you know, I give it a certain
00:24:43
number of success points based on the yards and the down and distance, right? Success in my percentages is you have to
00:24:49
get 45% of needed yards on first down, 60% on second down, and 100% on third or fourth down, but there's like partial
00:24:58
numbers. It's it's not it's not it doesn't go zero and one. It goes like
00:25:03
zero and 2 and 0.5 and 7, etc. And then you get extra bonus for the extra yardage. So a play will come out with
00:25:11
something like 2.3 success points if it's like a 10 yard pass on first and 10
00:25:16
or something. And then I compare that to what's the average for first and 10
00:25:21
passes in that area of the field. >> Yeah. Okay. So that's what I meant. I
00:25:24
was also asking what do you condition on when you compare a play? So it would be
00:25:29
similar >> distance area of the field uh and then adjust for the score, right? You know,
00:25:34
obviously it's easier to gain yardage if you're losing by 40 late in the game.
00:25:40
>> So, this is actually a pretty opponent adjustment based on the quality of the
00:25:44
opponent. So if I just look at this as a big contingency table, this is a very highdimensional contingency table
00:25:52
because I mean there's not that many downs so that's not high dimension but
00:25:56
distance maybe have them uh blocked or you know bundled into ranges or something and then there's maybe there's
00:26:03
score difference and then there's maybe >> what I try to do is smooth the curves.
00:26:08
>> Okay. So there is a smoothing that's done but I didn't have like I I don't
00:26:12
think it makes sense to say that eight yard gains are more common than seven yard gains. It's more likely that our
00:26:20
sample size isn't big enough. It what what's logical is to say that you know
00:26:26
you gain one more than you gain two you gain two more than you gain three you gain three you know etc. like it's it or
00:26:32
I guess four is the average and then the curve goes like this, right? But like um
00:26:37
you know for judging between first and 20 and first and 25, you sort of group that together because there aren't
00:26:44
enough actual like >> first and 21s that are out. You know, there aren't that many first and sevens
00:26:50
that aren't in the red zone. >> Makes a lot of sense. So let me get to
00:26:54
the last topic for today that I want to talk to you about which is as I mentioned in when I introduced you to
00:26:59
start with you're one of the I I imagine a tremendous honor for you and one welldeserved as a member of the
00:27:06
Associated Press that votes uh for both the regular season award and the allp pro team. Um one of the topics you
00:27:12
brought up in your notes was the growing role of analytics in awards voting. So that could mean many things. It could
00:27:18
mean um the number of people that you and I out of those 50 would consider as analytics oriented is going up. That's
00:27:24
one way to measure it. Another way to measure it is um you have knowledge of the analytics that people are using.
00:27:32
Like here are the stats that people use or those are provided in some way. What did you mean when you said the growing
00:27:38
role of analytics in awards voting? >> More the first than the second, but
00:27:42
they're definitely both true. I was the first analytics person to be on the panel.
00:27:48
uh starting in 2021, but they've done a lot of changing of the panel over the
00:27:52
last few years. So now you have Mina Kines >> and you have Sam Monson who used to be
00:27:58
with PFF and you have Doug Ferrar who's more of a film guy but understands analytics used to write for me at
00:28:04
Football Outsider. The other thing though is if you think about it, people who are not necessarily
00:28:10
analytics people have access to more analytics now than they used to. Think about a guy like Dan Orlovski,
00:28:18
right? Dan is a former quarterback. He is a film guy, but he works for ESPN and he's very conversant with all of the
00:28:26
ESPN advanced stats that ESPN stats and info does. And I'm going to guess that
00:28:32
when Dan puts his ballot together, he's going to use some of that stuff. Now,
00:28:36
he's more probably going to use his eyes and his scouting, but he's going to use
00:28:40
some of that stuff that didn't exist a few years ago, or at least there was nobody on the panel that would use any
00:28:46
of that a few years ago. >> Right. >> So, I think even the people who are not
00:28:49
necessarily analytics people are going to use more analytics than they would have used 10 years ago.
00:28:55
>> Can you point I don't want to talk about this year because I know that's not only
00:28:58
is it off limits, you shouldn't talk about it because you're a voter, but we
00:29:01
can talk about the past. So can you talk about instances at least in your own mind that somebody was like wow if it
00:29:10
wasn't for analytics that person never would have been voted all pro or never
00:29:14
would have been an or maybe that's off limits too just like I was just not that
00:29:17
your vote I'm not asking for your vote or anything like that >> I'm thinking because I have an answer
00:29:21
when it comes to my vote but I'm trying I don't know about for the whole allpro
00:29:29
team or the whole or specific awards that I think of anyone like that, but I can think of one where I was very
00:29:35
different. I'm very famous among, you know, people who understand this stuff.
00:29:41
When Lamar Jackson won MVP in 2023, he got 49 of the 50 votes and I was the exception.
00:29:50
>> So, can you tell Well, now that you said that, can you tell us about that and
00:29:53
what what did you vote? And >> I wrote a whole thing about how no matter what stat you looked at, he came
00:30:00
in below Josh Allen and Dak Prescott. >> My advanced stats, ESPN's advanced
00:30:07
stats, PFF grades, like SIS, anything you looked at, he came in below those guys. And I was like, I just can't vote
00:30:15
for him for MVP when he's not number one and or two or two in any staff. And so I
00:30:22
went with Josh Allen. Wow. Well, let me just say uh for those people that follow us on whether you
00:30:28
want to call it X or Twitter, W Moneyball, um there's no doubt I'm going
00:30:32
to be if we still call it tweeting, I'm going to be tweeting that out because
00:30:36
that in itself is fascinating. It's also fascinating to me that since let's
00:30:41
assume whether people use it or not as a separate issue, that had to have been sort of known by the voters. And so that
00:30:48
that to me is very interesting. Let me ask I'm just asking you for speculation.
00:30:53
If that were to happen today, 5 years from today, you agree the vote you're I
00:30:59
mean we would both hope the vote wouldn't be 49 to1. >> Today it's still there's still
00:31:04
narrative, right? Cuz Allen beat Jackson last year even though last year the opposite was true. Jackson had better
00:31:10
numbers. >> Interesting. But in five years, I don't know if somebody whose numbers are as
00:31:17
low as what Cam Newton's were in 2015 is going to be able to win the MVP. I don't
00:31:23
know. I mean, you you'll have you'll certainly have places where the person
00:31:27
who finishes third or fourth will beat the person who finishes first in a stat, especially because different metrics
00:31:33
have different players number one. But five or six years from now, you may not have a situation where you have a guy
00:31:40
who's 10th or 11th in stats, but he gets a narrative going, >> right? >> And his team wins a lot of games because
00:31:48
of their defense and he gets a narrative going and then he wins the award. You may not have that anymore in a few
00:31:53
years. Well, let me just my last question now is um can you tell me a little bit about
00:31:58
what you're what you guys are working on that you can talk I'm not necessarily
00:32:01
the nuts and bolts but what are you guys doing at FTN Fantasy that our listeners
00:32:05
might be interested in like or you even personally like what's the big project
00:32:08
you're working on now >> I mean big projectwise I don't have anything in particular except you know
00:32:14
still always looking to improve things in the offseason you know improve predictive ability um you know the thing
00:32:21
I'm really proud of at FTN is because we do all of charting in our FTN stats hub.
00:32:26
If you're a subscriber to the site, ftnfantasy.com, you get DVOA for all kinds of charting
00:32:33
uh categories. So, you can look at who's the best against man coverage or zone
00:32:38
coverage or with two high safeties and which teams are the best with certain run concepts, outside zone, inside zone,
00:32:44
power, man, etc. I love that stuff and I love doing stuff to figure out what's
00:32:50
real and what is just really variable. And then working on the 1977 and then the other thing that FTN is
00:32:59
doing that I have nothing to do with but we are doing and is cool is basketball charting. and we're going to have a
00:33:06
bunch of basketball stuff coming out uh for people who enjoy basketball analy well Aaron I'd like to thank you for
00:33:14
joining me here today on the podcast edition of Wharton Moneyball I've been talking to Aaron Shatz chief analytics
00:33:19
officer FTN Fantasy uh apparently it's ftnfantasy.com right is the uh is the website you can
00:33:26
also read >> called FTN Fantasy we do a lot more than just fantasy it's just we had to get it
00:33:31
all under one ceiling and that's the ceiling we went Well, again, Aaron, thank you again for
00:33:37
joining me today on Wharton Moneyball. >> Hey, thanks for having me, man. >> Welcome back to Wharton Moneyball here
00:33:43
on the Wharton podcast network. This is Eric Bradlo, professor of marketing, statistics, and data science here at the
00:33:48
Wharton School. Uh, I just got done and finished talking to Aaron Shatz of FTN Fantasy, chief analytics officer. Had a
00:33:56
great discussion of lots of different topics. Um, and uh, some combination of myself, Audi Winer, Shane Jensen, and
00:34:03
Cade Massie are here every week on Wharton Moneyball. Uh, since it's just me, um, and there's the a quote second
00:34:09
half of the show, I did something interesting this week, which I hope all of our listeners uh, will want to will
00:34:15
be interested in. I decided that since I'm Wharton's vice dean of AI and
00:34:20
analytics, uh, why not use Generative AI to answer a bunch of questions that I have uh, I'd like to hear the answer to.
00:34:28
So, I'm going to tell you what I did. I'm going to tell you the the exact
00:34:31
prompt I typed in, and I'm going to tell you how, in my case, chat GPT. You could
00:34:35
use your own large language model, but Wharton gives us a free enterprise access to chat GPT. Um, I'll tell you
00:34:42
exactly what I said, what I asked it, what the prompt was, and what it found. And by the way, just for all of you that
00:34:47
want to know, this was chat GPT 5.2, which is the newest version. And I also used it in thinking mode trying to get
00:34:54
the if you'd like the highest quality answer for anybody that wants to reproduce what I did. So the first
00:35:00
question I asked it was give me a 95% confidence interval for the number of wins for the Oklahoma City Thunder in
00:35:09
the regular season. So let me say a couple things. I'm interested in that because they're 26 and three. They
00:35:16
certainly have the potential to break the all-time record of 73-9 by the I think it was 2017 Golden State
00:35:23
Warriors who lost the title as everyone remembers to LeBron in the finals. Um but I'm also interested into how much
00:35:29
uncertainty it gives it. Okay. And so the first thing is what this is first something that differs from older
00:35:37
versions of large language models. You may remember like when they first came out like we've only trained the date up
00:35:42
until January the 1st of two years ago. Well, that's all gone now. Now, I think
00:35:46
everybody knows that the first thing it said is as of December 23rd, their record is 26 and three. And they
00:35:51
literally played last night. So, they know that they played last night. That's
00:35:55
in the database. So, that's good. Um, it said it leaves 53 games in the regular
00:36:00
season. Sounds good. Um, then it did something more spe more specific and impressive and it started thinking and
00:36:08
it would come up on the screen what it was thinking about. It says, "Look, we're going to fit a beta binomial
00:36:14
predictive model." So, let me just be clear to everybody what that means. So,
00:36:19
you can treat the 29 games they're played as a binomial, a coin flipping model. 26 have come up head success.
00:36:27
Three have come up failed. That means they've lost. That's the binomial part.
00:36:31
And then we're going to put a prior distribution on that coin flip. And usually most people put a beta
00:36:38
distribution. You can view that as an equivalent number of wins and losses. So for example, if I said it's a beta coin
00:36:46
with a 1010, that would say my prior is 10 wins and 10 losses. I would add that to the 26 and three. I'd have 36 wins
00:36:54
and 13 losses and that would be my predicted win probability for the rest of the season for the 53 games. So first
00:37:02
I was impressed that it knew about basian inference that well. It knew that doing some sort of shrinkage, which a
00:37:09
beta binomial is going to do because no one's going to predict that they're
00:37:13
going to go, you know, 950 or whatever 26 and 3 is for the rest of the season. You're going to have to shrink it down
00:37:19
to some prior and a beta binomial gives you a formal way to do it because it gives you an effective sample size. We
00:37:26
know the observed sample size 29, but the prior is going to give us an effective sample size. It then said the
00:37:33
95% interval is from 64 wins to 78 wins. Now, first thing that surprised me was the 78 number. I mean, that number I
00:37:43
mean, they're not winning 78 games. They're not going 52 and one the rest of
00:37:48
the season, but I will say it also computed this simulation game by game, so it knew the schedule. 64, I think
00:37:56
that's just way too low. I mean, you say, well, what about injuries? Well, the first like month of the season they
00:38:01
played without their second best score, Jaden Jackson. So, they're not I mean,
00:38:05
maybe um I just thought it was an interesting interval. I guess it's centered around 71 or 70 or 71 which
00:38:13
seems quite plausible to me that they might do but either way. Um and then lastly it said for a reference a more
00:38:20
optimistic assuming their current win percentage continues exactly it leads to an interval of 69 to 77 wins which means
00:38:28
centered at 73 plus or minus 4. So you could view it as a standard deviation of two. You're going to take two standard
00:38:35
errors of either side. So you're 73 plus or minus 2 * 2 which gives you an interval from 69 to 77. Either way I was
00:38:44
impressed by its thinking. I was impressed by its use of the beta binomial. I could have queried it more
00:38:50
to use this prior. It would have done that calculation. It was simulating game by game. Not bad. Not bad. So the next
00:38:59
question, since we're living in the world of college football, I asked it, what is the probability that the four
00:39:06
semifinal college football teams are Indiana, OSU, Georgia, and Oregon. So that's the 1,
00:39:15
two, three, and the five seed. The reason I picked the five seed Oregon is that they're favored over Texas Tech.
00:39:20
And I said, give me an exact probability estimate and a 95% confidence interval.
00:39:25
So, first thing it said is um it recognized the matchup, so it told me the matchups. That's not that hard. Um
00:39:31
it then used the draft kings money line. It could have used lots of different money lines, but that's fine. It then
00:39:38
gave me the probability of each of the teams that I won winning. It then said, let's assume they're independent. It
00:39:45
multiplied them together, and it comes up with 17.94%. It then computed a 95% confidence
00:39:52
interval based on those proportions. Um, it then gave me a range of 16.72 to 19.68%.
00:40:01
U, that seems a little bit narrow to me. Um, and then it said here, if you want me to compute spreadbased probability
00:40:07
instead of money lines, it could do that type of computation. So again, this one was interesting because it rep
00:40:15
recognized that I was looking for a compound event. you know the probability of four things happening because I asked
00:40:20
for exactly all four of them. It used the it found the money line to compute those probabilities. It multiplied them
00:40:28
together um and it gave me a confidence interval based on these uh proportions and and what the likely uncertainty is
00:40:36
in those proportions. So I thought that was pretty impressive. And again it's
00:40:41
more to me that it's impressive compared to historically. If I had asked it six
00:40:46
months ago, a year ago, I don't think it would have given me as precise an answer. Um, and last but not least, I I
00:40:53
asked it um let's see here. I had one other thing that I asked it for. Ah, okay. Um, I asked it the question I
00:41:01
always like to ask, and this time I did it for the NFL. I asked it, how many teams do I need and what would those
00:41:10
teams be that would give a 50% probability to win the Super Bowl? In other words,
00:41:19
if I thought there were four teams that added up to 50%, I I'm asking it to tell
00:41:23
me the number four and who those teams are. Now, what's fascinating about this
00:41:26
is I didn't I hadn't even thought about this at the time. I just asked Aaron
00:41:30
Shatz that question if you remember during the first half of the show and he said in his mind it's almost two teams
00:41:37
the Rams and the Seahawks almost add up to 50% under his DVOA model. Um that's
00:41:43
not what uh but he also said under using betting lines they would be about 27%. And that's what um chat GPT 5.2 thinking
00:41:52
found. It said first it's converting bet 365 odds to an implied probability and
00:41:58
it gave me the formula of the implied probability which is the way we always do it. P equals 100 over the plus number
00:42:06
plus 100. So you take if it's plus 300 100 over 300 plus 100 is the implied probability. It normalizes across all 32
00:42:15
teams to take care of the vig the betting vig that's in there. And then what it says, it gave me the top six
00:42:22
teams that give 49.65%. Now to Aaron's credit, not surprisingly, the Rams and the Seahawks are at the
00:42:30
top. Although according to this, that would only be about 22% probability. It then gave the Chiefs, who aren't even
00:42:36
going to make the playoffs, the Bills, the Patriots, and the Broncos. And those six teams, it adds up to 50%. Now,
00:42:46
here's the question. Certainly, I think Aaron would take the No, of course, the
00:42:49
Chiefs aren't making the playoffs. The Bills, the Patriots, and Broncos are.
00:42:53
Um, and by the way, the seven team for us Eagles fans are the Eagles here. Um, but I thought this was fascinating.
00:43:00
Again, I just want to say for our listeners out there, use generative AI. I I want to say I get I this is not a
00:43:07
specially trained generative AI model. It's the one that if you have chat GPT
00:43:12
5.2 2 and possibly in thinking mode. Um, you have the same ability I do to type in what is the number of teams X that
00:43:21
are needed to get to 50% win probability, order them by the highest probability teams and tell me who the
00:43:28
teams are. That's all I typed in and this analysis came in and it said here and again it said closest to 50% using
00:43:36
the strict order of the teams is the number six. That's pretty impressive and
00:43:41
this is a statistic we've been talking about to understand uncertainty in the
00:43:45
league really for the 11 and a half years we've been here on Wharton Moneyball. So those are the three things
00:43:51
I did on chat GPT 5.2. I'll try to do some more before next week's show and
00:43:56
also post some on our Twitter account Moneyball. Um, just the last thing I wanted to talk about maybe in the um, in
00:44:04
the last couple minutes I'm going to talk today is um, since everybody knows
00:44:08
I'm a big tennis fan and we're trying to get very soon we're going to get Paul
00:44:11
Anakone on and he would be great to talk to about this. Um, as many people saw in
00:44:16
the last week Carlos Alcarez, the number one player in the world, the holder of six majors um, fired his coach. It'll be
00:44:23
very interesting to see what impact that has on his performance during the year because one of the things we've always
00:44:29
asked about is the impact uh of coaching and so you know he's keeping a lot of
00:44:34
his team but his main coach Juan Carlos Ferrero uh former number one in the world uh winner of I think the French um
00:44:42
is no longer his coach. It'll very interesting to see if there's any drop
00:44:45
off or if there's any narrative. So well remember he also has to beat other players and he has to beat especially
00:44:51
Yanx S. So you know can we decompose whether maybe S's just getting better or
00:44:57
is it that Alcarez got worse? I think this is the kind of thing that we're going to need advanced analytics for
00:45:02
because I'm not sure like he's never won the Australian Open. So we said, well,
00:45:06
let's say he loses. We can't just say, oh well, if he had his other coach. No.
00:45:10
So we're going to actually have to dive into the advanced analytics and understand, you know, is he actually
00:45:17
performing worse? And then of course there's also MA is match to match and week-toeek variation. So we're going to
00:45:23
have to disentangle that from it. It will not be an easy analysis. We're certainly going to have to watch it for
00:45:29
some period of time and maybe use last year as a if you like a control group and compare his performance at similar
00:45:36
tournaments on similar surfaces and see if there's some sort of market decline.
00:45:41
It's an interesting counterfactual on how you might look at the impact of coaching. So this has been uh this
00:45:47
week's show of Wharton Moneyball here on the Wharton podcast network. Again, I'm
00:45:51
Eric Bradlow. On behalf of myself, Kate Massie, Shane Jensen, and Aie Winer like
00:45:55
to thank you for joining us. I'd like to thank Marissa Rena, like to thank D
00:45:59
Patel, and as always, I'd like to thank Dion Simpkins, our associate producer,
00:46:03
sound engineer, the person that makes all of this happen. So, between now and next week, enjoy your statistics, enjoy
00:46:09
your sports. We'll see you next week here on the Wharton Podcast Network and
00:46:13
Wharton Moneyball.