Through Week 1, Matthew Stafford leads all quarterbacks (QBs) with 10.81 Yards per Attempt (Y/A) and the league average is 7.12 Y/A. Of course, 32 attempts does not a season make, so anyone who’s ever heard of regression to the mean realizes Stafford probably won’t be able to sustain his current pace. Or, in the language of measurement theory, it’s highly unlikely that his current Y/A performance is a reliable indicator of his “true” Y/A ability. At some point, however, it will be, so today’s mission is to answer the question “When does Y/A stabilize?”
Now, I’ve ranted before about how we shouldn’t be reinventing the wheel in NFL analytics, and Monte McNair’s already established the stabilization point of Y/A to be approximately 400 attempts. But before you shame me for my blatant hypocrisy, let me just say that (a) I’ve improved his analysis in several ways; and (b) this is part of a much larger project that, in time, you’ll see far exceeds the scope of McNair’s work.
It’s been a while, so here’s a reminder of what the analysis entailed:
- I collected data for all QBs with at least 400 total pass attempts from 2002 to 2013.
- To control for team effects, I included only those QBs that had 400+ (or 500+ or 600+, etc.) attempts for the same team.
- Starting with QBs that had 400+ attempts, I randomly selected two sets of 200 attempts for each QB, and calculated the correlation (r) between the two sets.
- I performed 25 iterations of Step 2 so that r converged.
- I repeated Steps 3 and 4 for QBs with 500+, 600+, 700+ attempts, and so on.
- For each attempts group, I calculated the number of attempts at which the explained variance, R2, would mathematically equal 0.5.1
- I calculated a weighted average of my Step 6 results.2
- I calculated the “true” Y/A for a hypothetical QB that’s posted a 7.00 Y/A through X number of attempts.
Here they are:
|Attempts||n||r||R2 = 0.50||Avg Y/A||Obs 7.0 Y/A|
As a reminder, here’s an example of how to read the table. There were 52 QBs who had (at least) two sets of 550 attempts for the same team, and those QBs had an average Y/A of 7.22. Given their split-half correlation of 0.58, Y/A stabilized at 396 attempts for this group. And given their 7.26 average Y/A, we can estimate that a QB with 7.00 Y/A after 550 attempts has a “true” Y/A of 7.11.
The “Wtd Average” row is the bottom line of the analysis, both literally and figuratively. Mimicking McNair’s previous results, I found that Y/A stabilizes at 396 attempts (aka “approximately 400”). Furthermore, we can use this total along with the league-average Y/A to estimate that a QB with 7.00 Y/A after 396 attempts has a “true” Y/A of 7.09.
And just to call attention to something I’ve glossed over in the past, take note that 7.09 is halfway between the hypothetical QB’s 7.00 Y/A and the average QB’s 7.18 Y/A. This is due to the underlying math that allows for our “half-skill/half-luck” conclusion about Y/A stability.3
I’m not going to be posting what follows every week, but it’s an example of what you can do with reliability analysis in terms of practical application.
Now that we know Y/A stabilizes at 396 attempts, and now that we have passing data from Week 1, we can start looking at real QBs rather than hypothetical ones. All we have to do is add 396 attempts of average Y/A performance to what each QB produced in Week 1. Doing so results in the following “True” Y/A rankings:4
|QB||Tm||Obs Y/A||Rk||True Y/A||Rk|
The first thing that jumps out is that the Top 3 in observed Y/A (“Obs Y/A”) are flipped in the True Y/A rankings. That’s because, although they produced similar Y/A stats, Ryan’s came over more attempts (43) than did Roethlisberger’s (34) and Stafford’s (32). Mathematically speaking, Ryan’s calculation was based on a regression to the mean of about 90%,5 whereas Roethlisberger’s and Stafford’s were based on about 93% regression to the mean.
A couple of other QBs who stand out are Michael Vick and Joe Flacco. Vick’s 0 pass yards against Oakland came on only 1 attempt, so practically all of his 0.00 observed Y/A is regressed towards the Y/A of an average QB. On the flip side, Flacco’s 5.56 observed Y/A was the 28th-ranked Y/A performance of Week 1 and it featured the most attempts (62). That combination resulted in an even lower True Y/A ranking despite gaining over a yard’s worth of it thanks to regression — or in this case progression — to the mean.
DT : IR :: TL : DR
Matthew Stafford had a great game against the Giants. But while he was able to sustain 10.81 Y/A over 32 attempts, he’s unlikely to as the season progresses. How many more attempts do we have to see before his Y/A becomes a reliable indicator over the long term? Well, in virtual lock-step agreement with previous research, I found that it takes 396 attempts before a QB’s Y/A represents 50% of his “true” Y/A ability and 50% luck.
The formula is (attempts/2)*[(1-r)/r]. ↩
Weighted by group size. ↩
The formula is [(7.00 * 396) + (7.18 * 396)]/ (396+396) = 5,615.28/792 = 7.09. ↩
The table is fully searchable and sortable, and you can change the number of displayed entries. ↩
90% represents 396 of the 396 + 43 = 439 attempts in the denominator. ↩