Here’s a challenge: Jump onto the Google machine and try to find out why a quarterback (QB) needs 224 pass attempts to qualify for official rankings. Better yet, try to find an explanation for any minimum requirement that the NFL uses to rank players at any position. Go ahead, I’ll wait…
I couldn’t find anything either, even after attempting to get an answer directly from the three organizations that might know something about it: the Elias Sports Bureau, Stats LLC, and the NFL itself.1
In my post on measurement validity and reliability, I argued that, as members of the NFL analytics community, it’s our responsibility to convince people that our stats are trustworthy. One way to do so involves showing that our methodological decisions are not arbitrary. Absent a publicly available explanation, the league’s mandatory minimums for official stat rankings fail this test.
Luckily, tackling this problem doesn’t require a peek behind the NFL curtain. Instead, all we need to do is a reliability analysis that (a) is similar to ones I’ve already published on I//R, and (b) has been used by sabermetricians to define MLB mandatory minimums for almost a decade.
The NFL says a QB needs 224 attempts to qualify for its Passer Rating (PR) rankings. Today, I’ll use reliability analysis to tell you what the mandatory minimum should be.
There was only one difference between the methods I used here and what I did previously to determine when PR “stabilizes.”2 In the context of stability, we were asking the question, “How many of an individual QB’s pass attempts must we observe before his PR is a reliable indicator of efficiency?” In order to speak of individuals, we need to put all QBs on the same playing field in terms of attempts — pun intended — so that our answer generalizes across all QBs, regardless of whether their name is Matthew Stafford, Matt Schaub, Matt Ryan, or Ryan Tannehill.
In the context of mandatory minimums, we’re asking a different question: What is the minimum number of pass attempts before we can reliably compare a group of QBs with at least that many attempts? And because our objective is comparing (aka ranking) a group of QBs rather than judging individual QBs, we don’t need them to have exactly X number of attempts; just at least X number of attempts.
Here’s an example of how this distinction manifests itself in practice. The past two seasons, Colin Kaepernick had 634 pass attempts with San Francisco and Russell Wilson had 800 with Seattle. As part of estimating when PR stabilizes, I randomly split Kaepernick’s attempts into two sets of 300, discarding the remaining 34, and I randomly split Wilson’s into two sets of 300, discarding the remaining 200. It was necessary to put Kaepernick and Wilson on equal footing so we could apply our stabilization point to both of them — and to all other QBs for that matter.
To estimate the mandatory minimum for PR, however, we don’t need Kaepernick or Wilson (or everyone else) to be on equal footing because ranking QBs assumes they’re different; otherwise, what’s the point? Therefore, this time around, I didn’t have to discard their extra attempts: I just split Kaepernick’s into two random sets of 317 and Wilson’s into two random sets of 400.
Here are my findings:
|Minimum Split||n||Average Split||r||R2 = 0.50|
In this table, “Minimum Split” is the smallest split size a QB needed to be included in a given group, n is the the number of QBs that were included, “Average Split” is the average split size for that group, r is the group’s split-half correlation, and “R2 = 0.50″ is their estimated mandatory minimum to qualify for PR rankings. So for instance, there were 74 QBs who had at least two sets of 400 attempts for the same team, and those 74 QBs actually averaged 973 attempts in each split. Their split-half correlation was 0.77, which implies that their mandatory minimum should be 119 attempts.
So is the NFL correct when they require at least 224 pass attempts to qualify for their PR rankings? Using the “Wtd Average” of each group’s results, we see that the unequivocal answer is no, and they’re not even close: The mandatory minimum for PR rankings should be 109 pass attempts.3
Thankfully, no PR title since 2002 went to the wrong QB because of the NFL’s seemingly arbitrary qualification standard. That said, if the mandatory minimum was 109 attempts instead of 224, here’s a snapshot of how the PR rankings would have differed over that period:
- 118 additional QB seasons would have qualified.
- 29 “non-qualifying” QB seasons would have ranked in the Top 20.
- 13 of those 29 would have ranked in the Top 10.
- 6 of those 13 would have ranked in the Top 5.
- Tim Rattay (96.6 in 2003), Alex Smith (104.1 in 2012), and Colin Kaepernick (98.3 in 2012) have been unwitting victims of the San Francisco Screwjob, as they’ve posted 3 of the Top 6 PRs among “non-qualifiers.”
- The highest finish among “non-qualifiers” would have been Marc Bulger ranking 2nd in 2002 with a 101.5 PR over 138 attempts.
DT: IR :: TL : DR
For QBs, is there something magical about 224 pass attempts? As far as I can tell, the NFL hasn’t explained their qualifying standards for official player rankings, so 224 might as well be a number pulled out of thin air. To break the magician’s code, I used reliability analysis.
It turns out that the magic number of attempts for a QB to be qualified for PR rankings should be 109, not 224. The unfortunate byproduct is that 118 QB seasons have been left out since 2002, 13 of which would have ranked in the Top 10 if not for the magic of 224.
To date, Stats LLC is the only one to respond, and their response was “Ask the NFL.” ↩
If you’re interested, Russell Carleton provides a more thorough explanation of the distinction in the sabermetrics study I linked two paragraphs ago. ↩
Well, at least it should be in today’s NFL. Because my sample was restricted to the 32-team era from 2002 to 2013, I can’t say anything about the appropriate mandatory minimum for prior eras. I suppose that’s a post for another day. ↩