According to Pro Football Focus’s (PFF) charting, Cam Newton led all qualifying quarterbacks in average depth of target (aDOT) during the 2016 regular season; this after ranking second in 2015. On the other end of the spectrum, Alex Smith had the lowest aDOT in 2015 and the third-lowest in 2016. Observations like these are crack to my reliability-addled brain, so my id immediately sought out to examine aDOT’s “stickiness” (aka stability aka predictiveness). My superego didn’t stand a chance because stoking a fire under the crack pipe was increased public interest over the past three falls in a stat called “air yards,” which is statistically identical to “depth of target:”1
As far as I can tell, increased interest for the 2016 season can be traced back to Josh Hermsmeyer of Rotoviz demonstrating that both a receiver’s air yards per target (aYPT) and their market share of team air yards have impressive predictive utility in the context of fantasy football. But as the above graph suggests, the in-season boom started in 2014 after years of stagnant interest. Which is weird because air yards has been around as a stat since now-ESPN Analytics Specialist, Brian Burke, introduced air yards in June 2007 and produced air yards-based quarterback rankings in August 2007. What’s more, separating passing yardage into yards before the catch and yards after the catch (YAC) came even earlier to the football analytics world, when Aaron Schatz of Football Outsiders (FO) started their Game Charting Project in 2006.
Between then and now, several stats have been created based on this distinction:
- In August 2010, Bill Barnwell — then at FO, now at ESPN — created YAC+, which adjusted a receiver’s yardage based on — among other relevant factors — his distance downfield at the point of the reception.
- In August 2011, Dean Oliver — then at ESPN, now at TruMedia — introduced Total QBR, a major feature of which is weighting air yards and YAC differently when evaluating a quarterback.
- In April 2012, backed by the deep pockets of Neil Hornsby, PFF’s army of game charters produced the data required for Mike Clay — then at PFF, now at ESPN — to introduce (aDOT).
So here we are in February 2017, and interest in air yards is at its highest point in (at least) six years — which got me thinking. Yes, Josh has shown that aYPT is consistent from season to season for wide receivers, but
- no one has subjected aDOT (or air yards) to the kind of split-half reliability analysis that’s the hallmark of this site; and
- no one has done a reliability analysis on aDOT (or air yards) across all four offensive skill positions (i.e., quarterback, running back, wide receiver, and tight end).
Therefore, what follows is the first in an eight-part series investigating the reliability of aDOT and YAC at the skill positions; first up, aDOT for quarterbacks.
As a reminder, here’s my procedure applied to today’s reliability analysis:
- I collected aDOT data for all QBs that had at least 8 games played for the same team from 2006 to 2016.
- Starting with QBs that played 8+ games for the same team, I randomly selected two sets of 4 games for each QB, and calculated their aDOT in both sets.
- I calculated the split-half correlation (r) between the two randomly-selected sets of games.
- I performed 25 iterations of Steps 2 and 3 so that r converged.
- I repeated Steps 2-4 in 8-game intervals, from 16+ games all the way to 72+ games.
- For each “games played” interval, I calculated
- I calculated a weighted average of the results from Step 6.4
Below is the stability table for aDOT. If you’re unfamiliar with how to read it, click here.
|Games||n||r||R2 = 0.50||Avg aDOT||Obs 9.00 aDOT|
Cutting to the chase, focus on the bottom row labeled “Wtd Average.” There you’ll find that quarterback aDOT takes 10 games to stabilize. In statistical terms, this means that it takes 10 games for a quarterback’s aDOT to represent 50 percent skill and 50 percent luck. In practical terms, it means that, if a quarterback’s aDOT is 9.00 after 10 games, then you should expect it to regress exactly halfway towards the league average of 8.89. That’s where the bottom row’s 8.94 “true” aDOT figure comes from.5 Translating this to aimed throws, defined by PFF as “pass attempts less batted balls, spikes, throw aways, and hits,” aDOT takes 286 aimed throws to stabilize given my sample average of 28.6 per game.
Now, the fact that aDOT stabilizes in 10 games (or 286 aimed throws) is all well and good, but how does that compare to other quarterback stats? Well, I previously found that yards per attempt (YPA) stabilizes in 396 throws. Obviously, that’s throws, not aimed throws, so it’s a bit of an apples-to-oranges comparison. That said, barring batted balls, spikes, throw aways, and hits comprising more than 25 percent of throws for the average quarterback,6 aDOT stabilizes faster than YPA.
As a final way to demonstrate the reliability of aDOT in action, below is a table showing actual and “true” aDOT for every quarterback in 2016 with at least 100 aimed throws per PFF:
|Robert Griffin III||CLE||5||136||9.1||15||8.9||19|
|Alex D. Smith||KC||15||464||6.9||38||7.6||38|
DT : IR :: TL : DR
Recent interest in air yards has motivated me to examine the reliability of passing stats that distinguish between a quarterback’s responsibility for yardage and that of his receivers. In this first installment of an eight-part series, I found that it takes 10 games (or 286 aimed throws) for a quarterback’s aDOT to stabilize. Given the assumption that at least 75 percent of attempts are aimed throws, this is faster than it takes for YPA to stabilize (396 attempts), which means aDOT is a more reliable indicator of quarterback passing skill.
CORRECTION: A previous version of this post incorrectly stated that aYPT does not include air yards on incomplete targets.
Per Josh Hermsmeyer, the correlation between these two stats is .997. ↩
The formula is (Games/2)*[(1-r)/r]. ↩
The formula is [(Observed Performance * Games) + (League-Average Performance * Stabilization Point)] / (Observations + Stabilization Point) ↩
Weighted by group size. ↩
with intentional rounding, of course ↩
i.e., 396 throws minus 286 aimed throws, all divided by 396 ↩