When Does Average Depth of Target Stabilize for Quarterbacks?

According to Pro Football Focus’s (PFF) charting, Cam Newton led all qualifying quarterbacks in average depth of target (aDOT) during the 2016 regular season; this after ranking second in 2015. On the other end of the spectrum, Alex Smith had the lowest aDOT in 2015 and the third-lowest in 2016. Observations like these are crack to my reliability-addled brain, so my id immediately sought out to examine aDOT’s “stickiness” (aka stability aka predictiveness). My superego didn’t stand a chance because stoking a fire under the crack pipe was increased public interest over the past three falls in a stat called “air yards,” which is statistically identical to “depth of target:”1

As far as I can tell, increased interest for the 2016 season can be traced back to Josh Hermsmeyer of Rotoviz demonstrating that both a receiver’s air yards per target (aYPT) and their market share of team air yards have impressive predictive utility in the context of fantasy football. But as the above graph suggests, the in-season boom started in 2014 after years of stagnant interest. Which is weird because air yards has been around as a stat since now-ESPN Analytics Specialist, Brian Burkeintroduced air yards in June 2007 and produced air yards-based quarterback rankings in August 2007. What’s more, separating passing yardage into yards before the catch and yards after the catch (YAC) came even earlier to the football analytics world, when Aaron Schatz of Football Outsiders (FO) started their Game Charting Project in 2006.

Between then and now, several stats have been created based on this distinction:

So here we are in February 2017, and interest in air yards is at its highest point in (at least) six years — which got me thinking. Yes, Josh has shown that aYPT is consistent from season to season for wide receivers, but

  1. no one has subjected aDOT (or air yards) to the kind of split-half reliability analysis that’s the hallmark of this site; and
  2. no one has done a reliability analysis on aDOT (or air yards) across all four offensive skill positions (i.e., quarterback, running back, wide receiver, and tight end).

Therefore, what follows is the first in an eight-part series investigating the reliability of aDOT and YAC at the skill positions; first up, aDOT for quarterbacks.

Methods

As a reminder, here’s my procedure applied to today’s reliability analysis:

  1. I collected aDOT data for all QBs that had at least 8 games played for the same team from 2006 to 2016.
  2. Starting with QBs that played 8+ games for the same team, I randomly selected two sets of 4 games for each QB, and calculated their aDOT in both sets.
  3. I calculated the split-half correlation (r) between the two randomly-selected sets of games.
  4. I performed 25 iterations of Steps 2 and 3 so that r converged.
  5. I repeated Steps 2-4 in 8-game intervals, from 16+ games all the way to 72+ games.
  6. For each “games played” interval, I calculated
    1. the number of games at which the aDOT variance explained, R2, would mathematically equal 0.5.2
    2. the “true” aDOT for a hypothetical QB that’s had an observed performance of 9.00 aDOT through that number of games.3
  7. I calculated a weighted average of the results from Step 6.4

Results

Below is the stability table for aDOT. If you’re unfamiliar with how to read it, click here.

GamesnrR2 = 0.50Avg aDOTObs 9.00 aDOT
Wtd Average108.898.94
41830.3098.848.92
81020.6158.948.97
12740.51128.908.95
16540.59118.908.95
20420.63128.908.95
24370.69118.908.95
28270.72118.918.95
32220.62208.968.98
36170.65198.928.96

Cutting to the chase, focus on the bottom row labeled “Wtd Average.” There you’ll find that quarterback aDOT takes 10 games to stabilize. In statistical terms, this means that it takes 10 games for a quarterback’s aDOT to represent 50 percent skill and 50 percent luck. In practical terms, it means that, if a quarterback’s aDOT is 9.00 after 10 games, then you should expect it to regress exactly halfway towards the league average of 8.89. That’s where the bottom row’s 8.94 “true” aDOT figure comes from.5 Translating this to aimed throws, defined by PFF as “pass attempts less batted balls, spikes, throw aways, and hits,” aDOT takes 286 aimed throws to stabilize given my sample average of 28.6 per game.

Now, the fact that aDOT stabilizes in 10 games (or 286 aimed throws) is all well and good, but how does that compare to other quarterback stats? Well, I previously found that yards per attempt (YPA) stabilizes in 396 throws. Obviously, that’s throws, not aimed throws, so it’s a bit of an apples-to-oranges comparison. That said, barring batted balls, spikes, throw aways, and hits comprising more than 25 percent of throws for the average quarterback,6 aDOT stabilizes faster than YPA.

As a final way to demonstrate the reliability of aDOT in action, below is a table showing actual and “true” aDOT for every quarterback in 2016 with at least 100 aimed throws per PFF:

PlayerTmGAimedaDOTRkTrue aDOTRk
Cam NewtonCAR1546011.0110.21
Jameis WinstonTB1653110.8310.12
Matt BarkleyCHI721010.929.73
Carson PalmerARI1555210.069.64
Marcus MariotaTEN1541510.149.65
Tyrod TaylorBUF153969.879.46
Ryan FitzpatrickNYJ143759.879.47
Josh McCownCLE514910.149.38
Ben RoethlisbergerPIT144879.599.29
Kirk CousinsWAS165609.4119.210
Brock OsweilerHOU154729.2129.111
Trevor SiemianDEN144379.2129.012
Jay CutlerCHI51269.599.013
Russell WilsonSEA164929.1159.014
Aaron RodgersGB165709.0178.915
Philip RiversSD165169.0178.916
Andrew LuckIND155009.0178.917
Bryce PettyNYJ61259.2128.918
Robert Griffin IIICLE51369.1158.919
Matt RyanATL165028.9208.920
Blaine GabbertSF61528.7218.821
Blake BortlesJAX165788.7218.722
Dak PrescottDAL164308.6238.723
Brian HoyerCHI61918.3278.624
Case KeenumLA103018.4248.625
Tom BradyNE123918.4248.626
Andy DaltonCIN165118.4248.527
Colin KaepernickSF113068.3278.528
Cody KesslerCLE91748.0328.529
Ryan TannehillMIA133598.2308.530
Eli ManningNYG165728.3278.531
Derek CarrOAK155338.2308.432
Carson WentzPHI165557.9338.233
Matthew StaffordDET165577.9338.234
Jared GoffLA71907.3378.235
Joe FlaccoBAL166247.8358.136
Drew BreesNO166257.7368.037
Alex D. SmithKC154646.9387.638
Sam BradfordMIN155136.6397.439

DT : IR :: TL : DR

Recent interest in air yards has motivated me to examine the reliability of passing stats that distinguish between a quarterback’s responsibility for yardage and that of his receivers. In this first installment of an eight-part series, I found that it takes 10 games (or 286 aimed throws) for a quarterback’s aDOT to stabilize. Given the assumption that at least 75 percent of attempts are aimed throws, this is faster than it takes for YPA to stabilize (396 attempts), which means aDOT is a more reliable indicator of quarterback passing skill.

CORRECTION: A previous version of this post incorrectly stated that aYPT does not include air yards on incomplete targets.

Email to someoneShare on Facebook0Tweet about this on TwitterShare on Reddit0

  1. Per Josh Hermsmeyer, the correlation between these two stats is .997. 

  2. The formula is (Games/2)*[(1-r)/r]

  3. The formula is [(Observed Performance * Games) + (League-Average Performance * Stabilization Point)] / (Observations + Stabilization Point)  

  4. Weighted by group size. 

  5. with intentional rounding, of course 

  6. i.e., 396 throws minus 286 aimed throws, all divided by 396 

Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *