Based on the confirmatory factory analysis (CFA) I introduced in Part 1 and tested in Part 2, a model of QB Quality implied by the Adjusted Net Yards per Attempt (ANY/A) formula fits real-world data well, and that goodness of fit generalizes across multiple samples of QBs. That said, two of its components, Interceptions per Dropback (INTs/Db) and Sack Yards per Dropback (SkYds/Db) are far worse indicators of QB Quality than are Passing Yards per Dropback (Yds/Db) and Touchdowns per Dropback (TDs/Db).
In this final installment, I’m going to discuss an additional piece of evidence showing why INTs/Db and SkYds/Db are bad indicators of QB Quality (insofar as that’s what ANY/A’s measuring). From there, I’ll finish up by presenting CFA-implied ANY/As for qualifying QBs in 2014, and highlighting instances where those values differ considerably from actual ANY/A.
Modification Indices and Correlated Errors
As the name implies, CFA is a confirmatory technique: You should only use it when you have a well-founded theory about the phenomenon of interest, as well as a large body of previous research that supports it. However, the best laid plans of mice and men often go awry, especially when no one’s ever applied CFA (or structural equation modeling, in general) to the specific question at hand, which is the case here as far as I can tell.
The good news is that, no matter which software you use, a CFA results output tells you specifically where the model went wrong via what are called modification indices (MIs). We’ve already seen from factor loadings in Part 2 that INTs/Db and SkYds/Db aren’t good indicators of ANY/A; modification indices give an additional clue as to the nature of the that problem:
Without going into unnecessary detail, the “M.I.” column in this results table suggests that (a) the fit of our theory-based model of QB Quality could be improved if we included a residual error correlation (aka what the term “WITH” means in Mplus syntax) between SkYds/Db and INTs/Db, and (b) said correlation would equal approximately 0.17.
But what is an error correlation in the context of CFA? Well, in general, it can mean a variety of things, but with respect to ANY/A it suggests that there’s some QB trait underlying INTs/Db and SkYds/Db other than the one underlying Yds/Db and TDs/Db. Stated differently, the skills that make a QB throw for yards and touchdowns might very well be different than the skills that make them avoid throwing for drive-killing INTs and taking drive-killing sack yardage.
2014 Estimated ANY/A
With the above caveat aside, it nevertheless remains the case that the underlying ANY/A model, which says that some combination of Yds/Db, TDs/Db, INTs/Db, and SkYds/Db indicates QB Quality, adequately fits real-world data from randomly-assigned qualifying QB seasons since the “Mel Blount Rule.” Therefore, the final analytical exercise in this series is to apply said model to data from the 2014 NFL season. To wit, below are ANY/As for last season’s qualifying QBs based on (a) the standard ANY/A math, and (b) my CFA math: (It’s sorted by the difference between CFA ANY/A and actual ANY/A, but it’s fully sortable and searchable.)
|QB-OFF||Act ANY/A||Rk||CFA ANY/A||Rk||+/-||Rk|
To be clear about the “Diff” column in the above table, it doesn’t show how much a QB overachieved or underachieved, per se. Rather, it shows how much a QB’s ANY/A would change if we calculated it based on my CFA, rather than the formula derived at Pro-Football Reference.
But if it remains unclear to you, here’s the take-home message: Actual ANY/A overvalues a QB’s ability to avoid interceptions and sack yardage. How so, you ask? Well, 8 of the Top 10 in “Diff” also ranked in the Top 10 of this past season’s INT/Db rankings, and 4 of those 8 — Tom Brady, Carson Palmer, Ben Roethlisberger, and Drew Stanton — also finished in the Top 10 of this past season’s SkYds/Db rankings. Meanwhile, the CFA ANY/A rankings basically tracked alongside the 2014 rankings for Yds/Db and TD/Db.
In terms of evaluating individual QBs according to a logical amalgam of the above, with their high CFA ANY/A and high Actual ANY/A, Brady, Roethlisberger, and Aaron Rodgers were the best in the world last season.
IR : DT :: TL : DR
My CFA test of ANY/A suggested that INTs/Db and SkYds/Db weren’t good indicators of QB quality: Randomized real-world data samples said they minimally contributed to QB quality and may very well have represented another QB trait entirely. That was further evinced by the fact that only 3 of the Top 10 QBs in CFA-implied ANY/A — Brady, Roethlisbeger, and Rodgers — were able to produce an Actual ANY/A that withstood these negative effects.
Going forward, we should consider ANY/A to be a highly reliable metric that’s only half-valid. Pro Football Reference has an alternative QB metric, Adjusted Yards per Attempt (AY/A), that may prove to be just as reliable, but more valid.
Note to self: Possible idea for future Modeling Monday content.