Day 1 of the 2018 NFL draft is shaping up to be QB-centric, with five passers projected to come off the board. Owing to this, the commentary that’s entered my social media orbit over the past month has been completion percentage-centric. As I’ve never paid much attention to the draft, I was surprised to have a human emotion upon consuming said commentary. It didn’t take long to figure out why: No one knows, statistically speaking, how much college completion percentage is skill vs. luck. If it’s mostly luck, then the commentariat is wasting their time, which suggests I should unfollow. But if it’s mostly skill, then maybe I should take heed and start supporting their Patreons.
As is my wont, this meant that I had to do a reliability analysis of college completion percentage (Comp%). And then I figured, I might as well assess the reliability of college Yards per Attempt (YPA) and Adjusted Yards per Attempt (AY/A) too. ((For the unaware, I couldn’t assess ANY/A because the NCAA — in their infite wisdom — still counts sacks as running plays to this day.)) What follows is a post detailing these statistical inquiries, their results, and their implications, both theoretical and applicable to the 2018 draft.
Methods
It’s been a while, and also some of you may be new to this, so here are the details of how I went about performing this reliability analysis:
- I collected data for all Football Bowl Subdivision (FBS) QBs that had at least 8 games with 14 or more pass attempts from 2000 to 2017. ((Data courtesy of and qualifying threshold per Sports Reference.))
- To control for team effects, I included only those QBs that played 8+ games for the same team.
- Starting with QBs that played 8+ games, I randomly selected two sets of 4 games for each QB, and calculated their Comp%, YPA, and AY/A in both sets.
- For both of these metrics, I calculated its split-half correlation (r) between the two randomly-selected sets of games.
- I performed 25 iterations of Step 4 so that r converged.
- I repeated Steps 3-5, increasing the QB inclusion criteria in 8-game intervals, from 8+ games all the way to 48+ games. ((Long-time readers will notice that this step proceeds through 72+ games in an NFL context. The reason for stopping at 48+ games is that it’s impossible to play 72 games in college , and 48 games approximates a four-year starter.))
- For each “games played” group, I calculated the number of games at which the variance explained in each metric, R2, would mathematically equal 0.5. ((The formula is (Observations/2)*[(1-r)/r].))
- I calculated the True Comp%, True YPA, and True AY/A for a hypothetical QB that’s had an observed performance of 60.0% Comp%, 9.75 YPA, and 10.50 AY/A through 8, 16, 24, etc. number of games. ((The formula is [(Observed Performance * Observations) + (League-Average Performance * Stabilization Point)] / (Observations + Stabilization Point) ))
- I calculated a weighted average of the results from Steps 7 and 8. ((Weighted by group size.))
Results
First up, I’ll answer the question, “How many games-worth of stats do we need to see before an FBS QB’s YPA represents 50 percent signal (aka skill) and 50 percent noise (aka luck)?”
| Game Split | n | r | R2 = 0.50 | Avg Y/A | Obs 9.75 Y/A |
|---|---|---|---|---|---|
| 4 | 1,086 | 0.29 | 10 | 7.17 | 8.46 |
| 8 | 884 | 0.45 | 10 | 7.21 | 8.48 |
| 12 | 567 | 0.54 | 10 | 7.36 | 8.56 |
| 16 | 349 | 0.58 | 12 | 7.43 | 8.59 |
| 20 | 157 | 0.62 | 12 | 7.43 | 8.59 |
| 24 | 12 | 0.65 | 13 | 7.75 | 8.75 |
| Wtd Average | 10 | 7.26 | 8.51 |
For those unfamiliar or who have (understandably) forgotten, here’s how to read the above results table. The “bottom line” statistic is the stabilization point, which is displayed in the “R2 = 0.50″ column of the “Wtd Average” row. In this case, said statistic reveals that college YPA takes 10 games to stabilize, which means a YPA based on 10 games represents 50 percent signal and 50 percent noise. Alternatively, it means that said YPA should be regressed exactly halfway towards the mean to determine the FBS QB’s “true” (i.e., noise-independent) YPA.
The rest of the table’s “Wtd Average” row shows that a hypothetical FBS QB with a YPA of 9.75 after 10 games has a True YPA of 8.51, which is exactly halfway between 9.75 and 7.26 (i.e., the overall average YPA in my sample).
And if we want, we can increase granularity by exploiting the knowledge that the QBs in my sample averaged 25.6 attempts per game. ((720,834 attempts across 28,174 QB games)). Doing so reveals that YPA takes 264 attempts to stabilize. Interestingly, this is lower/quicker than the 396-attempt stabilization point I found for NFL YPA.
So what happens when we adjust YPA for touchdowns and interceptions so as to produce AY/A?
| Game Split | n | r | R2 = 0.50 | Avg AY/A | Obs 10.50 AY/A |
|---|---|---|---|---|---|
| 4 | 1,086 | 0.28 | 10 | 6.78 | 8.64 |
| 8 | 884 | 0.43 | 11 | 6.87 | 8.69 |
| 12 | 567 | 0.52 | 11 | 7.13 | 8.81 |
| 16 | 349 | 0.54 | 14 | 7.24 | 8.87 |
| 20 | 157 | 0.59 | 14 | 7.26 | 8.88 |
| 24 | 12 | 0.76 | 8 | 7.83 | 9.17 |
| Wtd Average | 11 | 6.95 | 8.73 |
Given how connected AY/A is to YPA mathematically, the similarity of results makes perfect sense. This extends to attempts too, as the mathematical translation to 284 attempts is negligibly different from YPA’s 264 attempts.
Furthermore, if adjusting for touchdowns and interceptions doesn’t have a meaningful impact on reliability with respect to FBS QBs, then why even have a stat that adjusts FBS QB YPA for touchdowns and interceptions? If I had to guess, this situation is the (understandable) result of early FBS analytics taking its cue from NFL analytics just as early NFL analytics took its cue from MLB analytics (e.g., Pythagorean wins).
Moving on, here are my reliability analysis results for Comp%:
| Game Split | n | r | R2 = 0.50 | Avg Comp Pct | Obs 60.0% Comp Pct |
|---|---|---|---|---|---|
| 4 | 1,086 | 0.38 | 6 | 58.6% | 59.3% |
| 8 | 884 | 0.56 | 6 | 58.9% | 59.5% |
| 12 | 567 | 0.64 | 7 | 59.9% | 60.0% |
| 16 | 349 | 0.71 | 6 | 60.3% | 60.2% |
| 20 | 157 | 0.75 | 7 | 60.4% | 60.2% |
| 24 | 12 | 0.88 | 3 | 62.5% | 61.2% |
| Wtd Average | 7 | 59.2% | 59.6% |
Lo and behold, FBS Comp% stabilizes faster than FBS YPA and FBS AY/A. In terms of games, the table shows 7 versus the previous 10 and 11. And translating this result into pass attempts, 167 is considerably lower than 264 and 284. Therefore, we can conclude that Comp% is the “stickiest” of these three stats; the most reflective of statistical signal vis-a-vis noise.
However, two caveats apply. First, as Josh Hermsmeyer has shown, and Bill Barnwell has applied to this year’s crop of FBS QBs, Comp% must be placed in the context of target depth (i.e., adjusted for air yards). My analysis did not do so, which is a clear limitation. That said, as you’ll see shortly, insofar as using this result to attach a “True Comp%” number to each QB in the upcoming draft, the rankings are similar enough to Bill’s to render this caveat moot (for now).
Second, and I want to make this point abundantly clear, I’m reporting a reliability analysis here, not a validity analysis. Just because FBS Comp% is “sticky,” doesn’t necessarily mean it predicts success at the NFL level. It’s said that “reliability sets the ceiling for validity.” Applied to our current statistical consideration(s), this means that FBS Comp% can’t predict NFL Comp% better than FBS Comp% predicts itself from one game to the next. Therefore, the best I can say based on my results is that FBS Comp% should be — must be — converted to True Comp% in predictive models. ((See Footnote 3 below.))
Speaking of which, my last topic of discussion wishes to take a 30,000-foot view of QB projection. In the results above, it’s clear that FBS Comp% is the most reliable metric of the three. Astute observers will have also noticed that it increases commensurate with the number of games played (i.e., as the “Game Split” column increases). Hell, it does for YPA and AY/A too, albeit to a lesser extent. Of course, this makes sense, as we would hope QBs get better on average with experience.
All of which is to say that the original idea underlying Football Outsiders’ original QB projection system — the Lewin Career Forecast — seems sound from a reliability perspective, even if it hasn’t stood the test of time from a predictive perspective. The two most important variables identified by Lewin over a decade ago were Comp% and Games Started. Perhaps the cutoffs — 60% Comp% and 37 games started — that arose later were arbitrary. Perhaps the 65% R2 he reported in 2007 suffered from overfitting. Nevertheless, those devils in the details should not obscure the overarching angel of a fact that the combination of Comp% and sample size (Read: Experience) reliably indicates FBS QB skill.
Are they predictive? This analysis can’t answer that. Reliability sets the ceiling on validity, so the point here is that these variables were a decade ago — and continue to be — a good place to start.
Statistics wonkery out of the way, below are True Comp%, True YPA, and True AY/A for the 16 FBS QBs who currently have a draft round designation by NFLDraftScout.com:
| Player | NFLDS Rd | True Comp Pct | Rk | True YPA | Rk | True AY/A | Rk |
|---|---|---|---|---|---|---|---|
| Josh Allen | 1 | 56.9% | 16 | 7.65 | 12 | 7.48 | 12 |
| Baker Mayfield | 1 | 69.8% | 1 | 10.62 | 1 | 11.86 | 1 |
| Sam Darnold | 1 | 64.9% | 6 | 8.54 | 6 | 8.72 | 7 |
| Josh Rosen | 1 | 60.9% | 10 | 7.98 | 10 | 7.99 | 11 |
| Lamar Jackson | 1 | 57.0% | 15 | 8.33 | 8 | 8.48 | 9 |
| Mason Rudolph | 2 | 63.2% | 8 | 9.41 | 2 | 9.87 | 2 |
| Mike White | 3-4 | 66.4% | 3 | 8.75 | 4 | 9.35 | 4 |
| Luke Falk | 4 | 68.3% | 2 | 7.05 | 13 | 7.35 | 13 |
| Chase Litton | 5-6 | 60.8% | 11 | 6.96 | 14 | 6.99 | 14 |
| Nic Shimonek | 6 | 66.4% | 4 | 8.04 | 9 | 8.55 | 8 |
| Kurt Benkert | 6-7 | 57.5% | 14 | 6.29 | 16 | 6.32 | 16 |
| Logan Woodside | 6-7 | 65.1% | 5 | 9.02 | 3 | 9.65 | 3 |
| Riley Ferguson | 7-FA | 63.1% | 9 | 8.68 | 5 | 9.27 | 5 |
| J.T. Barrett | 7-FA | 63.5% | 7 | 7.79 | 11 | 8.39 | 10 |
| John Wolford | 7-FA | 59.7% | 13 | 6.90 | 15 | 6.38 | 15 |
| Danny Etling | 7-FA | 59.7% | 12 | 8.43 | 7 | 8.84 | 6 |
Again, this post was on reliability (i.e., measurement), not validity (i.e., prediction), but I’m compelled to opine based solely on the former:
- Baker Mayfield laps the field in all three “true” stats, as he did based on Barnwell’s aforementioned air yards-adjusted stats.
- Josh Allen’s “true” stats aren’t commensurate with a No. 1 pick.
- Lamar Jackson’s and Josh Rosen’s “true” stats resemble Allen’s more than Mayfield’s.
- Mason Rudolph.
DT : IR :: TL : DR
It takes about half-a-season for completion percentage to stabilize for college QBs, whereas it takes about a full season for YPA and AY/A. Also, the more games a college QB plays, the “stickier” these stats get. This finding doesn’t necessarily mean college completion percentage is predictive of NFL success, especially when it’s not adjusted for depth of target (aka air yards). That’s because predicting NFL success is a validity question, not a reliability question. Based on my results, the answer to said question is to use True Comp%, True YPA, and True ANY/A in predictive models rather than their raw counterparts.