In 2005, Aaron Schatz of Football Outsiders authored an essay in the inaugural issue of the Journal of Quantitative Analysis in Sports (JQAS), wherein he listed the 10 biggest problems facing NFL stat analysts at the time. Seven of the 10 were topic areas of future research, but the crucial three spoke to a fundamental need for better data:
- Standardizing and adding detail to play-by-play
- Opportunities on defense
Nearly a decade later, these problems have been solved for the most part. Aside from a handful of official scorers who still haven’t gotten the memo(s), play-by-play data is standardized across NFL stadiums, and various details have been added (e.g., individual snap counts, air yards, etc.). Meanwhile, public access to all-22 coaches’ film has made the game charting projects by Football Outsiders, ESPN Stats & Info, and Pro Football Focus both easier and more information-dense. At this point, unless the NFL releases tactical data (e.g., actual play calls, individual player assignments, etc.) or deploys tracking technology (a la SportVU and PITCHf/x) — neither of which is happening soon, if ever — we are currently in possession of the best data we’re going to have for the foreseeable future.
If so, then what are the problems facing NFL analytics now? In my view, they’re as follows:
- Using more “state of the art” statistical techniques
- Assessing the reliability and validity of existing statistics
- Stop reinventing the wheel
Let’s take these one at a time.
It frustrates me that much of the NFL research I see on the internet continues to rely on bivariate correlations and ordinary least squares (OLS) regression, both of which were brainchildren born over a century ago. Although this is fine for people who don’t know any advanced techniques, those of us who can break the chain of MS Excel should do so more often.
And you know what? It seems I’m not the only one who believes this. Keith Goldner introduced his Markov model of football in May 2011, while people like Boris Chen, Trey Causey, and Chase Stuart have applied mixture modeling, machine learning, and Bayes Theorem, respectively, on their sites in the past year or so. I applaud them, and wish to join the revolution.
The past decade has seen an arms race in the development of new statistics to evaluate teams and players. It’s been a boon to the NFL experience, but it’s also spawned a monster in the shape of a fan who doesn’t know which numbers to trust. One reason, in my opinion? After new measures are created, developers, readers, and consumers rarely provide sufficient evidence to convince skeptics why they should trust those measures. (Again, there are exceptions.)
To be clear, I’m not talking about bringing certain statistics out of their “black boxes.” Capitalism isn’t just a red herring in the online stats community, so I understand concealing the finer details of sausage-making. Rather, my frustration is that social researchers of all stripes (e.g., economists, psychologists, educators, etc.) spend a great deal of time assessing and reassessing the validity and reliability (i.e., the trustworthiness) of their measures — even ones that require a licensing fee to use. The typical outcome of these labors is that the cream of the crop gains the widest acceptance. That’s an industry standard we should strive for, and we can achieve it by doing more convincing, not less, especially when the huddled NFL masses are yearning to trust statistics.
A related point regards validating models, not just measures. For instance, one thing you want to see from, let’s say, an analysis predicting NFL wins, is the use of both an in-sample test (i.e., a calibration model) and out-of-sample test (i.e., a validation model). Again, exceptions exist, but it’s more often the case that I see someone perform a regression analysis on a full sample, and then that’s that. Maybe the model gets updated with new data later, but that doesn’t change what may have been invalid the first time around.
Those who know me are aware of my No. 1 bugaboo in NFL analytics: Someone comes along and creates yet another draft value chart, as if that hasn’t been done to death already. Now, don’t get me wrong, if the new wheel is an innovation — for instance, it’s the result of advanced statistical techniques — I’m all in favor of it and eager to read about it. Rather, the problem I’m identifying here is two-fold:
- There are so many research questions that haven’t been answered or haven’t even been attempted to be answered. Therefore, we shouldn’t waste valuable brain power and internet space writing about old news.
- Members of the internet NFL analytics community often seem to be disconnected from academia.
Checking to see if a topic’s been covered before is the first step in any research project — journalistic, scientific, or otherwise — so it eludes me how Google often eludes people that write on the internet. On the other hand, I am sympathetic to a lack of awareness of peer-reviewed research. Staying current with academia means getting behind the publishing paywall, which isn’t a universal capability. That said, there are many (ostensibly open) NFL research questions that have already been addressed in the academic literature, largely unbeknownst to the internet masses.
DT : IR :: TL : DR
In NFL analytics, the era of small data is over, but the era of big data is still a long way off. Therefore, we should spend this era of medium data focusing more on (a) optimizing our knowledge and (b) separating the statistical wheat from the chaff. The purpose of this site is to help move the needle in that direction, in solidarity with like-minded others. I don’t know everything (or most things), but I feel obligated to use whatever limited expertise I have in a manner that pushes the envelope. To this end, I’ll be doing the following in the coming days, weeks, months, and years:
- Applying advanced statistical techniques
- Evaluating the trustworthiness of existing stats
- Connecting our internet brain to our academic brain