There’s a disconnect between internet and academic research in NFL analytics, and that disconnect motivates the type of content you’ll be seeing in this post every Wednesday.
Regardless of whether or not you have a voracious reading appetite, many of you are probably familiar with books like Carroll et al’s (1988) The Hidden Game of Football, Albert et al.’s (2005) Anthology of Statistics in Sports, Moskowitz’s (2012) Scorecasting, and Winston’s (2012) Mathletics. Each of these seminal works either introduced and/or summarized scholarly NFL research, and their popularity indicates that the knowledge of stats-savvy fans is built on a solid foundation.
And yet, when those same stats-savvy fans — this author included — want to learn something new, they turn primarily to the internet. The (now-defunct) pro-football-reference (PFR) blog was a gold mine. If one wants to wax nostalgic about the PFR blog or read new research answering a wide variety of questions, one goes to Football Perspective. If one wants to read new research about game strategy, one goes to Advanced NFL Analytics. Every now and then, Jason Lisk, Football Outsiders, or Pro Football Focus either publish new research or embed it within a current events piece. To boot, similar sites — this one included — emerge onto the landscape quite frequently.
But what about academia? Have those ivory tower denizens nothing to offer us? Obviously, that’s a rhetorical question; of course they do. Did you know there’s (at least) 20 years-worth of peer-reviewed NFL research on field goal accuracy, beating the point spread, and racial bias in salary compensation?
I can think of two reasons why academic knowledge has gone largely ignored — or at least largely uncited — as analytics websites have been reinventing the wheel. First, much of said knowledge is hiding behind either a library database login or a paywall (the revenues of which probably finance ivory tower renovations). For instance, I just linked to three journal articles of the barricaded kind. Second, the nature of the academic publication process is such that a researcher dreams up an analysis, but the results don’t get published until six years later, and then a follow-up study revealing said research was hokum isn’t published until eight years after that. In short, these things take time, and the fast-paced information age undermines patience.
So, job No. 1a of this weekly article will be to bring the ebony of internet sites and the ivory (tower) of academic research into more perfect harmony with respect to a shared knowledge base. You may deem this to be a trivial task, so I’ll provide three examples that vividly illustrate why it isn’t one:
- In 2015, internet sites are still selling the idea that their game predictions can beat Vegas against the spread despite 30 years of academic research showing it’s highly likely they can’t.
- In 2011, academics on the internet were still reinventing the draft pick value wheel despite the general form being chiseled out of stone by other academics in 2005 and fashioned into steel-belted radials on the internet in 2007.
- And here’s my personal favorite: A host of internet writers believe that combine metrics predict a college player’s subsequent NFL performance despite damn near every academic study over the past 10 years showing that (at best) they only predict a player’s draft slot.
To its credit, Stats in the Wild highlights academic research from time to time. I wish to join them.
On the flip side, Job No. 1b will be to critique — and I mean that in the classical, non-valent sense — the metrics and research in an increasingly evergreen forest of NFL analytics websites. If my personal consumption of this content is any indication, there’s at least one new metric/study being posted each week; but are they trustworthy?
Anyone who has listened to the Three Cone Drill Podcast can confirm that trustworthiness is a big deal to me. In academia, the point is moot: As I mentioned above, the peer review process — for the most part — accurately separates the wheat from the chaff. In online NFL analytics, however, there are no such gatekeepers, and few (if any) barriers to publishing.
Berri and Bradbury (2010) wrote at length about the trustworthiness of online sports analytics content, concluding that
No matter the pedigree of the participants or the overall quality of ideas, [blogs] are no substitute for the formal peer-review process that typically governs economics research. Even after academics gain notoriety for their academic work, online commentary is not the same as academic research, even if the commentary is offered by an already-accepted reputational brand…Regardless of the usefulness of commentary, blogs should be considered a source of ideas and inspiration for further research rather than a serious research outlet. Good ideas developed on blogs should be written up, tested rigorously, and then published through normal channels.
Truth be told, I actually concur with this sentiment: Blogs — including this one — are no substitute for the rigors of the academic peer-review process. Where I differ with Berri and Bradbury, however, is that, insofar as there exist people in the sports analytics blogosphere (like myself) who have experience with peer review, those people do a disservice to knowledge by relegating online research to the status of second-class citizen. Rather, as “peers” in a mutual quest to better explain/predict our sport(s) of interest, it seems more productive for the more-academic among us to simply add a dash of peer review to the online mix: If an online study is shit, we call it out as shit; if it’s gold, we champion it as gold; if it’s somewhere in between shit and gold, we say as much and offer constructive criticism. Or perhaps we write it up and test it rigorously ourselves, but publish it outside “normal channels.”
DT : IR :: TL : DR
I feel a great disturbance in The Force: Academics and online researchers don’t seem to operate in the same cognitive space. Generally speaking, the former minimizes online research, while the latter minimizes — or is unaware of — the power of peer review. My Wisdom Wednesday posts will endeavor to bridge that gap.