If you haven’t read Part 1, go do that first by clicking here.
Today, I’m going to answer the third sub-question I posed there: How does the accuracy of way-too-early spreads change as the season progresses (both in terms of actual spreads and actual game results)?
As a more accessible example than last time of why this question is relevant to gauging the reliability of Cantor Gaming (CG) Technologies’ way-too-early point spreads, consider Nate Silver’s forecast of the 2016 presidential election, which currently gives Hillary Clinton a 48-42 edge in the popular vote. Just as it’s the dead period of the NFL offseason right now, it’s the dead period of the election cycle. Just as NFL training camps have yet to begin, the party conventions and proper general election campaining has yet to begin. And just as the testing ground of NFL preseason games hasn’t happened, so haven’t the sanctioned debates between Clinton and Donald Trump.
Due to this lack of vital information that can only be gleaned from important events taking place in the future, it stands to reason that, just as Silver’s forecast on July 13 should be farther from the actual election result on November 8 than his forecast on November 7, so too should CG’s forecast of point spreads on April 21 be farther from the actual spreads on September 8 than the spreads they release on September 7. In both contexts, if the way-too-early forecasts are more accurate than the day-ahead forecast, then there’s something wrong with how they incorporate new information. On the other hand, if the way-too-early forecasts are wildly more inaccurate than the day-ahead forecast, then there’s something wrong with the fundamentals underlying their initial forecasts.
Unlike in election forecasting, unfortunately, football forecasting doesn’t end upon kickoff of Panthers-Broncos in Week 1 (analogous to Clinton-Trump on election day). No, there are plenty of important events that will inevitably happen between Week 1 and Week 16 — arguably even more important than those prior to Week 1 — that dig CG’s way-too-early spreads into a deeper and deeper informational hole the farther into the season we get. This represents the second layer of their time-to-event disadvantage: Even if way-too-early spreads are accurate indicators of Week 1 spreads, it stands to reason that said accuracy is likely to decrease by Week 16, perhaps substantially given the randomness of NFL events.
What bloggers and fantasy football projectors want to see is way-too-early spreads being neither more accurate in Week 16 than Week 1 nor wildly more inaccurate. (Bettors want to see the opposite.) In short, if we’re to trust way-too-early spreads as reliable indicators of actual spreads, we’re aiming for the Goldilocks zone: Accurate in Week 1, and a gradual increase towards slightly less accuracy by Week 16.
In-Season Accuracy Results
Without further ado, here’s how accurately CG spreads predicted actual spreads over the course of the season from 2012 to 2015, as measured by root mean squared error (RMSE):
In this chart, the black dots are the actual values, the solid red line is the trend line, and those hieroglyphics inside the red box represent the equation for the trend line. And remember, the lower the RMSE, the higher the accuracy/reliability.
There are two clear takeaways from the chart. First, the reliability of way-too-early spreads for forecasting actual spreads is highest in Week 1. Whether we reference the actual RMSE (2.3) or the trend line (2.5), we can expect way-too-early spreads to be within 2.5 points of the actual spread. Second, there’s a gradual decrease in accuracy (aka gradual increase in RMSE) as the season progresses, culminating in an actual RMSE in Week 16 of 5.8 and a trend line RMSE of 5.2.
Whether you choose to apply the actual results or the trend line results in your various endeavors, these results imply that way-too-early spreads meet the two criteria I listed in Part 1 for being reliable indicators of actual spreads: a) accurate early in the season and b) resilient to time-related accuracy drop-offs. Truth be told, given what I said earlier about how much information way-too-early spreads don’t have, this result is remarkable — at least to me.
At this point, I’d be remiss if I didn’t revisit my discussion at the end of Part 1 about the distinction between perception and reality in NFL game results, and how way-too-early spreads are better indicators of the former than the latter. To that end, here’s how the accuracy of way-too-early spreads and actual spreads changes over the course of the season in terms of predicting actual margins of victory: (Reminder: Overall RMSE was 14.3 for way-too-early spreads and 13.7 for actual spreads.)
Woah! There are two interesting lessons to be learned here. First, way-too-early spreads are as accurate as actual spreads in terms of predicting actual Week 1 margins of victory. Furthermore, this result provides even more evidence of my belief perseverance hypothesis from Part 1: Between CG’s release and Week 1, the betting public’s perceptions of relative team strength don’t seem to change, despite four months-worth of additional information.
Second, although actual spreads have a lower RMSE than way-too-early spreads in 13 of 16 weeks, there’s no meaningful difference between the two through Week 11. It isn’t until Week 12 that their accuracy diverges, with actual spreads winning out thereafter. To me, it’s mind-boggling that it takes that long to diverge given the information advantage actual spreads have over way-too-early spreads.
DT : TL :: IR : DR
In this analysis, I’ve provided three pieces of evidence about the reliability of way-too-early spreads, and therefore the validity of their various applications:
- They’re reliable indicators of how the betting public subjectively perceives NFL matchups, not the objective reality of those matchups.
- Their reliability profile suggests that the process underlying CG’s way-too-early spreads is sound.
- In terms of actual margins of victory, it’s not until Week 12 that their accuracy becomes meaningfully worse than actual spreads.