By: Doug Ludemann
Predictive Metrics – Testing the Model
Predictive Metrics – Testing the ModelBy: Doug Ludemann
Predictive Metrics - Outline
In order to test our Age Adjusted model, I wanted to compare its results to that of the other two models, the simple model we derived from first down reception efficiency data, as well as the consensus projections available at Fantasy Pros.
With the trendlines in place, you can see that the age-adjusted model (AgeAdj) has a very slight edge in terms of the R2 value over our previous model, but the difference is probably not relevant, let alone significant in a statistical sense. Nor have we improved our model in terms of the degree of "fit" over the Fantasy Pros Projection numbers.
Thus we probably have a valid concept on our hands in terms of the age adjustment formula, but more research will have to be done before than information is actionable in any appreciable way.
I also added a single white line to the visualization below, which symbolizes a "perfect" fit of the data set. Admittedly, this chart is a bit messy but I feel it's useful to put your eyes on the actual data spread, even if it's those trendlines that are ultimately more important to the underlying mathematics with which we are considering.
However, I want to stress that simply because I was unable to recreate the degree of "fit" as with Fantasy Pros consensus model, there is no reason to believe that this has been a failed endeavor. Far from it, in fact.
First, to reiterate what I'd asserted in a previous post, all told I have a few hours invested in this project...with much of that time trying to coerce this blasted software to crunch far more numbers than it was designed to accommodate. This is as compared to the Fantasy Pro projections that are an aggregate of many different expert's projections. The sheer magnitude of the time invested in those aggregate rankings absolutely dwarfs what I've invested in this little math experiment. Additionally, the gulf of difference between the amount of time I invested versus the aggregate rankings is not equivalent to the relatively small difference in model fit (R2) which are relatively similar.
Second, R2 isn't the only way to compare data sets. For example, you can take the difference between the projected and the actual production and average out the magnitude of those difference (both positive and negative) across an entire sample of players using a Mean Absolute Error function. You can see from the table below that the Simple Algorithm performed better than the Fantasy Pros projection in terms of MAE, whereas my Age Adjusted model performed quite poorly by comparison.
Another commonly used metric to compare projection models to data is the Mean Absolute Percentage Error, which is very similar to the MAE except instead of error expressed in terms of actual magnitude, it is expressed in terms of the percent of the error. In the table below you can see that the Fantasy Pros aggregate rankings actually showed the lowest mean absolute percent error, with the Simple Algorithm finishing a full lap behind.
Perhaps the most useful of the model comparison metrics is the Root Mean Squared Error or RMSE, which is just as it sounds, the square root of the magnitude of the difference in values of the two models squared.
That jumble of words aside, the RMSE describes the magnitude of the error between your model and your data set.
You can see from the chart above that the Simple Algorithm once again outperforms the Fantasy Pros model, with the Age Adjusted model once again pulling up the rear.
I also calculated the Variance between the model and the actual 2020 fantasy production. Taking the average (or mean) of the variance for each model produced our Mean Variance (MV) metric.
You can see that once again the Simple Algorithm outproduced the field in terms of the Variance inherent in its projections with our Age Adjusted model once again failing to make the grade.
It's pertinent to stress that the Fantasy Pros projections performed admirably across all metrics. I certainly am not trying to give the impression that this simple predictive algorithm that we've derived is superior to the aggregate consensus projections. Rather I'm simply trying to reinforce the great degree of predictability inherent in the first down reception efficiency metric.
So what have we learned here?
That I didn't put much effort into my predictive algorithm? Definitely.
That we broke fantasy football with this new information. Probably not.
That we have reason to believe that this particular efficiency metric is one of the more important we should consider when trying to project various receiver's fantasy production? Certainly.
That statistical correlation doesn't win championships. Plausibly.
Look, this isn't rocket science. We've learned that first down receptions per game are more predictive than aDOT, Tgts/RR, Tgts/G, and Rec/G in terms of the correlation to fantasy production. They're also stickier year over year as compared to TDs/G.
And honestly it makes sense. We're talking about a lot of statistics that are all "baked into one" with the first down receptions stat. There's a target, which is correlated to fantasy scoring. There's a reception, also correlated to fantasy scoring. There's the implied yards produced because presumably that first down reception will be a bit deeper on average than your typical target. So it's really no surprise that the efficiency with which you catch first downs is highly correlated to fantasy production.
Additionally, teams generally target their most talented players with their first down targets. These are the guys who are going to see productive volume week-to-week, thus making it amongst the stickiest of stats. It is for these reasons that I'm planning on scoping out first down per game efficiency metrics to help me set my DFS rosters and identify trade targets.
Of course, this begs the question. Is a similar degree of correlation evident for runners?
As you can see from the chart above which represents rushes that resulted in first downs for the 2020 NFL season charted versus fantasy scoring, and I think you'll agree, the correlation is quite strong.
You may do with this information what you wish. I know what I'm going to do with it.
If you liked this article, please help by sharing it with a friend.
Good luck this year.
There's just no getting around it. The best Dynasty Geeks content is behind that login. More stats. More context. More snide remarks. More voids shouted at.
Subscribing will get you access to more of the good ish.
Hey, pride comes before the fall. Money makes the world go round, and any gratuity that you might feel is prudent would be greatly appreciated.
If you enjoyed the article, I ask that you share and please donate to help keep the Dynasty Geeks blog in the black.