Validity of an nflWAR statistic

[minutemancl]

I was reading yet another article today about the devaluation of RBs, especially as it pertains to the draft. It touched on the regular points every other one of these articles does: it is unwise from a economic perspective as the guaranteed money and average of the contract a top-5 pick RB makes is substantially more at their position than at another, and that a RB by committee, using players acquired through free agency and late in the draft, can you get you similar production.

However, one thing about this article stood out. The author mentioned an nflWAR statistic used in a graph they presented. They used it when comparing RBs picked high in the draft compared to those taken later. WAR, wins above replacement, is a statistic that has been used in baseball for a while now ever since teams started utilizing sabermetrics. WAR in baseball always made sense to me though: in any given instance, there are a limited number of factors that contribute to the success or failure of an at-bat. It doesn't get much more intricate or complicated than the relationship between 2 players on the field, the pitcher and the batter. (Obviously if the batter makes contact, fielding is factored in, but unless Arod is running the bases, the result of the play there depends on the player fielding the ball, the player getting the ball thrown to them for a tag or force out, and the base runner. Three factors that can all be measured. There is no one blocking the base runner from running or trying to steal the ball away from the infielder). I've always thought that creating a similar analytical approach to football would be impossible. There are 22 players on the field at a time and with so many moving pieces, it is impossible to create a reproducible method of assigning a wins above replacement grade to each player. If you want the full story on how a player performs and how good or bad they are, you watch film exclusively for that player and grade their snaps based on their responsibilities. With 10 other players all affecting the individual performance of 1 player, there are too many confounding variables to get a reliable grade based on analytics alone. 

So after hearing about this nflWAR, I decided to look it up myself. What I found was 3 people (students? adjuncts?) from Carnegie Mellon University who created a reproducible nflWAR statistic for offensive players. You can find it yourself here: https://arxiv.org/abs/1802.00998. 

It is an in depth 43 page pdf that goes into the whole process of creating their nflWAR. If you have the time, please read it and give me your thoughts. As to the actual thread title, here is what I noticed and gathered. 

The nflWAR statistic as it is posed here, as i postured before and continue to believe, is highly flawed. I don't think it will ever be not flawed. The authors make a good attempt though. For one, they limit their nflWAR only to those offensive players who directly touch the ball on a play: QBs, RBs, TEs, WRs. Offensive skill players. They acknowledge that the OL plays a part in the success or failure of a play, but due to their goal of making this reproducible and only reliant on publicly available information, they can only give a grade for the OL as a whole with no acknowledgement as to the actual players playing those positions. This means it doesn't include the quality of those players, much less how or why they lost their rep. This is the biggest issue for me on why this fails, and the reason for that should be obvious. Another facet this statistic seems to fail in is in situations like these: the play call is a pass play to the x receiver running a slant. The receiver gets way too much depth in his route, allowing a heady corner to jump the route and get an interception. Overall, there doesn't seem to be built in accommodations for teammates failing in their responsibilities which therefore reflects on a player who can be graded. 

Read it over and let me know what you guys think about this. Am I wrong and is it actually a pretty good and effective way to measure a player's worth? Or do you think like me that there are too many confounding variables that are unaccounted for? For what its worth, I think this is a really impressive effort that really only falters in that one area. The authors account for a lot of areas you don't have to in baseball, such as down and distance and time remaining. 

[ninjapirate]

nfl game sample sizes are too limited and players aren't individuals so a war stat for the nfl will be nothing short of useless.

[JAF-N72EX]

After reading this, I applaud the effort. But the methodology itself is not practical for what they're aiming to do here. Using NFL.com play-by-data to try an evaluate each player without adding other contextual factors behind it is bad idea alone from the jump. But then, to take this information and try an correlate that into wins is only digging a deeper hole. 

5 hours ago, minutemancl said:

For one, they limit their nflWAR only to those offensive players who directly touch the ball on a play: QBs, RBs, TEs, WRs. Offensive skill players.

It's funny that you say this because as I was reading this I felt like I was reading an article based more on fantasy football than the actual game. And the methods that they apply here I think could possibly work very well for fantasy football. Not for sure, I don't play FF, but with a few tweaks it could be useful. But FF has no place in real football whatsoever.

5 hours ago, minutemancl said:

The nflWAR statistic as it is posed here, as i postured before and continue to believe, is highly flawed. I don't think it will ever be not flawed. The authors make a good attempt though. For one, they limit their nflWAR only to those offensive players who directly touch the ball on a play: QBs, RBs, TEs, WRs. Offensive skill players. They acknowledge that the OL plays a part in the success or failure of a play, but due to their goal of making this reproducible and only reliant on publicly available information, they can only give a grade for the OL as a whole with no acknowledgement as to the actual players playing those positions. This means it doesn't include the quality of those players, much less how or why they lost their rep. This is the biggest issue for me on why this fails, and the reason for that should be obvious. Another facet this statistic seems to fail in is in situations like these: the play call is a pass play to the x receiver running a slant. The receiver gets way too much depth in his route, allowing a heady corner to jump the route and get an interception. Overall, there doesn't seem to be built in accommodations for teammates failing in their responsibilities which therefore reflects on a player who can be graded.

This is where I have to defend them. It's up to the user to fully understand the purposes, goals, and objectives to what they are reading beforehand and also understands it's limitations based on that. This was only meant to be an analytical project. Not a scouting project. Therefore, that is all that you should have expected to come from this.

Analytics and scouting are not the same. People get these two things confused with one another way to often.

[tom cody]

I think with so many different people doing different on the field, a War stat for the NFL would be hard to do. I've done some research into different sports and player value and from what I've seen ranking football players is harder than the other sports. 

[Shanedorf]

2 things:

1) All models are wrong

2) Some of them are useful

[GSUeagles14]

17 hours ago, ninjapirate said:

nfl game sample sizes are too limited and players aren't individuals so a war stat for the nfl will be nothing short of useless.

couldnt say it any better. 16 games isnt really big enough to draw a conclusion and even if it was every player is son dependent on multiple other players its an absolute waste to try and come up with a "war" stat for football.

[Broncofan]

21 hours ago, ninjapirate said:

nfl game sample sizes are too limited and players aren't individuals so a war stat for the nfl will be nothing short of useless.

And in baseball, you can break down a defensive play to the players in question (range, making the catch/throw, error, etc.).  And they fare against their peers.   You can break down an at-bat, the result, and you can even break down baserunning.   You can clearly evaluate the pitcher.  Pretty much all plays can be broken down to individual responsibilities, other than a catcher's gamecalling (but you can even break down how well they frame the ball lol).

Football WAR is limited to those positions because those are the only areas the same methodology can be applied to.   But it still doesn't cover for missed reads, slow reads, WR's in the wrong spot, or taking the wrong option route, or a RB hitting the wrong hole.   At least it doesn't have a transparent way of doing it.   And then you get into how each part of the team functions.   

Great idea, but wrong sport.   If anything, this could be applied way easier to hockey or baseball, with only 5 (or 6 if you include the goalie) players on each team on the playing field/ice.