Alternate title: Tim Tebow broke WAR in 2012 (with help from Tyrod Taylor and Terrelle Pryor, all of whom are in the replacement QB set in 2012). Replacement-level rush_iPA by season starting in 2011: .22, .50 (!!!), -0.01, -.31, -.16.
Example of the problem (with thanks for making all of this publicly available, otherwise I'd never be able to look at it this closely): Russell Wilson had rush_iPA of 0.49 in 2012 (82 rushes+32 sacks) and rush_iPA of 0.20 in 2015 (87 rushes+46 sacks). But because the replacement-level for rushing varies so much, he had -1.2 rush_iPAR in 2012 and 47.2 rush_iPAR in 2015, despite being a better rusher in 2012.
The unreasonably high value of replacement-level rush_iPA in 2012 tanks everyone's WAR, and no QB even reaches 2 WAR. WAR tells us that in 2012, Doug Martin created more wins than Russell Wilson, Ben Roethlisberger, or Tony Romo, and that Drew Brees was below replacement-level. The current calculation of replacement-level QBs creates two problems: (1) it is impossible to compare WAR across seasons because replacement-level changes too much from season-to-season and (2) comparisons of QBs to other positions within a given season depend too much on the replacement-level QB set.
As I've said on twitter, I don't think it makes sense to calculate replacement-level rushing and passing separately. One problem this creates is that it gives rushing EPA and passing EPA different inherent values, which seems intuitively wrong (when a QB chooses whether to throw or scramble, the result of the play counts the same in real life regardless of the choice he made). Another problem is that it implicitly assumes that if, say, Cam Newton were replaced by a replacement-level QB, that QB would rush the ball 121 likes Newton did in 2017.
A possible solution is to come up with one value for replacement-level iPA for QBs and calculate everything relative to that. I'd also consider averaging replacement-level over several years (with more recent years carrying more weight) to get rid of the noise.
Alternate title: Tim Tebow broke WAR in 2012 (with help from Tyrod Taylor and Terrelle Pryor, all of whom are in the replacement QB set in 2012). Replacement-level rush_iPA by season starting in 2011: .22, .50 (!!!), -0.01, -.31, -.16.
Example of the problem (with thanks for making all of this publicly available, otherwise I'd never be able to look at it this closely): Russell Wilson had rush_iPA of 0.49 in 2012 (82 rushes+32 sacks) and rush_iPA of 0.20 in 2015 (87 rushes+46 sacks). But because the replacement-level for rushing varies so much, he had -1.2 rush_iPAR in 2012 and 47.2 rush_iPAR in 2015, despite being a better rusher in 2012.
The unreasonably high value of replacement-level rush_iPA in 2012 tanks everyone's WAR, and no QB even reaches 2 WAR. WAR tells us that in 2012, Doug Martin created more wins than Russell Wilson, Ben Roethlisberger, or Tony Romo, and that Drew Brees was below replacement-level. The current calculation of replacement-level QBs creates two problems: (1) it is impossible to compare WAR across seasons because replacement-level changes too much from season-to-season and (2) comparisons of QBs to other positions within a given season depend too much on the replacement-level QB set.
As I've said on twitter, I don't think it makes sense to calculate replacement-level rushing and passing separately. One problem this creates is that it gives rushing EPA and passing EPA different inherent values, which seems intuitively wrong (when a QB chooses whether to throw or scramble, the result of the play counts the same in real life regardless of the choice he made). Another problem is that it implicitly assumes that if, say, Cam Newton were replaced by a replacement-level QB, that QB would rush the ball 121 likes Newton did in 2017.
A possible solution is to come up with one value for replacement-level iPA for QBs and calculate everything relative to that. I'd also consider averaging replacement-level over several years (with more recent years carrying more weight) to get rid of the noise.