Re: A new ranking systen for college hockey
Thanks to all. I'm going to add more to the paper about nonuniformity and controlling for PP and ENG, so I won't add much here. In short, I don't really think it's necessary, but it's worth a look. And thanks for the suggestion of using aggregate PP and PK, FS23. That won't really work for technical reasons, but it helped clarify in my head how to describe the issue.
Slurpees: Any statistics-based model assumes that the model applies for the whole dataset, or explicitly invokes some changing parameter over time. There is no reasonable way in this sort of model to account for injuries, personnel changes, or any of that stuff. It essentially just assumes that this year's history is who you are. Of course, so does every other system we're discussing: KRACH and PWR, for example, though human polls can take account of anything they choose. Obviously, a team that sustains injuries to critical players won't be as good as the power rating under this sort of system indicates. And there's no real way in this sort of model to figure out how much worse. To do that, you'd need a model that worked at the player level, not the team level. There are some diagnostics you could use to see if a team is underperforming relative to the way they performed a month ago, but I don't think that's an ideal use for this kind of model, because humans will spot patterns that are really just random occurrences.
JF_Gophers: First, I can state that the methodology has not been tweaked or adjusted in any way to get a result. And the methodology is so simple (in concept -- to get practical numbers you need a pretty good computer and expensive software) there's really not any scope for doing so. Second, this methodology, without knowing the score of a single game (it only knows what one team scored and who their opponent is) managed to rank the teams in such a way that 13 of the 15 non-playin teams in the tournament were in the top 15. So PWR depends on quality wins and wins and head-to-head comparisons, but it gets almost the same results as a method which doesn't know the complete score of any game. For those who haven't looked at the paper, here are the top 15 teams in rank order:
1. North Dakota
2. Miami
3. Yale
4. Boston College
5. Michigan
6. Nebraska-Omaha
7. Union
8. Notre Dame
9. Denver
10. Wisconsin
11. New Hampshire
12. Minnesota Duluth
13. Merrimack
14. Western Michigan
15. St. Cloud State
For a method that doesn't look at the winner of a single game, that's a pretty good list, IMO, other than Wisconsin, for whom I'm going to add a section in the paper. Other than Air Force, the two teams (CC and RPI) who made it in in lieu of SCS and Wisconsin, are ranked 17th and 20th respectively. Pretty close. And note that this methodology got all four of the top seeds, albeit in a different order.
Finally, Red Cows: I'm in complete agreement, and the Bill James article you cited was an important article to me way back in 1982 when it was published. For those who don't go back that far, that was the year an Atlanta Braves team who wasn't very good the year before rattled off a dozen wins or so to start the season. As that article pointed out (and is fully in the spirit of this methodology) any team can go on a hot streak. To figure out if they're any good, you need to look at whether they're killing people or squeaking by to make an informed judgment. That Braves team, to many people's surprise, ended up winning the National League West that year and the James article was an attempt to retrospectively figure out whether or not we should have figured that out at the time. Thanks for reminding me of it.
