Octonion Power Rankings

[Patman]

Re: Octonion Power Rankings

It predicted 11/15 (73.3%) of the NCAA D1 tournament outcomes correctly last year, fitting on only non-tournament games. Is that good or bad?

Its a small sample size. I've thought about yearly correlation but then you get into the trickery of time-series behavior. I'm a little leery of a mixed-effect model for this... and with the constraint my thoughts would be that it would be hard to calculate... while one can always recode the predictors with respect to constraint, that a "2" may appear in the data is a little leery.

Also, if you are using 58 separate random effects, I would be shocked if the computation wasn't difficult in some measure.... oh, wait... random effect within year and then moving value for each. I suppose but I'm not sure what's gained or if model strength actually borrows over. "Nested within NCAA divisions" so every team by year comes from the same pool for offense and then again for defense?

As the complexity increases the ease of modeling by standard software also increases. GLMM models can be nasty if you throw enough into the works.

[Hammer]

I am sorry. I had thought that Minny and Ferris were #1s. Interesting that they each defeated #1 seeds from their own conference...

No. Ferris beat #3 seed Denver, then #4 Cornell to get to Tampa, then #1 Union to get to the final.

[octonion]

Re: Octonion Power Rankings

Sure, one tournament is a small sample. What would be your metric for measuring relative predictive ability? Octonion is likely better than KRACH even now, as KRACH doesn't account for winning margins. That's why KRACH has Quinnipiac ranked over Minnesota.

[goblue78]

Re: Octonion Power Rankings

Relative predictability for any set of games is measured as sumof(f(prob(win)-win)), where the sum is taken over all out-of-sample games and f() is a penalty function which is always positive and, ideally punishes longshots, like f(x)=x^2. Thus, a "pure" PWR based metric that predicts the higher seed always wins (or a pure metric based on any ranking system that predicts the higher seed will always win, generates either f(1) or f(0) for every game. Your system, by contrast, will generate f(1-pr(win)) when you get it right and f(pr(win)) when you get it wrong. Thus, predicting a game as 50-50 will generate f(.5) no matter what the outcome. Nicely, the binomial nature of the probabilities also will tell you whether or not your predictive score is within the tolerance of the probabilities: for any game with true probability p, the standard error is sqrt(p(1-p)). Since games are independent, you can then form a compounded expected predicted accuracy over N games which rises as the square root of N. This will at least tell you something about whether or not you've overfitted the model.

One more thing: it's true that your model uses score and KRACH doesn't, but KRACH uses who won the game and your method doesn't. Unless the Poisson model is exactly right (and it isn't), it's up in the air which is more accurate for predicting the outcome of actual games. I tested this by resimulating the season using only the Poisson model and looking at team records: try it, you will find some surprises.

[Patman]

Sure, one tournament is a small sample. What would be your metric for measuring relative predictive ability? Octonion is likely better than KRACH even now, as KRACH doesn't account for winning margins. That's why KRACH has Quinnipiac ranked over Minnesota.

Thank you for not responding to substantive issues, as a journal referee I would not be pleased. Remember that in the future should you proceed.

[goblue78]

Re: Octonion Power Rankings

To make my previous point concrete, let's take your statement: "That's why KRACH has rated Minnesota over Quinnipiac." But the question is: which one is better, and by how much? KRACH currently has ratings of 162.619 for QPac and 153.051 for Minny. This means that the probability of a Quinnipiac win over Minnesota on neutral ice is 162.619/(153.051+162.619) or 51.5 percent. I'm not sure exactly how your Poisson model works, of course, but if works like a typical Poisson model with offensive factors of 1.536 and 1.916 for QPac and Minny, with defensive factors of .398 and .453, then on neutral ice we get a Poisson score of 1.536-.453= 1.083 for QPac's goals and 1.916-.398 = 1.518 for Minny, then we get a probability of a QPac victory of 37.7 percent (I have taken the liberty of breaking ties proportional to net strength, but that should be relatively robust.) OK. So that's a pretty big difference between two methods right? One has QPac with a 37.7 percent chance of winning head to head and the other method has 51.5 percent. But it's really hard to separate those two probabilties even if they played 10 times. using a likelihood ratio of 8 to signify strong evidence, 0 wins would be evidence that you had the better argument and 9 or 10 wins would suggest KRACH did, the remainder of the cases would not be very strong evidence one way or the other. Even using a weak evidence standard of a likelihood ratio of 4, 3-6 QPac wins would be inconclusive. And that's playing 10 times!

[Back2BackU-MnPride2002]

Re: Octonion Power Rankings

It predicted 11/15 (73.3%) of the NCAA D1 tournament outcomes correctly last year, fitting on only non-tournament games. Is that good or bad?

So, is this just something you are doing for fun this season, or is this something you are taking seriously? Can I expect to look up where my team ranks in the Octonion Ratings week to week next season and into the future?!

[octonion]

Re: Octonion Power Rankings

Hah - no, there's a time to respond to more substantial issues. Usually that's the weekends.

[octonion]

Re: Octonion Power Rankings

Hockey tends to be more weakly predictive than other sports, but you can pool the predictions and results over a large number of teams and years to compare the relative accuracy of different methods. You can still be wrong, of course, as team strengths aren't even constant within the same game. But is it a useful approximation?

[octonion]

Re: Octonion Power Rankings

I'm a professional sports analyst, but I'm an amateur at ice hockey analytics (so just for fun). I need to enhance it with some type of informative prior distribution to handle early-season games to make it more useful.

[Back2BackU-MnPride2002]

Re: Octonion Power Rankings

I like your rating system better than the PWR and RPI for sure. And from what I can tell it seems better than the KRACH as well. But for now I still favor Massey's Rating system. Your's mirrors his in many ways, but there are still some notable differences.


Your system seems to really not like MA-Lowell, Yale & BU. But really likes AFA, Ferris St, Mich Tech and Union?

Be interesting to figure out why that is?!


Massey's biggest differences with your system is where he has Miami(#7) & Union(#20).

[Back2BackU-MnPride2002]

Re: Octonion Power Rankings

I'm a professional sports analyst, but I'm an amateur at ice hockey analytics (so just for fun). I need to enhance it with some type of informative prior distribution to handle early-season games to make it more useful.

So how do you get a job as a professional sports analyst?! lol Do you need a degree from MIT or something like that?

[octonion]

Re: Octonion Power Rankings

My code is up on GitHub and publicly accessible (at least for now). I'll likely make it private in another week or so.

https://github.com/octonion/hockey

[Back2BackU-MnPride2002]

Re: Octonion Power Rankings

Poisson model, pooled NCAA divisions, home/away/neutral factors.

div = NCAA division
str = team strength
ofs = offensive strength
dfs = defensive strength
sos = strength of schedule

  rk |             school              | div |  str  |  ofs  |  dfs  |  sos  
-----+---------------------------------+-----+-------+-------+-------+-------
   1 | minnesota                       |   1 | 4.230 | 1.916 | 0.453 | 1.489
   2 | quinnipiac                      |   1 | 3.858 | 1.536 | 0.398 | 1.460
   3 | miami                           |   1 | 3.469 | 1.402 | 0.404 | 1.458
   4 | minnesota-state                 |   1 | 3.443 | 1.753 | 0.509 | 1.502
   5 | north-dakota                    |   1 | 3.321 | 1.727 | 0.520 | 1.526
   6 | denver                          |   1 | 3.259 | 1.837 | 0.564 | 1.548
   7 | st-cloud-state                  |   1 | 3.216 | 1.720 | 0.535 | 1.499
   8 | new-hampshire                   |   1 | 3.173 | 1.687 | 0.532 | 1.470
   9 | umass-lowell                    |   1 | 3.113 | 1.565 | 0.503 | 1.428
  10 | notre-dame                      |   1 | 3.084 | 1.605 | 0.521 | 1.482
  11 | union                           |   1 | 3.012 | 1.570 | 0.521 | 1.448
  12 | rensselaer                      |   1 | 2.931 | 1.551 | 0.529 | 1.496
  13 | wisconsin                       |   1 | 2.868 | 1.369 | 0.477 | 1.496
  14 | providence                      |   1 | 2.863 | 1.544 | 0.539 | 1.482
  15 | boston-college                  |   1 | 2.808 | 1.761 | 0.627 | 1.437
  16 | western-michigan                |   1 | 2.786 | 1.295 | 0.465 | 1.459
  17 | nebraska-omaha                  |   1 | 2.679 | 1.760 | 0.657 | 1.495
  18 | air-force                       |   1 | 2.673 | 1.534 | 0.574 | 1.349
  19 | ferris-state                    |   1 | 2.625 | 1.463 | 0.558 | 1.475
  20 | dartmouth                       |   1 | 2.618 | 1.526 | 0.583 | 1.462
  21 | cornell                         |   1 | 2.541 | 1.377 | 0.542 | 1.502
  22 | yale                            |   1 | 2.527 | 1.576 | 0.624 | 1.486
  23 | niagara                         |   1 | 2.508 | 1.494 | 0.596 | 1.354
  24 | colorado-college                |   1 | 2.507 | 1.798 | 0.717 | 1.567
  25 | michigan-tech                   |   1 | 2.493 | 1.611 | 0.646 | 1.516

When do you update your standings?

[goblue78]

Re: Octonion Power Rankings

I'd be interested in an update as well, just to see how much a single 10-0 game improves Quinnipiac's numbers.