Patman
Rodent of Unusual Size
Alright, I thought we had played a game
USCHO Fan Forum
This is a sample guest message. Register a free account today to become a member! Once signed in, you'll be able to participate on this site by adding your own topics and posts, as well as connect with other members through your own private inbox!
-
The USCHO Fan Forum has migrated to a new plaform, xenForo. Most of the function of the forum should work in familiar ways. Please note that you can switch between light and dark modes by clicking on the gear icon in the upper right of the main menu bar. We are hoping that this new platform will prove to be faster and more reliable. Please feel free to explore its features.
Pairwise Analysts, Please Come Help Us 2013
- Thread starter Numbers
- Start date
Alright, I thought we had played a game
Re: Pairwise Analysts, Please Come Help Us 2013
If there's interest in any particular output/results, or even some new type of interactivity so you can explore the data yourself, let me know and I'll see what I can do. I'll probably get my first post of the season up this week.
Basically, there are three kinds of variably scheduled games in college hockey:
* Best of three series
* Tournaments where games are played by winners/losers of other games
* Conference tournaments. After the first round, they all fall into the 2nd bullet above. In the play-in rounds they depend on standings and may involve ranking and reordering the bracket. Each is different, which is a little annoying; but the real killer is the CCHA which uses shootout results that I don't have in my games database.
I can simulate everything except across the conference tournament break. So, for now I simulate up to the end of the regular season, then once the tournaments are seeded I can simulate to the end of the conference tournaments.
If there's interest in any particular output/results, or even some new type of interactivity so you can explore the data yourself, let me know and I'll see what I can do. I'll probably get my first post of the season up this week.
Basically, there are three kinds of variably scheduled games in college hockey:
* Best of three series
* Tournaments where games are played by winners/losers of other games
* Conference tournaments. After the first round, they all fall into the 2nd bullet above. In the play-in rounds they depend on standings and may involve ranking and reordering the bracket. Each is different, which is a little annoying; but the real killer is the CCHA which uses shootout results that I don't have in my games database.
Last edited:
Re: Pairwise Analysts, Please Come Help Us 2013
Burgie12: You're right about Yale and its TUC games, but you left out the games that have been already played that might become TUC games again if Holy Cross, UMass and Colorado College creep back in... they're even closer than RPI and Princeton. That's what I think makes the PWR interesting, if maddening to analyze: the fact that Yale's chances might well depend on which of Colorado College and Holy Cross has a better season from here on out. One is very good for the Elis and one is very bad. BUT... you don't want any team to sink TOO far below the TUC line or it begins eating into your RPI, whether you beat them or lost to them. In an ideal world, every team you lost to should have an RPI of .4999 to maximize your chances of getting in.
Burgie12: You're right about Yale and its TUC games, but you left out the games that have been already played that might become TUC games again if Holy Cross, UMass and Colorado College creep back in... they're even closer than RPI and Princeton. That's what I think makes the PWR interesting, if maddening to analyze: the fact that Yale's chances might well depend on which of Colorado College and Holy Cross has a better season from here on out. One is very good for the Elis and one is very bad. BUT... you don't want any team to sink TOO far below the TUC line or it begins eating into your RPI, whether you beat them or lost to them. In an ideal world, every team you lost to should have an RPI of .4999 to maximize your chances of getting in.
Re: Pairwise Analysts, Please Come Help Us 2013
Thanks for your analysis, "just win baby!' helps in all cases as well.
FlagDUDE08
Banned
Re: Pairwise Analysts, Please Come Help Us 2013
True, Yale was better off losing in the ECAC tourney a couple of years ago to get in the dance, forgot about that one.
True, Yale was better off losing in the ECAC tourney a couple of years ago to get in the dance, forgot about that one.
Re: Pairwise Analysts, Please Come Help Us 2013
The only real statement that can be made for Yale this far out from the end of the season is that they'll need to perform well, especially against the firm TUCs (Dartmouth, Quinnipiac, etc.) and they should cheer for the teams that they played (UMass, BC, and especially DU and CC) to do well, too.
I was just trying to look at their schedule going ahead and how heavily back-loaded it is.
Even if Holy Cross does slip back into the TUCs, it's just one loss. In order for Yale to stay in the field, they'll have to keep their RPI high. To do that, they're going to need a lot of wins down the stretch against good-quality teams, which will take care of their TUC Record whether the Crusaders are above the cliff or not.
Within reason. There are always exceptions.
The only real statement that can be made for Yale this far out from the end of the season is that they'll need to perform well, especially against the firm TUCs (Dartmouth, Quinnipiac, etc.) and they should cheer for the teams that they played (UMass, BC, and especially DU and CC) to do well, too.
Hockey Dogg
Less vision than a HE official
Re: Pairwise Analysts, Please Come Help Us 2013
I could be wrong, but I think there is an error in the current standings on USCHO. BU has a 4-6 record vs TUC on the main PWR rankings page, but if you look at the individual comparisons it is showing a BU record of 4-5 vs. TUC and thus is not awarding a PWR point for TUC in all BU comparisons.
I could be wrong, but I think there is an error in the current standings on USCHO. BU has a 4-6 record vs TUC on the main PWR rankings page, but if you look at the individual comparisons it is showing a BU record of 4-5 vs. TUC and thus is not awarding a PWR point for TUC in all BU comparisons.
I wonder if that 4-5 record is in a comparison with a TUC they have played. In that case, that game does not count on the TUC record.
Re: Pairwise Analysts, Please Come Help Us 2013
Right, H2H games are not also included in their TUC record. So, when looking at the DU / BU comparison, they will only have a 4-5-0 record. And, in the BC / BU comparison, it is 3-4-0, but against Miami (for example) it does come into play.
Patman
Rodent of Unusual Size
Jim, lets talk after I get something written up... Figuring out the playoffs is mostly a game of efficiently programming tie-breaking rules.
I am not a computer scientist nor a true programmer (and I really get sick of needing to be a jack of all trades.)
I think if the work can be parceled in some way then something useful can be done.
Re: Pairwise Analysts, Please Come Help Us 2013
Jim and Pat,
Thanks for all you fellows bring to discussion like this. Can I ask a few questions? I seem to remember from some discussion last year that it is impossible to do a full-up "odds" considering every possibility, because the number of games is too many, and the PWR would have to be calculated at the end of each combination of games.
So, if that's right, it seems to me that Jim does what is called a series of "Monte Carlo" runs. Now, if I understand that right, that means he uses some kind of comparison between each team that gives odds of winning each game (KRACH would work nice here), and uses some random number generator or other programming tool to 'pick' the winner of each game, in accord with those odds. Do that for every game, add up the PWR at the end. Then repeat. Do it a bunch of times (1000? I think), and you can say the "odds" of UND getting a #1 seed are (insert number here in %). Does that seem right?
If so, that is a nice tool to have available to those of us curious about this matter. Because, those odds will fill the rest of the games. And, in the end, we will see some things changing from the present PWR that we might not guess at ahead of time.
For example, I suspect that Lowell will come out lower than they are right now, for reasons discussed above. BU perhaps, too. So, that "Monte Carlo" odds calculator will help us learn who is in a more advantaged situation, and who in a less advantaged situation, that we can see just from the PWR numbers now (I mean the # of comparisons won).
Thanks again fellows. My natural interest in these things is why I named myself "Numbers."
Jim and Pat,
Thanks for all you fellows bring to discussion like this. Can I ask a few questions? I seem to remember from some discussion last year that it is impossible to do a full-up "odds" considering every possibility, because the number of games is too many, and the PWR would have to be calculated at the end of each combination of games.
So, if that's right, it seems to me that Jim does what is called a series of "Monte Carlo" runs. Now, if I understand that right, that means he uses some kind of comparison between each team that gives odds of winning each game (KRACH would work nice here), and uses some random number generator or other programming tool to 'pick' the winner of each game, in accord with those odds. Do that for every game, add up the PWR at the end. Then repeat. Do it a bunch of times (1000? I think), and you can say the "odds" of UND getting a #1 seed are (insert number here in %). Does that seem right?
If so, that is a nice tool to have available to those of us curious about this matter. Because, those odds will fill the rest of the games. And, in the end, we will see some things changing from the present PWR that we might not guess at ahead of time.
For example, I suspect that Lowell will come out lower than they are right now, for reasons discussed above. BU perhaps, too. So, that "Monte Carlo" odds calculator will help us learn who is in a more advantaged situation, and who in a less advantaged situation, that we can see just from the PWR numbers now (I mean the # of comparisons won).
Thanks again fellows. My natural interest in these things is why I named myself "Numbers."
Re: Pairwise Analysts, Please Come Help Us 2013
Yeah, that's essentially what I do. The products I currently produce from the monte carlos are area charts of the probabilities of each PWR ranking for each team based on how many games that team wins. e.g. this time of year, area charts like these; then as the tournament approaches, full tables of the remaining possible outcomes.
Yeah, that's essentially what I do. The products I currently produce from the monte carlos are area charts of the probabilities of each PWR ranking for each team based on how many games that team wins. e.g. this time of year, area charts like these; then as the tournament approaches, full tables of the remaining possible outcomes.
Last edited:
Re: Pairwise Analysts, Please Come Help Us 2013
A Monte Carlo simulation of the remainder of the season (with PWR calculations at the end of each simulated season) is fairly simple to do, but there are, it seems to me, two big problems (and one small) with it. First, KRACH doesn't predict ties, or at least I don't see any simple way to get it to do so, and ties are a nontrivial component of PWR. In addition, the KRACH predictions should at least be modified for home ice, although that's fairly simple to do as long as you have a good way to make the estimate. Second, we would be fixing KRACH at today's level which really introduces a whole new set of uncertainties. It makes no sense (theoretically or in terms of computer effort) to dynamically update KRACH for pseudodata, but you will find yourself in the position of making predictions that you'd never make in real life, where teams with great records (in the pseudodata) are getting thumped by lesser teams because of lucky runs. This problem is in some ways philosophical rather than practical, but it's a big one.
One other fairly sizeable problem is programming the playoff rules in every conference, which is really a pain.
A Monte Carlo simulation of the remainder of the season (with PWR calculations at the end of each simulated season) is fairly simple to do, but there are, it seems to me, two big problems (and one small) with it. First, KRACH doesn't predict ties, or at least I don't see any simple way to get it to do so, and ties are a nontrivial component of PWR. In addition, the KRACH predictions should at least be modified for home ice, although that's fairly simple to do as long as you have a good way to make the estimate. Second, we would be fixing KRACH at today's level which really introduces a whole new set of uncertainties. It makes no sense (theoretically or in terms of computer effort) to dynamically update KRACH for pseudodata, but you will find yourself in the position of making predictions that you'd never make in real life, where teams with great records (in the pseudodata) are getting thumped by lesser teams because of lucky runs. This problem is in some ways philosophical rather than practical, but it's a big one.
One other fairly sizeable problem is programming the playoff rules in every conference, which is really a pain.
Last edited:
Re: Pairwise Analysts, Please Come Help Us 2013
GoBlue,
I totally understand what you are saying. I think there are a couple of things to mention here.
1) What is the goal? Is it to get the 'best' possible prediction of the rest of the season? Some might have interest in that. If that is what we want, then, yes, all the things you write about are very valid concerns.
2) However, my interest is not in 'predicting.' Past years have left me the impression that predicting is impossible. Someone will go on a run of Wins or Losses that we don't anticipate. One year recently, the #1 team as of the beginning of January failed to even make the field. So, my interest is more like this:
2a) The PWR is really basically RPI, with a few tweaks thrown in. Let's predict that everyone's RPI basically stays the same. Then, what effect will the TUC records and the ComOpp records have on individual comparisons? Remember, if Team A has a better RPI than Team B, they can still lose the compare if they lose both TUC and ComOpp. Therefore, the TUC and ComOpp records are like 'hidden' features.
2b) In some cases (Qunnipiac is a good example right now), a team's 2nd half schedule does not have as much Schedule Strength as the 1st half. That means that their record will need to be better to keep their RPI up to the same level. Can we make note of that, too?
These are the kind of things I am interested in.
When it is Conference Tourney time, then it becomes really important. It still is impossible to make full up predictions, but it becomes possible at that point to analyze specific comparisons. And, some of those shed real light on the final field.
I hope all that is clear.
And, it is why I am interested in what Jim and Pat have in mind, because those Monte Carlo runs are basically like, "Let's play the rest of the season and keep everyone's RPI the same. Then, how does the PWR come out?"
GoBlue,
I totally understand what you are saying. I think there are a couple of things to mention here.
1) What is the goal? Is it to get the 'best' possible prediction of the rest of the season? Some might have interest in that. If that is what we want, then, yes, all the things you write about are very valid concerns.
2) However, my interest is not in 'predicting.' Past years have left me the impression that predicting is impossible. Someone will go on a run of Wins or Losses that we don't anticipate. One year recently, the #1 team as of the beginning of January failed to even make the field. So, my interest is more like this:
2a) The PWR is really basically RPI, with a few tweaks thrown in. Let's predict that everyone's RPI basically stays the same. Then, what effect will the TUC records and the ComOpp records have on individual comparisons? Remember, if Team A has a better RPI than Team B, they can still lose the compare if they lose both TUC and ComOpp. Therefore, the TUC and ComOpp records are like 'hidden' features.
2b) In some cases (Qunnipiac is a good example right now), a team's 2nd half schedule does not have as much Schedule Strength as the 1st half. That means that their record will need to be better to keep their RPI up to the same level. Can we make note of that, too?
These are the kind of things I am interested in.
When it is Conference Tourney time, then it becomes really important. It still is impossible to make full up predictions, but it becomes possible at that point to analyze specific comparisons. And, some of those shed real light on the final field.
I hope all that is clear.
And, it is why I am interested in what Jim and Pat have in mind, because those Monte Carlo runs are basically like, "Let's play the rest of the season and keep everyone's RPI the same. Then, how does the PWR come out?"
Last edited:
Re: Pairwise Analysts, Please Come Help Us 2013
For the home-ice advantage, the probability is shown by:
The log of the probability of each game is added and the KRACH and theta values are adjusted to maximize the sum (this can be done by Excel Solver or by someone with coding experience in a different language, such as R).
The problem I have with this method is it seems to apply the same home-ice advantage to each team, which is obviously false. I thought I had seen another paper discussing ways to introduce home-field advantage to Bradley-Terry, but I can't find it right now.
To include the possibility of ties in Bradley-Terry:
And, the same thing is done here, we take the log of the probability of the result of each game and adjust the KRACH and theta values to maximize their sum.
I have been calculating team's ratings based on the tie-adjusted Bradley-Terry system and here are their rankings, for comparison (Θ = 1.43):
Using these adjusted ratings, a team has a 17.6% chance of tying itself, 15.8% against a team with a KRACH two times its own, 13.1% against 3x's, etc. For example, a UMass v UAA game would no longer have an (approximately) 66.7 / 33.3% win split. It would now have a 57.9 / 26.3 / 15.8% W/L/T split for UMass.
Surprisingly, the estimated tie rate is actually pretty accurate. There have been 83 ties so far this season and using the theta of 1.43, if you add up the probability of a tie in every game that has been played so far, the total is 83.30.
I know that was way more math intensive than most people were looking for, but hopefully it was useful to some.
There are adaptations of Bradley-Terry (which is known by college hockey fans as KRACH) that account for ties or home-ice advantage. I haven't seen them used in conjunction, but it shouldn't be overly difficult to account for. There was a paper published by three Taiwanese professors that details the additional factors used for either adaptation (PDF).
For the home-ice advantage, the probability is shown by:
Code:
Θq1
-------- if T1 is home
Θq1 + q2
P(T1 beats T2) = {
q1
-------- if T2 is home
q1 + Θq2
q1 = B-T Ranking of T1
q2 = B-T Ranking of T2
Θ>0 = strength of home-field advantageThe problem I have with this method is it seems to apply the same home-ice advantage to each team, which is obviously false. I thought I had seen another paper discussing ways to introduce home-field advantage to Bradley-Terry, but I can't find it right now.
To include the possibility of ties in Bradley-Terry:
Code:
q1
P(T1 beats T2) = --------
q1 + Θq2
q2
P(T2 beats T1) = --------
Θq1 + q2
(Θ^2 - 1)(q1)(q2)
P(T1 ties T2) = --------------------
(q1 + Θq2)(Θq1 + q2)
q1 = B-T Ranking of T1
q2 = B-T Ranking of T2
Θ>1 = "threshold" parameter within which teams can be considered equal aka a tie occursI have been calculating team's ratings based on the tie-adjusted Bradley-Terry system and here are their rankings, for comparison (Θ = 1.43):
Code:
Boston College 600.23
New Hampshire 586.81
Quinnipiac 461.19
Minnesota 459.13
Notre Dame 417.67
Boston University 387.60
Denver 322.94
North Dakota 294.09
Dartmouth 256.29
Yale 252.97
Miami 219.21
Nebraska-Omaha 216.85
UMass Lowell 209.39
Western Michigan 208.80
Minnesota State 195.70
St. Cloud State 179.53
Cornell 153.99
Colgate 150.95
Wisconsin 144.62
Northern Michigan 141.57
Providence 133.96
Colorado College 131.42
Lake Superior 128.97
Robert Morris 122.56
Minnesota Duluth 120.11
Ohio State 120.06
Union 119.94
Niagara 117.32
Ferris State 108.67
Alaska 106.87
Massachusetts 103.18
Rensselaer 89.64
Merrimack 87.57
Michigan Tech 87.34
Princeton 83.70
Vermont 81.27
Bemidji State 79.79
Holy Cross 77.24
Michigan State 76.62
Harvard 76.16
Bowling Green 72.58
St. Lawrence 67.70
Brown 63.15
Michigan 57.40
Northeastern 56.93
Mercyhurst 53.39
Alaska Anchorage 52.53
Maine 49.83
Connecticut 44.90
Air Force 41.93
Clarkson 39.21
Canisius 38.93
Bentley 33.75
RIT 24.93
Army 24.91
Penn State 22.51
American International 12.25
Alabama-Huntsville 7.03
Sacred Heart 2.66Surprisingly, the estimated tie rate is actually pretty accurate. There have been 83 ties so far this season and using the theta of 1.43, if you add up the probability of a tie in every game that has been played so far, the total is 83.30.
Even if you are updating KRACH after every weekend or so with the pseudodata, it's still going to underpredict the upset runs that teams will go on while riding a hot goalie
Yes it is. Especially since the AHA doesn't really publish their info, HEA's guidelines could be interpreted in three different ways, and the CCHA uses shootouts.
I know that was way more math intensive than most people were looking for, but hopefully it was useful to some.
Last edited:
Re: Pairwise Analysts, Please Come Help Us 2013
burgie12: Nice work. As I said, the home ice multiplier is a fairly simple procedure. And the tie methodology is quite intuitive so long as the 1.43 is right. I'll need to think a minute or two about the best method of including both in a unified way, but that looks pretty easy.
numbers: if all you really want to know is whether RPI under- or overstates your chances, you mostly just need to look and see how the RPI column and PWR rating differ. For example, right now, they line up for the first 10 teams, and they line up for the first 14 except for a reversal of 11 and 12. That's not always the case by any means, but it shows how close RPI is to PWR, at least until every team has 10 TUC games. Overcoming an RPI disadvantage requires really both a COpp and TUC advantage in most cases where the teams haven't played head-to-head, or where the head-to-head results are inconclusive. While there is something to be said for BU's disadvantage going forward (which to date hasn't cost them a single spot, assuming their tie with NoDak is broken by RPI) it shows that even a massive TUC disadvantage will rarely cost you more than a place or two for a good RPI team. It can have a bigger effect for lower teams since the RPIs are much closer together -- but nobody cares because nobody other than conference winners with RPIs below about 14 or so are going to the dance anyway.
burgie12: Nice work. As I said, the home ice multiplier is a fairly simple procedure. And the tie methodology is quite intuitive so long as the 1.43 is right. I'll need to think a minute or two about the best method of including both in a unified way, but that looks pretty easy.
numbers: if all you really want to know is whether RPI under- or overstates your chances, you mostly just need to look and see how the RPI column and PWR rating differ. For example, right now, they line up for the first 10 teams, and they line up for the first 14 except for a reversal of 11 and 12. That's not always the case by any means, but it shows how close RPI is to PWR, at least until every team has 10 TUC games. Overcoming an RPI disadvantage requires really both a COpp and TUC advantage in most cases where the teams haven't played head-to-head, or where the head-to-head results are inconclusive. While there is something to be said for BU's disadvantage going forward (which to date hasn't cost them a single spot, assuming their tie with NoDak is broken by RPI) it shows that even a massive TUC disadvantage will rarely cost you more than a place or two for a good RPI team. It can have a bigger effect for lower teams since the RPIs are much closer together -- but nobody cares because nobody other than conference winners with RPIs below about 14 or so are going to the dance anyway.
Patman
Rodent of Unusual Size
Monte Carlo is a method that is employed when more direct means of estimation is untenable. As such, since the PWR is a function of an entire season the resulting non-MC estimator would be ridiculous and not worth the time. As such, Monte Carlo is an estimation strategy.
Personally, I think it could shed some light one what does and doesn't influence the procedure. Give me a few nights. I started on a doc late last night after a uconn alum event. Given the way I work I'll have quite a bit of detail but not nearly as much as if it were a technical spec. I want to get things out there.
Until then, there is some measure of reading from formula but there is a bit of tea-leaf reading... I'd like to take the tea leaves out, myself.
Patman
Rodent of Unusual Size
This topic in the stats community is a bit over bludgeoned. Rutter uses a Bayesian hierarchical model on the B-T formulation (or rather logistic regression form). One could choose other link functions... A former Mich stat student used normals, other links are available, I'd be curious about non-parametric links as the differences between link choice is somewhat esoteric (and I stumped a student in a phd defense on this despite her working on link fcns... It was her 2nd phd, I have no remorse.)
Aside from that you also have latent variable models where it envisions more of a "tug-of-war" with the tie being some zone in the middle (Albyn? Jones does this for soccer.). This is before you get to various score-based models.
Fact of the matter, options never end and I have a few ideas of my own I'd like to see. I am aware of one paper that used nonparametrics to address football prediction.
It doesn't really end.