Pairwise Analysts, Please Come Help Us 2013

[goblue78]

Re: Pairwise Analysts, Please Come Help Us 2013

Numbers: Maybe this an example of the sort of thing you're thinking about. Let me know if I'm on the right track. Look at Yale-NoDak. Currently, NoDak leads in RPI (by a smidge) and also has an edge in Common Opponents. Neither has 10 TUC games. This gives NoDak a 2-0 edge. But the Common Opponent edge will be determined by the two games left between NoDak and DU. Yale's number is done at 2 (1-0 vs DU, 1-0 vs. CC, 0-1 vs HC). NoDak has two more games with DU left. Their final ComOpp rating will be (2-0 vs. HC [1], 2-2 vs CC [0.5], plus the rating achieved from their current 1-0-1 with DU combined with the next two games) But unless they lose both of them, or tie one and lose one, they still have the common opponent matchup won. The important thing is that if you give them two DU losses, though, holding everything else constant (which is just for discussion's sake) they'd probably fall behind Yale in RPI anyway. And clearly the TUC comparison could go either way since both teams have a bunch of TUC games left, and both are around the .500 TUC level. So even though there are possibilties here, I suspect that RPI will win out in the end.

(I am reminded that this becomes more difficult by the fact that NoDak could face CC and/or DU again in the playoffs.) This makes it even more complicated.

 

[LynahFan]

Re: Pairwise Analysts, Please Come Help Us 2013

Surprisingly, the estimated tie rate is actually pretty accurate. There have been 83 ties so far this season and using the theta of 1.43, if you add up the probability of a tie in every game that has been played so far, the total is 83.30.

Why is this surprising? Didn't you "fit" the 1.43 parameter based on the fact that there have been, in fact, 83 ties? If you let your iteration run longer (to get more decimals on the 1.43), I would expect the prediction of the number of ties to be *exactly* 83.0 ties.

 

[FlagDUDE08]

Re: Pairwise Analysts, Please Come Help Us 2013

So even though there are possibilties here, I suspect that RPI will win out in the end.

I didn't know we had that good of a shot...

 

[burgie12]

Re: Pairwise Analysts, Please Come Help Us 2013

burgie12: Nice work. As I said, the home ice multiplier is a fairly simple procedure. And the tie methodology is quite intuitive so long as the 1.43 is right.

1.43 is right for this season, but it isn't a constant value. At the end of last season, it was 1.36. If you were to calculate the B-T Ratings for the EPL, it's 2.15.

numbers: if all you really want to know is whether RPI under- or overstates your chances, you mostly just need to look and see how the RPI column and PWR rating differ. For example, right now, they line up for the first 10 teams, and they line up for the first 14 except for a reversal of 11 and 12.

That's only because there are so few teams that have played ten games against TUCs. That's a function of how early in the season it is, not the fact that RPI is actually accurate compared to the PWR. Yes, I know you know that, I'm just making sure that it's stated plainly.

Overcoming an RPI disadvantage requires really both a COpp and TUC advantage in <s>most cases</s> every case where the teams haven't played head-to-head, or where the head-to-head results are inconclusive. While there is something to be said for BU's disadvantage going forward (which to date hasn't cost them a single spot, assuming their tie with NoDak is broken by RPI) because Dartmouth and North Dakota haven't played enough TUC games, not because their RPI is holding them up. As I said above, they will drop two comparison wins just by playing out the rest of the regular season and could easily drop four. <s>it shows that even a massive TUC disadvantage will rarely cost you more than a place or two for a good RPI team.</s>

The rest of my response is in-line with the rest of your post.

This topic in the stats community is a bit over bludgeoned. Rutter uses a Bayesian hierarchical model on the B-T formulation (or rather logistic regression form). One could choose other link functions... A former Mich stat student used normals, other links are available, I'd be curious about non-parametric links as the differences between link choice is somewhat esoteric (and I stumped a student in a phd defense on this despite her working on link fcns... It was her 2nd phd, I have no remorse.)

Aside from that you also have latent variable models where it envisions more of a "tug-of-war" with the tie being some zone in the middle (Albyn? Jones does this for soccer.). This is before you get to various score-based models.

Fact of the matter, options never end and I have a few ideas of my own I'd like to see. I am aware of one paper that used nonparametrics to address football prediction.

It doesn't really end.

Not being a mathemtician... *woosh* Rutter uses the properties of a normal distribution to calculate his model. I like it, especially since it can provide a definitive difference between teams at opposite ends of the spectrum without pushing ratings into the thousands, but its ability to predict ties is a bit off.

 

[burgie12]

Re: Pairwise Analysts, Please Come Help Us 2013

Why is this surprising? Didn't you "fit" the 1.43 parameter based on the fact that there have been, in fact, 83 ties? If you let your iteration run longer (to get more decimals on the 1.43), I would expect the prediction of the number of ties to be *exactly* 83.0 ties.

It's out to as many decimal places as Excel Solver will allow it (1.42611606730253 to be exact). Solver is maximizing the sum of the probabilities of the games (well, technically the logs of these probabilities) having occurred as they did given the ratings that the teams end up with. When I set Sum(P(ties)) to be equal to the number of ties as a constraint, it freaks out, saying that the solution is unfeasible. And, I'm surprised, because it was closer in its prediction in the number of ties than the Mease / Rutter Rankings (83.32 predicted), which I thought would be much more accurate in keeping the ties constraint reasonable.

ETA: Well, that and the fact that when I was originally calculating the B-T total number of predicted ties, I was doing it wrong and I was off by 15%. So, when I actually calculated it properly, to see it come so close to accurate, I was a bit surprised.

 

[Umileated]

Re: Pairwise Analysts, Please Come Help Us 2013

I believe what has been used by some is to use KRACH ratings to figure out the odds of teams winning games and proactively adjust the PWR to make a prediction as to who has the best odds of making it to the tournament. I wonder how accurate KRACH is in predicting the winner of games based upon various ranges of chance of winning (50%-60%, 60%-70%, etc.)?

I payed around with the CHODR quite a bit last year. I only came up with a non-iterative excel sheet that tends to be "close enough" around this time of year. The guy who came up with it keeps track of individual game predicitions.

http://it.stlawu.edu/~chodr/results.html

Says here that he was 68% on the week, and is 62% on the season. If you look further into the predictions, I imagine a betting man would have hedged better than 68% on these predictions

 

[Umileated]

Re: Pairwise Analysts, Please Come Help Us 2013

I really ought to get to work... Don't be surprised if a PDF shows in my signature in line w the grander idea that I always seem to talk about.

Interested.

 

[Numbers]

Re: Pairwise Analysts, Please Come Help Us 2013

Numbers: Maybe this an example of the sort of thing you're thinking about. Let me know if I'm on the right track. Look at Yale-NoDak. Currently, NoDak leads in RPI (by a smidge) and also has an edge in Common Opponents. Neither has 10 TUC games. This gives NoDak a 2-0 edge. But the Common Opponent edge will be determined by the two games left between NoDak and DU. Yale's number is done at 2 (1-0 vs DU, 1-0 vs. CC, 0-1 vs HC). NoDak has two more games with DU left. Their final ComOpp rating will be (2-0 vs. HC [1], 2-2 vs CC [0.5], plus the rating achieved from their current 1-0-1 with DU combined with the next two games) But unless they lose both of them, or tie one and lose one, they still have the common opponent matchup won. The important thing is that if you give them two DU losses, though, holding everything else constant (which is just for discussion's sake) they'd probably fall behind Yale in RPI anyway. And clearly the TUC comparison could go either way since both teams have a bunch of TUC games left, and both are around the .500 TUC level. So even though there are possibilties here, I suspect that RPI will win out in the end.

(I am reminded that this becomes more difficult by the fact that NoDak could face CC and/or DU again in the playoffs.) This makes it even more complicated.

Without commenting directly on this analysis:
That's exactly what I am thinking about. The idea is that, for example, right now, Lowell looks pretty strong. #13 in the PWR, and their RPI is close to the #12 team. And, they have a nice little gap in RPI to the next group of teams. You would say, "Strong on the bubble, if the season plays out as it has been going..." But, their comparison wins with the WCHA teams are all really iffy, when the TUC record comes into play.

I like all the math on this site. What Pat and Jim are doing, as far as some kind of prediction, is clever, and interesting (How do they do that???). For myself, I keep coming back to "What happens in all the individual comparisons?" Because that is actually where the rubber meets the road.

I am reminded of a few years ago. I think 2 years ago. Minny had a really down year. It came to WCHA playoff time, and Anchorage came to Minneapolis. At the start of the week, this site, and also CHN were giving Minny almost no chance to make the field. When you looked really close at all the individual compares, however, all they needed was to not be swept, and things would come out fine, because of the brackets in all the other conferences. As it happened, Anchorage swept them (at Mariucci, no less - it was really bad), and so they didn't qualify. But, it showed me that there is lots of analysis available that is lacking, because it doesn't look close enough at the individual compares.

 

[Patman]

Re: Pairwise Analysts, Please Come Help Us 2013

Well, with me I'm interested in cross-tabs... Such as how many loss equivalents can a team absorb.

The reality is, a lot of this is computation intensive and not easy. I tried coding up pair wise in R once... Wasn't fun nor quick but I'm sure I could do it better now.

 

[goblue78]

Re: Pairwise Analysts, Please Come Help Us 2013

Well, numbers, if this is the way you're thinking, it doesn't take much math at all to get there. You can look at any of these comparisons yourself in fairly quick order (on the pairwise grid layout -- just click on any pairwise comparison). Do that and look at the remaining schedules (and make some guesses about the playoffs and anybody can play along.) I just don't think there's any way to that systematically without predicting the outcome of just about every game going forward. (And I mean predicting in the statistical sense of a probabilistic prediction.)

 

[jeteye1717]

If all you really want to know is whether RPI under- or over-states your [team's] chances, you mostly just need to look and see how the RPI column and PWR rating differ. For example, right now, they line up for the first 10 teams, and they line up for the first 14 except for a reversal of 11 and 12. That's not always the case by any means, but it shows how close RPI is to PWR, at least until every team has 10 TUC games.

Spot on ... Right now, neither Minnesota nor NoDak has played the 10 requisite TUC games (nor H2H), meaning their respective PWR rank is being driven exclusively by RPI (since even if they lose a COp point in any comparison, any resultant tie will then be broken by RPI). But that all changes this weekend, when the Team-Formerly-Known-as-the-Fighting-Sioux faces off with the Gophers at Mariucci. Absent some team re-ordering at the TUC Cliff, come late Saturday night both teams will see their TUC records jump to 10+, meaning the TUC component of the PWR ranking will come into play for both teams for the first time all season. With 3/4 of the PWR criteria now being evaluated for each comparison (and 4/4 including H2H 'twixt Minny and NoDak), an elevated RPI won't be enough to overcome losses in both COp and TUC criteria for certain comparisons. It will be interesting to see how Minnesota's 8-0-0 NC record holds up when the TUC test gets underway. Will it be enough to make up their current deficit of 2 PWR points behind the UNH-BC-Q tie at the top? Time (and their play on the ice) will tell ...

While there is something to be said for BU's disadvantage going forward (which to date hasn't cost them a single spot, assuming their tie with NoDak is broken by RPI) it shows that even a massive TUC disadvantage will rarely cost you more than a place or two for a good RPI team. It can have a bigger effect for lower teams since the RPIs are much closer together -- but nobody cares because nobody other than conference winners with RPIs below about 14 or so are going to the dance anyway.

Both good points, but I'll note that even amongst the top teams it's not all that unusual for team RPIs to be quite close together, often separated by the equivalent of a given game ending in an OT tie vs. a win/loss. Just a single goal (if scored, it's a win/loss; if not, a tie) is often enough to flip several comparisons, meaning in/out for some teams on the proverbial 'bubble,' or a switch in seeding band if at the 4/5, 8/9, or 12/13 margin. That in turn potentially impacts opponent assignment, regional venue, last change (or not), etc. Yet, despite how some of us obsess over the minutia, Al Davis still said it best - 'Just Win, Baby' ...

 

[Priceless]

Re: Pairwise Analysts, Please Come Help Us 2013

Spot on ... Right now, neither Minnesota nor NoDak has played the 10 requisite TUC games (nor H2H), meaning their respective PWR rank is being driven exclusively by RPI (since even if they lose a COp point in any comparison, any resultant tie will then be broken by RPI). But that all changes this weekend, when the Team-Formerly-Known-as-the-Fighting-Sioux faces off with the Gophers at Mariucci. Absent some team re-ordering at the TUC Cliff, come late Saturday night both teams will see their TUC records jump to 10+, meaning the TUC component of the PWR ranking will come into play for both teams for the first time all season. With 3/4 of the PWR criteria now being evaluated for each comparison (and 4/4 including H2H 'twixt Minny and NoDak), an elevated RPI won't be enough to overcome losses in both COp and TUC criteria for certain comparisons. It will be interesting to see how Minnesota's 8-0-0 NC record holds up when the TUC test gets underway. Will it be enough to make up their current deficit of 2 PWR points behind the UNH-BC-Q tie at the top? Time (and their play on the ice) will tell ...

If slack.net were up (it appears the whole site is down, not just Whalen's page) it would be easy to see how it holds up by removing the 10 game TUC requirement (or lowering the threshold) or by predicting the results. Hopefully the site comes back soon.

 

[JimDahl]

Re: Pairwise Analysts, Please Come Help Us 2013

I like all the math on this site. What Pat and Jim are doing, as far as some kind of prediction, is clever, and interesting (How do they do that???). For myself, I keep coming back to "What happens in all the individual comparisons?" Because that is actually where the rubber meets the road.

The two are complementary. Pages like this -- BU PWR Details -- are what I made for myself for exactly the purpose of manually diving into the individual comparisons. It's been through a lot of revisions and, though admittedly dense, is pretty much the exact information I found myself compiling when trying to figure out what might happen to a team's PWR. If you think about it differently or have suggestions, I'd love to hear it.

The simulations tip you off to probabilities and moves that are likely under certain conditions. For example, wondering why the simulations showed BU in such a tenuous position vs. its peers, I dove into the BU PWR details page and realized that their horrible TUC was about to doom them just as their RPI likely peaked due to a high SOS.

But that all changes this weekend, when the Team-Formerly-Known-as-the-Fighting-Sioux faces off with the Gophers at Mariucci. Absent some team re-ordering at the TUC Cliff, come late Saturday night both teams will see their TUC records jump to 10+, meaning the TUC component of the PWR ranking will come into play for both teams for the first time all season. With 3/4 of the PWR criteria now being evaluated for each comparison (and 4/4 including H2H 'twixt Minny and NoDak), an elevated RPI won't be enough to overcome losses in both COp and TUC criteria for certain comparisons. It will be interesting to see how Minnesota's 8-0-0 NC record holds up when the TUC test gets underway. Will it be enough to make up their current deficit of 2 PWR points behind the UNH-BC-Q tie at the top? Time (and their play on the ice) will tell ...

That's one of the questions that the previously discussed monte carlo simulations are pretty good at answering. I do so here. (tldr -- Minnesota is fine, UND needs some points).

 

[LTsatch]

Re: Pairwise Analysts, Please Come Help Us 2013

'Just Win, Baby' ...

I said the same thing earlier, but an early conference tournament loss sometimes actually helps a team make the show. Sad but true. Even though my favorite team was a recipient of this "loophole" in the past.

 

[Patman]

Re: Pairwise Analysts, Please Come Help Us 2013

I said the same thing earlier, but an early conference tournament loss sometimes actually helps a team make the show. Sad but true. Even though my favorite team was a recipient of this "loophole" in the past.

Frankly, I prefer that to the alternative of not having a system... as regrettable as it may be.

 

[Numbers]

Re: Pairwise Analysts, Please Come Help Us 2013

The two are complementary. Pages like this -- BU PWR Details -- are what I made for myself for exactly the purpose of manually diving into the individual comparisons. It's been through a lot of revisions and, though admittedly dense, is pretty much the exact information I found myself compiling when trying to figure out what might happen to a team's PWR. If you think about it differently or have suggestions, I'd love to hear it.

The simulations tip you off to probabilities and moves that are likely under certain conditions. For example, wondering why the simulations showed BU in such a tenuous position vs. its peers, I dove into the BU PWR details page and realized that their horrible TUC was about to doom them just as their RPI likely peaked due to a high SOS.



That's one of the questions that the previously discussed monte carlo simulations are pretty good at answering. I do so here. (tldr -- Minnesota is fine, UND needs some points).

Jim,

Thanks for chiming in. I think that my interest or "focus" is similar to yours, but I don't have the needed computer background or resources to do all that programming myself. I call myself "Numbers" because numbers have always been my interest. But, in the days of my higher education, life took a major turn....

I like all the predictions and so on that you guys do, they are interesting to me. And, that PWR Details is the kind of thing that I think is the real 'meat and potatoes' of this matter. So, yeah, you've got what I am looking for. Thanks again. I think the way you put up the details is a little hard to read. In the specific case of BU, I agree totally with your post. They will need a higher RPI or win %age in the 2nd half to hang onto their present PWR position. That is really what we are looking for.

Also, thanks for quoting me on your blog. At least, I think. I think I am reading that you agree with me. The 8-0-0 that Minnesota has so far helps with their TUC record because of BC and NoDame (also note that Mich State, not presently a TUC) is climbing, and Minnesota has wins against them, too. And, that 8-0-0 record helps in compares with HE, CCHA teams. I think that you are agreeing, right? If not, please help me understand where I am wrong.

 

[JimDahl]

Re: Pairwise Analysts, Please Come Help Us 2013

In the specific case of BU, I agree totally with your post. They will need a higher RPI or win %age in the 2nd half to hang onto their present PWR position. That is really what we are looking for.

It's slightly worse for BU than that. Their RPI will actually fall unless they significantly pick up their winning percentage. Their RPI is currently at a bit of a high because of SOS. You'll probably find this as dense if not worse than the PWR details pages, but I also post the game-by-game calculation of RPI including the effects of future games, e.g. BU RPI details. (RPI is actually quite straightforward for this sort of analysis because it doesn't matter against whom you win or lose, just who you play and your record).

I like all the predictions and so on that you guys do, they are interesting to me. And, that PWR Details is the kind of thing that I think is the real 'meat and potatoes' of this matter. So, yeah, you've got what I am looking for. Thanks again. I think the way you put up the details is a little hard to read.

Well, that's precisely my question for you -- what would a guy like you like to see? I'm trying to use you for free market research A decade or so ago I would do this sort of PWR analysis manually by combing the USCHO tables, but that required a lot of clicks and transcribing. I managed to cut my time investments by an order of magnitude with the PWR details pages, but they're admittedly ridiculously dense. Realizing that most people just wanted information like "UND needs to win 75% of its remaining games to be safely in the tournament", I added the simulations. That clearly requires significant faith in the forecaster, so I also try to bridge the two with analysis like the post (e.g. the simulations reveal X, here are the PWR details of why that seems to be the case). It sounds like you think I'm missing a step in between, some sort of PWR detail light?

The 8-0-0 that Minnesota has so far helps with their TUC record because of BC and NoDame (also note that Mich State, not presently a TUC) is climbing, and Minnesota has wins against them, too. And, that 8-0-0 record helps in compares with HE, CCHA teams. I think that you are agreeing, right? If not, please help me understand where I am wrong

Yeah, I thought you were just commenting on COP so was trying to say that while that's important, in this case TUC also is. For out-of-conference comparisons you're usually looking at RPI, COP, and TUC. The fact that Minnesota still does quite well even if they finish out the season .500, and thus suffer a significant decline in RPI, means that their impressive TUC is also a very important contributor to their stability. In citing you I was mostly just trying to link into this thread, since it inspired some of the analysis in that post.

 

[Numbers]

Re: Pairwise Analysts, Please Come Help Us 2013

Well, that's precisely my question for you -- what would a guy like you like to see? I'm trying to use you for free market research A decade or so ago I would do this sort of PWR analysis manually by combing the USCHO tables, but that required a lot of clicks and transcribing. I managed to cut my time investments by an of magnitude with the PWR details pages, but they're admittedly ridiculously dense. Realizing that most people just wanted information like "UND needs to win 75% of its remaining games to be safely in the tournament", I added the simulations. That clearly requires significant faith in the forecaster, so I also try to bridge the two with analysis like the post (e.g. the simulations reveal X, here are the PWR details of why that seems to be the case). It sounds like you think I'm missing a step in between, some sort of PWR detail light?

Hey, This is fun. No one ever asks me what I would like to see.

I looked again at the BU PWR page.
1)I think what I would prefer would be if the order of the compares went according to the current PWR rankings. That's how we are accustomed to seeing them. And, it makes it a little easier to pick out the ones that might flip - because they are 'in the middle' or some other way of describing that.
2)I think I like the green/red. It makes it easy to pick out, for example, that BU loses its compare to Ak in spite of a higher RPI.
3)And, the one other thing might be a list of who the ComOpp are. I know there is only so much room, but who the ComOpp are comes in handy in conference tourney time, in case there is repeat of those games, that might affect the compare.

Oh, and late edit 4) How about also a graph of the probabilities for the final PWR in toto. You have "If Minny wins 14 - this is the range, and the probability....." I would like to see, "Right now, the probability is a% for #1 seed, b% for #2, etc....." In other words, add together all the curves on the chart, with respect to the various probability that "Minny wins 10 games." If that makes sense at all. I have a tendency to be wordy, and thus confusing.

 

[Patman]

Re: Pairwise Analysts, Please Come Help Us 2013

Alright, this is an extremely early revision and I will tinker with it as my life wants or allows. I've said before that I stopped working on ranking systems because I want to live my life. That still comes first... but life still has its down time.

There are a ton of holes in this document to be sure.

NCAA Men's Division 1 Simulation Primer

edit: to me the model is the easy part, I can build a good one in the span of an afternoon (maybe only an hour) in R. Its the rest of it that freaks me out.

Everything in a nutshell without the long write-up...

how to handle in-season tournaments given various OT formats
how to efficiently compute conference rankings and tie-breaking procedures
building up structures for conference tournament simulation
efficient computation of pairwise
cobble together statistics on...
--league ranking
--league points
--opponents by conference playoff round
--chance of winning the rounds
--break-out tables for league results remaining (if 9-0-0 then... blah)
--pairwise predictions
--seeding predictions
--seeding contingencies for #effective wins... #effective losses (an effective win being wins plus half ties)
--????

Research to be done on all non-model aspects. Discussion of model choices? Multiple model choices? Baseball prospectus at last check uses three different models for MLB prediction.

 

[burgie12]

Re: Pairwise Analysts, Please Come Help Us 2013

1)I think what I would prefer would be if the order of the compares went according to the current PWR rankings. That's how we are accustomed to seeing them. And, it makes it a little easier to pick out the ones that might flip - because they are 'in the middle' or some other way of describing that.

This would be my only request, but I understand why they're currently in alphabetical order.

Additionally, I love that the shade of the background changes depending on by how much the team is winning (or losing) the specific factor.

3)And, the one other thing might be a list of who the ComOpp are. I know there is only so much room, but who the ComOpp are comes in handy in conference tourney time, in case there is repeat of those games, that might affect the compare.

The Common Opponents are listed if you click on the "COP" against the individual team, just like the games against TUCs are listed if you click that link. I especially like the way that the TUC pop-up is structured, showing that some teams may be very close to falling off of the cliff. This may seem like an odd request, but is there an intelligent way to show the records against teams just below the cliff? Perhaps, only teams within 0.005 RPI?