How to improve the Pairwise

[Shirtless Guy]

Re: How to improve the Pairwise

If the fourth criteria is one in which all or most teams will have data, it will result in fewer ties overall, because H2H comes up in maybe 1/3 of team comparisons and often less (depending on a team's conference they're playing maybe as few as 12 or as many as 20 opponents over the course of the season). That means that currently, most comparisons are decided only by RPI and common opponents, and since the RPI is the tiebreaker, that means most comparisons are really decided only by RPI because common opponents alone isn't enough to overcome RPI.

To elaborate on this there are a total of 1770 total comparisons, 1332 (>75%)were based solely on RPI because there was no H2H comparison. Of those 1332, 244 (18.3%) resulted in comparison being awarded on the RPI tiebreaker because the other team won the ComOpp comparison. 576 had the potential to end up a tie if an additional criteria was added but I highly doubt it would, especially if you did added a 2nd RPI value, like QWB+aRPI and aRPI each as their own criteria.

 

[fightingsiouxfan16]

Re: How to improve the Pairwise

I brought up injuries or missing key players (World juniors, suspensions) and when(young inexperienced in the fall(freshmen) vs spring) you play strong/weak teams are big factors.
Not sure how you would do this mathematically. Maybe weighting factors?

The biggest thing would be to get the NCAA to let them play more games (non-conference) later in the year. It is weighted way too heavily on results early in the year.

SCSU swept Minnesota in Nov but I don't know if we get the same result if we play them in Feb.

I think SCSCU would still beat them. Minn too me doesnt look like minnesota of the past. Michigan is very impressive. I would love to see more non conference games for the weaker leagues. when it comes to the NCHC and HE they play very tough conference games so to add tougher non conference games might be brutal. Like you said world juniors hurt but what if the schools dont let them go. Only let the kids go that are in juniors and it wont impact college hockey. Now im sure thatll hurt college hockey too. Providence lost a 3OT game to Lowell and dropped 2 spots which i thought was a little much. It was a great game and a deflection off a players skate to end the game.

 

[Shirtless Guy]

Re: How to improve the Pairwise

copied from the other thread since Tipsy wants this answer as well as J.D.

I'll play along. If there is a problem with it, can you identify a team you think should be in over some other team? If so, why? Just curious.

Obviously this year, my preference would be that MTU deserved to make the cut, having the 8th best W%, top ten in scoring offense, scoring defense and PP. I know they didn't have a very difficult schedule on paper and if they won one more game (either won the GLI champ over MI or gotten one win in their horrible weekend against UAA) they'd be in right now.

I will say that I have had these concerns with the current system since it was instituted 2 years ago. There needs to be a way for any team to flip a comparison without playing head to head, without it, we put too much faith in RPI getting things right. The RPI calculator as-is was basically set at those values solely to minimize the number of negative impact games, not because .25/.21/.59 does a better job than .25/.50/.25 at determining the best teams. RPI obviously is flawed at it's core since a victory over certain teams can lower a team's RPI.

As I broke down early, nearly 20% of all comparisons that didn't have a head to head component finished tied 1-1 with the tiebreaker going to the flawed RPI value.

 

[Shirtless Guy]

Re: How to improve the Pairwise

If any likes the ideas that get presented, I can run through the last 3 year's values and see how much things change. Granted, in the end the point isn't to make things change necessarily, just to do a better job.

 

[ExileOnDaytonStreet]

Re: How to improve the Pairwise

I think the OppOpp is weighted too high and I'd find some way to lower it slightly.

I think this reveals part of the problem with RPI in general.

The goal of a "simple math" system is, IMO, that it's as objective and removed from influence as possible. That means removing subjective criteria or "fudge factors" used to try to better fit historical trends.

The coefficients used to calculate RPI, as it turns out, have a HUGE impact on the results and therefore deciding what those coefficients should be introduces a lot of subjectivity into their rankings.

Therefore, RPI isn't really a very ideal metric. Even if it's still better than a dozen guys sealed off in a dark, smoky room trying to do what "feels right".

Edit: And this is before you get the "certain victories can lower your RPI" problem that our esteemed topless friend has pointed out.

 

[LtPowers]

Re: How to improve the Pairwise

Wouldn't that favor teams who are closer to .500 in conference but play in a tough conference because everyone else gets a better OppOpp W% value, the largest component of RPI now, while a top team in a weaker conference gets penalized for winning most of their games within their conference, driving down the OppOpp W% component?

Yes, and in fact I was thinking of starting a thread on this very topic. The top teams in Atlantic Hockey (and, I would guess, nWCHA and B1G) are routinely underrated in the PWR due to the necessities of their conference schedules. They play weak teams multiple times each, and that's extremely likely to result in "bad" losses just due to the vagaries of chance (a bounce here, a hot goalie there).


Powers &8^]

 

[FlagDUDE08]

Re: How to improve the Pairwise

Since the last change, I have always found it strange that only head to head games can flip a comparison from the team with the higher RPI, because a superior common opponent record isn't enough by itself because tiebreaker goes to RPI. With the RPI calculation so flawed and skewed specifically for hockey to avoid negative impact games, it doesn't make sense to put so much emphasis on something so arbitrary. Does anyone have a suggestion on how to change the current formulation to find a 4th criteria to allow flipping a comparison without head to head games? My first thought was to incorporate RPI w/QWB and RPI w/o QWB since not every team gets equal shots at QWB and only so much of that can be the responsibility of the team itself. If we are so confident in the RPI we choose to use, it would make sense to use both numbers in some fashion.

Other thoughts? I'd like to gather a few ideas and incorporate them into the last 3 years of the current system and see how it would have changed the field.

Of course we should win all the tiebreakers!

 

[Ralph Baer]

Re: How to improve the Pairwise

Of course we should win all the tiebreakers!

The problem is that we have to tie first.

 

[Tipsy McStagger]

Re: How to improve the Pairwise

copied from the other thread since Tipsy wants this answer as well as J.D.

Obviously this year, my preference would be that MTU deserved to make the cut, having the 8th best W%, top ten in scoring offense, scoring defense and PP. I know they didn't have a very difficult schedule on paper and if they won one more game (either won the GLI champ over MI or gotten one win in their horrible weekend against UAA) they'd be in right now.

I will say that I have had these concerns with the current system since it was instituted 2 years ago. There needs to be a way for any team to flip a comparison without playing head to head, without it, we put too much faith in RPI getting things right. The RPI calculator as-is was basically set at those values solely to minimize the number of negative impact games, not because .25/.21/.59 does a better job than .25/.50/.25 at determining the best teams. RPI obviously is flawed at it's core since a victory over certain teams can lower a team's RPI.

As I broke down early, nearly 20% of all comparisons that didn't have a head to head component finished tied 1-1 with the tiebreaker going to the flawed RPI value.

Part of this is the WCHA's own fault for having a 28 game conference schedule and only leaving 6 non-conference games. You aren't leaving yourself much margin for error.

As for Michigan Tech being left out, sure their numbers look impressive, but when you look at who they played against it isn't that impressive. They played the 47th ranked team in RPI 3 times and went to OT each time. Swept at home against Anchorage. In games against teams that made the tournament, they went 2-3 against Ferris, and lost to both Michigan and Yale.

 

[Shirtless Guy]

Re: How to improve the Pairwise

Part of this is the WCHA's own fault for having a 28 game conference schedule and only leaving 6 non-conference games. You aren't leaving yourself much margin for error.

As for Michigan Tech being left out, sure their numbers look impressive, but when you look at who they played against it isn't that impressive. They played the 47th ranked team in RPI 3 times and went to OT each time. Swept at home against Anchorage. In games against teams that made the tournament, they went 2-3 against Ferris, and lost to both Michigan and Yale.

Did not lose to Yale and I hardly would use FSU as their barameter. MTU had a winning record against everyone in conference except their horrible weekend with UAA, FSU and 1-1 @ BSU, but your point stands...The WCHA does not leave themselves ONLY 6 NC games, they all have at least 8 and quite a few 10 expect for the Alaska schools because of the exemption. To be fair, they shouldn't have to give up home dates to make their situations better...They need to get home games if they give up conference games. The fact that the WCHA did really bad in their games with the NCHC was a death blow to MTU and the fact that BGSU and MSU didn't do better to be quality opponents. How many one goal games did the good teams have against bad teams? MTU still won 2 of those 3 with Michigan State.

I'm honestly not having this discussion solely because MTU was left out but I guess I can admit that I probably wouldn't have started this thread if they were in. I think the system is flawed even if it has done a decent job of selecting the field over the last 3 years. I've had this discussion with friends the last couple years and this is mainly sparked by digging deep enough to create my own calculator in February.

 

[SJHovey]

Re: How to improve the Pairwise

I take it that we don't want to (re) introduce another comparison factor, like record in last 20 games?
reward teams coming in hot?

I remember. The old "record in the last 16 games" factor used by the selection committee gave them a lot of flexibility.

I'm not a huge fan of this, even though I'm sure my team may have benefited from it a time or two.

Because most teams play primarily conference foes in the second half of the season, it's not a great "comparison" between teams, except of course teams from the same conference.

It also can paint a misleading picture. Say you have a pretty good team, with strong first half record, but then suffers some injuries. Maybe those players don't get back until about 4 games left in the regular season, at which point the team seems to have righted the ship and returned to form.

 

[Tipsy McStagger]

Re: How to improve the Pairwise

Did not lose to Yale and I hardly would use FSU as their barameter. MTU had a winning record against everyone in conference except their horrible weekend with UAA, FSU and 1-1 @ BSU, but your point stands...The WCHA does not leave themselves ONLY 6 NC games, they all have at least 8 and quite a few 10 expect for the Alaska schools because of the exemption. To be fair, they shouldn't have to give up home dates to make their situations better...They need to get home games if they give up conference games. The fact that the WCHA did really bad in their games with the NCHC was a death blow to MTU and the fact that BGSU and MSU didn't do better to be quality opponents. How many one goal games did the good teams have against bad teams? MTU still won 2 of those 3 with Michigan State.

I'm honestly not having this discussion solely because MTU was left out but I guess I can admit that I probably wouldn't have started this thread if they were in. I think the system is flawed even if it has done a decent job of selecting the field over the last 3 years. I've had this discussion with friends the last couple years and this is mainly sparked by digging deep enough to create my own calculator in February.

I am not saying RPI is perfect. When I evaluate a team, I like to look at good wins and bad losses. To me, Tech doesn't really have any good wins. It is kinda tough to when you have 28 of 34 games where you aren't playing good teams. Even if we switched it to KRACH, Tech is only 17th, Mankato 23rd, BGSU 26th.

 

[CLS]

Re: How to improve the Pairwise

I remember. The old "record in the last 16 games" factor used by the selection committee gave them a lot of flexibility.

I'm not a huge fan of this, even though I'm sure my team may have benefited from it a time or two.

Because most teams play primarily conference foes in the second half[i of the season, it's not a great "comparison" between teams, except of course teams from the same conference.

It also can paint a misleading picture. Say you have a pretty good team, with strong first half record, but then suffers some injuries. Maybe those players don't get back until about 4 games left in the regular season, at which point the team seems to have righted the ship and returned to form.

Excellent point and to me it emphasizes two different theories as to who should get the higher seed. Emphasizing the last n games to me is saying that the higher seed should go to the team more likely to win the tournament. Emphasizing every game equally is saying that the higher seed should be a reward for an entire season's work.

 

[Patman]

Excellent point and to me it emphasizes two different theories as to who should get the higher seed. Emphasizing the last n games to me is saying that the higher seed should go to the team more likely to win the tournament. Emphasizing every game equally is saying that the higher seed should be a reward for an entire season's work.

The main reason they got rid of it was the over emphasis on conference play. That was when non-conf games were scarce especially amongst the western schools. Though the notion is sound... How do you measure current performance?

 

[Shirtless Guy]

Re: How to improve the Pairwise

I am not saying RPI is perfect. When I evaluate a team, I like to look at good wins and bad losses. To me, Tech doesn't really have any good wins. It is kinda tough to when you have 28 of 34 games where you aren't playing good teams. Even if we switched it to KRACH, Tech is only 17th, Mankato 23rd, BGSU 26th.

Once again, I'm not trying to make this about MTU...but if you must go there...lets look at something. Wiscconsin, Army, Colgate, Western Michigan, Connecticut and LSSU finished 40-44 in the final PWR...if I add one game to the season for MTU against each of those 6 teams and run the pwr predictor, MTU gets the biggest bump from beating WMU (0.0028) and the smallest against Connecticut (0.0018). If those teams are all very similar according final RPI, should beating them result in that much variation which compounds throughout a season? A win over WMU is worth at least 0.0006 more than anyone else in that range with everyone being between 0.0018 and 0.0022.

 

[CLS]

Re: How to improve the Pairwise

The main reason they got rid of it was the over emphasis on conference play. That was when non-conf games were scarce especially amongst the western schools. Though the notion is sound... How do you measure current performance?

Which notion? To me trying to measure current performance is saying that you're in the "more likely to win the tournament" camp. I don't worry about current performance because I'm in the "reward for a full season's work" camp, so the notion of "current performance" is irrelevant.

 

[goblue78]

Re: How to improve the Pairwise

I feel your pain, Shirtless, but my take on it (as a professional statistician when I'm not watching hockey) is that there is no possible system which can reliably separate the 14th best team from the 15th best team in a league where there is only a modest amount of mixing between conferences. I have no problem living with the notion that there are probably six teams in any given year that any system would accept, another 4 or 5 that any half-decent system accepts, and that the rest of the spots (after AQ) are always going to be methodology-dependent in a way that will be fairly arbitrary. I think it's fairly easy to look at the just misses (MTU, Cornell and Minnesota) and make cases for them getting in (well, not Cornell... I hate them) but you have to ask who are you going to throw out? If MTU is in, Duluth is out. I think it's difficult to come up with objective criteria where that becomes a no-brainer.

When Yale was the last team in 2013, I figured out that there was a game played between Western Michigan and St. Lawrence before Yale's season even started in which an OT goal by St. Lawrence knocked WMU out of a tournament that Yale won. http://board.uscho.com/showthread.p...y-Tournament&p=5686238&viewfull=1#post5686238 Once you get down bast the top 5 or 6 (and even within that group amongst each other) it's all pretty arbitrary. i don't think of the Pairwise as the definitive way to pick the best 16-x teams so much as an objective and not completely stupid way to do it. So that's a long roundabout way of agreeing with Patman. To make it better, you'd better have some pretty solid and irrefutable ideas about what better means in evaluating teams that clearly aren't the best, but still have a shot to win a one-and-done four round tournament.

 

[joecct]

get conferences to reduce number of conference games to have more interconference matchups to allow for better cross conference data...

Great idea! Think people will fill the barn to watch Clarkson - Bemidji?

Ideally an 8 team conference with 28 conference games works best. That gives 6 OOC that you can min/max out costs/revenues while giving a shot to your RPI and PWR.

 

[ExileOnDaytonStreet]

Re: How to improve the Pairwise

Great idea! Think people will fill the barn to watch Clarkson - Bemidji?

Ideally an 8 team conference with 28 conference games works best. That gives 6 OOC that you can min/max out costs/revenues while giving a shot to your RPI and PWR.

It does produce wild(er) swings in RPI and PWR when you have a more limited sample size for OOC games, though. Really makes those wins a lot more valuable.

 

[Shirtless Guy]

Re: How to improve the Pairwise

Anything done must be able to be done by a fastidious layperson. As awful as things are, the RPI can be computed in a long afternoon with a simple calculator.

I completely disagree with this...took me 2 weeks of my free time with help from Jim Dahl to get through all the ins and outs of QWB, weighting games, etc. This is far from something a layperson can understand and compute with a long afternoon and a calculator. If it was simply the old RPI without weights, maybe. I think we're already past that point and to choose to not change a system simply because it's too complex is silly. A layperson can understand a system without being able to figure out the values themselves in an afternoon.

I feel your pain, Shirtless, but my take on it (as a professional statistician when I'm not watching hockey) is that there is no possible system which can reliably separate the 14th best team from the 15th best team in a league where there is only a modest amount of mixing between conferences. I have no problem living with the notion that there are probably six teams in any given year that any system would accept, another 4 or 5 that any half-decent system accepts, and that the rest of the spots (after AQ) are always going to be methodology-dependent in a way that will be fairly arbitrary. I think it's fairly easy to look at the just misses (MTU, Cornell and Minnesota) and make cases for them getting in (well, not Cornell... I hate them) but you have to ask who are you going to throw out? If MTU is in, Duluth is out. I think it's difficult to come up with objective criteria where that becomes a no-brainer.

When Yale was the last team in 2013, I figured out that there was a game played between Western Michigan and St. Lawrence before Yale's season even started in which an OT goal by St. Lawrence knocked WMU out of a tournament that Yale won. http://board.uscho.com/showthread.p...y-Tournament&p=5686238&viewfull=1#post5686238 Once you get down bast the top 5 or 6 (and even within that group amongst each other) it's all pretty arbitrary. i don't think of the Pairwise as the definitive way to pick the best 16-x teams so much as an objective and not completely stupid way to do it. So that's a long roundabout way of agreeing with Patman. To make it better, you'd better have some pretty solid and irrefutable ideas about what better means in evaluating teams that clearly aren't the best, but still have a shot to win a one-and-done four round tournament.

I agree with you that it is difficult to find any system that is flawless in determining the top 10 non-AQ teams, especially in the case of college hockey when so few of the games intermingle between east and west. The biggest issue is the lack of that overlap and the ability to properly compare MTU to Cornell as an example. Under the current System Cornell wins on RPI but loses CoOp, but the only CoOp was Yale...so that is no way to flip a comparison...

I really try hard here to get the point across that this isn't specifically about MTU. I can tell that most are not believing me...