Someone below mentioned the cult aspect, which is a good place to start. That's part of what I was going for with the "power imbalance" - though who knows how much of that is simply because polygamy is illegal and therefore only practiced by cults.
But anyone who's ever lived with 2 or more roommates at a time knows what I mean to some extent. The way problems are handled in a one-on-one situation is different than a three-way (or more) negotiation. As the number of people in a relationship goes up, there is an increased likelihood that some person in the relationship is less a willing party than someone coerced, at least for some actions. Maybe that goes to the consent element, but I think it's a legitimate concern. If a polygamist could show that concern is not real, I'd be willing to listen to it.
As far as the practical matters, it's not "because this is how it's always been done" but "because the it would cost a boatload of time, money, and legislation to work this out, because the current system is not setup to handle multiple spouses." The government clearly has an interest in its own finances. Whether that interest is sufficient to stave off polygamy likely depends on how it would be classified. It's doubtful polyamory would be classified as a sexual orientation rather than a preference (just as preferences for blondes, or cougars, or poolboys are not classified as such), though obviously that could change over time. In that case, the government wins because it would have a legitimate interest, and under the rational basis test that's sufficient.
And part of the point of civil marriage is that it covers all those rights and responsibilities without needing to consult a lawyer. If I'm in a coma, my wife has the power to make medical decisions. It's simple and efficient. If there are two spouses, what then if they disagree? If there are three others, does majority vote rule or does it have to be unanimous?
I'm sure there are people out there that could make a strong argument for a return to the Biblical days of Solomon and his 700 wives, but I'm not one of them because I simply don't care. I happen to think the government could win court cases against it without resorting to the lame reasons put forth in the gay marriage cases, but I'm not invested in that.