To be honest, sports are a different segment when it comes to the argument here. At "elite" levels, the right to compete is a privilege that WILL be abused. It's the simple fact that male biology is better equipped to handle physical exertion better than female biology. There is no denying that. Combine it with a "win at all costs" attitude and you have a ripe atmosphere for exploitation.
As an amateur athlete, should this effect you MT? I'd mostly say no. That's what logic says. You running the YOUR AD HERE 10k on a random Saturday should 100% not be infringed on. But when you start talking elite high school athletics, NCAA, and pro/international competition, it becomes a completely different animal.
I really hope that you don't take this as a personal attack. I support you being you in 99.98% of what you do. Elite athletics fall into that .02% however. And there is supporting science behind the reasoning. It's simply biological and economic reasoning. No social, psychological, or other sciences that are in play with this aspect. Obviously this is drastically different when looking at what the LGBTQ community need to fight through on a daily basis. And I get that you don't want this to be the starting point of a "slippery slope". But elite athletics are far more likely to be exploited than open choice of bathrooms.