As per usual, it's about laziness. Shortening what are two things into the same wordlet. If people could actually be arsed to say 'transvestite' and 'transsexual' instead of just 'trans' every time, it would clear up a lot of confusion. The same applies to the 'gender-fluid' -why not just say 'androgynous' and have done with it.
Clearly a male transvestite, sans gender dysphoria, is still physically and psychologically male and should not be allowed into female-only spaces. They may be in danger in some male-only spaces, but are as well-equipped to defend themselves as any other man. That being said, is there, or has there ever been, any such thing as a 'safe space' for anyone? For surely the world is full of perils.
A male sex offender, unless his offences have been solely against males, should never be confined with female prisoners whatever they claim their gender identity to be. That's called common sense and is too seldom applied nowadays..
The main actual - as opposed to perceived - issue is that of transsexuals in the process of transition. One is inclined, at first look, to say that socially-transitioned people should use the facilities assigned to their original physical sex until their surgical reassignment is complete. However, this causes difficulties. They would be in the same position as a transvestites, physically, but psychologically/emotionally would be in a much worse case. To be purely practical, I would advocate encouraging such people to use disabled facilities pro tem, as these are generally more private and less crowded.
You will not stop predatory men dressing as women to invade female spaces because this is a wholly different thing and changing labels and rules won't help. What you need are attendants in restrooms and so on, so that there will always be a witness and somebody to help.
As to the rest of the gender-confused, flexible, neutral, whatever-I-feel-like-today crowd, they need to sort themselves out! Some are bandwagon-jumpers, some need help, some will figure it out for themselves, and there are probably a fair few who stand in need of a kick up the arse! But that's people for you.