Imagine a hairy bearded person with large biceps from taking male hormones and working out entering the ladies room!  A man in the ladies room!  Eeek! 
 That's exactly what you'll get if transmen have to use the room of their birth gender.