I think that's a valid point.  Public restrooms are designed with less privacy in the U.S. than elsewhere, but there's no real reason for that.  The biggest threat to a woman's privacy in a public bathroom with stalls is from other people's toddlers. 

There are ways of dealing with people's fears about either bathrooms or locker rooms that don't involve discriminating against groups of people.