Why fuck is that any of their business?

Its no different to asking if you wear womens underwear to work.

Besides follow it up after the joke with a "well if you speak to my references you'll note that I have been 100% accountible for my work and personal behavior".