One is led to believe that the housekeepers told people.  So that is how the story got out. 

JPII never denied this after it went public.

Would you have had someone take signed affidavits from the Pope's Housekeepers?

Since JPII always supported Opus Dei which demands its women clean not only their own women's barracks but also the men's, it is likely true.  

So if you were to find out Pandora, that he did definitely say it, would you support his statement as a loving and equitable statement made to these women from JPII?    or finally admit this Pope was sexist and that this bias against women is gender degrading?