We show that the diagnostic classifiers used to establish gender and age bias in <cite>Elazar and Goldberg (2018)</cite> (a) rely only on samplespecific patterns to predict gender and age, and (b) do therefore not generalize to new samples or domains.