Recent progress in word embedding techniques has been achieved with contextualized word embeddings (Peters et al., 2018) which provide different vector representations for the same word in different contexts. While gender bias has been studied, detected and partially addressed for standard word embeddings techniques<cite> (Bolukbasi et al., 2016</cite>; Zhao et al., 2018a; Gonen and Goldberg, 2019) , it is not the case for the latest techniques of contextualized word embeddings.