Although successful, adversarial training has been criticized for not being stable and failing to converge, inspiring researchers to propose nonadversarial methods more recently (Xu et al., 2018a; Hoshen and Wolf, 2018; Alvarez-Melis and Jaakkola, 2018; Artetxe et al., 2018b) . In particular, Artetxe et al. (2018b) show that the adversarial methods of <cite>Conneau et al. (2018)</cite> and Zhang et al. (2017a,b) fail for many language pairs.