Self-training methods such as ELMo (Peters et al., 2018) , GPT (Radford et al., 2018) , BERT<cite> (Devlin et al., 2019)</cite> , XLM (Lample and Conneau, 2019) , and XLNet have brought significant performance gains, but it can be challenging to determine which aspects of the methods contribute the most.