We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task, we also analyze BERT and find that it exhibits systematic deficiencies, e.g. worse alignment for open-class parts-of-speech and word pairs written in different scripts, that are corrected by the alignment procedure. These results support contextual alignment as a useful concept for understanding large multilingual pre-trained models. Figure 1: t-SNE (Maaten & Hinton, 2008) visualization of the embedding space of multilingual BERT for English-German word pairs (left: pre-alignment, right: post-alignment). Each point is a different instance of the word in the Europarl corpus. This figure suggests that BERT begins already somewhat aligned out-of-the-box but becomes much more aligned after our proposed procedure. Embedding alignment was originally studied for word vectors with the goal of enabling cross-lingual transfer, where the embeddings for two languages are in alignment if word translations, e.g. cat and Katze, have similar representations (Mikolov et al., 2013a; Smith et al., 2017) . Recently, large pretrained models have largely subsumed word vectors based on their accuracy on downstream tasks, partly due to the fact that their word representations are context-dependent, allowing them to more richly capture the meaning of a word (Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018) . Therefore, with the same goal of cross-lingual transfer but for these more complex models, we might consider contextual embedding alignment, where we observe whether word pairs within parallel sentences, e.g. cat in "The cat sits" and Katze in "Die Katze sitzt," have similar representations. One model relevant to these questions is multilingual BERT, a version of BERT pre-trained on 104 languages that achieves remarkable transfer on downstream tasks. For example, after the model is fine-tuned on the English MultiNLI training set, it achieves 74.3% accuracy on the test set in Spanish, which is only 7.1% lower than the English accuracy (Devlin et al., 2018; Conneau et al., 2018b) . Furthermore, while the model transfers better to languages similar to English, it still achieves reasonable accuracies even on languages with different scripts. However, given the way that multilingual BERT was pre-trained, it is unclear why we should expect such high zero-shot performance. Compared to monolingual BERT which exhibits no zero-shot transfer, multilingual BERT differs only in that (1) during pre-training (i.e. masked word prediction), each batch contains sentences from all of the languages, and (2) it uses a single shared vocabulary, formed by WordPiece on the concatenated monolingual corpora (Devlin et al., 2019) . Therefore, we might wonder: (1) How can we better understand BERT's multilingualism? (2) Can we further improve BERT's cross-lingual transfer? In this paper, we show that contextual embedding alignment is a useful concept for addressing these questions. First, we propose a contextual version of word retrieval to evaluate the degree of alignment, where a model is presented with two parallel corpora, and given a word within a sentence in one corpus, it must find the correct word and sentence in the other. Using this metric of alignment, we show that multilingual BERT achieves zero-shot transfer because its embeddings are partially aligned, as depicted in Figure 1 , with the degree of alignment predicting the degree of downstream transfer. Next, using between 10K and 250K sentences per language from the Europarl corpus as parallel data (Koehn, 2005) , we propose a fine-tuning-based alignment procedure and show that it significantly improves BERT as a multilingual model. Specifically, on zero-shot XNLI, where the model is trained on English MultiNLI and tested on other languages (Conneau et al., 2018b) , the aligned model improves accuracies by 2.78% on average over the base model, and it remarkably matches translate-train models for Bulgarian and Greek, which approximate the fully-supervised setting. To put our results in the context of past work, we also use word retrieval to compare our finetuning procedure to two alternatives: (1) fastText augmented with sentence and aligned using rotations (Bojanowski et al., 2017; Rücklé et al., 2018; Artetxe et al., 2018) , and (2) BERT aligned using rotations (Aldarmaki & Diab, 2019; Schuster et al., 2019; Wang et al., 2019) . We find that when there are multiple occurences per word, fine-tuned BERT outperforms fastText, which outperforms rotation-aligned BERT. This result supports the intuition that contextual alignment is more difficult than its non-contextual counterpart, given that a rotation, at least when applied naively, is no longer sufficient to produce strong alignments. In addition, when there is only one occurrence per word, fine-tuned BERT matches the performance of fastText. Given that context disambiguation is no longer necessary, this result suggests that our fine-tuning procedure is able to align BERT at the type level to a degree that matches non-contextual approaches. Finally, we use the contextual word retrieval task to conduct finer-grained analysis of multilingual BERT, with the goal of better understanding its strengths and shortcomings. Specifically, we find that base BERT has trouble aligning open-class compared to closed-class parts-of-speech, as well as word pairs that have large differences in usage frequency, suggesting insight into the pre-training procedure that we explore in Section 5. Together, these experiments support contextual alignment as an important task that provides useful insight into large multilingual pre-trained models. Given that the degree of alignment is causally predictive of downstream cross-lingual transfer, contextual alignment proves to be a useful concept for understanding and improving multilingual pretrained models. Given small amounts of parallel data, our alignment procedure improves multilingual BERT and corrects many of its systematic deficiencies. Contextual word retrieval also provides useful new insights into the pre-training procedure, opening up new avenues for analysis. Table 5 : Zero-shot accuracy on the XNLI test set with more languages, where we use 20K parallel sentences for each language paired with English. This result confirms that the alignment method works for distant languages and a variety of parallel corpora, including Europarl, MultiUN, and Tanzil, which contains sentences from the Quran (Koehn, 2005; Eisele & Chen, 2010; Tiedemann, 2012) . A APPENDIX
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they both improve multilingual BERT's zero-shot XNLI transfer and provide useful insights into the model.
{"src": "scitldr-AIC_train", "paper_id": "r1xCMyBtPS"}