Word alignments are useful for tasks like statistical and neural machine translation (NMT) and annotation projection. Statistical word aligners perform well, as do methods that extract alignments jointly with translations in NMT. However, most approaches require parallel training data and quality decreases as less training data is available. We propose word alignment methods that require little or no parallel data. The key idea is to leverage multilingual word embeddings – both static and contextualized – for word alignment. Our multilingual embeddings are created from monolingual data only without relying on any parallel data or dictionaries. We find that traditional statistical aligners are outperformed by contextualized embeddings – even in scenarios with abundant parallel data. For example, for a set of 100k parallel sentences, contextualized embeddings achieve a word alignment F1 that is more than 5% higher (absolute) than eflomal. Word alignment is essential for statistical machine translation and useful in NMT, e.g., for imposing priors on attention matrices (Liu et al., 2016; Alkhouli and Ney, 2017; Alkhouli et al., 2018) or for decoding (Alkhouli et al., 2016; Press and Smith, 2018) . Further, word alignments have been successfully used in a range of tasks such as typological analysis (Lewis and Xia, 2008; Östling, 2015) , annotation projection (Yarowsky et al., 2001; Padó and Lapata, 2009 ) and creating multilingual embeddings (Guo et al., 2016) . Statistical word aligners such as the IBM models (Brown et al., 1993) and their successors (e.g., fastalign (Dyer et al., 2013) , GIZA++ (Och and Ney, 2003) , eflomal (Östling and Tiedemann, 2016) ) are widely used for alignment. With the rise of NMT (Bahdanau et al., 2014) , attempts have been made to interpret attention matrices as soft word alignments (Koehn and Knowles, 2017; Ghader and Monz, 2017) . Several methods create alignments from attention matrices (Peter et al., 2017; Li et al., 2018; Zenkel et al., 2019) or pursue a multitask approach for alignment and translation (Chen et al., 2016; Garg et al., 2019) . However, most systems require parallel data and their performance deteriorates when parallel text is scarce (cf. Tables 1-2 in (Och and Ney, 2003) ). Recent unsupervised multilingual embedding algorithms that use only monolingual data provide high quality static and contextualized embeddings (Conneau et al., 2018; Devlin et al., 2019; Pires et al., 2019) . Our key idea is to leverage these embeddings for word alignments -without relying on parallel data. Requiring no or little parallel data is advantageous in many scenarios, e.g., in the low-resource case and in domain-specific settings without parallel data. A lack of parallel data cannot be easily remedied: mining parallel sentences is possible (cf. (Schwenk et al., 2019) ) but assumes that monolingual corpora contain parallel sentences. Contributions: (1) We propose two new alignment methods based on the matrix of embedding similarities. (2) We propose two post-processing algorithms that handle null words and integrate positional information. (3) We show that word alignments obtained from multilingual BERT outperform strong statistical word aligners like eflomal. (4) We investigate the differences between word and subword processing for alignments and find subword processing to be preferable. Upon acceptance we will publish the source code. We presented word aligners based on contextualized (resp. static) embeddings that perform better than (resp. comparably with) statistical word aligners. Our method is the first that does not require parallel data and is particularly useful for scenarios where a medium number of parallel sentences need to be aligned, but no additional parallel data is available. For a set of 100k parallel sentences, contextualized embeddings achieve an alignment F 1 that is 5% higher (absolute) than eflomal.
We use representations trained without any parallel data for creating word alignments.
{"src": "scitldr-AIC_train", "paper_id": "SFIxZr3RRpr"}