Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation. However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to  hold in practice (Søgaard et al., 2018), and fundamentally limits their performance. This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective. Given the abundance of parallel data available (Tiedemann, 2012), we propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations. Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches, as well as convincingly outscores mapping methods while maintaining parity with jointly trained methods on word-translation. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference. As an additional advantage, our bilingual method also improves the quality of monolingual word vectors despite training on much smaller datasets.   We make our code and models publicly available.
 Cross-lingual representations-such as embeddings of words and phrases into a single comparable feature space-have become a key technique in multilingual natural language processing. They offer strong promise towards the goal of a joint understanding of concepts across languages, as well as for enabling the transfer of knowledge and machine learning models between different languages. Therefore, cross-lingual embeddings can serve a variety of downstream tasks such as bilingual lexicon induction, cross-lingual information retrieval, machine translation and many applications of zero-shot transfer learning, which is particularly impactful from resource-rich to low-resource languages. Existing methods can be broadly classified into two groups (Ruder et al., 2017) : mapping methods leverage existing monolingual embeddings which are treated as independent, and apply a postprocess step to map the embeddings of each language into a shared space, through a linear transformation (Mikolov et al., 2013b; Conneau et al., 2017; Joulin et al., 2018) . On the other hand, joint methods learn representations concurrently for multiple languages, by combining monolingual and cross-lingual training tasks (Luong et al., 2015; Coulmance et al., 2015; Gouws et al., 2015; Vulic & Moens, 2015; Chandar et al., 2014; Hermann & Blunsom, 2013) . While recent work on word embeddings has focused almost exclusively on mapping methods, which require little to no cross-lingual supervision, (Søgaard et al., 2018) establish that their performance is hindered by linguistic and domain divergences in general, and for distant language pairs in particular. Principally, their analysis shows that cross-lingual hubness, where a few words (hubs) in the source language are nearest cross-lingual neighbours of many words in the target language, and structural non-isometry between embeddings do impose a fundamental barrier to the performance of linear mapping methods. (Ormazabal et al., 2019) propose using joint learning as a means of mitigating these issues. Given parallel data, such as sentences, a joint model learns to predict either the word or context in both source and target languages. As we will demonstrate with results from our algorithm, joint methods yield compatible embeddings which are closer to isomorphic, less sensitive to hubness, and perform better on cross-lingual benchmarks. Contributions. We propose the BI-SENT2VEC algorithm, which extends the SENT2VEC algorithm (Pagliardini et al., 2018; Gupta et al., 2019) to the cross-lingual setting. We also revisit TRANS-GRAM Coulmance et al. (2015) , another joint learning method, to assess the effectiveness of joint learning over mapping-based methods. Our contributions are • On cross-lingual sentence-retrieval and monolingual word representation quality evaluations, BI-SENT2VEC significantly outperforms competing methods, both jointly trained as well as mapping-based ones while preserving state-of-the-art performance on cross-lingual word retrieval tasks. For dis-similar language pairs, BI-SENT2VEC outperform their competitors by an even larger margin on all the tasks hinting towards the robustness of our method. • BI-SENT2VEC performs on par with a multilingual RNN based sentence encoder, LASER (Artetxe & Schwenk, 2018) , on MLDoc (Schwenk & Li, 2018) , a zero-shot crosslingual transfer task on documents in multiple languages. Compared to LASER, our method improves computational efficiency by an order of magnitude for both training and inference, making it suitable for resource or latency-constrained on-device cross-lingual NLP applications. • We verify that joint learning methods consistently dominate state-of-the-art mapping methods on standard benchmarks, i.e., cross-lingual word and sentence retrieval. • Training on parallel data additionally enriches monolingual representation quality, evident by the superior performance of BI-SENT2VEC over FASTTEXT embeddings trained on a 100× larger corpus. We make our models and code publicly available. In the following section, we discuss the results on monolingual and cross-lingual benchmarks, presented in Tables 1 -5 , and a data ablation study for how the model behaves with increasing parallel corpus size in Figure 2 -3. The most impressive outcome of our experiments is improved crosslingual sentence retrieval performance, which we elaborate on along with word translation in the next subsection. We introduce a cross-lingual extension of an existing monolingual word and sentence embedding method. The proposed model is tested at three levels of linguistic granularity: words, sentences and documents. The model outperforms all other methods by a wide margin on the cross-lingual sentence retrieval task while maintaining parity with the best-performing methods on word translation tasks. Our method achieves parity with LASER on zero-shot document classification, despite being a much simpler model. We also demonstrate that training on parallel data yields a significant improvement in the monolingual word representation quality. The success of our model on the bilingual level calls for its extension to the multilingual level especially for pairs which have little or no parallel corpora. While the amount of bilingual/multilingual parallel data has grown in abundance, the amount of monolingual data available is practically limitless. Consequently, we would like to explore training cross-lingual embeddings with a large amount of raw text combined with a smaller amount of parallel data. We used ParaCrawl v4.0 corpora for training BI-SENT2VEC, SENT2VEC,BIVEC,VECMAP and TRANSGRAM embeddings except for En-Ru pair for which we used OpenSubtitles and Tanzil corpora combined. MUSE and RCSLS vectors were trained from FASTTEXT vectors obtained from Wikipedia dumps (Grave et al., 2018a
Joint method for learning cross-lingual embeddings with state-of-art performance for cross-lingual tasks and mono-lingual quality
{"src": "scitldr-AIC_train", "paper_id": "SJlJSaEFwS"}