The scarcity of labeled training data often prohibits the internationalization of NLP models to multiple languages.   Cross-lingual understanding has made progress in this area using language universal representations. However, most current approaches focus on the problem as one of aligning language and do not address the natural domain drift across languages and cultures.   In this paper, We address the domain gap in the setting of semi-supervised cross-lingual document classification, where labeled data is available in a source language and only unlabeled data is available in the target language.   We combine a state-of-the-art unsupervised learning method, masked language modeling pre-training, with a recent method for semi-supervised learning, Unsupervised Data Augmentation (UDA), to simultaneously close the language and the domain gap.   We show that addressing the domain gap in cross-lingual tasks is crucial.   We improve over strong baselines and achieve a new state-of-the-art for cross-lingual document classification. Recent advances in Natural Language Processing have enabled us to train high-accuracy systems for many language tasks. However, training an accurate system still requires a large amount of training data. It is inefficient to collect data for a new task and it is virtually impossible to annotate a separate data set for each language. To go beyond English and a few popular languages, we need methods that can learn from data in one language and apply it to others. Cross-Lingual Understanding (XLU) has emerged as a field concerned with learning models on data in one language and applying it to others. Much of the work in XLU focuses on the zero-shot setting, which assumes that labeled data is available in one source language (usually English) and not in any of the target languages in which the model is evaluated. The labeled data can be used to train a high quality model in the source language. One then relies on general domain parallel corpora and monolingual corpora to learn to 'transfer' from the source language to the target language. Transfer methods can explicitly rely on machine translation models built from such parallel corpora. Alternatively, one can use such corpora to learn language universal representations to produce features to train a model in one language, which one can directly apply to other languages. Such representations can be in the form of cross-lingual word embeddings, contextual word embeddings, or sentence embeddings (Ruder et al. (2017) ; Lample & Conneau (2019) ; Schwenk & Douze (2017) ). Using such techniques, recent work has demonstrated reasonable zero-shot performance for crosslingual document classification (Schwenk & Li (2018) ) and natural language inference (Conneau et al. (2018) ). What we have so far described is a simplified view of XLU, which focuses solely on the problem of aligning languages. This view assumes that, if we had access to a perfect translation system, and translated our source training data into the target language, the resulting model would perform as well as if we had collected a similarly sized labeled dataset directly in our target language. Existing work in XLU to date also works under this assumption. However, in real world applications, we must also bridge the domain gap across different languages, as well as the language gap. No task is ever identical in two languages, even if we group them under the same label, e.g. 'news document classification' or 'product reviews sentiment analysis'. A Chinese customer might express sentiment differently than his American counterpart. Or French news might simply cover different topics than English news. As a result, any approach which ignores this domain drift will fall short of native in-language performance in real world XLU. In this paper, we propose to jointly tackle both language and domain transfer. We consider the semi-supervised XLU task, where in addition to labeled data in a source language, we have access to unlabeled data in the target language. Using this unlabeled data, we combine the aforementioned cross-lingual methods with recently proposed unsupervised domain adaptation and weak supervision techniques on the task of cross-lingual document classification (XLDC). In particular, we focus on two approaches for domain adaptation. The first method is based on masked language model (MLM) pre-training (as in Devlin et al. (2018) ) using unlabeled target language corpora. Such methods have been shown to improve over general purpose pre-trained models such as BERT in the weakly supervised setting (Lee et al. (2019) ; Han & Eisenstein (2019) ). The second method is unsupervised data augmentation (UDA) (Xie et al. (2019) ), where synthetic paraphrases are generated from the unlabeled corpus, and the model is trained on a label consistency loss. While both of these techniques were proposed previously, in both cases there are some open questions when applying them on the cross-lingual problems. For instance when performing data augmentation, one could generate augmented paraphrases in either the source or the target language or both. We experiment with various approaches and provide guidelines with ablation studies. Furthermore, we find that the value of additional labeled data in the source language is limited due to the train-test discrepancy of XLDC tasks. We propose to alleviate this issue by using self-training technique to do the domain adaptation from the source language into the target language. By combining these methods, we are able to reduce error rates by an average 44% over a strong XLM baseline, setting a new state-of-the-art for cross-lingual document classification. In this paper, we tackled the domain mismatch challenge in cross-lingual document classification -an important, yet often overlooked problem in cross-lingual understanding. We provided evidence for the existence and importance of this problem, even when utilizing strong pre-trained cross-lingual representations. We proposed a framework combining cross-lingual transfer techniques with three domain adaptation methods; unsupervised data augmentation, masked language model pre-training and self-training, which can leverage unlabeled data in the target language to moderate the domain gap. Our results show that by removing the domain discrepancy, we can close the performance gap between crosslingual transfer and monolingual baselines almost completely for the document classification task. We are also able to improve the state-of-the-art in this area by a large margin. While document classification is by no means the most challenging task for XLU, we believe the strong gains that we demonstrated could be extended to other cross-lingual tasks, such as cross-lingual question answering and event detection. Developing cross-lingual methods which are competitive with in-language models for real world, semantically challenging NLP problems remains an open problem and subject of future research. The experiments in this paper are based on the PyTorch (Paszke et al., 2017) and Pytext (Aly et al., 2018) package. We use the Adam (Kingma & Ba, 2014) as the optimizer. For all experiments, we grid search the learning rate in the set {5 × 10 −6 , 1 × 10 −5 , 2 × 10 −5 }. When using UDA method, we also try the three different annealing strategies introduced in the UDA paper (Xie et al., 2019) , and the λ in (1) is always set as 1. The batch size in the Ft and UDA+Self method is 128. In the UDA method, the batch size is 16 for the labeled data and 80 for the unlabeled data. Due to the limitation of the GPU memory, in all experiments, we set the length of samples as 256, and cut the input tokens exceeding this length. Finally, we report the results with the best hyper-parameters. As for the augmentation process, we sweep the temperature which controls the diversity of beam search in translation. The best temperature for "en-de, en-fr, en-es" and "en-ru" are 1.0 and 0.6, the sampling space is the whole vocabulary. In the "en-zh" setting, the temperature is 1.0 and the sampling space is the top 100 tokens in the vocabulary. We note that this uses the Facebook production translation models, and results could vary when other translation systems are applied. For reproducibility, we will release the augmented datasets that we generated.
Semi-supervised Cross-lingual Document Classification
{"src": "scitldr-AIC_train", "paper_id": "ryeot3EYPB"}