Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, Tomas Mikolov &ndash; Facebook AI Research; École polytechnique fédérale de Lausanne EPFL, Switzerland
Learning word vectors for 157 languages
Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance. A key ingredient to the successful application of these representations is to train t hem on very large corpora, and use these pre-trained models in downstream tasks. In this paper, we describe how we trained such high qualit y word representations for 157 languages. We used two sources of data to train these models: the free online encyclopedia Wikip edia and data from the common crawl project. We also introduce three new word analogy datasets to evaluate these word vectors, for Fren ch, Hindi and Polish. Finally, we evaluate our pre-trained word vectors on 10 languages for which evaluation datasets exists, sho wing very strong performance compared to previous models.
Facebook AI Research; École polytechnique fédérale de Lausanne EPFL, Switzerland
nlp/word-embeddings
The common crawl is a non profit organization which crawls the web and makes the resulting data publicly available. This large scale corpus was previously used to estimate n-gram language models (Buck et al., 2014) or to learn English word vectors (Pennington et al., 2014). To the best of our knowledge, it was not used yet to learn word vectors for a large set of languages. The data is distributed either as raw HTML pages, or as WET files which contain the extracted text data, converted to UTF-8. We decided to use the extracted text data, as it is much smaller in size, and easier to process (no need to remove HTML). We downloaded the May 2017 crawl, corresponding to roughly 24 terabytes of raw text data.