We present a family of neural-network--inspired models for computing continuous word representations, specifically designed to exploit both monolingual and multilingual text. This framework allows us to perform unsupervised training of embeddings that exhibit higher accuracy on syntactic and semantic compositionality, as well as multilingual semantic similarity, compared to previous models trained in an unsupervised fashion. We also show that such multilingual embeddings, optimized for semantic similarity, can improve the performance of statistical machine translation with respect to how it handles words not present in the parallel data.
Because our approach enables us to exploit both monolingual and parallel data simultaneously, the resulting distributed representation can be successfully used to learn translations for terms that appear in the monolingual data only (). This represents the neural word-embedding equivalent of a long line of research based on word-surface patterns, starting with earlier attempts @cite_42 , and continuing with @cite_23 @cite_35 , and complemented by approaches based on probabilistic models @cite_26 . Our approach has the advantages of achieving this effect in a completely unsupervised fashion, without exploiting surface patterns, and benefiting from the smoothness properties associated with continuous word representations.
@cite_35 @cite_42 @cite_23 @cite_26
2084277454 2139812240 2140406733
This paper presents novel improvements to the induction of translation lexicons from monolingual corpora using multilingual dependency parses. We introduce a dependency-based context model that incorporates long-range dependencies, variable context sizes, and reordering. It provides a 16 relative improvement over the baseline approach that uses a fixed context window of adjacent words. Its Top 10 accuracy for noun translation is higher than that of a statistical translation model trained on a Spanish-English parallel corpus containing 100,000 sentence pairs. We generalize the evaluation to other word-types, and show that the performance can be increased to 18 relative by preserving part-of-speech equivalencies during translation. Common algorithms for sentence and word-alignment allow the automatic identification of word translations from parallel texts. This study suggests that the identification of word translations should also be possible with non-parallel and even unrelated texts. The method proposed is based on the assumption that there is a correlation between the patterns of word co-occurrences in texts of different languages. We present a method for learning bilingual translation lexicons from monolingual corpora. Word types in each language are characterized by purely monolingual features, such as context counts and orthographic substrings. Translations are induced using a generative model based on canonical correlation analysis, which explains the monolingual lexicons in terms of latent matchings. We show that high-precision lexicons can be learned in a variety of language pairs and from a range of corpus types.