Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> 5.2.2 <s> In this paper we explore the novel idea of building a single universal reordering model from English to a large number of target languages. To build this model we exploit typological features of word order for a large number of target languages together with source (English) syntactic features and we train this model on a single combined parallel corpus representing all (22) involved language pairs. We contribute experimental evidence for the usefulness of linguistically defined typological features for building such a model. When the universal reordering model is used for preordering followed by monotone translation (no reordering inside the decoder), our experiments show that this pipeline gives comparable or improved translation performance with a phrase-based baseline for a large number of language pairs (12 out of 22) from diverse language families. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> 5.2.2 <s> We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. We apply these to the problem of modeling phone sequences---a domain in which universal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representations, and extrinsic evaluation in two downstream applications that make use of phonetic features show (i) that polyglot models better generalize to held-out data than comparable monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually. <s> BIB002
Multi-lingual Biasing. Some papers leverage typological features to gear the shared parameters of a joint multilingual model toward the properties of a specific language. BIB001 develop a reordering algorithm that estimates the permutation probabilities of aligned word pairs in multi-lingual parallel texts. The best sequence of permutations is inferred via k-best graph search in a finite state automaton, producing a lattice. This algorithm, which receives lexical, morphological, and syntactic features of the source word pairs and typological features of the target language as input, was shown to benefit a downstream machine translation task. The joint multilingual parser of  shares hidden-layer parameters across languages, and combines both language-invariant and language-specific features in its copious lexicalized input feature set. This transition-based parser selects the next action z (e.g., SHIFT) from a pool of possible actions given its current state p t , as defined in Equation (3): P(z|p t ) is defined in terms of a set of iteratively manipulated, densely represented data structures: a buffer b t , a stack s t , and an action history a t . The hidden representation of these modules are the output of stack-LSTMs, which are in turn fed with input word feature representations (stack and buffer) and action representations (history). The shared parameters are biased toward a particular language through language embeddings l it . The language embeddings consist of (a non-linear transformation of) either a mere one-hot identity vector or a vector of typological properties taken from WALS. In particular, they are added to both input feature and action vectors, to affect the three above-mentioned modules individually, and concatenated to the hidden module representations, to affect the entire parser state. The resulting state representation is propagated through an action-specific layer parametrized by g t and q t , and activated by a softmax function σ over actions. Similarly, typological features have been used to bias input and hidden states of language models. For example, BIB002 proposed a multilingual phonemelevel language model where an input phoneme x and a language vector at time t are linearly mapped to a local context representation and then passed to a global LSTM. This hidden representation G t is factored by a non-linear transformation of typological features t , as shown in Equation (4): As described in Equation (5), G t is then vectorized and mapped to a probability distribution of possible next phonemes φ t . The phoneme vectors, learned by the language model in an end-to-end manner, were demonstrated to benefit two downstream applications: lexical borrowing identification and speech synthesis. Moreover, typological features (in the form of implicational universals) can guide the design of Bayesian networks.  assign part-of-speech labels to word clusters acquired in an unsupervised fashion. The underlying network is acyclic and directed, and is converted to a join-tree network to handle multiple parents  . For instance, the sub-graph for the ordering of numerals and nouns is intertwined also with properties of adjectives and adpositions. The final objective maximizes the probability of a tag T i and a feature set Φ i , given the implicational universals U as argmax T P({Φ i , T i } n i=1 |U).