Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Data Selection, Synthesis, and Preprocessing <s> We consider a very simple, yet effective, approach to cross language adaptation of dependency parsers. We first remove lexical items from the treebanks and map part-of-speech tags into a common tagset. We then train a language model on tag sequences in otherwise unlabeled target data and rank labeled source data by perplexity per word of tag sequences from less similar to most similar to the target. We then train our target language parser on the most similar data points in the source labeled data. The strategy achieves much better results than a non-adapted baseline and state-of-the-art unsupervised dependency parsing, and results are comparable to more complex projection-based cross language adaptation algorithms. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Data Selection, Synthesis, and Preprocessing <s> We propose a simple cross-language parser adaptation strategy for discriminative parsers and apply it to easy-first transition-based dependency parsing (Goldberg and Elhadad, 2010). We evaluate our parsers on the Indo-European corpora in the CoNLL-X and CoNLL 2007 shared tasks. Using the remaining languages as source data we average under-fitted weights learned from each source language and apply the resulting linear classifier to the target language. Of course some source languages and some sentences in these languages are more relevant than others for the target language in question. We therefore explore improvements of our cross-language adaptation model involving source language and instance weighting, as well as unsupervised model selection. Overall our cross-language adaptation strategies provide better results than previous strategies for direct transfer, with near-linear time parsing and much faster training times than other approaches. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Data Selection, Synthesis, and Preprocessing <s> We present \textit{AutoExtend}, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The synset/lexeme embeddings obtained live in the same vector space as the word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet as a lexical resource, but AutoExtend can be easily applied to other resources like Freebase. AutoExtend achieves state-of-the-art performance on word similarity and word sense disambiguation tasks. <s> BIB003 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Data Selection, Synthesis, and Preprocessing <s> Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank). <s> BIB004 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Data Selection, Synthesis, and Preprocessing <s> We release Galactic Dependencies 1.0---a large set of synthetic languages not found on Earth, but annotated in Universal Dependencies format. This new resource aims to provide training and development data for NLP methods that aim to adapt to unfamiliar languages. Each synthetic treebank is produced from a real treebank by stochastically permuting the dependents of nouns and/or verbs to match the word order of other real languages. We discuss the usefulness, realism, parsability, perplexity, and diversity of the synthetic languages. As a simple demonstration of the use of Galactic Dependencies, we consider single-source transfer, which attempts to parse a real target language using a parser trained on a "nearby" source language. We find that including synthetic source languages somewhat increases the diversity of the source pool, which significantly improves results for most target languages. <s> BIB005
Another way in which typological features are used in NLP is to guide data selection. This procedure is crucial for (1) language transfer methods, as it guides the choice of the most suitable source languages and examples; and (2) multilingual joint models, in order to weigh the contribution of each language and example. The selection is typically carried out through general language similarity metrics. For instance,  base their selection on the URIEL language typology database, considering information about genealogical, geographic, syntactic, and phonetic properties. This facilitates language transfer of grapheme-to-phoneme models, by guiding the choice of source languages and aligning phoneme inventories. Metrics for source selection can also be extracted in a data-driven fashion, without explicit reference to structured taxonomies. For instance, BIB003 estimate the Kullback-Leibler divergence between PoS trigram distributions for delexicalized parser transfer. In order to approximate the divergence in syntactic structures between languages,  utilize the Jaccard distance between morphological feature sets and the tree edit distance of delexicalized dependency parses of translationally equivalent sentences. A priori and bottom-up approaches can also be combined. For delexicalized parser transfer, Agić (2017) relies on a weighted sum of distances based on (1) the PoS divergence defined by BIB003 ; (2) the character-based identity prediction of the target language; and (3) the Hamming distance from the target language typological vector. In fact, they have different weaknesses: Language identity (and consequently typology) fails to abstract away from language scripts. On the other hand, the accuracy of PoS-based metrics deteriorates easily in scenarios with scarce amounts of data. Source language selection is a special case of source language weighting where weights are one-hot vectors. However, weights can also be gradient and consist of real numbers. BIB002 adapt delexicalized parsers by weighting every 584 training instance based on the inverse of the Hamming distance between typological (or genealogical) features in source and target languages. An equivalent bottom-up approach is developed by BIB001 , who weighs source language sentences based on the perplexity between their coarse PoS tags and the predictions of a sequential model trained on the target language. Alternatively, the lack of target annotated data can be alleviated by synthesizing new examples, thus boosting the variety and amount of the source data. For instance, the Galactic Dependency Treebanks stem from real trees whose nodes have been permuted probabilistically, according to the word orders of nouns and verbs in other languages BIB005 . Synthetic trees improve the performance of model transfer for parsing when the source is chosen in a supervised way (performance on target development data) and in an unsupervised way (coverage of target PoS sequences). Rather than generating new synthetic data,  leverage typological features to pre-process treebanks in order to reduce their variation in language transfer tasks. In particular, they adapt source trees to the typology of a target language with respect to several constructions in a rule-based fashion. For instance, relative clauses in Arabic (Afro-Asiatic) with an indefinite antecedent drop the relative pronoun, which is mandatory in Portuguese (Indo-European). Hence, the pronoun has to be added, or deleted in the other direction. Feeding pre-processed syntactic trees to lexicalized syntax-based neural models, such as feature-based recurrent encoders (Sennrich and Haddow 2016) or TreeLSTMs BIB004 , achieves state-ofthe-art results in Neural Machine Translation and cross-lingual sentence similarity classification.