Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Comparison <s> We propose a simple cross-language parser adaptation strategy for discriminative parsers and apply it to easy-first transition-based dependency parsing (Goldberg and Elhadad, 2010). We evaluate our parsers on the Indo-European corpora in the CoNLL-X and CoNLL 2007 shared tasks. Using the remaining languages as source data we average under-fitted weights learned from each source language and apply the resulting linear classifier to the target language. Of course some source languages and some sentences in these languages are more relevant than others for the target language in question. We therefore explore improvements of our cross-language adaptation model involving source language and instance weighting, as well as unsupervised model selection. Overall our cross-language adaptation strategies provide better results than previous strategies for direct transfer, with near-linear time parsing and much faster training times than other approaches. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Comparison <s> It has been established that incorporating word cluster features derived from large unlabeled corpora can significantly improve prediction of linguistic structure. While previous work has focused primarily on English, we extend these results to other languages along two dimensions. First, we show that these results hold true for a number of languages across families. Second, and more interestingly, we provide an algorithm for inducing cross-lingual clusters and we show that features derived from these clusters significantly improve the accuracy of cross-lingual structure prediction. Specifically, we show that by augmenting direct-transfer systems with cross-lingual cluster features, the relative error of delexicalized dependency parsers, trained on English treebanks and transferred to foreign languages, can be reduced by up to 13%. When applying the same method to direct transfer of named-entity recognizers, we observe relative improvements of up to 26%. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Comparison <s> We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. We apply these to the problem of modeling phone sequences---a domain in which universal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representations, and extrinsic evaluation in two downstream applications that make use of phonetic features show (i) that polyglot models better generalize to held-out data than comparable monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually. <s> BIB003
In light of the performance of the described methods, to what extent can typological features benefit downstream NLP tasks and applications? To answer this key question, consider the performance scores of each model reported in Figure 10 . Each model has been evaluated in the original paper in one (or more) of the three main settings, with otherwise identical architecture and hyper-parameters: with gold database features (Typology), with latently inferred typological features (Data-driven), or without both (Baseline). It is evident that typology-enriched models consistently outperform baselines across several NLP tasks. Indeed, the scores are higher for metrics that increase (Unlabeled Attachment Score, F1 Score, and BLEU) and lower for metrics that decrease (Word Error Rate, Mean Average Error, and Perplexity) with better predictions. Nevertheless, improvements tend to be moderate, and only a small number of experiments support them with statistical significance tests. In general, it appears that they fall short of the potential usefulness of typology: in § 6 we analyze the possible reasons for this. Some of the experiments we have surveyed investigate the effect of substituting typological features with features related to Genealogy and Language Identity (e.g., one-hot encoding of languages). Based on the results in Figure 10 , it is unclear whether typology should be preferred, as it is sometimes rivaled by other types of features. In particular, it is typology that excels according to BIB003 , genealogy according to BIB001 and BIB002 , and language identity according to  . However, drawing conclusions from the last experiment seems incautious: In § 4.2, we argued that their selection of features (presented in Figure 5 ) is debatable because of low diversification or noise. Moreover, it should be emphasized that one-hot language encoding is limited to the Feature Baseline Data−driven Genealogy Language Identity Typology