Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 4 <s> Recent studies have shown the potential benefits of leveraging resources for resource-rich languages to build tools for similar, but resource-poor languages. We examine what constitutes "similarity" by comparing traditional phylogenetic language groups, which are motivated largely by genetic relationships, with language groupings formed by clustering methods using typological features only. Using data from the World Atlas of Language Structures (WALS), our preliminary experiments show that typologically-based clusters look quite different from genetic groups, but perform as good or better when used to predict feature values of member languages. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 4 <s> We introduce polyglot language models, recurrent neural network models trained to predict symbol sequences in many different languages using shared representations of symbols and conditioning on typological information about the language to be predicted. We apply these to the problem of modeling phone sequences---a domain in which universal symbol inventories and cross-linguistically shared feature representations are a natural fit. Intrinsic evaluation on held-out perplexity, qualitative analysis of the learned representations, and extrinsic evaluation in two downstream applications that make use of phonetic features show (i) that polyglot models better generalize to held-out data than comparable monolingual models and (ii) that polyglot phonetic feature representations are of higher quality than those learned monolingually. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 4 <s> This work examines the impact of crosslinguistic transfer on grammatical errors in English as Second Language (ESL) texts. Using a computational framework that formalizes the theory of Contrastive Analysis (CA), we demonstrate that language specific error distributions in ESL writing can be predicted from the typological properties of the native language and their relation to the typology of English. Our typology driven model enables to obtain accurate estimates of such distributions without access to any ESL data for the target languages. Furthermore, we present a strategy for adjusting our method to low-resource languages that lack typological documentation using a bootstrapping approach which approximates native language typology from ESL texts. Finally, we show that our framework is instrumental for linguistic inquiry seeking to identify first language factors that contribute to a wide range of difficulties in second language acquisition. <s> BIB003
Feature sets used in a sample of typologically informed experiments for dependency parsing. The numbers refer to WALS ordering (Dryer and Haspelmath 2013). features in WALS with the exception of phonological features. BIB002 selected 190 binarized phonological features from URIEL (Littel, Mortensen, and Levin 2016) . These features encoded the presence of single segments, classes of segments, minimal contrasts in a language inventory, and the number of segments in a class. For instance, they record whether a language allows two sounds to differ only in voicing, such as /t/ and /d/. Finally, a small number of experiments adopted the entire feature inventory of typological databases, without any sort of pre-selection. In particular,  and  extracted all the features in WALS, whereas  extracted all the features in URIEL.  did not resort to basic typological features, but rather to "several hundred [implicational universals] applicable to syntax" drawn from the Universal Archive  . Typological attributes that are extracted from typological databases are typically represented as feature vectors in which each dimension encodes a feature value. This feature representation is often binarized BIB001 : For each possible value v of each database attribute a, a new feature is created with value 1 if it corresponds to the actual value for a specific language and 0 otherwise. Note that this increases the number of features by a factor of 1 ||a|| ||a|| i=1 ||v a i ||. Although binarization helps harmonizing different features and different databases, it muddies the different types of typological variables. To what extent do the limitations of typological databases mentioned in § 4.1 affect the feature sets surveyed in this section? The coverage is generally broad for the languages used in these experiments, as they tend to be well-documented. For instance, on average, 79.8% of the feature values are populated for the 14 languages appearing in BIB003 , as opposed to 17% for all the languages in WALS. It is hard to assess at a large scale how informative a set of typological features is. However, these can be meaningfully compared with genealogical information. Ideally, these two properties should not be completely equivalent (otherwise they would be redundant), 5 but at the same time they should partly overlap (language cognates inherit typological properties from the same ancestors). In Figure 5 , we show two feature sets appearing in  , each depicted as a heatmap. Each row represents a language in the data; each cell is colored according to the feature value, ranging from 0 to 1. In particular, the feature set of Figure 5 In order to compare the similarities of the typological feature vectors among languages, we clustered languages hierarchically based on such vectors. 6 Intuitively, the more this hierarchy resembles their actual family tree, the more redundant the typological information is. This is the case for Figure 5(b) , where the lowest-lever clusters correspond exactly to a genus or family (top-down: Romance, Slavic, Germanic, Celtic, Uralic). Still, the language vectors belonging to the same cluster display some microvariations in individual features. On the other hand, 5(a) shows clusters differing from language genealogy: for instance, English and Czech are merged, although they belong to different genera (Germanic and Slavic). However, this feature set fails to account for fine-grained differences among related languages: For instance, French, Spanish, and Italian receive the same encoding. 7 To sum up, this section's survey on typological feature sets reveals that most experiments have taken into account a small number of databases and features therein. However, several studies did utilize a larger set of coherent features or full databases. Although well-documented languages do not suffer much from coverage issues, we showed how difficult it is to select typological features that are non-redundant with genealogy, fully discriminative, and informative. The next section addresses these problems, proposing automatic prediction as a solution.