Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> Recent studies have shown the potential benefits of leveraging resources for resource-rich languages to build tools for similar, but resource-poor languages. We examine what constitutes "similarity" by comparing traditional phylogenetic language groups, which are motivated largely by genetic relationships, with language groupings formed by clustering methods using typological features only. Using data from the World Atlas of Language Structures (WALS), our preliminary experiments show that typologically-based clusters look quite different from genetic groups, but perform as good or better when used to predict feature values of member languages. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> Abstract This paper discusses a multidimensional probabilistic semantic map of lexical motion verb stems based on data collected from parallel texts (viz. translations of the Gospel according to Mark) for 100 languages from all continents. The crosslinguistic diversity of lexical semantics in motion verbs is illustrated in detail for the domain of ‘go’, ‘come’, and ‘arrive’ type contexts. It is argued that the theoretical bases underlying probabilistic semantic maps from exemplar data are the isomorphism hypothesis (given any two meanings and their corresponding forms in any particular language, more similar meanings are more likely to be expressed by the same form in any language), similarity semantics (similarity is more basic than identity), and exemplar semantics (exemplar meaning is more fundamental than abstract concepts). <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> We propose to bring together two kinds of linguistic resources—interlinear glossed text (IGT) and a language-independent precision grammar resource—to automatically create precision grammars in the context of language documentation. This paper takes the first steps in that direction by extracting major-constituent word order and case system properties from IGT for a diverse sample of languages. <s> BIB003 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> In this paper we propose a probabilistic graphical model as an innovative framework for studying typological universals. We view language as a system and linguistic features as its components whose relationships are encoded in a Directed Acyclic Graph (DAG). Taking discovery of the word order universals as a knowledge discovery task we learn the graphical representation of a word order sub-system which reveals a finer structure such as direct and indirect dependencies among word order features. Then probabilistic inference enables us to see the strength of such relationships: given the observed value of one feature (or combination of features), the probabilities of values of other features can be calculated. Our model is not restricted to using only two values of a feature. Using imputation technique and EM algorithm it can handle missing values well. Model averaging technique solves the problem of limited data. In addition the incremental and divide-and-conquer method addresses the areal and genetic effects simultaneously instead of separately as in Daume III and Campbell (2007). <s> BIB004 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> This paper presents a comparison of classification methods for linguistic typology for the purpose of expanding an extensive, but sparse language resource: the World Atlas of Language Structures (WALS) (Dryer and Haspelmath, 2013). We experimented with a variety of regression and nearest-neighbor methods for use in classification over a set of 325 languages and six syntactic rules drawn from WALS. To classify each rule, we consider the typological features of the other five rules; linguistic features extracted from a word-aligned Bible in each language; and genealogical features (genus and family) of each language. In general, we find that propagating the majority label among all languages of the same genus achieves the best accuracy in label pre- diction. Following this, a logistic regression model that combines typological and linguistic features offers the next best performance. Interestingly, this model actually outperforms the majority labels among all languages of the same family. <s> BIB005 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> We show how to predict the basic word-order facts of a novel language given only a corpus of part-of-speech (POS) sequences. We predict how often direct objects follow their verbs, how often adjectives follow their nouns, and in general the directionalities of all dependency relations. Such typological properties could be helpful in grammar induction. While such a problem is usually regarded as unsupervised learning, our innovation is to treat it as supervised learning, using a large collection of realistic synthetic languages as training data. The supervised learner must identify surface features of a language's POS sequence (hand-engineered or neural features) that correlate with the language's deeper structure (latent trees). In the experiment, we show: 1) Given a small set of real languages, it helps to add many synthetic languages to the training data. 2) Our system is robust even when the POS sequences include noise. 3) Our system on this task outperforms a grammar induction baseline by a large margin. <s> BIB006 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 9 <s> Linguistic typology studies the range of structures present in human language. The main goal of the field is to discover which sets of possible phenomena are universal, and which are merely frequent. For example, all languages have vowels, while most---but not all---languages have an /u/ sound. In this paper we present the first probabilistic treatment of a basic question in phonological typology: What makes a natural vowel inventory? We introduce a series of deep stochastic point processes, and contrast them with previous computational, simulation-based approaches. We provide a comprehensive suite of experiments on over 200 distinct languages. <s> BIB007
Accuracy of different approaches (see legend on the right) in predicting missing values of WALS typological features (specified on the vertical axis). of predicting each feature relative to a specific strategy: For instance, the correct value of passive voice is harder to predict than word order, as claimed by BIB003 and seen in Figure 9 . However, some papers carry out comparisons of the different strategies within the same experimental setting. According to BIB005 , propagation from the genus majority value outperforms logistic regression among word-order typological features. On the other hand, Georgi, Xia, and BIB001 argue that typologybased clusters are to be preferred in general. This apparent contradiction stems from the nature of the target features: Genealogy excels in word order features because of their diachronic stability. As they tend to be preserved over time, they are often shared by all members of a family. In turn, majority value propagation is surpassed by supervised classification when evaluated on the entire WALS feature set  . In general, there appears to be no "one-size-fits-all" algorithm. For instance, BIB005 outperform BIB006 for object-verb order (83A) but are inferior to it for adposition-noun (85A). In fact, each strategy is suited for different features, and requires different resources. Based on Figure 9 , the extraction of information from morphosyntactic annotation is well suited for word order features, whereas distributional heuristics from multi-parallel texts are more informative about lexicalization patterns. On the other hand, unsupervised propagation and supervised learning are general-purpose strategies. Moreover, the first two presuppose some annotated and/or parallel texts, whereas the second two need pre-existing database documentation. Strategies may be preferred, according to which resources are available for a specific language. Many strategies have a common weakness, however, as they postulate incorrectly that language samples are independent and identically distributed BIB004 BIB007 . This is not the case, due to the interactions of family, area, and implicational universals. The solutions adopted to mitigate this weakness vary: BIB006 balance the data distribution with synthetic examples, whereas Takamura, Nagata, and Kawasaki (2016) model family and area interactions explicitly. However, according to Murawaki (2017), these interactions have different degrees of impact on typological features. In particular, inter-feature dependencies are more influential than inter-language dependencies, and horizontal diffusibility (borrowing from neighbors) is more prominent than vertical stability (inheriting from ancestors). Finally, a potential direction for future investigation emerges from this section's survey. In addition to missing value completion, automatic prediction often also accounts for the variation internal to each language. However, some strategies go even further, and "open the way for a typology where generalizations can be made without there being any need to reduce the attested diversity of categorization patterns to discrete types" BIB002 . In fact, language vectors (Malaviya, Neubig, and Littell 2017; Bjerva and Augenstein 2018) and distributional information from multiparallel texts (Asgari and Schütze 2017) are promising insofar they capture latent properties of languages in a bottom-up fashion, preserving their gradient nature. This offers an alternative to hand-crafted database features: In § 6.3 we make a case for integrating continuous, data-driven typological representations into NLP algorithms.