Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> A log slasher includes an elongated log receiving bunk and a saw assembly movable longitudinally along the bunk to saw logs supported in the bunk into shortened lengths. The saw is pivotally supported at one end on a traveling trolley for movement with the trolley along the bunk and in a vertical plane toward and away from the bunk. A guide moves with the saw to hold the saw against whip as the saw is moving into, and away from, a cut. A lift crane lifts logs into and out of the bunk and includes clam fingers having opposite angled surfaces adjacent the free ends thereof for engaging logs of different diameter. The trolley drive and that for moving the saw toward and away from the bunk are carried on the trolley and an articulated arm assembly, extendible and contractible in response to trolley movement, supports the power supply conduits in a compact arrangement throughout the range of movement of the trolley and saw. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> A standard form of analysis for linguistic typology is the universal implication. These implications state facts about the range of extant languages, such as “if objects come after verbs, then adjectives come after nouns.” Such implications are typically discovered by painstaking hand analysis over a small sample of languages. We propose a computational model for assisting at this process. Our model is able to discover both well-known implications as well as some novel implications that deserve further study. Moreover, through a careful application of hierarchical analysis, we are able to cope with the well-known sampling problem: languages are not independent. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> In this paper we explore the potential for identifying computationally relevant typological features from a multilingual corpus of language data built from readily available language data collected off the Web. Our work builds on previous structural projection work, where we extend the work of projection to building individual CFGs for approximately 100 languages. We then use the CFGs to discover the values of typological parameters such as word order, the presence or absence of definite and indefinite determiners, etc. Our methods have the potential of being extended to many more languages and parameters, and can have significant effects on current research focused on tool and resource development for low-density languages and grammar induction from raw corpora. <s> BIB003 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> Abstract Word-order typology often uses the linear order of binary grammatical pairs in sentences to classify a language. The present paper proposes a method based on dependency treebanks as a typological means. This paper investigates 20 languages using treebanks with different sizes from 16 K to 1 million dependencies. The results show that some languages are more head-initial or head-final than others, but all contain head-initial and head-final elements. The 20 languages can be arranged on a continuum with complete head-initial and head-final patterns as the two ends. Some data about subject–verb, object–verb and adjective–noun are extracted from the treebanks for comparison with the typological studies based on the traditional means, the results are similar. The investigation demonstrates that the proposed method is valid for positioning a language in the typological continuum and the resources from computational linguistics can also be used in language typology. <s> BIB004 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> We propose to bring together two kinds of linguistic resources—interlinear glossed text (IGT) and a language-independent precision grammar resource—to automatically create precision grammars in the context of language documentation. This paper takes the first steps in that direction by extracting major-constituent word order and case system properties from IGT for a diverse sample of languages. <s> BIB005 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> In this paper we propose a probabilistic graphical model as an innovative framework for studying typological universals. We view language as a system and linguistic features as its components whose relationships are encoded in a Directed Acyclic Graph (DAG). Taking discovery of the word order universals as a knowledge discovery task we learn the graphical representation of a word order sub-system which reveals a finer structure such as direct and indirect dependencies among word order features. Then probabilistic inference enables us to see the strength of such relationships: given the observed value of one feature (or combination of features), the probabilities of values of other features can be calculated. Our model is not restricted to using only two values of a feature. Using imputation technique and EM algorithm it can handle missing values well. Model averaging technique solves the problem of limited data. In addition the incremental and divide-and-conquer method addresses the areal and genetic effects simultaneously instead of separately as in Daume III and Campbell (2007). <s> BIB006 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no changes to the model architecture from a standard NMT system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model. On the WMT’14 benchmarks, a single multilingual model achieves comparable performance for English→French and surpasses state-of-the-art results for English→German. Similarly, a single multilingual model surpasses state-of-the-art results for French→English and German→English on WMT’14 and WMT’15 benchmarks, respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. Our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages. <s> BIB007 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> One central mystery of neural NLP is what neural models "know" about their subject matter. When a neural machine translation system learns to translate from one language to another, does it learn the syntax or semantics of the languages? Can this knowledge be extracted from the system to fill holes in human scientific knowledge? Existing typological databases contain relatively full feature specifications for only a few hundred languages. Exploiting the existence of parallel texts in more than a thousand languages, we build a massive many-to-one neural machine translation (NMT) system from 1017 languages into English, and use this to predict information missing from typological databases. Experiments show that the proposed method is able to infer not only syntactic, but also phonological and phonetic inventory features, and improves over a baseline that has access to information about the languages' geographic and phylogenetic neighbors. <s> BIB008 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> We show how to predict the basic word-order facts of a novel language given only a corpus of part-of-speech (POS) sequences. We predict how often direct objects follow their verbs, how often adjectives follow their nouns, and in general the directionalities of all dependency relations. Such typological properties could be helpful in grammar induction. While such a problem is usually regarded as unsupervised learning, our innovation is to treat it as supervised learning, using a large collection of realistic synthetic languages as training data. The supervised learner must identify surface features of a language's POS sequence (hand-engineered or neural features) that correlate with the language's deeper structure (latent trees). In the experiment, we show: 1) Given a small set of real languages, it helps to add many synthetic languages to the training data. 2) Our system is robust even when the POS sequences include noise. 3) Our system on this task outperforms a grammar induction baseline by a large margin. <s> BIB009 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Heuristics Based on Morphosyntactic Annotation. <s> A core part of linguistic typology is the classification of languages according to linguistic properties, such as those detailed in the World Atlas of Language Structure (WALS). Doing this manually is prohibitively time-consuming, which is in part evidenced by the fact that only 100 out of over 7,000 languages spoken in the world are fully covered in WALS. ::: We learn distributed language representations, which can be used to predict typological properties on a massively multilingual scale. Additionally, quantitative and qualitative analyses of these language embeddings can tell us how language similarities are encoded in NLP models for tasks at different typological levels. The representations are learned in an unsupervised manner alongside tasks at three typological levels: phonology (grapheme-to-phoneme prediction, and phoneme reconstruction), morphology (morphological inflection), and syntax (part-of-speech tagging). ::: We consider more than 800 languages and find significant differences in the language representations encoded, depending on the target task. For instance, although Norwegian Bokm{\aa}l and Danish are typologically close to one another, they are phonologically distant, which is reflected in their language embeddings growing relatively distant in a phonological task. We are also able to predict typological features in WALS with high accuracies, even for unseen language families. <s> BIB010
Morphosyntactic feature values can be extracted via heuristics from morphologically and syntactically annotated texts. For example, word order features can be calculated by counting the average direction of dependency relations or constituency hierarchies BIB004 . Consider the tree of a sentence in Welsh from BIB005 in Figure 6 . The relative order of verbsubject, and verb-object can be deduced from the position of the relevant nodes VBD, NN S , and NN O (highlighted). Morphosyntactic annotation is often unavailable for resource-lean languages. In such cases, it can be projected from a source language to a target language through language transfer. For instance, Östling (2015) projects source morphosyntactic annotation directly to several languages through a multilingual word alignment. After the alignment and projection, word order features are calculated by the average direction of dependency relations. Similarly,  transfer PoS annotation with a model transfer technique relying on multilingual embeddings, created through monolingual mapping (see § 3.3). After the projection, they predict feature values with a multiclass support vector machine using PoS tag n-gram features. Finally, typological information can be extracted from Interlinear Glossed Texts (IGT). Such collections of example sentences are collated by linguists and contain grammatical glosses with morphological information. These can guide alignment between the example sentence and its English translation. BIB003 and BIB005 project chunking information from English and train context free grammars on target languages. After collapsing identical rules, they arrange them by frequency and infer word order features. Language similarity measures can also rely on a distributed representation of each language. These language vectors are trained end-to-end as part of neural models for downstream tasks such as many-to-one Neural Machine Translation (NMT). In particular, language vectors can be obtained from artificial trainable tokens concatenated to every input sentence, similar to BIB007 , or from the aggregated values of the hidden states of a neural encoder. Using these language representations, typological feature values are propagated using k nearest neighbors BIB010 or predicted with logistic regression BIB008 . Language vectors can be conceived as data-driven, continuous typological representations of a language, and as such provide an alternative to manually crafted typological representations. Similar to the analysis carried out in § 4.2, we can investigate how much language vectors align with genealogical information. Figure 7 compares continuous representations based on artificial tokens (Figure 7(a) ) and encoder hidden states (Figure 7(b) ) with vectors of discrete WALS features from URIEL (Figure 7(c) ). All the representations are reduced to two dimensions with t-Distributed Stochastic Neighbor Embedding (t-SNE), and color-coded based on their language family. As the plots demonstrate, the information encoded in WALS vectors is akin to genealogical information, partly because of biases introduced by family-based propagation of missing values (Littel, Mortensen, and Levin 2016) (see § 4.3.2). On the other hand, artificial tokens and encoder hidden states cannot be reduced to genealogical clusters. Yet, their ability to predict missing values is not inferior to WALS features (as detailed in § 4.3.5). This implies that discrete and continuous representations appear to capture different aspects of the cross-lingual variation, while both being informative. For this reason, they are possibly complementary and could be combined in the future. As an alternative to unsupervised propagation, one can learn an explicit model for predicting feature values through supervised classification. For instance, Takamura, Nagata, and Kawasaki (2016) use logistic regression with WALS features and evaluate this model in a cross-validation setting where one language is held out in each fold. BIB009 provide supervision to a feed-forward neural network with windows of PoS tags from natural and synthetic corpora. Supervised learning of typology can also be guided by non-typological information (see § 2). Within the Bayesian framework, Murawaki (2017) exploits not only typological but also genealogical and areal dependencies among languages to represent each language as a binary latent parameter vector through a series of autologistic models. Eisner (2017, 2018 ) develop a point-process generative model of vowel inventories (represented as either IPA symbols or acoustic formants) based on some universal cognitive principles: dispersion (phonemes are as spread out as possible in the acoustic space) and focalization (some positions in the acoustic space are preferred due to the similarity of the main formants). An alternative approach to supervised prediction of typology is based on learning implicational universals of the kind pioneered by BIB001 , with probabilistic models from existing typological databases. Using such universals, features can be deduced by modus ponens. For instance, once it has been established that the presence of "High consonant/vowel ratio" and "No front-rounded vowels" implies "No tones," the latter feature value can be deduced from the premises if those are known. BIB002 propose a Bayesian model for learning typological universals that predicts implications between features based on the intuition that their likelihood does not equal their prior probability, but rather is constrained by other features. BIB006 casts this problem as knowledge discovery, where language features are encoded in a directed acyclic graph. The strength of implication universals is represented as weights associated with the edges of this graph.