Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> Since its first publication, "Language Universals and Linguistic Typology" has become established as the leading introductory account of one of the most productive areas of linguistics-the analysis, comparison, and classification of the common features and forms of the organization of languages. Adopting an approach to the subject pioneered by Greenberg and others, Bernard Comrie is particularly concerned with syntactico-semantic universals, devoting chapters to word order, case making, relative clauses, and causative constructions. His book is informed throughout by the conviction that an exemplary account of universal properties of human language cannot restrict itself to purely formal aspects, nor focus on analysis of a single language. Rather, it must also consider language use, relate formal properties to testable claims about cognition and cognitive development, and treat data from a wide range of languages. This second edition has been revised and updated to take full account of new research in universals and typology in the past decade, and more generally to consider how the approach advocated here relates to recent advances in generative grammatical theory. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> 1. Introduction 2. Typological classification 3. Implicational universals 4. Markedness in typology 5. Grammatical hierarchies 6. Prototypes and the interaction of typological patterns 7. External motives and the typology of form-function relations 8. Diachronic typology 9. Linguistic explanation in the dynamic paradigm Notes References Map of languages cited Author index Language index Subject index. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> This paper considers the problem of automatically inducing role-semantic annotations in the FrameNet paradigm for new languages. We introduce a general framework for semantic projection which exploits parallel texts, is relatively inexpensive and can potentially reduce the amount of effort involved in creating semantic resources. We propose projection models that exploit lexical and syntactic information. Experimental results on an English-German parallel corpus demonstrate the advantages of this approach. <s> BIB003 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> For centuries, the deep connection between languages has brought about major discoveries about human communication. In this paper we investigate how this powerful source of information can be exploited for unsupervised language learning. In particular, we study the task of morphological segmentation of multiple languages. We present a nonparametric Bayesian model that jointly induces morpheme segmentations of each language under consideration and at the same time identifies cross-lingual morpheme patterns, or abstract morphemes. We apply our model to three Semitic languages: Arabic, Hebrew, Aramaic, as well as to English. Our results demonstrate that learning morphological models in tandem reduces error by up to 24% relative to monolingual models. Furthermore, we provide evidence that our joint model achieves better performance when applied to languages from the same family. <s> BIB004 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> In this position paper, I argue that in order to create truly language-independent NLP systems, we need to incorporate linguistic knowledge. The linguistic knowledge in question is not intricate rule systems, but generalizations from linguistic typology about the range of variation in linguistic structures across languages. <s> BIB005 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> For centuries, scholars have explored the deep links among human languages. In this thesis, we present a class of probabilistic models that exploit these links as a form of naturally occurring supervision. These models allow us to substantially improve performance for core text processing tasks, such as morphological segmentation, part-of-speech tagging, and syntactic parsing. Besides these traditional NLP tasks, we also present a multilingual model for lost language decipherment. We test this model on the ancient Ugaritic language. Our results show that we can automatically uncover much of the historical relationship between Ugaritic and Biblical Hebrew, a known related language. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.) <s> BIB006 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> Language independence is commonly presented as one of the advantages of modern, machine-learning approaches to NLP, and it is an important type of scalability. In this position paper, I critically review the widespread approaches to achieving and evaluating language independence in the field of com- putational linguistics and argue that, on the one hand, we are not truly evaluating language independence with any systematicity and on the other hand, that truly language-independent technology requires more linguistic sophistication than is the norm. <s> BIB007 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> We describe a novel approach for inducing unsupervised part-of-speech taggers for languages that have no labeled training data, but have translated text in a resource-rich language. Our method does not assume any knowledge about the target language (in particular no tagging dictionary is assumed), making it applicable to a wide array of resource-poor languages. We use graph-based label propagation for cross-lingual knowledge transfer and use the projected labels as features in an unsupervised model (Berg-Kirkpatrick et al., 2010). Across eight European languages, our approach results in an average absolute improvement of 10.4% over a state-of-the-art baseline, and 16.7% over vanilla hidden Markov models induced with the Expectation Maximization algorithm. <s> BIB008 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> Abstract This paper explores the ways in which the field of natural language processing (NLP) can and does benefit from work in linguistic typology. I describe the recent increase in interest in multilingual natural language processing and give a high-level overview of the field. I then turn to a discussion of how linguistic knowledge in general is incorporated in NLP technology before describing how typological results in particular are used. I consider both rule-based and machine learning approaches to NLP and review literature on predicting typological features as well as that which leverages such features. <s> BIB009 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Introduction <s> Hierarchical attention networks have recently achieved remarkable performance for document classification in a given language. However, when multilingual document collections are considered, training such models separately for each language entails linear parameter growth and lack of cross-language transfer. Learning a single multilingual model with fewer parameters is therefore a challenging but potentially beneficial objective. To this end, we propose multilingual hierarchical attention networks for learning document structures, with shared encoders and/or shared attention mechanisms across languages, using multi-task learning and an aligned semantic space as input. We evaluate the proposed models on multilingual document classification with disjoint label sets, on a large dataset which we provide, with 600k news documents in 8 languages, and 5k labels. The multilingual models outperform monolingual ones in low-resource as well as full-resource settings, and use fewer parameters, thus confirming their computational efficiency and the utility of cross-language transfer. <s> BIB010
The world's languages may share universal features at a deep, abstract level, but the structures found in real-world, surface-level texts can vary significantly. This crosslingual variation has challenged the development of robust, multilingually applicable Natural Language Processing (NLP) technology, and as a consequence, existing NLP is still largely limited to a handful of resource-rich languages. The architecture design, training, and hyper-parameter tuning of most current algorithms are far from being language-agnostic, and often inadvertently incorporate language-specific biases BIB005 BIB007 . In addition, most state-of-the-art machine learning models rely on supervision from (large amounts of) labeled data-a requirement that cannot be met for the majority of the world's languages BIB006 . Over time, approaches have been developed to address the data bottleneck in multilingual NLP. These include unsupervised models that do not rely on the availability of manually annotated resources BIB004 Vulić, De Smet, and Moens 2011, inter alia) and techniques that transfer data or models from resource-rich to resource-poor languages BIB003 BIB008 Täckström, McDonald, and Uszkoreit 2012, inter alia) . Some multilingual applications, such as Neural Machine Translation and Information Retrieval, have been facilitated by learning joint models that learn from several languages  Johnson et al. 2017, inter alia) or via multilingual distributed representations of words and sentences (Mikolov, Le, and Sutskever 2013, inter alia) . Such techniques can lead to significant improvements in performance and parameter efficiency over monolingual baselines BIB010 . Another, highly promising source of information for modeling cross-lingual variation can be found in the field of Linguistic Typology. This discipline aims to systematically compare and document the world's languages based on the empirical observation of their variation with respect to cross-lingual benchmarks BIB001 BIB002 . Research efforts in this field have resulted in large typological databasesfor example, most prominently the World Atlas of Language Structures (WALS) (Dryer Submission received: 30 June 2018; revised version received: 20 March 2019 ; accepted for publication: 12 June 2019. https://doi.org/10.1162/COLI_a_00357 560 . Such databases can serve as a source of guidance for feature choice, algorithm design, and data selection or generation in multilingual NLP. Previous surveys on this topic have covered earlier research integrating typological knowledge into NLP  BIB009 . However, there is still no consensus on the general effectiveness of this approach. For instance,  has argued that data-driven machine learning should not need to commit to any assumptions about categorical and manually defined language types as defined in typological databases. In this article, we provide an extensive survey of typologically informed NLP methods to date, including the more recent neural approaches not previously surveyed in this area. We consider the impact of typological (including both structural and semantic) information on system performance and discuss the optimal sources for such information. Traditionally, typological information has been obtained from hand-crafted databases and, therefore, it tends to be coarse-grained and incomplete. Recent research has focused on inferring typological information automatically from multilingual data (Asgari and Schütze 2017, inter alia) , with the specific purpose of obtaining a more complete and finer-grained set of feature values. We survey these techniques and discuss ways to integrate their predictions into the current NLP algorithms. To the best of our knowledge, this has not yet been covered in the existing literature. In short, the key questions our paper addresses can be summarized as follows: (i) Which NLP tasks and applications can benefit from typology? (ii) What are the advantages and limitations of currently available typological databases? Can data-driven inference of typological features offer an alternative source of information? (iii) Which methods have been proposed to incorporate typological information in NLP systems, and how should such information be encoded? (iv) To what extent does the performance of typology-savvy methods surpass typology-agnostic baselines? How does typology compare with other criteria of language classification, such as genealogy? (v) How can typology be harnessed for data selection, rule-based systems, and model interpretation? We start this survey with a brief overview of Linguistic Typology ( § 2) and multilingual NLP ( § 3). After these introductory sections we proceed to examine the development of typological information for NLP, including that in hand-crafted typological databases and that derived through automatic inference from linguistic data ( § 4). In the same section, we also describe typological features commonly selected for application in NLP. In § 5 we discuss ways in which typological information has been integrated into NLP algorithms, identifying the main trends and comparing the performance of a range of methods. Finally, in § 6 we discuss the current limitations in the use of typology in NLP and propose novel research directions inspired by our findings.