Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> A New Typology: Gradience and Context-Sensitivity <s> One central mystery of neural NLP is what neural models "know" about their subject matter. When a neural machine translation system learns to translate from one language to another, does it learn the syntax or semantics of the languages? Can this knowledge be extracted from the system to fill holes in human scientific knowledge? Existing typological databases contain relatively full feature specifications for only a few hundred languages. Exploiting the existence of parallel texts in more than a thousand languages, we build a massive many-to-one neural machine translation (NMT) system from 1017 languages into English, and use this to predict information missing from typological databases. Experiments show that the proposed method is able to infer not only syntactic, but also phonological and phonetic inventory features, and improves over a baseline that has access to information about the languages' geographic and phylogenetic neighbors. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> A New Typology: Gradience and Context-Sensitivity <s> A core part of linguistic typology is the classification of languages according to linguistic properties, such as those detailed in the World Atlas of Language Structure (WALS). Doing this manually is prohibitively time-consuming, which is in part evidenced by the fact that only 100 out of over 7,000 languages spoken in the world are fully covered in WALS. ::: We learn distributed language representations, which can be used to predict typological properties on a massively multilingual scale. Additionally, quantitative and qualitative analyses of these language embeddings can tell us how language similarities are encoded in NLP models for tasks at different typological levels. The representations are learned in an unsupervised manner alongside tasks at three typological levels: phonology (grapheme-to-phoneme prediction, and phoneme reconstruction), morphology (morphological inflection), and syntax (part-of-speech tagging). ::: We consider more than 800 languages and find significant differences in the language representations encoded, depending on the target task. For instance, although Norwegian Bokm{\aa}l and Danish are typologically close to one another, they are phonologically distant, which is reflected in their language embeddings growing relatively distant in a phonological task. We are also able to predict typological features in WALS with high accuracies, even for unseen language families. <s> BIB002
As shown in § 4.2, most of the typology-savvy algorithms thus far exploited features extracted from manually crafted databases. However, this approach is riddled with several shortcomings, which are reflected in the small performance improvements observed in § 5.4. Luckily, these shortcomings may potentially be averted through the use of methods that allow typological information to emerge from the data in a bottom-up fashion, rather than being predetermined. In what follows we advocate for such a datadriven approach, based on several considerations. First, typological databases provide incomplete documentation of the cross-lingual variation, in terms of features and languages. Raw textual data, which is easily accessible for many languages and is cost-effective, may provide a valid alternative that can facilitate automatic learning of more complete knowledge. Second, database information is approximate, as it is restricted to the majority strategy within a language. However, in theory each language allows for multiple strategies in different contexts and with different frequencies, hence databases risk hindering models from learning less-likely but plausible patterns  . Inferring typological information from text would enable a system to discover patterns within individual examples, including both the frequent and the infrequent ones. Thirdly, typological features in databases are discrete, utilizing predefined categories devised to make high-level generalizations across languages. However, several categories in natural language are gradient (see for instance the discussion on semantic categorization in § 2), hence they are better captured by continuous features. In addition to being psychologically motivated, this sort of gradient representation is also more compatible with machine learning algorithms and particularly with deep neural models that naturally operate with real-valued multidimensional word embeddings and hidden states. To sum up, the automatic development of typological information and its possible integration into machine learning algorithms have the potential to solve an important bottleneck in polyglot NLP. Current manually curated databases consist of incomplete, approximate, and discrete features that are intended to reflect contextual and gradient information implicitly present in text. These features are fed to continuous, probabilistic, and contextual machine learning models-which do not form a natural fit for the typological features. Instead, we believe that modeling cross-lingual variation directly from textual data can yield typological information that is more suitable for machine learning. Several techniques surveyed in § 4.3 are suited to serve this purpose. In particular, the extraction from morphosyntactic annotation (Liu 2010, inter alia) and alignments from multi-parallel texts (Asgari and Schütze 2017, inter alia) provide information about typological constructions at the level of individual examples. Moreover, language vectors BIB001 BIB002 and alignments from multi-parallel texts preserve the gradient nature of typology through continuous representations. The successful integration of these components would affect the way multilingual feature engineering is performed. As opposed to using binary vectors of typological features, the information about language-internal variation could be encoded as realvalued vectors, where each dimension is a possible strategy for a given construction and its relative frequency within a language. As an alternative, selective sharing and multilingual biasing could be performed at the level of individual examples rather than languages as a whole. In particular, model parameters could be transferred among similar examples; and input/hidden representations could be conditioned on contextual typological patterns. Finally, focusing on the various instantiations of a particular type rather than considering languages as indissoluble blocks would enhance data selection, similar to what Søgaard (2011) achieved using PoS n-grams for similarity measurement. The selection of similar sentences rather than similar languages as source data in language transfer is likely to yield large improvements, as demonstrated by  for parsing in an oracle setting. Finally, the bottom-up development of typological features may also address radically resource-less languages that lack even raw textual data in a digital format. For this group, which still constitutes a large portion of the world's languages, there are often available reference grammars written by field linguists, which are the ultimate source for typological databases. These grammars could be queried automatically, and fine-grained typological information could be harvested through information extraction techniques.