Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 10 <s> It has been established that incorporating word cluster features derived from large unlabeled corpora can significantly improve prediction of linguistic structure. While previous work has focused primarily on English, we extend these results to other languages along two dimensions. First, we show that these results hold true for a number of languages across families. Second, and more interestingly, we provide an algorithm for inducing cross-lingual clusters and we show that features derived from these clusters significantly improve the accuracy of cross-lingual structure prediction. Specifically, we show that by augmenting direct-transfer systems with cross-lingual cluster features, the relative error of delexicalized dependency parsers, trained on English treebanks and transferred to foreign languages, can be reduced by up to 13%. When applying the same method to direct transfer of named-entity recognizers, we observe relative improvements of up to 26%. <s> BIB001 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 10 <s> We present a novel algorithm for multilingual dependency parsing that uses annotations from a diverse set of source languages to parse a new unannotated language. Our motivation is to broaden the advantages of multilingual learning to languages that exhibit significant differences from existing resource-rich languages. The algorithm learns which aspects of the source languages are relevant for the target language and ties model parameters accordingly. The model factorizes the process of generating a dependency tree into two steps: selection of syntactic dependents and their ordering. Being largely language-universal, the selection component is learned in a supervised fashion from all the training languages. In contrast, the ordering decisions are only influenced by languages with similar properties. We systematically model this cross-lingual sharing using typological features. In our experiments, the model consistently outperforms a state-of-the-art multi-lingual parser. The largest improvement is achieved on the non Indo-European languages yielding a gain of 14.4%. <s> BIB002 </s> Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing <s> Figure 10 <s> Accurate multilingual transfer parsing typically relies on careful feature engineering. In this paper, we propose a hierarchical tensor-based approach for this task. This approach induces a compact feature representation by combining atomic features. However, unlike traditional tensor models, it enables us to incorporate prior knowledge about desired feature interactions, eliminating invalid feature combinations. To this end, we use a hierarchical structure that uses intermediate embeddings to capture desired feature combinations. Algebraically, this hierarchical tensor is equivalent to the sum of traditional tensors with shared components, and thus can be effectively trained with standard online algorithms. In both unsupervised and semi-supervised transfer scenarios, our hierarchical tensor consistently improves UAS and LAS over state-of-theart multilingual transfer parsers and the traditional tensor model across 10 different languages. 1 <s> BIB003
Performance of the surveyed algorithms for the tasks detailed in joint multilingual learning setting: Because it does not convey any information, it is of no avail in language transfer. Finally, let us consider the effectiveness of the methods described in §5.2 with respect to incorporating typological features in NLP models. In case of selective sharing, the tensor-based discriminative model BIB003 outperforms the graph-based discriminative model BIB001 , which in turn surpasses the generative model BIB002 . With regard to biasing multilingual models, there is a clear tendency toward letting typological features interact not merely with the input representation, but also with deeper levels of abstraction such as hidden layers. Overall, this comparison supports the claim that typology can potentially aid in designing the architecture of algorithms, engineering their features, and selecting and pre-processing their data. Nonetheless, this discussion also revealed that many challenges lie ahead for each of these goals to be accomplished fully. We discuss them in the next section.