When a bilingual student learns to solve word problems in math, we expect the student to be able to solve these problem in both languages the student is fluent in, even if the math lessons were only taught in one language. However, current representations in machine learning are language dependent. In this work, we present a method to decouple the language from the problem by learning language agnostic representations and therefore allowing training a model in one language and applying to a different one in a zero shot fashion. We learn these representations by taking inspiration from linguistics, specifically the Universal Grammar hypothesis and learn universal latent representations that are language agnostic (Chomsky, 2014; Montague, 1970). We demonstrate the capabilities of these representations by showing that the models trained on a single language using language agnostic representations achieve very similar accuracies in other languages. Anecdotally speaking, fluent bilingual speakers rarely face trouble translating a task learned in one language to another. For example, a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages. Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory BID1 BID28 BID6 . The fundamental question we're interested in answering is on the learnability of these shared representations within a statistical framework.We approached this problem from a linguistics perspective. Languages have vastly varying syntactic features and rules. Linguistic Relativity studies the impact of these syntactic variations on the formations of concepts and theories BID5 . Within this framework of study, the two schools of thoughts are linguistic determinism and weak linguistic influence. Linguistic determinism argues that language entirely forms the range of cognitive processes, including the creation of various concepts, but is generally agreed to be false BID18 BID5 . Although there exists some weak linguistic influence, it is by no means fundamental BID0 . The superfluous nature of syntactic variations across languages brings forward the argument of principles and parameters (PnP) which hypothesizes the existence of a small distributed parameter representation that captures the syntactic variance between languages denoted by parameters (e.g. head-first or head-final syntax), as well as common principles shared across all languages BID12 . Universal Grammar (UG) is the study of principles and the parameters that are universal across languages BID29 .The ability to learn these universalities would allow us to learn representations of language that are fundamentally agnostic of the specific language itself. Doing so would allow us to learn a task in one language and reap the benefits of all other languages without needing multilingual datasets. We take inspiration from the UG hypothesis and learn latent representations that are language agnostic which allow us to solve downstream problems in new languages without the need of any language-specific training data. We do not make any claims about the Universal Grammar hypothesis, but simply take inspiration from it. Universal Grammar also comments on the learnability of grammar, stating that statistical information alone is not enough to learn grammar and some form of native language faculty must exist, sometimes titled the poverty of stimulus (POS) argument BID10 BID23 ). The goal of our paper is not to make a statement on the Universal Grammar hypothesis. But from a machine learning perspective, we're interested in extracting informative features. That being said it is of interest to what extent language models capture grammar and furthermore the extent to which models trained with our objective learn grammar. One way to measure universality is by studying perplexity of our multi-lingual language model as we increase the number of languages. To do so we trained 6 UG-WGAN models on the following languages: English, Russian, Arabic, Chinese, German, Spanish, French. We maintain the same procedure as described above. The hidden size of the language model was increased to 1024 with 16K BPE tokens being used. The first model was trained on English Russian, second was trained on English Russian Arabic and so on. For Arabic we still trained from left to right even though naturally the language is read from right to left. We report the results in FIG3 . As we increase the number of languages the perplexity gap between constrained and unconstrained UG-WGAN (λ = 0.0) decreases which implies while controlling capacity, our constrained (universal λ = 0.1) language model, models language (almost) as well as jointly trained language models with no universal constraints (λ = 0.0).Furthermore , the heatmap in FIG3 shows the perplexity gap of UG-WGAN trained on any combination of 2 languages from our set of 7. We can treat the perplexities as a loose measure of distance λ = 0.0 λ = 0.1 en earth's oxide is a monopoly that occurs towing of the carbon-booed trunks, resulting in a beam containing of oxygen through the soil, salt, warm waters, and the different proteins.the practice of epimatic behaviours may be required in many ways of all non-traditional entities.the groove and the products are numeric because they are called "pressibility" (ms) nutrients containing specific different principles that are available from the root of their family, including a wide variety of molecular and biochemical elements. a state line is a self-government environment for statistical cooperation, which is affected by the monks of canada, the east midland of the united kingdom.however, compared to the listing of special definitions, it has evolved to be congruent with structural introductions, allowing to form the chemical form.the vernacular concept of physical law is not as an objection (the whis) but as a universal school.es la revista ms reciente vari el manuscrito originalmente por primera vez en la revista publicada en 1994.en el municipio real se localiza al mar del norte y su entorno en escajros alto, con mayor variedad de cclica poblacin en forma de cerca de 1070 km2.de hecho la primera cancin de "blebe cantas", pahka zanjiwtryinvined cot de entre clases de fanticas, apareci en el ornitlogo sello triusion, jr., en la famosa publicacin playboy de john allen.fue el ltimo habitantes de suecia, con tres hijos, atasaurus y aminkinano (nuestra).The names of large predators in charlesosaurus include bird turtles hibernated by aerial fighters and ignored fish.jaime en veracruz fue llamado papa del conde mayor de valdechio, hijo de diego de ziga. We see from Figure 2 that perplexity worsens proportional to λ. We explore the differences by sampling sentences from an unconstrained language model and λ = 0.1 language model trained towards English and Spanish in Table 3 . In general there is a very small difference between a language model trained with our objective and one without. The constrained model tends to make more gender mistakes and mistakes due to Plural-Singular Form in Spanish. In English we saw virtually no fundamental differences between the language models. One explanation of this phenomena comes from the autonomy of syntax argument, which argues that semantics have no weight on syntax BID16 . Our hypothesis is that both models learn syntax well, but the models with better perplexity generate sentences with better or clearer semantic meaning. Although completely learning grammar from statistical signals might be improbable, we can still extract useful information. In this paper we introduced an unsupervised approach toward learning language agnostic universal representations by taking inspiration from the Universal Grammar hypothesis. We showed that we can use these representations to learn tasks in one language and automatically transfer them to others with no additional training. Furthermore we studied the importance of the Wasserstein constraint through the λ hyper-parameter. And lastly we explored the difference between a standard multi-lingual language model and UG-WGAN by studying the generated outputs of the respective language models as well as the perplexity gap growth with respect to the number of languages.
By taking inspiration from linguistics, specifically the Universal Grammar hypothesis, we learn language agnostic universal representations which we can utilize to do zero-shot learning across languages.
{"src": "scitldr-AIC_train", "paper_id": "r1l9Nj09YQ"}