We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to receive rewards. It interactively learns the teacher’s language from scratch based on two language use cases: sentence-directed navigation and question answering. It learns simultaneously the visual representations of the world, the language, and the action control. By disentangling language grounding from other computational routines and sharing a concept detection function between language grounding and prediction, the agent reliably interpolates and extrapolates to interpret sentences that contain new word combinations or new words missing from training sentences. The new words are transferred from the answers of language prediction. Such a language ability is trained and evaluated on a population of over 1.6 million distinct sentences consisting of 119 object words, 8 color words, 9 spatial-relation words, and 50 grammatical words. The proposed model significantly outperforms five comparison methods for interpreting zero-shot sentences. In addition, we demonstrate human-interpretable intermediate outputs of the model in the appendix. Some empiricists argue that language may be learned based on its usage (Tomasello, 2003) . Skinner (1957) suggests that the successful use of a word reinforces the understanding of its meaning as well as the probability of it being used again in the future. BID3 emphasizes the role of social interaction in helping a child develop the language, and posits the importance of the feedback and reinforcement from the parents during the learning process. This paper takes a positive view of the above behaviorism and tries to explore some of the ideas by instantiating them in a 2D virtual world where interactive language acquisition happens. This interactive setting contrasts with a common learning setting in that language is learned from dynamic interactions with environments instead of from static labeled data.Language acquisition can go beyond mapping language as input patterns to output labels for merely obtaining high rewards or accomplishing tasks. We take a step further to require the language to be grounded BID13 . Specifically, we consult the paradigm of procedural semantics BID24 which posits that words, as abstract procedures, should be able to pick out referents. We will attempt to explicitly link words to environment concepts instead of treating the whole model as a black box. Such a capability also implies that, depending on the interactions with the world, words would have particular meanings in a particular context and some content words in the usual sense might not even have meanings in our case. As a result, the goal of this paper is to acquire "in-context" word meanings regardless of their suitability in all scenarios.On the other hand, it has been argued that a child's exposure to adult language provides inadequate evidence for language learning BID7 , but some induction mechanism should exist to bridge this gap (Landauer & Dumais, 1997) . This property is critical for any AI system to learn an infinite number of sentences from a finite amount of training data. This type of generalization problem is specially addressed in our problem setting. After training, we want the agent to generalize to interpret zero-shot sentences of two types: Testing ZS2 sentences contain a new word ("watermelon") that never appears in any training sentence but is learned from a training answer. This figure is only a conceptual illustration of language generalization; in practice it might take many training sessions before the agent can generalize. (Due to space limitations, the maps are only partially shown.) 1) interpolation, new combinations of previously seen words for the same use case, or 2) extrapolation, new words transferred from other use cases and models.In the following, we will call the first type ZS1 sentences and the second type ZS2 sentences. Note that so far the zero-shot problems, addressed by most recent work BID14 BID4 of interactive language learning, belong to the category of ZS1. In contrast, a reliable interpretation of ZS2 sentences, which is essentially a transfer learning (Pan & Yang, 2010) problem, will be a major contribution of this work.We created a 2D maze-like world called XWORLD FIG0 ), as a testbed for interactive grounded language acquisition and generalization. 1 In this world, a virtual agent has two language use cases: navigation (NAV) and question answering (QA). For NAV, the agent needs to navigate to correct places indicated by language commands from a virtual teacher. For QA, the agent must correctly generate single-word answers to the teacher's questions. NAV tests language comprehension while QA additionally tests language prediction. They happen simultaneously: When the agent is navigating, the teacher might ask questions regarding its current interaction with the environment. Once the agent reaches the target or the time is up, the current session ends and a new one is randomly generated according to our configuration (Appendix B). The ZS2 sentences defined in our setting require word meanings to be transferred from single-word answers to sentences, or more precisely, from language prediction to grounding. This is achieved by establishing an explicit link between grounding and prediction via a common concept detection function, which constitutes the major novelty of our model. With this transferring ability, the agent is able to comprehend a question containing a new object learned from an answer, without retraining the QA pipeline. It is also able to navigate to a freshly taught object without retraining the NAV pipeline.It is worthwhile emphasizing that this seemingly "simple" world in fact poses great challenges for language acquisition and generalization, because:The state space is huge. Even for a 7ˆ7 map with 15 wall blocks and 5 objects selected from 119 distinct classes, there are already octillions (10 27 ) of possible different configurations, not to mention the intra-class variance of object instances (see FIG0 in the appendix). For two configurations that only differ in one block, their successful navigation paths could be completely different. This requires an accurate perception of the environment. Moreover, the configuration constantly changes from session to session, and from training to testing. In particular, the target changes across sessions in both location and appearance.The goal space implied by the language for navigation is huge. For a vocabulary containing only 185 words, the total number of distinct commands that can be said by the teacher conforming to our defined grammar is already over half a million. Two commands that differ by only one word could imply completely different goals. This requires an accurate grounding of language. The environment demands a strong language generalization ability from the agent. The agent has to learn to interpret zero-shot sentences that might be as long as 13 words. It has to "plug" the meaning of a new word or word combination into a familiar sentential context while trying to still make sense of the unfamiliar whole. The recent work BID14 BID4 addresses ZS1 (for short sentences with several words) but not ZS2 sentences, which is a key difference between our learning problem and theirs.We describe an end-to-end model for the agent to interactively acquire language from scratch and generalize to unfamiliar sentences. Here "scratch" means that the model does not hold any assumption of the language semantics or syntax. Each sentence is simply a sequence of tokens with each token being equally meaningless in the beginning of learning. This is unlike some early pioneering systems (e.g., SHRDLU BID23 and ABIGAIL (Siskind, 1994) ) that hard-coded the syntax or semantics to link language to a simulated world-an approach that presents scalability issues. There are two aspects of the interaction: one is with the teacher (i.e., language and rewards) and the other is with the environment (e.g., stepping on objects or hitting walls). The model takes as input RGB images, sentences, and rewards. It learns simultaneously the visual representations of the world, the language, and the action control. We evaluate our model on randomly generated XWORLD maps with random agent positions, on a population of over 1.6 million distinct sentences consisting of 119 object words, 8 color words, 9 spatial-relation words, and 50 grammatical words. Detailed analysis (Appendix A) of the trained model shows that the language is grounded in such a way that the words are capable to pick out referents in the environment. We specially test the generalization ability of the agent for handling zero-shot sentences. The average NAV success rates are 84.3% for ZS1 and 85.2% for ZS2 when the zero-shot portion is half, comparable to the rate of 90.5% in a normal language setting. The average QA accuracies are 97.8% for ZS1 and 97.7% for ZS2 when the zero-shot portion is half, almost as good as the accuracy of 99.7% in a normal language setting. We have presented an end-to-end model of a virtual agent for acquiring language from a 2D world in an interactive manner, through the visual and linguistic perception channels. After learning, the agent is able to both interpolate and extrapolate to interpret zero-shot sentences that contain new word combinations or even new words. This generalization ability is supported by an explicit grounding strategy that disentangles the language grounding from the subsequent languageindependent computations. It also depends on sharing a detection function between the language grounding and prediction as the core computation. This function enables the word meanings to transfer from the prediction to the grounding during the test time. Promising language acquisition and generalization results have been obtained in the 2D XWORLD. We hope that this work can shed some light on acquiring and generalizing language in a similar way in a 3D world.Thomas Landauer and Susan Dumais. A solution to plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104, 1997 . blue brown gray green orange purple red yellow apple armadillo artichoke avocado banana bat bathtub beans bear bed bee beet beetle bird blueberry bookshelf broccoli bull butterfly cabbage cactus camel carpet carrot cat centipede chair cherry circle clock coconut corn cow crab crocodile cucumber deer desk dinosaur dog donkey dragon dragonfly duck eggplant elephant fan fig fireplace fish fox frog garlic giraffe glove goat grape greenonion greenpepper hedgehog horse kangaroo knife koala ladybug lemon light lion lizard microwave mirror monitor monkey monster mushroom octopus onion ostrich owl panda peacock penguin pepper pig pineapple plunger potato pumpkin rabbit racoon rat rhinoceros rooster seahorse seashell seaurchin shrimp snail snake sofa spider square squirrel stairs star strawberry tiger toilet tomato triangle turtle vacuum wardrobe washingmachine watermelon whale wheat zebra east north northeast northwest south southeast southwest west blue brown gray green orange purple red yellow apple armadillo artichoke avocado banana bat bathtub beans bear bed bee beet beetle bird blueberry bookshelf broccoli bull butterfly cabbage cactus camel carpet carrot cat centipede chair cherry circle clock coconut corn cow crab crocodile cucumber Channel mask x feat . We inspect the channel mask x feat which allows the model to select certain feature maps from a feature cube h and predict an answer to the question s. We randomly sample 10k QA questions and compute x feat for each of them using the grounding module L. We divide the 10k questions into 134 groups, where each group corresponds to a different answer. 4 Then we compute an Euclidean distance matrix D where entry Dri, js is the average distance between the x feat of a question from the ith group and that from the jth group FIG6 .
Training an agent in a 2D virtual world for grounded language acquisition and generalization.
{"src": "scitldr-AIC_train", "paper_id": "H1UOm4gA-"}