How can we teach artificial agents to use human language flexibly to solve problems in a real-world environment? We have one example in nature of agents being able to solve this problem: human babies eventually learn to use human language to solve problems, and they are taught with an adult human-in-the-loop. Unfortunately, current machine learning methods (e.g. from deep reinforcement learning) are too data inefficient to learn a language in this way (3). An outstanding goal is finding an algorithm with a suitable ‘language learning prior’ that allows it to learn human language, while minimizing the number of required human interactions.

 In this paper, we propose to learn such a prior in simulation, leveraging the increasing amount of available compute for machine learning experiments (1). We call our approach Learning to Learn to Communicate (L2C). Specifically, in L2C we train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations unseen during training, including populations of humans. To show the promise of the L2C framework, we conduct some preliminary experiments in a Lewis signaling game (4), where we show that agents
 trained with L2C are able to learn a simple form of human language (represented by a hand-coded compositional language) in fewer iterations than randomly initialized agents. Language is one of the most important aspects of human intelligence; it allows humans to coordinate and share knowledge with each other. We will want artificial agents to understand language as it is a natural means for us to specify their goals.So how can we train agents to understand language? We adopt the functional view of language BID16 that has recently gained popularity (8; 14) : agents understand language when they can use language to carry out tasks in the real world. One approach to training agents that can use language in their environment is via emergent communication, where researchers train randomly initialized agents to solve tasks requiring communication (7; 16 ). An open question in emergent communication is how the resulting communication protocols can be transferred to learning human language. Existing approaches attempt to do this using auxiliary tasks, for example having agents predict the label of an image in English while simultaneously playing an image-based referential game BID11 . While this works for learning the names of objects, it's unclear if simply using an auxiliary loss will scale to learning the English names of complex concepts, or learning to use English to interact in an grounded environment.One approach that we know will work (eventually) for training language learning agents is using a human-in-the-loop, as this is how human babies acquire language. In other words, if we had a good enough model architecture and learning algorithm, the human-in-the-loop approach should work. However, recent work in this direction has concluded that current algorithms are too sample inefficient to effectively learn a language with compositional properties from humans (3). Human guidance is expensive, and thus we would want such an algorithm to be as sample efficient as possible. An open problem is thus to create an algorithm or training procedure that results in increased sampleefficiency for language learning with a human-in-the-loop.In this paper, we present the Learning to Learn to Communicate (L2C) framework, with the goal of training agents to quickly learn new (human) languages. The core idea behind L2C is to leverage the increasing amount of available compute for machine learning experiments (1) to learn a 'language learning prior' by training agents via meta-learning in Figure 1 . Diagram of the L2C framework. An advantage of L2C is that agents can be trained in an external environment (which grounds the language), where agents interact with the environment via actions and language. Thus, (in theory) L2C could be scaled to learn complicated grounded tasks involving language.simulation. Specifically, we train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations unseen during training, including populations of humans. The L2C framework has two main advantages: (1) permits for agents to learn language that is grounded in an environment with which the agents can interact (i.e. it is not limited to referential games); and (2) in contrast with work from the instruction following literature (2), agents can be trained via L2C to both speak (output language to help accomplish their goal) and listen (map from the language to a goal or sequence of actions).To show the promise of the L2C framework, we provide some preliminary experiments in a Lewis signaling game BID3 . Specifically , we show that agents trained with L2C are able to learn a simple form of human language (represented by a hand-coded compositional language) in fewer iterations than randomly initialized agents. These preliminary results suggest that L2C is a promising framework for training agents to learn human language from few human interactions.
We propose to use meta-learning for more efficient language learning, via a kind of 'domain randomization'. 
{"src": "scitldr-AIC_train", "paper_id": "rkxJrcHo2V"}