The emergence of language in multi-agent settings is a promising research direction to ground natural language in simulated agents. If AI would be able to understand the meaning of language through its using it, it could also transfer it to other situations flexibly. That is seen as an important step towards achieving general AI. The scope of emergent communication is so far, however, still limited. It is necessary to enhance the learning possibilities for skills associated with communication to increase the emergable complexity. We took an example from human language acquisition and the importance of the empathic connection in this process. We propose an approach to introduce the notion of empathy to multi-agent deep reinforcement learning. We extend existing approaches on referential games with an auxiliary task for the speaker to predict the listener's mind change improving the learning time. Our experiments show the high potential of this architectural element by doubling the learning speed of the test setup. Natural language is not as rule-based as researchers in supervised language learning would prefer. There are limitless context-dependent notions to it, and flexible language use is considered as a necessary aspect of general AI. Originally, natural language emerged through a necessity to achieve successful coordination. Hence, a general AI would need to understand the functional aspects of language and learn communication through interaction (Wittgenstein, 1958; Wagner et al., 2003) . These considerations led to the research field of emergent communication and the attempt to ground natural language through reinforcement learning. Deep reinforcement learning has achieved some impressive results over the last years (Arulkumaran et al., 2017) . One of its principal aspects is the ability to extract features from high dimensional input data without manual preprocessing. This capability is especially useful if the necessary representation is unknown to the designer. Classical deep reinforcement learning approaches rely on a large number of training examples, mainly because the sparse reward hardly provides enough feedback to shape the deep layers. These deep layers are responsible for the embedding of input data into a meaningful representation. Therefore, it takes many training steps before a useful representation emerges; if it converges at all. According to the theory of the predictive mind (Hohwy, 2013) , the human brain generates richer feedback through learning several unsupervised prediction tasks while training on the main task. The purpose of these predictions is to produce more and more expressive models and representations of the world. Oh et al. (2015) achieved a far more expressive representation of their visual inputs by learning an auxiliary prediction task. The sole purpose of the auxiliary net is to predict the change in the visual input given the last movement action. Training this net does not directly affect the original task, but it refines the visual representation to reflect the concepts of a 3D world. Hermann et al. (2017) used predictive tasks to ground natural language, but only focused on better understanding an existent language. We transfer the auxiliary prediction to the task of active communication. This goes along with the theory of mind (Premack & Woodruff, 1978; Schaafsma et al., 2015) stating that an essential part of intelligence in interaction emerges through predicting the mental state of the interaction partner. We let the speaker train an auxiliary net that tries to predict how the speaker's utterance will change the listener's hidden state. That resembles humans empathetic way of understanding what a message will do to the listener. We assume this leads to a more communication effective representation of the sensory input; in other words, the input encoding becomes more communicatable. The effect is visible in the essential acceleration of learning successes in developing a shared language. Our main contribution is an elegant extension to multi-agent deep reinforcement learning (MADRL) algorithms aiming to emerge a communication. It resembles an empathic connection between speaker and listener, which leads to faster convergence to a shared language. We doubled the learning speed of a MADRL algorithm playing a referential game by introducing this auxiliary prediction task to the speaking agent. We attribute the improvement to the richer gradients in the lower layers of the neural network to embed the input.
An auxiliary prediction task can speed up learning in language emergence setups.
{"src": "scitldr-AIC_train", "paper_id": "Hke1gySFvB"}