The task of visually grounded dialog involves learning goal-oriented cooperative dialog between autonomous agents who exchange information about a scene through several rounds of questions and answers. We posit that requiring agents to adhere to rules of human language while also maximizing information exchange is an ill-posed problem, and observe that humans do not stray from a common language, because they are social creatures and have to communicate with many people everyday, and it is far easier to stick to a common language even at the cost of some efficiency loss. Using this as inspiration, we propose and evaluate a multi-agent dialog framework where each agent interacts with, and learns from, multiple agents, and show that this results in more relevant and coherent dialog (as judged by human evaluators) without sacrificing task performance (as judged by quantitative metrics). Intelligent assistants like Siri and Alexa are increasingly becoming an important part of our daily lives, be it in the household, the workplace or in public places. As these systems become more advanced, we will have them interacting with each other to achieve a particular goal BID9 . We want these conversations to be interpretable to humans for the sake of transparency and ease of debugging. Having the agents communicate in natural language is one of the most universal ways of ensuring interpretability. This motivates our work on goal-driven agents which interact in coherent language understandable to humans.To that end, this paper builds on work by BID2 on goal-driven visual dialog agents. The task is formulated as a conversation between two agents, a Question (Q-) and an Answer (A-) bot. The A-Bot is given an image, while the QBot is given only a caption to the image. Both agents share a common objective, which is for the Q-Bot to form an accurate mental representation of the unseen image using which it can retrieve, rank or generate that image. This is facilitated by the exchange of 10 pairs of questions and answers between the two agents, using a shared vocabulary. BID2 trained the agents first in isolation via supervision from the VisDial dataset BID1 , followed by making them interact and adapt to each other via reinforcement learning by optimizing for better task performance. While trying to maximize performance, the agents learn to communicate in non-grammatical and semantically meaningless sentences in order to maximize the exchange of information. This reduces transparency of the AI system to human observers and is undesirable. We address this problem by proposing a multi-agent dialog framework where each agent interacts with multiple agents. This is motivated by our observation that humans adhere to syntactically and semantically coherent language, which we hypothesize is because they have to interact with an entire community, and having a private language for each person would be extremely inefficient. We show that our multi-agent (with multiple Q-Bots and multiple A-Bots) dialog system results in more coherent and human-interpretable dialog between agents, without compromising on task performance, which also validates our hypothesis. This makes them seem more helpful, transparent and trustworthy. We will make our code available as open-source. 1 In this paper we propose a novel Multi-Agent Dialog Framework (MADF), inspired from human communities, to improve the dialog quality of AI agents. We show that training 2 agents with supervised learning can lead to uninformative and repetitive dialog. Furthermore, we observe that the task performance (measured by the image retrieval percentile scores) for the system trained via supervision only deteriorates as dialog round number increases. We hypothesize that this is because the agents were trained in isolation and never allowed to interact during supervised learning, which leads to failure during testing when they encounter out of distribution samples (generated by the other agent, instead of ground truth) for the first time. We show how allowing a single pair of agents to interact and learn from each other via reinforcement learning dramatically improve their percentile scores, which additionally does not deteriorate over multiple rounds of dialog, since the agents have interacted with one another and been exposed to the other's generated questions or answers. However, the agents, in an attempt to improve task performance end up developing their own private language which does not adhere to the rules and conventions of human languages, and generates nongrammatical and non-sensical statements. As a result, the dialog system loses interpretability and sociability. Figure 4: Two randomly selected images from the VisDial dataset followed by the ground truth (human) and generated dialog about that image for each of our 4 systems (SL, RL-1Q,1A, RL-1Q,3A, RL-3Q,1A). These images were also used in the human evaluation results shown in Table 2 .multi-agent dialog framework based on self-play reinforcement learning, where a single A-Bot is allowed to interact with multiple Q-Bots and vice versa. Through a human evaluation study, we show that this leads to significant improvements in dialog quality measured by relevance, grammar and coherence. This is because interacting with multiple agents prevents any particular pair from maximizing performance by developing a private language, since it would harm performance with all the other agents.
Social agents learn to talk to each other in natural language towards a goal
{"src": "scitldr-AIC_train", "paper_id": "BJxX9mx8y7"}