With the ever increasing demand and the resultant reduced quality of services, the focus has shifted towards easing network congestion to enable more efficient flow in systems like traffic, supply chains and electrical grids. A step in this direction is to re-imagine the traditional heuristics based training of systems as this approach is incapable of modelling the involved dynamics. While one can apply Multi-Agent Reinforcement Learning (MARL) to such problems by considering each vertex in the network as an agent, most MARL-based models assume the agents to be independent. In many real-world tasks, agents need to behave as a group, rather than as a collection of individuals. In this paper, we propose a framework that induces cooperation and coordination amongst agents, connected via an underlying network, using emergent communication in a MARL-based setup. We formulate the problem in a general network setting and demonstrate the utility of communication in networks with the help of a case study on traffic systems. Furthermore, we study the emergent communication protocol and show the formation of distinct communities with grounded vocabulary. To the best of our knowledge, this is the only work that studies emergent language in a networked MARL setting. Co-existing intelligent agents affect each other in non-trivial ways. Consider for example, two agents playing a modified variant of archery in two dimensions. Agent A controls the speed at which the arrow is released but it can only shoot along y-axis. Agent B controls wind speed along x-axis. The arrow drifts along the direction of wind with a magnitude proportional to wind speed. A target is specified by (x, y) coordinates and the agents must act cooperatively to shoot the target. In this setup, the optimal action for agent A depends on the current policy of agent B and vice versa. Any change in one agent's policy modifies the other agent's perception about the environment dynamics. Formally, this issue is referred to as non-stationarity of environment in a multi-agent setup. This non-stationarity makes the learning problem hard and approaches that try to independently learn optimal behavior for agents do not perform well in practice (Tan, 1993) . Thus, it is important to develop models that have been tailored towards training multiple agents simultaneously. In this paper, we focus on a specific multi-agent setup where the agents are connected to each other via an underlying fixed network topology. We cast the problem in the multi-agent reinforcement learning (MARL) framework and assume that agents are rewarded by the environment based on their actions and their goal is to cooperatively maximize their rewards. We further assume that the agents have been endowed with the ability to communicate with one another along the network edges to achieve cooperation. However, the communication protocol is not fixed and the agents must learn a protocol to communicate with each other in order to maximize their rewards. Communication is essential in a multi-agent setup. In many practical scenarios, agents may only observe a small portion of the global environment state and they must take actions based on their local observations. As discussed above, agents affect each other in non-trivial ways through their actions. Thus, for achieving long term cooperation, it is essential for agents to be able to share their intents to complement the information provided by the local observation of each agent. Communication provides the ability to do so to the agents. Many real world problems can be cast in this framework and we provide a number of concrete examples after formally defining the problem setup in Section 2. For clarity of exposition and to be more concrete, in this paper, we focus on a particular real world problem as a case study, the problem of intelligently managing traffic. We present the traffic management problem as a particular instantiation of the abstract multi-agent reinforcement learning problem that we have informally defined above (see Section 2 for a formal definition). In this context, the agents correspond to traffic lights and the underlying network is the network of roads. Agents receive rewards from the environment based on factors like queue length at the traffic junction and must communicate with each other to cooperatively maximize their rewards, thereby ensuring a smooth flow of traffic. We propose a MARL-based traffic system that allows coordination between traffic signals (agents) via: (i) inter-agent communication; and (ii) a cooperative reward structure. At each time-step, the agents communicate with their immediate neighbours in the underlying network by broadcasting a message in the form of a discrete symbol (each message corresponds to a word, represented by a binary vector in our experiments). Over time, the agents are trained to exploit this broadcasted message to coordinate with each other and maximize their rewards. As the agents are trained, a language emerges between pairs of agents. Since the agents learn the communication protocol themselves, our approach is different from methods that use a fixed protocol for communication, like smart-grids. We empirically demonstrate the utility of communication in this setup and also investigate a cooperative reward structure over the network of traffic junctions. Our model uses a query-based soft attention mechanism to help the agents come up with more complex cooperative strategies. We perform extensive experimental evaluation to demonstrate that (i) our method outperforms baseline approaches; (ii) communication is useful, (iii) communication is grounded in actions taken by the agents, and (iv) the cooperative reward structure promotes communication and hence coordination. In this paper, we proposed an approach to mitigate network congestion with the help of traffic networks. Though extensive experiments, we demonstrated the benefit of emergent communication to optimize traffic flow over existing MARL approaches. Additionally, we performed qualitative studies on the emergent language and showed that it is grounded in actions. Human communication is discrete in nature and can, in general, be represented by categorical variables. Additionally, discrete variables are more interpretable which makes it well suited for real life problems like traffic management, where one needs transparency. However, the use of discrete latent variables render the neural network non-differentiable. The Gumbel Softmax gives a differentiable sample from a discrete distribution by approximating the hard one-hot vector into a soft version. The Gumbel distribution has two parameters: µ and β. The standard Gumbel distribution where µ and β are 0,1 respectively has probability density function: G(0, 1) = e −(z+e −z ) . Suppose, for a given agent, our model (Communicator) outputs a Multinomial distribution of message bits with logits : p = (p 1 , . . . , p d ) where d = 8. These logits are functions of inputs and weights which need to be trained. A simple way to draw samples from a discrete distribution would be to use the Gumbel-max trick (Jang et al., 2016) as shown, Here, the noise in form of z i is independently sampled from standard Gumbel distribution and are obtained as z i = − log(− log(u i )), u i being i. β is the temperature parameter, which, in our experiments, is set to 0.5. As β > 0, we obtain welldefined gradients ∂m i /∂p i with respect to parameters p i . Gumbel-Softmax can produce a 0.9999-hot vector instead of a hard one-hot vector depending on β. Since we want the communication to be discrete, we employ the Straight-Through version of the Gumbel-Softmax estimator with a simple reformulation of the form, Here,m i is the one-hot vector ofm i i.e. such thatm i = I{arg max im i = k, k ≤ k }. We use binary 8-bit messages, therefore k = 1 and k is fixed during the training process. Now, detach prevents the gradients from flowing through that node, hence, ∇ p i m i = ∇ p imi . This makes the communication discrete in the forward pass even as the gradients flow is smooth during the backward pass. The final output message of agent is given as m = (m 1 , . . . , m d ). A.3 COMMUNICATION . Action embeddings from matrix U (orthonormal matrix) of agent 0 corresponding to messages from all agents. As highlighted by the red circles, the action embeddings of agent i in response to messages from neighbours U i (as highlighted in gray at the centres of the plots) are overlapped. The color bar represents different agents. Here, we plots are for agents 0, 1, 8, 9. Similar trends are observed for rest of the agents as well.
A framework for studying emergent communication in a networked multi-agent reinforcement learning setup.
{"src": "scitldr-AIC_train", "paper_id": "H1lUp1BYDH"}