In Multi-Agent Reinforcement Learning (MARL), multiple agents learn and interact in the same environment. In this paper, we will focus on cooperative environments [1]}, where agents have the same goal and they need to collaborate with each other to achieve it. One naive approach in MARL is training agents independently using unmodified single-agent RL techniques [2]}, [3]}. However, it does not have any convergence guaranties due to the inherent non-stationarity of multi-agent environments resulting from the constantly changing policies of other agents [4]}, [5]}. On the other side of the spectrum, we could look at the multi-agent problem as a single-agent one. This yields a fully centralized approach that controls all agents simultaneously based on global information. Consequently, a centralized solution cannot scale to a large number of agents due to the resultant exponential growth of the action space, known as the familiar 'curse of dimensionality' [6]}. This leaves us with one viable technique to train agents in multi-agent environments, namely Centralized Training with Decentralized Execution. We also explicitly highlight the communication technique, which is considered part of CTDE [7]}, that allows an additional message channel during execution for better coordination.
