All methods use the same architecture based on the independent actor-critic algorithm with n-step returns and asynchronous environments [1]}. Each agent has an observation and message encoder to process observations and received messages. For methods with communication, each agent has a communication head to produce a message based on encoded observations. For policy learning, a GRU [2]} is used for partial observability before the policy and value heads. Each head is a 3-layer fully-connected neural network. We perform spectral normalization in the penultimate layer for each head to improve training stability [3]}. Details for architecture and hyperparameters used can be found in appendix  REF
