Figure REF  illustrates the components of the architecture used in this work, similar to [1]}. A message head is only used for algorithms with communication, namely CACL, AE COMM and DIAL. The Grounding Module refers to mechanisms to ground the messages produced by the message head, used in CACL and AE COMM. Unless specified otherwise, we fix all hidden layers to be a size of 32. The observation encoder and message encoder output values of size 32 and 16 respectively. Messages received are concatenated before passing to message encoders. For all the methods with communication, they produce messages of length 4 with a sigmoid function as activation. All models are trained with the Adam optimizer [2]}.
