One major difference between single- and multi-agent environment dynamics is the varying number of agents on the map. Even if the starting number of agents is always the same, it is natural that agents will have varying trajectory lengths depending on the task. Thus, it is essential to handle the states of the agents that are no longer present in the environment during the training phase. Absorbing state [1]} is an imaginary state that does not yield a reward for any action and does not lead to any other state, creating an infinite loop to itself. In our implementation, we add absorbing states after the end of an agents trajectory, so that it does not get a reward for future actions and so other agents can see that that agent is no longer on the map. We do not use observation loss to learn the representation of those states as no meaningful observation can be provided by the environment. Instead, we allow the world model to implicitly learn those representations only from predicting the discounts and rewards that are both equal to zero.
