Instead of using Reinforce [1]} for policy updates as in Dreamer, we found that for multi-agent environments PPO updates (Eq. REF ) yield better results. We do not use gradients backpropagated through the dynamics model as they hinder the performance, which is likely due to its high noisiness in multi-agent environments. We use parameter sharing [2]} for the actor network to accelerate training.
