In the multi-agent setting, the overall environment ceases to be stationary from the viewpoint of individual agents, since all the agents are concurrently updating their policies. For this reason, RL algorithms designed/tested for single-agent domains with stationary environments fail in the MA setting. Value-learning methods like Q-Learning ([1]}) largely ignore the non-stationarity of the process resulting in unstable learning save for the most simple problems ([2]}). On the other hand, policy-gradient (PG) methods, while theoretically capable of handling non-stationary environments, have only been used to study interesting yet simplified problems ([3]}, [4]}). PG methods remain affected by the large-variance of the gradient-estimator in high-dimensional spaces ([5]}, [6]}); additionally, these methods cannot reuse past-experience of the agents, making them woefully sample-inefficient and impractical for complex tasks with high-dimensional (e.g image, lidar) observations.
