Distributed Influence-Augmented Local Simulators for Parallel MARL in Large Networked Systems
Due to its high sample complexity, simulation is, as of today, critical for
the successful application of reinforcement learning. Many real-world problems,
however, exhibit overly complex dynamics, which makes their full-scale
simulation computationally slow. In this paper, we show how to decompose large
networked systems of many agents into multiple local components such that we
can build separate simulators that run independently and in parallel. To
monitor the influence that the different local components exert on one another,
each of these simulators is equipped with a learned model that is periodically
trained on real trajectories. Our empirical results reveal that distributing
the simulation among different processes not only makes it possible to train
large multi-agent systems in just a few hours but also helps mitigate the
negative effects of simultaneous learning.