Experience replay buffer is a popular technique for off-policy learning that has been shown to improve sample efficiency compared to on-policy algorithms [1]}. However, in multi-agent environments collected experience quickly becomes irrelevant due to the inherent non-stationarity [2]}. Methods have been proposed by [3]} to alleviate non-stationarity using techniques such as importance sampling and fingerprinting. However, these techniques can suffer from large variance as the agents progressively change their policies [4]}. Another approach is to use on-policy algorithms and a world model that approximates the environment's dynamics, which can be used to roll out imaginary trajectories starting from the states in the buffer for current policies, thus mitigating any issues with irrelevant experience [5]}.
