@inproceedings{
zhu2024madiff,
title={{MAD}iff: Offline Multi-agent Learning with Diffusion Models},
author={Zhengbang Zhu and Minghuan Liu and Liyuan Mao and Bingyi Kang and Minkai Xu and Yong Yu and Stefano Ermon and Weinan Zhang},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=PvoxbjcRPT}
}
Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle with extrapolation errors in offline settings, while supervised learning methods are constrained by model expressiveness. Recently, diffusion models (DMs) have shown promise in overcoming these limitations in single-agent learning, but their application in multi-agent scenarios remains unclear. Generating trajectories for each agent with independent DMs may impede coordination, while concatenating all agents’ information can lead to low sample efficiency. Accordingly, we propose MADiff, which is realized with an attention-based diffusion model to model the complex coordination among behaviors of multiple agents. To our knowledge, MADiff is the first diffusion-based multi-agent learning framework, functioning as both a decentralized policy and a centralized controller. During decentralized executions, MADiff simultaneously performs teammate modeling, and the centralized controller can also be applied in multi-agent trajectory predictions. Our experiments demonstrate that MADiff outperforms baseline algorithms across various multi-agent learning tasks, highlighting its effectiveness in modeling complex multi-agent interactions.
MADiff: Offline Multi-agent Learning with Diffusion Models
Zhengbang Zhu Minghuan Liu Liyuan Mao Bingyi Kang Minkai Xu Yong Yu Stefano Ermon Weinan Zhang