CM3: Cooperative Multi goal Multi stage Multi agent Reinforcement Learning
A variety of cooperative multi agent control problems require agents to achieve individual goals while contributing to collective success. This multi goal multi agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two stage curriculum, in which single agent goal attainment is learned prior to learning multi agent cooperation, and we derive a new multi goal multi agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi goal multi agent problems: cooperative navigation in difficult formations, negotiating multi vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.
ICLR