AHAC: Actor Hierarchical Attention Critic for Multi Agent Reinforcement Learning
Deep reinforcement learning has made significant progress in multi agent tasks in recent years. However, most previous studies focus on solving full cooperative tasks, which do not perform well in mixed tasks. In mixed tasks, the agent needs to comprehensively consider the information provided by friends and enemies to learn its strategy, and its strategy is sensitive to the received information. There is a great necessity to efficiently learn information representation for mixed tasks. To this end, we present an approach that conducts information representation learning for multiple agents using hierarchical attention mechanism. Our approach adopts the framework of centralized training and decentralized execution. It applies hierarchical attention to centrally computed critics, so critics process the received information more accurately and assist actors to choose better actions. The hierarchical attention critic uses two different attention levels, the agent level and the group level, to assign different weights to information of friends and enemies respectively and then summarize them at each time step. It can achieve more effective and scalable learning in mixed tasks. In addition, our approach uses recurrent neural networks that process sequence input information more efficiently. Experimental results show that our approach is not only applicable to cooperative environments but also better in mixed environments. Especially in the predator prey task, our approach receives twice as much reward as baselines.
2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC)