Human learning relies on retrospecting our detailed memory of the past [1]}, [2]}. For example, while exploring a new scenic area, we do not just remember a multitude of specific spots there, but can recall the paths that connect them with junctions and turns. However, there is no MARL method that can explicitly recall the past and identify key states that lead to future rewards. Such “mental time travel” [3]} ability is vital for tackling the challenges in OBMAS. Inspired by the recent progress in RL with episodic memory [4]}, [5]}, [6]} that is based on the memory prosthesis proposed by neuroscientists [1]}, [2]}, we propose our method of episodic memory representation for MARL. Unlike previous episodic memory methods that train a parameterized memory by either augmenting the policy inputs for execution [4]} or regularizing the TD learning [6]} for RL, our method utilizes the levelled graph data structure [11]}, a well established structure for data storage and retrieval, to represent an agent's individual episodic memory.
<FIGURE>