Multi-Agent RL. Many MARL methods focus on factorizing the global Q value to train agents' policies via CTDE [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}. However, these existing works assume actions are executed synchronously. Messias et al. [8]} propose an event-driven, asynchronous formulation of the multi-agent POMDP. However, the assumption of free communication [9]} is limited and the asynchronous execution [10]} in the paper is confined to the design of events and did not propose methods on solving challenging credit assignment issue in OBMAS. Recently, Amato et al. [11]} and Xiao et al. [12]}, [13]} propose macro-action methods, which are similar to hierarchical methods. Macro-actions are manually designed via abstracting primitive actions. However, macro-action methods mainly focus on macro-action selection during multi-timestep decision-making and assume the environment can use manually pre-defined methods for state transition. Unfortunately, the above works either focus on synchronous actions or defining specific asynchronous execution components with human knowledge. Learning coordination in OBMAS remains a challenge.
