While actions durations are ubiquitous, existing works only focus on single-agent settings, i.e., delay, in RL. Many approaches [1]}, [2]}, [3]} augment the state space with the queuing actions to be executed into the environment. However, such state-augmentation trick leads to exponentially increasing training samples with the growing action duration, making training intractable [4]}. Chen et al. [5]} extend the delayed MDP [2]} and propose Delayed Markov Game for MARL. However, on one hand, such state-augmentation treatment is confined to short delays, e.g., one timestep delay; on the other hand, the delayed timestep of the actions is privileged information, which is not available in many scenarios. Recent works on macro-actions [7]}, [8]} introduce asynchronous actions by designing macro-actions with prior environment knowledge. Macro-actions are different from options in hierarchical RL (HRL) [9]}, [10]} in that the later is not manually designed but learned. The key difference between macro-actions and off-beat actions is that macro-actions are high-level actions while off-beat actions are primitive actions. Unfortunately, the inter-agent credit assignment is still a challenge of HRL in OBMAS and the asynchronous We clarify the term asynchronous: actions that simultaneously committed into the environment by all agents in MARL will not complete their respective action durations at the same time in future timesteps. nature of off-beat actions undermines the temporal credit assignment of centralized training, causing poor sample efficiency and unsatisfactory performance (more discussions can be found in the related works sections in Sec. ).
