Action Delay in RL. Conventionally, the execution of actions in RL is instantaneous and the execution duration is neglected. Katsikopoulos et al. [1]} propose the Delayed MDP where actions have delays and Walsh et al [2]} propose a model-based method for the Delayed MDP.
To optimize the delayed MDP, many RL approaches [2]}, [4]}, [5]}, [6]} augment the state space with the queuing actions to be executed into the environment. However, this state-augmentation trick is intractable [7]}. Chen et al. [8]} extend the delayed MDP [4]} and propose a Delayed Markov Game. However, the state-augmentation treatment is confined to short delays and neglects the off-beat actions in multi-agent scenarios. Recently, Bouteiller et al. [10]} apply replay buffer correction method.
However, the delayed timestep is privileged information. It is not available for agents in many scenarios. Simply applying this single-agent trajectory correction in MARL cannot attain satisfactory performance due to off-beat actions; devising inter-agent trajectory correction methods for OBMAS is non-trivial.
