The problem of off-beat actions in MARL has yet to be investigated and tackled.
Training MARL policies in OBMAS is challenging: (i) Each agent's actions can have a variety of execution durations, which augments the order of complexity of OBMAS during decentralized execution, resulting in failure of the coordination; (ii) The action durations are unknown to agents during individual executions, and communication is constrained and not always feasible, making it non-trivial to model the environment; (iii) During training, both the temporal credit assignment with TD-learning [1]} and the inter-agent credit assignment with value decomposition methods [2]} cannot perform well due to the displaced rewards in multi-agent replay.
With off-beat actions, the nonstationarity issue, which mainly stems from rewards' time dependency on the agents' past actions, is exacerbated.
