Actor-critic algorithms have been shown to benefit from learning centralized joint critics alongside decentralised policies [1]}.
As the critics are not needed during execution, this approach is inherently decentralizable.
COMA [1]} extends this approach with a counterfactual multi-agent critic baseline based on temporal difference errors [3]} in order to facilitate multi-agent credit assignment.
Joint \(Q\) -learning can also be made decentralizable. Value Decomposition Networks [4]} decompose joint state-action value functions into sums of decentralised utility functions that can be used during greedy execution.
QMIX [5]} extends this additive decomposition to arbitrary centralized monotonic mixing networks.
Both centralized joint critics and factored joint value functions can reap some benefits of centralized joint learning,
while bypassing the joint action space explosion, imposing an effective prior on multi-agent credit assignment, and mitigating practical learning pathologies associated with centralized joint learning on popular benchmark environment StarCraft II [6]}.
Value factorisation has also been successfuly transferred to continuous action spaces and combined with actor critic approaches [7]}.
