Multi-Agent Reinforcement Learning. MARL aims to learn optimal policies for all the agents in the team. With TD-learning and a global Q value proxy \(Q^{\operatorname{tot}}\)  for the optimal \(Q^{*}\) , \(\lbrace Q_i\rbrace ^{N}_{i=1}\)  are optimized via minimizing the loss [1]}, [2]}: \(\theta ^{*} = \arg \min _{\theta ^{*}} \mathcal {L} (\theta ) :=\mathbb {E}_{D^{\prime } \sim \mathcal {D}}[(y_{t}^{\text{tot}}-Q_{\theta }^{\text{tot }}(s_{t}, u_{t}))^{2}]\) , where \(y_{t}^{\mathrm {tot}}=r_{t}+\gamma \max _{u^{\prime }} Q_{\bar{\theta }}^{\mathrm {tot}}(s_{t+1}, u^{\prime })\) 
and \(\theta \)  is the parameters of the agents. \(\bar{\theta }\)  is the parameter of the target \(Q^{\operatorname{tot}}\)  and is periodically copied from \(\theta \) . \(D^{\prime }\)  is a sample from the replay buffer \(\mathcal {D}\) .
