In this paper, we use PPO to learn decentralized policies \(\pi ^a\)  for agents with individual policy clipping where each agent's independent policy updates are clipped based on the objective defined in Equation REF . We consider a variant of the advantage function based on independent learning, where each agent \(a\)  learns a local observation based critic \(V_{\phi }(z_t^a)\)  parameterised by \(\phi \)  using Generalized Advantage Estimation (GAE) [1]} with discount factor \(\gamma =0.99\)  and \(\lambda =0.95\) .
The network parameters \(\phi , \theta \)  are shared across critics, and actors, respectively.
We also add an entropy regularization term to the final policy loss [2]}.
For each agent \(a\) , we have its advantage estimation as follows:
\(A^{a}_t = \sum _{l=0}^{h} (\gamma \lambda )^l \delta _{t+l}^{a},\) 
