Policy Robustness Evaluation using PPO
[1]
Input: Number of actors \(N\) , number of steps in each policy iteration \(T\) , number of epochs \(K\)  and minibatch size \(M\) 
Obtain training vector \(\textbf {k}\) , and evaluation vector \(\textbf {k}^{\prime }\)  using cluster analysis of the predefined uncertainty distribution \(\mathcal {K}\) 
\(iteration=1,2,\ldots \) 
\(actor=1,2,\ldots ,N\) 
[t]@X@
 Run policy \(\pi _{\theta }\)  in environment for \(T\)  time steps (which corresponds to in total \(E\)  episodes where the environment permeability is set to a sample from training vector \(\textbf {k}\) , at the beginning of every episode ) 


Compute value function estimates \(\hat{V}_1,\ldots ,\hat{V}_T\)  using critic network
Compute advantage function estimates \(\hat{A}_1,\ldots ,\hat{A}_T\) 
Optimize \(J_{ppo}(\theta )\) (Eq. (REF )), with \(K\)  epochs and minibatch size \(M\le NT\) 
\(\theta _{old}\leftarrow \theta \) 
[t]@X@
 compute and record training return \(\textbf {R}^{\pi (a|s; \textbf {k} \Rightarrow \textbf {k})}\) , and evaluation return \(\textbf {R}^{\pi (a|s; \textbf {k} \Rightarrow \textbf {k}^{\prime })}\) 





 The objective function for integrated actor-critic network, in the PPO implementation, is the summation of actor loss term (Eq. (REF )), value loss term and entropy loss term. Readers are referred to [1]} for the detailed definition of value loss term and entropy loss term.
