Policy Robustness Evaluation using A2C
[1]
Input: Number of actors \(N\) , and number of steps in each policy iteration \(T\) 
Obtain training vector \(\textbf {k}\) , and evaluation vector \(\textbf {k}^{\prime }\)  using cluster analysis of the predefined uncertainty distribution \(\mathcal {K}\) 
\(iteration=1,2,\ldots \) 
\(actor=1,2,\ldots ,N\) 
[t]@X@
 Run policy \(\pi _{\theta }\)  in environment for \(T\)  time steps (which corresponds to in total \(E\)  episodes where the environment permeability is set to a sample from training vector \(\textbf {k}\) , at the beginning of every episode ) 


Compute value function estimates \(\hat{V}_1,\ldots ,\hat{V}_T\)  using critic network
Compute advantage function estimates \(\hat{A}_1,\ldots ,\hat{A}_T\) 
Optimize \(J_{a2c}(\theta )\) (Eq. (REF )), with single epoch and batch size \(NT\) 
\(\theta _{old}\leftarrow \theta \) 
[t]@X@
 compute and record training return \(\textbf {R}^{\pi (a|s; \textbf {k} \Rightarrow \textbf {k})}\) , and evaluation return \(\textbf {R}^{\pi (a|s; \textbf {k} \Rightarrow \textbf {k}^{\prime })}\) 





In practice, a single integrated neural network is used for both, actor and critic networks. As a result, objective function for automatic differentiation is summation of Eq. (REF ) and value loss function for critic network. Please refer Algorithm S2 from [1]} for the formulation of value loss function.
