where, \(\hat{A}_t\)  is an estimator for advantage function at timestep \(t\)  and the term, \(\hat{\mathbb {E}}_t[\cdots ]\) , is empirical average over finite batch of samples collected through agent-environment interactions.
Gradient estimator of policy network, \(\nabla _{\theta }J(\theta )\) , is obtained by differentiating Eq. (REF ) which is done with automatic differentiation algorithm. The advantage function estimator, \(\hat{A}_t\) , is computed using generalized advantage estimator [1]} which is derived from the value function \(V_t\) .
The value function estimator \(\hat{V}_t\)  is learned through a separate neural network termed as the \(critic\)  network. Definitions of advantage and value functions are described in .
Algorithm REF  illustrates a broad outline for implementation of A2C algorithm in this study.
In order to reduce computational time, the iterative data sampling for objective function is performed in parallel on \(N\)  processors followed by a synchronous gradient update.
Note that in every policy iteration in total \(T\)  control steps are run where environment is reset with a new permeability sample from the \(\textbf {k}\)  after every terminal step of the episode.
