where \(\alpha _{LR}\)  is the learning rate. Here \(\hat{g}(\theta ) \triangleq \nabla _\theta L^{PG}(\theta ) \approx \nabla _\theta L(\theta )\)  is a sample average approximation of the policy gradient, obtained by differentiating a surrogate objective function \(L^{PG}(\theta )\) . This surrogate objective function is created over a finite batch of samples in an algorithm that alternates between sampling and optimization. Several forms of the surrogate objective are available [1]}, one of the simplest beingFor taking the derivative of \(L^{PG}\) , \(\log \pi _\theta (a|s)\)  can be analtyically expressed in terms of \(\mu _\theta (s)\)  and \(\sigma _\theta (s)\)  and the dependence of these terms on \(\theta \)  is computed through automatic differentiation.
\(\begin{aligned}L^{PG}(\theta ) &= \frac{1}{N}\sum _{i=1}^{N}\left(\sum _{h=0}^H R_h^{(i)} \log \pi _\theta (a_h^{(i)}|s_h^{(i)}) \right)\end{aligned}\) 
