Traditionally, the policy is optimized via a vanilla policy gradient algorithm by direct policy search over the entire exploration space which is known to be inefficient. blackInstead, we use trust region policy optimization (TRPO) to improve policy optimization by maximizing a surrogate objective over a trust-region [1]}. Accordingly, (REF ) can be re-written
\(\max _{\theta \in {R}^d} J(\pi _\theta )= {E}_{\pi _{\theta }} \left[ \frac{\pi _{\theta ^{\mathsf {new}}}(a|s)}{\pi _{\theta }(a|s)} A^{\pi _{\theta }}(s,a)\right],\) 
\(subject~to~~~~ {E}_s \left[KL \left(\pi _{\theta }(.|s)|| \pi _{\theta ^{\mathsf {new}}}(.|s)\right) \right] \le \epsilon _{KL},\) 
