TRPO [1]} is a class of policy-gradient methods that restricts the update of policies to within the trust region of the behavior policy by enforcing a KL divergence constraint on policy update at each iteration. Formally, TRPO optimizes the following:
\(&\max _{\theta } \qquad \qquad \mathbb {E}_{s_t, u_t} \left[ \frac{\pi _{\theta }(u_t | s_t)}{\pi _{\theta _{old}}(u_t | s_t)} A(s_t, u_t) \right] \\&\textrm {subject to} \qquad \,\, \mathbb {E}_{s_t, u_t}\left[\textrm {KL}\left(\pi _{\theta _{old}}, \pi _{\theta }\right)\right] \le \delta ,\) 
