
where \(\theta _{old}\)  are the policy parameters before the update and \(A(s_t, u_t)\)  is an approximation of advantage function.
The formulation is computationally expensive as it requires the computation of multiple Hessian-vector products for nonlinear conjugate gradients when approximating the KL constraint.
To resolve this, Proximal Policy Optimization [1]} approximates the trust region constraints by policy ratio clippings, i.e. the policy loss becomes:
\(\mathcal {L}(\theta ) = \mathbb {E}_{s_t, u_t} \left[ \min \left( \frac{\pi _{\theta }(u_t | s_t)}{\pi _{\theta _{old}}(u_t | s_t)} A(s_t, u_t), \textrm {clip}\left(\frac{\pi _{\theta }(u_t | s_t)}{\pi _{\theta _{old}}(u_t | s_t)}, 1-\epsilon , 1+\epsilon \right) A(s_t, u_t) \right) \right].\) 
