The accurate calculation of empirical expectation in Eq. REF  requires large number of samples. Furthermore, the performance of policy gradient methods is highly sensitive to the learning rate leading to difficulty in obtaining stable and steady improvement. The PPO algorithm introduces a clipped surrogate objective function [1]} to avoid excessive update in policy parameters in a simplified way as follows
\(J^{{\rm clip}}(w) = \mathbb {E}\big [ {\rm min}(r_t(w) {A}^{\pi _w}(s,a), {\rm clip} (r_t(w),1-\epsilon ,1+\epsilon ) {A}^{{\pi _w}}(s,a) ) \big ], \) 
