Here, if we use only one network for the policy, the excessive modification occurs during the training stage. Thus, we use the clipping surrogate method as follows [1]}:
\(\begin{split}\mathcal {L}^{\operatornamewithlimits{\mathrm {clip}}} (s, a; \theta _{\pi }) = \mathbb {E} \Bigg [ & \min \Big (p^t_\theta A^\pi (s, a), \\& \operatornamewithlimits{\mathrm {clip}}(p^t_\theta , 1-\epsilon , 1+\epsilon )A^\pi (s, a)\Big ) \Bigg ],\end{split}\) 
